If a page renders content server-side, you don't need Selenium. You need: - requests / httpx — fetch HTML - BeautifulSoup / lxml — parse it - pandas / json — store it This is 80% of all scraping you'll ever do. Faster, cheaper, more reliable than launching a browser. Reach for browsers (tomorrow's topic) only when JS is required for content to appear.
Three rules I follow on every scraper: 1. Respect robots.txt — read it; honour Disallow. 2. Rate limit — sleep 0.5-2s between requests; back off on errors. 3. Identify yourself — User-Agent: 'YourBot/1.0 (contact: you@x.com)'. Sites can block, but they can also reach you. For public data this keeps you on the right side of TOS. For paid APIs / private data, get permission first. The 'I scraped LinkedIn' story rarely ends well.
Reusable. Sets a User-Agent, retries with backoff, parses with BS4, returns dicts. Drop in URLs and a parse_fn. Most one-off scrapes are 50 lines on top of this.
Don't re-hit a server every time you fix a parser bug. Cache responses locally: - requests-cache library — drop-in cache for requests - Or write to disk by hash(url) Faster iteration. Kinder to the site. Survives flaky networks. Once the parser is right, run live. Set TTL based on how stale data is acceptable — minutes for prices, hours for product info, days for static content.
Day 78 done. - HTTP + parse covers most cases - Rate limit, robots, UA - Polite scraper template - Cache while developing Tomorrow (Day 79): Playwright. When JS rendering is required.