Web scraping is the automated extraction of structured observations from web pages. A quant might collect advertised prices, product availability or job postings as alternative data. The important output is a dated, interpretable dataset: a page that downloads successfully can still produce duplicate offers, mismatched units or information that was unavailable at a backtest's decision time.
Scraping is a collection method, not a trading signal. An advertised price is not a completed sale; a job advertisement is not a hire. Turning either into a useful forecast requires a separate measurement argument and validation.
From a response to an observation
A small collection pipeline has distinct steps:
- Fetch: request an authorised page over HTTP, checking the response status, final URL, content type and encoding. Use finite timeouts and the service's documented collection limits. Python's
urllib.requestdocumentation describes the response object, headers, status and timeout parameter. - Parse: interpret the returned HTML structure. A parser exposes tags, attributes and text; it does not know which number represents the desired economic quantity. Python's
HTMLParsercalls handlers for these elements. Downloaded HTML may omit values inserted later by browser scripts. - Extract and validate: map selected elements into named fields, then check identifiers, numerical types, currencies, units, timestamps and missing values. Retain the response snapshot where permitted, a content hash and the parser version so a surprising observation can be traced back.
- Publish a usable dataset: apply declared identity, revision and availability rules before aggregating observations into a feature.
An API provides a machine-oriented interface, commonly returning structured records such as JSON. HTML scraping infers structure from a presentation intended for readers. Prefer a suitable documented API or authorised download when available: it can provide explicit field meanings and versioning, although its schema, limits and historical coverage still need checking. The APIs lesson develops response validation with an offline fixture; its public preview is available and the full lesson requires membership.
Three returned rows, two offers
Consider an original fictional catalogue snapshot. All prices are USD per item, for the same declared pack-size convention, and the goal is an equally weighted mean across distinct offers. Two collection requests encounter the same offer on overlapping pages; a third encounters another offer. Both copies of Q1 represent the same quote version.
| Retrieved at, UTC | Quote identifier | Advertised price, USD/item |
|---|---|---|
| 10:01 | Q1 | 20 |
| 10:02 | Q1 | 20 |
| 10:03 | Q2 | 30 |
The row mean is USD/item. It gives Q1 twice the weight of Q2. After removing the repeated quote, the intended mean is USD/item. This is a mean of two offers, not a sales-weighted price or a consumer price index.
Original fictional snapshot. Repeated retrieval is not an additional offer; the identity rule determines the weights in the aggregate.
The following offline Python example extracts attributes from deliberately simple authored HTML. It makes no network requests. The attributes supply the schema; this parser is not a general website collector.
from html.parser import HTMLParser
from statistics import mean
class OfferParser(HTMLParser):
def __init__(self):
super().__init__()
self.rows = []
def handle_starttag(self, tag, attrs):
if tag == "offer":
fields = dict(attrs)
if fields["currency"] != "USD":
raise ValueError("expected USD")
self.rows.append((fields["quote-id"], int(fields["price"])))
html = '''<offer quote-id="Q1" price="20" currency="USD"></offer>
<offer quote-id="Q1" price="20" currency="USD"></offer>
<offer quote-id="Q2" price="30" currency="USD"></offer>'''
parser = OfferParser()
parser.feed(html)
parser.close()
unique = {}
for quote_id, price in parser.rows:
if quote_id in unique and unique[quote_id] != price:
raise ValueError("conflicting quote version")
unique[quote_id] = price
print(len(parser.rows), len(unique))
print(f"{mean(p for _, p in parser.rows):.2f} {mean(unique.values()):.2f}")
# 3 2
# 23.33 25.00Deduplicate by the observation's identity, not by its value: two different offers can legitimately share a price. Real keys may combine seller, product, location, quote time and version. A new price for Q1 tomorrow is a new observation; a conflicting price for the same version should trigger investigation rather than silently keeping the last row. Preserve the raw retrieval ledger even when the analytical table removes duplicates.
The page date cannot supply the availability clock
Suppose every offer above displays a quote time of 10:00, but collection finishes at 10:03 and validation finishes at 10:04 UTC. Under a rule requiring the complete validated snapshot, the two-offer mean becomes usable at 10:04. A 10:02 decision cannot consume it. A policy using partial snapshots would need its own explicit universe and cutoff rules.
For backtesting, retain separate fields for the period or event measured, any stated publication time, retrieval time and the time processing makes the record usable. A retrieved page may already contain revisions to an old date. Today's version cannot establish yesterday's contents: that substitution creates look-ahead bias. The St. Louis Fed's real-time-period documentation illustrates the broader distinction between today's view of history and information known in an earlier period; it does not timestamp a scraper's own receipt.
The Observation, Event and Availability Timestamps lesson develops this ordering through decisions, orders and fills. Its public preview is available; the full teaching requires membership.
A stable selector can still collect a different quantity
A redesign may move the price into a different element, producing missing values. A subtler schema drift changes meaning while keeping the element name: the price becomes a subscription instalment, switches currency, excludes tax or refers to a smaller pack. Monitor missingness and field structure, but also enforce economic units and inspect changed response samples. Do not convert a missing price to zero or assume a surviving number has the old meaning.
Coverage matters too. Pagination, regional settings, out-of-stock omissions and personalised displays can change the observed universe. A fall in mean advertised prices might reflect a different product mix rather than price reductions. Define the target population and track inclusion rules to assess selection bias.
Collection rules and access rights are separate checks
RFC 9309, published in September 2022, specifies the Robots Exclusion Protocol: crawler rules communicated through
robots.txt. Its introduction explicitly states that those rules are not access authorisation, and its security section distinguishes them from access controls. Respect the applicable crawler rules while separately establishing the permitted access and use, including relevant terms, data licences and privacy requirements. Public visibility alone does not settle those questions.Use an authorised collection route, identify the collector appropriately, honour service limits and stop to resolve blocked or challenged access. Do not bypass authentication or anti-bot controls. The permissions and obligations for an actual dataset depend on its source, use and jurisdiction; a generic scraper cannot determine them from
robots.txt.Sources
- Koster, Illyes, Zeller and Sassman, RFC 9309: Robots Exclusion Protocol, September 2022, §§1, 2.2–2.4 and 3.
- Python documentation,
urllib.request: opening URLs,urlopenresponse and timeout documentation. - Python documentation,
html.parser: simple HTML and XHTML parser, class overview, methods and parser example. - Federal Reserve Bank of St. Louis, FRED API real-time periods, introduction.