Keep the differences in the database

I got into FPV drones around 2018 — YouTube videos first, for rather longer than I would like to admit, then a cart full of parts, then a first build that actually flew.

Then it crashed. They all crash; that is most of the hobby. You clip a gate, misjudge a dive, lose the video feed behind a tree, and something breaks. So the hobby is not really about flying. It is about keeping a small fleet of fragile things repaired — an arm, a motor, an ESC, a stack, another set of props.

There is no Amazon for any of that. The parts come from dozens of small independent shops spread across different countries, each with its own catalog, its own currency, its own idea of how to write “in stock,” and no API. So every repair started the same way: a dozen browser tabs open at once, working out who actually had the part, what it cost, and whether they would ship it to me. That is a tedious question to answer by hand more than about twice, and I answered it by hand for years.

Five years later I finally built the thing that answers it. The backend read a couple of dozen of those shops on a schedule and normalized what came back into one searchable catalog with stock and prices attached. That is the entire product, and it is also the entire problem: every source is different, every source can change without telling you, and you control none of them.

Two decisions carried it. Neither is clever. Both are the kind of thing that is obvious in hindsight and easy to get wrong on the first pass, because the first instinct in both cases points the other way.

The first instinct is one class per store

The natural shape for “twenty sites, twenty layouts” is a base class and twenty subclasses. Each one knows how to find a price on its site. It reads well, it demos well, and it means every new shop is a code change, a review and a deploy. After the tenth one you are maintaining twenty files that differ only in string literals.

The alternative is to notice that the sites vary in values, not in behavior. Every one of them has a product name somewhere in the markup, and a price, and usually an image and a stock indicator. What differs is which tag and which class those live in. That is data, so it went in the database:

class Store(ScrapableItem, StoreBase, Base, table=True):
    ...
    # each Field() also carries an sa_column_kwargs comment,
    # elided here for width
    product_name_class: str = Field(nullable=False)
    product_name_css_is_class: bool = Field(default=True, nullable=False)
    product_name_tag: str = Field(nullable=False)
    product_price_class: str = Field(nullable=False)
    product_price_css_is_class: bool = Field(default=True, nullable=False)
    product_price_tag: str = Field(nullable=False)
    # ... then the same three columns again for image, thumbnail,
    # availability, variations and description

And the scraper stopped knowing anything about any particular shop:

for field in fields:
    style_class = getattr(self.store, f"product_{field}_class")
    html_tag = getattr(self.store, f"product_{field}_tag")
    selector = (
        "class"
        if getattr(self.store, f"product_{field}_css_is_class")
        else "id"
    )

    if not bool(style_class) or not bool(html_tag):
        continue

    soup_obj = soup.find(html_tag, {selector: style_class})

Three getattr calls against a naming convention. That is the whole mechanism. There is one scraper class for every store on the platform, and adding a shop is a row in a table, entered through the admin, live immediately. No deploy, no review, no new file.

I want to be honest about what this costs, because a wide table of about twenty-five nullable configuration columns is a schema most reviewers would call a smell, and they would not be wrong in general. The reason it is the right smell here is that the variation is unbounded in count and tightly bounded in kind. There will always be more shops. There will not be a shop that needs a fourth axis of variation beyond tag, selector and whether the selector is a class or an id. When those two things are true, a configuration table beats a class hierarchy. When the second one stops being true — when some store needs three selectors chained, or a price computed from two elements — the table is the wrong tool and you are about to start encoding a language in columns. I never hit that, but it was always where the design would break first.

Two of those columns are worth pausing on

Most of the store columns are mechanical. Two encode decisions.

Locale, with a comment I am glad I wrote down at the time:

locale: Locale = Field(
    sa_column=Column(
        Enum(Locale),
        comment="If the store uses , as decimal separator choose it_IT",
    ),
)

A German shop writes 1.234,56. A US shop writes 1,234.56. These are the same number and they are also, to a naive parser, two numbers that differ by roughly a factor of a thousand. Nothing crashes. There is no exception, no null, no alert — you just quietly start telling people a €1,234 motor costs €1.23, and the only symptom is that it ranks first on a price comparison page. Recording the store’s locale and handing the parsing to locale.atof is not sophisticated, but it moves the failure from silent-and-wrong to loud-and-early, which is the only move that matters.

Whether the store needs a browser, which is also a column:

scrape_with_js: bool = Field(default=False, nullable=False)

The instinct here is to make it a runtime fallback: try a plain HTTP request, and if it comes back without the price, escalate to a headless browser. That is what I would have written first, and it is worse. A store that renders its catalog client-side needs the browser on every page, so a fallback design pays the full cost of a failed cheap attempt every single time, forever, in exchange for never having to know anything. Recording the answer once per store means the common case stays a single HTTP request and the expensive path is taken deliberately.

The cost of that choice is a human has to set the flag, and a store that switches to client-side rendering next year will break until somebody flips it. That is a real cost, and the only reason it is acceptable is the machinery in the next section, which notices.

Decide what each kind of breakage means before you meet it

This is the half that actually kept the thing alive, and it is the half I would argue for hardest in a system that talks to anything it does not own.

A scraper against sites you control is a parsing problem. A scraper against sites you do not control is a classification problem: something will come back wrong every single day, and almost all of the engineering is in deciding, in advance, which wrongness means what. There were three levels.

At the field level, missing is normal and cheap:

if not soup_obj:
    logger.warning(f"Nothing found when searching for {field}!")
    continue

A shop that redesigns its description block should not cost me its whole catalog. A product with no description is still a product.

Name and price are different, because a product without a price is not a product, it is a row that will make the search results wrong. Those raise:

if not data.get("price"):
    raise ProductPriceNotFound(
        f"Cannot create product without price: {url}; {data=}"
    )

Which brings the interesting bit, the function that decides what a raised exception actually means:

async def scrape_or_deactivate(
    db: AsyncSession, scraper: StoreScraper, url: str, fields: List
) -> Optional[Product]:
    try:
        return await scraper.scrape(url, fields)
    except (URLNotFound, TimeoutError) as e:
        logger.warning(f"DEACTIVATING PRODUCT! {e}")
        await ProductManager.deactivate(db, product_link=url)
        return
    except (ProductNameNotFound, ProductPriceNotFound) as e:
        return
    except Exception as e:
        msg = f"Unexpected error when creating or updating product {url}: {e}"
        await send_log_to_telegram(msg, "error")
        return

Four outcomes, and the reason I like this function is that it refuses to collapse them:

• The page is gone. This is not an error, it is news — the shop discontinued the part. The right response is to deactivate the product so it stops appearing in results, and to say nothing to anybody.
• The page is there but we could not read a name or a price. Skip it, silently. One malformed product page out of two hundred is a Tuesday, not an incident.
• Something else happened. We do not know what this is, so a human gets a message.

That third branch is the one worth stealing. Most integrations I have reviewed have two categories, “worked” and “failed”, and they alert on both or neither. The distinction that matters is not success versus failure — it is expected failure versus failure you have never seen before. Only one of those should wake anybody up. The same split shows up in a document pipeline where an LLM had to fail loudly rather than confidently, and the reasoning is identical: silence is only safe for the failures you have already reasoned about.

Letting a store fall out on its own

Skipping individual products handles noise. It does not handle a shop rebuilding its site over a weekend, where every product now fails and the store is quietly rotting. So there is a level above:

stores = await StoreManager.get_stores_with_less_than_n_products(
    db, n=20
)
for store in stores:
    success, error_message = await store_scraper.ping_website()

    if not success:
        store.is_parsable = False
        store.reason_could_not_be_parsed = error_message
        await db.commit()
        continue

Low product count is the symptom. The check separates the shop is unreachable from the shop is up but we cannot read it any more, and writes the verdict onto the store row along with the reason.

Then the part that makes the whole design pay off, which is not in the scraper at all — it is in the relationship between a country and its stores:

stores: List["Store"] = Relationship(
    back_populates="country",
    sa_relationship_kwargs={
        "primaryjoin": "and_(Country.id == Store.country_id, "
                       "Store.is_active.is_(True), "
                       "Store.is_parsable.is_(True))"
    },
)

A store that stopped being readable disappears from every user-facing query, and from the scraping cron, by flipping one boolean. Nobody deploys. The site keeps working with one fewer shop in it, which is a much better product than a site with one shop full of garbage prices. The failure mode of the whole system is fewer results, and that was deliberate: a price comparison that is missing a store is mildly less useful, and a price comparison that shows a wrong price is worse than useless.

Where I got it wrong

Two things, and both are in the store-level check I just praised.

The threshold is a magic number, and it is the same magic number twice. The cron selects stores with fewer than 20 active products, then asks the importer for at most 20 links, then deactivates the store if it created fewer than 20 products:

importer = ProductImporter(db, store=store)
await importer.import_product(limit=20)

if importer.link_processed > 0 and importer.products_created_or_update < 20:
    store.is_parsable = False
    store.reason_could_not_be_parsed = "Cannot import new products"

Since the importer is capped at 20 links, the only way to survive that check is a perfect 20 out of 20. One discontinued item without a price — exactly the case the product-level logic is designed to shrug off — takes the entire store offline. And a genuinely small shop with 15 products in its sitemap can never pass, so it lands in the suspicious bucket every run, forever. The threshold wants to be a ratio against links actually processed, and it is a two-line change I never made.

The diagnosis gets thrown away at the exact moment it becomes useful. Look at what gets written to the row: "Cannot import new products". Meanwhile, thirty lines away, the field-level code logged precisely which selector stopped matching. That is the one fact a human needs to fix the store, and it goes to a log file while the store row — the thing somebody will actually open in the admin when they see the alert — gets a sentence that tells them nothing. Carrying the failed field names up into reason_could_not_be_parsed would have turned a fifteen-minute investigation into a ten-second one.

Both of these are small. Neither was hard. I mention them because the pattern I am recommending is only worth anything if the recovery path is as carefully thought through as the detection path, and in this system it was not.

Being a guest on somebody else’s server

Worth stating plainly, because it shaped several decisions and it is the part of this genre that usually goes unmentioned: these were small shops, and the cost of my traffic landed on them.

The catalog is read from each store’s own sitemap.xml, not by crawling search result pages. That is the surface the shop publishes for machines to read, and it comes with lastmod, so the crawl sorts by it and fetches what changed first. The schema still carries a set of search_url / search_tag / search_class columns, all marked DEPRECATED! — the archaeology of moving off search-page crawling and onto the front door.

Beyond that: work per run is bounded (200 links on an import, not the whole catalog), and retries are bounded (max_tries=3 on an exponential backoff, then give up). The crons are staggered by continent rather than all firing at once — Europe updates at 04:00 and 12:00, America at 06:00 and 14:00, Asia at 08:00 and 16:00, Oceania at 10:00 and 18:00, with imports on alternating weekdays. And is_parsable means a broken store stops being requested at all, which is the politest possible response to a shop that has clearly changed something.

None of that is generosity. A shop that blocks you is a shop that is gone from your product.

What actually transfers

Strip the scraping and this is a vendor integration problem, which is a much more common thing to be holding. Any time you are reading from many sources you do not control — regional partner APIs, customer SFTP drops, a dozen CRMs with the same three concepts under different names — the two decisions are the same:

Describe each source as data, not as code, for as long as the sources vary in values rather than in behavior. Know where that stops being true, because it does.

Classify the failures before you meet them. Not “did it work,” but: is this thing gone, is this one record malformed, is this whole source broken, or is this something I have never seen? Those want four different responses, and the last one is the only one that should page anybody.

The project shipped and ran on a cron from June 2023 to October 2024, when I stopped maintaining it. Plenty of it I would write differently now — the thresholds above, for a start. These two decisions are the ones I would make the same way again.