AGENTS.md — df-product-db (Damn Filters Product Database)

Pipeline is Python-stdlib-only (3.10+), SQLite-backed product-data project for the Damn Filters Shopify catalog. Builds a normalized DB from Matrixify exports and classifies products into dimensional "classes" (seeds for DF house part numbers). Optional read-only / dev tools (ruff, duckdb, openpyxl) are declared in requirements-optional.txt — the pipeline itself stays stdlib-only.

A class = same OD × ID × length × shape × media × top endcap × bottom endcap × build configuration (8 attributes), matched at nominal dimension. Media is part of the class, not a variant suffix. Dimensions, both endcaps, and build are product-level; media is the only variant-level attribute — non-primary variants need a media-specific class ID, never "inherit". The old "gate" term is retired — say "endcap". The dimension-merge allowance is the tolerance window — never "band" (a material dust-collector construction term) or "slack" (the DF comms app).

Reserved words

One meaning each — keep them out of other contexts so they don't confuse:

• "anchor" — an Anchor SKU only: a competitive loss-leader, the top-tier product that gets the most attention, budget, and ad-performance care. Never in a dimensional-classification context ("anchor products", "this SKU anchors the class" are wrong) — the classification's reference product is the "class reference product" (the modal nominal member of the class), never an "anchor".
• "bands" — the straps/bindings used in filter construction; nothing else.
• "grade" — house grade (a–f) or natural grade (A–F) only; always name which, never "grade" unqualified.

Known traps

• OD == ID is a parse artifact, not a real filter — a signature keyed on equal diameters lumps genuinely different filters into one class. Verify OD > ID before trusting a signature.
• Placeholder SKUs carry the real part number in the title. Some SKUs are placeholder strings (generated, not real OEM part numbers). For those, the OEM part number is in the title — read the title, not the SKU.
• Variant SKUs are supposed to be unique (Shopify requires it) — edit_db.csv has a few defects. A duplicated or blank SKU is a data defect, not the norm (a handful of duplicate SKUs and blank-SKU rows exist today; the data.duplicate_primary_sku WARN invariant flags primary-row duplicates). Because of these, join by Variant ID, never a dict keyed on SKU.
• Physical measurement beats scrape beats body-copy summary line — but it's rare and fallible. A warehouse measurement of the actual bin item is the top dims authority when you have one. But measuring filters is not common practice (only occasionally requested, from warehouse or vendor), and any single measurement carries human-error risk — verify, don't trust one reading blindly. A competitor scrape flagged "may mirror DF" (circular) must NOT be applied without physical/DFI/OEM corroboration. A body's "Dimensions: a x b x L" summary line can contradict its own labeled OD/ID fields — the labeled fields + physical bin win, not the summary line (22111's summary said 10.37 while the labeled ID and the bin were 9.48; the "correction" to 10.37 was reverted 2026-09-04).

Read first

• docs/domain-model.md — canonical business rules (read before touching data)
• docs/column-map.md — authoritative 0-indexed column offsets
• docs/grade-pricing.md — grade × anchor combos + regrade/reprice rules (read before touching grades/prices)
• docs/open-questions.md — live backlog (do not resolve without Viktor/Rachel)
• README.md — quickstart
• CHANGELOG.md — append-only dated pass log

Names. "Canon" = this repo/pipeline (df-product-db). "DFI" and "EFI" are the same vendor-confidential reference dataset (dfi/, legacy name EFI — the files still say efi_*); it is authoritative — among the highest-authority sources we have, much more authoritative than our own data. We trust it.

All docs/ (one line each): domain-model.md (canonical rules) · column-map.md (offsets) · grade-pricing.md (grades + pricing) · media-taxonomy.md (media catalog) · open-questions.md (live backlog) · metafield-definitions.md (Shopify metafields) · final-export-fields.md (final-export plan) · n8n-metafield-contract.md (n8n order MANIFEST) · donaldson-media-mapping.md (Donaldson substrate mapping) · duckdb-analysis.md (read-only SQL) · architecture.md (schema + model) · HANDOFF-2026-09-02-competitor-study.md (competitor-study runbook) · HANDOFF-2026-09-05-dims-drift-codex.md (external dims-drift handoff).

Counts are DERIVED — never written in prose

Class counts, in-stock counts, and the dcf-default backlog are derived — run python3 src/classification_report.py and python3 src/invariants.py for the current numbers. If a number must be cited, write date + source (e.g. "N,NNN classes (report YYYY-MM-DD)"). Enforced by docs.no_hardcoded_counts in src/invariants.py.

archive/ is historical — do not cite

archive/ (incl. archive/hermes-skill-notes/) holds superseded docs and old pass notes. It is a record, not a source: never import rules, counts, or terminology from it into live docs or code. Current truth = docs/domain-model.md (rules) + CHANGELOG.md (decisions) + imports/class-overrides.csv (data).

Locked rules (Viktor — do not override)

• Class = OD × ID × length × shape × media × top endcap × bottom endcap × build. Media is part of the class; the bottom endcap is 3-state (open / closed / closed+bolt); the top endcap is always open; build is class-defining (construction bands-vs-cage, prewrap, stainless, hardware).
• Bottom endcap ALWAYS a distinct class — no exceptions. open / closed / closed+bolt are three part numbers, three classes, 100% of the time. In-stock requires config match too (both endcaps + build match the bin, not just class + media).
• Flange is a top-endcap feature and always splits a class. Ovals never have flanges ("flange" on an oval = noise). Gasket is a top-endcap feature and only matters on flanged filters (square vs round gasket further distinguishes).
• Shape family = round / oval / conical ("panel" is non-cartridge, not a shape). For ovals the LARGER cross-section number is the OD.
• Tolerance window (A2, 2026-09-01) — tolerance(d) = clamp(1/16″ (0.0625″), 0.005·d, 1/4″ (0.25″)) with a shared budget (per-axis ratio ≤ 1.35, sum ≤ 2.0). The fixed 1/16″ rule survives as the floor; longer dimensions (length) get proportionally more leeway. Applies to round, oval, conical.
• bdf.Length is authoritative for cartridges; bdf.Height is not.
• x grade = non-primary ("upgrade") variant; an in-stock class is a, never x.
• Bin 10 out of stock; bins 1–9 in stock.
• DFI is AUTHORITATIVE (Viktor 2026-09-07) — among the highest-authority sources we have, much more authoritative than our own data; we trust it. It has quirks (a few dims differ from ours — probably right, to be checked), but for the most part DFI data is authoritative, no question. The DFI cross-references are already in the DFI data: each EFI part line carries manufacturer + part number (Donaldson P…, Camfil …, Clark …, Nordson B…), and every cross-reference on one EFI line is one class — a gold mine for verifying SKUs and their direct cross-references. Parse manufacturer+PN → EFI_PN → class and join our SKUs to it. The ~109 dims-disagree rows are checked, not auto-applied.
• Media matching is EXACT — no cross-media collapsing. The _DAMN2/_DAMN3 suffix is a DF-internal phantom-variant marker — never in class IDs or DF part numbers.
• Classes are CARTRIDGE-ONLY (non-cartridge → no class, no dcf-default, blank). Grades are primarily cartridge-scoped.
• Blank bdf.Shape = Round (APPLIED 2026-09-01 — 403 backfilled), unless "conical"/"oval" in title/body — and any contrary signal flags for review (D10), not auto-round.
• SB-only (spun-bond primary) — A1, 2026-09-01. A product whose body-copy spec states "media … is Spun Bond" is a spunbond product → sells ONLY the spunbond media (no 80/20/NF upgrade ladder), same discipline as exotics. Handling: a number of spunbond SKUs are exempt (kept); for the rest, DELETE the 80/20/NF variants once the spunbond primary status is verified OR the OEM official spec confirms spunbond. Two guards: never delete an in-stock variant, and never delete a variant whose natural grade is A/B or outranks the SB row it replaces.
• Exotics: exempt a few, delete the rest. Exotic-media products sell only the exotic media (no standard 80/20/NF/SB ladder). A small number are exempt — high-order (A/C) "downgrade exemptions" kept as a standard ladder for legitimate customer downgrades (logged in sku_status_notes as "Downgrade exemption … revisit"). The rest have their standard variants DELETEd, especially no-orders and natural D/F grades.
• Spunbond (_DAMN3-SB) not currently stocked (drop-ship) — a temporary business state, not a rule. There are no rules about what media can or cannot be stocked; what to stock is an ownership/management decision.
• Oval in stock = cls-NR015 only. stock-id #10 out of stock (its old class cls-0453 merged into cls-0126 by the 2026-09-01 merge sweep).
• Grade model (2026-09-01) = house grade (a–f) × natural grade (A–F) × anchor (*). Combo premiums apply AFTER a catalog-wide +3% on cartridges (in-stock bin pricing excluded). "Should not exist" combos are misgrades to correct: dA/fA → a (in stock) / b (not), dB/fB → b, fC → c. Base price is two-track (~65% margin ⇒ cost/0.35) — see docs/grade-pricing.md.
• Media taxonomy (2026-09-01): 3 standard media — 80/20 / NF / SB — plus isolated exotics + wide-pleat nanofiber (wide-pleat-nf). 80/20 FR and 100% Cellulose collapse to 80/20 (leave FR language in body HTML). Wide-pleat ("Nanofiber on Spunbond Polyester" / "Wide Pleat Nanofiber*") is a distinct media — own class, bands (spunbond-substrate) construction, isolated like an exotic but NOT a pure exotic (market the wide-pleat aspect). Derivation (INTENDED, Viktor 2026-09-07): media = Option1 Value label, or the bdf value when available — the SKU suffix is never a media label. External resolution order: OEM/manufacturer (Donaldson or other) → DFI → Clark → competitor listings. NOT IMPLEMENTED — canonical_media() still falls to the SKU suffix for standard labels; see docs/open-questions.md §2.1.
• SKU media indicators are DIAGNOSTIC, never classification input (Viktor 2026-09-05). A bare SKU (no suffix) is, in almost all cases, the ORIGINAL OEM marketing SKU (the ad-buying SKU) — being bare does NOT mean 80/20. The three-media expansion ladder (_DAMN2/_DAMN3) is a PHANTOM SKU (a generated upgrade row), with exceptions (P191920: entered as NF, the 80/20 was removed after complaints, NF/SB kept). -orig = an open-box return resold at OEM price (vanity, not a real selling item). Media written into the SKU name (NF prefix like Clark's NF20xxx series; -NF/-SB/-N101 suffix) is industry part-numbering — either an OEM's own naming or a reseller "slapping a new media on an old part number" — and is UNRELIABLE. These indicators must NEVER assign media. Use them only to FLAG problems or CONFIRM accuracy (e.g. check AF554-NF against bare AF554), resolved via OEM/competitor/DFI lookup, not from the SKU.
• bdf.* (cols 72–87, incl. bdf.2nd_Descr at 72) DO NOT EDIT OR DELETE — Carl's customer-facing config claim. Exceptions (log EVERY change in imports/class-overrides.csv): (1) supervised geometry corrections (transposed OD/ID → OD > ID); (2) bottom-endcap backfill (bdf.Bottom/bdf.Bolt_Hole) from explicit body text — or blank → closed per A4 (Viktor 2026-09-01); (3) dims backfill from labeled "Outside/Inside Diameter:"/"Length:"; (4) shape backfill (blank bdf.Shape → Round per A5, Viktor 2026-09-01).

Data files

• imports/edit_db.csv — working file. TWO header rows + Top Row relocated to col 61 — never parse naively (see docs/column-map.md). CRLF; embedded bare LF in quoted Body HTML.
• imports/Matrixify_Full.csv — pristine export (read-only reference).
• imports/class-overrides.csv — durable supervised-correction log (re-applied by src/apply_overrides.py).

Pitfalls

• python3, not python.
• One commit per pass, not per script — edit_db.csv is large and each commit that rewrites it adds a full copy to .git history; commit once when a pass is done, not after every intermediate script.
• One writer at a time. Only one agent runs edit_db.csv passes. Before starting a pass, confirm the tree is clean — a dirty edit_db.csv means another pass is in flight; stop and coordinate.
• "Not in in-stock-abc-skus.csv" ≠ "out of stock" — that file lists catalog SKUs; a physical STOCK <dims> bin is tracked via sku_stock_id.
• Product-level vs variant-level: dims, bottom endcap, flange, and build live on the PRIMARY row (Db.product_row()); non-primary variant rows carry empty Body HTML. Only media is variant-level.
• Dated output paths must derive from datetime.now(), never hardcoded — a hardcoded dated filename silently overwrites a committed dated artifact.
• str.splitlines() splits on embedded bare LF in quoted Body HTML → false diffs; compare via csv.reader field-by-field, not string diff.