Data Sources
Canon is built from a handful of independent sources, each with a distinct job. None of them is authoritative on everything — the product database is where they converge and get reconciled.
flowchart TD
subgraph ID["Identity — catalog & provenance"]
MX["Matrixify export"]
ORR["Orrstein files"]
end
subgraph SPEC["Specs — OEM & competitor"]
DN["Donaldson catalog"]
CL["Clark cross-ref"]
SC["Scrape cache"]
OEM["OEM PDFs"]
end
subgraph VAL["Validation"]
DFI["DFI reference (legacy EFI)"]
end
subgraph OPS["Commercial — grade · price · stock"]
NG["Natural grade"]
PO["PO / vendor cost"]
BIN["In-stock bins"]
end
ID ~~~ SPEC
SPEC ~~~ VAL
VAL ~~~ OPS
ID --> DB
SPEC --> DB
VAL --> DB
OPS --> DB
DB["Product database"] --> OUT["Shopify export"]
DB --> N8N["n8n PO automation"]
Catalog base
- Matrixify export (
imports/Matrixify_Full.csv→imports/edit_db.csv) — the Shopify catalog: every product, variant, price, and Body HTML. The base layer the rest of the pipeline operates on. - Orrstein files (
imports/orrstien-files/) — the original spreadsheets Jim Orrstein used to build the catalog. Two workbooks: the original Shopify catalog (declares each SKU's original media) and the director spec/price directory. They split the catalog into original / director / later-add and are the provenance layer — the answer to "was this SKU always here, and what did it originally say?"
Specs — OEM & competitor
- Donaldson catalog (
data/donaldson_catalog.db) — a 136-page Donaldson Torit catalog ingested to a local DB: part number, media, area, pleat, and OD. - Clark cross-reference (
data/clark_crossref.db) — Clark Filter's DC cross-reference: OEM part number → Clark part number with full OD × height × ID and endcap. Provenance-confirmed — our bdf dims match it at OD 92% / ID 92% / Length 98% (measured 2026-09-06, Clark cross-reference provenance). - Scrape cache (
data/scrape_cache.db) — a growing, deduplicated collection of scraped OEM and competitor pages: dimensions, media specs, and pricing. Sourced through a 5-tier hierarchy (Tier-0 OEM down to secondary retailers). - OEM PDF specs — the wider OEM landscape (Camfil, AAF, Nordson, RoboVent, Imperial, Mikropor, …). Roughly 45% of primary cartridges have a true OEM spec source; the rest are reseller-consensus. (The 45% figure is an unverified estimate — derivation and denominator to be documented.)
Validation
- DFI reference (
dfi/, legacy name EFI — the files still sayefi_*) — vendor-confidential reference data: cost dataset, part-number cross-references, and media-grade lookups. Authoritative (Viktor 2026-09-07) — among the highest-authority sources we have, more authoritative than our own data. Each EFI part line carries manufacturer + part number (Donaldson P…, Camfil …, Clark …, Nordson B…), and every cross-reference on one EFI line is one class — the join key for verifying our SKUs and their direct cross-references. Its quirks (a few dims differ from ours, probably right) are checked, not auto-applied.
Commercial — grade, price, stock
- Natural grade (
imports/natural_grade.csv) — pageviews and orders from GA4 + Shopify, distilled into the A–F popularity grade. - PO / vendor cost — purchase-order-asserted unit costs, matched to vendor records; drives
vendor.cost/vendor.preferred. - In-stock bins — the physical "STOCK" bins, tracked by
sku_stock_idwith unit counts in QuickBooks; determines what's actually in stock (bins 1–9).
Our own corrections
Beyond the external sources, every supervised decision is logged — imports/class-overrides.csv and the dated pass CSVs — so a correction is never silent and can always be re-audited.