Damn Filters Product Database

A product data management system for the Damn Filters Shopify catalog. Builds a normalized SQLite DB from Matrixify exports, classifies products into dimensional "classes" (seeds for DF house part numbers), and grades them (house a–f × natural A–F) for Google Shopping bids. The pipeline (import, invariants, classification, pass scripts) is Python-stdlib-only (3.10+); optional read-only analysis tools are declared in requirements-optional.txt.

Quickstart

python3 src/cli.py import       # build data/products.db from imports/Matrixify_Full.csv
python3 src/cli.py query stats  # confirm it worked

The import takes ~3s and produces a ~48 MB data/products.db (gitignored — built locally from the CSV).

Current state (derived — never trust a prose count)

Run these for the live numbers:

python3 src/classification_report.py   # classes, in-stock, dcf-default
python3 src/invariants.py              # invariant checks (0 FAIL = clean)

Layout

imports/     source CSVs (edit_db.csv = working file; Matrixify_Full.csv = pristine;
             class-overrides.csv = supervised-correction log)
docs/        domain-model.md (canonical rules), column-map.md, open-questions.md,
             grade-pricing.md
src/         pipeline modules + tools/ + archive/ (one-shot passes)
generated/   output CSVs (gitignored)
data/        products.db (gitignored)

Domain reference

• docs/domain-model.md — canonical business rules (read first)
• docs/column-map.md — 0-indexed column offsets
• docs/grade-pricing.md — house × natural × anchor grades + pricing
• docs/open-questions.md — live backlog

Optional tools

Not required for the pipeline; see requirements-optional.txt.

• openpyxl — src/provenance_join.py reads the Orrstein source workbooks (imports/orrstien-files/, local-only).
• duckdb — read-only SQL analysis over edit_db.csv and the SQLite reference DBs (docs/duckdb-analysis.md has the verified setup). Never writes edit_db.csv.
• ruff — dev-only linter (config ruff.toml; CI runs ruff check). Catches unused imports, shadowed names, and undefined names. Never imported at runtime.

Refreshing the data

1. Back up imports/edit_db.csv, then replace imports/Matrixify_Full.csv with the new export.
2. Merge the new export INTO edit_db.csv — this is a supervised merge, not a copy (reference impl: src/tools/merge_matrixify_export.py). edit_db.csv carries the project's classification columns (61–67 and 104–149), supervised corrections, and the two-header + relocated-Top Row layout the pristine export does not. Join on Variant ID (never Variant SKU — duplicates + blanks), patch shared rows in place, and preserve the project-owned cells — the bdf.* block (72–90, sculpted dims + endcap backfills), Variant Cost (49, PO-cost backfill), and the Google Shopping pass-through block (63–65 product grade/group/class, 68–69 variant class/group, 70–71 cleared custom_label_3 + custom_product, 91–93 spare/anchor/variant grade) — and the whole appended block 104–149 must NOT be overwritten from the store copy (66/67 are empty TEMP columns). See docs/column-map.md for the authoritative offsets; verify the width against docs.column_count_matches and re-run python3 src/invariants.py (0 FAIL) before committing.
3. git add + commit + push.
4. Each install runs git pull then python3 src/cli.py import.

Private

Proprietary Damn Filters product catalog — keep private.