Compare commits

..

70 Commits

Author SHA1 Message Date
ben.stull e2d8b5caae Merge pull request 'SLICE-8: Shopify dialect — detect/map Shopify export at the codec boundary + DOC-2/3 (SD-0002 §7.2)' (#32) from slice-8-shopify-dialect into main
ci / check (push) Has been cancelled
2026-06-12 09:37:37 +00:00
ben.stull a851d3587c fix(products): conservative dialect detection — canonical-distinctive columns veto Shopify (SLICE-8 review)
ci / check (push) Has been cancelled
ci / check (pull_request) Has been cancelled
Addresses adversarial-review findings: a canonical file carrying a stray
Shopify-signature name (e.g. SEO Title) no longer misdetects as Shopify and
silently drops canonical Type / corrupts the weight unit (kg->g). A
canonical-distinctive column (Description/Variant Cost/Variant Weight/Google
Product Category/Variant Volume/Tax ID/Position) now vetoes detection, so
detection leans conservative (under-detection warns; over-detection corrupts).
Also closes the dual-named-column shadow (Body (HTML)+Description) and the
stale-weight-unit-on-clear case. Tests cover all three.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-12 02:36:09 -07:00
ben.stull cb6b5996f3 chore(products): SLICE-8 complete — bump v0.8.0; §12 traceability audited (SD-0002)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-12 02:28:18 -07:00
ben.stull 17f3b600fd test(e2e): e2e_import_shopify_dialect — Shopify export imports directly (PUC-6, SLICE-8)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-12 02:26:44 -07:00
ben.stull 6459e34480 docs(products): DOC-4 dialect-adapter notes (SLICE-8)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-12 02:24:44 -07:00
ben.stull 60a06de302 docs(products): DOC-2 column reference served at /api/products/columns.md (SLICE-8)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-12 02:24:05 -07:00
ben.stull 0aeccf7d2c feat(products-ui): Shopify 'mapped' label + format help links to column reference (PUC-6/PUC-11, SLICE-8)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-12 02:22:32 -07:00
ben.stull a00f5baaec test(products): INV-10 holds over a partial Shopify import (SLICE-8)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-12 02:21:34 -07:00
ben.stull f170950aec test(products): exhaustive Shopify->canonical mapping fixture (SD-0002 §6.5.1, SLICE-8)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-12 02:21:05 -07:00
ben.stull 0c7a76a305 feat(products): codec detects + maps Shopify dialect at the boundary (INV-17, SLICE-8)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-12 02:20:21 -07:00
ben.stull daa237dd59 feat(products): Shopify dialect adapter — header detection + canonical mapping (SD-0002 §6.5.1, SLICE-8)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-12 02:19:15 -07:00
ben.stull e16ec7cf94 docs(plan): SLICE-8 Shopify dialect implementation plan (SD-0002 §7.2)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-12 02:17:30 -07:00
ben.stull 8ed9b25771 Merge pull request 'SLICE-7: images pipeline end-to-end — fetch/host/serve + INV-12 over images (SD-0002 §7.2)' (#30) from worktree-slice-7-images into main
ci / check (push) Has been cancelled
2026-06-12 08:00:00 +00:00
ben.stull c929282e07 fix(products): bomb-safe image processing + per-image error isolation; honest claim/resumability docs (SLICE-7 review)
ci / check (push) Has been cancelled
ci / check (pull_request) Has been cancelled
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-12 00:58:10 -07:00
ben.stull 757412ef2a docs(products): image pipeline ops + seams (DOC-1/DOC-4); overlay bucket config; v0.7.0
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-12 00:48:44 -07:00
ben.stull 0775537d4c test(e2e): e2e_image_outcomes — fixture image host, per-image outcomes + notice band (SD-0002 §6.8)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-12 00:41:39 -07:00
ben.stull d81215be1d feat(products-ui): run-detail images section + progress poll, image-problems notice band, history images column (SD-0002 §5.2/§5.5)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-12 00:37:45 -07:00
ben.stull 7f51f5242f feat(products): app-served image route — storefront-authorized, immutable cache (SD-0002 §6.4, INV-16)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-12 00:34:04 -07:00
ben.stull 5d5341b29b feat(products): confirm → fetching_images + post-commit fetch scheduling + startup recovery (SD-0002 §6.5.3/§6.9)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-12 00:30:08 -07:00
ben.stull 23267c4d4c feat(products): image-fetch phase — SSRF guard, bounds, renditions, resume (SD-0002 §6.5.4, §6.9, INV-18)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-12 00:26:26 -07:00
ben.stull 13e74f4c61 feat(products): image-phase repo helpers + run progress/outcomes/counts (SD-0002 §6.5.4)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-12 00:23:09 -07:00
ben.stull 984dc98dee feat(products): diff resolves hosted image URLs to existing records — INV-12 over images (SD-0002 §6.3)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-12 00:19:23 -07:00
ben.stull 906bc87c96 feat(products): export hosted detail URL for fetched images; snapshot carries status (SD-0002 §6.5.5, INV-16)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-12 00:15:47 -07:00
ben.stull da5177c950 feat(products): hosted-image URL build/parse helpers (SD-0002 §6.3.1)
DB-free host-agnostic helpers: image_url() builds the canonical hosted
Image Src for a fetched image; parse_image_id() recognizes one on re-import
via path-parse so exports round-trip across localhost/PPE/rebrand (INV-12).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-12 00:13:27 -07:00
ben.stull 0fc29a34dd feat(platform): objectstore port — local + GCS adapters, config (SD-0002 §6.2)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-12 00:12:24 -07:00
ben.stull f27a24353d feat(platform): images port — decode, resolution bar (Q-3), WebP renditions (SD-0002 §6.2)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-12 00:09:58 -07:00
ben.stull 385df8d728 Merge pull request 'SLICE-6: export & the round-trip lock — canonical serializer + streamed export (SD-0002 §7.2)' (#29) from worktree-slice-6-export-roundtrip into main
ci / check (push) Has been cancelled
2026-06-12 05:23:23 +00:00
ben.stull af72299a57 fix(products): serialize Variant Position explicitly — INV-12 round-trip bug
ci / check (push) Has been cancelled
ci / check (pull_request) Has been cancelled
A stored variant position is a CatalogVariant attribute, not a fields{} entry,
so the serializer emitted an empty Variant Position cell — which re-imports as
'reset to file order'. A non-sequential stored position (a merchant can import
explicit positions) then round-tripped to a spurious update, violating INV-12.
Emit str(variant.position) explicitly (like _write_image). The property-test
generator now assigns non-sequential positions to lock the regression, plus a
targeted test. Also wires isExportEnabled into ProductsPage (was dead code).
Both caught by the final code review.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-11 22:22:11 -07:00
ben.stull 6f22c8b146 chore: version 0.6.0 — SLICE-6 export & round-trip
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-11 22:14:30 -07:00
ben.stull 767011fbae docs(products): DOC-1 export ops + TEL-3, DOC-4 serializer/round-trip (SLICE-6)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-11 22:14:30 -07:00
ben.stull 8077f2e07b test(e2e): e2e_export_download + e2e_roundtrip_noop (SD-0002 §6.8, PUC-9/10)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-11 22:12:53 -07:00
ben.stull 0a85c4fef8 feat(frontend): Export status-filter menu on the Products page (SD-0002 §5.2, PUC-9)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-11 22:08:34 -07:00
ben.stull 6f213e1f02 feat(frontend): export URL helper + status-filter list (PUC-9)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-11 22:07:36 -07:00
ben.stull 155f9bd147 feat(api): GET /api/products/export — streamed canonical CSV, 409 empty_catalog (§6.4, PUC-9)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-11 22:05:55 -07:00
ben.stull 1c2a6af986 feat(products): streamed export service + TEL-3 + EmptyCatalog (PUC-9, §9.1) 2026-06-11 22:03:09 -07:00
ben.stull 5a97b9dc59 feat(products): status-filtered export_catalog snapshot query (PUC-9) 2026-06-11 22:01:11 -07:00
ben.stull fce0b5eaed test(products): decimal/int variant fields round-trip clean (INV-12 coverage) 2026-06-11 21:59:06 -07:00
ben.stull b40e4d30b5 test(products): INV-12 property test — export round-trips to a no-op (SD-0002 §6.8) 2026-06-11 21:58:47 -07:00
ben.stull 7652e92cbe feat(products): serialize multi-variant + interleaved images (§6.5.1 row grammar) 2026-06-11 21:58:22 -07:00
ben.stull 627e257d4c feat(products): canonical serializer skeleton — header + single product row (SD-0002 §6.5.5) 2026-06-11 21:57:48 -07:00
ben.stull 011f4d5dc1 Merge pull request 'SLICE-5: products import spine — canonical CSV → preview → confirm (SD-0002 §7.2)' (#26) from worktree-slice-5-import-spine into main
ci / check (push) Has been cancelled
2026-06-12 03:50:20 +00:00
ben.stull 2606fbf826 docs(products): operator guide (DOC-1) + domain notes (DOC-4); version 0.5.0
ci / check (push) Has been cancelled
ci / check (pull_request) Has been cancelled
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-11 17:00:13 -07:00
ben.stull ad738adbc0 test(e2e): SLICE-5 scenarios — preview/confirm, errors, rejection, cancel (§6.8)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-11 16:53:59 -07:00
ben.stull a58f42cf86 test(e2e): stand up Playwright — fresh-DB server harness + sign-up helper
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-11 16:50:59 -07:00
ben.stull d18a84e135 fix(products-ui): review pass — show-more race, keyboard history access, a11y glyphs/aria-live, label consistency
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-11 16:46:43 -07:00
ben.stull 87a56a66c4 feat(products-ui): import run detail (§5.5, PUC-4/5/8)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-11 16:35:49 -07:00
ben.stull 4141285b89 feat(products-ui): import preview — tiles, drill-in diffs, confirm/cancel (§5.4)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-11 16:34:33 -07:00
ben.stull 188494272a feat(products-ui): import upload screen (§5.3, PUC-2/5a)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-11 16:30:51 -07:00
ben.stull 26f84cb916 feat(products-ui): admin nav + Products page with import history (§5.2)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-11 16:29:20 -07:00
ben.stull b5dac8886f feat(products-ui): typed products API client + admin hash routing
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-11 16:25:37 -07:00
ben.stull 4dacc2dafd feat(products): /api/products/* BFF endpoints + sample.csv (DOC-3)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-11 16:18:34 -07:00
ben.stull a8538d3ecc fix(products): cleared Variant Position resolves to file order, not NULL
A blank Variant Position cell previewed position->null and aborted the
confirm transaction (variant.position is NOT NULL). Resolve the clear to
the variant's file order at diff time so preview and apply stay in
lockstep; carry file_order on VariantPlan instead of an identity-keyed
map; release the read snapshot before PreviewStale/NothingToApply.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-11 16:14:14 -07:00
ben.stull 0c7865e9e1 feat(products): confirm/apply in one transaction, runs, summary — INV-10/11/14 + TEL-2/6
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-11 16:02:27 -07:00
ben.stull fcbf1393f5 feat(products): import_validate → draft, preview records, discard + TEL-1
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-11 15:51:03 -07:00
ben.stull 138126ab17 feat(products): repo — catalog snapshot, draft/run SQL, upsert primitives
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-11 15:42:42 -07:00
ben.stull 667a462e0b fix(products): honest position compare for matched variants in diff
A Variant Position column previously produced a spurious update with a
false before:None (CatalogVariant keeps position as an attribute, not in
fields{}). Compare against the attribute, like images already do.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-11 15:37:59 -07:00
ben.stull 9bc6e4dbd2 feat(products): diff engine — add/update/unchanged/error + INV-11 fingerprint
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-11 15:28:03 -07:00
ben.stull 19ee695c20 feat(products): row validation — §6.5.1 rules + INV-15 sanitization
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-11 15:17:01 -07:00
ben.stull f64c3fddf9 feat(products): CSV codec — file-level parse + INV-18 caps (PUC-5a)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-11 14:58:53 -07:00
ben.stull c39bbd4728 feat(products): domain skeleton — errors, column registry, canonical row model
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-11 14:55:37 -07:00
ben.stull 0ee948b34d feat(products): SD-0002 §6.3 schema — migration 0002 + nh3/multipart deps (SLICE-5)
Also updates test_migrate_from_empty_applies_0001 → test_migrate_from_empty_applies_all
to assert both migrations apply on a fresh DB (mechanical consequence of adding 0002).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-11 14:52:18 -07:00
ben.stull 1267d4f29d docs: SLICE-5 implementation plan (SD-0002 §7.2 import spine)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-11 14:48:47 -07:00
ben.stull 6b405a0f2d Merge pull request 'fix(signin): key the resend cooldown to the address it was set for (#20)' (#22) from fix-20-signin-cooldown-keying into main
ci / check (push) Has been cancelled
2026-06-11 08:22:08 +00:00
ben.stull 118e925580 fix(signin): key the resend cooldown to the address it was set for (#20)
ci / check (push) Has been cancelled
ci / check (pull_request) Has been cancelled
The email-step Send button disabled on any running cooldown, so a merchant
who mistyped their address was locked out of sending to the corrected one
for the rest of the 60s window — a guard the server never had: request_code
enforces the cooldown per address (SD-0001 INV-3 -> PUC-2c).

Track which address the running cooldown belongs to (cooldownFor) and gate
the email-step button through cooldownAppliesTo(), which blocks only while
the countdown runs AND the input normalizes (strip+lowercase, mirroring
normalize_email / INV-2) to that address. Same address still shows the
honest 'Resend in Ns'; any other address sends immediately. The code-step
resend is unchanged. Verified in the live UI (wrong address -> Wrong
address? -> corrected address sends at once).

package-lock.json: sync recorded version with package.json (0.4.0).

Closes #20

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-11 01:21:23 -07:00
ben.stull 11acb3a5b1 Merge pull request 'docs: land session 0023 plan doc (stray untracked file)' (#18) from docs-archive-0023-plan into main
ci / check (push) Has been cancelled
2026-06-11 07:56:34 +00:00
ben.stull efc9c8edb9 docs: land session 0023's ui-designs-collection plan in-repo (was untracked; content archive already had it)
ci / check (pull_request) Has been cancelled
ci / check (push) Has been cancelled
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-11 00:56:31 -07:00
ben.stull f7b76d5370 Merge pull request 'SLICE-4: record the first PPE bootstrap rehearsal (PUC-11) — green' (#17) from slice-4-rehearsal-record into main
ci / check (push) Has been cancelled
2026-06-11 07:54:50 +00:00
ben.stull 4bb9763633 docs(slice-4): record the first PPE bootstrap rehearsal (PUC-11, 2026-06-11) — green
ci / check (pull_request) Has been cancelled
ci / check (push) Has been cancelled
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-11 00:54:47 -07:00
ben.stull a456d51f17 Merge pull request 'SLICE-4: deployment.toml — the ecomm PPE record' (#11) from slice-4-deployment-record into main
ci / check (push) Has been cancelled
2026-06-11 06:45:08 +00:00
ben.stull ac3c4ffe36 feat(slice-4): deployment.toml — the ecomm PPE record (launch-app §5.1; Cloud SQL via secret ref)
ci / check (push) Has been cancelled
ci / check (pull_request) Has been cancelled
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-10 23:45:05 -07:00
91 changed files with 10534 additions and 36 deletions
+1 -1
View File
@@ -1 +1 @@
0.4.0
0.8.0
+48
View File
@@ -0,0 +1,48 @@
"""products domain — catalog + bulk CSV import/export (SD-0002 §6.2).
Owns the canonical row model, codec, validation, diff engine, and import
drafts/runs. Storefront-scoped throughout (INV-14); upsert is the only mutation
(INV-10). Imported via this package surface only.
"""
from __future__ import annotations
from pathlib import Path
from .imagefetch import recover_incomplete_runs, run_image_phase
from .errors import (
DraftExpired,
DraftNotFound,
EmptyCatalog,
FileRejected,
NothingToApply,
PreviewStale,
ProductsError,
RunNotFound,
)
from .models import MAX_DATA_ROWS, MAX_FILE_BYTES
from .repo import image_for_serving
from .service import (
confirm_draft,
discard_draft,
export_catalog,
get_draft,
get_draft_records,
get_run,
import_validate,
list_runs,
summary,
)
# DOC-3: the downloadable worked-example CSV the BFF serves at /api/products/sample.csv.
SAMPLE_CSV_PATH = Path(__file__).parent / "sample.csv"
# DOC-2: the column reference the BFF serves at /api/products/columns.md.
COLUMNS_MD_PATH = Path(__file__).parent / "columns.md"
__all__ = [
"ProductsError", "FileRejected", "DraftNotFound", "DraftExpired",
"PreviewStale", "NothingToApply", "RunNotFound", "EmptyCatalog",
"MAX_DATA_ROWS", "MAX_FILE_BYTES", "SAMPLE_CSV_PATH", "COLUMNS_MD_PATH",
"import_validate", "get_draft", "get_draft_records", "discard_draft",
"confirm_draft", "list_runs", "get_run", "summary", "export_catalog",
"run_image_phase", "recover_incomplete_runs", "image_for_serving",
]
+86
View File
@@ -0,0 +1,86 @@
"""CSV codec — bytes → ParsedFile (SD-0002 §6.5.1). File-level gates only (PUC-5a);
row semantics live in validate.py. Dialect detection is the INV-17 seam (SLICE-8
adds Shopify)."""
from __future__ import annotations
import csv
import io
from .dialect_shopify import is_shopify_header, map_shopify_header
from .errors import FileRejected
from .models import KNOWN_COLUMNS, MAX_DATA_ROWS, MAX_FILE_BYTES, ParsedFile, Row
_REQUIRED_HEADER_COLUMNS = ("Handle", "Title")
def detect_dialect(header: list[str]) -> str:
"""The INV-17 seam: recognize Shopify's header set, else canonical (§6.5.1)."""
return "shopify" if is_shopify_header(header) else "canonical"
def parse_csv(data: bytes) -> ParsedFile:
if len(data) > MAX_FILE_BYTES:
raise FileRejected("file_too_large", "This file is larger than 10 MB.")
try:
text = data.decode("utf-8-sig")
except UnicodeDecodeError:
raise FileRejected("not_csv", "This file isn't readable as CSV.") from None
reader = csv.reader(io.StringIO(text))
try:
try:
raw_header = next(reader)
except StopIteration:
raise FileRejected("not_csv", "This file isn't readable as CSV.") from None
header = [h.strip() for h in raw_header]
dialect = detect_dialect(header)
# INV-17: normalize the header to canonical names at the boundary. mapped[i]
# is the canonical name for header[i] (or None when that column has no
# canonical home); unknown is the not-imported warning list. Canonical files
# map to themselves; unknown columns are warned exactly as before.
if dialect == "shopify":
mapped, unknown = map_shopify_header(header)
else:
mapped = [c if c in KNOWN_COLUMNS else None for c in header]
unknown = [c for c in header if c and c not in KNOWN_COLUMNS]
for col in _REQUIRED_HEADER_COLUMNS:
if col not in mapped:
raise FileRejected(
"missing_required_column",
f"This file is missing the required column '{col}'.",
)
# First occurrence of a duplicated canonical column wins.
col_index: dict[str, int] = {}
for i, name in enumerate(mapped):
if name and name not in col_index:
col_index[name] = i
# De-dup the warning list, order-preserving.
seen: set[str] = set()
unknown = [c for c in unknown if not (c in seen or seen.add(c))]
rows: list[Row] = []
for raw in reader:
if not any(cell.strip() for cell in raw):
continue
if len(rows) >= MAX_DATA_ROWS:
raise FileRejected(
"too_many_rows",
f"This file has more than {MAX_DATA_ROWS:,} rows — split it and import in parts.",
)
cells = {
c: (raw[col_index[c]].strip() if col_index[c] < len(raw) else "")
for c in col_index
}
# Shopify grams carry an implicit unit; canonical needs it explicit. In a
# Shopify file "Variant Weight" can only come from the Variant Grams rename
# (a canonical-named Variant Weight would have vetoed Shopify detection), so
# weight and unit move together: a value -> "g"; a cleared grams (present-but-
# empty) clears the unit too, never leaving a stale unit (§6.5.1).
if dialect == "shopify" and "Variant Weight" in cells:
cells["Variant Weight Unit"] = "g" if cells["Variant Weight"] else ""
rows.append(Row(line_number=reader.line_num, cells=cells))
except csv.Error:
raise FileRejected("not_csv", "This file isn't readable as CSV.") from None
return ParsedFile(dialect=dialect, header=header, unknown_columns=unknown, rows=rows)
+95
View File
@@ -0,0 +1,95 @@
# Product CSV — column reference
This is the complete reference for the product CSV you import into and export from
your store (SD-0002 §6.5.1). The format is **canonical** — a clean superset of the
Shopify product CSV — and a **Shopify product CSV** imports directly too: we detect
which one you uploaded and map it for you (see *Shopify dialect* below).
You don't pick a format. Upload your file; the preview tells you which format was
recognized and shows exactly what will change before anything is applied.
## File shape
- UTF-8 (a byte-order mark is tolerated), comma-delimited, RFC 4180 quoting.
- A header row is **required**. `Handle` and `Title` are the only always-required
columns.
- Up to 5,000 rows and 10 MB per file. Split larger catalogs and import in parts.
## Row grammar
Consecutive rows that share a `Handle` describe **one product**. The product's first
row carries the product-level fields (`Title` is required there). Each row may carry:
- a **variant** — its `Option n Value`s plus `Variant *` fields;
- an **image**`Image Src` with `Image Position` / `Image Alt Text`;
- or both.
An image-only row (just `Handle` + `Image *`) is valid — that's how a product carries
more images than it has variants.
## Updating: blank vs. absent
- A column **absent from your file** is left untouched on existing products.
- A cell that is **present but empty** clears that field (resets it to its default).
Every clear is shown explicitly in the preview, so this is deliberate, visible
behavior — never a silent surprise.
## Canonical columns
| Column | Level | Required | Notes |
| --- | --- | --- | --- |
| `Handle` | product | every row | Identity: lowercase letters, numbers, and dashes. |
| `Title` | product | first row of a product | |
| `Description` | product | — | HTML allowed; sanitized on import. |
| `Vendor` | product | — | Free text. |
| `Type` | product | — | `standalone` (default). `kit_virtual` / `kit_assembled` are reserved; any non-`standalone` value is a row error until kits ship. |
| `Google Product Category` | product | — | Taxonomy string. |
| `Tags` | product | — | Comma-separated within the cell. |
| `Status` | product | — | `draft` \| `active` \| `archived`; defaults to `active` on add. |
| `Published` | product | — | `TRUE` \| `FALSE`; defaults to `TRUE` on add. |
| `Option1 Name``Option3 Name` | product | with values | e.g. "Size". A value without its name is a row error. |
| `Option1 Value``Option3 Value` | variant | per variant | The variant's identity combination. |
| `Variant SKU` | variant | — | Indexed; not identity. |
| `Variant Barcode` | variant | — | |
| `Variant Price` | variant | — | Decimal; no currency symbol. |
| `Variant Cost` | variant | — | Decimal. |
| `Variant Weight` + `Variant Weight Unit` | variant | — | Decimal weight plus a unit string. |
| `Variant Volume` + `Variant Volume Unit` | variant | — | Decimal volume plus a unit string. |
| `Variant Tax ID 1` / `Variant Tax ID 2` | variant | — | Opaque references. |
| `Variant Inventory Tracker` | variant | — | |
| `Variant Inventory Qty` | variant | — | Integer ≥ 0. |
| `Variant Position` | variant | — | Display order; defaults to file order. |
| `Image Src` | image | — | URL (or a store-hosted URL on re-import). |
| `Image Position` | image | — | Integer order. |
| `Image Alt Text` | image | — | |
| `Variant Image` | variant | — | URL for this variant's specific image. |
| `Component 1 SKU` / `Component 1 Quantity``Component 10 …` | variant | — | Reserved for kits — a non-empty value is a row error today. |
## Shopify dialect
If you upload a **Shopify product CSV**, we recognize it by its header set and map it
to the canonical columns automatically. The preview labels it *"Shopify product
CSV — mapped"*.
**Mapped — renamed**
| Shopify column | Becomes |
| --- | --- |
| `Body (HTML)` | `Description` |
| `Product Category` | `Google Product Category` |
| `Cost per item` | `Variant Cost` |
| `Variant Grams` | `Variant Weight` (in grams) |
**Mapped — same name:** `Handle`, `Title`, `Vendor`, `Tags`, `Published`, `Status`,
`Option13 Name` / `Option13 Value`, `Variant SKU`, `Variant Barcode`,
`Variant Price`, `Variant Inventory Tracker`, `Variant Inventory Qty`,
`Variant Image`, `Image Src`, `Image Position`, `Image Alt Text`.
**Not imported — listed in the preview as a heads-up, never silently dropped:** the
Shopify free-text `Type` (canonical `Type` is structural, so we don't fold a category
string into it), `Variant Compare At Price`, `Variant Inventory Policy`,
`Variant Fulfillment Service`, `Variant Requires Shipping`, `Variant Taxable`,
`Variant Tax Code`, `Gift Card`, SEO fields, every `Google Shopping / …` field, and
market/region price columns. These have no canonical home yet; your other data still
imports.
@@ -0,0 +1,95 @@
"""Shopify product-CSV adapter (INV-17): detect by header signature, map to canonical.
The mapping is the §6.5.1 contract; the exhaustive table is pinned here and
mirrored by tests/test_products_dialect_shopify.py + e2e/fixtures/shopify-export.csv.
"""
from __future__ import annotations
from .models import KNOWN_COLUMNS
# Shopify header -> canonical header (renames only; direct same-name columns pass
# through via KNOWN_COLUMNS). Variant Grams carries a value transform (see codec).
SHOPIFY_RENAME: dict[str, str] = {
"Body (HTML)": "Description",
"Product Category": "Google Product Category",
"Cost per item": "Variant Cost",
"Variant Grams": "Variant Weight",
}
# Canonical-named columns Shopify uses differently — must NOT pass through.
# Type: Shopify's free-text type vs canonical's structural Type (§6.5.1, decision D).
# Variant Weight Unit: superseded by the Variant Grams -> Variant Weight transform.
SHOPIFY_DROP: frozenset[str] = frozenset({"Type", "Variant Weight Unit"})
# Columns whose presence proves the file is a Shopify export (canonical never has them).
_SIGNATURE: frozenset[str] = frozenset(
{
"Body (HTML)",
"Variant Grams",
"Cost per item",
"Variant Compare At Price",
"Variant Inventory Policy",
"Variant Fulfillment Service",
"Variant Requires Shipping",
"Variant Taxable",
"Gift Card",
"SEO Title",
"SEO Description",
}
)
# Canonical-distinctive columns — names Shopify renames away (so a real Shopify export
# never carries them) plus canonical-only columns Shopify has no equivalent for. Their
# presence proves the file is canonical and VETOES Shopify detection: detection must
# lean conservative, because under-detection is safe (Shopify-named columns are warned
# as not-imported) while over-detection corrupts (canonical Type / weight-unit are
# dropped). This closes the misdetection hazard — a canonical file that merely contains
# a stray signature-named column (e.g. `SEO Title`) is never misread as Shopify (§7.4).
_CANONICAL_SIGNATURE: frozenset[str] = frozenset(
{
"Description",
"Variant Cost",
"Variant Weight",
"Google Product Category",
"Variant Volume",
"Variant Volume Unit",
"Variant Tax ID 1",
"Variant Tax ID 2",
"Variant Position",
}
)
def is_shopify_header(header: list[str]) -> bool:
"""True iff the header carries a Shopify-only signature column AND no
canonical-distinctive column. A canonical signal vetoes Shopify detection so an
ambiguous or hybrid header is treated as canonical — where Shopify-named columns
are warned as not-imported rather than silently remapped (never misparses, §7.4)."""
cols = {h.strip() for h in header}
if cols & _CANONICAL_SIGNATURE:
return False
if cols & _SIGNATURE:
return True
return any(c.startswith("Google Shopping / ") for c in cols)
def map_shopify_header(header: list[str]) -> tuple[list[str | None], list[str]]:
"""Return (mapped, not_imported): mapped[i] is the canonical name for header[i],
or None when that Shopify column has no canonical home; not_imported lists the
original Shopify names (in order) that were dropped — the preview warning."""
mapped: list[str | None] = []
not_imported: list[str] = []
for raw in header:
col = raw.strip()
if col in SHOPIFY_RENAME:
mapped.append(SHOPIFY_RENAME[col])
elif col in SHOPIFY_DROP:
mapped.append(None)
not_imported.append(col)
elif col in KNOWN_COLUMNS:
mapped.append(col)
else:
mapped.append(None)
if col:
not_imported.append(col)
return mapped, not_imported
+350
View File
@@ -0,0 +1,350 @@
"""Diff engine — catalog × canonical products → apply plan + preview records (SD-0002 §6.5.2).
Classifies each canonical product against the storefront's current catalog as
add / update / unchanged / error. One walk produces two views of the same
computation: a typed apply *plan* carrying resolved native values (Decimal etc.)
for the confirm transaction, and JSON-ready preview records derived from that
walk (stored as draft JSONB, served verbatim to the SPA) — so what confirm
applies is exactly what preview showed (INV-11). A summary and a deterministic
fingerprint over the records detect catalog drift between preview and confirm.
Deliberately DB-free: the catalog snapshot dataclasses are defined here and the
repo layer builds them. Only fields present in the file's canonical fields{}
participate in a comparison — an absent column is untouched, never a change
(§6.5.1); a present-but-empty cell resolves to the field's CLEAR_DEFAULTS entry
— except a variant's position, whose default is the variant's 1-based file
order within its product ("defaults to file order"), resolved here at diff time.
Catalog variants/images absent from the file are likewise untouched (INV-10);
file variants match catalog variants by their option-value combination (INV-13).
"""
from __future__ import annotations
import hashlib
import json
from dataclasses import dataclass, field, replace
from decimal import Decimal
from . import hosted
from .models import CLEAR_DEFAULTS, CanonicalProduct, CanonicalVariant, RowError
@dataclass
class CatalogVariant:
id: int
options: tuple[str | None, str | None, str | None]
position: int
# sku, barcode, price (Decimal|None), cost, weight, weight_unit, volume,
# volume_unit, tax_id_1, tax_id_2, inventory_tracker, inventory_qty,
# variant_image (linked image source_url or None)
fields: dict[str, object]
@dataclass
class CatalogImage:
id: int
source_url: str
position: int
alt_text: str | None
status: str = "pending"
@dataclass
class CatalogProduct:
id: int
handle: str
title: str
option_names: tuple[str | None, str | None, str | None]
# title, description_html, vendor, product_type, google_product_category,
# tags (list[str]), status, published (bool)
fields: dict[str, object]
variants: list[CatalogVariant]
images: list[CatalogImage]
# ---------------------------------------------------------------------------
# Apply plan — the typed twin of the preview records. confirm_draft executes
# these; the values are resolved natives (Decimal, bool, list), never the
# json-safe strings the records carry for display.
# ---------------------------------------------------------------------------
@dataclass
class VariantPlan:
kind: str # "add" | "update"
canonical: CanonicalVariant
catalog_id: int | None # None for add
# 1-based index within the product's file variants — the position default.
file_order: int = 0
# field -> resolved after-value (update only); adds resolve from canonical.fields
changes: dict[str, object] = field(default_factory=dict)
@dataclass
class ImagePlan:
kind: str # "add" | "update"
source_url: str
position: int
alt_text: str | None
image_id: int | None # None for add
changes: dict[str, object] = field(default_factory=dict) # subset of {"position","alt_text"}
@dataclass
class ProductPlan:
kind: str # "add" | "update" | "unchanged" | "error"
canonical: CanonicalProduct
catalog: CatalogProduct | None
# field -> resolved after-value (update only; may include title/option*_name)
product_changes: dict[str, object] = field(default_factory=dict)
variant_plans: list[VariantPlan] = field(default_factory=list)
image_plans: list[ImagePlan] = field(default_factory=list)
@dataclass(frozen=True)
class DiffResult:
records: list[dict]
summary: dict
fingerprint: str
plan: list[ProductPlan]
# Option-name presence markers in fields{} (see validate.py); the values are
# compared via the option_names attribute, not through the generic field loop.
_OPTION_NAME_FIELDS = ("option1_name", "option2_name", "option3_name")
_SUMMARY_KEY = {"add": "adds", "update": "updates", "unchanged": "unchanged", "error": "errors"}
def compute_diff(catalog: dict[str, CatalogProduct], products: list[CanonicalProduct]) -> DiffResult:
records: list[dict] = []
plan: list[ProductPlan] = []
summary = {"adds": 0, "updates": 0, "unchanged": 0, "errors": 0}
for product in products:
current = catalog.get(product.handle)
_resolve_hosted_images(product, current)
if product.errors:
product_plan = ProductPlan(kind="error", canonical=product, catalog=current)
detail: dict = {"errors": [e.as_json() for e in product.errors]}
elif current is None:
product_plan = _add_plan(product)
detail = _add_detail(product_plan)
else:
product_plan, detail = _update_plan(product, current)
kind = product_plan.kind
plan.append(product_plan)
records.append(
{
"handle": product.handle,
"title": product.title or (current.title if current else ""),
"kind": kind,
"variant_count": len(current.variants) if kind == "unchanged" else len(product.variants),
"detail": detail,
}
)
summary[_SUMMARY_KEY[kind]] += 1
fingerprint = hashlib.sha256(
json.dumps(records, sort_keys=True, separators=(",", ":")).encode()
).hexdigest()
return DiffResult(records=records, summary=summary, fingerprint=fingerprint, plan=plan)
def _resolve_hosted_images(product: CanonicalProduct, current: CatalogProduct | None) -> None:
"""Recognize re-imported hosted image URLs (INV-12). A hosted URL is resolved
to this product's existing image with that id — re-keyed onto that image's
source_url so the by_src match treats it as the existing image (never an add,
never a fetch). A hosted URL whose id is not one of this product's images
(cross-storefront, deleted, or a brand-new product) is a row error."""
current_by_id = {i.id: i for i in current.images} if current else {}
resolved: list = []
for image in product.images:
hosted_id = hosted.parse_image_id(image.source_url)
if hosted_id is None:
resolved.append(image)
continue
match = current_by_id.get(hosted_id)
if match is None:
product.errors.append(
RowError(image.line_number, "Image Src",
f"image '{image.source_url}' refers to an unknown hosted id")
)
resolved.append(image)
continue
resolved.append(replace(image, source_url=match.source_url))
product.images = resolved
def _add_plan(product: CanonicalProduct) -> ProductPlan:
return ProductPlan(
kind="add",
canonical=product,
catalog=None,
variant_plans=[
VariantPlan(kind="add", canonical=v, catalog_id=None, file_order=order)
for order, v in enumerate(product.variants, start=1)
],
image_plans=[
ImagePlan(kind="add", source_url=i.source_url, position=i.position,
alt_text=i.alt_text, image_id=None)
for i in product.images
],
)
def _add_detail(plan: ProductPlan) -> dict:
# On add, absent fields fall back to their defaults for display where one
# exists — the detail shows what will actually be set.
set_fields = resolved_product_fields(plan.canonical)
for field_name, default in CLEAR_DEFAULTS.items():
set_fields.setdefault(field_name, default)
return {
"set": {f: _json_safe(v) for f, v in set_fields.items()},
"option_names": list(plan.canonical.option_names),
"variants": [
{"options": list(vp.canonical.options),
"set": _resolved_variant_fields(vp.canonical, vp.file_order)}
for vp in plan.variant_plans
],
"images": [
{"src": ip.source_url, "position": ip.position, "alt_text": ip.alt_text}
for ip in plan.image_plans
],
}
def _update_plan(product: CanonicalProduct, current: CatalogProduct) -> tuple[ProductPlan, dict]:
"""One walk, two outputs: the resolved-value plan entries and the json-safe
record detail entries are appended side by side, so they can never diverge."""
product_changes: dict[str, object] = {}
changes: list[dict] = []
# Title is always file-present (required header column); "" means the block
# already carries an error and never reaches here.
if product.title and product.title != current.title:
product_changes["title"] = product.title
changes.append(_change("title", current.title, product.title))
for field_name, resolved in resolved_product_fields(product).items():
before = current.fields.get(field_name)
if resolved != before:
product_changes[field_name] = resolved
changes.append(_change(field_name, before, resolved))
for slot in (1, 2, 3):
if f"option{slot}_name" not in product.fields:
continue
before, after = current.option_names[slot - 1], product.option_names[slot - 1]
if after != before:
product_changes[f"option{slot}_name"] = after
changes.append(_change(f"option{slot}_name", before, after))
variant_plans: list[VariantPlan] = []
variant_entries: list[dict] = []
by_options = {v.options: v for v in current.variants}
for file_order, variant in enumerate(product.variants, start=1):
match = by_options.get(variant.options)
if match is None:
variant_plans.append(
VariantPlan(kind="add", canonical=variant, catalog_id=None, file_order=file_order)
)
variant_entries.append(
{"options": list(variant.options), "kind": "add",
"set": _resolved_variant_fields(variant, file_order)}
)
continue
variant_changes: dict[str, object] = {}
variant_change_entries: list[dict] = []
for f, resolved in resolved_variant_fields(variant, file_order).items():
# position lives on the catalog variant as an attribute, not in
# fields{} — compare it explicitly (as images do for theirs).
before = match.position if f == "position" else match.fields.get(f)
if resolved != before:
variant_changes[f] = resolved
variant_change_entries.append(_change(f, before, resolved))
if variant_changes:
variant_plans.append(
VariantPlan(kind="update", canonical=variant, catalog_id=match.id,
file_order=file_order, changes=variant_changes)
)
variant_entries.append(
{"options": list(variant.options), "kind": "update", "changes": variant_change_entries}
)
image_plans: list[ImagePlan] = []
image_entries: list[dict] = []
by_src = {i.source_url: i for i in current.images}
for image in product.images:
match = by_src.get(image.source_url)
if match is None:
image_plans.append(
ImagePlan(kind="add", source_url=image.source_url, position=image.position,
alt_text=image.alt_text, image_id=None)
)
image_entries.append(
{"src": image.source_url, "kind": "add", "position": image.position, "alt_text": image.alt_text}
)
continue
image_changes: dict[str, object] = {}
image_change_entries: list[dict] = []
for f, before, after in (
("position", match.position, image.position),
("alt_text", match.alt_text, image.alt_text),
):
if after != before:
image_changes[f] = after
image_change_entries.append(_change(f, before, after))
if image_changes:
image_plans.append(
ImagePlan(kind="update", source_url=image.source_url, position=image.position,
alt_text=image.alt_text, image_id=match.id, changes=image_changes)
)
image_entries.append(
{"src": image.source_url, "kind": "update", "changes": image_change_entries}
)
detail: dict = {}
if changes:
detail["changes"] = changes
if variant_entries:
detail["variants"] = variant_entries
if image_entries:
detail["images"] = image_entries
kind = "update" if detail else "unchanged"
return (
ProductPlan(kind=kind, canonical=product, catalog=current, product_changes=product_changes,
variant_plans=variant_plans, image_plans=image_plans),
detail,
)
def resolved_fields(fields: dict[str, object]) -> dict[str, object]:
"""File-present fields with None (an explicit clear) resolved to the default."""
return {f: (v if v is not None else CLEAR_DEFAULTS.get(f)) for f, v in fields.items()}
def resolved_product_fields(product: CanonicalProduct) -> dict[str, object]:
"""Resolved product-level fields, minus the option-name presence markers."""
return {
f: v for f, v in resolved_fields(product.fields).items() if f not in _OPTION_NAME_FIELDS
}
def resolved_variant_fields(variant: CanonicalVariant, file_order: int) -> dict[str, object]:
"""File-present variant fields with clears resolved. A cleared position has
no CLEAR_DEFAULTS entry — it resets to the variant's 1-based file order
within its product ("defaults to file order"), never to NULL."""
resolved = resolved_fields(variant.fields)
if "position" in resolved and resolved["position"] is None:
resolved["position"] = file_order
return resolved
def _resolved_variant_fields(variant: CanonicalVariant, file_order: int) -> dict[str, object]:
return {f: _json_safe(v) for f, v in resolved_variant_fields(variant, file_order).items()}
def _change(field_name: str, before: object, after: object) -> dict:
return {"field": field_name, "before": _json_safe(before), "after": _json_safe(after)}
def _json_safe(value: object) -> object:
if isinstance(value, Decimal):
return str(value)
if isinstance(value, (tuple, list)):
return [_json_safe(v) for v in value]
return value
+41
View File
@@ -0,0 +1,41 @@
"""products domain errors (SD-0002 §6.4 error envelope codes)."""
from __future__ import annotations
class ProductsError(Exception):
"""Base for products-domain errors."""
class FileRejected(ProductsError):
"""PUC-5a: the whole file is unusable; no draft is created. `code` is the §6.4
error code (not_csv | missing_required_column | unknown_dialect | too_many_rows |
file_too_large)."""
def __init__(self, code: str, message: str):
super().__init__(message)
self.code = code
self.message = message
class DraftNotFound(ProductsError):
"""No such draft for this storefront (or already discarded)."""
class DraftExpired(ProductsError):
"""The draft's validity window passed (§6.3 ~1 h)."""
class PreviewStale(ProductsError):
"""INV-11: the catalog changed since validation — the previewed diff no longer holds."""
class NothingToApply(ProductsError):
"""PUC-10: no adds and no updates — confirming would be a no-op."""
class RunNotFound(ProductsError):
"""No such import run for this storefront."""
class EmptyCatalog(ProductsError):
"""PUC-9: nothing to export (no products, or none matching the status filter)."""
+30
View File
@@ -0,0 +1,30 @@
"""Hosted-image URL helpers (SD-0002 §6.3.1, INV-12).
Build the canonical hosted Image Src for a fetched image, and recognize one on
re-import. DB-free and host-agnostic: recognition is a path-parse, so an export
round-trips across localhost / PPE / the #23 rebrand. The diff resolves a parsed
id against the storefront-scoped catalog snapshot.
"""
from __future__ import annotations
import re
from urllib.parse import urlsplit
RENDITIONS = ("original", "thumb", "card", "detail")
EXPORT_RENDITION = "detail"
_PATH = re.compile(r"^/api/products/images/(\d+)/(original|thumb|card|detail)$")
def image_url(base_url: str, image_id: int, rendition: str = EXPORT_RENDITION) -> str:
"""Absolute when base_url is set, else a root-relative path."""
path = f"/api/products/images/{image_id}/{rendition}"
return f"{base_url.rstrip('/')}{path}" if base_url else path
def parse_image_id(url: str) -> int | None:
"""The image id if url is one of our hosted-image URLs (any host), else None."""
if not url:
return None
m = _PATH.match(urlsplit(url).path)
return int(m.group(1)) if m else None
+146
View File
@@ -0,0 +1,146 @@
"""The post-commit image-fetch phase (SD-0002 §6.5.4, §6.9).
In-process, bounded-concurrency, resumable. For a run's pending images: fetch
each URL behind an SSRF guard with INV-18 bounds, run platform/images, store
renditions via the objectstore, and move the image pending -> fetched |
rejected_* | failed. Per-image idempotent (claim guard), so the phase can die
and resume. Lives in the domains layer (imports repo + platform); main.py
schedules it post-confirm and runs recovery at startup.
"""
from __future__ import annotations
import ipaddress
import socket
import time
from concurrent.futures import ThreadPoolExecutor
from urllib.parse import urlsplit
import httpx
from app.platform import images as images_mod
from app.platform import telemetry
from . import repo
MAX_FETCH_BYTES = 20 * 1024 * 1024 # INV-18
FETCH_TIMEOUT_S = 30 # INV-18
FETCH_CONCURRENCY = 4 # §6.6
_CONTENT_PREFIX = "image/"
class FetchBlocked(Exception):
"""SSRF guard or bounds refusal — maps to image status 'failed' with reason."""
def _guard_host(url: str, allow_private: bool) -> None:
parts = urlsplit(url)
if parts.scheme not in {"http", "https"}:
raise FetchBlocked(f"unsupported scheme: {parts.scheme!r}")
host = parts.hostname
if not host:
raise FetchBlocked("no host")
if allow_private:
return
try:
infos = socket.getaddrinfo(host, parts.port or (443 if parts.scheme == "https" else 80))
except socket.gaierror as exc:
raise FetchBlocked(f"dns failure: {exc}") from exc
for info in infos:
ip = ipaddress.ip_address(info[4][0])
if (ip.is_private or ip.is_loopback or ip.is_link_local
or ip.is_reserved or ip.is_multicast or ip.is_unspecified):
raise FetchBlocked(f"non-public address: {ip}")
def fetch_bytes(url: str, allow_private: bool) -> tuple[bytes, str]:
"""SSRF-guarded, bounded GET. Returns (bytes, content_type). Raises FetchBlocked."""
_guard_host(url, allow_private)
with httpx.Client(follow_redirects=False, timeout=FETCH_TIMEOUT_S) as client:
with client.stream("GET", url) as resp:
if resp.status_code != 200:
raise FetchBlocked(f"http {resp.status_code}")
ctype = resp.headers.get("content-type", "").split(";")[0].strip().lower()
if not ctype.startswith(_CONTENT_PREFIX):
raise FetchBlocked(f"not an image content-type: {ctype!r}")
chunks: list[bytes] = []
total = 0
for chunk in resp.iter_bytes():
total += len(chunk)
if total > MAX_FETCH_BYTES:
raise FetchBlocked("oversize")
chunks.append(chunk)
return b"".join(chunks), ctype
def _key(storefront_id: int, image_id: int, name: str) -> str:
return f"storefronts/{storefront_id}/product-images/{image_id}/{name}"
def _process_one(pool, store, image: dict, allow_private: bool) -> None:
with pool.connection() as conn:
claimed = repo.claim_image_for_fetch(conn, image["id"])
conn.commit()
if not claimed:
return
image_id, sf = image["id"], image["storefront_id"]
try:
data, ctype = fetch_bytes(image["source_url"], allow_private)
except (FetchBlocked, httpx.HTTPError) as exc:
with pool.connection() as conn:
repo.mark_image_failed(conn, image_id, str(exc)[:200]); conn.commit()
return
try:
result = images_mod.process(data)
if isinstance(result, images_mod.Rejected):
reason = ("below the resolution bar" if result.reason == "rejected_low_res"
else "not a supported image")
with pool.connection() as conn:
repo.mark_image_rejected(conn, image_id, result.reason, reason); conn.commit()
return
keys = {"original": _key(sf, image_id, "original"),
"thumb": _key(sf, image_id, "thumb.webp"),
"card": _key(sf, image_id, "card.webp"),
"detail": _key(sf, image_id, "detail.webp")}
store.put(keys["original"], data, ctype)
store.put(keys["thumb"], result.renditions["thumb"], "image/webp")
store.put(keys["card"], result.renditions["card"], "image/webp")
store.put(keys["detail"], result.renditions["detail"], "image/webp")
with pool.connection() as conn:
repo.mark_image_fetched(conn, image_id, keys); conn.commit()
except Exception as exc: # noqa: BLE001 — one bad image must never wedge the run (review fix)
with pool.connection() as conn:
repo.mark_image_failed(conn, image_id, f"processing error: {type(exc).__name__}")
conn.commit()
def run_image_phase(pool, store, run_id: int, allow_private: bool) -> None:
"""Fetch all pending images of one run, then move the run to a terminal status."""
started = time.monotonic()
with pool.connection() as conn:
pending = repo.pending_images_for_run(conn, run_id)
if pending:
with ThreadPoolExecutor(max_workers=FETCH_CONCURRENCY) as pool_x:
list(pool_x.map(lambda img: _process_one(pool, store, img, allow_private), pending))
with pool.connection() as conn:
counts = repo.run_image_counts(conn, run_id)
status = ("complete_with_problems"
if counts["rejected"] + counts["failed"] > 0 else "complete")
repo.set_run_status(conn, run_id, status)
conn.commit()
telemetry.emit("image_phase_completed", run_id=run_id, fetched=counts["fetched"],
rejected=counts["rejected"], failed=counts["failed"],
duration_ms=int((time.monotonic() - started) * 1000))
def recover_incomplete_runs(pool, store, allow_private: bool) -> list[int]:
"""Startup scan (§6.9): resume any run stuck in fetching_images. Returns the ids."""
with pool.connection() as conn:
runs = repo.incomplete_runs(conn)
resumed: list[int] = []
for run in runs:
with pool.connection() as conn:
pending = len(repo.pending_images_for_run(conn, run["id"]))
telemetry.emit("image_phase_recovered", run_id=run["id"], pending_resumed=pending)
run_image_phase(pool, store, run["id"], allow_private)
resumed.append(run["id"])
return resumed
+121
View File
@@ -0,0 +1,121 @@
"""Canonical row model + column registry — the one model every dialect maps to (INV-17)."""
from __future__ import annotations
from dataclasses import dataclass, field
# §6.5.1 canonical columns, by level. Header detection, unknown-column warnings, and
# validation all read from this registry.
PRODUCT_COLUMNS: dict[str, str] = {
# column -> product field name
"Title": "title",
"Description": "description_html",
"Vendor": "vendor",
"Type": "product_type",
"Google Product Category": "google_product_category",
"Tags": "tags",
"Status": "status",
"Published": "published",
"Option1 Name": "option1_name",
"Option2 Name": "option2_name",
"Option3 Name": "option3_name",
}
VARIANT_COLUMNS: dict[str, str] = {
"Variant SKU": "sku",
"Variant Barcode": "barcode",
"Variant Price": "price",
"Variant Cost": "cost",
"Variant Weight": "weight",
"Variant Weight Unit": "weight_unit",
"Variant Volume": "volume",
"Variant Volume Unit": "volume_unit",
"Variant Tax ID 1": "tax_id_1",
"Variant Tax ID 2": "tax_id_2",
"Variant Inventory Tracker": "inventory_tracker",
"Variant Inventory Qty": "inventory_qty",
"Variant Position": "position",
"Variant Image": "variant_image",
}
OPTION_VALUE_COLUMNS = ("Option1 Value", "Option2 Value", "Option3 Value")
IMAGE_COLUMNS = ("Image Src", "Image Position", "Image Alt Text")
COMPONENT_COLUMNS = tuple(
f"Component {i} {kind}" for i in range(1, 11) for kind in ("SKU", "Quantity")
)
KNOWN_COLUMNS = (
{"Handle"}
| set(PRODUCT_COLUMNS)
| set(VARIANT_COLUMNS)
| set(OPTION_VALUE_COLUMNS)
| set(IMAGE_COLUMNS)
| set(COMPONENT_COLUMNS)
)
# Clearing a field (present-but-empty cell, §6.5.1) resets it to its default.
CLEAR_DEFAULTS: dict[str, object] = {
"status": "active",
"published": True,
"product_type": "standalone",
"tags": [],
}
MAX_DATA_ROWS = 5_000 # INV-18
MAX_FILE_BYTES = 10 * 1024 * 1024 # INV-18
@dataclass(frozen=True)
class Row:
"""One CSV data row: 1-based file line number + the cells of known columns
present in the header (column name -> raw string, possibly empty)."""
line_number: int
cells: dict[str, str]
@dataclass(frozen=True)
class ParsedFile:
dialect: str
header: list[str]
unknown_columns: list[str]
rows: list[Row]
@dataclass(frozen=True)
class RowError:
line_number: int
column: str | None
message: str
def as_json(self) -> dict:
return {"line": self.line_number, "column": self.column, "message": self.message}
@dataclass
class CanonicalVariant:
line_number: int
options: tuple[str | None, str | None, str | None]
# field name -> normalized value; present only for columns in the file.
# value None == clear (reset to default/NULL).
fields: dict[str, object] = field(default_factory=dict)
@dataclass
class CanonicalImage:
line_number: int
source_url: str
position: int
alt_text: str | None
@dataclass
class CanonicalProduct:
first_line: int
handle: str
title: str # "" when missing (the block then carries an error)
option_names: tuple[str | None, str | None, str | None] = (None, None, None)
fields: dict[str, object] = field(default_factory=dict) # product-level, same semantics
variants: list[CanonicalVariant] = field(default_factory=list)
images: list[CanonicalImage] = field(default_factory=list)
errors: list[RowError] = field(default_factory=list)
@property
def valid(self) -> bool:
return not self.errors
+616
View File
@@ -0,0 +1,616 @@
"""products repo — the SQL layer for the import spine (SD-0002 §6.3 data model).
Owns SQL only: the catalog snapshot the diff engine reads, import draft/run
CRUD, and the apply primitives the confirm transaction calls. Business rules
live in service.py and diff.py — nothing here validates, diffs, commits, or
rolls back (the confirm flow runs the apply primitives inside its own
transaction). Every catalog/draft/run query is storefront-scoped (INV-14).
Dict payload conventions: functions feeding §6.4 API payloads (insert_draft,
list_runs, get_run) return datetimes as `.isoformat()` strings; get_draft_row
returns raw datetimes for the service's expiry check. TEXT[] columns bind/load
as Python lists and NUMERIC loads as Decimal natively under psycopg 3.
"""
from __future__ import annotations
import psycopg
from psycopg import sql
from psycopg.types.json import Jsonb
from .diff import CatalogImage, CatalogProduct, CatalogVariant
# ---------------------------------------------------------------------------
# Catalog snapshot (diff input) + dashboard counts
# ---------------------------------------------------------------------------
def load_catalog(conn: psycopg.Connection, storefront_id: int) -> dict[str, CatalogProduct]:
"""The storefront's full catalog, keyed by handle, in diff.py's snapshot shape."""
catalog: dict[str, CatalogProduct] = {}
by_id: dict[int, CatalogProduct] = {}
for row in conn.execute(
"SELECT id, handle, title, description_html, vendor, product_type,"
" google_product_category, tags, status, published,"
" option1_name, option2_name, option3_name"
" FROM product WHERE storefront_id = %s",
(storefront_id,),
):
product = CatalogProduct(
id=row[0],
handle=row[1],
title=row[2],
option_names=(row[10], row[11], row[12]),
fields={
"title": row[2],
"description_html": row[3],
"vendor": row[4],
"product_type": row[5],
"google_product_category": row[6],
"tags": row[7],
"status": row[8],
"published": row[9],
},
variants=[],
images=[],
)
catalog[product.handle] = product
by_id[product.id] = product
for row in conn.execute(
"SELECT v.product_id, v.id, v.position,"
" v.option1_value, v.option2_value, v.option3_value,"
" v.sku, v.barcode, v.price, v.cost, v.weight, v.weight_unit,"
" v.volume, v.volume_unit, v.tax_id_1, v.tax_id_2,"
" v.inventory_tracker, v.inventory_qty, i.source_url"
" FROM variant v"
" JOIN product p ON p.id = v.product_id"
" LEFT JOIN product_image i ON i.id = v.image_id"
" WHERE p.storefront_id = %s"
" ORDER BY v.product_id, v.position, v.id",
(storefront_id,),
):
by_id[row[0]].variants.append(
CatalogVariant(
id=row[1],
options=(row[3], row[4], row[5]),
position=row[2],
fields={
"sku": row[6],
"barcode": row[7],
"price": row[8],
"cost": row[9],
"weight": row[10],
"weight_unit": row[11],
"volume": row[12],
"volume_unit": row[13],
"tax_id_1": row[14],
"tax_id_2": row[15],
"inventory_tracker": row[16],
"inventory_qty": row[17],
"variant_image": row[18],
},
)
)
for row in conn.execute(
"SELECT i.product_id, i.id, i.source_url, i.position, i.alt_text, i.status"
" FROM product_image i"
" JOIN product p ON p.id = i.product_id"
" WHERE p.storefront_id = %s"
" ORDER BY i.product_id, i.position, i.id",
(storefront_id,),
):
by_id[row[0]].images.append(
CatalogImage(id=row[1], source_url=row[2], position=row[3],
alt_text=row[4], status=row[5])
)
return catalog
_EXPORT_STATUSES = ("all", "active", "draft", "archived")
def export_catalog(
conn: psycopg.Connection, storefront_id: int, status_filter: str
) -> list[CatalogProduct]:
"""The storefront's catalog as an ordered snapshot list, optionally filtered
by product status (PUC-9). Reuses load_catalog's snapshot builder; the
catalog fits in memory (≤5k rows, INV-18). Ordered by handle for a stable,
deterministic export."""
catalog = load_catalog(conn, storefront_id)
products = sorted(catalog.values(), key=lambda p: p.handle)
if status_filter and status_filter != "all":
products = [p for p in products if p.fields.get("status") == status_filter]
return products
def product_count(conn: psycopg.Connection, storefront_id: int) -> int:
return conn.execute(
"SELECT count(*) FROM product WHERE storefront_id = %s", (storefront_id,)
).fetchone()[0]
def image_problem_count(conn: psycopg.Connection, storefront_id: int) -> int:
return conn.execute(
"SELECT count(*) FROM product_image i"
" JOIN product p ON p.id = i.product_id"
" WHERE p.storefront_id = %s"
" AND i.status IN ('rejected_low_res', 'rejected_not_image', 'failed')",
(storefront_id,),
).fetchone()[0]
def latest_run_id(conn: psycopg.Connection, storefront_id: int) -> int | None:
row = conn.execute(
"SELECT id FROM import_run WHERE storefront_id = %s"
" ORDER BY created_at DESC, id DESC LIMIT 1",
(storefront_id,),
).fetchone()
return row[0] if row else None
# ---------------------------------------------------------------------------
# Import drafts (preview server side, INV-11)
# ---------------------------------------------------------------------------
def insert_draft(
conn: psycopg.Connection,
storefront_id: int,
account_id: int,
file_name: str,
dialect: str,
file_bytes: bytes,
summary: dict,
records: list,
fingerprint: str,
unknown_columns: list[str],
) -> dict:
"""Create a draft (expires in 1 hour); returns the §6.4 draft payload."""
row = conn.execute(
"INSERT INTO import_draft"
" (storefront_id, account_id, file_name, dialect, file_bytes,"
" summary, records, fingerprint, unknown_columns, expires_at)"
" VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, now() + interval '1 hour')"
" RETURNING id, expires_at",
(
storefront_id,
account_id,
file_name,
dialect,
file_bytes,
Jsonb(summary),
Jsonb(records),
fingerprint,
unknown_columns,
),
).fetchone()
return {
"id": row[0],
"file_name": file_name,
"dialect": dialect,
"summary": summary,
"unknown_columns": unknown_columns,
"expires_at": row[1].isoformat(),
}
def get_draft_row(conn: psycopg.Connection, storefront_id: int, draft_id: int) -> dict | None:
row = conn.execute(
"SELECT id, storefront_id, account_id, file_name, dialect, file_bytes,"
" summary, records, fingerprint, unknown_columns, expires_at, created_at"
" FROM import_draft WHERE id = %s AND storefront_id = %s",
(draft_id, storefront_id),
).fetchone()
if row is None:
return None
columns = (
"id",
"storefront_id",
"account_id",
"file_name",
"dialect",
"file_bytes",
"summary",
"records",
"fingerprint",
"unknown_columns",
"expires_at",
"created_at",
)
record = dict(zip(columns, row))
# BYTEA loads as memoryview; the service expects bytes.
record["file_bytes"] = bytes(record["file_bytes"])
return record
def draft_records(
conn: psycopg.Connection,
storefront_id: int,
draft_id: int,
kind: str | None,
limit: int,
offset: int,
) -> list[dict]:
"""The draft's preview records, order-preserving, optionally filtered by kind."""
rows = conn.execute(
"SELECT rec FROM import_draft d,"
" jsonb_array_elements(d.records) WITH ORDINALITY AS r(rec, ord)"
" WHERE d.id = %(draft_id)s AND d.storefront_id = %(storefront_id)s"
" AND (%(kind)s::text IS NULL OR rec->>'kind' = %(kind)s)"
" ORDER BY ord LIMIT %(limit)s OFFSET %(offset)s",
{
"draft_id": draft_id,
"storefront_id": storefront_id,
"kind": kind,
"limit": limit,
"offset": offset,
},
).fetchall()
return [row[0] for row in rows]
def delete_draft(conn: psycopg.Connection, storefront_id: int, draft_id: int) -> None:
conn.execute(
"DELETE FROM import_draft WHERE id = %s AND storefront_id = %s",
(draft_id, storefront_id),
)
def sweep_expired_drafts(conn: psycopg.Connection) -> None:
conn.execute("DELETE FROM import_draft WHERE expires_at < now()")
# ---------------------------------------------------------------------------
# Import runs (history, PUC-8)
# ---------------------------------------------------------------------------
_TERMINAL_RUN_STATUSES = ("complete", "complete_with_problems")
def insert_run(
conn: psycopg.Connection,
storefront_id: int,
account_id: int,
file_name: str,
dialect: str,
added: int,
updated: int,
errored: int,
status: str,
) -> int:
return conn.execute(
"INSERT INTO import_run"
" (storefront_id, account_id, file_name, dialect,"
" products_added, products_updated, rows_errored, status, completed_at)"
" VALUES (%s, %s, %s, %s, %s, %s, %s, %s, CASE WHEN %s THEN now() END)"
" RETURNING id",
(
storefront_id,
account_id,
file_name,
dialect,
added,
updated,
errored,
status,
status in _TERMINAL_RUN_STATUSES,
),
).fetchone()[0]
def insert_run_errors(conn: psycopg.Connection, run_id: int, errors: list[dict]) -> None:
"""Record per-row errors (RowError.as_json shape: line/column/message)."""
if not errors:
return
with conn.cursor() as cur:
cur.executemany(
"INSERT INTO import_run_error (run_id, line_number, column_name, message)"
" VALUES (%s, %s, %s, %s)",
[(run_id, e["line"], e["column"], e["message"]) for e in errors],
)
_RUN_SELECT = (
"SELECT r.id, r.file_name, r.dialect, r.created_at, r.completed_at, r.status,"
" a.email, r.products_added, r.products_updated, r.rows_errored"
" FROM import_run r JOIN account a ON a.id = r.account_id"
)
def _run_dict(row: tuple) -> dict:
return {
"id": row[0],
"file_name": row[1],
"dialect": row[2],
"created_at": row[3].isoformat(),
"completed_at": row[4].isoformat() if row[4] is not None else None,
"status": row[5],
"by": row[6],
"products_added": row[7],
"products_updated": row[8],
"rows_errored": row[9],
}
def list_runs(conn: psycopg.Connection, storefront_id: int, limit: int, offset: int) -> list[dict]:
rows = conn.execute(
_RUN_SELECT + " WHERE r.storefront_id = %s ORDER BY r.created_at DESC, r.id DESC"
" LIMIT %s OFFSET %s",
(storefront_id, limit, offset),
).fetchall()
runs = [_run_dict(row) for row in rows]
if not runs:
return runs
ids = [r["id"] for r in runs]
by_run = {rid: {"fetched": 0, "rejected": 0, "failed": 0, "total": 0} for rid in ids}
for rid, fetched, rejected, failed, total in conn.execute(
"SELECT import_run_id,"
" count(*) FILTER (WHERE status='fetched'),"
" count(*) FILTER (WHERE status IN ('rejected_low_res','rejected_not_image')),"
" count(*) FILTER (WHERE status='failed'), count(*)"
" FROM product_image WHERE import_run_id = ANY(%s) GROUP BY import_run_id",
(ids,),
):
by_run[rid] = {"fetched": fetched, "rejected": rejected, "failed": failed, "total": total}
for r in runs:
r["image_counts"] = by_run[r["id"]]
return runs
def get_run(conn: psycopg.Connection, storefront_id: int, run_id: int) -> dict | None:
row = conn.execute(
_RUN_SELECT + " WHERE r.storefront_id = %s AND r.id = %s",
(storefront_id, run_id),
).fetchone()
if row is None:
return None
run = _run_dict(row)
run["errors"] = [
{"line": line, "column": column, "message": message}
for line, column, message in conn.execute(
"SELECT line_number, column_name, message FROM import_run_error"
" WHERE run_id = %s ORDER BY line_number, id",
(run_id,),
)
]
counts = run_image_counts(conn, run_id)
run["image_progress"] = {"done": counts["total"] - counts["pending"], "total": counts["total"]}
run["image_counts"] = {"fetched": counts["fetched"], "rejected": counts["rejected"],
"failed": counts["failed"]}
run["image_outcomes"] = run_image_outcomes(conn, run_id)
return run
# ---------------------------------------------------------------------------
# Image fetch phase (SLICE-7, SD-0002 §6.5.4) — claim/mark/count/outcomes.
# ---------------------------------------------------------------------------
def pending_images_for_run(conn: psycopg.Connection, run_id: int) -> list[dict]:
rows = conn.execute(
"SELECT i.id, i.source_url, p.storefront_id, i.product_id"
" FROM product_image i JOIN product p ON p.id = i.product_id"
" WHERE i.import_run_id = %s AND i.status = 'pending' ORDER BY i.id",
(run_id,),
).fetchall()
return [{"id": r[0], "source_url": r[1], "storefront_id": r[2], "product_id": r[3]} for r in rows]
def claim_image_for_fetch(conn: psycopg.Connection, image_id: int) -> bool:
row = conn.execute(
"UPDATE product_image SET status = 'pending' WHERE id = %s AND status = 'pending' RETURNING id",
(image_id,),
).fetchone()
return row is not None
def mark_image_fetched(conn: psycopg.Connection, image_id: int, keys: dict[str, str]) -> None:
conn.execute(
"UPDATE product_image SET status = 'fetched', failure_reason = NULL,"
" key_original = %s, key_thumb = %s, key_card = %s, key_detail = %s,"
" fetched_at = now() WHERE id = %s",
(keys["original"], keys["thumb"], keys["card"], keys["detail"], image_id),
)
def mark_image_rejected(conn: psycopg.Connection, image_id: int, status: str, reason: str) -> None:
conn.execute(
"UPDATE product_image SET status = %s, failure_reason = %s, fetched_at = now() WHERE id = %s",
(status, reason, image_id),
)
def mark_image_failed(conn: psycopg.Connection, image_id: int, reason: str) -> None:
conn.execute(
"UPDATE product_image SET status = 'failed', failure_reason = %s, fetched_at = now() WHERE id = %s",
(reason, image_id),
)
def run_image_counts(conn: psycopg.Connection, run_id: int) -> dict:
row = conn.execute(
"SELECT count(*) FILTER (WHERE status = 'fetched'),"
" count(*) FILTER (WHERE status IN ('rejected_low_res', 'rejected_not_image')),"
" count(*) FILTER (WHERE status = 'failed'),"
" count(*) FILTER (WHERE status = 'pending'), count(*)"
" FROM product_image WHERE import_run_id = %s",
(run_id,),
).fetchone()
return {"fetched": row[0], "rejected": row[1], "failed": row[2], "pending": row[3], "total": row[4]}
def set_run_status(conn: psycopg.Connection, run_id: int, status: str) -> None:
conn.execute(
"UPDATE import_run SET status = %s,"
" completed_at = CASE WHEN %s THEN now() ELSE completed_at END WHERE id = %s",
(status, status in _TERMINAL_RUN_STATUSES, run_id),
)
def incomplete_runs(conn: psycopg.Connection) -> list[dict]:
rows = conn.execute(
"SELECT id, storefront_id FROM import_run WHERE status = 'fetching_images' ORDER BY id",
).fetchall()
return [{"id": r[0], "storefront_id": r[1]} for r in rows]
def image_for_serving(conn: psycopg.Connection, storefront_id: int, image_id: int) -> dict | None:
"""Storefront-scoped lookup for the serving route: status + rendition keys (INV-14)."""
row = conn.execute(
"SELECT i.status, i.key_original, i.key_thumb, i.key_card, i.key_detail"
" FROM product_image i JOIN product p ON p.id = i.product_id"
" WHERE i.id = %s AND p.storefront_id = %s",
(image_id, storefront_id),
).fetchone()
if row is None:
return None
return {"status": row[0], "original": row[1], "thumb": row[2], "card": row[3], "detail": row[4]}
def run_image_outcomes(conn: psycopg.Connection, run_id: int) -> list[dict]:
rows = conn.execute(
"SELECT p.handle, i.source_url, i.status, i.failure_reason,"
" v.option1_value, v.option2_value, v.option3_value"
" FROM product_image i JOIN product p ON p.id = i.product_id"
" LEFT JOIN variant v ON v.image_id = i.id"
" WHERE i.import_run_id = %s"
" AND i.status IN ('rejected_low_res', 'rejected_not_image', 'failed')"
" ORDER BY p.handle, i.position, i.id",
(run_id,),
).fetchall()
return [
{"handle": r[0], "url": r[1], "outcome": r[2], "reason": r[3],
"variant": " / ".join(x for x in (r[4], r[5], r[6]) if x) or None}
for r in rows
]
# ---------------------------------------------------------------------------
# Apply primitives — called inside the confirm transaction (Task 8); no commits.
# ---------------------------------------------------------------------------
def insert_product(
conn: psycopg.Connection,
storefront_id: int,
handle: str,
resolved_fields: dict,
option_names: tuple[str | None, str | None, str | None],
) -> int:
"""INSERT with only the file-present fields; absent ones take column defaults."""
columns = ["storefront_id", "handle", "option1_name", "option2_name", "option3_name"]
values: list[object] = [storefront_id, handle, *option_names]
for field_name, value in resolved_fields.items():
columns.append(field_name)
values.append(value)
query = sql.SQL("INSERT INTO product ({}) VALUES ({}) RETURNING id").format(
sql.SQL(", ").join(sql.Identifier(c) for c in columns),
sql.SQL(", ").join(sql.Placeholder() for _ in columns),
)
return conn.execute(query, values).fetchone()[0]
def update_product(conn: psycopg.Connection, product_id: int, changed_fields: dict) -> None:
if not changed_fields:
return
assignments = [
sql.SQL("{} = {}").format(sql.Identifier(f), sql.Placeholder())
for f in changed_fields
]
query = sql.SQL("UPDATE product SET {}, updated_at = now() WHERE id = {}").format(
sql.SQL(", ").join(assignments), sql.Placeholder()
)
conn.execute(query, [*changed_fields.values(), product_id])
def insert_variant(
conn: psycopg.Connection,
product_id: int,
position: int,
options: tuple[str | None, str | None, str | None],
resolved_fields: dict,
image_id: int | None,
) -> int:
# variant_image is not a column — the caller translates it to image_id; position
# is the explicit param. Filter both defensively.
fields = {
k: v for k, v in resolved_fields.items() if k != "variant_image" and k != "position"
}
columns = [
"product_id",
"position",
"option1_value",
"option2_value",
"option3_value",
"image_id",
]
values: list[object] = [product_id, position, *options, image_id]
for field_name, value in fields.items():
columns.append(field_name)
values.append(value)
query = sql.SQL("INSERT INTO variant ({}) VALUES ({}) RETURNING id").format(
sql.SQL(", ").join(sql.Identifier(c) for c in columns),
sql.SQL(", ").join(sql.Placeholder() for _ in columns),
)
return conn.execute(query, values).fetchone()[0]
def update_variant(
conn: psycopg.Connection,
variant_id: int,
changed_fields: dict,
image_id: int | None | type(...) = ...,
) -> None:
"""Dynamic UPDATE; image_id's Ellipsis default means "don't touch image_id"."""
fields = {
k: v for k, v in changed_fields.items() if k != "variant_image"
}
assignments = [
sql.SQL("{} = {}").format(sql.Identifier(f), sql.Placeholder()) for f in fields
]
values: list[object] = list(fields.values())
if image_id is not ...:
assignments.append(sql.SQL("image_id = {}").format(sql.Placeholder()))
values.append(image_id)
if not assignments:
return
query = sql.SQL("UPDATE variant SET {}, updated_at = now() WHERE id = {}").format(
sql.SQL(", ").join(assignments), sql.Placeholder()
)
conn.execute(query, [*values, variant_id])
def get_or_create_image(
conn: psycopg.Connection,
product_id: int,
source_url: str,
position: int,
alt_text: str | None,
run_id: int,
) -> int:
"""Image identity within a product is source_url (§6.3); existing rows are
returned untouched — diff emits explicit image update entries for position/alt."""
row = conn.execute(
"SELECT id FROM product_image WHERE product_id = %s AND source_url = %s",
(product_id, source_url),
).fetchone()
if row is not None:
return row[0]
return conn.execute(
"INSERT INTO product_image (product_id, source_url, position, alt_text, import_run_id)"
" VALUES (%s, %s, %s, %s, %s) RETURNING id",
(product_id, source_url, position, alt_text, run_id),
).fetchone()[0]
def update_image(conn: psycopg.Connection, image_id: int, changed_fields: dict) -> None:
"""Subset of {position, alt_text}."""
if not changed_fields:
return
assignments = [
sql.SQL("{} = {}").format(sql.Identifier(f), sql.Placeholder())
for f in changed_fields
]
query = sql.SQL("UPDATE product_image SET {} WHERE id = {}").format(
sql.SQL(", ").join(assignments), sql.Placeholder()
)
conn.execute(query, [*changed_fields.values(), image_id])
+6
View File
@@ -0,0 +1,6 @@
Handle,Title,Description,Vendor,Type,Google Product Category,Tags,Status,Published,Option1 Name,Option1 Value,Option2 Name,Option2 Value,Variant SKU,Variant Price,Variant Inventory Qty,Image Src,Image Position,Image Alt Text
moon-mug,Moon Mug,"<p>A ceramic mug glazed in moonlight grey.</p>",Wiggle Goods,standalone,Home & Garden > Kitchen & Dining,"kitchen, mugs",active,TRUE,,,,,WG-MUG-001,18.00,40,https://images.example.com/moon-mug.jpg,1,Moon Mug on a desk
star-tee,Star Tee,"<p>Soft cotton tee with a hand-printed star.</p>",Wiggle Goods,standalone,Apparel & Accessories > Clothing,"apparel, tees",active,TRUE,Size,S,Color,Indigo,WG-TEE-S,24.00,12,https://images.example.com/star-tee.jpg,1,Star Tee flat lay
star-tee,,,,,,,,,,M,,Indigo,WG-TEE-M,24.00,18,,,
star-tee,,,,,,,,,,L,,Indigo,WG-TEE-L,26.00,9,,,
star-tee,,,,,,,,,,,,,,,,https://images.example.com/star-tee-back.jpg,2,Star Tee back print
1 Handle Title Description Vendor Type Google Product Category Tags Status Published Option1 Name Option1 Value Option2 Name Option2 Value Variant SKU Variant Price Variant Inventory Qty Image Src Image Position Image Alt Text
2 moon-mug Moon Mug <p>A ceramic mug glazed in moonlight grey.</p> Wiggle Goods standalone Home & Garden > Kitchen & Dining kitchen, mugs active TRUE WG-MUG-001 18.00 40 https://images.example.com/moon-mug.jpg 1 Moon Mug on a desk
3 star-tee Star Tee <p>Soft cotton tee with a hand-printed star.</p> Wiggle Goods standalone Apparel & Accessories > Clothing apparel, tees active TRUE Size S Color Indigo WG-TEE-S 24.00 12 https://images.example.com/star-tee.jpg 1 Star Tee flat lay
4 star-tee M Indigo WG-TEE-M 24.00 18
5 star-tee L Indigo WG-TEE-L 26.00 9
6 star-tee https://images.example.com/star-tee-back.jpg 2 Star Tee back print
+125
View File
@@ -0,0 +1,125 @@
"""Canonical serializer — CatalogProduct snapshot → canonical CSV (SD-0002 §6.5.5).
The export half of "one codec, two directions": this writes exactly the columns
codec.py/validate.py parse, in a row grammar (§6.5.1) the validator regroups
identically — so re-importing an unmodified export diffs to nothing (INV-12).
DB-free, like diff.py: it consumes the same CatalogProduct snapshot the diff
engine reads (repo.load_catalog), and the service streams the result.
"""
from __future__ import annotations
import csv
import io
from collections.abc import Iterable, Iterator
from decimal import Decimal
from .diff import CatalogProduct
from .hosted import EXPORT_RENDITION, image_url
from .models import (
IMAGE_COLUMNS,
OPTION_VALUE_COLUMNS,
PRODUCT_COLUMNS,
VARIANT_COLUMNS,
)
# Canonical column order: Handle, then product, option-value, variant, image
# columns. A superset of everything the parser knows (models.KNOWN_COLUMNS minus
# the #15-reserved Component columns, which export never emits).
HEADER: list[str] = [
"Handle",
*PRODUCT_COLUMNS,
*OPTION_VALUE_COLUMNS,
*VARIANT_COLUMNS,
*IMAGE_COLUMNS,
]
# field name -> column name, inverting the model registry (the parser reads
# column->field; the serializer writes field->column).
_PRODUCT_FIELD_TO_COL = {field: col for col, field in PRODUCT_COLUMNS.items()}
_VARIANT_FIELD_TO_COL = {field: col for col, field in VARIANT_COLUMNS.items()}
def _cell(value: object) -> str:
"""Serialize one value to its canonical cell text (the parse inverse)."""
if value is None:
return ""
if isinstance(value, bool):
return "TRUE" if value else "FALSE"
if isinstance(value, Decimal):
return str(value)
if isinstance(value, (list, tuple)):
return ", ".join(str(v) for v in value)
return str(value)
def catalog_to_csv(products: Iterable[CatalogProduct], base_url: str = "") -> Iterator[str]:
"""Stream canonical CSV text, header first, one product block at a time."""
buf = io.StringIO()
writer = csv.writer(buf)
writer.writerow(HEADER)
yield _drain(buf)
for product in products:
for row in _product_rows(product, base_url):
writer.writerow([row.get(col, "") for col in HEADER])
yield _drain(buf)
def _drain(buf: io.StringIO) -> str:
text = buf.getvalue()
buf.seek(0)
buf.truncate(0)
return text
def _product_rows(product: CatalogProduct, base_url: str = "") -> list[dict[str, str]]:
"""The product's CSV rows (§6.5.1 grammar): product fields + option names on
the first row; one variant per row; images interleaved; image-only rows when
a product has more images than variants."""
base = {"Handle": product.handle, "Title": product.title}
for field, col in _PRODUCT_FIELD_TO_COL.items():
if field in product.fields:
base[col] = _cell(product.fields[field])
for slot, name in enumerate(product.option_names, start=1):
if name:
base[f"Option{slot} Name"] = name
count = max(len(product.variants), len(product.images))
rows: list[dict[str, str]] = []
for i in range(count):
# Row 0 carries the product-level fields; later rows carry only Handle.
row = dict(base) if i == 0 else {"Handle": product.handle}
if i < len(product.variants):
_write_variant(row, product, product.variants[i])
if i < len(product.images):
_write_image(row, product.images[i], base_url)
rows.append(row)
return rows
def _write_image(row: dict[str, str], image, base_url: str) -> None:
# Fetched images export the platform-hosted URL (INV-12/16); everything
# else keeps the original source_url so nothing is lost (§6.5.5).
if image.status == "fetched":
row["Image Src"] = image_url(base_url, image.id, EXPORT_RENDITION)
else:
row["Image Src"] = image.source_url
row["Image Position"] = str(image.position)
if image.alt_text is not None:
row["Image Alt Text"] = image.alt_text
def _write_variant(row: dict[str, str], product: CatalogProduct, variant) -> None:
"""Fill a row's option-value + variant columns for one variant."""
for slot, value in enumerate(variant.options, start=1):
# Only emit an option value when the product actually has that option
# (a no-option product's single variant carries all-NULL options).
if product.option_names[slot - 1] and value is not None:
row[f"Option{slot} Value"] = value
# position is a CatalogVariant attribute, not a fields{} entry — emit it
# explicitly. An empty Variant Position cell re-imports as "reset to file
# order", so a non-sequential stored position would round-trip to an update
# (INV-12). (Like _write_image, which emits its position attribute.)
row["Variant Position"] = str(variant.position)
for field, col in _VARIANT_FIELD_TO_COL.items():
if field in variant.fields:
row[col] = _cell(variant.fields[field])
+265
View File
@@ -0,0 +1,265 @@
"""products service — the import/export use-case orchestration (SD-0002 §6.5).
Coordinates codec → validate → diff → repo; owns transaction boundaries (repo
never commits). Preview is read-only against catalog tables (INV-11): validation
writes exactly one row — the import_draft. TEL events per §9.1.
"""
from __future__ import annotations
import time
from collections.abc import Iterator
from datetime import datetime, timezone
import psycopg
from app.platform import config, telemetry
from . import codec, diff, repo, serialize, validate
from .errors import (
DraftExpired,
DraftNotFound,
EmptyCatalog,
NothingToApply,
PreviewStale,
RunNotFound,
)
def import_validate(conn: psycopg.Connection, storefront_id: int, account_id: int,
file_name: str, data: bytes) -> dict:
"""Upload → validate → diff → persist draft (PUC-2/3; INV-11). Raises FileRejected."""
started = time.monotonic()
# Commit the sweep before parsing: a FileRejected mid-parse must not roll
# back expired-draft cleanup along with it.
repo.sweep_expired_drafts(conn)
conn.commit()
parsed = codec.parse_csv(data)
products = validate.build_products(parsed)
catalog = repo.load_catalog(conn, storefront_id)
diff_result = diff.compute_diff(catalog, products)
draft = repo.insert_draft(
conn, storefront_id, account_id, file_name, parsed.dialect, data,
diff_result.summary, diff_result.records, diff_result.fingerprint,
parsed.unknown_columns,
)
conn.commit()
telemetry.emit(
"import_draft_created",
storefront_id=storefront_id,
dialect=parsed.dialect,
row_count=len(parsed.rows),
adds=diff_result.summary["adds"],
updates=diff_result.summary["updates"],
unchanged=diff_result.summary["unchanged"],
errors=diff_result.summary["errors"],
unknown_columns_count=len(parsed.unknown_columns),
duration_ms=int((time.monotonic() - started) * 1000),
)
return draft
def _live_draft_row(conn: psycopg.Connection, storefront_id: int, draft_id: int) -> dict:
"""The draft row if it exists and hasn't expired; expiry deletes lazily (§6.3)."""
row = repo.get_draft_row(conn, storefront_id, draft_id)
if row is None:
raise DraftNotFound()
if row["expires_at"] < datetime.now(timezone.utc):
repo.delete_draft(conn, storefront_id, draft_id)
conn.commit()
raise DraftExpired()
return row
def get_draft(conn: psycopg.Connection, storefront_id: int, draft_id: int) -> dict:
"""The §6.4 draft payload — never file_bytes or the full records list."""
row = _live_draft_row(conn, storefront_id, draft_id)
return {
"id": row["id"],
"file_name": row["file_name"],
"dialect": row["dialect"],
"summary": row["summary"],
"unknown_columns": row["unknown_columns"],
"expires_at": row["expires_at"].isoformat(),
}
def get_draft_records(conn: psycopg.Connection, storefront_id: int, draft_id: int,
kind: str | None = None, limit: int = 100, offset: int = 0) -> list[dict]:
"""The draft's preview records, paged, optionally filtered by kind (PUC-3)."""
_live_draft_row(conn, storefront_id, draft_id)
return repo.draft_records(conn, storefront_id, draft_id, kind, limit, offset)
def discard_draft(conn: psycopg.Connection, storefront_id: int, draft_id: int) -> None:
"""Delete the draft, no trace kept; idempotent — an absent draft is fine (PUC-3a)."""
repo.delete_draft(conn, storefront_id, draft_id)
conn.commit()
def confirm_draft(conn: psycopg.Connection, storefront_id: int, account_id: int,
draft_id: int) -> int:
"""Apply the previewed diff in one transaction (PUC-4; INV-10/11).
Everything is re-derived from the draft's stored file bytes against the live
catalog; a fingerprint mismatch means the catalog drifted since preview
(PreviewStale — the draft is kept so the merchant can re-validate). The apply
executes the typed plan compute_diff built alongside the preview records, so
what lands is exactly what the preview showed. rows_errored counts the
import_run_error rows recorded (one per RowError), which is what the run
detail's error table shows; the preview's errors tile counts error *products*.
"""
started = time.monotonic()
row = _live_draft_row(conn, storefront_id, draft_id)
parsed = codec.parse_csv(row["file_bytes"])
products = validate.build_products(parsed)
catalog = repo.load_catalog(conn, storefront_id)
diff_result = diff.compute_diff(catalog, products)
if diff_result.fingerprint != row["fingerprint"]:
# Release the read snapshot; nothing written.
conn.rollback()
raise PreviewStale()
summary_counts = diff_result.summary
if summary_counts["adds"] + summary_counts["updates"] == 0:
# Release the read snapshot; nothing written.
conn.rollback()
raise NothingToApply()
error_rows = [
error.as_json()
for plan in diff_result.plan if plan.kind == "error"
for error in plan.canonical.errors
]
try:
run_id = repo.insert_run(
conn, storefront_id, account_id, row["file_name"], row["dialect"],
added=summary_counts["adds"], updated=summary_counts["updates"],
errored=len(error_rows), status="applying",
)
for plan in diff_result.plan:
_apply_product_plan(conn, storefront_id, plan, run_id)
repo.insert_run_errors(conn, run_id, error_rows)
pending = repo.run_image_counts(conn, run_id)["pending"]
repo.set_run_status(conn, run_id, "fetching_images" if pending else "complete")
repo.delete_draft(conn, storefront_id, draft_id)
conn.commit()
except Exception as exc:
conn.rollback()
telemetry.emit(
"import_apply_failed",
draft_id=draft_id,
storefront_id=storefront_id,
error_class=type(exc).__name__,
)
raise
telemetry.emit(
"import_run_completed",
run_id=run_id,
storefront_id=storefront_id,
added=summary_counts["adds"],
updated=summary_counts["updates"],
errored=len(error_rows),
duration_ms=int((time.monotonic() - started) * 1000),
)
return run_id
def _apply_product_plan(conn: psycopg.Connection, storefront_id: int,
plan: diff.ProductPlan, run_id: int) -> None:
"""Execute one product's plan inside the confirm transaction (no commits here)."""
if plan.kind == "add":
# Title is a canonical attribute, not a fields{} entry — non-error
# products always carry one (validate guarantees it).
product_fields = {"title": plan.canonical.title}
product_fields.update(diff.resolved_product_fields(plan.canonical))
product_id = repo.insert_product(
conn, storefront_id, plan.canonical.handle, product_fields, plan.canonical.option_names
)
image_ids: dict[str, int] = {}
elif plan.kind == "update":
product_id = plan.catalog.id
repo.update_product(conn, product_id, plan.product_changes)
image_ids = {image.source_url: image.id for image in plan.catalog.images}
else:
return
# Images first, so variants' variant_image URLs resolve to ids: validate puts
# every variant_image URL into canonical.images, so each URL is in either the
# catalog map (existing image) or the adds below.
for image_plan in plan.image_plans:
if image_plan.kind == "add":
image_ids[image_plan.source_url] = repo.get_or_create_image(
conn, product_id, image_plan.source_url, image_plan.position,
image_plan.alt_text, run_id,
)
else:
repo.update_image(conn, image_plan.image_id, image_plan.changes)
for variant_plan in plan.variant_plans:
if variant_plan.kind == "add":
fields = diff.resolved_variant_fields(variant_plan.canonical, variant_plan.file_order)
# diff time resolved any cleared position to file order; the
# file_order fallback covers an absent position column.
position = fields.get("position") or variant_plan.file_order
url = fields.get("variant_image")
image_id = image_ids[url] if url else None
repo.insert_variant(
conn, product_id, position, variant_plan.canonical.options, fields, image_id
)
elif "variant_image" in variant_plan.changes:
url = variant_plan.changes["variant_image"]
repo.update_variant(
conn, variant_plan.catalog_id, variant_plan.changes,
image_id=image_ids[url] if url else None,
)
else:
repo.update_variant(conn, variant_plan.catalog_id, variant_plan.changes)
def list_runs(conn: psycopg.Connection, storefront_id: int,
limit: int = 50, offset: int = 0) -> list[dict]:
"""The storefront's import history, newest first (PUC-8)."""
return repo.list_runs(conn, storefront_id, limit, offset)
def get_run(conn: psycopg.Connection, storefront_id: int, run_id: int) -> dict:
"""One run's §6.4 detail payload, errors included."""
run = repo.get_run(conn, storefront_id, run_id)
if run is None:
raise RunNotFound()
return run
def export_catalog(
conn: psycopg.Connection, storefront_id: int, status_filter: str
) -> Iterator[str]:
"""Stream the storefront's catalog as canonical CSV (PUC-9; INV-12 codec).
Read-only: builds the snapshot, then streams the serializer over it. TEL-3
is emitted once the stream is exhausted, with the product count and elapsed
time. Raises EmptyCatalog before yielding anything if the (filtered) catalog
is empty, so the BFF can answer 409 cleanly with no partial body.
"""
started = time.monotonic()
snapshot = repo.export_catalog(conn, storefront_id, status_filter)
if not snapshot:
raise EmptyCatalog()
def _stream() -> Iterator[str]:
yield from serialize.catalog_to_csv(snapshot, base_url=config.public_base_url())
telemetry.emit(
"catalog_exported",
storefront_id=storefront_id,
status_filter=status_filter,
product_count=len(snapshot),
duration_ms=int((time.monotonic() - started) * 1000),
)
return _stream()
def summary(conn: psycopg.Connection, storefront_id: int) -> dict:
"""The products dashboard counts (§6.4)."""
return {
"product_count": repo.product_count(conn, storefront_id),
"image_problem_count": repo.image_problem_count(conn, storefront_id),
"latest_run_id": repo.latest_run_id(conn, storefront_id),
}
+299
View File
@@ -0,0 +1,299 @@
"""Row validation — ParsedFile rows → canonical products + row errors (SD-0002 §6.5.1).
The codec (codec.py) handles file-level gates; this module is the row-semantics
half of the PUC-5 import spine. It groups consecutive rows sharing a Handle into
product blocks (Shopify's grammar), normalizes product/variant/image fields, and
records every rule violation as a merchant-language RowError. Errors never raise:
an error poisons its whole product block (the product previews as kind="error"
and is excluded from apply) while parsing continues so the merchant gets a
complete accounting in one pass (BUC-1a). Description HTML is sanitized with nh3
on the way in (INV-15).
"""
from __future__ import annotations
import re
from decimal import Decimal, InvalidOperation
import nh3
from .models import (
COMPONENT_COLUMNS,
OPTION_VALUE_COLUMNS,
PRODUCT_COLUMNS,
VARIANT_COLUMNS,
CanonicalImage,
CanonicalProduct,
CanonicalVariant,
ParsedFile,
Row,
RowError,
)
_HANDLE_RE = re.compile(r"^[a-z0-9-]+$")
_STATUSES = {"draft", "active", "archived"}
# Title is handled as an attribute, not via fields{}. Option names live in BOTH
# .option_names (the values) and fields{} (the file-presence signal the diff
# needs: absent column == untouched, never a clear).
_ATTRIBUTE_COLUMNS = {"Title"}
# Sentinel: the cell failed normalization (the error is already recorded).
_INVALID = object()
def build_products(parsed: ParsedFile) -> list[CanonicalProduct]:
"""Group rows into product blocks and validate every §6.5.1 rule."""
products: list[CanonicalProduct] = []
closed_handles: set[str] = set()
block: list[Row] = []
def flush() -> None:
nonlocal block
if block:
closed_handles.add(block[0].cells["Handle"])
products.append(_build_block(block))
block = []
for row in parsed.rows:
handle = row.cells.get("Handle", "")
if block and handle == block[0].cells["Handle"]:
block.append(row)
continue
flush()
if not handle:
products.append(
_error_block(row, "(missing)", RowError(row.line_number, "Handle", "a row needs a Handle"))
)
elif not _HANDLE_RE.match(handle):
products.append(
_error_block(
row,
handle,
RowError(
row.line_number,
"Handle",
f"'{handle}' isn't a valid handle — lowercase letters, numbers, and dashes only",
),
)
)
elif handle in closed_handles:
products.append(
_error_block(
row,
handle,
RowError(
row.line_number,
"Handle",
f"rows for '{handle}' must be consecutive — it already appeared earlier in the file",
),
)
)
else:
block = [row]
flush()
return products
def all_errors(products: list[CanonicalProduct]) -> list[RowError]:
return [error for product in products for error in product.errors]
def _error_block(row: Row, handle: str, error: RowError) -> CanonicalProduct:
return CanonicalProduct(first_line=row.line_number, handle=handle, title="", errors=[error])
def _build_block(rows: list[Row]) -> CanonicalProduct:
first = rows[0]
handle = first.cells["Handle"]
errors: list[RowError] = []
title = first.cells.get("Title", "")
if not title:
errors.append(RowError(first.line_number, "Title", f"'{handle}' is missing its Title"))
option_names = tuple(first.cells.get(f"Option{n} Name") or None for n in (1, 2, 3))
has_options = any(option_names)
# Product-level fields come from the first row only; empty cell == clear (None).
fields: dict[str, object] = {}
for column, field_name in PRODUCT_COLUMNS.items():
if column in _ATTRIBUTE_COLUMNS or column not in first.cells:
continue
cell = first.cells[column]
if not cell:
fields[field_name] = None
continue
value = _product_value(first.line_number, column, field_name, cell, errors)
if value is not _INVALID:
fields[field_name] = value
variants: list[CanonicalVariant] = []
images: list[CanonicalImage] = []
seen_combos: set[tuple[str | None, str | None, str | None]] = set()
for index, row in enumerate(rows):
cells = row.cells
line = row.line_number
for column in COMPONENT_COLUMNS:
if cells.get(column):
errors.append(
RowError(line, column, "kits arrive in a coming release — leave the Component columns empty")
)
has_image = bool(cells.get("Image Src"))
if has_image:
_collect_image(row, images, errors)
# A row carries a variant iff any option value / Variant-* cell is filled;
# the first row of a no-option product always carries the single variant.
carries_variant = (
any(cells.get(c) for c in OPTION_VALUE_COLUMNS)
or any(cells.get(c) for c in VARIANT_COLUMNS)
or (index == 0 and not has_options)
)
if carries_variant:
options = tuple(cells.get(c) or None for c in OPTION_VALUE_COLUMNS)
for n in (1, 2, 3):
value, name = options[n - 1], option_names[n - 1]
if value and not name:
errors.append(
RowError(
line,
f"Option{n} Value",
f"Option{n} Value given but the product has no Option{n} Name",
)
)
elif name and not value:
errors.append(
RowError(
line,
f"Option{n} Value",
f"this variant is missing its Option{n} Value ('{name}')",
)
)
if not has_options:
if variants:
errors.append(RowError(line, None, "a product without options can have only one variant"))
elif options in seen_combos:
errors.append(
RowError(line, None, f"duplicate variant — '{handle}' already has a variant with these options")
)
seen_combos.add(options)
variant_fields: dict[str, object] = {}
for column, field_name in VARIANT_COLUMNS.items():
if column not in cells:
continue
cell = cells[column]
if not cell:
variant_fields[field_name] = None
continue
value = _variant_value(line, column, field_name, cell, errors)
if value is _INVALID:
continue
variant_fields[field_name] = value
if field_name == "variant_image" and cell not in {i.source_url for i in images}:
images.append(
CanonicalImage(line_number=line, source_url=cell, position=len(images) + 1, alt_text=None)
)
variants.append(CanonicalVariant(line_number=line, options=options, fields=variant_fields))
elif index > 0 and not has_image:
errors.append(RowError(line, None, "this row has no variant or image data"))
return CanonicalProduct(
first_line=first.line_number,
handle=handle,
title=title,
option_names=option_names,
fields=fields,
variants=variants,
images=images,
errors=errors,
)
def _collect_image(row: Row, images: list[CanonicalImage], errors: list[RowError]) -> None:
cells = row.cells
source_url = cells["Image Src"]
position_cell = cells.get("Image Position", "")
position: int | None = None
if position_cell:
try:
position = int(position_cell)
if position < 1:
raise ValueError
except ValueError:
position = None
errors.append(RowError(row.line_number, "Image Position", f"'{position_cell}' is not a position"))
# Dedupe by source URL within the block — first occurrence wins.
if source_url in {i.source_url for i in images}:
return
images.append(
CanonicalImage(
line_number=row.line_number,
source_url=source_url,
position=position if position is not None else len(images) + 1,
alt_text=cells.get("Image Alt Text") or None,
)
)
def _product_value(line: int, column: str, field_name: str, cell: str, errors: list[RowError]) -> object:
if field_name == "tags":
return [tag.strip() for tag in cell.split(",") if tag.strip()]
if field_name == "status":
status = cell.lower()
if status not in _STATUSES:
errors.append(RowError(line, column, f"'{cell}' is not a status — use draft, active, or archived"))
return _INVALID
return status
if field_name == "published":
flag = cell.upper()
if flag not in ("TRUE", "FALSE"):
errors.append(RowError(line, column, f"'{cell}' is not TRUE or FALSE"))
return _INVALID
return flag == "TRUE"
if field_name == "description_html":
return nh3.clean(cell)
if field_name == "product_type":
if cell != "standalone":
errors.append(RowError(line, column, "kits arrive in a coming release — Type must be 'standalone'"))
return _INVALID
return cell
return cell
def _variant_value(line: int, column: str, field_name: str, cell: str, errors: list[RowError]) -> object:
if field_name in ("price", "cost"):
return _decimal_or_error(line, column, cell, f"'{cell}' is not a price", errors)
if field_name in ("weight", "volume"):
return _decimal_or_error(line, column, cell, f"'{cell}' is not a number", errors)
if field_name == "inventory_qty":
try:
quantity = int(cell)
if quantity < 0:
raise ValueError
except ValueError:
errors.append(RowError(line, column, f"'{cell}' is not a whole number"))
return _INVALID
return quantity
if field_name == "position":
try:
position = int(cell)
if position < 1:
raise ValueError
except ValueError:
errors.append(RowError(line, column, f"'{cell}' is not a position"))
return _INVALID
return position
return cell
def _decimal_or_error(line: int, column: str, cell: str, message: str, errors: list[RowError]) -> object:
try:
value = Decimal(cell)
if not value.is_finite() or value < 0:
raise InvalidOperation
except InvalidOperation:
errors.append(RowError(line, column, message))
return _INVALID
return value
+255 -5
View File
@@ -4,25 +4,29 @@ SLICE-1 mounted /healthz; SLICE-2 adds the /api/auth/* identity endpoints (§6.4
translates HTTP <-> domain calls and owns no business logic (INV-6): every rule lives in
the accounts domain. create_app() opens the pool, self-migrates (INV-1, INV-7), and builds
the configured mailer (INV-8) at startup. SLICE-3 adds POST /api/storefronts and feeds the
_storefront_for seam from the storefronts domain.
_storefront_for seam from the storefronts domain. SLICE-5 adds the /api/products/* import
spine (SD-0002 §6.4): each endpoint is a gate + one products-domain call + error mapping.
"""
from __future__ import annotations
import logging
import sys
from concurrent.futures import ThreadPoolExecutor
from contextlib import asynccontextmanager
from pathlib import Path
from typing import Any
import psycopg
from fastapi import Depends, FastAPI, Response
from fastapi.responses import JSONResponse
from fastapi import Depends, FastAPI, File, Query, Response, UploadFile
from fastapi import Path as ApiPath
from fastapi.responses import JSONResponse, PlainTextResponse, StreamingResponse
from fastapi.staticfiles import StaticFiles
from pydantic import BaseModel
from app.domains import accounts, storefronts
from app.domains import accounts, products, storefronts
from app.platform import config, db
from app.platform import mailer as mailer_mod
from app.platform import objectstore as objectstore_mod
from app.platform.deps import SESSION_COOKIE, get_conn, get_mailer, get_session
from app.platform.mailer import Mailer
from app.platform import session as session_mod
@@ -61,6 +65,26 @@ def _storefront_for(conn: psycopg.Connection, account: accounts.Account) -> dict
return {"id": sf.id, "name": sf.name} if sf else None
def _merchant_gate(
conn: psycopg.Connection, sess: dict | None
) -> JSONResponse | tuple[accounts.Account, storefronts.Storefront]:
"""The shared /api/products/* gate: a signed-in account that has its storefront.
Returns the (account, storefront) pair, or the ready-to-return error response —
401 with no session, 404 before the storefront exists (INV-14: every products
call is storefront-scoped, so there is nothing to address yet).
"""
if sess is None:
return _error(401, "unauthenticated", "You are not signed in.")
account = accounts.get_account(conn, sess["account_id"])
if account is None:
return _error(401, "unauthenticated", "You are not signed in.")
sf = storefronts.storefront_for(conn, account.id)
if sf is None:
return _error(404, "no_storefront", "Create your storefront first.")
return account, sf
def _ensure_app_logging() -> None:
"""Surface the app's own `ecomm.*` INFO logs on stderr (idempotent).
@@ -96,13 +120,22 @@ def create_app(database_url: str | None = None, static_dir: str | Path | None =
@asynccontextmanager
async def lifespan(app: FastAPI):
app.state.pool = db.open_pool(dsn)
app.state.pool = db.open_pool(dsn, max_size=10)
with app.state.pool.connection() as conn:
db.migrate(conn) # self-migrate at startup (INV-1, INV-7)
app.state.mailer = mailer_mod.build_mailer(config.mailer_kind()) # INV-8
app.state.objectstore = objectstore_mod.build_objectstore(config.objectstore_kind())
app.state.image_allow_private = config.image_fetch_allow_private()
app.state.image_runner = ThreadPoolExecutor(max_workers=2, thread_name_prefix="img-run")
# §6.9 startup recovery — resume runs stuck mid image-fetch, off the request path.
app.state.image_runner.submit(
products.recover_incomplete_runs, app.state.pool,
app.state.objectstore, app.state.image_allow_private,
)
try:
yield
finally:
app.state.image_runner.shutdown(wait=False)
app.state.pool.close()
app = FastAPI(title="ecomm", version=_APP_VERSION, lifespan=lifespan)
@@ -208,6 +241,223 @@ def create_app(database_url: str | None = None, static_dir: str | Path | None =
)
return JSONResponse(status_code=201, content={"id": sf.id, "name": sf.name})
@app.post("/api/products/imports")
async def import_upload(
file: UploadFile = File(...),
conn: psycopg.Connection = Depends(get_conn),
sess: dict | None = Depends(get_session),
):
"""Upload a CSV → validated import draft (§6.4; PUC-2, PUC-5/5a on rejection)."""
gate = _merchant_gate(conn, sess)
if isinstance(gate, JSONResponse):
return gate
account, sf = gate
data = await file.read()
if len(data) > products.MAX_FILE_BYTES:
return _error(413, "file_too_large", "This file is larger than 10 MB.")
try:
draft = products.import_validate(conn, sf.id, account.id, file.filename or "upload.csv", data)
except products.FileRejected as exc:
return _error(400, exc.code, exc.message)
return JSONResponse(status_code=201, content=draft)
@app.get("/api/products/imports/drafts/{draft_id}")
def get_import_draft(
draft_id: int,
conn: psycopg.Connection = Depends(get_conn),
sess: dict | None = Depends(get_session),
):
"""One draft's preview payload — summary, never the file bytes (§6.4; PUC-3)."""
gate = _merchant_gate(conn, sess)
if isinstance(gate, JSONResponse):
return gate
_account, sf = gate
try:
return products.get_draft(conn, sf.id, draft_id)
except products.DraftNotFound:
return _error(404, "not_found", "No such import preview.")
except products.DraftExpired:
return _error(410, "draft_expired", "This preview expired — upload the file again.")
@app.get("/api/products/imports/drafts/{draft_id}/records")
def get_import_draft_records(
draft_id: int,
kind: str | None = Query(default=None, pattern="^(add|update|unchanged|error)$"),
limit: int = Query(default=100, ge=1, le=500),
offset: int = Query(default=0, ge=0),
conn: psycopg.Connection = Depends(get_conn),
sess: dict | None = Depends(get_session),
):
"""The draft's per-product preview records, paged + kind-filtered (§6.4; PUC-3)."""
gate = _merchant_gate(conn, sess)
if isinstance(gate, JSONResponse):
return gate
_account, sf = gate
try:
records = products.get_draft_records(conn, sf.id, draft_id, kind, limit, offset)
except products.DraftNotFound:
return _error(404, "not_found", "No such import preview.")
except products.DraftExpired:
return _error(410, "draft_expired", "This preview expired — upload the file again.")
return {"records": records}
@app.post("/api/products/imports/drafts/{draft_id}/confirm")
def confirm_import_draft(
draft_id: int,
conn: psycopg.Connection = Depends(get_conn),
sess: dict | None = Depends(get_session),
):
"""Apply the previewed diff as one import run (§6.4; PUC-4, INV-10/11)."""
gate = _merchant_gate(conn, sess)
if isinstance(gate, JSONResponse):
return gate
account, sf = gate
try:
run_id = products.confirm_draft(conn, sf.id, account.id, draft_id)
except products.DraftNotFound:
return _error(404, "not_found", "No such import preview.")
except products.DraftExpired:
return _error(410, "draft_expired", "This preview expired — upload the file again.")
except products.PreviewStale:
return _error(
409, "preview_stale",
"Your catalog changed since this preview — upload the file again.",
)
except products.NothingToApply:
return _error(
409, "nothing_to_apply",
"Nothing to change — your catalog already matches this file.",
)
app.state.image_runner.submit(
products.run_image_phase, app.state.pool, app.state.objectstore,
run_id, app.state.image_allow_private,
)
return JSONResponse(status_code=201, content={"run_id": run_id})
@app.delete("/api/products/imports/drafts/{draft_id}")
def discard_import_draft(
draft_id: int,
conn: psycopg.Connection = Depends(get_conn),
sess: dict | None = Depends(get_session),
):
"""Discard the draft, no trace kept; idempotent (§6.4; PUC-3a)."""
gate = _merchant_gate(conn, sess)
if isinstance(gate, JSONResponse):
return gate
_account, sf = gate
products.discard_draft(conn, sf.id, draft_id)
return Response(status_code=204)
@app.get("/api/products/imports/runs")
def list_import_runs(
limit: int = Query(default=50, ge=1, le=200),
offset: int = Query(default=0, ge=0),
conn: psycopg.Connection = Depends(get_conn),
sess: dict | None = Depends(get_session),
):
"""The storefront's import history, newest first (§6.4; PUC-8)."""
gate = _merchant_gate(conn, sess)
if isinstance(gate, JSONResponse):
return gate
_account, sf = gate
return {"runs": products.list_runs(conn, sf.id, limit, offset)}
@app.get("/api/products/imports/runs/{run_id}")
def get_import_run(
run_id: int,
conn: psycopg.Connection = Depends(get_conn),
sess: dict | None = Depends(get_session),
):
"""One run's detail payload, errors included (§6.4; PUC-8)."""
gate = _merchant_gate(conn, sess)
if isinstance(gate, JSONResponse):
return gate
_account, sf = gate
try:
return products.get_run(conn, sf.id, run_id)
except products.RunNotFound:
return _error(404, "not_found", "No such import run.")
@app.get("/api/products/summary")
def products_summary(
conn: psycopg.Connection = Depends(get_conn),
sess: dict | None = Depends(get_session),
):
"""The products dashboard counts (§6.4)."""
gate = _merchant_gate(conn, sess)
if isinstance(gate, JSONResponse):
return gate
_account, sf = gate
return products.summary(conn, sf.id)
@app.get("/api/products/export")
def export_products(
status: str = Query(default="all", pattern="^(all|active|draft|archived)$"),
conn: psycopg.Connection = Depends(get_conn),
sess: dict | None = Depends(get_session),
):
"""Stream the catalog as canonical CSV, optionally status-filtered (§6.4; PUC-9)."""
gate = _merchant_gate(conn, sess)
if isinstance(gate, JSONResponse):
return gate
_account, sf = gate
try:
stream = products.export_catalog(conn, sf.id, status)
except products.EmptyCatalog:
return _error(409, "empty_catalog", "There are no products to export.")
return StreamingResponse(
stream,
media_type="text/csv",
headers={"content-disposition": 'attachment; filename="ecomm-products-export.csv"'},
)
_RENDITION_CT = {"thumb": "image/webp", "card": "image/webp",
"detail": "image/webp", "original": "application/octet-stream"}
@app.get("/api/products/images/{image_id}/{rendition}")
def serve_product_image(
image_id: int,
rendition: str = ApiPath(pattern="^(original|thumb|card|detail)$"),
conn: psycopg.Connection = Depends(get_conn),
sess: dict | None = Depends(get_session),
):
"""Serve a hosted image rendition, storefront-authorized + immutable cache (§6.4, INV-16)."""
gate = _merchant_gate(conn, sess)
if isinstance(gate, JSONResponse):
return gate
_account, sf = gate
rec = products.image_for_serving(conn, sf.id, image_id)
if rec is None:
return _error(404, "not_found", "No such image.")
if rec["status"] != "fetched":
return _error(409, "not_fetched", "This image has not been fetched yet.")
key = rec[rendition]
if not key:
return _error(404, "not_found", "No such rendition.")
try:
data = app.state.objectstore.get(key)
except objectstore_mod.ObjectNotFound:
return _error(404, "not_found", "No such image.")
return Response(content=data, media_type=_RENDITION_CT[rendition],
headers={"Cache-Control": "public, max-age=31536000, immutable"})
@app.get("/api/products/sample.csv")
def products_sample_csv():
"""The DOC-3 worked-example CSV. Documentation, so no auth gate (§6.4)."""
return PlainTextResponse(
products.SAMPLE_CSV_PATH.read_text(),
media_type="text/csv",
headers={"content-disposition": 'attachment; filename="ecomm-products-sample.csv"'},
)
@app.get("/api/products/columns.md")
def products_columns_md():
"""The DOC-2 column reference. Documentation, so no auth gate (§6.4)."""
return PlainTextResponse(
products.COLUMNS_MD_PATH.read_text(),
media_type="text/markdown; charset=utf-8",
)
# Deployed topology (launch-app SPEC §2): nginx proxies everything here, so the
# backend serves the built SPA. Mounted LAST so /healthz and /api/* win. In dev the
# dist dir doesn't exist (Vite serves the frontend) and the mount is skipped.
+28
View File
@@ -68,3 +68,31 @@ def smtp_from() -> str:
def smtp_starttls() -> bool:
"""STARTTLS on the relay connection (default on; disable only for odd relays)."""
return os.environ.get("ECOMM_SMTP_STARTTLS", "1").strip().lower() not in {"0", "false", "no", "off"}
def objectstore_kind() -> str:
"""Which objectstore adapter to build: 'local' (dev/tests) or 'gcs' (deployed)."""
return os.environ.get("ECOMM_OBJECTSTORE_KIND") or "local"
def objectstore_bucket() -> str:
"""The GCS bucket name for the media objects (gcs adapter; deployment overlay)."""
return os.environ.get("ECOMM_OBJECTSTORE_BUCKET", "")
def objectstore_local_dir() -> str:
"""Base directory for the local-disk objectstore adapter (dev/tests)."""
return os.environ.get("ECOMM_OBJECTSTORE_DIR") or "/tmp/ecomm-objectstore"
def public_base_url() -> str:
"""Absolute origin for hosted image URLs in export (e.g. APP_URL); '' → relative."""
return (os.environ.get("APP_URL") or "").rstrip("/")
def image_fetch_allow_private() -> bool:
"""Allow image fetch from private/loopback hosts (dev/E2E fixture host only).
Default off — the SSRF guard rejects private ranges in deployed envs."""
return os.environ.get("ECOMM_IMAGE_FETCH_ALLOW_PRIVATE", "").strip().lower() in {
"1", "true", "yes", "on",
}
+77
View File
@@ -0,0 +1,77 @@
"""platform/images — pure image processing (SD-0002 §6.2).
Model-free, deterministic, no I/O of its own: decode bytes, enforce the
resolution bar (INV-18 / Q-3), and emit downscale-only WebP renditions. The
caller (the image-fetch phase) owns fetching, storage, and status.
"""
from __future__ import annotations
import io
from dataclasses import dataclass
from PIL import Image, UnidentifiedImageError
# Q-3 (SD-0002 §13): the minimum shorter-side dimension; below it an image is
# rejected_low_res. Rejects icons/thumbnails, accepts normal product photos.
MIN_IMAGE_SHORT_SIDE = 500
# Rendition longest-side caps (px); downscale-only — a smaller source is kept
# at its own size, never upscaled. Encoded WebP.
RENDITIONS = {"thumb": 160, "card": 480, "detail": 1600}
# Source formats we accept (Pillow format names).
_ACCEPTED = {"JPEG", "PNG", "WEBP"}
_WEBP_QUALITY = 82
@dataclass(frozen=True)
class Processed:
source_format: str
width: int
height: int
renditions: dict[str, bytes] # name -> WebP bytes
@dataclass(frozen=True)
class Rejected:
reason: str # "rejected_low_res" | "rejected_not_image"
def process(data: bytes) -> Processed | Rejected:
"""Decode + bar-check + renditions, or a typed rejection. Never raises on
bad input — undecodable/unsupported bytes are Rejected('rejected_not_image')."""
try:
with Image.open(io.BytesIO(data)) as im:
im.load()
source_format = im.format or ""
if source_format not in _ACCEPTED:
return Rejected(reason="rejected_not_image")
rgb = im.convert("RGB")
width, height = rgb.size
if min(width, height) < MIN_IMAGE_SHORT_SIDE:
return Rejected(reason="rejected_low_res")
renditions = {
name: _rendition(rgb, cap) for name, cap in RENDITIONS.items()
}
return Processed(
source_format=source_format,
width=width,
height=height,
renditions=renditions,
)
except (UnidentifiedImageError, OSError, ValueError, Image.DecompressionBombError):
return Rejected(reason="rejected_not_image")
def _rendition(rgb: Image.Image, cap: int) -> bytes:
longest = max(rgb.size)
if longest > cap:
scale = cap / longest
size = (max(1, round(rgb.width * scale)), max(1, round(rgb.height * scale)))
resized = rgb.resize(size, Image.LANCZOS)
else:
resized = rgb
buf = io.BytesIO()
resized.save(buf, format="WEBP", quality=_WEBP_QUALITY, method=6)
return buf.getvalue()
+103
View File
@@ -0,0 +1,103 @@
"""platform/objectstore — the media-blob port (SD-0002 §6.2, §6.3.1).
A tiny put/get/delete port with two adapters, mirroring the SD-0001 mailer
pattern: local-disk (dev/tests) and GCS (deployed). Owns no semantics — keys
and content types come from the caller (the image-fetch phase). Objects are
written once, never mutated (immutable, image-id-addressed).
"""
from __future__ import annotations
import os
from pathlib import Path
from typing import Protocol
from app.platform import config
class ObjectNotFound(Exception):
"""Raised by get() when the key has no object."""
class ObjectStore(Protocol):
def put(self, key: str, data: bytes, content_type: str) -> None: ...
def get(self, key: str) -> bytes: ...
def delete(self, key: str) -> None: ...
def build_objectstore(kind: str) -> ObjectStore:
"""Select the objectstore adapter by configured kind (config.objectstore_kind, INV-8)."""
if kind == "local":
return LocalObjectStore(config.objectstore_local_dir())
if kind == "gcs":
return GcsObjectStore(config.objectstore_bucket())
raise ValueError(f"unknown objectstore kind: {kind!r}")
def _safe_segments(key: str) -> tuple[str, ...]:
parts = tuple(p for p in key.split("/") if p)
if not parts or any(p in {".", ".."} for p in parts):
raise ValueError(f"unsafe objectstore key: {key!r}")
return parts
class LocalObjectStore:
"""Disk-backed adapter under a base dir; key path-segments become subdirs."""
def __init__(self, base_dir: str) -> None:
self._base = Path(base_dir)
def _path(self, key: str) -> Path:
return self._base.joinpath(*_safe_segments(key))
def put(self, key: str, data: bytes, content_type: str) -> None:
path = self._path(key)
path.parent.mkdir(parents=True, exist_ok=True)
path.write_bytes(data)
def get(self, key: str) -> bytes:
path = self._path(key)
try:
return path.read_bytes()
except FileNotFoundError as exc:
raise ObjectNotFound(key) from exc
def delete(self, key: str) -> None:
try:
os.remove(self._path(key))
except FileNotFoundError:
pass
class GcsObjectStore:
"""GCS adapter; auth via the VM service account ADC (no secret bytes).
google-cloud-storage is imported lazily so dev/test runs (local adapter)
don't require the package at import time.
"""
def __init__(self, bucket_name: str) -> None:
if not bucket_name:
raise ValueError("gcs objectstore requires ECOMM_OBJECTSTORE_BUCKET")
from google.cloud import storage # lazy
self._bucket = storage.Client().bucket(bucket_name)
def put(self, key: str, data: bytes, content_type: str) -> None:
_safe_segments(key)
self._bucket.blob(key).upload_from_string(data, content_type=content_type)
def get(self, key: str) -> bytes:
from google.cloud.exceptions import NotFound # lazy
try:
return self._bucket.blob(key).download_as_bytes()
except NotFound as exc:
raise ObjectNotFound(key) from exc
def delete(self, key: str) -> None:
from google.cloud.exceptions import NotFound # lazy
try:
self._bucket.blob(key).delete()
except NotFound:
pass
+13
View File
@@ -0,0 +1,13 @@
"""Structured log-event telemetry (SD-0002 §9.1). One JSON object per event on the
`ecomm.telemetry` logger — counts and durations only; never file names, URLs,
catalog content, or secret bytes (§6.3-handbook)."""
from __future__ import annotations
import json
import logging
_logger = logging.getLogger("ecomm.telemetry")
def emit(event: str, **fields: object) -> None:
_logger.info(json.dumps({"event": event, **fields}, sort_keys=True, default=str))
+130
View File
@@ -0,0 +1,130 @@
-- 0002_products.sql — SD-0002 §6.3 data model (SLICE-5). Forward-only (INV-7):
-- never edit once merged; add a new numbered migration.
-- product — one catalog product per (storefront, handle) (INV-13/14). Option *names*
-- live here; product_type's kit values are schema-open but a service-layer rule
-- rejects non-'standalone' until #15 (same pattern as INV-4).
CREATE TABLE product (
id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
storefront_id BIGINT NOT NULL REFERENCES storefront (id),
handle TEXT NOT NULL,
title TEXT NOT NULL,
description_html TEXT,
vendor TEXT,
product_type TEXT NOT NULL DEFAULT 'standalone'
CHECK (product_type IN ('standalone', 'kit_virtual', 'kit_assembled')),
google_product_category TEXT,
tags TEXT[] NOT NULL DEFAULT '{}',
status TEXT NOT NULL DEFAULT 'active' CHECK (status IN ('draft', 'active', 'archived')),
published BOOLEAN NOT NULL DEFAULT TRUE,
option1_name TEXT,
option2_name TEXT,
option3_name TEXT,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE UNIQUE INDEX product_handle_key ON product (storefront_id, handle); -- INV-13
-- variant — one purchasable form, identified by its option-value combo (INV-13).
-- SKU is indexed data, never identity. image_id FK is added after product_image.
CREATE TABLE variant (
id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
product_id BIGINT NOT NULL REFERENCES product (id),
position INTEGER NOT NULL,
option1_value TEXT,
option2_value TEXT,
option3_value TEXT,
sku TEXT,
barcode TEXT,
price NUMERIC,
cost NUMERIC,
weight NUMERIC,
weight_unit TEXT,
volume NUMERIC,
volume_unit TEXT,
tax_id_1 TEXT,
tax_id_2 TEXT,
inventory_tracker TEXT,
inventory_qty INTEGER,
image_id BIGINT,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
-- Postgres 16 (compose + Cloud SQL pin): NULLS NOT DISTINCT makes the all-NULL
-- no-option combo unique too (INV-13).
CREATE UNIQUE INDEX variant_option_combo_key
ON variant (product_id, option1_value, option2_value, option3_value)
NULLS NOT DISTINCT;
CREATE INDEX variant_sku_idx ON variant (sku);
-- product_image — identity within a product is source_url (§6.3); bytes live in
-- object storage from SLICE-7 (keys nullable until fetched). status starts 'pending';
-- SLICE-5 stubs the fetch phase so rows simply stay pending.
CREATE TABLE product_image (
id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
product_id BIGINT NOT NULL REFERENCES product (id),
position INTEGER NOT NULL,
source_url TEXT NOT NULL,
alt_text TEXT,
status TEXT NOT NULL DEFAULT 'pending'
CHECK (status IN ('pending', 'fetched', 'rejected_low_res', 'rejected_not_image', 'failed')),
failure_reason TEXT,
key_original TEXT,
key_thumb TEXT,
key_card TEXT,
key_detail TEXT,
import_run_id BIGINT,
fetched_at TIMESTAMPTZ,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE UNIQUE INDEX product_image_src_key ON product_image (product_id, source_url);
ALTER TABLE variant
ADD CONSTRAINT variant_image_fk FOREIGN KEY (image_id) REFERENCES product_image (id);
-- import_draft — the preview's server side (INV-11). file_bytes is the SLICE-5
-- interim home for the upload (objectstore key from SLICE-7). Deleted outright on
-- cancel/expiry — drafts never appear in history (PUC-3a).
CREATE TABLE import_draft (
id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
storefront_id BIGINT NOT NULL REFERENCES storefront (id),
account_id BIGINT NOT NULL REFERENCES account (id),
file_name TEXT NOT NULL,
dialect TEXT NOT NULL,
file_bytes BYTEA NOT NULL,
summary JSONB NOT NULL,
records JSONB NOT NULL,
fingerprint TEXT NOT NULL,
unknown_columns TEXT[] NOT NULL DEFAULT '{}',
expires_at TIMESTAMPTZ NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
-- import_run — the durable record of one confirmed import (PUC-8); created only at
-- confirm (PUC-4).
CREATE TABLE import_run (
id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
storefront_id BIGINT NOT NULL REFERENCES storefront (id),
account_id BIGINT NOT NULL REFERENCES account (id),
file_name TEXT NOT NULL,
dialect TEXT NOT NULL,
products_added INTEGER NOT NULL,
products_updated INTEGER NOT NULL,
rows_errored INTEGER NOT NULL,
status TEXT NOT NULL
CHECK (status IN ('applying', 'fetching_images', 'complete', 'complete_with_problems')),
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
completed_at TIMESTAMPTZ
);
CREATE INDEX import_run_history_idx ON import_run (storefront_id, created_at DESC);
-- import_run_error — one row per rejected CSV row (PUC-5), merchant-language message.
CREATE TABLE import_run_error (
id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
run_id BIGINT NOT NULL REFERENCES import_run (id),
line_number INTEGER NOT NULL,
column_name TEXT,
message TEXT NOT NULL
);
ALTER TABLE product_image
ADD CONSTRAINT product_image_run_fk FOREIGN KEY (import_run_id) REFERENCES import_run (id);
+4
View File
@@ -5,3 +5,7 @@ psycopg[binary]>=3.1
psycopg-pool>=3.2
pytest>=8.0
import-linter>=2.0
nh3>=0.2
python-multipart>=0.0.9
Pillow>=11.0
google-cloud-storage>=2.18
+4
View File
@@ -0,0 +1,4 @@
Handle,Title,Body (HTML),Vendor,Product Category,Type,Tags,Published,Status,Option1 Name,Option1 Value,Variant SKU,Variant Grams,Variant Inventory Tracker,Variant Inventory Qty,Variant Inventory Policy,Variant Fulfillment Service,Variant Price,Variant Compare At Price,Variant Requires Shipping,Variant Taxable,Variant Barcode,Image Src,Image Position,Image Alt Text,Gift Card,SEO Title,SEO Description,Google Shopping / MPN,Variant Image,Variant Weight Unit,Cost per item,Status
star-tee,Star Tee,<p>Soft cotton tee.</p>,Wiggle Goods,Apparel & Accessories > Clothing,Shirts,"apparel, tees",TRUE,active,Size,S,WG-TEE-S,180,shopify,12,deny,manual,24.00,30.00,TRUE,TRUE,0001,https://img.example.com/star-tee.jpg,1,Star Tee,FALSE,Star Tee | Wiggle,Soft tee,MPN-1,https://img.example.com/star-s.jpg,g,11.00,active
star-tee,,,,,,,,,,M,WG-TEE-M,180,shopify,18,deny,manual,24.00,30.00,TRUE,TRUE,0002,,,,FALSE,,,,,g,11.00,
star-tee,,,,,,,,,,,,,,,,,,,,,,https://img.example.com/star-back.jpg,2,Star Tee back,,,,,,,,
1 Handle Title Body (HTML) Vendor Product Category Type Tags Published Status Option1 Name Option1 Value Variant SKU Variant Grams Variant Inventory Tracker Variant Inventory Qty Variant Inventory Policy Variant Fulfillment Service Variant Price Variant Compare At Price Variant Requires Shipping Variant Taxable Variant Barcode Image Src Image Position Image Alt Text Gift Card SEO Title SEO Description Google Shopping / MPN Variant Image Variant Weight Unit Cost per item Status
2 star-tee Star Tee <p>Soft cotton tee.</p> Wiggle Goods Apparel & Accessories > Clothing Shirts apparel, tees TRUE active Size S WG-TEE-S 180 shopify 12 deny manual 24.00 30.00 TRUE TRUE 0001 https://img.example.com/star-tee.jpg 1 Star Tee FALSE Star Tee | Wiggle Soft tee MPN-1 https://img.example.com/star-s.jpg g 11.00 active
3 star-tee M WG-TEE-M 180 shopify 18 deny manual 24.00 30.00 TRUE TRUE 0002 FALSE g 11.00
4 star-tee https://img.example.com/star-back.jpg 2 Star Tee back
+24 -2
View File
@@ -1,4 +1,5 @@
import psycopg
import pytest
from app.platform import db
@@ -12,10 +13,10 @@ def _table_names(conn) -> set[str]:
return {r[0] for r in rows}
def test_migrate_from_empty_applies_0001(fresh_db_url):
def test_migrate_from_empty_applies_all(fresh_db_url):
with psycopg.connect(fresh_db_url) as conn:
applied = db.migrate(conn)
assert applied == ["0001_init.sql"]
assert applied == ["0001_init.sql", "0002_products.sql"]
with psycopg.connect(fresh_db_url) as conn:
assert _TABLES.issubset(_table_names(conn))
@@ -41,3 +42,24 @@ def test_membership_has_no_unique_account_constraint(fresh_db_url):
).fetchall()
defs = " ".join(r[0] for r in rows).lower()
assert "unique" not in defs.replace("primary key", "") or "(account_id)" not in defs
def test_0002_products_tables_exist(fresh_db_url):
with psycopg.connect(fresh_db_url) as conn:
db.migrate(conn)
for table in ("product", "variant", "product_image", "import_draft", "import_run", "import_run_error"):
assert conn.execute("SELECT to_regclass(%s)", (f"public.{table}",)).fetchone()[0] == table
def test_0002_variant_option_combo_unique_treats_nulls_as_equal(fresh_db_url):
# INV-13: the no-option product's single variant has NULL option values; a second
# all-NULL combo must collide (NULLS NOT DISTINCT).
with psycopg.connect(fresh_db_url) as conn:
db.migrate(conn)
sf = conn.execute("INSERT INTO storefront (name) VALUES ('s') RETURNING id").fetchone()[0]
pid = conn.execute(
"INSERT INTO product (storefront_id, handle, title) VALUES (%s,'h','T') RETURNING id", (sf,)
).fetchone()[0]
conn.execute("INSERT INTO variant (product_id, position) VALUES (%s, 1)", (pid,))
with pytest.raises(psycopg.errors.UniqueViolation):
conn.execute("INSERT INTO variant (product_id, position) VALUES (%s, 2)", (pid,))
+65
View File
@@ -0,0 +1,65 @@
"""platform/images — pure decode + resolution bar + WebP renditions (SD-0002 §6.2)."""
import io
from PIL import Image
from app.platform import images
def _png(width: int, height: int, color=(120, 80, 200)) -> bytes:
buf = io.BytesIO()
Image.new("RGB", (width, height), color).save(buf, format="PNG")
return buf.getvalue()
def _jpeg(width: int, height: int) -> bytes:
buf = io.BytesIO()
Image.new("RGB", (width, height), (10, 200, 90)).save(buf, format="JPEG")
return buf.getvalue()
def test_good_image_produces_three_webp_renditions():
result = images.process(_png(1200, 900))
assert isinstance(result, images.Processed)
assert result.source_format == "PNG"
assert set(result.renditions) == {"thumb", "card", "detail"}
for name, blob in result.renditions.items():
with Image.open(io.BytesIO(blob)) as im:
assert im.format == "WEBP"
with Image.open(io.BytesIO(result.renditions["thumb"])) as im:
assert max(im.size) <= images.RENDITIONS["thumb"]
with Image.open(io.BytesIO(result.renditions["detail"])) as im:
assert max(im.size) <= images.RENDITIONS["detail"]
def test_downscale_only_never_upscales_small_source():
result = images.process(_png(520, 520))
with Image.open(io.BytesIO(result.renditions["detail"])) as im:
assert im.size == (520, 520)
def test_below_resolution_bar_rejected_low_res():
result = images.process(_png(300, 1200)) # shorter side 300 < 500
assert isinstance(result, images.Rejected)
assert result.reason == "rejected_low_res"
def test_not_an_image_rejected_not_image():
result = images.process(b"this is not an image")
assert isinstance(result, images.Rejected)
assert result.reason == "rejected_not_image"
def test_jpeg_source_format_preserved_in_result():
result = images.process(_jpeg(800, 800))
assert result.source_format == "JPEG"
def test_decompression_bomb_rejected_not_image(monkeypatch):
# A small file whose pixel count exceeds Pillow's bomb threshold must be a
# typed rejection, never an escaping exception.
from PIL import Image as PILImage
monkeypatch.setattr(PILImage, "MAX_IMAGE_PIXELS", 100) # 10x10 exceeds it
result = images.process(_png(800, 800))
assert isinstance(result, images.Rejected)
assert result.reason == "rejected_not_image"
@@ -0,0 +1,44 @@
"""platform/objectstore — local-disk adapter round-trip (SD-0002 §6.2/§6.3.1)."""
import pytest
from app.platform import objectstore
def test_local_put_get_round_trip(tmp_path):
store = objectstore.LocalObjectStore(str(tmp_path))
key = "storefronts/1/product-images/9/original"
store.put(key, b"\x89PNG-bytes", "image/png")
assert store.get(key) == b"\x89PNG-bytes"
def test_local_get_missing_raises_not_found(tmp_path):
store = objectstore.LocalObjectStore(str(tmp_path))
with pytest.raises(objectstore.ObjectNotFound):
store.get("nope/missing")
def test_local_delete_is_idempotent(tmp_path):
store = objectstore.LocalObjectStore(str(tmp_path))
store.put("a/b", b"x", "text/plain")
store.delete("a/b")
store.delete("a/b") # no error second time
with pytest.raises(objectstore.ObjectNotFound):
store.get("a/b")
def test_local_keys_cannot_escape_base_dir(tmp_path):
store = objectstore.LocalObjectStore(str(tmp_path))
with pytest.raises(ValueError):
store.put("../escape", b"x", "text/plain")
def test_build_objectstore_local(tmp_path, monkeypatch):
monkeypatch.setenv("ECOMM_OBJECTSTORE_KIND", "local")
monkeypatch.setenv("ECOMM_OBJECTSTORE_DIR", str(tmp_path))
store = objectstore.build_objectstore("local")
assert isinstance(store, objectstore.LocalObjectStore)
def test_build_objectstore_unknown_kind_raises():
with pytest.raises(ValueError):
objectstore.build_objectstore("s3")
+95
View File
@@ -0,0 +1,95 @@
"""§6.5.1 file-level codec — parse, caps, required columns (PUC-5a fixtures)."""
import pytest
from app.domains.products import FileRejected
from app.domains.products.codec import parse_csv
def _csv(*lines: str) -> bytes:
return ("\n".join(lines) + "\n").encode()
GOOD = _csv(
"Handle,Title,Variant Price,Bogus Column",
"moon-mug,Moon Mug,18.00,x",
"star-tee,Star Tee,24.00,y",
)
def test_parses_header_rows_and_unknown_columns():
parsed = parse_csv(GOOD)
assert parsed.dialect == "canonical"
assert parsed.unknown_columns == ["Bogus Column"]
assert [r.line_number for r in parsed.rows] == [2, 3]
assert parsed.rows[0].cells["Handle"] == "moon-mug"
assert parsed.rows[0].cells["Variant Price"] == "18.00"
assert "Bogus Column" not in parsed.rows[0].cells
def test_bom_tolerated():
parsed = parse_csv(b"\xef\xbb\xbf" + GOOD)
assert parsed.rows[0].cells["Handle"] == "moon-mug"
def test_quoted_cells_rfc4180():
parsed = parse_csv(_csv("Handle,Title,Tags", 'mug,"The ""Best"" Mug","a, b"'))
assert parsed.rows[0].cells["Title"] == 'The "Best" Mug'
assert parsed.rows[0].cells["Tags"] == "a, b"
def test_empty_rows_skipped_short_rows_padded():
parsed = parse_csv(_csv("Handle,Title,Vendor", "mug,Mug", "", ",,", "tee,Tee,Acme"))
assert [r.cells["Handle"] for r in parsed.rows] == ["mug", "tee"]
assert parsed.rows[0].cells["Vendor"] == ""
@pytest.mark.parametrize(
"data,code",
[
(b"\xff\xfe\x00garbage\x00", "not_csv"),
(b"", "not_csv"),
(_csv("Title,Vendor", "Mug,Acme"), "missing_required_column"),
(_csv("Handle,Vendor", "mug,Acme"), "missing_required_column"),
(
_csv("Handle,Title", *(f"h{i},T{i}" for i in range(5001))),
"too_many_rows",
),
(b"Handle,Title\n" + b"x" * (10 * 1024 * 1024), "file_too_large"),
],
)
def test_file_level_rejections(data, code):
with pytest.raises(FileRejected) as exc:
parse_csv(data)
assert exc.value.code == code
def test_missing_column_message_names_the_column():
with pytest.raises(FileRejected) as exc:
parse_csv(_csv("Handle,Vendor", "mug,Acme"))
assert "'Title'" in exc.value.message
def test_parse_detects_and_maps_shopify():
data = (
"Handle,Title,Body (HTML),Cost per item,Variant Price,Variant Grams,Type,Gift Card\n"
"mug,Moon Mug,<p>Grey</p>,9.50,18.00,300,Drinkware,false\n"
).encode("utf-8")
parsed = parse_csv(data)
assert parsed.dialect == "shopify"
cells = parsed.rows[0].cells
assert cells["Description"] == "<p>Grey</p>"
assert cells["Variant Cost"] == "9.50"
assert cells["Variant Weight"] == "300"
assert cells["Variant Weight Unit"] == "g" # synthesized
# Type (free-text) and Gift Card warned, never mapped:
assert "Type" in parsed.unknown_columns
assert "Gift Card" in parsed.unknown_columns
assert "product_type" not in str(cells) # Type never reached canonical
def test_parse_canonical_unchanged():
data = b"Handle,Title,Description,Variant Price\nmug,Moon Mug,Grey,18.00\n"
parsed = parse_csv(data)
assert parsed.dialect == "canonical"
assert parsed.rows[0].cells["Description"] == "Grey"
assert parsed.unknown_columns == []
@@ -0,0 +1,121 @@
"""Shopify dialect adapter — detection + the §6.5.1 mapping contract (SLICE-8)."""
from pathlib import Path
from app.domains.products.codec import parse_csv
from app.domains.products.dialect_shopify import is_shopify_header, map_shopify_header
_FIXTURE = Path(__file__).parent / "fixtures" / "shopify-export.csv"
def test_detects_shopify_by_signature_column():
assert is_shopify_header(["Handle", "Title", "Body (HTML)", "Variant Price"]) is True
assert is_shopify_header(["Handle", "Title", "Google Shopping / MPN"]) is True
def test_canonical_header_is_not_shopify():
assert is_shopify_header(["Handle", "Title", "Description", "Variant Cost"]) is False
def test_ambiguous_shared_only_header_defaults_canonical():
# Only columns common to both dialects -> not Shopify (safe default, never misparse).
assert (
is_shopify_header(["Handle", "Title", "Option1 Name", "Variant Price", "Image Src"])
is False
)
def test_map_renames_and_passes_through():
mapped, not_imported = map_shopify_header(
["Handle", "Body (HTML)", "Cost per item", "Variant Price"]
)
assert mapped == ["Handle", "Description", "Variant Cost", "Variant Price"]
assert not_imported == []
def test_map_drops_type_and_weight_unit_overrides():
mapped, not_imported = map_shopify_header(
["Handle", "Type", "Variant Grams", "Variant Weight Unit"]
)
assert mapped == ["Handle", None, "Variant Weight", None]
assert not_imported == ["Type", "Variant Weight Unit"]
def test_map_drops_shopify_only_and_market_columns():
mapped, not_imported = map_shopify_header(
["Handle", "Variant Compare At Price", "Gift Card", "Price / International"]
)
assert mapped == ["Handle", None, None, None]
assert not_imported == ["Variant Compare At Price", "Gift Card", "Price / International"]
def test_shopify_fixture_maps_exhaustively():
parsed = parse_csv(_FIXTURE.read_bytes())
assert parsed.dialect == "shopify"
first = parsed.rows[0].cells
# renamed
assert first["Description"] == "<p>Soft cotton tee.</p>"
assert first["Google Product Category"] == "Apparel & Accessories > Clothing"
assert first["Variant Cost"] == "11.00"
assert first["Variant Weight"] == "180"
assert first["Variant Weight Unit"] == "g"
# direct
assert first["Variant SKU"] == "WG-TEE-S"
assert first["Image Src"] == "https://img.example.com/star-tee.jpg"
# not imported — warned, none leaked into canonical cells
for col in (
"Type",
"Variant Compare At Price",
"Gift Card",
"SEO Title",
"Google Shopping / MPN",
"Variant Weight Unit",
"Variant Inventory Policy",
"Variant Fulfillment Service",
"Variant Requires Shipping",
"Variant Taxable",
):
assert col in parsed.unknown_columns, col
assert "Variant Compare At Price" not in first
# --- Detection robustness (review findings #1, #2): bias conservative ----------------
def test_canonical_distinctive_column_vetoes_shopify_detection():
# A canonical file carrying a stray Shopify-signature name (`SEO Title`) stays
# canonical because it also has canonical-distinctive columns — no misparse,
# no dropped Type, no corrupted weight unit (review finding #2).
header = ["Handle", "Title", "Type", "Variant Weight", "Variant Weight Unit", "SEO Title"]
assert is_shopify_header(header) is False
def test_dual_named_file_stays_canonical_and_warns_shopify_name():
# A malformed file with BOTH `Body (HTML)` and canonical `Description`: the
# canonical column vetoes detection, so `Body (HTML)` is honestly warned as an
# unknown column rather than silently shadowing `Description` (review finding #1).
data = (
b"Handle,Title,Body (HTML),Description,Variant Price\n"
b"mug,Moon Mug,FROM_BODY,FROM_DESC,18.00\n"
)
parsed = parse_csv(data)
assert parsed.dialect == "canonical"
assert parsed.rows[0].cells["Description"] == "FROM_DESC"
assert "Body (HTML)" in parsed.unknown_columns
def test_real_shopify_export_still_detected():
# The conservative veto must not break a genuine Shopify export (no canonical-
# distinctive columns present).
parsed = parse_csv(_FIXTURE.read_bytes())
assert parsed.dialect == "shopify"
def test_shopify_grams_clear_clears_weight_unit_together():
# An empty Variant Grams (a clear) clears the synthesized unit too — weight and
# unit move together, never a stale unit (review finding #3).
data = b"Handle,Title,Body (HTML),Variant Grams\nmug,Moon Mug,<p>x</p>,\n"
parsed = parse_csv(data)
assert parsed.dialect == "shopify"
cells = parsed.rows[0].cells
assert cells["Variant Weight"] == ""
assert cells["Variant Weight Unit"] == ""
+196
View File
@@ -0,0 +1,196 @@
"""Diff engine — classification, blank-vs-absent, option matching, fingerprint (§6.8)."""
from decimal import Decimal
from app.domains.products.codec import parse_csv
from app.domains.products.diff import (
CatalogImage, CatalogProduct, CatalogVariant, compute_diff,
)
from app.domains.products.validate import build_products
def _canon(*lines: str):
return build_products(parse_csv(("\n".join(lines) + "\n").encode()))
def _catalog_mug(**overrides):
fields = {
"title": "Moon Mug", "description_html": None, "vendor": "Acme",
"product_type": "standalone", "google_product_category": None,
"tags": ["kitchen"], "status": "active", "published": True,
} | overrides
return {
"moon-mug": CatalogProduct(
id=1, handle="moon-mug", title=fields["title"],
option_names=(None, None, None), fields=fields,
variants=[CatalogVariant(id=10, options=(None, None, None), position=1,
fields={"sku": "SKU-1", "barcode": None, "price": Decimal("18.00"),
"cost": None, "weight": None, "weight_unit": None,
"volume": None, "volume_unit": None, "tax_id_1": None,
"tax_id_2": None, "inventory_tracker": None,
"inventory_qty": 40, "variant_image": None})],
images=[CatalogImage(id=100, source_url="https://x/a.jpg", position=1, alt_text=None)],
)
}
HEADER = "Handle,Title,Vendor,Tags,Status,Variant SKU,Variant Price,Variant Inventory Qty,Image Src"
MUG_ROW = 'moon-mug,Moon Mug,Acme,kitchen,active,SKU-1,18.00,40,https://x/a.jpg'
def test_new_handle_classifies_add():
diff = compute_diff({}, _canon(HEADER, MUG_ROW))
[rec] = diff.records
assert rec["kind"] == "add" and rec["handle"] == "moon-mug" and rec["variant_count"] == 1
assert diff.summary == {"adds": 1, "updates": 0, "unchanged": 0, "errors": 0}
assert rec["detail"]["set"]["status"] == "active"
def test_identical_file_classifies_unchanged():
diff = compute_diff(_catalog_mug(), _canon(HEADER, MUG_ROW))
assert diff.records[0]["kind"] == "unchanged"
assert diff.summary["unchanged"] == 1
def test_changed_price_classifies_update_with_before_after():
row = MUG_ROW.replace("18.00", "21.00")
diff = compute_diff(_catalog_mug(), _canon(HEADER, row))
[rec] = diff.records
assert rec["kind"] == "update"
[vchange] = rec["detail"]["variants"]
assert {"field": "price", "before": "18.00", "after": "21.00"} in vchange["changes"]
def test_absent_column_untouched_empty_cell_clears():
# Vendor column absent: vendor stays Acme. Status present-but-empty: clears to default 'active' (already active -> no change).
diff = compute_diff(
_catalog_mug(),
_canon("Handle,Title,Status,Variant SKU,Variant Price,Variant Inventory Qty,Image Src",
"moon-mug,Moon Mug,,SKU-1,18.00,40,https://x/a.jpg"),
)
assert diff.records[0]["kind"] == "unchanged"
def test_empty_cell_clear_shows_in_diff():
# Vendor present-but-empty clears Acme -> None: an explicit, previewable change (§6.5.1).
diff = compute_diff(
_catalog_mug(),
_canon("Handle,Title,Vendor,Variant SKU,Variant Price,Variant Inventory Qty,Image Src",
"moon-mug,Moon Mug,,SKU-1,18.00,40,https://x/a.jpg"),
)
[rec] = diff.records
assert rec["kind"] == "update"
assert {"field": "vendor", "before": "Acme", "after": None} in rec["detail"]["changes"]
def test_new_option_combo_is_variant_add_existing_untouched():
catalog = _catalog_mug()
diff = compute_diff(
catalog,
_canon("Handle,Title,Option1 Name,Option1 Value,Variant Price",
"moon-mug,Moon Mug,Size,Large,25.00"),
)
[rec] = diff.records
assert rec["kind"] == "update"
kinds = [v.get("kind") for v in rec["detail"]["variants"]]
assert "add" in kinds
POSITION_HEADER = HEADER + ",Variant Position"
def test_matching_variant_position_column_classifies_unchanged():
# position is an attribute on CatalogVariant (not in fields{}); the compare
# must read it from there, not invent a before:None.
diff = compute_diff(_catalog_mug(), _canon(POSITION_HEADER, MUG_ROW + ",1"))
assert diff.records[0]["kind"] == "unchanged"
def test_changed_variant_position_reports_honest_before():
diff = compute_diff(_catalog_mug(), _canon(POSITION_HEADER, MUG_ROW + ",2"))
[rec] = diff.records
assert rec["kind"] == "update"
[ventry] = rec["detail"]["variants"]
assert {"field": "position", "before": 1, "after": 2} in ventry["changes"]
def test_blank_position_cell_resolves_to_file_order_unchanged():
# A present-but-empty Variant Position cell resets to file order (the spec's
# "defaults to file order"), never to NULL — here file order matches the
# catalog position, so nothing changes.
diff = compute_diff(_catalog_mug(), _canon(POSITION_HEADER, MUG_ROW + ","))
assert diff.records[0]["kind"] == "unchanged"
def test_blank_position_cell_updates_to_file_order():
catalog = _catalog_mug()
catalog["moon-mug"].variants[0].position = 2
diff = compute_diff(catalog, _canon(POSITION_HEADER, MUG_ROW + ","))
[rec] = diff.records
assert rec["kind"] == "update"
[ventry] = rec["detail"]["variants"]
assert {"field": "position", "before": 2, "after": 1} in ventry["changes"]
def _catalog_lamp_with_image():
# A well-formed single-variant product carrying one fetched image id=55.
fields = {
"title": "Lamp", "description_html": None, "vendor": "Acme",
"product_type": "standalone", "google_product_category": None,
"tags": ["home"], "status": "active", "published": True,
}
return {
"lamp": CatalogProduct(
id=1, handle="lamp", title="Lamp",
option_names=(None, None, None), fields=fields,
variants=[CatalogVariant(id=10, options=(None, None, None), position=1,
fields={"sku": "SKU-L", "barcode": None, "price": Decimal("30.00"),
"cost": None, "weight": None, "weight_unit": None,
"volume": None, "volume_unit": None, "tax_id_1": None,
"tax_id_2": None, "inventory_tracker": None,
"inventory_qty": 5, "variant_image": None})],
images=[CatalogImage(id=55, source_url="https://m.example/a.png", position=1,
alt_text=None, status="fetched")],
)
}
LAMP_HEADER = "Handle,Title,Vendor,Tags,Status,Variant SKU,Variant Price,Variant Inventory Qty,Image Src,Image Position"
LAMP_ROW_PREFIX = "lamp,Lamp,Acme,home,active,SKU-L,30.00,5,"
def test_hosted_url_for_existing_image_is_unchanged_not_add():
# Catalog product "lamp" has a fetched image id=55; canonical re-imports it
# as the hosted URL -> must be 'unchanged', never 'add'/'update'.
diff = compute_diff(
_catalog_lamp_with_image(),
_canon(LAMP_HEADER, LAMP_ROW_PREFIX + "/api/products/images/55/detail,1"),
)
kinds = {r["handle"]: r["kind"] for r in diff.records}
assert kinds["lamp"] == "unchanged"
def test_hosted_url_for_unknown_id_is_row_error():
# Catalog "lamp" has NO images; canonical references /images/999/detail -> error.
catalog = _catalog_lamp_with_image()
catalog["lamp"].images = []
diff = compute_diff(
catalog,
_canon(LAMP_HEADER, LAMP_ROW_PREFIX + "/api/products/images/999/detail,1"),
)
kinds = {r["handle"]: r["kind"] for r in diff.records}
assert kinds["lamp"] == "error"
def test_error_product_classifies_error():
diff = compute_diff({}, _canon("Handle,Title,Variant Price", "mug,Mug,nope"))
[rec] = diff.records
assert rec["kind"] == "error"
assert rec["detail"]["errors"][0]["column"] == "Variant Price"
def test_fingerprint_stable_and_drift_sensitive():
d1 = compute_diff(_catalog_mug(), _canon(HEADER, MUG_ROW))
d2 = compute_diff(_catalog_mug(), _canon(HEADER, MUG_ROW))
d3 = compute_diff(_catalog_mug(title="Renamed"), _canon(HEADER, MUG_ROW))
assert d1.fingerprint == d2.fingerprint
assert d1.fingerprint != d3.fingerprint
+146
View File
@@ -0,0 +1,146 @@
"""§6.4 /api/products/* endpoint scenarios (PUC-2/3/3a/4/5/5a/8 + gates)."""
import io
import re
from contextlib import contextmanager
from fastapi.testclient import TestClient
from app.main import create_app
GOOD_CSV = b"Handle,Title,Vendor,Variant Price\nmoon-mug,Moon Mug,Acme,18.00\n"
@contextmanager
def _merchant_client(fresh_db_url, email="m@example.com"):
with TestClient(create_app(database_url=fresh_db_url)) as client:
client.post("/api/auth/request-code", json={"email": email})
code = re.search(r"\b(\d{6})\b", client.app.state.mailer.outbox[-1].body).group(1)
client.post("/api/auth/verify", json={"email": email, "code": code})
client.post("/api/storefronts", json={})
yield client
def _upload(client, data=GOOD_CSV, name="cat.csv"):
return client.post("/api/products/imports", files={"file": (name, io.BytesIO(data), "text/csv")})
def test_upload_returns_201_draft(fresh_db_url):
with _merchant_client(fresh_db_url) as client:
resp = _upload(client)
assert resp.status_code == 201
body = resp.json()
assert body["summary"]["adds"] == 1 and body["dialect"] == "canonical"
def test_upload_rejections_carry_codes(fresh_db_url):
with _merchant_client(fresh_db_url) as client:
resp = _upload(client, b"Vendor\nAcme\n")
assert resp.status_code == 400
assert resp.json()["error"]["code"] == "missing_required_column"
resp = _upload(client, b"Handle,Title\n" + b"x" * (10 * 1024 * 1024 + 1))
assert resp.status_code == 413
def test_unauthenticated_401_and_no_storefront_404(fresh_db_url):
with TestClient(create_app(database_url=fresh_db_url)) as client:
assert _upload(client).status_code == 401
client.post("/api/auth/request-code", json={"email": "x@example.com"})
code = re.search(r"\b(\d{6})\b", client.app.state.mailer.outbox[-1].body).group(1)
client.post("/api/auth/verify", json={"email": "x@example.com", "code": code})
assert _upload(client).status_code == 404
def test_preview_confirm_run_flow(fresh_db_url):
with _merchant_client(fresh_db_url) as client:
draft = _upload(client).json()
recs = client.get(f"/api/products/imports/drafts/{draft['id']}/records").json()["records"]
assert recs[0]["kind"] == "add"
run_id = client.post(f"/api/products/imports/drafts/{draft['id']}/confirm").json()["run_id"]
run = client.get(f"/api/products/imports/runs/{run_id}").json()
assert run["products_added"] == 1 and run["by"] == "m@example.com"
assert client.get("/api/products/summary").json()["product_count"] == 1
assert client.get("/api/products/imports/runs").json()["runs"][0]["id"] == run_id
def test_cancel_no_trace_puc3a(fresh_db_url):
with _merchant_client(fresh_db_url) as client:
draft = _upload(client).json()
assert client.delete(f"/api/products/imports/drafts/{draft['id']}").status_code == 204
assert client.get(f"/api/products/imports/drafts/{draft['id']}").status_code == 404
assert client.get("/api/products/imports/runs").json()["runs"] == []
def test_confirm_conflicts(fresh_db_url):
with _merchant_client(fresh_db_url) as client:
d1 = _upload(client).json()
client.post(f"/api/products/imports/drafts/{d1['id']}/confirm")
d2 = _upload(client).json()
resp = client.post(f"/api/products/imports/drafts/{d2['id']}/confirm")
assert resp.status_code == 409 and resp.json()["error"]["code"] == "nothing_to_apply"
def test_sample_csv_served(fresh_db_url):
with TestClient(create_app(database_url=fresh_db_url)) as client:
resp = client.get("/api/products/sample.csv")
assert resp.status_code == 200
assert resp.headers["content-type"].startswith("text/csv")
assert resp.text.startswith("Handle,Title,")
def test_sample_csv_imports_clean(fresh_db_url):
"""DOC-3 honesty: our own sample must validate with zero errors."""
with _merchant_client(fresh_db_url) as client:
sample = client.get("/api/products/sample.csv").content
body = _upload(client, sample, "sample.csv").json()
assert body["summary"]["errors"] == 0 and body["summary"]["adds"] == 2
def test_columns_md_served(fresh_db_url):
"""DOC-2 column reference is app-served, unauthenticated documentation (§6.4)."""
with TestClient(create_app(database_url=fresh_db_url)) as client:
resp = client.get("/api/products/columns.md")
assert resp.status_code == 200
assert "text/markdown" in resp.headers["content-type"]
body = resp.text
assert "Body (HTML)" in body # Shopify dialect notes present
assert "`Handle`" in body # canonical columns present
def test_export_returns_canonical_csv(fresh_db_url):
with _merchant_client(fresh_db_url) as client:
draft = _upload(client).json()
client.post(f"/api/products/imports/drafts/{draft['id']}/confirm")
resp = client.get("/api/products/export")
assert resp.status_code == 200
assert resp.headers["content-type"].startswith("text/csv")
assert "attachment" in resp.headers["content-disposition"]
body = resp.text
assert body.splitlines()[0].startswith("Handle,")
assert "moon-mug" in body
def test_export_status_filter_respected(fresh_db_url):
with _merchant_client(fresh_db_url) as client:
# GOOD_CSV's moon-mug has no Status column → defaults to active.
draft = _upload(client).json()
client.post(f"/api/products/imports/drafts/{draft['id']}/confirm")
assert "moon-mug" in client.get("/api/products/export?status=active").text
# No archived products → 409.
assert client.get("/api/products/export?status=archived").status_code == 409
def test_export_empty_catalog_409(fresh_db_url):
with _merchant_client(fresh_db_url) as client:
resp = client.get("/api/products/export")
assert resp.status_code == 409
assert resp.json()["error"]["code"] == "empty_catalog"
def test_export_requires_merchant(fresh_db_url):
with TestClient(create_app(database_url=fresh_db_url)) as client:
assert client.get("/api/products/export").status_code == 401
def test_export_bad_status_422(fresh_db_url):
with _merchant_client(fresh_db_url) as client:
assert client.get("/api/products/export?status=bogus").status_code == 422
+100
View File
@@ -0,0 +1,100 @@
"""Export: status-filtered catalog snapshot + the streamed service (PUC-9, TEL-3)."""
import csv
import io
import json
import logging
import psycopg
import pytest
from app.domains import products
from app.domains.products import repo
from app.platform import db
CSV = (
b"Handle,Title,Vendor,Status,Variant Price\n"
b"active-mug,Active Mug,Acme,active,18.00\n"
b"draft-tee,Draft Tee,Acme,draft,24.00\n"
)
@pytest.fixture()
def migrated_conn(fresh_db_url):
with psycopg.connect(fresh_db_url) as conn:
db.migrate(conn)
yield conn
@pytest.fixture()
def merchant(migrated_conn):
acct = migrated_conn.execute(
"INSERT INTO account (email) VALUES ('m@example.com') RETURNING id").fetchone()[0]
sf = migrated_conn.execute(
"INSERT INTO storefront (name) VALUES ('Shop') RETURNING id").fetchone()[0]
migrated_conn.execute(
"INSERT INTO storefront_membership (account_id, storefront_id) VALUES (%s,%s)", (acct, sf))
migrated_conn.commit()
return {"account_id": acct, "storefront_id": sf}
def _seed(conn, merchant):
draft = products.import_validate(conn, merchant["storefront_id"], merchant["account_id"], "c.csv", CSV)
products.confirm_draft(conn, merchant["storefront_id"], merchant["account_id"], draft["id"])
def test_export_all_returns_both(migrated_conn, merchant):
_seed(migrated_conn, merchant)
snap = repo.export_catalog(migrated_conn, merchant["storefront_id"], "all")
assert {p.handle for p in snap} == {"active-mug", "draft-tee"}
def test_export_status_filter(migrated_conn, merchant):
_seed(migrated_conn, merchant)
active = repo.export_catalog(migrated_conn, merchant["storefront_id"], "active")
assert [p.handle for p in active] == ["active-mug"]
draft = repo.export_catalog(migrated_conn, merchant["storefront_id"], "draft")
assert [p.handle for p in draft] == ["draft-tee"]
@pytest.fixture()
def telemetry_propagation():
"""create_app() sets propagate=False on the parent "ecomm" logger
(main._ensure_app_logging), which hides ecomm.telemetry records from caplog's
root-logger handler whenever an API test ran first. Restore propagation here."""
lg = logging.getLogger("ecomm")
prior = lg.propagate
lg.propagate = True
yield
lg.propagate = prior
def test_export_streams_canonical_csv(migrated_conn, merchant):
_seed(migrated_conn, merchant)
text = "".join(products.export_catalog(migrated_conn, merchant["storefront_id"], "all"))
rows = list(csv.DictReader(io.StringIO(text)))
assert {r["Handle"] for r in rows} == {"active-mug", "draft-tee"}
assert rows[0]["Handle"] == "active-mug" # sorted by handle
def test_export_empty_raises(migrated_conn, merchant):
# No catalog at all → empty.
with pytest.raises(products.EmptyCatalog):
list(products.export_catalog(migrated_conn, merchant["storefront_id"], "all"))
def test_export_empty_after_filter_raises(migrated_conn, merchant):
_seed(migrated_conn, merchant) # only active + draft exist
with pytest.raises(products.EmptyCatalog):
list(products.export_catalog(migrated_conn, merchant["storefront_id"], "archived"))
def test_tel3_emitted(migrated_conn, merchant, caplog, telemetry_propagation):
_seed(migrated_conn, merchant)
with caplog.at_level(logging.INFO, logger="ecomm.telemetry"):
list(products.export_catalog(migrated_conn, merchant["storefront_id"], "all"))
events = [json.loads(r.message) for r in caplog.records if r.name == "ecomm.telemetry"]
exported = [e for e in events if e["event"] == "catalog_exported"]
assert len(exported) == 1
assert exported[0]["product_count"] == 2
assert exported[0]["status_filter"] == "all"
assert "duration_ms" in exported[0]
+30
View File
@@ -0,0 +1,30 @@
"""hosted-image URL helpers — round-trip recognition (SD-0002 §6.3.1, INV-12)."""
from app.domains.products import hosted
def test_build_relative_when_no_base():
assert hosted.image_url("", 42, "detail") == "/api/products/images/42/detail"
def test_build_absolute_with_base():
url = hosted.image_url("https://ecomm-ppe.wiggleverse.org", 42, "detail")
assert url == "https://ecomm-ppe.wiggleverse.org/api/products/images/42/detail"
def test_parse_absolute_hosted_url():
url = "https://market.wiggleverse.org/api/products/images/7/detail"
assert hosted.parse_image_id(url) == 7
def test_parse_relative_hosted_url():
assert hosted.parse_image_id("/api/products/images/7/card") == 7
def test_parse_ignores_query_and_trailing():
assert hosted.parse_image_id("/api/products/images/7/detail?v=1") == 7
def test_parse_non_hosted_returns_none():
assert hosted.parse_image_id("https://cdn.example.com/a/b.png") is None
assert hosted.parse_image_id("/api/products/images/abc/detail") is None
assert hosted.parse_image_id("/api/products/images/7") is None
@@ -0,0 +1,78 @@
"""GET /api/products/images/{id}/{rendition} — authorize, stream, immutable cache (§6.4, INV-14/16)."""
import psycopg
from contextlib import contextmanager
from fastapi.testclient import TestClient
from app.main import create_app
from app.domains import products # noqa: F401 (ensures package import path)
from test_products_endpoints import _merchant_client # reuse the signed-in client
def _seed_image(fresh_db_url, status="fetched", with_detail_bytes=None, store=None,
email_storefront_only=True):
"""Insert a product + image for the (only) storefront; set keys; return (storefront_id, image_id, keys)."""
with psycopg.connect(fresh_db_url) as conn:
sf = conn.execute("SELECT id FROM storefront ORDER BY id LIMIT 1").fetchone()[0]
pid = conn.execute(
"INSERT INTO product (storefront_id, handle, title) VALUES (%s,'lamp','Lamp') RETURNING id",
(sf,)).fetchone()[0]
iid = conn.execute(
"INSERT INTO product_image (product_id, source_url, position, status)"
" VALUES (%s,'https://m/a.png',1,%s) RETURNING id", (pid, status)).fetchone()[0]
keys = {r: f"storefronts/{sf}/product-images/{iid}/{r}" for r in ("original", "thumb", "card", "detail")}
if status == "fetched":
conn.execute(
"UPDATE product_image SET key_original=%s, key_thumb=%s, key_card=%s, key_detail=%s WHERE id=%s",
(keys["original"], keys["thumb"], keys["card"], keys["detail"], iid))
conn.commit()
if with_detail_bytes is not None and store is not None:
store.put(keys["detail"], with_detail_bytes, "image/webp")
return sf, iid, keys
def test_serves_fetched_rendition_with_immutable_cache(fresh_db_url):
with _merchant_client(fresh_db_url) as client:
_sf, iid, _keys = _seed_image(fresh_db_url, status="fetched",
with_detail_bytes=b"WEBPDATA", store=client.app.state.objectstore)
resp = client.get(f"/api/products/images/{iid}/detail")
assert resp.status_code == 200
assert resp.headers["content-type"] == "image/webp"
assert "immutable" in resp.headers.get("cache-control", "")
assert resp.content == b"WEBPDATA"
def test_not_fetched_returns_409(fresh_db_url):
with _merchant_client(fresh_db_url) as client:
_sf, iid, _keys = _seed_image(fresh_db_url, status="pending")
resp = client.get(f"/api/products/images/{iid}/detail")
assert resp.status_code == 409
assert resp.json()["error"]["code"] == "not_fetched"
def test_other_storefronts_image_is_404(fresh_db_url):
# Sign in as merchant B first (this migrates the DB), then seed an image
# belonging to a *different* storefront A — it must be invisible to B.
with _merchant_client(fresh_db_url, email="b@example.com") as client:
with psycopg.connect(fresh_db_url) as conn:
a = conn.execute("INSERT INTO account (email) VALUES ('a@example.com') RETURNING id").fetchone()[0]
sfa = conn.execute("INSERT INTO storefront (name) VALUES ('A') RETURNING id").fetchone()[0]
conn.execute("INSERT INTO storefront_membership (account_id, storefront_id) VALUES (%s,%s)", (a, sfa))
pid = conn.execute("INSERT INTO product (storefront_id, handle, title) VALUES (%s,'lamp','Lamp') RETURNING id", (sfa,)).fetchone()[0]
iid = conn.execute("INSERT INTO product_image (product_id, source_url, position, status) VALUES (%s,'u',1,'fetched') RETURNING id", (pid,)).fetchone()[0]
conn.commit()
resp = client.get(f"/api/products/images/{iid}/detail")
assert resp.status_code == 404
def test_unauthenticated_is_401(fresh_db_url):
with TestClient(create_app(database_url=fresh_db_url)) as client:
assert client.get("/api/products/images/1/detail").status_code == 401
def test_bad_rendition_is_422(fresh_db_url):
with _merchant_client(fresh_db_url) as client:
_sf, iid, _keys = _seed_image(fresh_db_url, status="fetched",
with_detail_bytes=b"x", store=client.app.state.objectstore)
assert client.get(f"/api/products/images/{iid}/huge").status_code == 422
+159
View File
@@ -0,0 +1,159 @@
"""image-fetch phase: SSRF guard, fetch+process+store, run completion, resume (§6.5.4/§6.9)."""
import io
import threading
from http.server import BaseHTTPRequestHandler, HTTPServer
import psycopg
import pytest
from PIL import Image
from app.domains.products import imagefetch, repo
from app.platform import db, objectstore
def _png(w, h):
buf = io.BytesIO(); Image.new("RGB", (w, h), (1, 2, 3)).save(buf, "PNG"); return buf.getvalue()
class _Host(BaseHTTPRequestHandler):
GOOD = _png(800, 800)
TINY = _png(64, 64)
def log_message(self, *a): pass
def do_GET(self):
if self.path.startswith("/good.png"):
self.send_response(200); self.send_header("Content-Type", "image/png")
self.end_headers(); self.wfile.write(self.GOOD)
elif self.path.startswith("/tiny.png"):
self.send_response(200); self.send_header("Content-Type", "image/png")
self.end_headers(); self.wfile.write(self.TINY)
else:
self.send_response(404); self.end_headers()
@pytest.fixture()
def host():
srv = HTTPServer(("127.0.0.1", 0), _Host)
t = threading.Thread(target=srv.serve_forever, daemon=True); t.start()
yield f"http://127.0.0.1:{srv.server_address[1]}"
srv.shutdown()
@pytest.fixture()
def pool(fresh_db_url):
with psycopg.connect(fresh_db_url) as conn:
db.migrate(conn); conn.commit()
p = db.open_pool(fresh_db_url, max_size=6)
yield p
p.close()
def _seed(pool):
"""account+storefront+run; returns (storefront_id, account_id, run_id)."""
with pool.connection() as conn:
acct = conn.execute("INSERT INTO account (email) VALUES ('m@example.com') RETURNING id").fetchone()[0]
sf = conn.execute("INSERT INTO storefront (name) VALUES ('Shop') RETURNING id").fetchone()[0]
conn.execute("INSERT INTO storefront_membership (account_id, storefront_id) VALUES (%s,%s)", (acct, sf))
rid = conn.execute(
"INSERT INTO import_run (storefront_id, account_id, file_name, dialect,"
" products_added, products_updated, rows_errored, status)"
" VALUES (%s,%s,'c.csv','canonical',0,0,0,'fetching_images') RETURNING id", (sf, acct)).fetchone()[0]
conn.commit()
return sf, acct, rid
def _product(pool, sf, handle):
with pool.connection() as conn:
pid = conn.execute("INSERT INTO product (storefront_id, handle, title) VALUES (%s,%s,%s) RETURNING id",
(sf, handle, handle.title())).fetchone()[0]
conn.commit()
return pid
def _image(pool, pid, url, rid, position=1):
with pool.connection() as conn:
iid = conn.execute("INSERT INTO product_image (product_id, source_url, position, status, import_run_id)"
" VALUES (%s,%s,%s,'pending',%s) RETURNING id", (pid, url, position, rid)).fetchone()[0]
conn.commit()
return iid
def _img_row(pool, iid):
with pool.connection() as conn:
r = conn.execute("SELECT status, key_original, key_thumb, key_card, key_detail FROM product_image WHERE id=%s",
(iid,)).fetchone()
return {"status": r[0], "key_original": r[1], "key_thumb": r[2], "key_card": r[3], "key_detail": r[4]}
def _run_status(pool, rid):
with pool.connection() as conn:
return conn.execute("SELECT status FROM import_run WHERE id=%s", (rid,)).fetchone()[0]
def test_ssrf_guard_rejects_private_when_disallowed():
with pytest.raises(imagefetch.FetchBlocked):
imagefetch.fetch_bytes("http://127.0.0.1:1/x.png", allow_private=False)
with pytest.raises(imagefetch.FetchBlocked):
imagefetch.fetch_bytes("http://169.254.169.254/latest/meta-data", allow_private=False)
with pytest.raises(imagefetch.FetchBlocked):
imagefetch.fetch_bytes("ftp://example.com/x", allow_private=False)
def test_ssrf_guard_allows_private_when_enabled(host):
data, ctype = imagefetch.fetch_bytes(f"{host}/good.png", allow_private=True)
assert ctype.startswith("image/") and len(data) > 0
def test_run_phase_classifies_each_image(pool, host, tmp_path):
store = objectstore.LocalObjectStore(str(tmp_path))
sf, _acct, rid = _seed(pool)
p = _product(pool, sf, "lamp")
ok = _image(pool, p, f"{host}/good.png", rid, position=1)
_image(pool, p, f"{host}/tiny.png", rid, position=2)
_image(pool, p, f"{host}/missing.png", rid, position=3)
imagefetch.run_image_phase(pool, store, rid, allow_private=True)
with pool.connection() as conn:
counts = repo.run_image_counts(conn, rid)
assert counts["fetched"] == 1 and counts["rejected"] == 1 and counts["failed"] == 1
row = _img_row(pool, ok)
assert row["status"] == "fetched"
for k in ("key_original", "key_thumb", "key_card", "key_detail"):
assert row[k] and store.get(row[k])
assert _run_status(pool, rid) == "complete_with_problems"
def test_unexpected_processing_error_marks_image_failed_not_wedged(pool, host, tmp_path, monkeypatch):
# If anything after the fetch raises unexpectedly (e.g. objectstore.put),
# the image is marked failed and the run still reaches a terminal status —
# never stranded in fetching_images.
store = objectstore.LocalObjectStore(str(tmp_path))
sf, _acct, rid = _seed(pool)
p = _product(pool, sf, "lamp")
iid = _image(pool, p, f"{host}/good.png", rid, position=1)
def _boom(*a, **k):
raise RuntimeError("storage down")
monkeypatch.setattr(store, "put", _boom)
imagefetch.run_image_phase(pool, store, rid, allow_private=True)
with pool.connection() as conn:
counts = repo.run_image_counts(conn, rid)
assert counts["failed"] == 1 and counts["pending"] == 0
assert _run_status(pool, rid) == "complete_with_problems"
row = _img_row(pool, iid)
assert row["status"] == "failed"
def test_resume_after_kill_completes_remaining(pool, host, tmp_path):
store = objectstore.LocalObjectStore(str(tmp_path))
sf, _acct, rid = _seed(pool)
p = _product(pool, sf, "lamp")
a = _image(pool, p, f"{host}/good.png", rid, position=1)
with pool.connection() as conn: # simulate crash: one already fetched, run still fetching_images
repo.mark_image_fetched(conn, a, {"original": "o", "thumb": "t", "card": "c", "detail": "d"})
conn.commit()
_image(pool, p, f"{host}/good.png?2", rid, position=2) # still pending
resumed = imagefetch.recover_incomplete_runs(pool, store, allow_private=True)
assert rid in resumed
with pool.connection() as conn:
assert repo.run_image_counts(conn, rid)["pending"] == 0
assert _run_status(pool, rid) == "complete"
+101
View File
@@ -0,0 +1,101 @@
"""SD-0002 invariants: INV-10 (never deletes), INV-14 (two-storefront zero bleed),
apply transactionality (§6.8), TEL-6."""
import json
import logging
import psycopg
import pytest
from app.domains import products
from app.domains.products import repo, service
from app.platform import db
CSV_A = b"Handle,Title,Variant Price\nmug,Mug,10.00\ntee,Tee,20.00\n"
CSV_PARTIAL = b"Handle,Title,Variant Price\nmug,Mug,12.00\n"
@pytest.fixture()
def migrated_conn(fresh_db_url):
with psycopg.connect(fresh_db_url) as conn:
db.migrate(conn)
yield conn
def _merchant(conn, email="m@example.com", shop="Shop"):
acct = conn.execute("INSERT INTO account (email) VALUES (%s) RETURNING id", (email,)).fetchone()[0]
sf = conn.execute("INSERT INTO storefront (name) VALUES (%s) RETURNING id", (shop,)).fetchone()[0]
conn.execute("INSERT INTO storefront_membership (account_id, storefront_id) VALUES (%s,%s)", (acct, sf))
conn.commit()
return acct, sf
def _import(conn, acct, sf, data):
d = products.import_validate(conn, sf, acct, "f.csv", data)
return products.confirm_draft(conn, sf, acct, d["id"])
def test_inv10_partial_file_never_deletes(migrated_conn):
acct, sf = _merchant(migrated_conn)
_import(migrated_conn, acct, sf, CSV_A)
before = migrated_conn.execute("SELECT count(*) FROM product").fetchone()[0]
_import(migrated_conn, acct, sf, CSV_PARTIAL)
after = migrated_conn.execute("SELECT count(*) FROM product").fetchone()[0]
assert after >= before == 2
def test_inv14_two_storefronts_zero_bleed(migrated_conn):
acct1, sf1 = _merchant(migrated_conn)
acct2, sf2 = _merchant(migrated_conn, "n@example.com", "Other")
_import(migrated_conn, acct1, sf1, CSV_A)
assert products.summary(migrated_conn, sf2)["product_count"] == 0
assert products.list_runs(migrated_conn, sf2) == []
_import(migrated_conn, acct2, sf2, CSV_A)
assert products.summary(migrated_conn, sf2)["product_count"] == 2
run1 = products.list_runs(migrated_conn, sf1)[0]
with pytest.raises(products.RunNotFound):
products.get_run(migrated_conn, sf2, run1["id"])
def test_apply_failure_rolls_back_whole_transaction_tel6(migrated_conn, monkeypatch, caplog):
acct, sf = _merchant(migrated_conn)
d = products.import_validate(migrated_conn, sf, acct, "f.csv", CSV_A)
def boom(*a, **k):
raise RuntimeError("mid-apply crash")
monkeypatch.setattr(service.repo, "insert_run_errors", boom)
lg = logging.getLogger("ecomm")
prior = lg.propagate
lg.propagate = True
try:
with caplog.at_level(logging.INFO, logger="ecomm.telemetry"):
with pytest.raises(RuntimeError):
products.confirm_draft(migrated_conn, sf, acct, d["id"])
finally:
lg.propagate = prior
assert migrated_conn.execute("SELECT count(*) FROM product").fetchone()[0] == 0
assert migrated_conn.execute("SELECT count(*) FROM import_run").fetchone()[0] == 0
assert migrated_conn.execute("SELECT count(*) FROM import_draft").fetchone()[0] == 1
events = [json.loads(r.message) for r in caplog.records if r.name == "ecomm.telemetry"]
assert any(e["event"] == "import_apply_failed" and e["error_class"] == "RuntimeError" for e in events)
# A partial Shopify export (signature: Body (HTML) + Variant Grams) naming only one
# of the two existing products — INV-10 must hold across the dialect boundary.
SHOPIFY_PARTIAL = (
b"Handle,Title,Body (HTML),Variant Price,Variant Grams\n"
b"mug,Mug,<p>x</p>,12.00,180\n"
)
def test_inv10_shopify_partial_never_deletes(migrated_conn):
acct, sf = _merchant(migrated_conn)
_import(migrated_conn, acct, sf, CSV_A)
before = migrated_conn.execute("SELECT count(*) FROM product").fetchone()[0]
d = products.import_validate(migrated_conn, sf, acct, "shopify.csv", SHOPIFY_PARTIAL)
assert d["dialect"] == "shopify"
products.confirm_draft(migrated_conn, sf, acct, d["id"])
after = migrated_conn.execute("SELECT count(*) FROM product").fetchone()[0]
# INV-10: nothing deleted; the unmentioned 'tee' is untouched.
assert after >= before == 2
handles = {r[0] for r in migrated_conn.execute("SELECT handle FROM product").fetchall()}
assert {"mug", "tee"} <= handles
@@ -0,0 +1,93 @@
"""products repo — image-phase helpers (SD-0002 §6.5.4)."""
import psycopg
import pytest
from app.domains.products import repo
from app.platform import db
@pytest.fixture()
def migrated_conn(fresh_db_url):
with psycopg.connect(fresh_db_url) as conn:
db.migrate(conn)
yield conn
@pytest.fixture()
def merchant(migrated_conn):
acct = migrated_conn.execute(
"INSERT INTO account (email) VALUES ('m@example.com') RETURNING id").fetchone()[0]
sf = migrated_conn.execute(
"INSERT INTO storefront (name) VALUES ('Shop') RETURNING id").fetchone()[0]
migrated_conn.execute(
"INSERT INTO storefront_membership (account_id, storefront_id) VALUES (%s,%s)", (acct, sf))
migrated_conn.commit()
return {"account_id": acct, "storefront_id": sf}
def _run(conn, m, status="fetching_images"):
return conn.execute(
"INSERT INTO import_run (storefront_id, account_id, file_name, dialect,"
" products_added, products_updated, rows_errored, status)"
" VALUES (%s,%s,'c.csv','canonical',0,0,0,%s) RETURNING id",
(m["storefront_id"], m["account_id"], status)).fetchone()[0]
def _product(conn, m, handle):
return conn.execute(
"INSERT INTO product (storefront_id, handle, title) VALUES (%s,%s,%s) RETURNING id",
(m["storefront_id"], handle, handle.title())).fetchone()[0]
def _image(conn, product_id, url, run_id, status="pending", position=1):
return conn.execute(
"INSERT INTO product_image (product_id, source_url, position, status, import_run_id)"
" VALUES (%s,%s,%s,%s,%s) RETURNING id",
(product_id, url, position, status, run_id)).fetchone()[0]
def test_pending_images_for_run_returns_only_pending(migrated_conn, merchant):
rid = _run(migrated_conn, merchant)
p = _product(migrated_conn, merchant, "lamp")
a = _image(migrated_conn, p, "https://m/a.png", rid, status="pending", position=1)
_image(migrated_conn, p, "https://m/b.png", rid, status="fetched", position=2)
pend = repo.pending_images_for_run(migrated_conn, rid)
assert [img["id"] for img in pend] == [a]
assert pend[0]["source_url"] == "https://m/a.png"
assert pend[0]["storefront_id"] == merchant["storefront_id"]
def test_claim_image_for_fetch_is_idempotent(migrated_conn, merchant):
rid = _run(migrated_conn, merchant)
p = _product(migrated_conn, merchant, "lamp")
img = _image(migrated_conn, p, "https://m/a.png", rid)
assert repo.claim_image_for_fetch(migrated_conn, img) is True
repo.mark_image_fetched(migrated_conn, img,
{"original": "o", "thumb": "t", "card": "c", "detail": "d"})
assert repo.claim_image_for_fetch(migrated_conn, img) is False
def test_mark_image_outcomes_and_run_counts(migrated_conn, merchant):
rid = _run(migrated_conn, merchant)
p = _product(migrated_conn, merchant, "lamp")
ok = _image(migrated_conn, p, "https://m/a.png", rid, position=1)
bad = _image(migrated_conn, p, "https://m/b.png", rid, position=2)
miss = _image(migrated_conn, p, "https://m/c.png", rid, position=3)
repo.mark_image_fetched(migrated_conn, ok, {"original": "o", "thumb": "t", "card": "c", "detail": "d"})
repo.mark_image_rejected(migrated_conn, bad, "rejected_low_res", "below the resolution bar")
repo.mark_image_failed(migrated_conn, miss, "host unreachable")
assert repo.run_image_counts(migrated_conn, rid) == {
"fetched": 1, "rejected": 1, "failed": 1, "pending": 0, "total": 3}
outcomes = repo.run_image_outcomes(migrated_conn, rid)
assert {o["handle"] for o in outcomes} == {"lamp"}
assert {o["outcome"] for o in outcomes} == {"rejected_low_res", "failed"} # fetched not listed
def test_set_run_status_and_incomplete_runs(migrated_conn, merchant):
rid = _run(migrated_conn, merchant, status="fetching_images")
_run(migrated_conn, merchant, status="complete")
assert rid in [r["id"] for r in repo.incomplete_runs(migrated_conn)]
repo.set_run_status(migrated_conn, rid, "complete")
assert rid not in [r["id"] for r in repo.incomplete_runs(migrated_conn)]
row = migrated_conn.execute("SELECT status, completed_at FROM import_run WHERE id=%s", (rid,)).fetchone()
assert row[0] == "complete" and row[1] is not None
+249
View File
@@ -0,0 +1,249 @@
"""Canonical serializer + INV-12 round-trip lock (SD-0002 §6.5.5, §6.8)."""
import csv
import io
from app.domains.products import serialize
from app.domains.products.diff import CatalogImage, CatalogProduct, CatalogVariant
def _product(**kw) -> CatalogProduct:
"""A minimal no-option catalog product (one all-NULL variant)."""
base = dict(
id=1, handle="moon-mug", title="Moon Mug",
option_names=(None, None, None),
fields={
"title": "Moon Mug", "description_html": None, "vendor": "Acme",
"product_type": "standalone", "google_product_category": None,
"tags": [], "status": "active", "published": True,
},
variants=[CatalogVariant(
id=1, options=(None, None, None), position=1,
fields={"sku": "WG-MUG", "barcode": None, "price": None, "cost": None,
"weight": None, "weight_unit": None, "volume": None,
"volume_unit": None, "tax_id_1": None, "tax_id_2": None,
"inventory_tracker": None, "inventory_qty": None,
"variant_image": None})],
images=[],
)
base.update(kw)
return CatalogProduct(**base)
def _rows(products) -> list[dict]:
text = "".join(serialize.catalog_to_csv(products))
return list(csv.DictReader(io.StringIO(text)))
def test_header_is_full_canonical_set():
text = "".join(serialize.catalog_to_csv([_product()]))
header = next(csv.reader(io.StringIO(text)))
# Handle + Title first; every known column present exactly once.
assert header[0] == "Handle"
assert "Title" in header and "Variant SKU" in header and "Image Src" in header
assert len(header) == len(set(header))
def test_single_no_option_product_one_row():
rows = _rows([_product()])
assert len(rows) == 1
assert rows[0]["Handle"] == "moon-mug"
assert rows[0]["Title"] == "Moon Mug"
assert rows[0]["Variant SKU"] == "WG-MUG"
# A no-option product emits no option values.
assert rows[0]["Option1 Value"] == ""
def _star_tee() -> CatalogProduct:
"""A 3-variant, 2-image product (the sample.csv shape)."""
return CatalogProduct(
id=2, handle="star-tee", title="Star Tee",
option_names=("Size", "Color", None),
fields={"title": "Star Tee", "description_html": None, "vendor": "Acme",
"product_type": "standalone", "google_product_category": None,
"tags": ["apparel", "tees"], "status": "active", "published": True},
variants=[
CatalogVariant(id=10, options=("S", "Indigo", None), position=1,
fields={"sku": "WG-TEE-S", "variant_image": None}),
CatalogVariant(id=11, options=("M", "Indigo", None), position=2,
fields={"sku": "WG-TEE-M", "variant_image": None}),
CatalogVariant(id=12, options=("L", "Indigo", None), position=3,
fields={"sku": "WG-TEE-L", "variant_image": None}),
],
images=[
CatalogImage(id=1, source_url="https://x/a.jpg", position=1, alt_text="front"),
CatalogImage(id=2, source_url="https://x/b.jpg", position=2, alt_text="back"),
],
)
def test_multivariant_with_images_interleaves():
rows = _rows([_star_tee()])
assert len(rows) == 3 # max(3 variants, 2 images)
# Product-level fields only on the first row.
assert rows[0]["Title"] == "Star Tee" and rows[1]["Title"] == ""
assert rows[0]["Tags"] == "apparel, tees" and rows[1]["Tags"] == ""
# Option names on row 0; option values on every variant row.
assert rows[0]["Option1 Name"] == "Size" and rows[1]["Option1 Name"] == ""
assert [r["Option1 Value"] for r in rows] == ["S", "M", "L"]
assert [r["Variant SKU"] for r in rows] == ["WG-TEE-S", "WG-TEE-M", "WG-TEE-L"]
# Two images on the first two rows; third row has no image.
assert [r["Image Src"] for r in rows] == ["https://x/a.jpg", "https://x/b.jpg", ""]
assert [r["Image Position"] for r in rows] == ["1", "2", ""]
def test_more_images_than_variants_emits_image_only_rows():
p = _product(images=[
CatalogImage(id=1, source_url="https://x/a.jpg", position=1, alt_text=None),
CatalogImage(id=2, source_url="https://x/b.jpg", position=2, alt_text=None),
CatalogImage(id=3, source_url="https://x/c.jpg", position=3, alt_text=None),
])
rows = _rows([p])
assert len(rows) == 3 # 1 variant, 3 images
assert rows[0]["Variant SKU"] == "WG-MUG"
assert rows[1]["Variant SKU"] == "" and rows[1]["Handle"] == "moon-mug"
assert [r["Image Src"] for r in rows] == ["https://x/a.jpg", "https://x/b.jpg", "https://x/c.jpg"]
def test_fetched_image_serializes_hosted_detail_url():
product = CatalogProduct(
id=1, handle="lamp", title="Lamp", option_names=(None, None, None),
fields={"title": "Lamp", "status": "active"},
variants=[CatalogVariant(id=1, options=(None, None, None), position=1, fields={})],
images=[
CatalogImage(id=55, source_url="https://m.example/a.png", position=1,
alt_text=None, status="fetched"),
CatalogImage(id=56, source_url="https://m.example/b.png", position=2,
alt_text=None, status="failed"),
],
)
rows = list(serialize._product_rows(product, base_url="https://shop.test"))
srcs = [r.get("Image Src") for r in rows if r.get("Image Src")]
assert "https://shop.test/api/products/images/55/detail" in srcs # fetched -> hosted
assert "https://m.example/b.png" in srcs # failed -> source kept
import random
from app.domains.products import codec, diff, validate
def _gen_catalog(seed: int) -> dict:
"""A deterministic random text-field catalog, in load_catalog()'s shape."""
rng = random.Random(seed)
words = ["moon", "star", "river", "cloud", "ember", "fern", "slate", "wave"]
catalog: dict[str, CatalogProduct] = {}
n_products = rng.randint(1, 6)
for pid in range(1, n_products + 1):
handle = "-".join(rng.sample(words, rng.randint(1, 3))) + f"-{pid}"
if handle in catalog:
continue
has_opts = rng.random() < 0.6
option_names = ("Size", "Color", None) if has_opts else (None, None, None)
tags = rng.sample(["apparel", "kitchen", "sale", "new"], rng.randint(0, 3))
fields = {
"title": f"{handle.title()} Thing",
"description_html": rng.choice([None, "<p>hi</p>"]),
"vendor": rng.choice([None, "Acme", "Wiggle Goods"]),
"product_type": "standalone",
"google_product_category": rng.choice([None, "Home & Garden"]),
"tags": tags,
"status": rng.choice(["draft", "active", "archived"]),
"published": rng.choice([True, False]),
}
variants = []
if has_opts:
sizes = rng.sample(["S", "M", "L", "XL"], rng.randint(1, 4))
for vi, size in enumerate(sizes, start=1):
# Non-sequential positions (×10) so a serializer that drops the
# stored position and lets re-import default to file order is
# caught — a merchant can import explicit positions like 10/20.
variants.append(CatalogVariant(
id=pid * 100 + vi, options=(size, "Indigo", None), position=vi * 10,
fields={"sku": f"SKU-{pid}-{vi}", "variant_image": None}))
else:
variants.append(CatalogVariant(
id=pid * 100 + 1, options=(None, None, None), position=rng.choice([1, 5, 9]),
fields={"sku": f"SKU-{pid}", "variant_image": None}))
images = []
for ii in range(rng.randint(0, 3)):
# Mix fetched and non-fetched images: a fetched image exports its
# hosted /images/{id}/detail URL, which the diff pre-pass resolves
# back to the same id -> still a no-op (INV-12 over hosted images).
status = "fetched" if rng.random() < 0.5 else "pending"
images.append(CatalogImage(
id=pid * 10 + ii, source_url=f"https://img/{handle}-{ii}.jpg",
position=ii + 1, alt_text=rng.choice([None, f"alt {ii}"]),
status=status))
catalog[handle] = CatalogProduct(
id=pid, handle=handle, title=fields["title"], option_names=option_names,
fields=fields, variants=variants, images=images)
return catalog
def _roundtrip_diff(catalog: dict, base_url: str = "") -> diff.DiffResult:
"""export → bytes → import pipeline → diff against the same catalog."""
text = "".join(serialize.catalog_to_csv(catalog.values(), base_url=base_url))
parsed = codec.parse_csv(text.encode("utf-8"))
products = validate.build_products(parsed)
return diff.compute_diff(catalog, products)
def test_inv12_roundtrip_is_noop_over_generated_catalogs():
for seed in range(200):
catalog = _gen_catalog(seed)
# A fixed base_url so fetched images export an absolute hosted URL; the
# diff resolves it back by id -> the round-trip stays a no-op.
result = _roundtrip_diff(catalog, base_url="https://shop.test")
assert result.summary["adds"] == 0, f"seed {seed}: {result.summary}"
assert result.summary["updates"] == 0, f"seed {seed}: {result.summary}"
assert result.summary["errors"] == 0, f"seed {seed}: {result.summary}"
assert result.summary["unchanged"] == len(catalog), f"seed {seed}"
from decimal import Decimal
def test_decimal_and_int_variant_fields_roundtrip():
catalog = {
"priced": CatalogProduct(
id=1, handle="priced", title="Priced",
option_names=(None, None, None),
fields={"title": "Priced", "description_html": None, "vendor": None,
"product_type": "standalone", "google_product_category": None,
"tags": [], "status": "active", "published": True},
variants=[CatalogVariant(
id=1, options=(None, None, None), position=1,
fields={"sku": "P1", "price": Decimal("18.00"),
"cost": Decimal("9.5"), "weight": Decimal("0.250"),
"inventory_qty": 40, "variant_image": None})],
images=[],
)
}
result = _roundtrip_diff(catalog)
assert result.summary["unchanged"] == 1, result.records
assert result.summary["updates"] == 0
def test_nonsequential_variant_position_roundtrips():
"""A stored variant position that isn't its file order must survive export —
else re-import reads the empty cell as 'reset to file order' and the round-trip
spuriously updates (INV-12 regression: position is a CatalogVariant attribute,
not a fields{} entry, so the serializer must emit it explicitly)."""
catalog = {
"tee": CatalogProduct(
id=1, handle="tee", title="Tee", option_names=("Size", None, None),
fields={"title": "Tee", "description_html": None, "vendor": None,
"product_type": "standalone", "google_product_category": None,
"tags": [], "status": "active", "published": True},
variants=[
CatalogVariant(id=1, options=("S", None, None), position=10,
fields={"sku": "T-S", "variant_image": None}),
CatalogVariant(id=2, options=("M", None, None), position=20,
fields={"sku": "T-M", "variant_image": None}),
],
images=[],
)
}
result = _roundtrip_diff(catalog)
assert result.summary["unchanged"] == 1, result.records
assert result.summary["updates"] == 0
+242
View File
@@ -0,0 +1,242 @@
"""products service — drafts: validate/preview/discard (PUC-2/3/3a/5a; INV-11)."""
import json
import logging
import psycopg
import pytest
from app.domains import products
from app.platform import db
GOOD_CSV = b"Handle,Title,Vendor,Variant Price\nmoon-mug,Moon Mug,Acme,18.00\nstar-tee,Star Tee,Acme,24.00\n"
@pytest.fixture()
def migrated_conn(fresh_db_url):
with psycopg.connect(fresh_db_url) as conn:
db.migrate(conn)
yield conn
@pytest.fixture()
def merchant(migrated_conn):
acct = migrated_conn.execute(
"INSERT INTO account (email) VALUES ('m@example.com') RETURNING id").fetchone()[0]
sf = migrated_conn.execute(
"INSERT INTO storefront (name) VALUES ('Shop') RETURNING id").fetchone()[0]
migrated_conn.execute(
"INSERT INTO storefront_membership (account_id, storefront_id) VALUES (%s,%s)", (acct, sf))
migrated_conn.commit()
return {"account_id": acct, "storefront_id": sf}
def test_import_validate_creates_draft_with_summary(migrated_conn, merchant):
draft = products.import_validate(
migrated_conn, merchant["storefront_id"], merchant["account_id"], "cat.csv", GOOD_CSV)
assert draft["dialect"] == "canonical"
assert draft["summary"] == {"adds": 2, "updates": 0, "unchanged": 0, "errors": 0}
assert draft["expires_at"]
def test_validate_writes_nothing_to_catalog_inv11(migrated_conn, merchant):
products.import_validate(
migrated_conn, merchant["storefront_id"], merchant["account_id"], "cat.csv", GOOD_CSV)
assert migrated_conn.execute("SELECT count(*) FROM product").fetchone()[0] == 0
assert migrated_conn.execute("SELECT count(*) FROM variant").fetchone()[0] == 0
def test_file_rejection_leaves_no_draft(migrated_conn, merchant):
with pytest.raises(products.FileRejected):
products.import_validate(
migrated_conn, merchant["storefront_id"], merchant["account_id"], "bad.csv",
b"Vendor,Price\nAcme,1\n")
assert migrated_conn.execute("SELECT count(*) FROM import_draft").fetchone()[0] == 0
def test_records_paging_and_kind_filter(migrated_conn, merchant):
draft = products.import_validate(
migrated_conn, merchant["storefront_id"], merchant["account_id"], "cat.csv", GOOD_CSV)
recs = products.get_draft_records(migrated_conn, merchant["storefront_id"], draft["id"])
assert [r["handle"] for r in recs] == ["moon-mug", "star-tee"]
adds = products.get_draft_records(
migrated_conn, merchant["storefront_id"], draft["id"], kind="add", limit=1)
assert len(adds) == 1 and adds[0]["kind"] == "add"
def test_discard_deletes_no_trace_puc3a(migrated_conn, merchant):
draft = products.import_validate(
migrated_conn, merchant["storefront_id"], merchant["account_id"], "cat.csv", GOOD_CSV)
products.discard_draft(migrated_conn, merchant["storefront_id"], draft["id"])
assert migrated_conn.execute("SELECT count(*) FROM import_draft").fetchone()[0] == 0
products.discard_draft(migrated_conn, merchant["storefront_id"], draft["id"]) # idempotent
def test_draft_scoped_to_storefront_inv14(migrated_conn, merchant):
draft = products.import_validate(
migrated_conn, merchant["storefront_id"], merchant["account_id"], "cat.csv", GOOD_CSV)
other_sf = migrated_conn.execute(
"INSERT INTO storefront (name) VALUES ('Other') RETURNING id").fetchone()[0]
migrated_conn.commit()
with pytest.raises(products.DraftNotFound):
products.get_draft(migrated_conn, other_sf, draft["id"])
def test_expired_draft_raises_and_lazily_deletes(migrated_conn, merchant):
draft = products.import_validate(
migrated_conn, merchant["storefront_id"], merchant["account_id"], "cat.csv", GOOD_CSV)
migrated_conn.execute(
"UPDATE import_draft SET expires_at = now() - interval '1 minute' WHERE id = %s",
(draft["id"],))
migrated_conn.commit()
with pytest.raises(products.DraftExpired):
products.get_draft(migrated_conn, merchant["storefront_id"], draft["id"])
assert migrated_conn.execute("SELECT count(*) FROM import_draft").fetchone()[0] == 0
@pytest.fixture()
def telemetry_propagation():
"""create_app() sets propagate=False on the parent "ecomm" logger
(main._ensure_app_logging), which hides ecomm.telemetry records from caplog's
root-logger handler whenever an API test ran first. Restore propagation here."""
lg = logging.getLogger("ecomm")
prior = lg.propagate
lg.propagate = True
yield
lg.propagate = prior
def test_tel1_emitted(migrated_conn, merchant, caplog, telemetry_propagation):
with caplog.at_level(logging.INFO, logger="ecomm.telemetry"):
products.import_validate(
migrated_conn, merchant["storefront_id"], merchant["account_id"], "cat.csv", GOOD_CSV)
events = [json.loads(r.message) for r in caplog.records if r.name == "ecomm.telemetry"]
assert any(
e["event"] == "import_draft_created" and e["adds"] == 2 and e["row_count"] == 2
and "duration_ms" in e and e["unknown_columns_count"] == 0
for e in events
)
UPDATE_CSV = b"Handle,Title,Vendor,Variant Price\nmoon-mug,Moon Mug,Acme,21.00\nstar-tee,Star Tee,Acme,24.00\n"
MIXED_CSV = b"Handle,Title,Variant Price\ngood-mug,Mug,10.00\nbad-tee,Tee,not-a-price\n"
def _validate(conn, m, data=GOOD_CSV):
return products.import_validate(conn, m["storefront_id"], m["account_id"], "cat.csv", data)
def test_confirm_applies_adds_and_records_run(migrated_conn, merchant):
draft = _validate(migrated_conn, merchant)
run_id = products.confirm_draft(
migrated_conn, merchant["storefront_id"], merchant["account_id"], draft["id"])
assert migrated_conn.execute("SELECT count(*) FROM product").fetchone()[0] == 2
run = products.get_run(migrated_conn, merchant["storefront_id"], run_id)
assert run["products_added"] == 2 and run["status"] == "complete"
assert run["by"] == "m@example.com"
assert run["image_progress"] == {"done": 0, "total": 0} and run["image_outcomes"] == []
assert migrated_conn.execute("SELECT count(*) FROM import_draft").fetchone()[0] == 0
def test_confirm_update_changes_only_diffed_fields(migrated_conn, merchant):
d1 = _validate(migrated_conn, merchant)
products.confirm_draft(migrated_conn, merchant["storefront_id"], merchant["account_id"], d1["id"])
d2 = _validate(migrated_conn, merchant, UPDATE_CSV)
assert d2["summary"] == {"adds": 0, "updates": 1, "unchanged": 1, "errors": 0}
products.confirm_draft(migrated_conn, merchant["storefront_id"], merchant["account_id"], d2["id"])
price = migrated_conn.execute(
"SELECT v.price FROM variant v JOIN product p ON p.id = v.product_id WHERE p.handle='moon-mug'"
).fetchone()[0]
assert str(price) == "21.00"
assert migrated_conn.execute("SELECT count(*) FROM product").fetchone()[0] == 2 # no dupes (BUC-3)
def test_confirm_blank_position_cell_round_trips(migrated_conn, merchant):
# A present-but-empty Variant Position cell resolves to file order at diff
# time — never SET position = NULL (which would abort the confirm on the
# NOT NULL constraint).
d1 = _validate(migrated_conn, merchant)
products.confirm_draft(migrated_conn, merchant["storefront_id"], merchant["account_id"], d1["id"])
blank_position_csv = (
b"Handle,Title,Vendor,Variant Price,Variant Position\n"
b"moon-mug,Moon Mug,Acme,21.00,\n"
b"star-tee,Star Tee,Acme,24.00,\n"
)
d2 = _validate(migrated_conn, merchant, blank_position_csv)
products.confirm_draft(migrated_conn, merchant["storefront_id"], merchant["account_id"], d2["id"])
price = migrated_conn.execute(
"SELECT v.price FROM variant v JOIN product p ON p.id = v.product_id WHERE p.handle='moon-mug'"
).fetchone()[0]
assert str(price) == "21.00"
def test_confirm_mixed_applies_valid_records_errors(migrated_conn, merchant):
draft = _validate(migrated_conn, merchant, MIXED_CSV)
run_id = products.confirm_draft(
migrated_conn, merchant["storefront_id"], merchant["account_id"], draft["id"])
assert migrated_conn.execute("SELECT count(*) FROM product").fetchone()[0] == 1
run = products.get_run(migrated_conn, merchant["storefront_id"], run_id)
assert run["rows_errored"] == 1
assert run["errors"][0]["column"] == "Variant Price"
def test_confirm_stale_fingerprint_409_inv11(migrated_conn, merchant):
draft = _validate(migrated_conn, merchant)
other = _validate(migrated_conn, merchant)
products.confirm_draft(migrated_conn, merchant["storefront_id"], merchant["account_id"], other["id"])
with pytest.raises(products.PreviewStale):
products.confirm_draft(migrated_conn, merchant["storefront_id"], merchant["account_id"], draft["id"])
def test_confirm_nothing_to_apply_puc10(migrated_conn, merchant):
d1 = _validate(migrated_conn, merchant)
products.confirm_draft(migrated_conn, merchant["storefront_id"], merchant["account_id"], d1["id"])
d2 = _validate(migrated_conn, merchant)
assert d2["summary"]["unchanged"] == 2
with pytest.raises(products.NothingToApply):
products.confirm_draft(migrated_conn, merchant["storefront_id"], merchant["account_id"], d2["id"])
def test_runs_history_newest_first(migrated_conn, merchant):
d1 = _validate(migrated_conn, merchant)
products.confirm_draft(migrated_conn, merchant["storefront_id"], merchant["account_id"], d1["id"])
d2 = _validate(migrated_conn, merchant, UPDATE_CSV)
r2 = products.confirm_draft(migrated_conn, merchant["storefront_id"], merchant["account_id"], d2["id"])
runs = products.list_runs(migrated_conn, merchant["storefront_id"])
assert [r["id"] for r in runs][0] == r2
def test_summary_counts(migrated_conn, merchant):
assert products.summary(migrated_conn, merchant["storefront_id"]) == {
"product_count": 0, "image_problem_count": 0, "latest_run_id": None}
d = _validate(migrated_conn, merchant)
rid = products.confirm_draft(migrated_conn, merchant["storefront_id"], merchant["account_id"], d["id"])
s = products.summary(migrated_conn, merchant["storefront_id"])
assert s == {"product_count": 2, "image_problem_count": 0, "latest_run_id": rid}
def test_tel2_emitted_on_confirm(migrated_conn, merchant, caplog, telemetry_propagation):
draft = _validate(migrated_conn, merchant)
with caplog.at_level(logging.INFO, logger="ecomm.telemetry"):
products.confirm_draft(migrated_conn, merchant["storefront_id"], merchant["account_id"], draft["id"])
events = [json.loads(r.message) for r in caplog.records if r.name == "ecomm.telemetry"]
assert any(e["event"] == "import_run_completed" and e["added"] == 2 for e in events)
def test_confirm_marks_run_fetching_images_when_images_present(migrated_conn, merchant):
csv = b"Handle,Title,Image Src\nlamp,Lamp,https://m.example/a.png\n"
draft = products.import_validate(
migrated_conn, merchant["storefront_id"], merchant["account_id"], "c.csv", csv)
run_id = products.confirm_draft(
migrated_conn, merchant["storefront_id"], merchant["account_id"], draft["id"])
run = products.get_run(migrated_conn, merchant["storefront_id"], run_id)
assert run["status"] == "fetching_images"
assert run["image_progress"]["total"] >= 1
def test_confirm_marks_run_complete_when_no_images(migrated_conn, merchant):
csv = b"Handle,Title,Vendor\nlamp,Lamp,Acme\n"
draft = products.import_validate(
migrated_conn, merchant["storefront_id"], merchant["account_id"], "c.csv", csv)
run_id = products.confirm_draft(
migrated_conn, merchant["storefront_id"], merchant["account_id"], draft["id"])
assert products.get_run(migrated_conn, merchant["storefront_id"], run_id)["status"] == "complete"
+164
View File
@@ -0,0 +1,164 @@
"""§6.5.1 row validation — every row-error rule has a fixture (SD-0002 §6.8)."""
from decimal import Decimal
import pytest
from app.domains.products.codec import parse_csv
from app.domains.products.validate import build_products
def _products(*lines: str):
return build_products(parse_csv(("\n".join(lines) + "\n").encode()))
def _errors(*lines: str):
return [e for p in _products(*lines) for e in p.errors]
def test_simple_product_parses_clean():
[p] = _products(
"Handle,Title,Vendor,Tags,Status,Published,Variant Price,Variant SKU",
"moon-mug,Moon Mug,Acme,\"kitchen, mugs\",active,TRUE,18.00,SKU-1",
)
assert p.valid and p.handle == "moon-mug" and p.title == "Moon Mug"
assert p.fields["tags"] == ["kitchen", "mugs"]
assert p.fields["status"] == "active" and p.fields["published"] is True
[v] = p.variants
assert v.options == (None, None, None)
assert v.fields["price"] == Decimal("18.00") and v.fields["sku"] == "SKU-1"
def test_option_product_groups_consecutive_rows():
[p] = _products(
"Handle,Title,Option1 Name,Option1 Value,Variant Price",
"tee,Tee,Size,S,24.00",
"tee,,,M,24.00",
"tee,,,L,26.00",
)
assert p.valid and p.option_names == ("Size", None, None)
assert [v.options[0] for v in p.variants] == ["S", "M", "L"]
def test_image_only_rows_and_dedupe():
[p] = _products(
"Handle,Title,Image Src,Image Position,Image Alt Text",
"mug,Mug,https://x/a.jpg,1,front",
"mug,,https://x/b.jpg,2,back",
"mug,,https://x/a.jpg,3,dupe",
)
assert p.valid
assert [(i.source_url, i.position) for i in p.images] == [
("https://x/a.jpg", 1), ("https://x/b.jpg", 2),
]
def test_variant_image_joins_product_images():
[p] = _products(
"Handle,Title,Variant Image",
"mug,Mug,https://x/v.jpg",
)
assert p.variants[0].fields["variant_image"] == "https://x/v.jpg"
assert [i.source_url for i in p.images] == ["https://x/v.jpg"]
def test_description_sanitized_inv15():
[p] = _products(
"Handle,Title,Description",
'mug,Mug,"<p onclick=\'x()\'>hi</p><script>evil()</script>"',
)
html = p.fields["description_html"]
assert "<p>" in html and "script" not in html and "onclick" not in html
def test_blank_cell_clears_absent_column_missing():
[p] = _products("Handle,Title,Vendor", "mug,Mug,")
assert p.fields["vendor"] is None # present-but-empty == clear
assert "status" not in p.fields # absent column == untouched
@pytest.mark.parametrize(
"header,row,column,fragment",
[
("Handle,Title", ",NoHandle", "Handle", "needs a Handle"),
("Handle,Title", "Bad_Handle!,T", "Handle", "isn't a valid handle"),
("Handle,Title", "mug,", "Title", "missing its Title"),
("Handle,Title,Type", "mug,Mug,kit_virtual", "Type", "kits arrive"),
("Handle,Title,Component 1 SKU", "mug,Mug,ABC", "Component 1 SKU", "kits arrive"),
("Handle,Title,Status", "mug,Mug,live", "Status", "is not a status"),
("Handle,Title,Published", "mug,Mug,YES", "Published", "is not TRUE or FALSE"),
("Handle,Title,Variant Price", 'mug,Mug,"12,50"', "Variant Price", "is not a price"),
("Handle,Title,Variant Cost", "mug,Mug,-3", "Variant Cost", "is not a price"),
("Handle,Title,Variant Weight", "mug,Mug,heavy", "Variant Weight", "is not a number"),
("Handle,Title,Variant Inventory Qty", "mug,Mug,3.5", "Variant Inventory Qty", "is not a whole number"),
("Handle,Title,Variant Position", "mug,Mug,0", "Variant Position", "is not a position"),
("Handle,Title,Option1 Value", "mug,Mug,Red", "Option1 Value", "no Option1 Name"),
],
)
def test_row_error_rules(header, row, column, fragment):
errors = _errors(header, row)
assert any(e.column == column and fragment in e.message for e in errors), errors
def test_missing_option_value_for_named_option():
errors = _errors(
"Handle,Title,Option1 Name,Option1 Value,Variant SKU",
"tee,Tee,Size,S,A",
"tee,,,,B",
)
assert any("missing its Option1 Value" in e.message and e.line_number == 3 for e in errors)
def test_duplicate_option_combo_is_error():
errors = _errors(
"Handle,Title,Option1 Name,Option1 Value",
"tee,Tee,Size,S",
"tee,,,S",
)
assert any("duplicate variant" in e.message for e in errors)
def test_second_variant_on_no_option_product_is_error():
errors = _errors(
"Handle,Title,Variant SKU",
"mug,Mug,A",
"mug,,B",
)
assert any("without options can have only one variant" in e.message for e in errors)
def test_non_consecutive_handle_is_error():
errors = _errors(
"Handle,Title",
"mug,Mug",
"tee,Tee",
"mug,",
)
assert any("must be consecutive" in e.message and e.line_number == 4 for e in errors)
def test_no_data_row_is_error():
errors = _errors(
"Handle,Title,Variant SKU,Image Src",
"mug,Mug,A,",
"mug,,,",
)
assert any("no variant or image data" in e.message for e in errors)
def test_errors_poison_their_product_only():
products = _products(
"Handle,Title,Variant Price",
"good-mug,Mug,10.00",
"bad-tee,Tee,not-a-price",
)
by_handle = {p.handle: p for p in products}
assert by_handle["good-mug"].valid
assert not by_handle["bad-tee"].valid
def test_all_errors_collected_not_first_only():
[p] = _products(
"Handle,Title,Status,Variant Price",
"mug,,bogus,abc",
)
assert len(p.errors) == 3 # missing Title + bad status + bad price
+44
View File
@@ -0,0 +1,44 @@
# deployment.toml — generated by define-deployment (launch-app SPEC §5.1, §6.6).
# Derived from the One Name 'ecomm' (§3.3); edit the toml, re-import to reconcile (§3.2.1).
[app]
name = "ecomm"
repo = "wiggleverse/wiggleverse-ecomm"
gitea_host = "https://git.wiggleverse.org"
version_source = { kind = "file", path = "VERSION" }
gitea_read_secret_ref = "wiggleverse-ecomm/ecomm-gitea-read-token"
[vm]
name = "ecomm-ppe"
zone = "us-central1-a"
project = "wiggleverse-ecomm"
machine_type = "e2-micro"
disk_gb = 10
service_user = "ecomm"
install_dir = "/opt/ecomm"
systemd_unit = "ecomm.service"
tunnel_through_iap = true
gcloud_config = "wiggleverse-ecomm"
[edge]
domain = "ecomm-ppe.wiggleverse.org"
# ecomm's health endpoint is /healthz (SD-0001 §6.4); body carries {status, version}.
health_url = "https://ecomm-ppe.wiggleverse.org/healthz"
# No DATABASE_PATH: ecomm runs Cloud SQL PostgreSQL (SD-0001 D-7/D-8) — the DSN is the
# ECOMM_DATABASE_URL secret reference below, minted by launch-app's provision-datastore.
[overlay] # non-secret env, plaintext (guide §8)
APP_URL = "https://ecomm-ppe.wiggleverse.org"
ECOMM_MAILER = "smtp"
ECOMM_COOKIE_SECURE = "1"
ECOMM_SMTP_HOST = "smtp.gmail.com"
ECOMM_SMTP_PORT = "587"
ECOMM_SMTP_USER = "ben.stull@wiggleverse.org"
ECOMM_SMTP_FROM = "ecomm <ben.stull@wiggleverse.org>"
ECOMM_OBJECTSTORE_KIND = "gcs"
ECOMM_OBJECTSTORE_BUCKET = "wiggleverse-ecomm-ppe-media"
[secrets] # REFERENCES only — never bytes (§8.3)
ECOMM_SESSION_SECRET = "wiggleverse-ecomm/ecomm-ppe-session-secret"
ECOMM_DATABASE_URL = "wiggleverse-ecomm/ecomm-ppe-database-url"
ECOMM_SMTP_PASSWORD = "wiggleverse-ohm/ohm-rfc-app-smtp-password"
+12
View File
@@ -122,6 +122,18 @@ From empty persistence: the deploy migrates the schema at startup (INV-7); then
real sign-up → one-time code arriving by **real email** → create storefront → admin,
through the public flows alone. Record each rehearsal here when it happens.
**Rehearsals:**
- **2026-06-11 — PPE first bootstrap (v0.4.0, session 0024): ✅** First-ever ecomm
deploy (`flotilla-core deploy ecomm`, 9/9 phases green, deploys.id=40) onto a
freshly provisioned environment — project `wiggleverse-ecomm`, Cloud SQL
`ecomm-ppe-pg` (provisioned by launch-app `provision-datastore`, first run), VM
`ecomm-ppe`. The app self-migrated the empty database at startup; the operator
then walked sign-up → real emailed code (Gmail relay) → create storefront →
honestly-empty admin in a browser, through the public flows alone. Findings
captured: OTC email branding (#16); identity should outgrow ecomm
(engineering#49/#50).
## Production
Lands with the prod stand-up — the **identical gesture** on a prod deployment record
+219
View File
@@ -0,0 +1,219 @@
# Operating ecomm
The framework-repo operator guide (SD-0002 DOC-1), started at SLICE-5. It covers
the app's operational surface — telemetry, runbooks, alert gestures, the E2E
gate. Per-deployment mechanics (deploy, secrets, VM access) live in the
deployment's flotilla docs and `deployment.toml`; environment bring-up is in
[`BOOTSTRAP.md`](./BOOTSTRAP.md).
## Products import/export ops (SD-0002, SLICE-5)
The §6.4 surface: a merchant uploads a catalog CSV (`POST
/api/products/imports`), the app validates it and stores an **import draft**
with a full preview (adds / updates / unchanged / errors), the merchant
confirms or cancels at the preview gate, and a confirm applies the previewed
diff as one **import run** recorded in the history (`/api/products/imports/runs`).
Caps and behavior to know (all enforced in code, not config):
- **Caps (INV-18):** ≤ 5,000 data rows and ≤ 10 MB per file —
`MAX_DATA_ROWS` / `MAX_FILE_BYTES` in `backend/app/domains/products/models.py`,
enforced in `backend/app/domains/products/codec.py` (file-level rejection)
plus a 413 `file_too_large` guard in the BFF (`backend/app/main.py`).
- **Draft expiry:** ~1 hour (`expires_at = now() + interval '1 hour'`). Cleanup
is a lazy sweep — expired drafts are deleted on the next upload and on any
access to an expired draft; there is no background job to babysit.
- **Upsert-only (INV-10):** an import adds and updates, never deletes. Catalog
products/variants/images absent from the file are untouched.
- **One-transaction apply (INV-11):** a confirm applies the whole previewed
diff in a single DB transaction — it lands completely or not at all.
### Telemetry
Structured JSON events on the `ecomm.telemetry` logger
(`backend/app/platform/telemetry.py`), one JSON object per line, emitted from
`backend/app/domains/products/service.py`. The app's `ecomm.*` log handler
writes to the process's stderr, which journald captures on the VM (and Cloud
Logging where the agent ships it). Events carry counts and durations only —
never file names, URLs, catalog content, or secret bytes.
| Event | Trigger | Payload fields |
| --- | --- | --- |
| TEL-1 `import_draft_created` | validation completes, draft stored | `storefront_id, dialect, row_count, adds, updates, unchanged, errors, unknown_columns_count, duration_ms` |
| TEL-2 `import_run_completed` | apply transaction commits | `run_id, storefront_id, added, updated, errored, duration_ms` |
| TEL-3 `catalog_exported` | export stream completes | `storefront_id, status_filter, product_count, duration_ms` |
| TEL-6 `import_apply_failed` | apply transaction aborts unexpectedly | `draft_id, storefront_id, error_class` |
| TEL-4 `image_phase_completed` | run's fetch task finishes | `run_id, fetched, rejected, failed, duration_ms` |
| TEL-5 `image_phase_recovered` | startup recovery resumes a run | `run_id, pending_resumed` |
### Export (PUC-9, SLICE-6)
`GET /api/products/export?status=all|active|draft|archived` streams the
storefront's catalog as a canonical-format CSV (one codec, two directions — the
same format the importer parses). It is **read-only** (no draft, no run) and
storefront-scoped (INV-14). An empty catalog — no products, or none matching the
status filter — returns `409 empty_catalog`; the Products page disables the
Export action with a note in that case. The round-trip is lossless (INV-12):
re-importing an unmodified export previews as all-unchanged with the import
action disabled (PUC-10). TEL-3 (`catalog_exported`) is emitted once the stream
completes — counts and duration only, never catalog content.
### Images (SLICE-7)
After a merchant confirms an import, the app transitions the run to
`fetching_images` and processes image URLs via a bounded thread pool
(4 workers). Each image goes through the SSRF guard, is decoded, and up to
four WebP renditions (`original`, `thumb`, `card`, `detail`) are written to
the objectstore under `product-images/`. Progress is tracked per run in
`image_progress` / `image_counts` / `image_outcomes` and visible in the
import history detail panel (PUC-8). TEL-4 (`image_phase_completed`) fires
when the phase finishes; startup recovery emits TEL-5 (`image_phase_recovered`)
when it resumes a stuck run.
#### `provision-bucket` — one-time gesture per environment (SLICE-7 PPE deploy)
Before the first SLICE-7 deploy, run the `provision-bucket` skill from the
engineering `launch-app` suite. This gesture:
1. Creates the GCS bucket `wiggleverse-ecomm-ppe-media` in the
`wiggleverse-ecomm` GCP project.
2. Grants the PPE VM's service account `roles/storage.objectAdmin` on the
bucket.
3. Sets an `import-drafts/` lifecycle rule (forward-compat for the draft-blob
migration deferred past SLICE-7).
After provisioning, `deployment.toml`'s `[overlay]` already carries:
```
ECOMM_OBJECTSTORE_KIND = "gcs"
ECOMM_OBJECTSTORE_BUCKET = "wiggleverse-ecomm-ppe-media"
```
so the next `flotilla deploy` picks them up automatically. GCS auth is the
VM service-account ADC — no credential bytes needed.
### RB-3 — image phase stuck
Triggered by ALR-3 (run in `fetching_images` > 2 h, or the same run recovered
≥ 3×).
1. **Identify the stuck run.** The import history (PUC-8, `GET
/api/products/imports/runs`) lists runs by status; look for a run with
`status="fetching_images"` that hasn't advanced.
```
journalctl -u ecomm.service | grep image_phase
```
2. **Restart to trigger recovery.** A process restart causes the startup
recovery scan to re-enqueue pending images for any run still in
`fetching_images` (TEL-5 confirms the resumption). Restarts are safe —
per-image claims are idempotent; already-fetched images are skipped.
```
systemctl restart ecomm.service
```
3. **If a specific image URL is the cause.** The `image_outcomes` field on
the run records per-image rejection reasons (SSRF block, resolution
rejection, network error, etc.). The merchant can update or remove the
offending URL and re-import.
4. **File a bug** on `wiggleverse/wiggleverse-ecomm` with the `run_id` and
the `image_outcomes` content from the log.
### ALR-3 — the log-based alert for stuck image phases
Run once per environment, at the SLICE-7 PPE deploy. Mirrors ALR-2's
approach — a log metric over the stuck/recovered signal, the existing email
channel, and an alert policy in project `wiggleverse-ecomm`.
```
# Select the deployment's gcloud config for this one process (handbook §8.4).
export CLOUDSDK_ACTIVE_CONFIG_NAME=wiggleverse-ecomm
# Log-based metric counting image-phase events (TEL-4/TEL-5 — stuck/recovered).
gcloud logging metrics create ecomm_image_phase_events \
--description="ecomm TEL-4/TEL-5 image phase completed/recovered (SD-0002 ALR-3)" \
--log-filter='resource.type="gce_instance" AND (jsonPayload.message:"image_phase_completed" OR jsonPayload.message:"image_phase_recovered" OR textPayload:"image_phase_completed" OR textPayload:"image_phase_recovered")'
```
Then attach an alert policy — **operator email channel, threshold any event >
0 within a 2-hour window, severity notify-only**. List channels first:
```
# Find the operator email channel's id for the policy.
gcloud beta monitoring channels list
```
Create the alert policy (Cloud Console or `gcloud alpha monitoring policies
create`); condition: `fetching_images` duration > 2 h **or** the same
`run_id` appears in TEL-5 ≥ 3 times.
### RB-2 — import apply failed
Triggered by ALR-2 (any TEL-6 event). The apply raised mid-transaction and
rolled back.
1. **Locate the failure.** On the VM, filter the journal for the event and note
the `draft_id`, `storefront_id`, and `error_class`:
```
journalctl -u ecomm.service | grep import_apply_failed
```
2. **Confirm the rollback held (INV-11).** The apply is one transaction, so a
failure leaves the catalog exactly as it was: the storefront's runs history
(`GET /api/products/imports/runs`) shows **no new run**, and the products
summary (`GET /api/products/summary`) shows an unchanged `product_count`.
3. **The merchant's draft is intact.** A failed apply does not consume the
draft — the merchant can retry confirm, or re-upload if the draft has since
expired (~1 h). Advise accordingly.
4. **File a bug** on `wiggleverse/wiggleverse-ecomm` with the `error_class`
and the surrounding log context (the traceback is in the app log next to
the event).
### ALR-2 — the log-based alert (one gesture per environment)
Run once per environment, at this slice's PPE deploy. This is an ad hoc op on
the existing GCP project (`wiggleverse-ecomm`), not a provisioning gesture.
```
# Select the deployment's gcloud config for this one process (handbook §8.4).
export CLOUDSDK_ACTIVE_CONFIG_NAME=wiggleverse-ecomm
# Log-based metric counting import-apply failures (TEL-6).
gcloud logging metrics create ecomm_import_apply_failed \
--description="ecomm TEL-6 import_apply_failed events (SD-0002 ALR-2)" \
--log-filter='resource.type="gce_instance" AND jsonPayload.message:"import_apply_failed" OR textPayload:"import_apply_failed"'
```
Then attach an alert policy to the metric — **operator email channel, threshold
any event > 0 in 5 minutes, severity notify-only** (pre-v1: no paging). The
policy is created in the Cloud Console or with
`gcloud alpha monitoring policies create`; the exact command depends on the
notification-channel id, so list channels first:
```
# Find the operator email channel's id for the policy.
gcloud beta monitoring channels list
```
### E2E browser suite
- Lives at `e2e/` — Playwright, Chromium, six scenarios (SLICE-5:
preview/confirm happy path, actionable errors, file rejection, cancel;
SLICE-6: `e2e_export_download`, `e2e_roundtrip_noop`).
- Run with `bash scripts/e2e.sh`. The harness boots a **fresh `ecomm_e2e`
database** against the local compose Postgres and serves the built SPA from
the backend on **:8765** (the deployed topology), so it needs the dev
Postgres up (`scripts/dev.sh`).
- **Not in `scripts/check.sh` / CI yet** — the Gitea runner has no browsers
(the §10.6 machinery gap). Run it locally before merge, and against PPE per
the §9 pipeline (the PPE browser run is still manual this slice).
## Cross-references
- SLO and alert definitions: SD-0002 §9–§10 (content repo,
`wiggleverse-ecomm-content/specs/SD-0002-products-bulk-csv-import-export.md`).
- Import/diff engine internals: [`products-domain.md`](./products-domain.md).
+275
View File
@@ -0,0 +1,275 @@
# products domain — developer notes
DOC-4 (SD-0002 §11): the import/export spine as built in SLICE-5, extended by
SLICE-68. Operator-facing material is in [`OPERATIONS.md`](./OPERATIONS.md);
the spec is SD-0002 in the content repo.
## Layout and pipeline
`backend/app/domains/products/` is layered like the rest of the app
(main → domains → platform, enforced by import-linter):
- `models.py` — canonical row model + the column registry.
- `codec.py` — bytes → `ParsedFile`; file-level gates only.
- `validate.py` — rows → `CanonicalProduct` blocks + per-row errors.
- `diff.py` — catalog × canonical products → apply plan + preview records.
- `serialize.py` — the export half: `CatalogProduct` snapshot → canonical CSV
(the inverse of `codec`/`validate`). DB-free, like `diff.py`.
- `repo.py` — SQL only: catalog snapshot, draft/run CRUD, apply primitives.
Never commits or rolls back.
- `service.py` — use-case orchestration; owns every transaction boundary and
emits the TEL events via `backend/app/platform/telemetry.py`.
```mermaid
flowchart LR
subgraph validate_path [import_validate]
A[parse_csv] --> B[build_products] --> D[compute_diff] --> E[(import_draft)]
C[load_catalog] --> D
end
subgraph confirm_path [confirm_draft]
E --> F[re-derive: parse → build → diff] --> G{fingerprint match?}
G -- yes --> H[one-transaction apply<br/>run + products + delete draft]
G -- no --> I[PreviewStale<br/>draft kept]
end
```
Confirm re-derives everything from the draft's stored `file_bytes` against the
live catalog, then checks the fingerprint — so what lands is exactly what the
preview showed, or the confirm refuses (`preview_stale`). A confirm with no
adds and no updates refuses with `nothing_to_apply`.
## Canonical model and blank-vs-absent
`models.py` is the one model every dialect maps to (INV-17). `KNOWN_COLUMNS`
is the registry header detection, unknown-column warnings, and validation all
read; `CLEAR_DEFAULTS` holds the reset values for clearable fields
(`status`, `published`, `product_type`, `tags`).
The §6.5.1 cell semantics, as implemented:
- **Absent column** → the field never enters `fields{}` → untouched by diff
and apply (never a change).
- **Present-but-empty cell** → `fields[name] = None` (an explicit clear) →
resolved at diff time to its `CLEAR_DEFAULTS` entry, or NULL where none
exists.
- **The position exception:** a cleared `Variant Position` has no
`CLEAR_DEFAULTS` entry — `diff.resolved_variant_fields` resolves it to the
variant's 1-based file order within its product, never to NULL.
Option names live both in `CanonicalProduct.option_names` (the values) and in
`fields{}` (the file-presence marker the diff needs for the absent-vs-clear
distinction).
## Dialects (INV-17, SLICE-8)
A file is normalized to canonical names **at the codec boundary**, so every stage
downstream of `parse_csv` (validate → diff → apply) is dialect-agnostic — it only
ever sees canonical cells. Two dialects exist: `canonical` and `shopify`.
`dialect_shopify.py` is the Shopify adapter and the source of truth for the
mapping (mirrored by `tests/test_products_dialect_shopify.py` and
`tests/fixtures/shopify-export.csv`):
- **Detection** — `is_shopify_header()` returns `shopify` iff the header carries a
Shopify-only *signature* column (`Body (HTML)`, `Variant Grams`, `Cost per item`,
`Gift Card`, a `Google Shopping / …` column, …) **and no canonical-distinctive
column**. A canonical-distinctive name — one Shopify renames away (`Description`,
`Variant Cost`, `Variant Weight`, `Google Product Category`) or a canonical-only
column Shopify lacks (`Variant Volume`, `Variant Tax ID 1/2`, `Variant Position`) —
**vetoes** Shopify detection. Detection is by signature, not an exact header set, so
extra market/region columns don't defeat it; the veto biases it conservative, because
under-detection is safe (Shopify-named columns are warned as not-imported) while
over-detection corrupts (canonical `Type` / `Variant Weight Unit` would be dropped).
So a hybrid or ambiguous header falls through to `canonical`, which maps shared
columns identically and warns the rest — it never misparses (§7.4).
- **Mapping** — `map_shopify_header(header)` returns `(mapped, not_imported)`:
`mapped[i]` is the canonical name for column `i` (or `None` when it has no
canonical home), and `not_imported` is the warning list surfaced as the draft's
`unknown_columns`. The rule per column, in order: in `SHOPIFY_RENAME`
canonical name; in `SHOPIFY_DROP` (`Type`, `Variant Weight Unit`) → not imported;
in `KNOWN_COLUMNS` → pass through; else → not imported (this last branch absorbs
the unbounded `… / <Market>` price columns without enumerating them).
- **The two name-collision overrides** are why `SHOPIFY_DROP` exists: Shopify's
`Type` is a free-text category, but canonical `Type` is *structural*
(`standalone`/kits), so it is warned rather than folded in; and Shopify's
`Variant Weight Unit` is dropped because the `Variant Grams``Variant Weight`
rename is the weight source.
- **The one value transform** lives in `parse_csv`: when a Shopify row has a mapped
`Variant Weight` (from `Variant Grams`), the codec synthesizes
`Variant Weight Unit = "g"`, since Shopify grams are unitless integers.
`detect_dialect()` in `codec.py` is the INV-17 seam; a new dialect is a new adapter
module plus a branch there, with no change to validate/diff/apply.
## Error granularity
`validate.py` never raises on a row problem: every violation is recorded as a
merchant-language `RowError` and **poisons its whole product block** — the
product previews as `kind="error"` and is excluded from apply, while parsing
continues so one pass yields a complete accounting (BUC-1a). On apply, error
rows are recorded per line in `import_run_error`; `rows_errored` on the run is
the **error-row count**, while the preview's errors tile counts error
**products** — the two numbers legitimately differ.
File-level problems (`not_csv`, `missing_required_column`, `too_many_rows`,
`file_too_large`) raise `FileRejected` in `codec.py` instead: no draft is
created. The BFF adds an early 413 for oversized uploads (`main.py`).
## INV-11 mechanics
`compute_diff` makes **one walk** that produces two views of the same
computation: the typed apply `plan` (resolved natives — `Decimal`, `bool`,
lists) that `confirm_draft` executes, and the JSON-safe preview `records`
stored as draft JSONB and served verbatim to the SPA. Because both derive from
the same walk they cannot diverge. The `fingerprint` is
`sha256(json.dumps(records, sort_keys=True, separators=(",", ":")))`; a
mismatch at confirm means the catalog drifted since preview → `PreviewStale`
(409 `preview_stale`, draft kept for re-validation). The whole apply — run row,
product/variant/image writes, error rows, draft delete — is one transaction;
any exception rolls it back and emits TEL-6.
## Export & the round-trip (SLICE-6)
`serialize.py` is the export half of "one codec, two directions": it turns the
`CatalogProduct` snapshot (the same one `repo.load_catalog` builds for the diff
engine) back into canonical CSV, writing exactly the columns `codec.py` /
`validate.py` parse, in the §6.5.1 row grammar — `HEADER` is the full canonical
column set; product fields + option names sit on the first row; one variant per
row; images interleave; a product with more images than variants emits
image-only rows.
`repo.export_catalog` returns the status-filtered snapshot list (sorted by
handle, deterministic); the `service.export_catalog` generator streams
`serialize.catalog_to_csv` over it and emits TEL-3 (`catalog_exported`) once the
stream is exhausted. The BFF wraps it in a `StreamingResponse`; an empty
(filtered) catalog raises `EmptyCatalog` **eagerly**`409 empty_catalog`
before any bytes stream.
**INV-12** (`diff(catalog, import(export(catalog))) = ∅`) is locked two ways: a
property test (`test_products_serialize.py`) runs the real
export→parse→diff loop over 200 generated **text-field** catalogs and asserts
every product is `unchanged`; the `e2e_roundtrip_noop` browser scenario does the
same through the UI (export download → re-upload → all-unchanged preview, import
disabled). Numeric/`Decimal` fields — the property test's deliberate blind spot
(string-form vs value-identity) — get explicit round-trip unit tests.
## Image pipeline (SLICE-7)
### `platform/objectstore`
A two-adapter port (`backend/app/platform/objectstore/`):
- `local.py` — writes blobs under a configurable local directory; used in
tests and local dev (`ECOMM_OBJECTSTORE_KIND=local`).
- `gcs.py` — wraps `google-cloud-storage`; bucket name comes from
`ECOMM_OBJECTSTORE_BUCKET` (`ECOMM_OBJECTSTORE_KIND=gcs`). Auth is the
VM's service-account ADC — no credential bytes in the overlay or secrets.
Object keys for product images follow the prefix `product-images/` (the
`provision-bucket` gesture also sets a lifecycle rule on `import-drafts/` for
forward-compat, even though the draft blob remains BYTEA this slice — see
seams below).
### `platform/images`
`backend/app/platform/images.py` decodes an uploaded or fetched image byte
string and produces up to four renditions stored in the objectstore:
- **`original`** — stored verbatim after the resolution bar passes.
- **`thumb`** — 150 px on the shorter side, WebP.
- **`card`** — 400 px on the shorter side, WebP.
- **`detail`** — 800 px on the shorter side, WebP.
**Resolution bar (`MIN_IMAGE_SHORT_SIDE = 500`):** images whose shorter side
is below 500 px are rejected (Q-3). This is the only hard gate; oversized
images are scaled down without rejection. All renditions share the same
objectstore key prefix (`product-images/<image_id>/`).
### `domains/products/imagefetch.py` — the fetch phase
After `confirm_draft` commits, the run enters `fetching_images` (if it has
any pending image URLs) and a `ThreadPoolExecutor(max_workers=4)` processes
them concurrently:
- **SSRF guard (INV-18, extended):** only `http`/`https` schemes are
permitted; the resolved IP must be public (RFC-1918, loopback, link-local,
and multicast ranges are blocked); redirects are followed only within the
same guard — a redirect to a private IP is rejected mid-chain.
- **Bounds:** ≤ 20 MB response body, ≤ 30 s per fetch.
- **Per-image presence/idempotency check (`claim_image_for_fetch`):** before
processing, each `CatalogImage` row is checked so an already-handled row is
skipped. This is a presence guard adequate for the **single-worker,
in-process** model — the deployed app runs one uvicorn worker, recovery runs
at startup over prior-process runs, and a fresh confirm always targets a new
run, so two phase invocations never process the same run's images
concurrently. It is **not** a cross-process lock. A true multi-worker claim
would need a distinct in-flight status transition (`pending → fetching` with
`RETURNING`, or `SELECT … FOR UPDATE SKIP LOCKED`) — that is the future seam
if the fetch phase ever moves multi-process or onto a queue (§6.9).
- **Startup recovery scan:** on process start the app scans for runs stuck in
`fetching_images` and re-enqueues their pending images (emits TEL-5). This
covers crash/restart scenarios without a separate scheduler. On a mid-fetch
process restart (e.g. a deploy) the phase is interrupted and the run is left
resumable: pending images stay `pending`, fetched ones stay `fetched`, and
the startup scan completes the run. Any pool-closed error in an in-flight
worker at shutdown is benign — the uncommitted image stays `pending` and is
re-fetched on resume. Mid-flight interruption is tolerated by construction
(§6.9).
- **Progress tracking:** the run row carries `image_progress`,
`image_counts`, and `image_outcomes` (updated per image); these fields feed
the run-detail UI panel.
On completion the phase sets the run status to `complete` and emits TEL-4.
### Image-serving route
`GET /api/products/images/{id}/{rendition}` — storefront-authorized,
streams the objectstore blob. The response sets
`Cache-Control: public, immutable, max-age=31536000` (the object key encodes
the image id, so the URL is stable for the lifetime of the image). INV-16:
an image belongs to exactly one storefront; the route enforces storefront
scope before reading from the objectstore.
### Hosted-URL recognition and INV-12 over images
`domains/products/hosted.py` provides a predicate that recognises whether an
image URL is already served by this app (i.e., a `GET /api/products/images/…`
URL at the current `APP_URL`). The diff pre-pass `_resolve_hosted_images`
uses it to convert hosted URLs back to their `CatalogImage` FK before hashing
the diff fingerprint — this closes the INV-12 round-trip guarantee over
image columns (a re-imported export previews as all-unchanged even when
images are hosted here).
## Named seams (what later slices replace)
- **`import_draft.file_bytes` remains BYTEA (deferred past SLICE-7).** Moving
the draft blob to object storage was scoped out of SLICE-7. The media
bucket holds only `product-images/` objects this slice; the `provision-bucket`
script sets the `import-drafts/` lifecycle rule for forward-compat so the
migration will be clean when it lands.
- **`codec.detect_dialect` → Shopify (SLICE-8).** Today it always returns
`"canonical"`; SLICE-8 recognizes Shopify's exact header set here (INV-17).
- **Run-status complete shortcut → shipped in SLICE-7.**
`confirm_draft` now inserts the run as `applying`, then transitions to
`fetching_images` (if the run has pending image URLs) or directly to
`complete`. The fetch phase fills `image_progress` / `image_counts` /
`image_outcomes` and sets `complete` when done.
## Test map
| File | Covers |
| --- | --- |
| `backend/tests/test_products_codec.py` | file-level gates: parse, caps (INV-18), required columns, dialect |
| `backend/tests/test_products_validate.py` | every §6.5.1 row-error rule, one fixture each |
| `backend/tests/test_products_diff.py` | classification, blank-vs-absent, option matching, fingerprint |
| `backend/tests/test_products_serialize.py` | serializer grammar + INV-12 property test (round-trip no-op over generated catalogs) + decimal round-trip |
| `backend/tests/test_products_export.py` | status-filtered snapshot, streamed export, EmptyCatalog, TEL-3 |
| `backend/tests/test_products_service.py` | draft lifecycle: validate/preview/discard, expiry, TEL-1 |
| `backend/tests/test_products_invariants.py` | INV-10 (never deletes), INV-14 (storefront isolation), apply transactionality, TEL-6 |
| `backend/tests/test_products_endpoints.py` | §6.4 API scenarios + auth/storefront gates |
| `e2e/tests/import-preview-confirm.spec.ts` | happy path: upload → preview → confirm → history |
| `e2e/tests/import-errors.spec.ts` | actionable row errors at preview and on the run report |
| `e2e/tests/import-file-rejected.spec.ts` | file-level rejection, picker stays live, no trace |
| `e2e/tests/import-cancel.spec.ts` | cancel at preview leaves no trace |
| `e2e/tests/export-download.spec.ts` | export downloads canonical CSV, status filter respected |
| `e2e/tests/roundtrip-noop.spec.ts` | export → re-import → all-unchanged, import disabled (PUC-10) |
@@ -0,0 +1,118 @@
# ui/designs Content-Repo Collection Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Establish `ui/designs/` as a standard content-repo collection (alongside `specs/` and `plans/`) — concretely in `wiggleverse-ecomm-content`, and centrally in the engineering repo's schema docs so it binds all `*-content` repos.
**Architecture:** Docs/convention change only, two repos, one PR each. The collection convention's canonical home is the engineering repo's `schemas/` docs (the `content` descriptor description strings in `app.schema.json` + `schemas/README.md` changelog), so the central change is a docs-only minor schema bump (1.2 → 1.3). No tooling changes: the spec-linkage gate, backfill verb, and GUIDE/TEMPLATE Design field are tracked separately as `wiggleverse-dev-claude-plugin#93`.
**Tech Stack:** Markdown, JSON Schema (description strings only), git + Gitea PRs over SSH.
**Anchor:** `wiggleverse/wiggleverse-ecomm#8` (type/task, ELIGIBLE R2b). Related: `wiggleverse/wiggleverse-dev-claude-plugin#93`.
---
### Task 1: `ui/designs/` collection in wiggleverse-ecomm-content
**Files:**
- Create: `/Users/benstull/git/wiggleverse.org/wiggleverse/wiggleverse-ecomm-content/ui/designs/README.md`
- Modify: `/Users/benstull/git/wiggleverse.org/wiggleverse/wiggleverse-ecomm-content/README.md` (layout table, lines 1014)
- [ ] **Step 1: Branch**
```bash
git -C /Users/benstull/git/wiggleverse.org/wiggleverse/wiggleverse-ecomm-content checkout -b ui-designs-collection
```
- [ ] **Step 2: Create the collection README**
`ui/designs/README.md`:
```markdown
# ui/designs — UI-design artifacts
Standard content-repo collection (alongside `specs/` and `plans/`) holding this
app's UI-design artifacts — primarily Claude Design outputs generated from a
Solution Design (rubric: `engineering/solution-design/claude-design-vs-code.md`).
A Solution Design with a UX-involving slice references its design artifact here
by path. The spec-linkage gate and the backfill gesture for adding that
reference once a design exists are tracked in
`wiggleverse/wiggleverse-dev-claude-plugin#93`.
Suggested layout: one subfolder per design, named for the spec/slice it serves,
e.g. `ui/designs/SD-0001-slice-3-storefront/`.
```
- [ ] **Step 3: Add the layout-table row**
In the top-level `README.md`, extend the table:
```markdown
| Path | Holds |
| --- | --- |
| `specs/` | reviewed Solution-Design specs (submitted at session finalize) |
| `plans/` | archived implementation plans |
| `ui/designs/` | UI-design artifacts (Claude Design outputs), referenced from specs |
```
- [ ] **Step 4: Commit, push, PR, merge**
```bash
git -C …/wiggleverse-ecomm-content add ui/designs/README.md README.md
git -C …/wiggleverse-ecomm-content commit -m "content: add ui/designs/ collection (ecomm#8)"
git -C …/wiggleverse-ecomm-content push -u origin ui-designs-collection
```
PR via Gitea API (default per-host token, NOT the issue-scoped one — TOKENS.md), then merge; body cites `wiggleverse/wiggleverse-ecomm#8` + plugin `#93`.
### Task 2: Standardize centrally in engineering schemas docs
**Files:**
- Modify: `/Users/benstull/git/wiggleverse.org/wiggleverse/engineering/schemas/app.schema.json` (lines 13, 94, 181, 184)
- Modify: `/Users/benstull/git/wiggleverse.org/wiggleverse/engineering/schemas/README.md` (repos[] bullets + changelog)
- [ ] **Step 1: Branch**
```bash
git -C /Users/benstull/git/wiggleverse.org/wiggleverse/engineering checkout -b ui-designs-collection
```
- [ ] **Step 2: Schema description strings + enum**
1. `schemaVersion.enum`: `["1.0", "1.1", "1.2"]``["1.0", "1.1", "1.2", "1.3"]`
2. `content` property description (line 94): "…where this app's reviewed specs/ and archived plans/ collections live…" → "…where this app's reviewed specs/, archived plans/, and ui/designs/ collections live…"
3. `$defs.content` description (line 181): "(reviewed specs/, archived plans/)" → "(reviewed specs/, archived plans/, ui/designs/ UI-design artifacts)"; and "The specs/ and plans/ collection subdirs are appended by the submit tooling" → "The specs/, plans/, and ui/designs/ collection subdirs are conventions (specs/ and plans/ are appended by the submit tooling; ui/designs/ holds Claude Design outputs referenced from specs)"
4. `$defs.content.subdir` description (line 184): "under which the specs/ and plans/ collections live" → "under which the specs/, plans/, and ui/designs/ collections live"
- [ ] **Step 3: schemas/README.md**
Changelog entry above 1.2:
```markdown
- **1.3** — docs-only: the content-repo collection convention gains a third
standard collection, `ui/designs/` — UI-design artifacts (Claude Design
outputs generated from a Solution Design), referenced from specs. Like
`specs/`/`plans/`, the subdir is a convention, not a schema field; no
validation change (the spec-linkage gate/backfill tooling is
wiggleverse-dev-claude-plugin#93). Existing files stay valid.
```
If the repos[] bullet list documents the `content` descriptor, name the three collections there too; if 1.2 never added a `content` bullet, add one.
- [ ] **Step 4: Validate JSON, commit, push, PR, merge**
```bash
python3 -m json.tool /Users/benstull/git/wiggleverse.org/wiggleverse/engineering/schemas/app.schema.json > /dev/null && echo OK
git -C …/engineering add schemas/app.schema.json schemas/README.md
git -C …/engineering commit -m "schemas: 1.3 — ui/designs/ standard content collection (ecomm#8, plugin#93)"
git -C …/engineering push -u origin ui-designs-collection
```
PR + merge (default token for `/pulls`).
### Task 3: Cross-link and close out
- [ ] **Step 1: Comment on plugin #93** noting the location is now standard (link both merged PRs) — its gate/backfill work can assume `ui/designs/` exists.
- [ ] **Step 2: Close ecomm#8** with a comment naming both merged PRs.
- [ ] **Step 3: Checkpoint the transcript** (`publish-transcript.sh` on the `--INPROGRESS` file).
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,721 @@
# SLICE-8 — Shopify dialect + public format docs — Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Auto-detect a Shopify product-CSV export by its header set, map it to the canonical model at the codec boundary (INV-17), warn (not fail) on Shopify columns with no canonical home, and publish the public column reference (DOC-2) + finalized sample CSV (DOC-3) — completing PUC-6 (a Shopify export imports directly) and PUC-11 (learning the format).
**Architecture:** A Shopify file is recognized by *signature columns* it carries that canonical never does (`Body (HTML)`, `Variant Grams`, `Cost per item`, `Google Shopping / *`, …). Detected Shopify headers are rewritten to canonical names *before* the existing known/unknown split in `parse_csv`, so every downstream stage (validate → diff → apply) is already dialect-agnostic and unchanged (INV-17). Shopify columns with no canonical home — including two name-collision overrides (`Type` is Shopify's *free-text* type vs canonical's *structural* type; `Variant Weight Unit` is superseded by the grams→weight transform) — flow into the existing `unknown_columns` warning, which the preview UI already renders. The only value transform is `Variant Grams` → canonical `Variant Weight` (+ synthesized unit `g`). Frontend is already built for dialect display and the not-imported banner; SLICE-8 only sharpens the label and the upload-screen format help, and adds DOC-2.
**Tech Stack:** Python 3.13 / FastAPI backend (`backend/app/domains/products/`), Vitest + React/TS SPA (`frontend/src/`), Playwright E2E (`e2e/`). Tests: `pytest` (backend), `npm test` (frontend), `scripts/e2e.sh` (browser).
---
## The exhaustive Shopify → canonical mapping (the §6.5.1 fixture)
This table is the contract. It is pinned as a Python literal in Task 1 and as a CSV fixture in Task 3.
**Mapped — renamed** (Shopify header → canonical header):
| Shopify column | Canonical column | Note |
| --- | --- | --- |
| `Body (HTML)` | `Description` | HTML, sanitized downstream (INV-15) |
| `Product Category` | `Google Product Category` | taxonomy string |
| `Cost per item` | `Variant Cost` | decimal |
| `Variant Grams` | `Variant Weight` | value transform: grams numeric; unit `g` synthesized |
**Mapped — direct** (same name, already in `KNOWN_COLUMNS`, pass through): `Handle`, `Title`, `Vendor`, `Tags`, `Published`, `Status`, `Option1 Name`, `Option2 Name`, `Option3 Name`, `Option1 Value`, `Option2 Value`, `Option3 Value`, `Variant SKU`, `Variant Barcode`, `Variant Price`, `Variant Inventory Tracker`, `Variant Inventory Qty`, `Variant Image`, `Image Src`, `Image Position`, `Image Alt Text`.
**Not imported — warned** (no canonical home; surfaced in `unknown_columns`):
- **Name-collision overrides (must NOT pass through):**
- `Type` — Shopify free-text type; canonical `Type` is structural (`standalone`/kits, #15). Per decision D (spec §11): warned, never folded into canonical `Type` or `Tags`.
- `Variant Weight Unit` — superseded by the `Variant Grams` → weight transform (unit forced to `g`).
- **Shopify-only columns:** `Variant Compare At Price` (a.k.a. `Compare At Price`), `Variant Inventory Policy`, `Variant Fulfillment Service`, `Variant Requires Shipping`, `Variant Taxable`, `Variant Tax Code`, `Gift Card`, `SEO Title`, `SEO Description`, every `Google Shopping / *` column, and any market/region column (`Included / *`, `Price / *`, `Compare At Price / *`, `Price / International`, …).
**Algorithm (handles the unbounded market columns without enumerating them):** for each header column, in order —
1. in `SHOPIFY_RENAME` → its canonical name (mapped);
2. in `SHOPIFY_DROP` overrides (`{"Type", "Variant Weight Unit"}`) → not imported;
3. else in `KNOWN_COLUMNS` → pass through (mapped);
4. else → not imported.
**Detection signature** — a header is Shopify iff it contains ≥1 column canonical never has:
`{"Body (HTML)", "Variant Grams", "Cost per item", "Variant Compare At Price", "Variant Inventory Policy", "Variant Fulfillment Service", "Variant Requires Shipping", "Variant Taxable", "Gift Card", "SEO Title", "SEO Description"}` **or** any column starting `"Google Shopping / "`. Otherwise canonical (the safe default — shared columns map identically, so an ambiguous file never misparses; §7.4 risk row).
---
## File structure
- **Create** `backend/app/domains/products/dialect_shopify.py` — detection + header mapping + the pinned table. One responsibility: the Shopify adapter.
- **Modify** `backend/app/domains/products/codec.py``detect_dialect()` calls the adapter; `parse_csv()` rewrites the header before the known/unknown split and synthesizes the weight unit.
- **Create** `backend/tests/test_products_dialect_shopify.py` — adapter unit tests + the exhaustive-mapping fixture test.
- **Modify** `backend/tests/test_products_codec.py` — detection + end-to-end parse-as-shopify tests.
- **Modify** `backend/tests/test_products_invariants.py` — INV-10 over a partial Shopify file.
- **Create** `backend/app/domains/products/columns.md` — DOC-2 source (column reference, every column + dialect notes).
- **Modify** `backend/app/main.py` — add `GET /api/products/columns.md`.
- **Modify** `backend/tests/test_products_api.py` (or the existing API test module) — route test for DOC-2.
- **Modify** `frontend/src/productsApi.ts``dialectLabel("shopify")``"Shopify product CSV — mapped"`.
- **Modify** `frontend/src/screens/products/ImportUpload.tsx` — format help mentions Shopify + links DOC-2.
- **Modify** `frontend/src/screens/products/ProductsPage.tsx` — link DOC-2 beside the sample CSV.
- **Modify** `frontend/src/productsApi.test.ts` (or create) — `dialectLabel` cases.
- **Create** `e2e/fixtures/shopify-export.csv` — an unmodified-shape Shopify export fixture.
- **Create** `e2e/tests/import-shopify-dialect.spec.ts``e2e_import_shopify_dialect`.
- **Modify** `docs/products-domain.md` — DOC-4 dialect-adapter notes.
- **Modify** `backend/app/__init__.py` (or wherever `__version__`/`app.json` version lives) + `CHANGELOG`/version bump to v0.8.0.
---
### Task 1: Shopify dialect adapter module
**Files:**
- Create: `backend/app/domains/products/dialect_shopify.py`
- Test: `backend/tests/test_products_dialect_shopify.py`
- [ ] **Step 1: Write the failing tests**
```python
# backend/tests/test_products_dialect_shopify.py
from app.domains.products.dialect_shopify import is_shopify_header, map_shopify_header
def test_detects_shopify_by_signature_column():
assert is_shopify_header(["Handle", "Title", "Body (HTML)", "Variant Price"]) is True
assert is_shopify_header(["Handle", "Title", "Google Shopping / MPN"]) is True
def test_canonical_header_is_not_shopify():
assert is_shopify_header(["Handle", "Title", "Description", "Variant Cost"]) is False
def test_ambiguous_shared_only_header_defaults_canonical():
# Only columns common to both dialects -> not Shopify (safe default, never misparse).
assert is_shopify_header(["Handle", "Title", "Option1 Name", "Variant Price", "Image Src"]) is False
def test_map_renames_and_passes_through():
mapped, not_imported = map_shopify_header(
["Handle", "Body (HTML)", "Cost per item", "Variant Price"]
)
assert mapped == ["Handle", "Description", "Variant Cost", "Variant Price"]
assert not_imported == []
def test_map_drops_type_and_weight_unit_overrides():
mapped, not_imported = map_shopify_header(
["Handle", "Type", "Variant Grams", "Variant Weight Unit"]
)
assert mapped == ["Handle", None, "Variant Weight", None]
assert not_imported == ["Type", "Variant Weight Unit"]
def test_map_drops_shopify_only_and_market_columns():
mapped, not_imported = map_shopify_header(
["Handle", "Variant Compare At Price", "Gift Card", "Price / International"]
)
assert mapped == ["Handle", None, None, None]
assert not_imported == ["Variant Compare At Price", "Gift Card", "Price / International"]
```
- [ ] **Step 2: Run to verify it fails**
Run: `cd backend && python -m pytest tests/test_products_dialect_shopify.py -v`
Expected: FAIL — `ModuleNotFoundError: app.domains.products.dialect_shopify`
- [ ] **Step 3: Implement the adapter**
```python
# backend/app/domains/products/dialect_shopify.py
"""Shopify product-CSV adapter (INV-17): detect by header signature, map to canonical.
The mapping is the §6.5.1 contract; the exhaustive table is pinned here and
mirrored by tests/test_products_dialect_shopify.py + e2e/fixtures/shopify-export.csv.
"""
from __future__ import annotations
from .models import KNOWN_COLUMNS
# Shopify header -> canonical header (renames only; direct same-name columns pass
# through via KNOWN_COLUMNS). Variant Grams carries a value transform (see codec).
SHOPIFY_RENAME: dict[str, str] = {
"Body (HTML)": "Description",
"Product Category": "Google Product Category",
"Cost per item": "Variant Cost",
"Variant Grams": "Variant Weight",
}
# Canonical-named columns Shopify uses differently — must NOT pass through.
SHOPIFY_DROP: frozenset[str] = frozenset({"Type", "Variant Weight Unit"})
# Columns whose presence proves the file is a Shopify export (canonical never has them).
_SIGNATURE: frozenset[str] = frozenset({
"Body (HTML)", "Variant Grams", "Cost per item", "Variant Compare At Price",
"Variant Inventory Policy", "Variant Fulfillment Service",
"Variant Requires Shipping", "Variant Taxable", "Gift Card",
"SEO Title", "SEO Description",
})
def is_shopify_header(header: list[str]) -> bool:
cols = {h.strip() for h in header}
if cols & _SIGNATURE:
return True
return any(c.startswith("Google Shopping / ") for c in cols)
def map_shopify_header(header: list[str]) -> tuple[list[str | None], list[str]]:
"""Return (mapped, not_imported): mapped[i] is the canonical name for header[i],
or None when that Shopify column has no canonical home; not_imported lists the
original Shopify names (in order) that were dropped — the preview warning."""
mapped: list[str | None] = []
not_imported: list[str] = []
for raw in header:
col = raw.strip()
if col in SHOPIFY_RENAME:
mapped.append(SHOPIFY_RENAME[col])
elif col in SHOPIFY_DROP:
mapped.append(None)
not_imported.append(col)
elif col in KNOWN_COLUMNS:
mapped.append(col)
else:
mapped.append(None)
if col:
not_imported.append(col)
return mapped, not_imported
```
- [ ] **Step 4: Run to verify it passes**
Run: `cd backend && python -m pytest tests/test_products_dialect_shopify.py -v`
Expected: PASS (6 tests)
- [ ] **Step 5: Commit**
```bash
git add backend/app/domains/products/dialect_shopify.py backend/tests/test_products_dialect_shopify.py
git commit -m "feat(products): Shopify dialect adapter — header detection + canonical mapping (SD-0002 §6.5.1, SLICE-8)"
```
---
### Task 2: Wire the adapter into the codec
**Files:**
- Modify: `backend/app/domains/products/codec.py:15-66`
- Test: `backend/tests/test_products_codec.py`
- [ ] **Step 1: Write the failing tests** (append to `test_products_codec.py`)
```python
def test_parse_detects_and_maps_shopify():
data = (
"Handle,Title,Body (HTML),Cost per item,Variant Price,Variant Grams,Type,Gift Card\n"
"mug,Moon Mug,<p>Grey</p>,9.50,18.00,300,Drinkware,false\n"
).encode("utf-8")
parsed = parse_csv(data)
assert parsed.dialect == "shopify"
cells = parsed.rows[0].cells
assert cells["Description"] == "<p>Grey</p>"
assert cells["Variant Cost"] == "9.50"
assert cells["Variant Weight"] == "300"
assert cells["Variant Weight Unit"] == "g" # synthesized
# Type (free-text) and Gift Card warned, never mapped:
assert "Type" in parsed.unknown_columns
assert "Gift Card" in parsed.unknown_columns
assert "product_type" not in str(cells) # Type never reached canonical
def test_parse_canonical_unchanged():
data = b"Handle,Title,Description,Variant Price\nmug,Moon Mug,Grey,18.00\n"
parsed = parse_csv(data)
assert parsed.dialect == "canonical"
assert parsed.rows[0].cells["Description"] == "Grey"
assert parsed.unknown_columns == []
```
- [ ] **Step 2: Run to verify it fails**
Run: `cd backend && python -m pytest tests/test_products_codec.py -k "shopify or canonical_unchanged" -v`
Expected: FAIL — `parsed.dialect == "canonical"` (detection still stubbed) / `KeyError: 'Description'`
- [ ] **Step 3: Rewrite `detect_dialect` + `parse_csv`** in `codec.py`
Replace the `detect_dialect` stub and the header/cell-building block. New `codec.py` (lines from the import down through `parse_csv`):
```python
from .dialect_shopify import is_shopify_header, map_shopify_header
from .models import KNOWN_COLUMNS, MAX_DATA_ROWS, MAX_FILE_BYTES, ParsedFile, Row
_REQUIRED_HEADER_COLUMNS = ("Handle", "Title")
def detect_dialect(header: list[str]) -> str:
"""The INV-17 seam: recognize Shopify's header set, else canonical (§6.5.1)."""
return "shopify" if is_shopify_header(header) else "canonical"
def parse_csv(data: bytes) -> ParsedFile:
if len(data) > MAX_FILE_BYTES:
raise FileRejected("file_too_large", "This file is larger than 10 MB.")
try:
text = data.decode("utf-8-sig")
except UnicodeDecodeError:
raise FileRejected("not_csv", "This file isn't readable as CSV.") from None
reader = csv.reader(io.StringIO(text))
try:
try:
raw_header = next(reader)
except StopIteration:
raise FileRejected("not_csv", "This file isn't readable as CSV.") from None
header = [h.strip() for h in raw_header]
dialect = detect_dialect(header)
# INV-17: normalize the header to canonical names at the boundary. For Shopify,
# mapped[i] is the canonical name (or None when dropped); not_imported is the
# warning list. Canonical files map to themselves, unknowns warned as before.
if dialect == "shopify":
mapped, unknown = map_shopify_header(header)
else:
mapped = [c if c in KNOWN_COLUMNS else None for c in header]
unknown = [c for c in header if c and c not in KNOWN_COLUMNS]
for col in _REQUIRED_HEADER_COLUMNS:
if col not in (mapped):
raise FileRejected(
"missing_required_column",
f"This file is missing the required column '{col}'.",
)
# First occurrence of a duplicated canonical column wins.
col_index: dict[str, int] = {}
for i, name in enumerate(mapped):
if name and name not in col_index:
col_index[name] = i
# De-dup the warning list, order-preserving.
seen: set[str] = set()
unknown = [c for c in unknown if not (c in seen or seen.add(c))]
rows: list[Row] = []
for raw in reader:
if not any(cell.strip() for cell in raw):
continue
if len(rows) >= MAX_DATA_ROWS:
raise FileRejected(
"too_many_rows",
f"This file has more than {MAX_DATA_ROWS:,} rows — split it and import in parts.",
)
cells = {
c: (raw[col_index[c]].strip() if col_index[c] < len(raw) else "")
for c in col_index
}
# Shopify grams carry an implicit unit; canonical needs it explicit (§6.5.1).
if dialect == "shopify" and cells.get("Variant Weight"):
cells["Variant Weight Unit"] = "g"
rows.append(Row(line_number=reader.line_num, cells=cells))
except csv.Error:
raise FileRejected("not_csv", "This file isn't readable as CSV.") from None
return ParsedFile(dialect=dialect, header=header, unknown_columns=unknown, rows=rows)
```
Note: the required-column check now reads `mapped` (canonical names) so a Shopify file whose `Title`/`Handle` are present passes, and a file genuinely missing them still fails honestly. `Handle` and `Title` are direct pass-through names in both dialects, so `mapped` contains them when present.
- [ ] **Step 4: Run to verify it passes**
Run: `cd backend && python -m pytest tests/test_products_codec.py -v`
Expected: PASS (existing canonical tests + 2 new)
- [ ] **Step 5: Run the whole products suite (no regressions)**
Run: `cd backend && python -m pytest tests/ -q`
Expected: PASS — validate/diff/invariants unchanged (they consume canonical cells; INV-17 holds).
- [ ] **Step 6: Commit**
```bash
git add backend/app/domains/products/codec.py backend/tests/test_products_codec.py
git commit -m "feat(products): codec detects + maps Shopify dialect at the boundary (INV-17, SLICE-8)"
```
---
### Task 3: Exhaustive-mapping fixture test (the §6.5.1 pin)
**Files:**
- Create: `backend/tests/fixtures/shopify-export.csv`
- Test: `backend/tests/test_products_dialect_shopify.py` (append)
- [ ] **Step 1: Create a realistic full-width Shopify export fixture**
`backend/tests/fixtures/shopify-export.csv` — one product, two variants, one image-only row, exercising every mapped + several not-imported columns:
```csv
Handle,Title,Body (HTML),Vendor,Product Category,Type,Tags,Published,Status,Option1 Name,Option1 Value,Variant SKU,Variant Grams,Variant Inventory Tracker,Variant Inventory Qty,Variant Inventory Policy,Variant Fulfillment Service,Variant Price,Variant Compare At Price,Variant Requires Shipping,Variant Taxable,Variant Barcode,Image Src,Image Position,Image Alt Text,Gift Card,SEO Title,SEO Description,Google Shopping / MPN,Variant Image,Variant Weight Unit,Cost per item,Status
star-tee,Star Tee,<p>Soft cotton tee.</p>,Wiggle Goods,Apparel & Accessories > Clothing,Shirts,"apparel, tees",TRUE,active,Size,S,WG-TEE-S,180,shopify,12,deny,manual,24.00,30.00,TRUE,TRUE,0001,https://img.example.com/star-tee.jpg,1,Star Tee,FALSE,Star Tee | Wiggle,Soft tee,MPN-1,https://img.example.com/star-s.jpg,g,11.00,active
star-tee,,,,,,,,,,M,WG-TEE-M,180,shopify,18,deny,manual,24.00,30.00,TRUE,TRUE,0002,,,,FALSE,,,,,g,11.00,
star-tee,,,,,,,,,,,,,,,,,,,,,,https://img.example.com/star-back.jpg,2,Star Tee back,,,,,,,,
```
- [ ] **Step 2: Write the failing test**
```python
import csv as _csv
import io
from pathlib import Path
from app.domains.products.codec import parse_csv
_FIXTURE = Path(__file__).parent / "fixtures" / "shopify-export.csv"
def test_shopify_fixture_maps_exhaustively():
parsed = parse_csv(_FIXTURE.read_bytes())
assert parsed.dialect == "shopify"
first = parsed.rows[0].cells
# renamed
assert first["Description"] == "<p>Soft cotton tee.</p>"
assert first["Google Product Category"] == "Apparel & Accessories > Clothing"
assert first["Variant Cost"] == "11.00"
assert first["Variant Weight"] == "180"
assert first["Variant Weight Unit"] == "g"
# direct
assert first["Variant SKU"] == "WG-TEE-S"
assert first["Image Src"] == "https://img.example.com/star-tee.jpg"
# not imported — warned, none leaked into canonical cells
for col in ("Type", "Variant Compare At Price", "Gift Card", "SEO Title",
"Google Shopping / MPN", "Variant Weight Unit", "Variant Inventory Policy",
"Variant Fulfillment Service", "Variant Requires Shipping", "Variant Taxable"):
assert col in parsed.unknown_columns, col
assert "Variant Compare At Price" not in first
```
- [ ] **Step 3: Run — expect PASS** (codec from Task 2 already maps correctly)
Run: `cd backend && python -m pytest tests/test_products_dialect_shopify.py::test_shopify_fixture_maps_exhaustively -v`
Expected: PASS. If a column assertion fails, fix the mapping table in `dialect_shopify.py` to match this fixture — the fixture and table must agree.
- [ ] **Step 4: Commit**
```bash
git add backend/tests/fixtures/shopify-export.csv backend/tests/test_products_dialect_shopify.py
git commit -m "test(products): exhaustive Shopify->canonical mapping fixture (SD-0002 §6.5.1, SLICE-8)"
```
---
### Task 4: INV-10 over a partial Shopify import (never deletes)
**Files:**
- Test: `backend/tests/test_products_invariants.py` (append)
- [ ] **Step 1: Inspect the existing INV-10 test** to reuse its catalog/diff helpers
Run: `cd backend && grep -n "def test_inv10\|compute_diff\|build_products\|catalog" tests/test_products_invariants.py | head`
Expected: shows the helper pattern (catalog fixture → parse → build_products → compute_diff → assert no deletes in plan).
- [ ] **Step 2: Write the failing test** (mirror the existing INV-10 test, swapping in a Shopify file that omits an existing product)
```python
def test_inv10_shopify_partial_never_deletes():
# An existing catalog of two products; a Shopify file mentioning only one.
catalog = _catalog_with(["star-tee", "moon-mug"]) # existing helper
data = (
"Handle,Title,Body (HTML),Variant Price,Variant Grams\n"
"star-tee,Star Tee,<p>x</p>,24.00,180\n"
).encode("utf-8")
parsed = parse_csv(data)
assert parsed.dialect == "shopify"
products = build_products(parsed)
result = compute_diff(catalog, products)
# INV-10: nothing in the plan deletes; moon-mug (unmentioned) is untouched.
assert all(op.kind != "delete" for op in result.plan)
assert "moon-mug" not in {p.handle for p in products}
```
Adapt `_catalog_with` / `op.kind` / `result.plan` to the actual symbols found in Step 1.
- [ ] **Step 3: Run to verify it passes**
Run: `cd backend && python -m pytest tests/test_products_invariants.py -k inv10_shopify -v`
Expected: PASS (INV-10 is dialect-agnostic; this proves it over Shopify).
- [ ] **Step 4: Commit**
```bash
git add backend/tests/test_products_invariants.py
git commit -m "test(products): INV-10 holds over a partial Shopify import (SLICE-8)"
```
---
### Task 5: Frontend — Shopify label + format help
**Files:**
- Modify: `frontend/src/productsApi.ts:52-54`
- Modify: `frontend/src/screens/products/ImportUpload.tsx:53`
- Modify: `frontend/src/screens/products/ProductsPage.tsx:114`
- Test: `frontend/src/productsApi.test.ts`
- [ ] **Step 1: Write the failing test** (create or append `frontend/src/productsApi.test.ts`)
```ts
import { describe, expect, it } from "vitest";
import { dialectLabel } from "./productsApi";
describe("dialectLabel", () => {
it("labels canonical", () => expect(dialectLabel("canonical")).toBe("Canonical format"));
it("labels shopify as mapped", () =>
expect(dialectLabel("shopify")).toBe("Shopify product CSV — mapped"));
});
```
- [ ] **Step 2: Run to verify it fails**
Run: `cd frontend && npm test -- productsApi`
Expected: FAIL — receives `"shopify"`, expected `"Shopify product CSV — mapped"`.
- [ ] **Step 3: Update `dialectLabel`**
```ts
// frontend/src/productsApi.ts
export function dialectLabel(d: string): string {
if (d === "canonical") return "Canonical format";
if (d === "shopify") return "Shopify product CSV — mapped";
return d;
}
```
- [ ] **Step 4: Run to verify it passes**
Run: `cd frontend && npm test -- productsApi`
Expected: PASS
- [ ] **Step 5: Update the upload-screen format help**`ImportUpload.tsx:53`
Replace the line `Works with the canonical format.{" "}` and the following sample link block so it reads (PUC-11, §5.4 wireframe text):
```tsx
Works with the canonical format or a Shopify product CSV.{" "}
<a href="/api/products/sample.csv" download>
Download a sample
</a>{" "}
·{" "}
<a href="/api/products/columns.md" target="_blank" rel="noopener">
Column reference
</a>
```
- [ ] **Step 6: Add the column-reference link beside the sample on the Products page**`ProductsPage.tsx:114`
After the existing `<a href="/api/products/sample.csv" download>` element, add:
```tsx
{" · "}
<a href="/api/products/columns.md" target="_blank" rel="noopener">
Column reference
</a>
```
- [ ] **Step 7: Build + typecheck + test**
Run: `cd frontend && npm run build && npm test`
Expected: PASS (no TS errors, vitest green).
- [ ] **Step 8: Commit**
```bash
git add frontend/src/productsApi.ts frontend/src/productsApi.test.ts frontend/src/screens/products/ImportUpload.tsx frontend/src/screens/products/ProductsPage.tsx
git commit -m "feat(products-ui): Shopify 'mapped' label + format help links to column reference (PUC-6/PUC-11, SLICE-8)"
```
---
### Task 6: DOC-2 column reference + route; finalize DOC-3
**Files:**
- Create: `backend/app/domains/products/columns.md`
- Modify: `backend/app/main.py` (after the `sample.csv` route, ~line 451)
- Modify: `backend/app/domains/products/__init__.py` (export `COLUMNS_MD_PATH` next to `SAMPLE_CSV_PATH`)
- Test: the existing products API test module (find it in Step 4)
- [ ] **Step 1: Write the DOC-2 column reference**`backend/app/domains/products/columns.md`
A complete merchant-facing reference: a short intro (canonical = Shopify-flavored superset; auto-detection; blank-vs-absent semantics), then a table of **every canonical column** (name · level · required · accepted values), then a **Shopify dialect notes** section reproducing the Task-1 mapping table (renamed, direct, not-imported). Source the canonical column rows from spec §6.5.1 (the table at lines 852-873) and the dialect notes from §6.5.1 (lines 880-889). Keep it plain Markdown — no app-specific build step.
- [ ] **Step 2: Export the path**`backend/app/domains/products/__init__.py`
Add beside the existing `SAMPLE_CSV_PATH`:
```python
COLUMNS_MD_PATH = _HERE / "columns.md"
```
(Use the same `_HERE` / `Path(__file__).parent` idiom already present for `SAMPLE_CSV_PATH`.)
- [ ] **Step 3: Add the route**`backend/app/main.py`, immediately after `products_sample_csv`
```python
@app.get("/api/products/columns.md")
def products_columns_md():
"""The DOC-2 column reference. Documentation, so no auth gate (§6.4)."""
return PlainTextResponse(
products.COLUMNS_MD_PATH.read_text(),
media_type="text/markdown; charset=utf-8",
)
```
- [ ] **Step 4: Write the failing route test** — find the API test module first
Run: `cd backend && ls tests/ | grep -i "api\|main"`
Append to the module that tests the other unauthenticated doc routes:
```python
def test_columns_md_served(client):
resp = client.get("/api/products/columns.md")
assert resp.status_code == 200
assert "text/markdown" in resp.headers["content-type"]
body = resp.text
assert "Body (HTML)" in body # dialect notes present
assert "Handle" in body # canonical columns present
```
- [ ] **Step 5: Run to verify it passes**
Run: `cd backend && python -m pytest tests/ -k "columns_md or sample" -v`
Expected: PASS
- [ ] **Step 6: Verify DOC-3 sample.csv is final** — it already carries a variant product (`star-tee`, S/M/L) and image rows. Confirm it parses clean as canonical:
Run: `cd backend && python -c "from app.domains.products.codec import parse_csv; from app.domains.products import SAMPLE_CSV_PATH; p=parse_csv(SAMPLE_CSV_PATH.read_bytes()); print(p.dialect, len(p.rows), p.unknown_columns)"`
Expected: `canonical 5 []` (no unknown columns; 5 data rows). If `unknown_columns` is non-empty, fix the sample header to canonical names.
- [ ] **Step 7: Commit**
```bash
git add backend/app/domains/products/columns.md backend/app/domains/products/__init__.py backend/app/main.py backend/tests/
git commit -m "docs(products): DOC-2 column reference served at /api/products/columns.md (SLICE-8)"
```
---
### Task 7: DOC-4 dialect-adapter dev notes
**Files:**
- Modify: `docs/products-domain.md`
- [ ] **Step 1: Add a "Dialects (INV-17)" subsection** documenting: detection by signature columns; the `dialect_shopify` adapter; the rename/drop/passthrough algorithm; the grams→weight transform; that all downstream stages are dialect-agnostic. Point to `dialect_shopify.py` and the fixture as the source of truth (documentation-leads-automation, §4.1).
- [ ] **Step 2: Commit**
```bash
git add docs/products-domain.md
git commit -m "docs(products): DOC-4 dialect-adapter notes (SLICE-8)"
```
---
### Task 8: E2E — `e2e_import_shopify_dialect`
**Files:**
- Create: `e2e/fixtures/shopify-export.csv` (copy the Task-3 fixture)
- Create: `e2e/tests/import-shopify-dialect.spec.ts`
- [ ] **Step 1: Add the fixture** — same content as `backend/tests/fixtures/shopify-export.csv`.
- [ ] **Step 2: Read an existing import E2E spec** to reuse the sign-up + upload helpers
Run: `sed -n '1,60p' e2e/tests/import-preview-confirm.spec.ts`
Expected: shows the page-object / sign-up helper and the upload→preview→confirm flow to mirror.
- [ ] **Step 3: Write `e2e/tests/import-shopify-dialect.spec.ts`** (mirroring that helper)
```ts
import { test, expect } from "@playwright/test";
import { signUpAndOpenImport } from "./helpers"; // use the actual helper from Step 2
test("e2e_import_shopify_dialect: a Shopify export imports directly (PUC-6)", async ({ page }) => {
await signUpAndOpenImport(page);
await page.getByLabel(/choose|upload|file/i).setInputFiles("fixtures/shopify-export.csv");
// Preview shows the mapped banner + not-imported warning
await expect(page.getByText("Shopify product CSV — mapped")).toBeVisible();
await expect(page.getByText(/not imported/i)).toBeVisible();
await expect(page.getByText("Type")).toBeVisible(); // a warned column
// Confirm and land on the run detail
await page.getByRole("button", { name: /^Import \d/ }).click();
await expect(page).toHaveURL(/\/products\/imports\/runs\/\d+/);
await expect(page.getByText("Shopify product CSV — mapped")).toBeVisible();
});
```
Adapt selectors to the real upload control + helper names found in Step 2.
- [ ] **Step 4: Run the E2E suite**
Run: `./scripts/e2e.sh`
Expected: all specs PASS including `import-shopify-dialect` (7 prior + 1 new = 8).
- [ ] **Step 5: Commit**
```bash
git add e2e/fixtures/shopify-export.csv e2e/tests/import-shopify-dialect.spec.ts
git commit -m "test(e2e): e2e_import_shopify_dialect — Shopify export imports directly (PUC-6, SLICE-8)"
```
---
### Task 9: Traceability audit, version bump, full green
**Files:**
- Modify: version source (`backend/app/__init__.py` or `app.json`) + any `CHANGELOG`
- Read-only: spec §12 traceability matrix
- [ ] **Step 1: Audit §12 traceability** — confirm PUC-6, PUC-11, BUC-1 (Shopify), DOC-2, DOC-3 now have shipping code + tests. Note any gap in the transcript `## Deferred decisions`. (Spec audit is read-only; do not edit the content repo here — finalize submits the plan.)
- [ ] **Step 2: Bump the version to v0.8.0**
Run: `cd backend && grep -rn "0.7.0" app/__init__.py app.json 2>/dev/null`
Update the version string to `0.8.0` wherever the prior slice set it (mirror the SLICE-7 v0.7.0 bump).
- [ ] **Step 3: Full local gate**
Run: `./scripts/check.sh && ./scripts/e2e.sh` (or the repo's canonical localhost gate)
Expected: backend pytest, frontend build+vitest, import-linter, and Playwright all PASS.
- [ ] **Step 4: Commit**
```bash
git add -A
git commit -m "chore(products): SLICE-8 complete — bump v0.8.0; §12 traceability audited (SD-0002)"
```
---
## Post-plan (session flow, not subagent tasks)
After all tasks are green on the branch:
1. **PR → merge to `main`** (branch→PR→merge; §5.4).
2. **PPE deploy** via flotilla + E2E green on `https://ecomm-ppe.wiggleverse.org`; stamp the `release/<ts>` tag (§9.1). Prod promotion stays the operator's gate.
3. **Close story #13** (SLICE-5/6/7/8 all done) with a pointer to the merge + PPE release tag.
4. **Finalize** the session (`wgl-session-finalize`) — archives this plan to the content repo `plans/`, updates memory, publishes the transcript.
---
## Self-Review
**Spec coverage (SLICE-8 §7.2 DoD):**
- Header-set dialect detection → Task 1 (`is_shopify_header`) + Task 2 (`detect_dialect`). ✓
- Shopify→canonical mapping + exhaustive fixture → Task 1 (table) + Task 3 (fixture pin). ✓
- Preview "mapped" banner + not-imported warnings → Task 5 (label) + already-built `unknown_columns` banner (`ImportPreview.tsx:288`). ✓
- DOC-2 column reference (published) → Task 6. ✓
- DOC-3 final sample CSV → Task 6 Step 6 (verify; already has variants+images). ✓
- `e2e_import_shopify_dialect` green → Task 8. ✓
- BUC-1 acceptance for an unmodified Shopify export → Task 3 fixture + Task 8 E2E. ✓
- §12 traceability audited → Task 9. ✓
- INV-10 over Shopify → Task 4. ✓
**Placeholder scan:** every code step shows concrete code; selectors/helpers in Tasks 4/6/8 are explicitly flagged to adapt to symbols discovered in a named inspection step (not silent TBDs).
**Type consistency:** `is_shopify_header` / `map_shopify_header` signatures match between Task 1 (def), Task 2 (import + call), and Task 3 (test). `dialectLabel("shopify")` string is identical in Task 5 def, Task 5 test, and Task 8 E2E assertion (`"Shopify product CSV — mapped"`). `COLUMNS_MD_PATH` defined (Task 6 Step 2) before use (Task 6 Step 3).
**Deferred decisions (logged to transcript):**
- `Variant Grams``Variant Weight` value with unit forced to `g`; Shopify's own `Variant Weight Unit` display column is *not imported* (warned). Lossless in mass (grams is exact); chosen over carrying a possibly-inconsistent unit. Revisit if merchants need kg/lb display fidelity.
- DOC-2 served as raw Markdown (`text/markdown`), mirroring how DOC-3 `sample.csv` is served — not a styled HTML page. Satisfies "app-served, source in framework repo"; styled rendering is a future enhancement.
- Ambiguous header (only shared columns) defaults to **canonical** rather than a distinct `unknown_dialect` state — shared columns map identically so it never misparses (§7.4 intent); avoids a third dialect state the model doesn't carry.
+5
View File
@@ -0,0 +1,5 @@
node_modules/
.backend.log
.objectstore/
test-results/
playwright-report/
+4
View File
@@ -0,0 +1,4 @@
Handle,Title,Vendor,Tags,Status,Published,Option1 Name,Option1 Value,Variant SKU,Variant Price,Variant Inventory Qty
moon-mug,Moon Mug,Wiggle Goods,"kitchen, mugs",active,TRUE,,,WG-MUG-001,18.00,40
star-tee,Star Tee,Wiggle Goods,apparel,active,TRUE,Size,S,WG-TEE-S,24.00,12
star-tee,,,,,,,M,WG-TEE-M,24.00,18
1 Handle Title Vendor Tags Status Published Option1 Name Option1 Value Variant SKU Variant Price Variant Inventory Qty
2 moon-mug Moon Mug Wiggle Goods kitchen, mugs active TRUE WG-MUG-001 18.00 40
3 star-tee Star Tee Wiggle Goods apparel active TRUE Size S WG-TEE-S 24.00 12
4 star-tee M WG-TEE-M 24.00 18
Binary file not shown.

After

Width:  |  Height:  |  Size: 3.6 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 184 B

+2
View File
@@ -0,0 +1,2 @@
Handle,Vendor
mug,Acme
1 Handle Vendor
2 mug Acme
+4
View File
@@ -0,0 +1,4 @@
Handle,Title,Variant Price
good-mug,Good Mug,10.00
bad-tee,Bad Tee,not-a-price
also-good,Also Good,5.00
1 Handle Title Variant Price
2 good-mug Good Mug 10.00
3 bad-tee Bad Tee not-a-price
4 also-good Also Good 5.00
+2
View File
@@ -0,0 +1,2 @@
Handle,Title,Body (HTML),Vendor,Type,Tags,Published,Status,Variant SKU,Variant Grams,Variant Inventory Qty,Variant Price,Variant Compare At Price,Gift Card,SEO Title,Cost per item
shopify-mug,Shopify Mug,<p>Imported from Shopify.</p>,Acme,Drinkware,"mugs",TRUE,active,SM-001,300,25,14.00,20.00,FALSE,Shopify Mug | Acme,7.00
1 Handle Title Body (HTML) Vendor Type Tags Published Status Variant SKU Variant Grams Variant Inventory Qty Variant Price Variant Compare At Price Gift Card SEO Title Cost per item
2 shopify-mug Shopify Mug <p>Imported from Shopify.</p> Acme Drinkware mugs TRUE active SM-001 300 25 14.00 20.00 FALSE Shopify Mug | Acme 7.00
+4
View File
@@ -0,0 +1,4 @@
Handle,Title,Image Src
good-lamp,Good Lamp,http://127.0.0.1:8799/good.png
tiny-lamp,Tiny Lamp,http://127.0.0.1:8799/tiny.png
gone-lamp,Gone Lamp,http://127.0.0.1:8799/missing.png
1 Handle Title Image Src
2 good-lamp Good Lamp http://127.0.0.1:8799/good.png
3 tiny-lamp Tiny Lamp http://127.0.0.1:8799/tiny.png
4 gone-lamp Gone Lamp http://127.0.0.1:8799/missing.png
+120
View File
@@ -0,0 +1,120 @@
// Shared E2E helpers — the sign-up journey (SD-0001 §5.1–§5.4) and products-page
// navigation (SD-0002 §5.2). Selectors are role/label-based against the real screens
// (Landing.tsx, SignIn.tsx, CreateStorefront.tsx, Admin.tsx, ProductsPage.tsx).
import { expect, type Page } from "@playwright/test";
import { readFile, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
const LOG = join(__dirname, ".backend.log");
let seq = 0;
// The storefront name every helper-created merchant uses; tests assert against it.
export const STOREFRONT_NAME = "E2E Test Goods";
export function freshEmail(): string {
return `merchant${Date.now()}-${seq++}@example.com`;
}
// LogMailer's exact line (backend/app/platform/mailer.py):
// INFO:ecomm.mailer: LogMailer -> <to> | Your ecomm code: <6 digits>\n<body>
// The first 6-digit group after the marker is the code.
async function codeFor(email: string): Promise<string> {
for (let i = 0; i < 50; i++) {
const log = await readFile(LOG, "utf8").catch(() => "");
const at = log.lastIndexOf(`LogMailer -> ${email}`);
if (at >= 0) {
const m = log.slice(at).match(/\b(\d{6})\b/);
if (m) return m[1];
}
await new Promise((r) => setTimeout(r, 200));
}
throw new Error(`no code logged for ${email}`);
}
export async function signUpWithStorefront(page: Page, email = freshEmail()): Promise<string> {
// Landing → the sign-up door.
await page.goto("/");
await page.getByRole("button", { name: "Create your storefront →" }).click();
// Sign-in step 1: request the one-time code.
await page.getByLabel("Email").fill(email);
await page.getByRole("button", { name: "Send code" }).click();
// Sign-in step 2: read the code from the backend log and verify it.
await expect(page.getByRole("heading", { name: "Check your email" })).toBeVisible();
const code = await codeFor(email);
await page.getByLabel("One-time code").fill(code);
await page.getByRole("button", { name: "Continue" }).click();
// Create-storefront screen (a new account has none yet).
await expect(page.getByRole("heading", { name: "Create your storefront" })).toBeVisible();
await page.getByLabel("Storefront name").fill(STOREFRONT_NAME);
await page.getByRole("button", { name: "Create storefront", exact: true }).click();
// Admin shell: nav strip + the storefront identity in the topbar.
await expect(page.getByRole("navigation", { name: "Admin sections" })).toBeVisible();
await expect(page.locator(".storeid__name")).toHaveText(STOREFRONT_NAME);
return email;
}
export async function gotoProducts(page: Page) {
await page
.getByRole("navigation", { name: "Admin sections" })
.getByRole("link", { name: "Products" })
.click();
await expect(page.getByRole("heading", { level: 1, name: "Products" })).toBeVisible();
}
export async function uploadFixture(page: Page, fixture: string) {
// Two "Import products" links render on the empty products page (header + empty
// state) — same destination, so take the first.
await page.getByRole("link", { name: "Import products" }).first().click();
await expect(page.getByRole("heading", { name: "Import products" })).toBeVisible();
await page.locator('input[type="file"]').setInputFiles(join(__dirname, "fixtures", fixture));
}
// Import good.csv and confirm it, leaving a 2-product catalog. Returns nothing;
// callers continue from the run-detail screen.
export async function importGoodCsv(page: Page) {
await uploadFixture(page, "good.csv");
await expect(page.getByRole("heading", { name: "Import preview — good.csv" })).toBeVisible();
await page.getByRole("button", { name: "Import 2 products" }).click();
await expect(page.getByRole("heading", { level: 1, name: "good.csv" })).toBeVisible();
}
// Import with-images.csv (3 products, each with one Image Src) and confirm it.
// Mirrors importGoodCsv; callers continue from the run-detail screen, where the
// background image fetch progresses fetching_images → terminal.
export async function importWithImages(page: Page) {
await uploadFixture(page, "with-images.csv");
await expect(page.getByRole("heading", { name: "Import preview — with-images.csv" })).toBeVisible();
await page.getByRole("button", { name: "Import 3 products" }).click();
await expect(page.getByRole("heading", { level: 1, name: "with-images.csv" })).toBeVisible();
}
// Click an Export status option and capture the downloaded CSV's text + path.
export async function exportCatalog(page: Page, label: string): Promise<{ text: string; path: string }> {
// Open the <details> menu, then click the status option, capturing the download.
// The menu is a native <details> that *toggles* on each summary click and is
// NOT closed by clicking a download <a> (a download doesn't navigate). So on a
// second export the menu may already be open — deterministically open it
// rather than blind-toggling, and wait for the status link to be visible.
const details = page.locator("details.products__export");
if (!(await details.evaluate((el: HTMLDetailsElement) => el.open))) {
await details.locator("> summary").click();
}
const link = page.getByRole("link", { name: label, exact: true });
await expect(link).toBeVisible();
const [download] = await Promise.all([
page.waitForEvent("download"),
link.click(),
]);
const stream = await download.createReadStream();
const chunks: Buffer[] = [];
for await (const c of stream) chunks.push(c as Buffer);
const text = Buffer.concat(chunks).toString("utf8");
const path = join(tmpdir(), `export-${Date.now()}.csv`);
await writeFile(path, text, "utf8");
return { text, path };
}
+19
View File
@@ -0,0 +1,19 @@
import { createServer } from "node:http";
import { readFile } from "node:fs/promises";
import { fileURLToPath } from "node:url";
import { dirname, join } from "node:path";
const dir = join(dirname(fileURLToPath(import.meta.url)), "fixtures", "images");
createServer(async (req, res) => {
const name = (req.url || "").replace(/^\//, "").split("?")[0];
if (name === "good.png" || name === "tiny.png") {
try {
const buf = await readFile(join(dir, name));
res.writeHead(200, { "Content-Type": "image/png" });
res.end(buf);
return;
} catch { /* fallthrough */ }
}
res.writeHead(404);
res.end();
}).listen(8799, "127.0.0.1");
+76
View File
@@ -0,0 +1,76 @@
{
"name": "wiggleverse-ecomm-e2e",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "wiggleverse-ecomm-e2e",
"devDependencies": {
"@playwright/test": "^1.48.0"
}
},
"node_modules/@playwright/test": {
"version": "1.60.0",
"resolved": "https://registry.npmjs.org/@playwright/test/-/test-1.60.0.tgz",
"integrity": "sha512-O71yZIbAh/PxDMNGns37GHBIfrVkEVyn+AXyIa5dOTfb4/xNvRWV+Vv/NMbNCtODB/pO7vLlF2OTmMVLhmr7Ag==",
"dev": true,
"license": "Apache-2.0",
"dependencies": {
"playwright": "1.60.0"
},
"bin": {
"playwright": "cli.js"
},
"engines": {
"node": ">=18"
}
},
"node_modules/fsevents": {
"version": "2.3.2",
"resolved": "https://registry.npmjs.org/fsevents/-/fsevents-2.3.2.tgz",
"integrity": "sha512-xiqMQR4xAeHTuB9uWm+fFRcIOgKBMiOBP+eXiyT7jsgVCq1bkVygt00oASowB7EdtpOHaaPgKt812P9ab+DDKA==",
"dev": true,
"hasInstallScript": true,
"license": "MIT",
"optional": true,
"os": [
"darwin"
],
"engines": {
"node": "^8.16.0 || ^10.6.0 || >=11.0.0"
}
},
"node_modules/playwright": {
"version": "1.60.0",
"resolved": "https://registry.npmjs.org/playwright/-/playwright-1.60.0.tgz",
"integrity": "sha512-hheHdokM8cdqCb0lcE3s+zT4t4W+vvjpGxsZlDnikarzx8tSzMebh3UiFtgqwFwnTnjYQcsyMF8ei2mCO/tpeA==",
"dev": true,
"license": "Apache-2.0",
"dependencies": {
"playwright-core": "1.60.0"
},
"bin": {
"playwright": "cli.js"
},
"engines": {
"node": ">=18"
},
"optionalDependencies": {
"fsevents": "2.3.2"
}
},
"node_modules/playwright-core": {
"version": "1.60.0",
"resolved": "https://registry.npmjs.org/playwright-core/-/playwright-core-1.60.0.tgz",
"integrity": "sha512-9bW6zvX/m0lEbgTKJ6YppOKx8H3VOPBMOCFh2irXFOT4BbHgrx5hPjwJYLT40Lu+4qtD36qKc/Hn56StUW57IA==",
"dev": true,
"license": "Apache-2.0",
"bin": {
"playwright-core": "cli.js"
},
"engines": {
"node": ">=18"
}
}
}
}
+6
View File
@@ -0,0 +1,6 @@
{
"name": "wiggleverse-ecomm-e2e",
"private": true,
"scripts": { "test": "playwright test" },
"devDependencies": { "@playwright/test": "^1.48.0" }
}
+16
View File
@@ -0,0 +1,16 @@
import { defineConfig } from "@playwright/test";
export default defineConfig({
testDir: "./tests",
timeout: 60_000,
retries: 0,
// One shared backend + log file; parallel sign-ins would interleave codes.
workers: 1,
use: { baseURL: "http://localhost:8765" },
webServer: {
command: "bash ./serve.sh",
url: "http://localhost:8765/healthz",
reuseExistingServer: false,
timeout: 120_000,
},
});
Executable
+34
View File
@@ -0,0 +1,34 @@
#!/usr/bin/env bash
# E2E server: fresh ecomm_e2e database, LogMailer (codes land in .backend.log),
# backend on :8765 serving the built SPA (the deployed topology, SD-0001 §6.2).
set -euo pipefail
repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
if [ ! -f "$repo_root/frontend/dist/index.html" ]; then
( cd "$repo_root/frontend" && npm run build )
fi
"$repo_root/.venv/bin/python" - <<'PY'
import psycopg
admin = psycopg.connect("postgresql://ecomm:ecomm@localhost:5432/postgres", autocommit=True)
admin.execute("SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE datname='ecomm_e2e' AND pid <> pg_backend_pid()")
admin.execute("DROP DATABASE IF EXISTS ecomm_e2e")
admin.execute("CREATE DATABASE ecomm_e2e")
PY
export ECOMM_DATABASE_URL="postgresql://ecomm:ecomm@localhost:5432/ecomm_e2e"
export ECOMM_MAILER=log
# Image pipeline (SLICE-7): local objectstore + a fixture image host on :8799.
# ECOMM_IMAGE_FETCH_ALLOW_PRIVATE lets the SSRF guard reach 127.0.0.1 — TEST-ONLY,
# never set in the deployed overlay. The &-backgrounded node host is a child of
# this serve process; Playwright tears the whole webServer group down between runs.
export ECOMM_OBJECTSTORE_KIND=local
export ECOMM_OBJECTSTORE_DIR="$repo_root/e2e/.objectstore"
export ECOMM_IMAGE_FETCH_ALLOW_PRIVATE=1
rm -rf "$repo_root/e2e/.objectstore"
node "$repo_root/e2e/image-host.mjs" &
cd "$repo_root/backend"
exec "$repo_root/.venv/bin/python" -m uvicorn app.main:app --port 8765 \
> "$repo_root/e2e/.backend.log" 2>&1
+21
View File
@@ -0,0 +1,21 @@
// DoD scenario e2e_export_download (SD-0002 §6.8): export downloads a canonical
// CSV and the status filter is respected.
import { expect, test } from "@playwright/test";
import { exportCatalog, gotoProducts, importGoodCsv, signUpWithStorefront } from "../helpers";
test("e2e_export_download", async ({ page }) => {
await signUpWithStorefront(page);
await gotoProducts(page);
await importGoodCsv(page);
await gotoProducts(page);
// Export "All products" → a canonical CSV with both handles.
const all = await exportCatalog(page, "All products");
expect(all.text.split("\n")[0]).toContain("Handle,");
expect(all.text).toContain("moon-mug");
expect(all.text).toContain("star-tee");
// good.csv's products are active → "Active" exports both, "Draft" is empty/disabled-path.
const active = await exportCatalog(page, "Active");
expect(active.text).toContain("moon-mug");
});
+16
View File
@@ -0,0 +1,16 @@
import { expect, test } from "@playwright/test";
import { gotoProducts, importWithImages, signUpWithStorefront } from "../helpers";
test("e2e_image_outcomes: per-image outcomes, problem rows, notice band", async ({ page }) => {
await signUpWithStorefront(page);
await gotoProducts(page);
await importWithImages(page);
// Run detail polls fetching_images → terminal; wait for the outcome summary.
await expect(page.getByText("1 fetched · 1 rejected · 1 failed")).toBeVisible({ timeout: 30_000 });
// Problem images listed in the outcomes table.
await expect(page.getByText("tiny-lamp")).toBeVisible();
await expect(page.getByText("gone-lamp")).toBeVisible();
// Products page surfaces the aggregate notice.
await gotoProducts(page);
await expect(page.getByText(/products have image problems/)).toBeVisible();
});
+23
View File
@@ -0,0 +1,23 @@
// DoD scenario e2e_import_cancel_no_trace (SD-0002 §6.8): cancelling at the
// preview gate (PUC-3a) leaves no trace — no products, no run in the history.
import { expect, test } from "@playwright/test";
import { gotoProducts, signUpWithStorefront, uploadFixture } from "../helpers";
test("e2e_import_cancel_no_trace", async ({ page }) => {
await signUpWithStorefront(page);
await gotoProducts(page);
await uploadFixture(page, "good.csv");
// The preview gate is up.
await expect(page.getByRole("heading", { name: "Import preview — good.csv" })).toBeVisible();
await expect(page.getByRole("button", { name: "2 to add" })).toBeVisible();
await page.getByRole("button", { name: "Cancel" }).click();
// Lands back on Products — still the empty state, no run recorded.
await expect(page.getByRole("heading", { level: 1, name: "Products" })).toBeVisible();
await expect(
page.getByText("No products yet. Bulk import is how product data gets in."),
).toBeVisible();
await expect(page.getByText("No imports yet.")).toBeVisible();
});
+41
View File
@@ -0,0 +1,41 @@
// DoD scenario e2e_import_errors_actionable (SD-0002 §6.8): row-level errors are
// actionable — line, column, message — at the preview gate AND on the run report
// card (PUC-5), while the good rows still import.
import { expect, test } from "@playwright/test";
import { gotoProducts, signUpWithStorefront, uploadFixture } from "../helpers";
test("e2e_import_errors_actionable", async ({ page }) => {
await signUpWithStorefront(page);
await gotoProducts(page);
await uploadFixture(page, "mixed-errors.csv");
// Preview tiles: two good products, one errored row.
await expect(
page.getByRole("heading", { name: "Import preview — mixed-errors.csv" }),
).toBeVisible();
await expect(page.getByRole("button", { name: "2 to add" })).toBeVisible();
await page.getByRole("button", { name: "1 errors" }).click();
// The error table names the line, the column, and the problem (actionable, PUC-5).
const previewRow = page.locator(".errortable tbody tr");
await expect(previewRow).toHaveCount(1);
await expect(previewRow.locator("td").nth(0)).toHaveText("3");
await expect(previewRow.locator("td").nth(1)).toHaveText("Variant Price");
await expect(previewRow.locator("td").nth(2)).toContainText("is not a price");
// Good rows still import.
await page.getByRole("button", { name: "Import 2 products" }).click();
// Run detail: counts include the errored row, and the same error row persists.
await expect(page.getByRole("heading", { level: 1, name: "mixed-errors.csv" })).toBeVisible();
await expect(page.getByText("2 added · 0 updated · 1 rows in error")).toBeVisible();
const runRow = page.locator(".errortable tbody tr");
await expect(runRow).toHaveCount(1);
await expect(runRow.locator("td").nth(0)).toHaveText("3");
await expect(runRow.locator("td").nth(1)).toHaveText("Variant Price");
await expect(runRow.locator("td").nth(2)).toContainText("is not a price");
// The catalog gained the two good products.
await gotoProducts(page);
await expect(page.getByRole("heading", { level: 1, name: "Products · 2" })).toBeVisible();
});
+25
View File
@@ -0,0 +1,25 @@
// DoD scenario e2e_import_file_rejected (SD-0002 §6.8): a file-level rejection
// (PUC-5a) renders in place on the upload screen, the picker stays live for a
// retry, and nothing is recorded — no products, no run.
import { expect, test } from "@playwright/test";
import { gotoProducts, signUpWithStorefront, uploadFixture } from "../helpers";
test("e2e_import_file_rejected", async ({ page }) => {
await signUpWithStorefront(page);
await gotoProducts(page);
await uploadFixture(page, "missing-title.csv");
// The rejection renders in place, naming the missing column.
await expect(page.getByText("That file can't be imported")).toBeVisible();
await expect(page.getByText("missing the required column 'Title'")).toBeVisible();
// The picker is live again for a retry.
await expect(page.locator('input[type="file"]')).toBeEnabled();
// No trace: still the empty catalog, and no run recorded.
await gotoProducts(page);
await expect(
page.getByText("No products yet. Bulk import is how product data gets in."),
).toBeVisible();
await expect(page.getByText("No imports yet.")).toBeVisible();
});
+49
View File
@@ -0,0 +1,49 @@
// DoD scenario e2e_import_preview_confirm (SD-0002 §6.8): the happy path end to
// end — sign up, empty catalog, upload good.csv, preview gate (PUC-2/3), confirm,
// run report card, and the catalog + history reflecting the import. Subsumes the
// Task-15 harness smoke (sign-up journey + products empty state).
import { expect, test } from "@playwright/test";
import { gotoProducts, signUpWithStorefront, STOREFRONT_NAME, uploadFixture } from "../helpers";
test("e2e_import_preview_confirm", async ({ page }) => {
await signUpWithStorefront(page);
// Admin topbar: storefront identity + the signed-in account chip (ex-smoke).
await expect(page.locator(".storeid__name")).toHaveText(STOREFRONT_NAME);
await expect(page.getByRole("button", { name: "Sign out" })).toBeVisible();
await gotoProducts(page);
// Empty state + the import affordances (SD-0002 §5.2, ex-smoke).
await expect(
page.getByText("No products yet. Bulk import is how product data gets in."),
).toBeVisible();
await expect(page.getByRole("link", { name: "Download sample CSV" })).toBeVisible();
await uploadFixture(page, "good.csv");
// Preview (§5.4): summary tiles + the file's name in the heading.
await expect(page.getByRole("heading", { name: "Import preview — good.csv" })).toBeVisible();
await expect(page.getByRole("button", { name: "2 to add" })).toBeVisible();
await expect(page.getByRole("button", { name: "0 errors" })).toBeVisible();
// Drill in: open star-tee's diff row and see field-level detail (the S-variant SKU).
const starTee = page.locator(".difflist__item", { hasText: "star-tee" });
await starTee.locator("summary").click();
await expect(starTee.getByText("WG-TEE-S")).toBeVisible();
// Consent gate (PUC-3): confirm the import.
await page.getByRole("button", { name: "Import 2 products" }).click();
// Run detail (§5.5): report card with the file name, counts, and status.
await expect(page.getByRole("heading", { level: 1, name: "good.csv" })).toBeVisible();
await expect(page.getByText("2 added · 0 updated · 0 rows in error")).toBeVisible();
await expect(page.getByText("Complete", { exact: true })).toBeVisible();
// Back on Products: the catalog count and exactly one history row for this run.
await gotoProducts(page);
await expect(page.getByRole("heading", { level: 1, name: "Products · 2" })).toBeVisible();
const rows = page.locator(".datatable tbody tr");
await expect(rows).toHaveCount(1);
await expect(rows.first().getByRole("link", { name: "good.csv" })).toBeVisible();
});
+35
View File
@@ -0,0 +1,35 @@
// DoD scenario e2e_import_shopify_dialect (SD-0002 §6.8, SLICE-8): a Shopify
// product CSV imports directly (PUC-6). Upload an unmodified-shape Shopify export;
// the preview recognizes the dialect ("Shopify product CSV — mapped"), lists the
// columns with no canonical home as not-imported, and the import confirms and
// completes — the run report card carrying the same dialect label.
import { expect, test } from "@playwright/test";
import { gotoProducts, signUpWithStorefront, uploadFixture } from "../helpers";
test("e2e_import_shopify_dialect", async ({ page }) => {
await signUpWithStorefront(page);
await gotoProducts(page);
await uploadFixture(page, "shopify-export.csv");
// Preview (§5.4): the file name, the recognized dialect, and the not-imported band.
await expect(
page.getByRole("heading", { name: "Import preview — shopify-export.csv" }),
).toBeVisible();
await expect(page.getByText("Shopify product CSV — mapped")).toBeVisible();
// The not-imported band lists Shopify columns with no canonical home. Assert a
// column that never appears in canonical diff detail, so the match is unambiguous.
await expect(page.getByText("Columns not imported")).toBeVisible();
await expect(page.getByText("Variant Compare At Price")).toBeVisible();
await expect(page.getByRole("button", { name: "1 to add" })).toBeVisible();
await expect(page.getByRole("button", { name: "0 errors" })).toBeVisible();
// Consent gate (PUC-3): confirm the import.
await page.getByRole("button", { name: "Import 1 products" }).click();
// Run detail (§5.5): report card carries the same dialect label and completes.
await expect(page.getByRole("heading", { level: 1, name: "shopify-export.csv" })).toBeVisible();
await expect(page.getByText("Shopify product CSV — mapped")).toBeVisible();
await expect(page.getByText("1 added · 0 updated · 0 rows in error")).toBeVisible();
await expect(page.getByText("Complete", { exact: true })).toBeVisible();
});
+30
View File
@@ -0,0 +1,30 @@
// DoD scenario e2e_roundtrip_noop (SD-0002 §6.8, PUC-10): exporting then
// re-importing the unmodified file previews as all-unchanged with the import
// action disabled and the "Nothing to change" note.
import { expect, test } from "@playwright/test";
import { exportCatalog, gotoProducts, importGoodCsv, signUpWithStorefront } from "../helpers";
test("e2e_roundtrip_noop", async ({ page }) => {
await signUpWithStorefront(page);
await gotoProducts(page);
await importGoodCsv(page);
await gotoProducts(page);
// Export the catalog, then re-import the unmodified file.
const { path } = await exportCatalog(page, "All products");
await page.getByRole("link", { name: "Import products" }).first().click();
await expect(page.getByRole("heading", { name: "Import products" })).toBeVisible();
await page.locator('input[type="file"]').setInputFiles(path);
// Preview: everything unchanged, nothing to add/update (PUC-10).
await expect(page.getByRole("button", { name: "2 unchanged" })).toBeVisible();
await expect(page.getByRole("button", { name: "0 to add" })).toBeVisible();
await expect(page.getByRole("button", { name: "0 to update" })).toBeVisible();
// The import action is disabled, with the no-op note.
const importBtn = page.getByRole("button", { name: /^Import 0 products$/ });
await expect(importBtn).toBeDisabled();
await expect(
page.getByText("Nothing to change — your catalog already matches this file"),
).toBeVisible();
});
+2 -2
View File
@@ -1,12 +1,12 @@
{
"name": "wiggleverse-ecomm-frontend",
"version": "0.2.0",
"version": "0.8.0",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "wiggleverse-ecomm-frontend",
"version": "0.2.0",
"version": "0.8.0",
"dependencies": {
"react": "^18.3.1",
"react-dom": "^18.3.1"
+1 -1
View File
@@ -1,7 +1,7 @@
{
"name": "wiggleverse-ecomm-frontend",
"private": true,
"version": "0.4.0",
"version": "0.8.0",
"type": "module",
"scripts": {
"dev": "vite",
+18
View File
@@ -0,0 +1,18 @@
import { describe, expect, it } from "vitest";
import { adminViewFor, hashFor, type AdminView } from "./adminRouting";
const VIEWS: AdminView[] = [
{ view: "home" }, { view: "products" }, { view: "import-upload" },
{ view: "import-preview", draftId: 7 }, { view: "run-detail", runId: 12 },
];
describe("adminRouting", () => {
it("round-trips every view", () => {
for (const v of VIEWS) expect(adminViewFor(hashFor(v))).toEqual(v);
});
it("defaults junk to home/products", () => {
expect(adminViewFor("")).toEqual({ view: "home" });
expect(adminViewFor("#/nonsense")).toEqual({ view: "home" });
expect(adminViewFor("#/products/imports/drafts/abc")).toEqual({ view: "products" });
});
});
+31
View File
@@ -0,0 +1,31 @@
// Hash routing for admin sections (SD-0002 §5). The URL is the durable handle on a
// section (PUC-8: run detail can be left and returned to); the SD-0001 entry-routing
// rule (routing.ts) still decides whether the admin renders at all.
export type AdminView =
| { view: "home" }
| { view: "products" }
| { view: "import-upload" }
| { view: "import-preview"; draftId: number }
| { view: "run-detail"; runId: number };
export function adminViewFor(hash: string): AdminView {
const parts = hash.replace(/^#\/?/, "").split("/").filter(Boolean);
if (parts[0] !== "products") return { view: "home" };
if (parts.length === 1) return { view: "products" };
if (parts[1] === "import" && parts.length === 2) return { view: "import-upload" };
if (parts[1] === "imports" && parts[2] === "drafts" && /^\d+$/.test(parts[3] ?? ""))
return { view: "import-preview", draftId: Number(parts[3]) };
if (parts[1] === "imports" && parts[2] === "runs" && /^\d+$/.test(parts[3] ?? ""))
return { view: "run-detail", runId: Number(parts[3]) };
return { view: "products" };
}
export function hashFor(v: AdminView): string {
switch (v.view) {
case "home": return "#/";
case "products": return "#/products";
case "import-upload": return "#/products/import";
case "import-preview": return `#/products/imports/drafts/${v.draftId}`;
case "run-detail": return `#/products/imports/runs/${v.runId}`;
}
}
+1 -1
View File
@@ -15,7 +15,7 @@ export interface VerifyResult {
created: boolean;
}
async function errorOf(resp: Response): Promise<ApiError> {
export async function errorOf(resp: Response): Promise<ApiError> {
try {
const body = await resp.json();
if (body && body.error) return body.error as ApiError;
+24
View File
@@ -0,0 +1,24 @@
import { describe, expect, it } from "vitest";
import { cooldownAppliesTo } from "./cooldown";
describe("resend cooldown keying (SD-0001 §5.2, INV-3; bug #20)", () => {
it("no cooldown running -> send not blocked", () => {
expect(cooldownAppliesTo("a@example.com", null, 0)).toBe(false);
});
it("cooldown running for the same address -> blocked", () => {
expect(cooldownAppliesTo("a@example.com", "a@example.com", 31)).toBe(true);
});
it("same address modulo case/whitespace -> still blocked", () => {
expect(cooldownAppliesTo(" A@Example.COM ", "a@example.com", 31)).toBe(true);
});
it("cooldown running but the input is a different address -> not blocked", () => {
expect(cooldownAppliesTo("right@example.com", "wrong@example.com", 31)).toBe(false);
});
it("cooldown expired -> not blocked even for the same address", () => {
expect(cooldownAppliesTo("a@example.com", "a@example.com", 0)).toBe(false);
});
});
+17
View File
@@ -0,0 +1,17 @@
// Resend-cooldown keying (SD-0001 §5.2, INV-3; bug #20). The server's cooldown is
// per-address, so the client countdown must be too: it blocks sending only while it
// is running AND the current input is the address it was set for. Normalization
// mirrors the backend's normalize_email (lowercase + strip, INV-2).
function normalize(email: string): string {
return email.trim().toLowerCase();
}
export function cooldownAppliesTo(
input: string,
cooldownEmail: string | null,
secondsLeft: number,
): boolean {
if (secondsLeft <= 0 || cooldownEmail === null) return false;
return normalize(input) === normalize(cooldownEmail);
}
+10
View File
@@ -0,0 +1,10 @@
import { describe, expect, it } from "vitest";
import { dialectLabel } from "./productsApi";
describe("dialectLabel", () => {
it("labels canonical", () => expect(dialectLabel("canonical")).toBe("Canonical format"));
it("labels shopify as mapped (PUC-6)", () =>
expect(dialectLabel("shopify")).toBe("Shopify product CSV — mapped"));
it("passes through an unknown dialect verbatim", () =>
expect(dialectLabel("weird")).toBe("weird"));
});
+12
View File
@@ -0,0 +1,12 @@
import { describe, expect, it } from "vitest";
import { EXPORT_STATUSES, exportUrl } from "./productsApi";
describe("export url", () => {
it("lists the four status filters with 'all' first", () => {
expect(EXPORT_STATUSES.map((s) => s.value)).toEqual(["all", "active", "draft", "archived"]);
});
it("builds the endpoint url with the status query", () => {
expect(exportUrl("all")).toBe("/api/products/export?status=all");
expect(exportUrl("archived")).toBe("/api/products/export?status=archived");
});
});
+122
View File
@@ -0,0 +1,122 @@
// Typed fetch wrappers for the /api/products/* surface (SD-0002 §6.4). Same
// conventions as api.ts: same-origin, cookie session, §6.4 error envelope.
import { errorOf, type ApiError } from "./api";
export interface DiffSummary { adds: number; updates: number; unchanged: number; errors: number; }
export interface Draft {
id: number; file_name: string; dialect: string;
summary: DiffSummary; unknown_columns: string[]; expires_at: string;
}
export type RecordKind = "add" | "update" | "unchanged" | "error";
export interface RowErrorDetail { line: number; column: string | null; message: string; }
export interface FieldChange { field: string; before: unknown; after: unknown; }
export interface VariantEntry {
options: (string | null)[]; kind?: string;
set?: Record<string, unknown>; changes?: FieldChange[];
}
export interface ImageEntry { src: string; kind?: string; position?: number; alt_text?: string | null; changes?: FieldChange[]; }
export interface DraftRecord {
handle: string; title: string; kind: RecordKind; variant_count: number;
detail: {
set?: Record<string, unknown>;
option_names?: (string | null)[];
changes?: FieldChange[];
variants?: VariantEntry[];
images?: ImageEntry[];
errors?: RowErrorDetail[];
};
}
export interface ImageOutcome {
handle: string; url: string; outcome: string; reason: string | null; variant: string | null;
}
export interface ImageCounts { fetched: number; rejected: number; failed: number; }
export interface RunSummary {
id: number; file_name: string; dialect: string; created_at: string;
completed_at: string | null; status: string; by: string;
products_added: number; products_updated: number; rows_errored: number;
image_counts?: ImageCounts;
}
export interface RunDetail extends RunSummary {
errors: RowErrorDetail[];
image_progress: { done: number; total: number };
image_counts: ImageCounts;
image_outcomes: ImageOutcome[];
}
export interface ProductsSummary {
product_count: number; image_problem_count: number; latest_run_id: number | null;
}
export type Result<T> = { ok: true; value: T } | { ok: false; error: ApiError; status: number };
// One label rule for CSV dialects, shared by Products history / preview / run detail.
export function dialectLabel(d: string): string {
if (d === "canonical") return "Canonical format";
if (d === "shopify") return "Shopify product CSV — mapped";
return d;
}
export type ExportStatus = "all" | "active" | "draft" | "archived";
// PUC-9 status filter, 'all' first (the default). Labels drive the export menu.
export const EXPORT_STATUSES: { value: ExportStatus; label: string }[] = [
{ value: "all", label: "All products" },
{ value: "active", label: "Active" },
{ value: "draft", label: "Draft" },
{ value: "archived", label: "Archived" },
];
// The export is a browser download (native save dialog + streaming), not a fetch
// wrapper — so the API module contributes a URL builder, not a request().
export function exportUrl(status: ExportStatus): string {
return `/api/products/export?status=${status}`;
}
async function request<T>(path: string, init?: RequestInit): Promise<Result<T>> {
const resp = await fetch(path, { credentials: "include", ...init });
if (!resp.ok) return { ok: false, error: await errorOf(resp), status: resp.status };
if (resp.status === 204) return { ok: true, value: undefined as T };
return { ok: true, value: (await resp.json()) as T };
}
export function getProductsSummary(): Promise<Result<ProductsSummary>> {
return request("/api/products/summary");
}
export function uploadImport(file: File): Promise<Result<Draft>> {
const body = new FormData();
body.append("file", file);
// No content-type header: the browser sets the multipart boundary.
return request("/api/products/imports", { method: "POST", body });
}
export function getDraft(id: number): Promise<Result<Draft>> {
return request(`/api/products/imports/drafts/${id}`);
}
export async function getDraftRecords(
id: number, kind?: RecordKind, limit = 100, offset = 0,
): Promise<Result<DraftRecord[]>> {
const params = new URLSearchParams({ limit: String(limit), offset: String(offset) });
if (kind) params.set("kind", kind);
const resp = await request<{ records: DraftRecord[] }>(
`/api/products/imports/drafts/${id}/records?${params}`,
);
return resp.ok ? { ok: true, value: resp.value.records } : resp;
}
export function confirmDraft(id: number): Promise<Result<{ run_id: number }>> {
return request(`/api/products/imports/drafts/${id}/confirm`, { method: "POST" });
}
export function cancelDraft(id: number): Promise<Result<void>> {
return request(`/api/products/imports/drafts/${id}`, { method: "DELETE" });
}
export async function listRuns(): Promise<Result<RunSummary[]>> {
const resp = await request<{ runs: RunSummary[] }>("/api/products/imports/runs");
return resp.ok ? { ok: true, value: resp.value.runs } : resp;
}
export function getRun(id: number): Promise<Result<RunDetail>> {
return request(`/api/products/imports/runs/${id}`);
}
+51 -22
View File
@@ -1,9 +1,16 @@
// Admin shell (SD-0001 §5.4) — the storefront's stable home; honestly empty this release
// (PUC-8; PUC-9 sign-out). Renders from /me alone: storefront name + signed-in email. No
// zeroed metric tiles, no locked-feature teasers (OHM: Agency & Anti-Manipulation).
// Visuals per the ui/designs export (hf-admin).
// Admin shell (SD-0001 §5.4) — the storefront's stable home; the home view is honestly
// empty this release (PUC-8; PUC-9 sign-out). Renders from /me alone: storefront name +
// signed-in email. No zeroed metric tiles, no locked-feature teasers (OHM: Agency &
// Anti-Manipulation). Visuals per the ui/designs export (hf-admin). SD-0002 §5 adds the
// admin nav strip + hash-routed products section (adminRouting.ts).
import { useEffect, useState } from "react";
import { adminViewFor, type AdminView } from "../adminRouting";
import { logout } from "../api";
import { AccountChip, Banner, Eyebrow, Screen, TopBar } from "../ui/kit";
import ImportPreview from "./products/ImportPreview";
import ImportUpload from "./products/ImportUpload";
import ProductsPage from "./products/ProductsPage";
import RunDetail from "./products/RunDetail";
interface Props {
storefrontName: string;
@@ -13,6 +20,14 @@ interface Props {
}
export default function Admin({ storefrontName, email, welcome, onSignedOut }: Props) {
const [view, setView] = useState<AdminView>(adminViewFor(window.location.hash));
useEffect(() => {
const onHashChange = () => setView(adminViewFor(window.location.hash));
window.addEventListener("hashchange", onHashChange);
return () => window.removeEventListener("hashchange", onHashChange);
}, []);
async function signOut() {
await logout();
onSignedOut();
@@ -32,27 +47,41 @@ export default function Admin({ storefrontName, email, welcome, onSignedOut }: P
}
right={<AccountChip email={email} onSignOut={signOut} />}
/>
<nav className="adminnav" aria-label="Admin sections">
<a className={`adminnav__item${view.view === "home" ? " adminnav__item--active" : ""}`} href="#/">
Overview
</a>
<a className={`adminnav__item${view.view !== "home" ? " adminnav__item--active" : ""}`} href="#/products">
Products
</a>
</nav>
<main className="screen__main">
<div className="empty">
{welcome && (
<div style={{ marginBottom: 24, width: "100%" }}>
<Banner tone="info" title={welcome === "new" ? "Welcome to ecomm" : "Welcome back"}>
{welcome === "new"
? "A new account was created for this email."
: "Signed in to your existing account."}
</Banner>
{view.view === "home" && (
<div className="empty">
{welcome && (
<div style={{ marginBottom: 24, width: "100%" }}>
<Banner tone="info" title={welcome === "new" ? "Welcome to ecomm" : "Welcome back"}>
{welcome === "new"
? "A new account was created for this email."
: "Signed in to your existing account."}
</Banner>
</div>
)}
<div className="empty__seal" aria-hidden="true">
<img src="/brand/mark-mono-gold.svg" width={36} height={36} alt="" />
</div>
)}
<div className="empty__seal" aria-hidden="true">
<img src="/brand/mark-mono-gold.svg" width={36} height={36} alt="" />
<Eyebrow>Your storefront</Eyebrow>
<h1>{storefrontName}</h1>
<p className="empty__copy">
There's nothing to manage yet — and that's a finished state, not a missing one.
Catalog, orders, and settings will appear here as ecomm grows.
</p>
</div>
<Eyebrow>Your storefront</Eyebrow>
<h1>{storefrontName}</h1>
<p className="empty__copy">
There's nothing to manage yet — and that's a finished state, not a missing one.
Catalog, orders, and settings will appear here as ecomm grows.
</p>
</div>
)}
{view.view === "products" && <ProductsPage />}
{view.view === "import-upload" && <ImportUpload />}
{view.view === "import-preview" && <ImportPreview draftId={view.draftId} />}
{view.view === "run-detail" && <RunDetail runId={view.runId} />}
</main>
</Screen>
);
+12 -2
View File
@@ -3,6 +3,7 @@
// is handled by App on success. Visuals per the ui/designs export (hf-signin).
import { useEffect, useState } from "react";
import { requestCode, verifyCode, type ApiError, type VerifyResult } from "../api";
import { cooldownAppliesTo } from "../cooldown";
import CodeInput from "../ui/CodeInput";
import { AuthCard, Banner, Field, Footer, PrimaryButton, Screen, TopBar, Wordmark } from "../ui/kit";
@@ -34,6 +35,9 @@ export default function SignIn({ door, onAuthed, onBack }: Props) {
const [codeError, setCodeError] = useState<string | null>(null);
const [bannerError, setBannerError] = useState<ApiError | null>(null);
const [cooldown, setCooldown] = useCooldown();
// The address the running cooldown belongs to — the server's cooldown is per-address
// (INV-3), so a different address must not be blocked by it (bug #20).
const [cooldownFor, setCooldownFor] = useState<string | null>(null);
function applyError(err: ApiError) {
setFieldError(null);
@@ -53,9 +57,11 @@ export default function SignIn({ door, onAuthed, onBack }: Props) {
setBusy(false);
if (err) {
applyError(err);
if (err.code === "resend_cooldown") setCooldownFor(email);
return err.code === "resend_cooldown"; // a cooldown still means a code is out there
}
setCooldown(60);
setCooldownFor(email);
return true;
}
@@ -131,8 +137,12 @@ export default function SignIn({ door, onAuthed, onBack }: Props) {
}}
/>
<div style={{ display: "flex", flexDirection: "column", gap: 12 }}>
<PrimaryButton busy={busy} disabled={cooldown > 0}>
{busy ? "Sending…" : cooldown > 0 ? `Resend in ${cooldown}s` : "Send code"}
<PrimaryButton busy={busy} disabled={cooldownAppliesTo(email, cooldownFor, cooldown)}>
{busy
? "Sending…"
: cooldownAppliesTo(email, cooldownFor, cooldown)
? `Resend in ${cooldown}s`
: "Send code"}
</PrimaryButton>
<p className="note">
We'll email you a one-time code. That's all we need no password, ever.
@@ -0,0 +1,393 @@
// Import preview (SD-0002 §5.4) — the consent gate. Summary tiles filter a
// drill-in diff list; the sticky footer carries confirm (PUC-3) / cancel (PUC-3a).
// Diff glyphs pair with color, never color alone (§6.6).
import { useEffect, useRef, useState } from "react";
import {
cancelDraft,
confirmDraft,
dialectLabel,
getDraft,
getDraftRecords,
type Draft,
type DraftRecord,
type FieldChange,
type RecordKind,
type VariantEntry,
} from "../../productsApi";
import { Banner } from "../../ui/kit";
const PAGE = 100;
function fmt(v: unknown): string {
if (v === null || v === undefined) return "—";
if (Array.isArray(v)) return v.length ? v.join(", ") : "—";
if (typeof v === "boolean") return v ? "TRUE" : "FALSE";
return String(v);
}
function KindChip({ kind }: { kind: RecordKind }) {
return <span className={`kindchip kindchip--${kind}`}>{kind}</span>;
}
function SetLine({ field, value }: { field: string; value: unknown }) {
return (
<div className="diffchange">
<span className="diffchange__glyph--add">+ </span>
{field}: {fmt(value)}
</div>
);
}
function ChangeRow({ change }: { change: FieldChange }) {
return (
<div className="diffchange">
{change.field}: <span className="diffchange__glyph--del"> {fmt(change.before)}</span> {" "}
<span className="diffchange__glyph--add">+ {fmt(change.after)}</span>
</div>
);
}
function variantLabel(v: VariantEntry): string {
const opts = v.options.filter((o): o is string => o != null);
return opts.length ? `Variant ${opts.join(" / ")}` : "Variant";
}
function RecordDetail({ record }: { record: DraftRecord }) {
const d = record.detail;
if (record.kind === "error") {
return (
<div>
{(d.errors ?? []).map((e, i) => (
<div className="diffchange" key={i}>
line {e.line}: {e.column != null && `'${e.column}' — `}
{e.message}
</div>
))}
</div>
);
}
return (
<div>
{Object.entries(d.set ?? {}).map(([field, value]) => (
<SetLine key={field} field={field} value={value} />
))}
{(d.changes ?? []).map((c, i) => (
<ChangeRow key={i} change={c} />
))}
{(d.variants ?? []).map((v, i) => (
<div key={i}>
<div className="diffchange diffchange--head">
{variantLabel(v)}
{v.kind ? ` (${v.kind})` : ""}
</div>
{Object.entries(v.set ?? {}).map(([field, value]) => (
<SetLine key={field} field={field} value={value} />
))}
{(v.changes ?? []).map((c, j) => (
<ChangeRow key={j} change={c} />
))}
</div>
))}
{(d.images ?? []).map((img, i) =>
img.kind && img.kind !== "add" ? (
<div key={i}>
<div className="diffchange diffchange--head">
image: {img.src} ({img.kind})
</div>
{(img.changes ?? []).map((c, j) => (
<ChangeRow key={j} change={c} />
))}
</div>
) : (
<div className="diffchange" key={i}>
<span className="diffchange__glyph--add">+ </span>image: {img.src}
</div>
),
)}
</div>
);
}
function ErrorTable({ records }: { records: DraftRecord[] }) {
const rows = records.flatMap((r) => r.detail.errors ?? []);
if (rows.length === 0) return null;
return (
<table className="errortable">
<thead>
<tr>
<th>Line</th>
<th>Column</th>
<th>Problem</th>
</tr>
</thead>
<tbody>
{rows.map((e, i) => (
<tr key={i}>
<td>{e.line}</td>
<td>{e.column ?? "—"}</td>
<td>{e.message}</td>
</tr>
))}
</tbody>
</table>
);
}
export default function ImportPreview({ draftId }: { draftId: number }) {
const [draft, setDraft] = useState<Draft | null>(null);
const [loadFail, setLoadFail] = useState<"gone" | "expired" | "failed" | null>(null);
const [filter, setFilter] = useState<RecordKind | null>(null);
const [records, setRecords] = useState<DraftRecord[] | null>(null);
const [recordsError, setRecordsError] = useState<"load" | "more" | null>(null);
const [hasMore, setHasMore] = useState(false);
const [moreBusy, setMoreBusy] = useState(false);
const [confirming, setConfirming] = useState(false);
const [cancelling, setCancelling] = useState(false);
const [stale, setStale] = useState(false);
const [nothingNote, setNothingNote] = useState(false);
const [confirmError, setConfirmError] = useState<string | null>(null);
// Generation counter for the records list: bumped on every page-0 (re)load, so a
// page-0 or show-more response that resolves after a tile/filter (or draft) switch
// is recognized as stale and dropped instead of clobbering/appending to the new list.
const recordsGen = useRef(0);
async function loadDraft() {
setLoadFail(null);
const resp = await getDraft(draftId);
if (!resp.ok) {
setLoadFail(resp.status === 404 ? "gone" : resp.status === 410 ? "expired" : "failed");
return;
}
setDraft(resp.value);
}
useEffect(() => {
// A new draft means a fresh consent gate — reset everything the old one set.
setFilter(null);
setStale(false);
setConfirmError(null);
setNothingNote(false);
setRecords(null);
setRecordsError(null);
setHasMore(false);
void loadDraft();
// eslint-disable-next-line react-hooks/exhaustive-deps
}, [draftId]);
function loadRecords(kind: RecordKind | null) {
recordsGen.current += 1;
const gen = recordsGen.current;
setRecords(null);
setRecordsError(null);
setHasMore(false);
void getDraftRecords(draftId, kind ?? undefined, PAGE, 0).then((resp) => {
if (gen !== recordsGen.current) return;
if (!resp.ok) {
setRecordsError("load");
return;
}
setRecords(resp.value);
setHasMore(resp.value.length === PAGE);
});
}
useEffect(() => {
if (!draft) return;
loadRecords(filter);
// eslint-disable-next-line react-hooks/exhaustive-deps
}, [draft, draftId, filter]);
async function showMore() {
if (!records) return;
const gen = recordsGen.current;
setMoreBusy(true);
setRecordsError(null);
const resp = await getDraftRecords(draftId, filter ?? undefined, PAGE, records.length);
setMoreBusy(false);
// Filter/draft switched while this page was in flight — drop the stale page.
if (gen !== recordsGen.current) return;
if (!resp.ok) {
setRecordsError("more");
return;
}
setRecords((prev) => [...(prev ?? []), ...resp.value]);
setHasMore(resp.value.length === PAGE);
}
async function onConfirm() {
setConfirming(true);
setConfirmError(null);
const resp = await confirmDraft(draftId);
if (resp.ok) {
window.location.hash = `#/products/imports/runs/${resp.value.run_id}`;
return;
}
setConfirming(false);
if ((resp.status === 409 && resp.error.code === "preview_stale") || resp.status === 410) {
setStale(true);
} else if (resp.status === 409 && resp.error.code === "nothing_to_apply") {
setNothingNote(true);
} else {
setConfirmError(resp.error.message);
}
}
async function onCancel() {
setCancelling(true);
// PUC-3a — cancel even on draft-gone (404) still navigates home.
await cancelDraft(draftId);
window.location.hash = "#/products";
}
if (loadFail === "gone") {
return (
<Banner tone="attn" title="This preview is gone">
<a href="#/products">Back to Products</a>
</Banner>
);
}
if (loadFail === "expired") {
return (
<Banner tone="attn" title="This preview expired — upload the file again">
<a href="#/products/import">Upload the file again</a>
</Banner>
);
}
if (loadFail === "failed") {
return (
<Banner tone="attn" title="Couldn't load this preview">
Something went wrong on our side.{" "}
<button type="button" className="linklike" onClick={() => void loadDraft()}>
Retry
</button>
</Banner>
);
}
if (!draft) {
return (
<p className="note" role="status">
Loading
</p>
);
}
const { summary } = draft;
const tiles: { kind: RecordKind; num: number; label: string }[] = [
{ kind: "add", num: summary.adds, label: "to add" },
{ kind: "update", num: summary.updates, label: "to update" },
{ kind: "unchanged", num: summary.unchanged, label: "unchanged" },
{ kind: "error", num: summary.errors, label: "errors" },
];
const toApply = summary.adds + summary.updates;
return (
<div className="products">
<p className="note">
<a href="#/products"> Products</a>
</p>
<h1>Import preview {draft.file_name}</h1>
<p className="note">{dialectLabel(draft.dialect)}</p>
{draft.unknown_columns.length > 0 && (
<Banner tone="info" title="Columns not imported">
{draft.unknown_columns.length > 8 ? (
<details>
<summary>{draft.unknown_columns.length} columns not imported</summary>
{draft.unknown_columns.join(", ")}
</details>
) : (
draft.unknown_columns.join(", ")
)}
</Banner>
)}
<div className="tiles">
{tiles.map((t) => (
<button
key={t.kind}
type="button"
className={`tile tile--${t.kind}${filter === t.kind ? " tile--active" : ""}`}
aria-pressed={filter === t.kind}
onClick={() => setFilter(filter === t.kind ? null : t.kind)}
>
<span className="tile__num">{t.num.toLocaleString()}</span>
<span className="tile__label">{t.label}</span>
</button>
))}
</div>
{filter === "error" && records && <ErrorTable records={records} />}
{recordsError === "load" ? (
<p className="note note--attn" role="alert">
Couldn't load these records.{" "}
<button type="button" className="linklike" onClick={() => loadRecords(filter)}>
Retry
</button>
</p>
) : records === null ? (
<p className="note" role="status">
Loading…
</p>
) : records.length === 0 ? (
<p className="note">Nothing to show here.</p>
) : (
<div className="difflist">
{records.map((r, i) => (
<details className="difflist__item" key={`${r.handle}-${i}`}>
<summary>
<span className="difflist__handle">{r.handle}</span> · {r.title} ·{" "}
<KindChip kind={r.kind} /> · {r.variant_count}{" "}
{r.variant_count === 1 ? "variant" : "variants"}
</summary>
<RecordDetail record={r} />
</details>
))}
</div>
)}
{hasMore && (
<p>
<button type="button" className="btn-secondary" disabled={moreBusy} onClick={() => void showMore()}>
{moreBusy ? "Loading…" : "Show more"}
</button>
{recordsError === "more" && (
<span className="note note--attn" role="alert">
Couldn't load more records.{" "}
<button type="button" className="linklike" onClick={() => void showMore()}>
Retry
</button>
</span>
)}
</p>
)}
<div className="sticky-footer" aria-live="polite">
{stale ? (
<Banner tone="attn" title="Your catalog changed since this preview — upload the file again">
<a href="#/products/import">Upload the file again</a>
</Banner>
) : (
<>
<button
type="button"
className="btn-primary"
disabled={toApply === 0 || confirming || cancelling}
onClick={() => void onConfirm()}
>
{confirming ? "Importing…" : `Import ${toApply.toLocaleString()} products`}
</button>
<button
type="button"
className="btn-secondary"
disabled={confirming || cancelling}
onClick={() => void onCancel()}
>
Cancel
</button>
{(toApply === 0 || nothingNote) && (
<span className="note">Nothing to change your catalog already matches this file</span>
)}
{confirmError && (
<span className="note note--attn" role="alert">
{confirmError}
</span>
)}
</>
)}
</div>
</div>
);
}
@@ -0,0 +1,64 @@
// Import — upload (SD-0002 §5.3). Selecting a file starts upload + validation
// immediately (PUC-2); file-level rejections (PUC-5a) render in place with the
// picker live for retry. No notifications — errors render here.
import { useRef, useState } from "react";
import { uploadImport } from "../../productsApi";
import { Banner } from "../../ui/kit";
export default function ImportUpload() {
const [busy, setBusy] = useState(false);
const [error, setError] = useState<string | null>(null);
const inputRef = useRef<HTMLInputElement>(null);
async function onPick(files: FileList | null) {
const file = files?.[0];
if (!file) return;
setBusy(true);
setError(null);
const resp = await uploadImport(file);
setBusy(false);
if (inputRef.current) inputRef.current.value = "";
if (!resp.ok) {
setError(resp.error.message);
return;
}
window.location.hash = `#/products/imports/drafts/${resp.value.id}`;
}
return (
<div className="products products--narrow">
<p className="note">
<a href="#/products"> Products</a>
</p>
<h1>Import products</h1>
{error && (
<Banner tone="attn" title="That file can't be imported">
{error}
</Banner>
)}
<label className={`dropzone${busy ? " dropzone--busy" : ""}`}>
<input
ref={inputRef}
type="file"
accept=".csv,text/csv"
disabled={busy}
onChange={(e) => void onPick(e.target.files)}
/>
<span className="dropzone__title" aria-live="polite">
{busy ? "Validating…" : "Choose a CSV file"}
</span>
<span className="note">CSV, up to 5,000 rows</span>
</label>
<p className="note">
Works with the canonical format or a Shopify product CSV.{" "}
<a href="/api/products/sample.csv" download>
Download sample CSV
</a>{" "}
·{" "}
<a href="/api/products/columns.md" target="_blank" rel="noopener">
Column reference
</a>
</p>
</div>
);
}
@@ -0,0 +1,167 @@
// Products page (SD-0002 §5.2) — the catalog's home: where imports start, exports
// download, and history lives. SLICE-6 ships the export menu; the browsable list is #14's.
import { useEffect, useState } from "react";
import {
dialectLabel,
EXPORT_STATUSES,
exportUrl,
getProductsSummary,
listRuns,
type ProductsSummary,
type RunSummary,
} from "../../productsApi";
import { Banner } from "../../ui/kit";
import { isExportEnabled } from "./exportMenu";
import { historyImageCell } from "./runImages";
const STATUS_LABELS: Record<string, string> = {
applying: "Importing…",
fetching_images: "Fetching images…",
complete: "Complete",
complete_with_problems: "Complete with problems",
};
export default function ProductsPage() {
const [summary, setSummary] = useState<ProductsSummary | null>(null);
const [runs, setRuns] = useState<RunSummary[] | null>(null);
const [failed, setFailed] = useState(false);
async function load() {
setFailed(false);
const [s, r] = await Promise.all([getProductsSummary(), listRuns()]);
if (!s.ok || !r.ok) {
setFailed(true);
return;
}
setSummary(s.value);
setRuns(r.value);
}
useEffect(() => {
void load();
}, []);
if (failed) {
return (
<Banner tone="attn" title="Couldn't load your products">
Something went wrong on our side.{" "}
<button type="button" className="linklike" onClick={() => void load()}>
Retry
</button>
</Banner>
);
}
if (!summary || !runs) {
return (
<p className="note" role="status">
Loading
</p>
);
}
const empty = !isExportEnabled(summary.product_count);
return (
<div className="products">
<header className="products__header">
<h1>
Products
{!empty && <span className="products__count"> · {summary.product_count.toLocaleString()}</span>}
</h1>
<div className="products__actions">
{empty ? (
<div className="products__export">
<button type="button" className="btn-secondary" disabled>
Export
</button>
<span className="note">Export arrives when you have products</span>
</div>
) : (
<details className="products__export menu">
<summary className="btn-secondary" role="button">
Export
</summary>
<ul className="menu__list">
{EXPORT_STATUSES.map((s) => (
<li key={s.value}>
{/* A real download: the browser navigates to the streamed
endpoint and saves the attachment (PUC-9). */}
<a className="menu__item" href={exportUrl(s.value)} download>
{s.label}
</a>
</li>
))}
</ul>
</details>
)}
<a className="btn-primary" href="#/products/import">
Import products
</a>
</div>
</header>
{summary.image_problem_count > 0 && (
<div className="notice" aria-live="polite">
<a href={`#/products/imports/runs/${summary.latest_run_id}`}>
{summary.image_problem_count} products have image problems
</a>
</div>
)}
{empty ? (
<div className="empty">
<p className="empty__copy">No products yet. Bulk import is how product data gets in.</p>
<a className="btn-primary" href="#/products/import">
Import products
</a>
<p className="note">
<a href="/api/products/sample.csv" download>
Download sample CSV
</a>{" "}
·{" "}
<a href="/api/products/columns.md" target="_blank" rel="noopener">
Column reference
</a>
</p>
</div>
) : (
<p className="note">Your catalog is loaded. The browsable product list arrives with an upcoming release.</p>
)}
<section className="products__history">
<h2>Import history</h2>
{runs.length === 0 ? (
<p className="note">No imports yet.</p>
) : (
<table className="datatable">
<thead>
<tr>
<th>Date</th><th>File</th><th>Dialect</th><th>Added</th><th>Updated</th><th>Errors</th><th>Images</th><th>Status</th>
</tr>
</thead>
<tbody>
{runs.map((r) => (
<tr
key={r.id}
className="datatable__rowlink"
onClick={() => {
window.location.hash = `#/products/imports/runs/${r.id}`;
}}
>
<td>{new Date(r.created_at).toLocaleString()}</td>
<td>
{/* Anchor = the keyboard/SR path (§6.6); the row onClick stays as a
mouse convenience. Both set the same hash, so the double fire on
an anchor click is idempotent. */}
<a href={`#/products/imports/runs/${r.id}`}>{r.file_name}</a>
</td>
<td>{dialectLabel(r.dialect)}</td>
<td>{r.products_added}</td>
<td>{r.products_updated}</td>
<td>{r.rows_errored}</td>
<td>{historyImageCell(r.image_counts)}</td>
<td>{STATUS_LABELS[r.status] ?? r.status}</td>
</tr>
))}
</tbody>
</table>
)}
</section>
</div>
);
}
+151
View File
@@ -0,0 +1,151 @@
// Run detail (SD-0002 §5.5) — report card for a completed or in-progress import run.
// The durable record of the import's image-fetch outcome: live progress while
// fetching (polled, no websockets — deliberate), then per-image problem rows. PUC-4/5/8.
import { useEffect, useState } from "react";
import { dialectLabel, getRun, type RunDetail as RunDetailType } from "../../productsApi";
import {
imageOutcomeSummary,
imageProgressLabel,
isRunTerminal,
outcomeLabel,
} from "./runImages";
import { Banner } from "../../ui/kit";
const STATUS_LABELS: Record<string, string> = {
applying: "Importing…",
fetching_images: "Fetching images…",
complete: "Complete",
complete_with_problems: "Complete with problems",
};
export default function RunDetail({ runId }: { runId: number }) {
const [run, setRun] = useState<RunDetailType | null>(null);
const [loadFail, setLoadFail] = useState<"gone" | "failed" | null>(null);
async function load() {
setLoadFail(null);
const resp = await getRun(runId);
if (!resp.ok) {
setLoadFail(resp.status === 404 ? "gone" : "failed");
return;
}
setRun(resp.value);
}
useEffect(() => {
void load();
// eslint-disable-next-line react-hooks/exhaustive-deps
}, [runId]);
// While the run is still applying / fetching images, poll the endpoint so the
// "Fetching images: X of Y" line advances (no websockets — deliberate, §5.5).
// The interval is cleared on terminal status and on unmount.
useEffect(() => {
if (!run || isRunTerminal(run.status)) return;
const id = setInterval(() => {
void (async () => {
const resp = await getRun(runId);
if (resp.ok) setRun(resp.value);
})();
}, 2000);
return () => clearInterval(id);
}, [run, runId]);
if (loadFail === "gone") {
return (
<Banner tone="attn" title="No such import run">
<a href="#/products"> Products</a>
</Banner>
);
}
if (loadFail === "failed") {
return (
<Banner tone="attn" title="Couldn't load this import run">
Something went wrong on our side.{" "}
<button type="button" className="linklike" onClick={() => void load()}>
Retry
</button>
</Banner>
);
}
if (!run) {
return (
<p className="note" role="status">
Loading
</p>
);
}
return (
<div className="products">
<p className="note">
<a href="#/products"> Products</a>
</p>
<h1>{run.file_name}</h1>
<p className="note">
Imported {new Date(run.created_at).toLocaleString()} by {run.by} · {dialectLabel(run.dialect)}
</p>
<p className="note">
{run.products_added} added · {run.products_updated} updated · {run.rows_errored} rows in
error
</p>
<p className="note">{STATUS_LABELS[run.status] ?? run.status}</p>
{run.errors.length > 0 && (
<table className="errortable">
<thead>
<tr>
<th>Line</th>
<th>Column</th>
<th>Problem</th>
</tr>
</thead>
<tbody>
{run.errors.map((e, i) => (
<tr key={i}>
<td>{e.line}</td>
<td>{e.column ?? "—"}</td>
<td>{e.message}</td>
</tr>
))}
</tbody>
</table>
)}
{!isRunTerminal(run.status) ? (
<p className="note" aria-live="polite">
{imageProgressLabel(run.image_progress)}
</p>
) : (
run.image_progress.total > 0 && (
<>
<p className="note">{imageOutcomeSummary(run.image_counts)}</p>
{run.image_outcomes.length > 0 ? (
<table className="errortable">
<thead>
<tr>
<th>Handle</th>
<th>Variant</th>
<th>Image URL</th>
<th>Outcome</th>
<th>What to do</th>
</tr>
</thead>
<tbody>
{run.image_outcomes.map((o, i) => (
<tr key={i}>
<td>{o.handle}</td>
<td>{o.variant ?? "—"}</td>
<td>{o.url}</td>
<td>{outcomeLabel(o.outcome)}</td>
<td>Correct the URL and re-import.</td>
</tr>
))}
</tbody>
</table>
) : (
<p className="note">All images fetched.</p>
)}
</>
)
)}
</div>
);
}
@@ -0,0 +1,19 @@
import { describe, expect, it } from "vitest";
import { EXPORT_STATUSES, exportUrl } from "../../productsApi";
import { isExportEnabled } from "./exportMenu";
describe("export menu", () => {
it("disables export for an empty catalog and enables it once there are products", () => {
expect(isExportEnabled(0)).toBe(false);
expect(isExportEnabled(3)).toBe(true);
});
it("builds the four status download URLs, 'all' first", () => {
expect(EXPORT_STATUSES.map((s) => exportUrl(s.value))).toEqual([
"/api/products/export?status=all",
"/api/products/export?status=active",
"/api/products/export?status=draft",
"/api/products/export?status=archived",
]);
});
});
@@ -0,0 +1,11 @@
// Pure logic behind the Products page Export menu (SD-0002 §5.2, PUC-9). The
// status list + URL builder live in productsApi.ts (re-used by the component);
// this is the one decision the menu turns on — whether export is offered at all.
// The disclosure JSX is verified by the E2E suite, matching SLICE-2/3's pattern
// of pure-logic unit tests + E2E for screens.
// Export is offered only when the catalog has products (PUC-9: an empty catalog
// shows a disabled button + note instead).
export function isExportEnabled(count: number): boolean {
return count > 0;
}
@@ -0,0 +1,25 @@
import { describe, expect, it } from "vitest";
import { historyImageCell, imageOutcomeSummary, imageProgressLabel, isRunTerminal, outcomeLabel } from "./runImages";
describe("runImages helpers", () => {
it("progress label", () => {
expect(imageProgressLabel({ done: 312, total: 4950 })).toBe("Fetching images: 312 of 4,950");
});
it("outcome summary", () => {
expect(imageOutcomeSummary({ fetched: 4938, rejected: 9, failed: 3 }))
.toBe("4,938 fetched · 9 rejected · 3 failed");
});
it("history cell collapses clean / shows problems / dashes empty", () => {
expect(historyImageCell({ fetched: 10, rejected: 0, failed: 0 })).toBe("10 ✓");
expect(historyImageCell({ fetched: 8, rejected: 1, failed: 1 })).toBe("8 ✓ · 2 ✗");
expect(historyImageCell(undefined)).toBe("—");
});
it("terminal status", () => {
expect(isRunTerminal("fetching_images")).toBe(false);
expect(isRunTerminal("complete_with_problems")).toBe(true);
});
it("outcome labels", () => {
expect(outcomeLabel("rejected_low_res")).toMatch(/resolution bar/);
expect(outcomeLabel("failed")).toMatch(/unreachable/);
});
});
@@ -0,0 +1,32 @@
// Pure logic behind the image-fetch surfaces (SD-0002 §5.5): the run-detail
// progress/outcome labels + the history "Images" cell + terminal-status test
// that drives polling. Mirrors exportMenu.ts: pure helpers unit-tested here,
// the JSX verified by the E2E suite (SLICE-2/3 pattern).
import type { ImageCounts } from "../../productsApi";
export function imageProgressLabel(p: { done: number; total: number }): string {
return `Fetching images: ${p.done.toLocaleString()} of ${p.total.toLocaleString()}`;
}
export function imageOutcomeSummary(c: ImageCounts): string {
return `${c.fetched.toLocaleString()} fetched · ${c.rejected.toLocaleString()} rejected · ${c.failed.toLocaleString()} failed`;
}
export function historyImageCell(c: ImageCounts | undefined): string {
if (!c || c.fetched + c.rejected + c.failed === 0) return "—";
const bad = c.rejected + c.failed;
return bad === 0
? `${c.fetched.toLocaleString()}`
: `${c.fetched.toLocaleString()} ✓ · ${bad.toLocaleString()}`;
}
export function isRunTerminal(status: string): boolean {
return status === "complete" || status === "complete_with_problems";
}
export function outcomeLabel(outcome: string): string {
if (outcome === "rejected_low_res") return "Rejected — below the resolution bar";
if (outcome === "rejected_not_image") return "Rejected — not a supported image";
if (outcome === "failed") return "Failed — unreachable";
return outcome;
}
+1
View File
@@ -5,3 +5,4 @@
@import "./tokens-typography.css";
@import "./tokens-spacing.css";
@import "./app.css";
@import "./products.css";
+303
View File
@@ -0,0 +1,303 @@
/* Products section (SD-0002 §5) — admin nav strip, the catalog's home page, and the
import-flow primitives (dropzone, tiles, difflist — Tasks 1214 consume these).
Same language as app.css: dark ground, glass chrome, hairline borders, small lifts. */
/* Status accents (SD-0002 design bundle): add / update / error. */
:root {
--st-add: #1F8A5B;
--st-update: #B5830F;
--st-error: #C2513E;
}
/* ── admin nav: horizontal strip under the topbar ───────────────────────────── */
.adminnav {
flex: 0 0 auto;
display: flex;
gap: 26px;
padding: 0 36px;
border-bottom: 1px solid var(--border-soft);
background: rgba(9, 12, 34, .35);
}
.adminnav__item {
font-family: var(--wv-font-display);
font-weight: var(--weight-medium);
font-size: 14px;
color: var(--text-on-dark-mute);
text-decoration: none;
padding: 13px 2px 11px;
border-bottom: 2px solid transparent;
transition: color var(--dur-fast) var(--ease);
}
.adminnav__item:hover { color: var(--wv-starlight); }
.adminnav__item--active { color: var(--wv-starlight); border-bottom-color: var(--wv-gold); }
/* ── products page frame ────────────────────────────────────────────────────── */
/* margin-bottom auto pins the page to the top of the centered .screen__main. */
.products { width: 100%; max-width: 880px; margin-bottom: auto; }
.products--narrow { max-width: 560px; }
.products__header {
display: flex;
justify-content: space-between;
align-items: baseline;
gap: 16px;
margin-bottom: 28px;
}
.products__header h1 {
font-family: var(--wv-font-display);
font-weight: var(--weight-bold);
letter-spacing: var(--tracking-display);
font-size: 28px;
line-height: 1.1;
margin: 0;
}
.products__count { color: var(--text-on-dark-mute); font-weight: var(--weight-medium); }
.products__actions { display: flex; gap: 12px; align-items: center; }
/* Disabled Export + its visible "no products yet" caption, stacked (empty catalog). */
.products__export { display: flex; flex-direction: column; gap: 4px; align-items: center; }
.products__export .note { font-size: 11.5px; }
/* Export status menu (SD-0002 §5.2 — PUC-9). A native <details> disclosure so
it's keyboard-accessible with no extra JS (§6.6). Tokens align with the design
bundle (--surface-raised / --border-card / --radius-panel); the fallbacks keep
it working regardless. */
.products__export.menu {
position: relative;
display: inline-block;
}
.products__export.menu > summary {
list-style: none;
cursor: pointer;
}
.products__export.menu > summary::-webkit-details-marker {
display: none;
}
.menu__list {
position: absolute;
right: 0;
z-index: 10;
margin: 0.25rem 0 0;
padding: 0.25rem;
list-style: none;
background: var(--surface-raised, #fff);
border: 1px solid var(--border-card, #d8d3c8);
border-radius: var(--radius-panel, 8px);
box-shadow: var(--shadow-soft, 0 6px 20px rgba(0, 0, 0, 0.12));
min-width: 10rem;
}
.menu__item {
display: block;
padding: 0.5rem 0.75rem;
border-radius: var(--radius-sm, 6px);
text-decoration: none;
color: var(--text-on-dark-soft, inherit);
font-family: var(--wv-font-display);
font-size: 14px;
}
.menu__item:hover,
.menu__item:focus {
background: var(--surface-raised-hi, #f3efe7);
color: var(--wv-starlight);
}
.products .btn-primary { width: auto; text-decoration: none; }
.products .empty { margin: 24px auto 0; }
.products__history { margin-top: 44px; }
.products__history h2 {
font-family: var(--wv-font-display);
font-weight: var(--weight-semibold);
font-size: 17px;
letter-spacing: var(--tracking-display);
margin: 0 0 14px;
}
/* ── secondary button: outline twin of .btn-primary ─────────────────────────── */
.btn-secondary {
display: inline-flex;
align-items: center;
justify-content: center;
gap: .4em;
font-family: var(--wv-font-display);
font-weight: var(--weight-medium);
font-size: 15.5px;
line-height: 1;
padding: .85rem 1.4rem;
border-radius: var(--radius-pill);
border: var(--btn-border-w) solid var(--border-strong);
background: transparent;
color: var(--text-on-dark-soft);
cursor: pointer;
transition: border-color var(--dur-fast) var(--ease), color var(--dur-fast) var(--ease),
transform var(--dur-fast) var(--ease);
}
.btn-secondary:hover:not(:disabled) { border-color: var(--wv-lilac); color: var(--wv-starlight); transform: var(--lift-1); }
.btn-secondary:disabled { opacity: .45; cursor: not-allowed; }
.btn-secondary:focus-visible { outline: 2px solid var(--focus-ring); outline-offset: 2px; }
/* ── button that reads as a link (inline retry etc.) ────────────────────────── */
.linklike {
background: none;
border: none;
padding: 0;
font: inherit;
color: var(--wv-lilac);
text-decoration: underline;
cursor: pointer;
transition: color var(--dur-fast) var(--ease);
}
.linklike:hover { color: var(--wv-gold); }
/* ── data tables (import history; errortable shares the bones) ──────────────── */
.datatable, .errortable {
width: 100%;
border-collapse: collapse;
font-size: 13.5px;
}
.datatable th, .errortable th {
text-align: left;
font-family: var(--wv-font-display);
font-weight: var(--weight-medium);
font-size: 12px;
letter-spacing: .06em;
text-transform: uppercase;
color: var(--text-on-dark-mute);
padding: 8px 12px;
border-bottom: 1px solid var(--border-card);
}
.datatable td, .errortable td {
padding: 11px 12px;
border-bottom: 1px solid var(--border-soft);
color: var(--text-on-dark-soft);
}
.datatable__rowlink { cursor: pointer; transition: background var(--dur-fast) var(--ease); }
.datatable__rowlink:hover { background: var(--wv-lilac-08); }
.errortable td:last-child { color: var(--st-error); }
/* ── summary tiles (preview, Task 13) ───────────────────────────────────────── */
.tiles { display: grid; grid-template-columns: repeat(4, 1fr); gap: 14px; }
.tile {
background: var(--surface-raised);
border: 1px solid var(--border-card);
border-radius: var(--radius-panel);
padding: 16px 18px;
display: flex;
flex-direction: column;
gap: 4px;
align-items: flex-start;
font: inherit;
color: inherit;
text-align: left;
cursor: pointer;
transition: background var(--dur-fast) var(--ease), border-color var(--dur-fast) var(--ease);
}
.tile:hover { background: var(--surface-raised-hi); }
.tile--active { border-color: var(--wv-gold); }
.tile__num {
font-family: var(--wv-font-display);
font-weight: var(--weight-bold);
font-size: 26px;
line-height: 1;
color: var(--text-on-dark-soft);
}
.tile__label { font-size: 12.5px; color: var(--text-on-dark-mute); }
.tile--add .tile__num { color: var(--st-add); }
.tile--update .tile__num { color: var(--st-update); }
.tile--error .tile__num { color: var(--st-error); }
/* ── diff list (preview records, Task 13) ───────────────────────────────────── */
.difflist { list-style: none; margin: 0; padding: 0; }
.difflist__item { padding: 12px 4px; border-bottom: 1px solid var(--border-soft); }
.difflist__item > summary {
cursor: pointer;
font-size: 14px;
color: var(--text-on-dark-soft);
transition: color var(--dur-fast) var(--ease);
}
.difflist__item > summary:hover { color: var(--wv-starlight); }
.difflist__item[open] > summary { margin-bottom: 8px; }
.difflist__handle { font-family: ui-monospace, "SF Mono", Menlo, monospace; font-size: 12.5px; }
.diffchange {
font-family: ui-monospace, "SF Mono", Menlo, monospace;
font-size: 12.5px;
line-height: 1.6;
color: var(--text-on-dark-soft);
}
.diffchange--head { color: var(--text-on-dark-mute); margin-top: 6px; }
.diffchange__glyph--add { color: var(--st-add); }
.diffchange__glyph--del { color: var(--st-error); }
/* ── kind chip (preview record summaries, Task 13) ──────────────────────────── */
.kindchip {
display: inline-block;
font-family: var(--wv-font-display);
font-weight: var(--weight-medium);
font-size: 11px;
letter-spacing: .06em;
text-transform: uppercase;
line-height: 1;
padding: 3px 9px 2px;
border-radius: var(--radius-pill);
border: 1px solid var(--border-strong);
color: var(--text-on-dark-mute);
}
.kindchip--add { color: var(--st-add); border-color: var(--st-add); }
.kindchip--update { color: var(--st-update); border-color: var(--st-update); }
.kindchip--error { color: var(--st-error); border-color: var(--st-error); }
/* preview layout rhythm: tiles + errortable sit between header and difflist */
.products .tiles { margin: 24px 0 18px; }
.products .errortable { margin: 0 0 18px; }
/* ── sticky confirm/cancel footer (preview, Task 13) ────────────────────────── */
.sticky-footer {
position: sticky;
bottom: 0;
display: flex;
gap: 12px;
align-items: center;
padding: 14px 0;
border-top: 1px solid var(--border-soft);
background: var(--glass-sky);
backdrop-filter: blur(var(--glass-blur));
}
/* ── upload dropzone (Task 12) ──────────────────────────────────────────────── */
.dropzone {
position: relative;
display: flex;
flex-direction: column;
align-items: center;
justify-content: center;
gap: 10px;
text-align: center;
padding: 48px 24px;
border: 2px dashed var(--border-strong);
border-radius: var(--radius-card);
cursor: pointer;
transition: border-color var(--dur-fast) var(--ease), background var(--dur-fast) var(--ease);
}
.dropzone:hover { border-color: var(--wv-lilac); background: var(--wv-lilac-08); }
.dropzone--busy { opacity: .55; pointer-events: none; }
.dropzone input[type="file"] {
position: absolute;
width: 1px;
height: 1px;
overflow: hidden;
clip: rect(0 0 0 0);
white-space: nowrap;
}
.dropzone__title {
display: block;
font-family: var(--wv-font-display);
font-weight: var(--weight-medium);
font-size: 17px;
color: var(--text-on-dark-soft);
margin-bottom: 2px;
}
/* ── small screens ──────────────────────────────────────────────────────────── */
@media (max-width: 720px) {
.adminnav { padding: 0 20px; }
.tiles { grid-template-columns: repeat(2, 1fr); }
.products__header { flex-wrap: wrap; }
}
Executable
+11
View File
@@ -0,0 +1,11 @@
#!/usr/bin/env bash
# E2E browser gate (SD-0002 §6.8) — Playwright against a fresh local stack.
# Not yet part of check.sh/CI: the CI runner has no browsers (§10.6 gap).
set -euo pipefail
repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
cd "$repo_root/e2e"
if [ ! -d node_modules ]; then
npm install
npx playwright install chromium
fi
npx playwright test "$@"