wiggleverse-ecomm/docs/OPERATIONS.md

Operating ecomm

The framework-repo operator guide (SD-0002 DOC-1), started at SLICE-5. It covers the app's operational surface — telemetry, runbooks, alert gestures, the E2E gate. Per-deployment mechanics (deploy, secrets, VM access) live in the deployment's flotilla docs and deployment.toml; environment bring-up is in BOOTSTRAP.md.

Products import/export ops (SD-0002, SLICE-5)

The §6.4 surface: a merchant uploads a catalog CSV (POST /api/products/imports), the app validates it and stores an import draft with a full preview (adds / updates / unchanged / errors), the merchant confirms or cancels at the preview gate, and a confirm applies the previewed diff as one import run recorded in the history (/api/products/imports/runs).

Caps and behavior to know (all enforced in code, not config):

• Caps (INV-18): ≤ 5,000 data rows and ≤ 10 MB per file — MAX_DATA_ROWS / MAX_FILE_BYTES in backend/app/domains/products/models.py, enforced in backend/app/domains/products/codec.py (file-level rejection) plus a 413 file_too_large guard in the BFF (backend/app/main.py).
• Draft expiry: ~1 hour (expires_at = now() + interval '1 hour'). Cleanup is a lazy sweep — expired drafts are deleted on the next upload and on any access to an expired draft; there is no background job to babysit.
• Upsert-only (INV-10): an import adds and updates, never deletes. Catalog products/variants/images absent from the file are untouched.
• One-transaction apply (INV-11): a confirm applies the whole previewed diff in a single DB transaction — it lands completely or not at all.

Telemetry

Structured JSON events on the ecomm.telemetry logger (backend/app/platform/telemetry.py), one JSON object per line, emitted from backend/app/domains/products/service.py. The app's ecomm.* log handler writes to the process's stderr, which journald captures on the VM (and Cloud Logging where the agent ships it). Events carry counts and durations only — never file names, URLs, catalog content, or secret bytes.

Event	Trigger	Payload fields
TEL-1 import_draft_created	validation completes, draft stored	storefront_id, dialect, row_count, adds, updates, unchanged, errors, unknown_columns_count, duration_ms
TEL-2 import_run_completed	apply transaction commits	run_id, storefront_id, added, updated, errored, duration_ms
TEL-3 catalog_exported	export stream completes	storefront_id, status_filter, product_count, duration_ms
TEL-6 import_apply_failed	apply transaction aborts unexpectedly	draft_id, storefront_id, error_class
TEL-4 image_phase_completed	run's fetch task finishes	run_id, fetched, rejected, failed, duration_ms
TEL-5 image_phase_recovered	startup recovery resumes a run	run_id, pending_resumed

Export (PUC-9, SLICE-6)

GET /api/products/export?status=all|active|draft|archived streams the storefront's catalog as a canonical-format CSV (one codec, two directions — the same format the importer parses). It is read-only (no draft, no run) and storefront-scoped (INV-14). An empty catalog — no products, or none matching the status filter — returns 409 empty_catalog; the Products page disables the Export action with a note in that case. The round-trip is lossless (INV-12): re-importing an unmodified export previews as all-unchanged with the import action disabled (PUC-10). TEL-3 (catalog_exported) is emitted once the stream completes — counts and duration only, never catalog content.

Images (SLICE-7)

After a merchant confirms an import, the app transitions the run to fetching_images and processes image URLs via a bounded thread pool (4 workers). Each image goes through the SSRF guard, is decoded, and up to four WebP renditions (original, thumb, card, detail) are written to the objectstore under product-images/. Progress is tracked per run in image_progress / image_counts / image_outcomes and visible in the import history detail panel (PUC-8). TEL-4 (image_phase_completed) fires when the phase finishes; startup recovery emits TEL-5 (image_phase_recovered) when it resumes a stuck run.

provision-bucket — one-time gesture per environment (SLICE-7 PPE deploy)

Before the first SLICE-7 deploy, run the provision-bucket skill from the engineering launch-app suite. This gesture:

1. Creates the GCS bucket wiggleverse-ecomm-ppe-media in the wiggleverse-ecomm GCP project.
2. Grants the PPE VM's service account roles/storage.objectAdmin on the bucket.
3. Sets an import-drafts/ lifecycle rule (forward-compat for the draft-blob migration deferred past SLICE-7).

After provisioning, deployment.toml's [overlay] already carries:

ECOMM_OBJECTSTORE_KIND   = "gcs"
ECOMM_OBJECTSTORE_BUCKET = "wiggleverse-ecomm-ppe-media"

so the next flotilla deploy picks them up automatically. GCS auth is the VM service-account ADC — no credential bytes needed.

RB-3 — image phase stuck

Triggered by ALR-3 (run in fetching_images > 2 h, or the same run recovered ≥ 3×).

1. Identify the stuck run. The import history (PUC-8, GET /api/products/imports/runs) lists runs by status; look for a run with status="fetching_images" that hasn't advanced.

   journalctl -u ecomm.service | grep image_phase
   

2. Restart to trigger recovery. A process restart causes the startup recovery scan to re-enqueue pending images for any run still in fetching_images (TEL-5 confirms the resumption). Restarts are safe — per-image claims are idempotent; already-fetched images are skipped.

   systemctl restart ecomm.service
   

3. If a specific image URL is the cause. The image_outcomes field on the run records per-image rejection reasons (SSRF block, resolution rejection, network error, etc.). The merchant can update or remove the offending URL and re-import.

4. File a bug on wiggleverse/wiggleverse-ecomm with the run_id and the image_outcomes content from the log.

ALR-3 — the log-based alert for stuck image phases

Run once per environment, at the SLICE-7 PPE deploy. Mirrors ALR-2's approach — a log metric over the stuck/recovered signal, the existing email channel, and an alert policy in project wiggleverse-ecomm.

# Select the deployment's gcloud config for this one process (handbook §8.4).
export CLOUDSDK_ACTIVE_CONFIG_NAME=wiggleverse-ecomm

# Log-based metric counting image-phase events (TEL-4/TEL-5 — stuck/recovered).
gcloud logging metrics create ecomm_image_phase_events \
  --description="ecomm TEL-4/TEL-5 image phase completed/recovered (SD-0002 ALR-3)" \
  --log-filter='resource.type="gce_instance" AND (jsonPayload.message:"image_phase_completed" OR jsonPayload.message:"image_phase_recovered" OR textPayload:"image_phase_completed" OR textPayload:"image_phase_recovered")'

Then attach an alert policy — operator email channel, threshold any event > 0 within a 2-hour window, severity notify-only. List channels first:

# Find the operator email channel's id for the policy.
gcloud beta monitoring channels list

Create the alert policy (Cloud Console or gcloud alpha monitoring policies create); condition: fetching_images duration > 2 h or the same run_id appears in TEL-5 ≥ 3 times.

RB-2 — import apply failed

Triggered by ALR-2 (any TEL-6 event). The apply raised mid-transaction and rolled back.

1. Locate the failure. On the VM, filter the journal for the event and note the draft_id, storefront_id, and error_class:

   journalctl -u ecomm.service | grep import_apply_failed
   

2. Confirm the rollback held (INV-11). The apply is one transaction, so a failure leaves the catalog exactly as it was: the storefront's runs history (GET /api/products/imports/runs) shows no new run, and the products summary (GET /api/products/summary) shows an unchanged product_count.

3. The merchant's draft is intact. A failed apply does not consume the draft — the merchant can retry confirm, or re-upload if the draft has since expired (~1 h). Advise accordingly.

4. File a bug on wiggleverse/wiggleverse-ecomm with the error_class and the surrounding log context (the traceback is in the app log next to the event).

ALR-2 — the log-based alert (one gesture per environment)

Run once per environment, at this slice's PPE deploy. This is an ad hoc op on the existing GCP project (wiggleverse-ecomm), not a provisioning gesture.

# Select the deployment's gcloud config for this one process (handbook §8.4).
export CLOUDSDK_ACTIVE_CONFIG_NAME=wiggleverse-ecomm

# Log-based metric counting import-apply failures (TEL-6).
gcloud logging metrics create ecomm_import_apply_failed \
  --description="ecomm TEL-6 import_apply_failed events (SD-0002 ALR-2)" \
  --log-filter='resource.type="gce_instance" AND jsonPayload.message:"import_apply_failed" OR textPayload:"import_apply_failed"'

Then attach an alert policy to the metric — operator email channel, threshold any event > 0 in 5 minutes, severity notify-only (pre-v1: no paging). The policy is created in the Cloud Console or with gcloud alpha monitoring policies create; the exact command depends on the notification-channel id, so list channels first:

# Find the operator email channel's id for the policy.
gcloud beta monitoring channels list

E2E browser suite

• Lives at e2e/ — Playwright, Chromium, six scenarios (SLICE-5: preview/confirm happy path, actionable errors, file rejection, cancel; SLICE-6: e2e_export_download, e2e_roundtrip_noop).
• Run with bash scripts/e2e.sh. The harness boots a fresh ecomm_e2e database against the local compose Postgres and serves the built SPA from the backend on :8765 (the deployed topology), so it needs the dev Postgres up (scripts/dev.sh).
• Not in scripts/check.sh / CI yet — the Gitea runner has no browsers (the §10.6 machinery gap). Run it locally before merge, and against PPE per the §9 pipeline (the PPE browser run is still manual this slice).

Cross-references

• SLO and alert definitions: SD-0002 §9–§10 (content repo, wiggleverse-ecomm-content/specs/SD-0002-products-bulk-csv-import-export.md).
• Import/diff engine internals: products-domain.md.