wiggleverse-ecomm/docs/OPERATIONS.md

# Operating ecomm

The framework-repo operator guide (SD-0002 DOC-1), started at SLICE-5. It covers
the app's operational surface — telemetry, runbooks, alert gestures, the E2E
gate. Per-deployment mechanics (deploy, secrets, VM access) live in the
deployment's flotilla docs and `deployment.toml`; environment bring-up is in
[`BOOTSTRAP.md`](./BOOTSTRAP.md).

## Products import/export ops (SD-0002, SLICE-5)

The §6.4 surface: a merchant uploads a catalog CSV (`POST
/api/products/imports`), the app validates it and stores an **import draft**
with a full preview (adds / updates / unchanged / errors), the merchant
confirms or cancels at the preview gate, and a confirm applies the previewed
diff as one **import run** recorded in the history (`/api/products/imports/runs`).

Caps and behavior to know (all enforced in code, not config):

- **Caps (INV-18):** ≤ 5,000 data rows and ≤ 10 MB per file —
  `MAX_DATA_ROWS` / `MAX_FILE_BYTES` in `backend/app/domains/products/models.py`,
  enforced in `backend/app/domains/products/codec.py` (file-level rejection)
  plus a 413 `file_too_large` guard in the BFF (`backend/app/main.py`).
- **Draft expiry:** ~1 hour (`expires_at = now() + interval '1 hour'`). Cleanup
  is a lazy sweep — expired drafts are deleted on the next upload and on any
  access to an expired draft; there is no background job to babysit.
- **Upsert-only (INV-10):** an import adds and updates, never deletes. Catalog
  products/variants/images absent from the file are untouched.
- **One-transaction apply (INV-11):** a confirm applies the whole previewed
  diff in a single DB transaction — it lands completely or not at all.

### Telemetry

Structured JSON events on the `ecomm.telemetry` logger
(`backend/app/platform/telemetry.py`), one JSON object per line, emitted from
`backend/app/domains/products/service.py`. The app's `ecomm.*` log handler
writes to the process's stderr, which journald captures on the VM (and Cloud
Logging where the agent ships it). Events carry counts and durations only —
never file names, URLs, catalog content, or secret bytes.

| Event | Trigger | Payload fields |
| --- | --- | --- |
| TEL-1 `import_draft_created` | validation completes, draft stored | `storefront_id, dialect, row_count, adds, updates, unchanged, errors, unknown_columns_count, duration_ms` |
| TEL-2 `import_run_completed` | apply transaction commits | `run_id, storefront_id, added, updated, errored, duration_ms` |
| TEL-3 `catalog_exported` | export stream completes | `storefront_id, status_filter, product_count, duration_ms` |
| TEL-6 `import_apply_failed` | apply transaction aborts unexpectedly | `draft_id, storefront_id, error_class` |
| TEL-4 `image_phase_completed` | run's fetch task finishes | `run_id, fetched, rejected, failed, duration_ms` |
| TEL-5 `image_phase_recovered` | startup recovery resumes a run | `run_id, pending_resumed` |

### Export (PUC-9, SLICE-6)

`GET /api/products/export?status=all|active|draft|archived` streams the
storefront's catalog as a canonical-format CSV (one codec, two directions — the
same format the importer parses). It is **read-only** (no draft, no run) and
storefront-scoped (INV-14). An empty catalog — no products, or none matching the
status filter — returns `409 empty_catalog`; the Products page disables the
Export action with a note in that case. The round-trip is lossless (INV-12):
re-importing an unmodified export previews as all-unchanged with the import
action disabled (PUC-10). TEL-3 (`catalog_exported`) is emitted once the stream
completes — counts and duration only, never catalog content.

### Images (SLICE-7)

After a merchant confirms an import, the app transitions the run to
`fetching_images` and processes image URLs via a bounded thread pool
(4 workers). Each image goes through the SSRF guard, is decoded, and up to
four WebP renditions (`original`, `thumb`, `card`, `detail`) are written to
the objectstore under `product-images/`. Progress is tracked per run in
`image_progress` / `image_counts` / `image_outcomes` and visible in the
import history detail panel (PUC-8). TEL-4 (`image_phase_completed`) fires
when the phase finishes; startup recovery emits TEL-5 (`image_phase_recovered`)
when it resumes a stuck run.

#### `provision-bucket` — one-time gesture per environment (SLICE-7 PPE deploy)

Before the first SLICE-7 deploy, run the `provision-bucket` skill from the
engineering `launch-app` suite. This gesture:

1. Creates the GCS bucket `wiggleverse-ecomm-ppe-media` in the
   `wiggleverse-ecomm` GCP project.
2. Grants the PPE VM's service account `roles/storage.objectAdmin` on the
   bucket.
3. Sets an `import-drafts/` lifecycle rule (forward-compat for the draft-blob
   migration deferred past SLICE-7).

After provisioning, `deployment.toml`'s `[overlay]` already carries:

```
ECOMM_OBJECTSTORE_KIND   = "gcs"
ECOMM_OBJECTSTORE_BUCKET = "wiggleverse-ecomm-ppe-media"
```

so the next `flotilla deploy` picks them up automatically. GCS auth is the
VM service-account ADC — no credential bytes needed.

### RB-3 — image phase stuck

Triggered by ALR-3 (run in `fetching_images` > 2 h, or the same run recovered
≥ 3×).

1. **Identify the stuck run.** The import history (PUC-8, `GET
   /api/products/imports/runs`) lists runs by status; look for a run with
   `status="fetching_images"` that hasn't advanced.

   ```
   journalctl -u ecomm.service | grep image_phase
   ```

2. **Restart to trigger recovery.** A process restart causes the startup
   recovery scan to re-enqueue pending images for any run still in
   `fetching_images` (TEL-5 confirms the resumption). Restarts are safe —
   per-image claims are idempotent; already-fetched images are skipped.

   ```
   systemctl restart ecomm.service
   ```

3. **If a specific image URL is the cause.** The `image_outcomes` field on
   the run records per-image rejection reasons (SSRF block, resolution
   rejection, network error, etc.). The merchant can update or remove the
   offending URL and re-import.

4. **File a bug** on `wiggleverse/wiggleverse-ecomm` with the `run_id` and
   the `image_outcomes` content from the log.

### ALR-3 — the log-based alert for stuck image phases

Run once per environment, at the SLICE-7 PPE deploy. Mirrors ALR-2's
approach — a log metric over the stuck/recovered signal, the existing email
channel, and an alert policy in project `wiggleverse-ecomm`.

```
# Select the deployment's gcloud config for this one process (handbook §8.4).
export CLOUDSDK_ACTIVE_CONFIG_NAME=wiggleverse-ecomm

# Log-based metric counting image-phase events (TEL-4/TEL-5 — stuck/recovered).
gcloud logging metrics create ecomm_image_phase_events \
  --description="ecomm TEL-4/TEL-5 image phase completed/recovered (SD-0002 ALR-3)" \
  --log-filter='resource.type="gce_instance" AND (jsonPayload.message:"image_phase_completed" OR jsonPayload.message:"image_phase_recovered" OR textPayload:"image_phase_completed" OR textPayload:"image_phase_recovered")'
```

Then attach an alert policy — **operator email channel, threshold any event >
0 within a 2-hour window, severity notify-only**. List channels first:

```
# Find the operator email channel's id for the policy.
gcloud beta monitoring channels list
```

Create the alert policy (Cloud Console or `gcloud alpha monitoring policies
create`); condition: `fetching_images` duration > 2 h **or** the same
`run_id` appears in TEL-5 ≥ 3 times.

### RB-2 — import apply failed

Triggered by ALR-2 (any TEL-6 event). The apply raised mid-transaction and
rolled back.

1. **Locate the failure.** On the VM, filter the journal for the event and note
   the `draft_id`, `storefront_id`, and `error_class`:

   ```
   journalctl -u ecomm.service | grep import_apply_failed
   ```

2. **Confirm the rollback held (INV-11).** The apply is one transaction, so a
   failure leaves the catalog exactly as it was: the storefront's runs history
   (`GET /api/products/imports/runs`) shows **no new run**, and the products
   summary (`GET /api/products/summary`) shows an unchanged `product_count`.
3. **The merchant's draft is intact.** A failed apply does not consume the
   draft — the merchant can retry confirm, or re-upload if the draft has since
   expired (~1 h). Advise accordingly.
4. **File a bug** on `wiggleverse/wiggleverse-ecomm` with the `error_class`
   and the surrounding log context (the traceback is in the app log next to
   the event).

### ALR-2 — the log-based alert (one gesture per environment)

Run once per environment, at this slice's PPE deploy. This is an ad hoc op on
the existing GCP project (`wiggleverse-ecomm`), not a provisioning gesture.

```
# Select the deployment's gcloud config for this one process (handbook §8.4).
export CLOUDSDK_ACTIVE_CONFIG_NAME=wiggleverse-ecomm

# Log-based metric counting import-apply failures (TEL-6).
gcloud logging metrics create ecomm_import_apply_failed \
  --description="ecomm TEL-6 import_apply_failed events (SD-0002 ALR-2)" \
  --log-filter='resource.type="gce_instance" AND jsonPayload.message:"import_apply_failed" OR textPayload:"import_apply_failed"'
```

Then attach an alert policy to the metric — **operator email channel, threshold
any event > 0 in 5 minutes, severity notify-only** (pre-v1: no paging). The
policy is created in the Cloud Console or with
`gcloud alpha monitoring policies create`; the exact command depends on the
notification-channel id, so list channels first:

```
# Find the operator email channel's id for the policy.
gcloud beta monitoring channels list
```

### E2E browser suite

- Lives at `e2e/` — Playwright, Chromium, six scenarios (SLICE-5:
  preview/confirm happy path, actionable errors, file rejection, cancel;
  SLICE-6: `e2e_export_download`, `e2e_roundtrip_noop`).
- Run with `bash scripts/e2e.sh`. The harness boots a **fresh `ecomm_e2e`
  database** against the local compose Postgres and serves the built SPA from
  the backend on **:8765** (the deployed topology), so it needs the dev
  Postgres up (`scripts/dev.sh`).
- **Not in `scripts/check.sh` / CI yet** — the Gitea runner has no browsers
  (the §10.6 machinery gap). Run it locally before merge, and against PPE per
  the §9 pipeline (the PPE browser run is still manual this slice).

## Cross-references

- SLO and alert definitions: SD-0002 §9–§10 (content repo,
  `wiggleverse-ecomm-content/specs/SD-0002-products-bulk-csv-import-export.md`).
- Import/diff engine internals: [`products-domain.md`](./products-domain.md).