CataSync feeds 3/6: engine named feeds, field repeater, presets, XML/CSV builder
Imported from GitHub issue [El-SaMa/oma#246](https://github.com/El-SaMa/oma/issues/246) by @El-SaMa.
---
Part of the Product feeds epic. Needs "Shop catalog snapshot" live.
## What
Any number of **named feeds per shop**, each fully customizable (fields repeater, filters, format). Example set on niletech: `hinta.fi`, `Hintaseuranta`, `Hintaopas (Prisjakt)`, `Google Merchant`. Feeds can be created from a preset, duplicated, edited, paused and deleted.
## Data model
Migration `ops/catasync/engine/migrations/016_feeds.sql`:
```sql
CREATE TABLE IF NOT EXISTS feeds (
id CHAR(36) PRIMARY KEY,
site_id CHAR(36) NOT NULL,
name VARCHAR(80) NOT NULL,
destination ENUM('url','google') NOT NULL DEFAULT 'url',
format ENUM('xml','csv') NOT NULL DEFAULT 'xml',
preset VARCHAR(40) NULL,
token_hash CHAR(64) NOT NULL, -- sha256 of the public token; the token itself is shown once
enabled TINYINT(1) NOT NULL DEFAULT 1,
settings JSON NOT NULL,
fields JSON NOT NULL, -- ordered repeater rows
filters JSON NOT NULL,
built_at DATETIME NULL, item_count INT NULL, build_ms INT NULL, last_error VARCHAR(300) NULL,
created_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
updated_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
UNIQUE KEY (site_id, name)
);
```
- `settings` XML: `{ rootTag, itemTag, namespaces: {prefix: uri}, wrapper: "rss" | "plain" }`. CSV: `{ delimiter: "," | ";" | "\t", header: true }`. Both: `backorderDays` (default **24**), `idPrefix` (default empty), `includeVariations` (default true), `shipping: { "wave-products": 9.90, "default": 6.21, "freeFrom": 100, "freeExcludeClasses": ["wave-products"] }` (editable per feed).
- `fields` row: `{ name, kind: "source" | "text" | "template", source?, value?, transform?: "none" | "strip_html" | "truncate" | "upper" | "lower" | "price_number" | "price_with_currency" | "yes_no" | "date_iso", maxLength?, required: boolean, cdata?: boolean }`. Max 60 rows. `name` may carry a namespace prefix (`g:price`) for XML; for CSV it is the column header. When `source` is `gallery` the row repeats once per image (for `g:additional_image_link`).
- `filters`: `{ stock: ["instock","onbackorder"], requireImage: true, requireDescription: true, minPrice?, maxPrice?, categoriesInclude?: string[], categoriesExclude?: string[], shippingClassesExclude?: string[], catasyncOnly: false, readyOnly: false }`.
## Sources available to rows
- From `shop_items.data`: every connector field (`id, sku, ean, mpn, name, short_description, description, permalink, image, gallery, regular_price, sale_price, price, currency, stock_status, stock_quantity, weight, shipping_class, categories, brand, energy_class, lead_text, item_group_id`) plus `category_path_last`.
- Computed:
- `availability` (Google words `in_stock` / `out_of_stock` / `backorder`), `availability_fi` (`Varastossa` / `Tilaustuote` / `Loppu`)
- `availability_date`: today + `backorderDays` for backorder (ISO 8601), else empty. Computed at build time, never stored.
- `delivery_days`: instock `1-3` (no class, f9-products, also-products), `5-10` (wave-products); backorder `14-24`
- `shipping_price`: from `settings.shipping` by shipping class; 0 when price >= `freeFrom` and the class is not in `freeExcludeClasses`
- `condition` (`new`), `supplier` (CataSync offer supplier or `own`), `id_with_prefix`
- Templates use `{source}` placeholders, e.g. `{brand} {name}`.
## Builder
- New module `ops/catasync/engine/src/feedsout.mjs` (not the supplier `feeds.mjs`): `buildFeed(feedId) -> { contentType, body, count, skipped: { reason: n } }`. Streams rows from `shop_items` for the site, applies filters, rows and transforms. XML: proper escaping (or CDATA when the row says so), UTF-8, invalid control chars removed. Items missing a `required` field are skipped and counted.
- Cache the built body on disk at `/var/lib/catasync/feeds/<feedId>.xml|csv` (create the dir in `install.sh`, owned by the service user). Rebuild after each catalog sync that changed the site's data, and on feed edit. Serve the cached file.
- Presets in `ops/catasync/engine/src/feedpresets.mjs`, all editable after creation:
- `google_xml`: RSS 2.0 with `xmlns:g="http://base.google.com/ns/1.0"`, rows `g:id, title, description, link, g:image_link, g:additional_image_link (gallery), g:availability, g:availability_date, g:price, g:sale_price, g:brand, g:gtin, g:mpn, g:condition, g:item_group_id, g:product_type, g:shipping_weight, g:shipping (country FI + price)`; filters require image and description.
- `hinta_fi`, `hintaseuranta`, `hintaopas_prisjakt`: start from the Google XML layout plus `delivery_days` and `shipping_price` rows. These sites generally accept Google-format feeds, but **check each site's own merchant feed spec before go-live** (merchant pages / ask their support) and adjust the preset; note the result in the handbook.
- `csv_basic`: `id;name;price;url;image;ean;brand;availability;delivery_days;shipping_price`, delimiter `;`.
- `empty`: no rows.
- Routes in `server.mjs`:
- `GET /v1/sites/:id/feeds`
- `POST /v1/sites/:id/feeds` (create from preset or a body; returns the **token once**)
- `POST /v1/feeds/:id` (update), `POST /v1/feeds/:id/duplicate` (new name, new token once), `POST /v1/feeds/:id/rotate-token` (token once), `DELETE /v1/feeds/:id`, `POST /v1/feeds/:id/rebuild`
- `GET /v1/feeds/:id/preview?limit=5` (rendered sample + skipped counts)
- `GET /v1/feeds/by-token/:token` (sha256 lookup, enabled feeds only, returns the cached file and content type; 404 otherwise)
- Error codes: `feed_name_taken`, `feed_fields_invalid`, `feed_too_many_fields`, `feed_not_found`, `feed_disabled`, `feed_preset_unknown`. Add them to `KNOWN_ERRORS` in `services/core-api/src/catasync/engine.ts` in issue 4.
- Tests (pure functions): XML escaping, CDATA, gallery repeat, templates, transforms, filters, required skip, CSV quoting, availability date.
## Acceptance
- [ ] Engine tests pass (`node --test test/*.test.mjs`); engine rolled.
- [ ] On maxfort: one feed from each preset; preview rows correct; XML passes `xmllint --noout`; CSV opens in a spreadsheet.
- [ ] Handbook section "Product feeds" (model, presets, sources list, filters, tokens).
## Binding rules (read before coding)
- Read `CLAUDE.md` and every `.cursor/rules/*.mdc` with `alwaysApply: true` first. They win over your defaults.
- **No OpenAI / AI calls anywhere in this feature.** CataSync AI switch (`ai_enabled`) stays off. Never turn it on.
- Smallest correct change. Match the surrounding code style (engine = plain ESM `.mjs`, mysql2 pool from `src/db.mjs`; core-api = TypeScript).
- Every staff action goes through the admin MCP (`admin-mcp-first.mdc`): executor in `services/core-api/src/agent-tools/tools-catasync.ts`, confirm handler in `services/core-api/src/platform/confirm.ts` for writes, registry entry, zod schema in `services/mcp-server/src/tools.ts`, test in `tools-catasync.test.ts`. The UI calls the same tool (via `apps/web/lib/catasync/admin-actions.ts`), never a separate path.
- UI: vertical side nav only (never a second tab row), forms in modals, shadow cards, full width (`portal-chrome.mdc`).
- Every string: `next-intl` key with native **en, fi, sv** (Finnish must read native, not translated). No em/en dashes in prose. Run `node scripts/i18n-validate.mjs`.
- Handbook: update `apps/web/content/admin-docs/products/catasync.md` (new section) for every behaviour you add.
- Secrets (tokens, Google tokens, feed tokens) never in logs, commits, handbook or chat.
- Commits: author **ELSAMA** only, **no Co-authored-by / AI attribution**. Push to `main` right after each commit (rebase on `origin/main` first).
- Deploy: engine with `bash scripts/catasync-engine-roll.sh figaro`; OMA only via the train `bash scripts/deploy-quick.sh` (never set `OMA_DEPLOY_SERVICES`, never build on the VM). Check the live SHA contains your commit.
- WordPress plugin changes: bump version, pass `scripts/wp-plugin-gate-remote.sh ops/catasync/plugins/lafu-catasync-connector` with **0 errors 0 warnings** + security check (`wp-plugin-dod.mdc`). Install on **maxfort.fi first** (test shop), only then niletech.
- Never change niletech prices, statuses, shipping rows or the Code Snippets / table-rate plugins as part of this work.
- Clean up test data and temp files when done.
