CataSync feeds 3/6: engine named feeds, field repeater, presets, XML/CSV builder

Imported from GitHub issue El-SaMa/oma#246 by @El-SaMa.

---

Part of the Product feeds epic. Needs "Shop catalog snapshot" live.

What

Any number of named feeds per shop, each fully customizable (fields repeater, filters, format). Example set on niletech: hinta.fi, Hintaseuranta, Hintaopas (Prisjakt), Google Merchant. Feeds can be created from a preset, duplicated, edited, paused and deleted.

Data model

Migration ops/catasync/engine/migrations/016_feeds.sql:

CREATE TABLE IF NOT EXISTS feeds (
  id CHAR(36) PRIMARY KEY,
  site_id CHAR(36) NOT NULL,
  name VARCHAR(80) NOT NULL,
  destination ENUM('url','google') NOT NULL DEFAULT 'url',
  format ENUM('xml','csv') NOT NULL DEFAULT 'xml',
  preset VARCHAR(40) NULL,
  token_hash CHAR(64) NOT NULL,       -- sha256 of the public token; the token itself is shown once
  enabled TINYINT(1) NOT NULL DEFAULT 1,
  settings JSON NOT NULL,
  fields JSON NOT NULL,               -- ordered repeater rows
  filters JSON NOT NULL,
  built_at DATETIME NULL, item_count INT NULL, build_ms INT NULL, last_error VARCHAR(300) NULL,
  created_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
  updated_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
  UNIQUE KEY (site_id, name)
);

• settings XML: { rootTag, itemTag, namespaces: {prefix: uri}, wrapper: "rss" | "plain" }. CSV: { delimiter: "," | ";" | "\t", header: true }. Both: backorderDays (default 24), idPrefix (default empty), includeVariations (default true), shipping: { "wave-products": 9.90, "default": 6.21, "freeFrom": 100, "freeExcludeClasses": ["wave-products"] } (editable per feed).
• fields row: { name, kind: "source" | "text" | "template", source?, value?, transform?: "none" | "strip_html" | "truncate" | "upper" | "lower" | "price_number" | "price_with_currency" | "yes_no" | "date_iso", maxLength?, required: boolean, cdata?: boolean }. Max 60 rows. name may carry a namespace prefix (g:price) for XML; for CSV it is the column header. When source is gallery the row repeats once per image (for g:additional_image_link).
• filters: { stock: ["instock","onbackorder"], requireImage: true, requireDescription: true, minPrice?, maxPrice?, categoriesInclude?: string[], categoriesExclude?: string[], shippingClassesExclude?: string[], catasyncOnly: false, readyOnly: false }.

Sources available to rows

• From shop_items.data: every connector field (id, sku, ean, mpn, name, short_description, description, permalink, image, gallery, regular_price, sale_price, price, currency, stock_status, stock_quantity, weight, shipping_class, categories, brand, energy_class, lead_text, item_group_id) plus category_path_last.
• Computed:
  • availability (Google words in_stock / out_of_stock / backorder), availability_fi (Varastossa / Tilaustuote / Loppu)
  • availability_date: today + backorderDays for backorder (ISO 8601), else empty. Computed at build time, never stored.
  • delivery_days: instock 1-3 (no class, f9-products, also-products), 5-10 (wave-products); backorder 14-24
  • shipping_price: from settings.shipping by shipping class; 0 when price >= freeFrom and the class is not in freeExcludeClasses
  • condition (new), supplier (CataSync offer supplier or own), id_with_prefix
• Templates use {source} placeholders, e.g. {brand} {name}.

Builder

• New module ops/catasync/engine/src/feedsout.mjs (not the supplier feeds.mjs): buildFeed(feedId) -> { contentType, body, count, skipped: { reason: n } }. Streams rows from shop_items for the site, applies filters, rows and transforms. XML: proper escaping (or CDATA when the row says so), UTF-8, invalid control chars removed. Items missing a required field are skipped and counted.
• Cache the built body on disk at /var/lib/catasync/feeds/<feedId>.xml|csv (create the dir in install.sh, owned by the service user). Rebuild after each catalog sync that changed the site's data, and on feed edit. Serve the cached file.
• Presets in ops/catasync/engine/src/feedpresets.mjs, all editable after creation:
  • google_xml: RSS 2.0 with xmlns:g="http://base.google.com/ns/1.0", rows g:id, title, description, link, g:image_link, g:additional_image_link (gallery), g:availability, g:availability_date, g:price, g:sale_price, g:brand, g:gtin, g:mpn, g:condition, g:item_group_id, g:product_type, g:shipping_weight, g:shipping (country FI + price); filters require image and description.
  • hinta_fi, hintaseuranta, hintaopas_prisjakt: start from the Google XML layout plus delivery_days and shipping_price rows. These sites generally accept Google-format feeds, but check each site's own merchant feed spec before go-live (merchant pages / ask their support) and adjust the preset; note the result in the handbook.
  • csv_basic: id;name;price;url;image;ean;brand;availability;delivery_days;shipping_price, delimiter ;.
  • empty: no rows.
• Routes in server.mjs:
  • GET /v1/sites/:id/feeds
  • POST /v1/sites/:id/feeds (create from preset or a body; returns the token once)
  • POST /v1/feeds/:id (update), POST /v1/feeds/:id/duplicate (new name, new token once), POST /v1/feeds/:id/rotate-token (token once), DELETE /v1/feeds/:id, POST /v1/feeds/:id/rebuild
  • GET /v1/feeds/:id/preview?limit=5 (rendered sample + skipped counts)
  • GET /v1/feeds/by-token/:token (sha256 lookup, enabled feeds only, returns the cached file and content type; 404 otherwise)
• Error codes: feed_name_taken, feed_fields_invalid, feed_too_many_fields, feed_not_found, feed_disabled, feed_preset_unknown. Add them to KNOWN_ERRORS in services/core-api/src/catasync/engine.ts in issue 4.
• Tests (pure functions): XML escaping, CDATA, gallery repeat, templates, transforms, filters, required skip, CSV quoting, availability date.

Acceptance

• Engine tests pass (node --test test/*.test.mjs); engine rolled.
• On maxfort: one feed from each preset; preview rows correct; XML passes xmllint --noout; CSV opens in a spreadsheet.
• Handbook section "Product feeds" (model, presets, sources list, filters, tokens).

Binding rules (read before coding)

• Read CLAUDE.md and every .cursor/rules/*.mdc with alwaysApply: true first. They win over your defaults.
• No OpenAI / AI calls anywhere in this feature. CataSync AI switch (ai_enabled) stays off. Never turn it on.
• Smallest correct change. Match the surrounding code style (engine = plain ESM .mjs, mysql2 pool from src/db.mjs; core-api = TypeScript).
• Every staff action goes through the admin MCP (admin-mcp-first.mdc): executor in services/core-api/src/agent-tools/tools-catasync.ts, confirm handler in services/core-api/src/platform/confirm.ts for writes, registry entry, zod schema in services/mcp-server/src/tools.ts, test in tools-catasync.test.ts. The UI calls the same tool (via apps/web/lib/catasync/admin-actions.ts), never a separate path.
• UI: vertical side nav only (never a second tab row), forms in modals, shadow cards, full width (portal-chrome.mdc).
• Every string: next-intl key with native en, fi, sv (Finnish must read native, not translated). No em/en dashes in prose. Run node scripts/i18n-validate.mjs.
• Handbook: update apps/web/content/admin-docs/products/catasync.md (new section) for every behaviour you add.
• Secrets (tokens, Google tokens, feed tokens) never in logs, commits, handbook or chat.
• Commits: author ELSAMA only, no Co-authored-by / AI attribution. Push to main right after each commit (rebase on origin/main first).
• Deploy: engine with bash scripts/catasync-engine-roll.sh figaro; OMA only via the train bash scripts/deploy-quick.sh (never set OMA_DEPLOY_SERVICES, never build on the VM). Check the live SHA contains your commit.
• WordPress plugin changes: bump version, pass scripts/wp-plugin-gate-remote.sh ops/catasync/plugins/lafu-catasync-connector with 0 errors 0 warnings + security check (wp-plugin-dod.mdc). Install on maxfort.fi first (test shop), only then niletech.
• Never change niletech prices, statuses, shipping rows or the Code Snippets / table-rate plugins as part of this work.
• Clean up test data and temp files when done.