keyword-researcher: brittle JSON parsing dropped output

Status: ✅ Fixed
Severity: Medium
Component: packages/agents/src/workers/keyword-researcher.worker.ts

Symptom

On the first job run (and occasionally later), the keyword researcher saved no keyword groups to the DB and left deliverables in pending_review with no data. Logs showed no error — the worker appeared to succeed.

Root Cause

The postProcess function stripped markdown fences using anchored regexes:


const cleaned = output.replace(/^```(?:json)?\n?/i, "").replace(/\n?```$/i, "").trim();

These only matched fences at the absolute start/end of the string. Claude frequently writes a brief preamble before the JSON block (e.g. “Here are the keyword groups for your campaign:\n\n```json\n{…}”):

Prose before JSON → regex misses the fence → JSON.parse fails on the prose → falls back silently, saving nothing
Fenced JSON mid-text → same failure
Trailing prose after closing brace → parse succeeds but misses trailing text (rare but possible)

The catch block saved the deliverable in pending_review without groups, so no exception was raised and the failure was invisible.

Fix

keyword-researcher.worker.ts:

Replaced the brittle regex with extractJson<AgentOutput>(output) imported from activity.worker.ts
extractJson uses brace-depth scanning to find the first complete {...} object regardless of surrounding text or fences
Added promptSuffix: OUTPUT_FORMAT_INSTRUCTION via createContentWorker options — appended to every prompt, it tells Claude to start with { and end with } with no prose or fences

content.worker.ts:

Added promptSuffix?: string to ContentWorkerOptions — appended after all context sections. Available to any content worker that needs strict output-format enforcement.

Variables

The groups array shape that must be returned:


{
  "groups": [
    {
      "name": "string",
      "purpose": "string",
      "keywords": [
        { "keyword": "string", "searchVolume": "string?", "difficulty": "string?", "intent": "string?", "category": "string?" }
      ]
    }
  ]
}