Gap 3: No Hallucination Detection

Problem

AgentRun.transcript stores the full execution log of every tool call and LLM turn, but it is never analysed. An agent calling search_knowledge.js with the same query five times in a row, getting the same result each time, is not detected. The run completes, the bloated repetitive content goes to DM review, and the DM has no signal that the agent struggled.

From the Reflexion framework (Shinn et al., 2023), two heuristics reliably detect agent failure:

Inefficiency: the agent takes too many steps without making progress toward a final output
Hallucination: the agent repeats the same action with identical arguments and receives identical observations

Both are detectable from the transcript without any additional LLM call.

Concrete example

The blog-writer for a thin-content tenant calls search_knowledge.js "brand voice guidelines" at step 2, gets back 3 results, then calls it again at step 5 with the same query, gets the same 3 results, then calls it again at step 9. The final blog post heavily paraphrases those 3 chunks. AgentRun.toolsUsed = 3 — looks normal. No flag is raised.

What to Build

1. Transcript analyser function

After every run completes, parse the transcript to detect hallucination signals:


interface TranscriptEvent {
  type: "tool_use" | "tool_result" | "text" | "thinking";
  content: string;
  metadata?: { toolName?: string; args?: string; result?: string };
  timestamp: string;
}
 
function detectHallucination(transcript: TranscriptEvent[]): {
  detected: boolean;
  reason?: "repeated_tool_call" | "excessive_steps" | "no_progress";
  details?: string;
} {
  const toolCalls = transcript.filter(e => e.type === "tool_use");
 
  // Check for repeated identical tool calls
  const seen = new Map<string, number>();
  for (const call of toolCalls) {
    const key = `${call.metadata?.toolName}::${call.metadata?.args}`;
    const count = (seen.get(key) ?? 0) + 1;
    seen.set(key, count);
    if (count >= 3) {
      return {
        detected: true,
        reason: "repeated_tool_call",
        details: `Called "${call.metadata?.toolName}" with identical args ${count} times`,
      };
    }
  }
 
  // Check for excessive steps with no substantial output
  if (toolCalls.length > 15) {
    return { detected: true, reason: "excessive_steps", details: `${toolCalls.length} tool calls` };
  }
 
  return { detected: false };
}

2. Add hallucinationDetected field to AgentRun


model AgentRun {
  // ... existing fields
  hallucinationDetected Boolean  @default(false)
  hallucinationReason   String?
}

3. Route flagged runs to DM review with warning

In setup.worker.ts and blog-writer.worker.ts, after saving the run result:


const hallucinationCheck = detectHallucination(result.transcript);
 
if (hallucinationCheck.detected) {
  await db.agentRun.update({
    where: { id: agentRunId },
    data: {
      hallucinationDetected: true,
      hallucinationReason: hallucinationCheck.details,
    },
  });
 
  // Flag the content for DM — surface a warning badge in the review UI
  await db.blogPost.update({
    where: { id: blogPostId },
    data: { qualityFlags: { push: "hallucination_suspected" } },
  });
}

The DM portal review page shows a warning badge on flagged content so reviewers know to scrutinise it more carefully.

4. Admin visibility

Surface hallucinationDetected in the manage portal’s Agent Runs table. This lets admins identify which agent roles are hallucination-prone and tune their prompts or skill sets.

Files to Change

New file: packages/agents/src/lib/hallucination-detector.ts
packages/agents/src/workers/blog-writer.worker.ts — call detector after run completion
packages/agents/src/workers/setup.worker.ts — call detector for context-file-writer step
packages/db/prisma/schema.prisma — add hallucinationDetected + hallucinationReason to AgentRun
apps/dashboard/src/app/(dashboard)/... DM review UI — surface warning badge

Gap 6: Critic agent quality gate (complementary — critic catches content quality issues; hallucination detector catches execution issues)
Gap 12: Tool usage analytics (detailed tool call logs enable better detection)