​

Deep Research Agent

​

What You’ll Learn

• Multi-stage workflow: Memory retrieval → Refine → Decompose → Execute → Citations → Gap Filling → Synthesis → Verification
• Adaptive patterns: React, Parallel, and Hybrid execution based on complexity
• Advanced features: Entity filtering, gap-filling, context compression, react-per-task mode
• Quality controls: Citation requirements, coverage enforcement, claim verification
• Language matching: Automatically responds in the user’s query language

​

Prerequisites

• Shannon stack running (Docker Compose)
• Gateway reachable at http://localhost:8080
• Auth defaults:
  • Docker Compose: authentication is disabled by default (GATEWAY_SKIP_AUTH=1).
  • Local builds: authentication is enabled by default. Set GATEWAY_SKIP_AUTH=1 to disable auth, or include an API key header -H "X-API-Key: $API_KEY".

​

Workflow Architecture

• Memory-aware: Uses conversation history for contextual research
• Entity-focused: Filters results when entity is detected (e.g., specific company/product)
• Self-correcting: Gap filling ensures comprehensive coverage
• Quality-enforced: Per-area coverage validation (≥600 chars, ≥2 citations per section)

​

Quick Start (HTTP)

# Submit a research task (default “standard” strategy)
curl -X POST http://localhost:8080/api/v1/tasks \
  -H "Content-Type: application/json" \
  -d '{
    "query": "Latest breakthroughs in quantum error correction",
    "context": { "force_research": true }
  }'

# Submit and get a ready-to-use SSE stream URL (recommended)
# Tip: include force_research to route to ResearchWorkflow (otherwise router may select Supervisor)
curl -s -X POST http://localhost:8080/api/v1/tasks/stream \
  -H "Content-Type: application/json" \
  -d '{
        "query": "What are the main transformer architecture trends in 2025?",
        "context": { "force_research": true }
      }' | jq

​

Stream Events (SSE)

# After the /tasks/stream call, use "stream_url" from the response:
curl -N "http://localhost:8080/api/v1/stream/sse?workflow_id=task-..."

• LLM_OUTPUT: final synthesis text
• DATA_PROCESSING: progress/usage
• WORKFLOW_COMPLETED: completion

​

Execution Patterns

​

React Pattern (complexity < 0.5)

# Example: Basic conceptual query triggers React
curl -X POST http://localhost:8080/api/v1/tasks \
  -H "Content-Type: application/json" \
  -d '{
    "query": "What is quantum computing?",
    "context": { "force_research": true }
  }'

​

Parallel Pattern (no dependencies)

# Example: Multi-aspect comparison triggers Parallel
curl -X POST http://localhost:8080/api/v1/tasks \
  -H "Content-Type: application/json" \
  -d '{
    "query": "Compare LangChain and AutoGen frameworks: architecture, pricing, community, performance",
    "context": { "force_research": true }
  }'

​

Hybrid Pattern (has dependencies)

# Example: Query with logical dependencies triggers Hybrid
curl -X POST http://localhost:8080/api/v1/tasks \
  -H "Content-Type: application/json" \
  -d '{
    "query": "Analyze company X market strategy, then evaluate competitive response, then forecast next moves",
    "context": { "force_research": true }
  }'

​

React Per Task (deep research mode)

1. Manual: context.react_per_task = true
2. Auto-enable: complexity > 0.7 AND strategy in {deep, academic}

# Example: Enable react_per_task for iterative per-area research
curl -X POST http://localhost:8080/api/v1/tasks \
  -H "Content-Type: application/json" \
  -d '{
    "query": "Comprehensive analysis of transformer architecture evolution 2017-2025",
    "context": {
      "force_research": true,
      "react_per_task": true,
      "react_max_iterations": 6
    },
    "research_strategy": "deep"
  }'

​

Strategy Presets

# Quick strategy (minimal iterations)
curl -X POST http://localhost:8080/api/v1/tasks \
  -H "Content-Type: application/json" \
  -d '{
    "query": "What is quantum error correction?",
    "context": { "force_research": true },
    "research_strategy": "quick"
  }'

# Deep strategy with overrides
curl -X POST http://localhost:8080/api/v1/tasks \
  -H "Content-Type: application/json" \
  -d '{
    "query": "Compare LangChain and AutoGen frameworks",
    "context": {
      "force_research": true,
      "react_per_task": true,
      "react_max_iterations": 6
    },
    "research_strategy": "deep",
    "enable_verification": true
  }'

# Academic strategy (auto-enables react_per_task for complexity > 0.7)
curl -X POST http://localhost:8080/api/v1/tasks \
  -H "Content-Type: application/json" \
  -d '{
    "query": "Survey of attention mechanisms in modern NLP architectures",
    "context": { "force_research": true },
    "research_strategy": "academic"
  }'

• max_concurrent_agents (1–20): Controls maximum concurrency for parallel subtasks (only applies when multiple subtasks are executed)
• react_max_iterations (2–8, default 5): Controls ReAct loop depth per task (independent parameter, no prerequisites)

• react_per_task (bool): Enable mini ReAct loops per subtask (auto-enabled for complexity > 0.7 + deep/academic)
• enable_verification (bool): Enable claim verification against citations
• budget_agent_max (int): Optional per-agent token budget with enforcement

• gap_filling_enabled (bool): Enable/disable gap filling
• gap_filling_max_gaps (1–20): Maximum number of gaps to detect
• gap_filling_max_iterations (1–5): Maximum retry iterations per gap
• gap_filling_check_citations (bool): Check citation density as gap indicator

​

Python SDK

​

CLI

python -m shannon.cli --base-url http://localhost:8080 \
  submit "Latest quantum computing breakthroughs" \
  --force-research \
  --research-strategy deep --enable-verification

​

Programmatic

from shannon import ShannonClient

client = ShannonClient(base_url="http://localhost:8080")
handle = client.submit_task(
    "Compare LangChain and AutoGen frameworks",
    context={
        "force_research": True,
        "research_strategy": "deep",
        "react_max_iterations": 6,
        "enable_verification": True,
    },
)
final = client.wait(handle.task_id)
print(final.result)
client.close()

​

Response Format

{
  "task_id": "task-00000000-0000-0000-0000-000000000000",
  "status": "TASK_STATUS_COMPLETED",
  "result": "... final synthesis text ...",
  "metadata": {
    "citations": [
      {
        "url": "https://example.com/article",
        "title": "Source Title",
        "source": "example.com",
        "quality_score": 0.92,
        "credibility_score": 0.85
      }
    ],
    "verification": {
      "total_claims": 10,
      "overall_confidence": 0.78,
      "supported_claims": 7,
      "unsupported_claims": [
        "Claim text lacking sufficient evidence ..."
      ],
      "claim_details": [
        {
          "claim": "Concrete claim text ...",
          "confidence": 0.81,
          "supporting_citations": [1, 3],
          "conflicting_citations": []
        }
      ]
    }
  },
  "created_at": "2025-11-07T12:00:00Z",
  "updated_at": "2025-11-07T12:02:00Z",
  "usage": {
    "input_tokens": 1234,
    "output_tokens": 5678,
    "total_tokens": 6912,
    "estimated_cost": 0.0234
  },
  "model_used": "gpt-5-2025-08-07",
  "provider": "openai"
}

• result is the final synthesized markdown/text.
• metadata.citations lists collected sources (all are included in the Sources section of the result).
• metadata.verification appears when verification is enabled and completed.
• In Gateway status responses, created_at / updated_at reflect response time; authoritative run timing and totals are persisted in the database (events/timeline).
• usage summarizes token usage and may include cost. Gateway status may expose estimated_cost; workflow metadata may include cost_usd.
• model_used and provider reflect the model/provider chosen during synthesis.

​

Advanced Features

​

Memory Retrieval

• Combines recent (last 5 messages) + semantic (top 5 relevant, similarity ≥ 0.75)
• Enables coherent multi-turn research with conversation context
• Fallback: Session memory (last 20 messages)

• Auto-triggers for sessions >20 messages
• LLM summarizes conversation (target: 37.5% of window)
• Prevents context window overflow in long conversations

​

Entity Filtering

• Detects canonical name, exact search queries, official domains, disambiguation terms
• Example: canonical_name: "Acme Analytics", official_domains: ["acme.com"]

• Scoring: domain match +0.6, alias in URL +0.4, text match +0.4
• Threshold: 0.3 (any single match passes)
• Safety floor: minKeep=8 (backfilled by quality × credibility)
• Official domains always preserved (bypass threshold)

• Filters off-entity tool results (e.g., removes “Mind Inc” when searching for “Acme Mind”)
• Keeps reasoning-only outputs; prunes off-entity tool-driven results

​

Gap Filling

• Missing section headings (### Area Name)
• Gap indicator phrases (“limited information”, “insufficient data”)
• Low citation density (< 2 inline citations per section)

1. Build targeted queries: "Find detailed information about: <area>"
2. Execute focused ReAct loops (max 3 iterations per gap)
3. Re-collect citations (global deduplication with original citations)
4. Re-synthesize with combined evidence using large tier

Query: "Analyze company X market strategy, competitive landscape, and leadership"

Initial synthesis coverage:
- Market strategy: ✓ 800 chars, 3 citations
- Competitive landscape: ✓ 650 chars, 2 citations
- Leadership: ✗ 200 chars, 0 citations  ← Gap detected

Gap filling triggers:
1. Targeted query: "Find detailed information about: company X leadership"
2. ReAct loop executes focused search
3. Re-synthesize with combined evidence

​

Citations

• Extracted from web_search and web_fetch tool outputs
• URL/DOI normalization and deduplication
• Scoring: quality (recency + completeness) × credibility (domain reputation)
• Diversity enforcement (max 3 per domain)

• Minimum 6 inline citations per report (clamped by available, floor 3)
• Per-area requirement: ≥2 inline citations per section
• Sources section lists all collected citations:
  • “Used inline” (cited in text)
  • “Additional source” (collected but not cited)

​

Synthesis Continuation

1. Must end with sentence punctuation (., !, ?, 。)
2. No dangling conjunctions (and, but, however, therefore...)
3. Every research area has subsection with ≥600 chars and ≥2 citations

• Adaptive margin: min(25% of effective_max_completion, 300 tokens)
• Prompt: “Continue from last sentence; maintain headings and citation style”
• Uses large tier for quality (gpt-4.1/opus)

​

Verification

• Identifies factual assertions from synthesis
• Cross-references against collected citations

• Weighted by citation credibility
• Flags conflicts and unsupported claims
• Per-claim details: supporting/conflicting citations

{
  "verification": {
    "total_claims": 10,
    "overall_confidence": 0.78,
    "supported_claims": 7,
    "unsupported_claims": ["Claim lacking evidence..."],
    "claim_details": [
      {
        "claim": "Transformer architecture introduced in 2017",
        "confidence": 0.95,
        "supporting_citations": [1, 3],
        "conflicting_citations": []
      }
    ]
  }
}

​

Language Matching

• Detects user query language (heuristic)
• Synthesis responds in the same language
• Supports English, Chinese, Japanese, Korean, Arabic, Russian, Spanish, French, German (heuristic‑based)
• Generic instructions ensure robustness across languages

​

Behavior Guarantees

​

Tips & Best Practices

​

Troubleshooting

• Invalid web_search search_type: Sanitized (falls back to auto)
• Duplicate citations: Handled via URL/DOI normalization
• Entity detection: Performed during refinement when present in the query

• Memory retrieval: memory_retrieval_v1
• Session memory fallback: session_memory_v1
• Context compression: context_compress_v1
• Gap filling: gap_filling_v1

​

Next Steps

• GitHub: https://github.com/Kocoro-lab/Shannon
• Discord: https://discord.gg/NB7C2fMcQR
• Company: https://kocoro.ai