Document Intelligence & RAG — Architecture¶

Audience: Architects, tech leads, senior engineers evaluating the v2 document agent and its delta against the v1 DocumentSearchAgent.

Context and Purpose¶

The document agent is Swisper's interface between the planner and the user's document corpus — uploaded files, ingested email attachments, and Swisper-generated documents. It must do three things at once:

1. Stay in the FC loop. Document operations have to interleave freely with memory recall, calendar lookups, and other tools in a single planner turn. "Find the insurance policy Thomas sent and remind me when it expires" must not need two agent hand-offs.
2. Honour workspace boundaries. Every retrieval is scoped to a workspace and avatar. There is no cross-workspace leakage path, even on a misconfigured planner call.
3. Keep retrieval honest. Vector-only cosine similarity misses exact matches (policy numbers, names, dates). Hybrid retrieval and a Discovery Engine reranker close that gap, with all components individually feature-flagged for safe rollout.

The v1 DocumentSearchAgent (Pattern A delegation, reactive LangGraph subgraph) cannot satisfy any of these without architectural changes. The v2 agent is a clean break.

v1 vs v2 — What Changed¶

The v1 path is preserved on feature/voice-v2-gemini-live-and-friends so today's UAT stays unbroken; v2 is selected in the supervisor when the flag is on.

v2 Architecture Overview¶

graph TD
    subgraph Planner["Agentic Supervisor (FC loop)"]
        P["global_planner_node"]
        TN["tool_node"]
    end

    subgraph DocTools["Document tools (Pattern 2 — agents/document/)"]
        SD["search_documents"]
        GD["get_document"]
        LD["list_documents"]
        CD["create_document"]
    end

    subgraph Services["Domain services"]
        DSS["DocumentSearchService<br/>(services/document_search.py)"]
        IDX["IndexingService<br/>(services/file_indexing.py)"]
        RR["RerankerService<br/>(services/reranker.py)"]
        FE["FactAndEntityExtraction<br/>Service"]
    end

    subgraph Gateways["Gateways / external"]
        LLM["SwisperLLMAdapter<br/>(embedding lanes)"]
        DocAI["Document AI<br/>(layout parser)"]
        DE["Vertex Discovery Engine<br/>(reranker)"]
    end

    subgraph Storage["Persistence"]
        PG[("AlloyDB:<br/>documents, document_chunks,<br/>attachment_chunks (pgvector + tsvector)")]
        S3[("MinIO / GCS<br/>(encrypted blobs)")]
    end

    subgraph Jobs["Background"]
        EDFJ["ExtractDocumentFactsJob"]
        REEMB["ReembedDocumentChunksJob"]
    end

    P -->|tool_calls| TN
    TN --> SD & GD & LD & CD

    SD --> DSS
    GD & LD --> PG
    CD --> IDX

    DSS --> LLM
    DSS --> PG
    DSS --> RR --> DE

    IDX --> DocAI & LLM & PG & S3
    IDX -. IndexingCompleteEvent .-> EDFJ --> FE --> PG
    REEMB --> LLM --> PG

Wiring: agents/agentic_supervisor/agent.py builds the tool list per request via _get_document_tools() (lines ~290–325). The factory agents/document/factory.py::build_document_tools() returns the four BaseTool instances, each capturing the request's workspace_id, avatar_id, and db_session so the planner can call them without passing context.

Components and Responsibilities¶

Tool Catalogue (Planner-Facing)¶

The planner sees four tools when DOCUMENT_SEARCH_V2_ENABLED=True and a workspace context exists. Tool descriptions are loaded from each BaseTool.description; the planner picks among them based on user intent.

search_documents — primary retrieval tool¶

SearchDocumentsInput:
    query: str                  # natural-language query
    top_k: int = 5              # 1..20

Returns a numbered list of evidence entries. Each entry contains source kind (document | email_attachment), source UUID, optional Section: heading path, optional Pages:, optional parent email subject/sender, relevance score, and the chunk text.

Internal flow (DocumentSearchService.search): 1. Embed query on the document_embedding lane. 2. (Optional) apply EncryptionContext.transform_query for encrypted-at-rest corpora. 3. Run the configured fusion strategy on DocumentChunk (workspace-scoped). Always run dense search on AttachmentChunk (no tsvector column). 4. Merge, sort by distance, keep top 3 × top_k candidates. 5. If DOCUMENT_RERANKER_ENABLED=True, rerank via Discovery Engine; on failure return fusion order. 6. Return top top_k.

get_document¶

GetDocumentInput:
    document_id: str            # document UUID

Returns title, filename, type, size, created timestamp, summary, and processing status. Returns a "not found in this workspace" message if the workspace check fails — the result is identical to "does not exist" so the planner cannot probe across workspaces.

list_documents¶

ListDocumentsInput:
    limit: int = 20             # 1..100

Returns a header line plus one row per document with ID, title, filename, type, size, and created timestamp. Workspace-scoped; ordered by created_at desc.

create_document¶

CreateDocumentInput:
    title: str                  # 1..500 chars
    content: str                # Markdown, min 1 char

Writes the content as <sanitised_title>.md through IndexingService.process_and_index_document() with source_type=GENERATED. The document is searchable on the next search_documents call. Errors return a string starting with "Failed to create document"; success returns the new document_id.

How the planner uses them¶

The planner is encouraged (via tool descriptions) to:

• Reach for search_documents whenever the user asks about uploaded content, attachments, or generated notes.
• Compose with memory tools — e.g. recall("Thomas insurance") followed by search_documents("home insurance Thomas") and use the union to answer.
• Use get_document after search_documents only when the user wants metadata (title, summary, status) about a hit.
• Use list_documents for "what do I have on file?" or as a fallback when search returns nothing.
• Use create_document only when the user explicitly asks Swisper to save a structured artefact (meeting notes, summary, report) — not for chat replies.

Embedding Lanes¶

Two cohorts, two lanes — enforced at adapter level via agent_configuration rows.

Hard rule (Spec §6.9): A query MUST be embedded on the same lane as the corpus it targets. DocumentSearchService._embed_query is the only place a document query is embedded; it always uses document_embedding. Don't paste SwisperLLMAdapter("__service__", "embedding") into search code paths.

The Vector(2000) column width is unchanged across both lanes — the migration from Azure embeddings to gemini-embedding-2-preview is content-only and runs through ReembedDocumentChunksJob.

Hybrid Retrieval¶

Three fusion strategies, selected via DOCUMENT_FUSION_STRATEGY (with DOCUMENT_HYBRID_RETRIEVAL_ENABLED=False forcing dense_only regardless):

Lexical signal: AlloyDB tsvector column on DocumentChunk.search_vector, populated against plaintext before PGPString encrypts chunk_text. Attachments don't carry a tsvector and always run dense.

After fusion, the top 3 × top_k candidates flow to the reranker (when enabled). The reranker score replaces relevance_score in the returned EvidenceResults.

Workspace + Avatar Scoping¶

Every read and every write goes through a workspace boundary:

• Search: DocumentSearchService adds WHERE workspace_id = :workspace_id on every chunk query (both document and attachment).
• Get/List: tools check Document.workspace_id against the bound workspace ID. Mismatches return "not found", not a permission error — no information disclosure.
• Create: IndexingService writes Document.workspace_id, Document.avatar_id, and propagates workspace_id to every DocumentChunk.
• Background jobs: ExtractDocumentFactsJob reads Document.workspace_id and stores facts under the same workspace.

Indices: ix_documents_workspace_id and ix_document_chunks_workspace_id make these filters cheap.

Document → Fact Pipeline¶

After indexing, a document with fact_extraction_status='queued' and a classification on the configurable allowlist becomes input for ExtractDocumentFactsJob:

1. Pull up to MAX_CHUNK_PASSAGES = 5 chunks, each truncated to MAX_PASSAGE_CHARS = 800.
2. Hand the joined text to FactAndEntityExtractionService.extract_and_store() with the document's workspace_id/avatar_id and source_ref_type='document', source_ref_id=<document_id> provenance.
3. Facts are deduplicated semantically (0.92 threshold) against existing facts, embedded on the embedding lane (intentionally different from the document lane), and linked to Person records by entity disambiguation.

Failure isolation: extraction failure does not block document availability. The document remains searchable via search_documents regardless of fact extraction state.

Feature flag: DOCUMENT_FACT_EXTRACTION_ENABLED gates the whole job. While off, documents stay in queued; flipping the flag picks them up on the next run.

Data Model — What's New in v2¶

Document (table documents):

DocumentChunk (table document_chunks):

AttachmentChunk keeps the same shape and joins to Attachment → Email for parent-email provenance in evidence rows.

Configuration & Feature Flags¶

All keys are reachable via ConfigurationService (DB-backed) with environment-variable fallbacks defined in swisper/core/config.py.

Recommended rollout sequence: turn on DOCUMENT_SEARCH_V2_ENABLED first (verify tool selection in traces), then DOCUMENT_HYBRID_RETRIEVAL_ENABLED with rrf, then DOCUMENT_FACT_EXTRACTION_ENABLED once chunk re-embedding has completed.

How to Use It¶

From the planner¶

You don't call this module directly — the supervisor exposes it. Verify in a request trace that Built N document tools for workspace=<id> is logged (see agentic_supervisor/agent.py:320) and that the planner emits a tool_call for one of the four document tools instead of delegate_to_document_search_agent.

From a service / job¶

Read paths go through DocumentSearchService. Construct it with a Session and call search(query, workspace_id, avatar_id, top_k); the service handles embedding, fusion, and reranking. Do not instantiate RAGService for new code — it is the v1 implementation and is being retired with the agents/doc/ agent.

Write paths go through IndexingService.process_and_index_document(file_content, filename, workspace_id, avatar_id, user_id, source_type). Pass the right DocumentSourceType so downstream consumers (e.g. fact extraction) can apply type-aware policies.

From a test¶

Real-LLM tests should embed against the document_embedding lane to match the corpus. Fixtures should populate workspace_id, avatar_id, and (for hybrid) the search_vector column. The plaintext-first ordering in IndexingService._populate_search_vectors is the only correct way — manual fixture rows that bypass it will not match to_tsvector queries.

From the UI¶

The frontend keeps calling the existing /api/v1/documents/... REST endpoints (see api/routes/documents.py). Those routes still drive the v1 RAG service today. Migration to v2 is tracked separately and does not block the planner-side cutover; the chat-driven path described above is the v2 critical path.

Failure Modes & Degradation¶

Spec Coverage / Gap Analysis¶

Comparing what's shipped on feature/voice-v2-gemini-live (and integration sub-branches) against spec_FEAT_ALL_document_intelligence_v1.md:

The two non-trivial gaps that change the planner's behaviour are: (1) the missing analyze_document tool — the planner has no way to request deep single-document analysis without falling back to v1 — and (2) the absent self-correcting loop, which means a poor first-cycle query is returned as-is. Both are tractable extensions of the current shape (no architectural rework needed).

Known Trade-offs and Debt¶