Fact System — Architecture¶

This content was migrated from Documentation/fact_entity_preference_extraction.md and restructured into audience sections. Review for accuracy against the current codebase.

Context and Purpose¶

The Fact System exists to give Swisper persistent, personalized memory. Without it, every conversation would start from zero — the assistant would not know the user's name, family, allergies, or preferences.

The driving requirements behind its design are:

• Automatic extraction — Users should not have to explicitly save information. The system learns from natural conversation
• Entity-first attribution — Every fact must be correctly linked to the right person. Misattributed facts (storing a son's hobby under a colleague's name) are worse than missing facts
• Non-blocking extraction — Fact extraction must not add latency to the user's response. It runs in the background while the response streams
• Semantic retrieval — Facts must be retrievable by meaning, not just exact text. "What does Martin eat?" should surface dietary facts even if stored as "Martin is vegetarian"

Architecture Overview¶

The Fact System spans multiple nodes in the Global Supervisor graph and a set of backend services for persistence and retrieval.

flowchart TB
    subgraph Extraction["Extraction Pipeline"]
        MSG([User Message]) --> IC[Intent Classification]
        IC -->|has_extractable_facts| FE[Fact Extraction Node]
        IC -->|entities detected| ER[Entity Resolution Node]
        FE --> EM[Extraction Merge Node]
        ER --> EM
        EM -->|conflict detected| FCR[Fact Conflict Resolution]
        EM -->|no conflict| PERSIST
    end

    subgraph Retrieval["Retrieval Pipeline"]
        IC -->|needs_semantic_retrieval| SR[Semantic Retrieval]
        IC -->|temporal query| TR[Temporal Retrieval]
        SR --> MA[Memory Assembly]
        TR --> MA
    end

    subgraph Storage["Persistence Layer"]
        PERSIST[Fact Persistence Service]
        PERSIST --> PG[(PostgreSQL + pgvector)]
        PERSIST --> EMBED[Embedding Generation]
        EMBED -->|Vertex AI| PG
        MA --> CACHE[(Redis Cache)]
    end

    subgraph Output["To UI Response"]
        MA --> UI[UI Response Nodes]
        UI -->|facts woven into response| RESP([User Response])
    end

Flow summary: Intent classification determines which pipeline stages run. Fact extraction and entity resolution run in parallel, then merge. The merge step links facts to entities, detects conflicts, and persists. Retrieval pulls relevant facts via vector search and temporal queries, assembling them for the UI response nodes.

Component Responsibilities¶

Data Model¶

Fact Schema¶

Entity Schema (Persons Table)¶

Fact Attribution Model¶

Key Design Decisions¶

1. Skip Ambiguous Entities Over Guessing¶

• Chosen: When entity resolution is ambiguous (confidence < 0.8), skip both the entity and associated facts entirely
• Rejected: Store with NULL entity; store with best-guess entity; create a new entity
• Rationale: Orphan facts (NULL entity, not about user) are unusable and pollute the database. Wrong attribution is worse than missing data. The system loses some valid facts but prevents entity explosion and misattribution

2. Background Extraction (Non-Blocking)¶

• Chosen: Fact extraction runs as a background task, not blocking the response
• Rejected: Synchronous extraction before response generation
• Rationale: Extraction takes 3–4.5 seconds (LLM call + storage). Making users wait that long would be unacceptable. Current-turn information is available directly from the message; extracted facts are needed for future turns

3. Unified LLM Call for Facts + Entities + Topics¶

• Chosen: Single LLM call extracts all three outputs
• Rejected: Separate LLM calls for each extraction type
• Rationale: More token-efficient and faster. The LLM has full context to make coherent entity-fact linking decisions within a single pass

4. Semantic Deduplication Over String Matching¶

• Chosen: LLM-based entity resolution with red-flag detection and coherence analysis
• Rejected: String matching, fuzzy matching, phonetic matching
• Rationale: "Martin (son, 8 years old)" and "Martin (colleague, accountant)" share the same name but are clearly different people. Only semantic understanding can detect age-career contradictions and relationship impossibilities

Interfaces and Contracts¶

Known Trade-offs and Debt¶