Token Usage & Analytics — Architecture¶

Audience: Architects, tech leads, senior engineers.

Context and Purpose¶

LLM API calls are the primary operational cost in Swisper. Without tracking, it is impossible to attribute costs to users, features, or specific pipeline nodes. The Token Usage system provides this attribution by instrumenting every LLM call with token-count recording, then persisting the data for analytics.

The driving architectural requirements are:

• Zero impact on request latency — Token recording must not add database writes to the critical path. This is achieved via Redis accumulation during the request.
• Per-node granularity — It must be possible to determine how many tokens each graph node consumes, not just the total per request.
• Separate job tracking — Background job token usage must be tracked separately from interactive usage, since jobs run outside the user request lifecycle.

Architecture Overview¶

graph TD
    subgraph Request["Request Lifecycle"]
        SI["session_init"] -->|initialize_tracking| RH["Redis Hash"]
        LLM1["LLM Call 1"] -->|record_llm_usage| RH
        LLM2["LLM Call N"] -->|record_llm_usage| RH
        STREAM["streaming.py"] -->|persist_to_postgres| PG["PostgreSQL: token_usages"]
    end

    subgraph Job["Background Job Lifecycle"]
        JI["Job start"] -->|initialize_tracking| RH2["Redis Hash"]
        JLLM["LLM Calls"] -->|record_llm_usage| RH2
        JE["Job end"] -->|persist_background_job_to_postgres| PG2["PostgreSQL: background_job_token_usages"]
    end

    subgraph Analytics["Analytics API"]
        API["GET /analytics/token-usage/..."] --> PG & PG2
    end

Component Responsibilities¶

Data Model¶

TokenUsage (table: token_usages)¶

One row per interactive request (conversation turn).

Indexes: created_at, chat_id, user_id + created_at (composite), node_type_breakdown (GIN).

BackgroundJobTokenUsage (table: background_job_token_usages)¶

One row per background job run.

Indexes: user_id + created_at (composite), job_type, llm_type_breakdown (GIN).

Two-Tier Storage¶

Redis (hot tier)¶

• Key pattern: token_usage:{correlation_id}
• Type: Redis hash
• Fields: user_id, chat_id, structured_output_tokens, streaming_tokens, embedding_tokens, total_prompt_tokens, total_completion_tokens, total_tokens, llm_call_count, node_type_breakdown (JSON string), start_time
• TTL: 3600 seconds (1 hour)
• Purpose: Fast incremental updates during request processing. Each record_llm_usage() call increments counters via HINCRBY and updates the JSONB breakdown.

PostgreSQL (cold tier)¶

• Tables: token_usages, background_job_token_usages
• Write timing: Once per request/job, at completion
• Purpose: Permanent storage for analytics queries. Redis key is deleted after successful persistence.

Analytics API¶

All endpoints are under /api/v1/analytics/ and require superuser authentication.

All endpoints accept optional start and end datetime query parameters for time-range filtering.

Key Design Decisions¶

Decision: Redis accumulation during request, PostgreSQL on completion

• Chosen: Accumulate in Redis hash, flush to PostgreSQL once per request
• Rejected: Direct PostgreSQL writes per LLM call, in-memory accumulation
• Rationale: Direct DB writes would add latency to every LLM call. In-memory accumulation would lose data on process crash. Redis provides fast atomic increments with durability acceptable for the 1-hour window, and a single PostgreSQL write on completion keeps the transaction count low.

Decision: JSONB for node-type breakdown

• Chosen: Store per-node token usage as a JSONB field on each row
• Rejected: Separate child table with one row per node per request
• Rationale: JSONB avoids a high-cardinality child table (dozens of nodes × thousands of requests). PostgreSQL GIN indexes make JSONB aggregations efficient. The trade-off is that schema changes to the breakdown format require careful migration.

Decision: Separate table for background job usage

• Chosen: background_job_token_usages with job_type and llm_type_breakdown
• Rejected: Shared table with nullable chat_id / job_type
• Rationale: Job and interactive usage have different metadata (jobs have job_type and processing_duration_ms; interactive has chat_id and request_duration_ms). Separate tables avoid nullable columns and allow independent indexing strategies.

Known Trade-offs and Debt¶