Large first calls are routine
Across the sampled development data: minimum 27.6k, median 96.1k, p90 172.2k, and maximum 212.0k tokens. Prompt caching can reduce compute and latency, but the model still has to attend over the full input.
Context architecture for Personal Demi
A concrete design for giving the model the smallest trustworthy working set without losing task, meeting, approval, or proactive-agent state.
Recommendation
Proposed direction
Use a deterministic state path in front of a hybrid bounded context assembler.
Answer canonical state questions directly from authoritative projections. For open-ended reasoning and actions, compose the turn from mandatory current state, a bounded raw conversation tail, progressively loaded tool schemas, and optional retrieved evidence. Both paths use the same sources and authorization boundary.
A fast path is not a second source of truth. It is a deterministic renderer over the same materialized projections. Every model-visible item still needs a source, freshness, authority, reason for admission, and token cost. Authorization and exact approval parameters are never delegated to semantic retrieval or lossy summaries.
There is no settled industry standard. This recommendation is the convergence of current mechanisms and evidence, not a vendor recipe. Its thresholds must be validated against Demi traces.
Latest Main prompt audit
The first call is carrying a general agent into a state lookup
For the request “Are there any tasks pending on me?”, only the current request, the materialized Work Index, and the task-state interpretation rules are essential. The latest main prompt at 9ef2aeb8 sends a much larger general-purpose surface before the model can answer.
~48.3ktokens before dynamic context or conversation history
42 tools~27.9k tokens; 95.5% of schema characters irrelevant here
96.1kmedian first-call tokens across 258 recent development turns
58.9%of sampled first calls were at or above 82k tokens
| First-call segment | Approximate size | Relevance to this request | Design implication |
|---|---|---|---|
| Current request + Work Index + task-state rules | Small, data-dependent | 5/5 | Make this the complete input to the canonical state path. |
| Evidence and no-bluff policy | Part of shared policy | 3/5 | Keep a compact invariant; enforce authoritative reads in code. |
| Time, routing, and response-format guidance | Several overlapping blocks | 1–2/5 | Include only fields that change the answer; use one output contract. |
| 42 tool schemas | ~27.9k tokens | 0/5 | Expose zero tools for this path. Make load_tool_group genuinely progressive elsewhere. |
| 42-item skill catalog | ~3.7k tokens | 0/5 | Keep the catalog out of routine state turns; load a skill by intent. |
| Identity, priorities, style, people files, context tree | Unbounded in several lanes | 0/5 | Retrieve personal context only when it can change the response or action. |
| Meeting context | Up to two 40k transcript copies | 0/5 | Deduplicate the two injection paths and admit transcript slices only for meeting intents. |
Pending means pending
Compute pending = tasks where pending != "none". If empty, answer “Nothing needs you right now.” Do not pad the response with recently completed work or infer completion from chat history.
Scope and confidence: these measurements are first model calls from recent development usage, not a verified reconstruction of the exact screenshot trace. Tool results fetched after inference starts are excluded. The full segment inventory and source mapping are in the word-by-word prompt audit.
Six problems currently collapsed into “context”
Current state
What is waiting, running, blocked, scheduled, or awaiting approval now. Read from authoritative projections, not memory search.
Conversation continuity
The recent turns needed to resolve references, corrections, tone, and the active line of thought.
Historical evidence
Older messages, transcripts, files, and provider results. Retrieve only when the request needs them.
Capability discovery
Which tools can help. Start with namespace summaries; load full schemas after the relevant group is selected.
Execution continuity
Goals, decisions, attempts, blockers, outputs, and exact artifact references for long-running work.
Authorization
What Demi may do and the precise action being approved. Preserve exact typed data and enforce outside the model.
Architectural options
| Approach | How it works | Where it wins | Primary failure | Verdict |
|---|---|---|---|---|
| A. Large prompt + caching | Send broad history, state, and tool schemas every turn; cache the stable prefix. | Lowest implementation complexity. High recall when the model actually attends to the right item. | Cost and latency grow; stale and irrelevant facts compete with the request. Caching reduces recompute, not attention load. | Reject |
| B. Intent router + specialist prompts | Classify the request, then select one narrow prompt and tool bundle. | Cheap, predictable, and easy to reason about for clean single-domain intents. | Misroutes ambiguous or multi-domain requests; policy and cross-domain state get duplicated. | Use only as a hint |
| C. Retrieval-first / vector RAG | Embed all state and history; retrieve top-k chunks for each turn. | Scales optional historical evidence and fuzzy recall. | False negatives are silent. Semantic similarity is a poor authority model for exact, fresh, pending state. | Evidence lane only |
| D. One agent per domain | Tasks, meetings, approvals, and chat each own a prompt, memory, and tool set. | Strong isolation and domain specialization. | Handoffs and shared decisions become distributed-systems problems; state and policy diverge. | For execution, not context |
| E. Hybrid bounded assembler | Typed state is deterministic; history is bounded; evidence is retrieved; tools are progressive; artifacts remain external. | Balances correctness, latency, inspectability, and cross-domain work. | Requires provenance, projections, budgets, evaluation, and observability infrastructure. | Recommend |
Key design choice: use deterministic inclusion where omission would create a false statement or unsafe action; use retrieval where omission is recoverable with another tool call.
Proposed architecture
Request envelopeUser message, surface, thread, attachments, explicit entity ids, pending approval
→
Scope resolver + context planner
1 · Stable policy core
2 · Typed current-state projections
3 · Recent raw conversation tail
4 · Retrieved evidence
5 · Active tool groups
→
Model turnReceives ContextEnvelope + manifest; may request missing context or tool groups
Outside the prompt: event log · tasks · meetings/transcripts · approval ledger · provider artifacts · user memory · tool registry
ResolverRules over explicit ids, surface, active task, meeting phase, and pending approval. A classifier may add candidates but cannot remove mandatory state.
PlannerBuilds independent budget lanes, deduplicates, applies authority/freshness rules, and records why each item was included or omitted.
RetrieverHybrid lexical + semantic + metadata + time filters over optional evidence. Never the only path to current tasks, approvals, or meeting state.
Fallback
get_context(type, ids, query, time_range) and load_tool_group(namespace) let the model recover from deliberate under-inclusion.Data contracts, not prompt concatenation
The assembler should return structured lanes plus an audit manifest. Rendering them into provider-specific messages is the final adapter step.
type ContextEnvelope = {
request: UserRequest
scope: {
user_id: ID
thread_id: ID
surface: "main" | "task" | "meeting" | "approval" | "proactive"
task_id?: ID
meeting_id?: ID
approval_id?: ID
}
policy_version: string
current_state: ContextItem[]
recent_history: ContextItem[]
retrieved_evidence: ContextItem[]
active_tool_groups: string[]
budget: { limit: number; used: number; by_lane: Record<string, number> }
omissions: { lane: string; reason: string; recover_with?: ToolCall }[]
}
type ContextItem = {
source_type: "projection" | "event" | "artifact" | "memory" | "message"
source_id: ID
authority: "live" | "canonical" | "derived" | "user_claim"
observed_at: Timestamp
expires_at?: Timestamp
sensitivity: "normal" | "private" | "restricted"
token_estimate: number
admission_reason: string
content: unknown
}Conflict order: live provider or exact approval record → canonical materialized projection → task/meeting record → retrieved memory → chat claim or generated summary. Newer does not automatically beat more authoritative; surface disagreements to the model.
Assembly algorithm
Reserve immutable lanes.
Allocate the policy/safety core and request envelope first. These cannot be displaced by history or tool output.
Resolve scope deterministically.
Use surface, explicit ids, reply targets, attachments, active task, meeting phase, and pending approval. Emit ambiguity instead of inventing scope.
Load mandatory current state.
Read exact projections for in-scope tasks, meetings, approvals, and proactive triggers. Apply freshness TTLs and authority tags.
Add a bounded raw tail.
Preserve the most recent turns verbatim. Attach the latest structured checkpoint for older execution state.
Select tool namespaces.
Rules pre-load mandatory groups; a compact catalog exposes other groups. Full schemas load only when selected.
Retrieve optional evidence.
Search with entity, tenant, time, source, and sensitivity filters; combine lexical and semantic scores; rerank; deduplicate against current state.
Allocate by lane.
Trim within lanes rather than globally. Drop reloadable outputs and low-authority history before current state, corrections, approvals, or the user request.
Emit envelope + manifest.
Log tokens by component, admitted source ids, omissions, freshness, retrieval queries, active tools, latency, and downstream outcome.
Initial guardrails, not industry facts
| Lane | Starting budget | Overflow behavior |
|---|---|---|
| Policy + request | 3–4k tokens | Never evict; refactor the core if it exceeds budget. |
| Tool discovery + schemas | 0.5–1k index; ≤3k loaded | Unload unused groups; fetch another group on demand. |
| Current state | 1–2k | Keep exact in-scope records; collapse unrelated lists to counts/ids. |
| Recent raw history | 4–6k | Move oldest turns behind the latest structured checkpoint. |
| Retrieved evidence | 4–8k | Rerank, diversify sources, retain stable references for reload. |
| Total initial context | 12–16k default; p95 ≤20–24k | >40k requires a recorded reason and evaluation bucket. |
Admission rules by surface
| Surface | Always include | Load only when needed | Never rely on |
|---|---|---|---|
| Main chat / status | Compact work index: active, blocked, pending input, next scheduled event; freshness timestamp. | Exact task record, provider evidence, or older conversation after reference. | A generated recap as proof that nothing is pending. |
| Task | Task brief, current phase, pending input, checkpoint, recent raw tail, artifact ids. | Old attempts, large outputs, external provider state. | Whole main-chat history. |
| Meeting | Meeting id, participants, phase, agenda/goal, current notes, commitments. | Transcript slices and related account/task history. | A semantic match when an exact meeting id exists. |
| Approval | Exact proposed action, parameters, target, requester, expiry, policy decision, approval status. | Supporting evidence referenced by the approval. | Summaries, embeddings, or model memory for authorization. |
| Proactive run | Trigger evidence, user preferences, relevant task/routine, cooldown, last notification/outcome. | Evidence needed to validate the opportunity. | General chat history as permission or evidence of urgency. |
Memory and compaction
Do not make a rolling prose summary the durable memory. Use three layers:
Recent raw tail
Verbatim turns for local reference resolution, corrections, conversational intent, and tone.
Structured checkpoint
Goal, plan, decisions, unresolved blockers, pending user input, attempted actions, failures, and artifact ids.
External evidence
Full messages, transcripts, provider results, and files addressed by stable ids and reloaded only when needed.
WhenCheckpoint at meaningful milestones and before pressure thresholds, not after every turn.
ProtectUser corrections, unresolved commitments, exact ids, approval parameters, authoritative timestamps, and relevant failed attempts.
Memory writesSeparate from response generation. Store typed, source-linked claims with version, confidence, sensitivity, and invalidation path.
DeletionSupport supersession and forgetting explicitly. A stale preference must not survive merely because it is semantically similar.
Provider compaction may be an adapter, not the source of truth. Opaque summaries are acceptable for transient continuity only if canonical checkpoints and full-fidelity references remain under Demi’s control.
Failure behavior is part of the architecture
Missing context
Call get_context with a typed request. If the source is unavailable, say so; do not convert absence from the prompt into absence in the world.
Stale state
Refresh records past their TTL. Live authoritative state wins, and the model receives both the new value and the superseded claim when the conflict matters.
Retrieval miss
Retry with exact keywords, metadata, and a broader time window. Then return a scoped inability, not a confident negative.
Budget overflow
Discard reloadable tool output, duplicate evidence, then low-ranked history. Never drop the request, policy, exact approval, or mandatory current state.
Ambiguous entity
Return candidates with ids and distinguishing fields. Ask only when choosing one would materially change the answer or action.
Planner degradation
Fall back to core + request + current surface projection + recent tail + context-fetch tool. Record the degraded path.
Evaluation gate
Optimize for reliable behavior, not just fewer tokens. Build a replay set from production traces and score each competency separately.
| Scenario family | Must demonstrate | Primary measures |
|---|---|---|
| Status and pending work | Exact active/blocked/waiting state; abstain when a source is unavailable. | State accuracy, unsupported negatives, projection freshness. |
| Corrections and updates | New value supersedes old without erasing useful history. | Update accuracy, stale-claim rate, conflict disclosure. |
| Meetings and approvals | Resolve exact entity; preserve commitments and authorization parameters. | Entity accuracy, authorization regressions, evidence traceability. |
| Long tasks | Recover goal, checkpoint, blockers, artifacts, and significant failed attempts. | Compaction recovery, long-range completion, repeated-work rate. |
| Proactive runs | Use fresh trigger evidence and preferences without inventing urgency or permission. | Precision, actionability, false-positive notifications, cooldown compliance. |
| Tool discovery | Select the right group and recover when it was not preloaded. | Tool-group miss rate, recovery turns, schema tokens, latency. |
| Retrieval stress | Find exact, temporal, cross-session, and contradictory evidence; abstain on misses. | Recall@k, reranker recall, source diversity, false certainty. |
Release gate: no regression on state/approval correctness; lower p50 and p95 input tokens; bounded latency; measurable recovery from deliberate context omission. Budget numbers change only from these results.
Implementation sequence
0 · MeasureLog prompt composition by source and lane; build trace replay/evals. Without this, “context optimization” is opinion.
1 · State pathAdd deterministic renderers for task status, approvals, reminders, credits, and current connections. Target 4–8k tokens and zero tools.
2 · ProjectMaterialize typed, authoritative current-state projections for tasks, meetings, approvals, and proactive triggers. Inject a fresh full projection, not historical deltas.
3 · DiscoverMake optional tool groups inactive by default. Keep only recovery and universally required tools visible on open-ended turns.
4 · BoundCollapse duplicate policy and response contracts, remove meeting duplication, add structured checkpoints, and cap raw history.
5 · RetrieveAdd relevance-gated personal context and hybrid retrieval for optional historical evidence after exact-state paths are measured.
Keep one authority model: deterministic answers and agentic turns must read the same projections and share backend policy. The fast path is an alternate execution path, not alternate state or authorization logic.
Inspiration and evidence
What the sources actually support
Evidence labels matter. Peer-reviewed benchmarks are the strongest support for measured failure modes, but remain workload/model-specific. 2026 preprints are useful directional evidence, not settled practice. Vendor docs prove a mechanism is available; they do not prove it is optimal.
MemoryAgentBench · ICLR 2026
Evaluates accurate retrieval, test-time learning, long-range understanding, and selective forgetting. Long-context, RAG, and external-memory agents each leave material gaps. Supports evaluating competencies separately, not choosing one universal memory technique.
LongMemEval · ICLR 2025
Tests information extraction, multi-session reasoning, temporal reasoning, knowledge updates, and abstention. Supports explicit indexing/retrieval/reading stages plus time-aware query construction.
Diagnosing and Mitigating Context Rot in Long-horizon Search · revised Aug 2026
Finds context-length-related premature termination and model-dependent mitigation results across long-horizon search. Supports bounded context and evaluation per model; does not establish a universal threshold or remedy.
MemGym · May 2026
Evaluates memory inside tool-use dialogue, research, coding, and computer-use agents while separating memory from reasoning and tool skill. Supports testing Demi on real agent workflows rather than isolated recall.
Memex(RL) · Mar 2026
Explores compact indexed experience summaries that point to external full-fidelity evidence. It motivates reference-preserving compaction, but its results do not make that a production standard.
StructMemEval · Feb 2026
Examines structured memory such as ledgers, lists, and trees. Supports typed checkpoints and projections where generic chunk retrieval struggles.
The Horizon Gap · Aug 2026
Separates long-horizon tasks, long-context models, and long-term memory systems across a broad literature review. Useful taxonomy and very current, but too recent to treat as settled consensus.
OpenAI tool search and Agents SDK tools · current docs checked Sep 2026
Show deferred tool and namespace loading. Evidence that progressive schemas are implementable, not that OpenAI’s defaults are correct for Demi.
Anthropic tool-context management and context editing · current docs checked Sep 2026
Distinguish tool search, programmatic calls, caching, clearing, and compaction. Useful implementation taxonomy; provider-specific and not independent validation.
Google ADK session state · current docs checked Sep 2026
Demonstrates separation between event history and mutable session scratchpad/state. An implementation reference, not comparative evidence.
Bottom line
The reliable conclusion is composition, not a product feature.
Current evidence favors separating authoritative state, recent raw context, optional retrieval, progressive capabilities, and durable external memory. It does not justify one vector store, one compactor, one provider, or one context-size number as the answer.