Large first calls are routine
Across the sampled development data: minimum 27.6k, median 96.1k, p90 172.2k, and maximum 212.0k tokens. Prompt caching can reduce compute and latency, but the model still has to attend over the full input.
Context architecture for Personal Demi
A concrete design for giving the model the smallest trustworthy working set without losing task, meeting, approval, or proactive-agent state.
Recommendation
Proposed direction
Give the general agent a compact operating model, a map of available context, and tools to acquire it.
The first prompt should teach the agent how to find authoritative state, evidence, procedures, and capabilities after it sees any request. Keep the context assembler as a budget and provenance layer. It may attach obvious scope, but it should not need to predict the full intent before the agent can work.
Teach the agent to fish. Always expose a small set of context-navigation primitives and a compact source manifest. Keep raw records, full tool schemas, and domain procedures behind those primitives. Authorization and exact approval parameters remain enforced outside the model.
There is no settled industry standard. This recommendation is the convergence of current mechanisms and evidence, not a vendor recipe. Its thresholds must be validated against Demi traces.
Latest Main prompt audit
Rate the prompt by whether it equips a general-purpose agent.
The design target is an agent that can handle unknown, ambiguous, and cross-domain requests. The question for every always-on segment is whether it helps the agent discover, acquire, verify, or use the right context across that full space.
~48.3ktokens before dynamic context or conversation history
42 tools~27.9k tokens of full execution schemas before capability selection
96.1kmedian first-call tokens across 258 recent development turns
58.9%of sampled first calls were at or above 82k tokens
| First-call capability | General-purpose relevance | Correct representation |
|---|---|---|
| Operating model: intent, evidence, authority, approvals, recovery | 5/5 | One compact, non-duplicated core with backend enforcement. |
| Context-source manifest + acquisition protocol | 5/5 | Always show what can be known, its authority and freshness, and how to fetch it. |
| Current request + bounded recent conversation | 5/5 | Always present verbatim so the agent can form its own acquisition plan. |
| Authoritative tasks, meetings, approvals, files, memory, and integrations | 5/5 | Advertise every source; attach only explicit surface state and fetch records on demand. |
| Capability and skill discovery | 4/5 | Compact categories, descriptions, and loading handles. |
| Full tool schemas and skill bodies | 2/5 | Load only after the agent selects the capability. |
| Raw memory, context trees, transcripts, and provider artifacts | 1/5 | Keep external; retrieve bounded, source-linked slices. |
| Duplicate policy, response contracts, and repeated context | 0/5 | Remove. |
Make absence require a lookup
The core should teach one reusable rule: before claiming that something does not exist, query the authoritative source named in the manifest. This fixes task-status answers and transfers to meetings, approvals, files, people, and integrations.
Scope and confidence: these measurements are first model calls from recent development usage. Tool results fetched after inference starts are excluded. The full prompt audit rates every component from the general-purpose-agent perspective.
Six problems currently collapsed into “context”
Current state
What is waiting, running, blocked, scheduled, or awaiting approval now. Read from authoritative projections, not memory search.
Conversation continuity
The recent turns needed to resolve references, corrections, tone, and the active line of thought.
Historical evidence
Older messages, transcripts, files, and provider results. Retrieve only when the request needs them.
Capability discovery
Which tools can help. Start with namespace summaries; load full schemas after the relevant group is selected.
Execution continuity
Goals, decisions, attempts, blockers, outputs, and exact artifact references for long-running work.
Authorization
What Demi may do and the precise action being approved. Preserve exact typed data and enforce outside the model.
Architectural options
| Approach | How it works | Where it wins | Primary failure | Verdict |
|---|---|---|---|---|
| A. Large prompt + caching | Send broad history, state, and tool schemas every turn; cache the stable prefix. | Lowest implementation complexity. High recall when the model actually attends to the right item. | Cost and latency grow; stale and irrelevant facts compete with the request. Caching reduces recompute, not attention load. | Reject |
| B. Intent router + specialist prompts | Classify the request, then select one narrow prompt and tool bundle. | Cheap, predictable, and easy to reason about for clean single-domain intents. | Misroutes ambiguous or multi-domain requests; policy and cross-domain state get duplicated. | Use only as a hint |
| C. Retrieval-first / vector RAG | Embed all state and history; retrieve top-k chunks for each turn. | Scales optional historical evidence and fuzzy recall. | False negatives are silent. Semantic similarity is a poor authority model for exact, fresh, pending state. | Evidence lane only |
| D. One agent per domain | Tasks, meetings, approvals, and chat each own a prompt, memory, and tool set. | Strong isolation and domain specialization. | Handoffs and shared decisions become distributed-systems problems; state and policy diverge. | For execution, not context |
| E. Agent-led bounded acquisition | The agent gets a context map and universal retrieval primitives; it chooses what to load while the runtime enforces budgets and authority. | Preserves generality without paying the full context cost on every turn. | Requires reliable discovery, recovery behavior, typed sources, and multi-step evaluation. | Recommend |
Key design choice: make all relevant context discoverable, not immediately present. Include exact state when the surface or explicit id makes it mandatory; otherwise give the agent a reliable path to acquire and verify it.
Proposed architecture
Initial envelopeUser request, bounded raw tail, obvious surface scope, compact source manifest
→
General agent operating loop
1 · Interpret the request
2 · Inspect available sources
3 · Acquire the minimum context
4 · Verify authority and freshness
5 · Load a capability or answer
→
Acquisition layer
get_context, source inspection, search, and load_tool_group return typed, bounded resultsOutside the prompt: event log · tasks · meetings/transcripts · approval ledger · provider artifacts · user memory · tool registry
ManifestA small typed map of available sources: description, authority, freshness, scope, approximate size, and retrieval handle. It tells the agent where to look without injecting the data.
AgentChooses context after seeing the request. The prompt teaches widening, exact lookup, freshness checks, conflict handling, and abstention when the source is unavailable.
AssemblerEnforces budgets, provenance, deduplication, authorization, and result shape. It can attach explicit ids and surface state, but does not need to solve intent routing.
RetrieverSupports exact ids, structured filters, lexical search, and semantic search. Current tasks, approvals, and meeting state retain authoritative structured lookup paths.
Data contracts, not prompt concatenation
The assembler should return structured lanes plus an audit manifest. Rendering them into provider-specific messages is the final adapter step.
type ContextEnvelope = {
request: UserRequest
scope: {
user_id: ID
thread_id: ID
surface: "main" | "task" | "meeting" | "approval" | "proactive"
task_id?: ID
meeting_id?: ID
approval_id?: ID
}
policy_version: string
available_context: ContextSource[]
current_state: ContextItem[]
recent_history: ContextItem[]
retrieved_evidence: ContextItem[]
active_tool_groups: string[]
budget: { limit: number; used: number; by_lane: Record<string, number> }
omissions: { lane: string; reason: string; recover_with?: ToolCall }[]
}
type ContextSource = {
name: "tasks" | "meetings" | "approvals" | "memory" | "files" | "integrations"
description: string
authority: "live" | "canonical" | "derived"
freshness: Timestamp | "on_fetch"
retrieval_tool: string
estimated_cost: "small" | "medium" | "large"
}
type ContextItem = {
source_type: "projection" | "event" | "artifact" | "memory" | "message"
source_id: ID
authority: "live" | "canonical" | "derived" | "user_claim"
observed_at: Timestamp
expires_at?: Timestamp
sensitivity: "normal" | "private" | "restricted"
token_estimate: number
admission_reason: string
content: unknown
}Conflict order: live provider or exact approval record → canonical materialized projection → task/meeting record → retrieved memory → chat claim or generated summary. Newer does not automatically beat more authoritative; surface disagreements to the model.
Agent-led acquisition algorithm
Reserve immutable lanes.
Allocate the policy/safety core and request envelope first. These cannot be displaced by history or tool output.
Attach a compact source manifest.
List what context and capabilities exist, their authority and freshness, and the primitive that loads each one.
Add a bounded raw tail.
Preserve recent turns verbatim and attach explicit ids or surface state that are unambiguously in scope.
Expose universal acquisition primitives.
Keep source inspection, exact context lookup, search, and tool-group loading available on every turn.
Let the agent plan acquisition.
After reading the request, the agent chooses a source, fetches the smallest useful slice, and widens only when the result is insufficient.
Validate each fetch.
The runtime enforces tenant, authorization, sensitivity, freshness, size, and provenance before returning a typed result.
Load capabilities progressively.
The agent sees compact capability descriptions first and receives full operation schemas only after selecting a group.
Emit envelope + acquisition trace.
Log which sources were considered, fetched, widened, unavailable, or omitted, plus tokens, latency, and outcome.
Initial guardrails, not industry facts
| Lane | Starting budget | Overflow behavior |
|---|---|---|
| Policy + request | 3–4k tokens | Never evict; refactor the core if it exceeds budget. |
| Tool discovery + schemas | 0.5–1k index; ≤3k loaded | Unload unused groups; fetch another group on demand. |
| Current state | 1–2k | Keep exact in-scope records; collapse unrelated lists to counts/ids. |
| Recent raw history | 4–6k | Move oldest turns behind the latest structured checkpoint. |
| Retrieved evidence | 4–8k | Rerank, diversify sources, retain stable references for reload. |
| Total initial context | 12–16k default; p95 ≤20–24k | >40k requires a recorded reason and evaluation bucket. |
Admission rules by surface
| Surface | Always include | Load only when needed | Never rely on |
|---|---|---|---|
| Main chat | Request, recent raw tail, source manifest, acquisition primitives, and explicit surface ids. | Work Index, exact task record, provider evidence, personal memory, or older conversation. | A generated recap or absence from the initial prompt as proof that nothing exists. |
| Task | Task brief, current phase, pending input, checkpoint, recent raw tail, artifact ids. | Old attempts, large outputs, external provider state. | Whole main-chat history. |
| Meeting | Meeting id, participants, phase, agenda/goal, current notes, commitments. | Transcript slices and related account/task history. | A semantic match when an exact meeting id exists. |
| Approval | Exact proposed action, parameters, target, requester, expiry, policy decision, approval status. | Supporting evidence referenced by the approval. | Summaries, embeddings, or model memory for authorization. |
| Proactive run | Trigger evidence, user preferences, relevant task/routine, cooldown, last notification/outcome. | Evidence needed to validate the opportunity. | General chat history as permission or evidence of urgency. |
Memory and compaction
Do not make a rolling prose summary the durable memory. Use three layers:
Recent raw tail
Verbatim turns for local reference resolution, corrections, conversational intent, and tone.
Structured checkpoint
Goal, plan, decisions, unresolved blockers, pending user input, attempted actions, failures, and artifact ids.
External evidence
Full messages, transcripts, provider results, and files addressed by stable ids and reloaded only when needed.
WhenCheckpoint at meaningful milestones and before pressure thresholds, not after every turn.
ProtectUser corrections, unresolved commitments, exact ids, approval parameters, authoritative timestamps, and relevant failed attempts.
Memory writesSeparate from response generation. Store typed, source-linked claims with version, confidence, sensitivity, and invalidation path.
DeletionSupport supersession and forgetting explicitly. A stale preference must not survive merely because it is semantically similar.
Provider compaction may be an adapter, not the source of truth. Opaque summaries are acceptable for transient continuity only if canonical checkpoints and full-fidelity references remain under Demi’s control.
Failure behavior is part of the architecture
Missing context
Call get_context with a typed request. If the source is unavailable, say so; do not convert absence from the prompt into absence in the world.
Stale state
Refresh records past their TTL. Live authoritative state wins, and the model receives both the new value and the superseded claim when the conflict matters.
Retrieval miss
Retry with exact keywords, metadata, and a broader time window. Then return a scoped inability, not a confident negative.
Budget overflow
Discard reloadable tool output, duplicate evidence, then low-ranked history. Never drop the request, policy, exact approval, or mandatory current state.
Ambiguous entity
Return candidates with ids and distinguishing fields. Ask only when choosing one would materially change the answer or action.
Planner degradation
Fall back to core + request + current surface projection + recent tail + context-fetch tool. Record the degraded path.
Evaluation gate
Optimize for reliable behavior, not just fewer tokens. Build a replay set from production traces and score each competency separately.
| Scenario family | Must demonstrate | Primary measures |
|---|---|---|
| Status and pending work | Exact active/blocked/waiting state; abstain when a source is unavailable. | State accuracy, unsupported negatives, projection freshness. |
| Corrections and updates | New value supersedes old without erasing useful history. | Update accuracy, stale-claim rate, conflict disclosure. |
| Meetings and approvals | Resolve exact entity; preserve commitments and authorization parameters. | Entity accuracy, authorization regressions, evidence traceability. |
| Long tasks | Recover goal, checkpoint, blockers, artifacts, and significant failed attempts. | Compaction recovery, long-range completion, repeated-work rate. |
| Proactive runs | Use fresh trigger evidence and preferences without inventing urgency or permission. | Precision, actionability, false-positive notifications, cooldown compliance. |
| Tool discovery | Select the right group and recover when it was not preloaded. | Tool-group miss rate, recovery turns, schema tokens, latency. |
| Retrieval stress | Find exact, temporal, cross-session, and contradictory evidence; abstain on misses. | Recall@k, reranker recall, source diversity, false certainty. |
Release gate: no regression on state/approval correctness; lower p50 and p95 input tokens; bounded latency; measurable recovery from deliberate context omission. Budget numbers change only from these results.
Implementation sequence
0 · MeasureLog prompt composition by source and lane; build trace replay/evals. Without this, “context optimization” is opinion.
1 · MapCreate a compact source manifest and four universal primitives: inspect sources, exact context lookup, search, and tool-group loading.
2 · TeachReplace domain prose with a short acquisition protocol: choose authority, fetch narrowly, widen on miss, verify before negatives, and abstain when unavailable.
3 · DeferMake optional tool groups, skill bodies, memory files, meeting transcripts, and provider schemas inactive until the agent requests them.
4 · BoundCollapse duplicate policy and response contracts, remove meeting duplication, add structured checkpoints, and cap raw history and tool results.
5 · OptimizeAdd intent hints, prefetch, and deterministic fast paths only where broad evals show they improve latency without reducing generality.
The first milestone is not a smarter router. It is an agent that can reliably discover and acquire missing context for unfamiliar requests. Prefetch and fast paths are optimizations over that complete loop.
Inspiration and evidence
What the sources actually support
Evidence labels matter. Peer-reviewed benchmarks are the strongest support for measured failure modes, but remain workload/model-specific. 2026 preprints are useful directional evidence, not settled practice. Vendor docs prove a mechanism is available; they do not prove it is optimal.
MemoryAgentBench · ICLR 2026
Evaluates accurate retrieval, test-time learning, long-range understanding, and selective forgetting. Long-context, RAG, and external-memory agents each leave material gaps. Supports evaluating competencies separately, not choosing one universal memory technique.
LongMemEval · ICLR 2025
Tests information extraction, multi-session reasoning, temporal reasoning, knowledge updates, and abstention. Supports explicit indexing/retrieval/reading stages plus time-aware query construction.
Diagnosing and Mitigating Context Rot in Long-horizon Search · revised Aug 2026
Finds context-length-related premature termination and model-dependent mitigation results across long-horizon search. Supports bounded context and evaluation per model; does not establish a universal threshold or remedy.
MemGym · May 2026
Evaluates memory inside tool-use dialogue, research, coding, and computer-use agents while separating memory from reasoning and tool skill. Supports testing Demi on real agent workflows rather than isolated recall.
Memex(RL) · Mar 2026
Explores compact indexed experience summaries that point to external full-fidelity evidence. It motivates reference-preserving compaction, but its results do not make that a production standard.
StructMemEval · Feb 2026
Examines structured memory such as ledgers, lists, and trees. Supports typed checkpoints and projections where generic chunk retrieval struggles.
The Horizon Gap · Aug 2026
Separates long-horizon tasks, long-context models, and long-term memory systems across a broad literature review. Useful taxonomy and very current, but too recent to treat as settled consensus.
OpenAI tool search and Agents SDK tools · current docs checked Sep 2026
Show deferred tool and namespace loading. Evidence that progressive schemas are implementable, not that OpenAI’s defaults are correct for Demi.
Anthropic tool-context management and context editing · current docs checked Sep 2026
Distinguish tool search, programmatic calls, caching, clearing, and compaction. Useful implementation taxonomy; provider-specific and not independent validation.
Google ADK session state · current docs checked Sep 2026
Demonstrates separation between event history and mutable session scratchpad/state. An implementation reference, not comparative evidence.
Bottom line
The reliable conclusion is composition, not a product feature.
Current evidence favors separating authoritative state, recent raw context, optional retrieval, progressive capabilities, and durable external memory. It does not justify one vector store, one compactor, one provider, or one context-size number as the answer.