Large first calls are routine
Across the sampled development data: minimum 27.6k, median 96.1k, p90 172.2k, and maximum 212.0k tokens. Prompt caching can reduce compute and latency, but the model still has to attend over the full input.
A concrete design for giving the model the smallest trustworthy working set without losing task, meeting, approval, or proactive-agent state.
The first prompt should teach the agent how to find authoritative state, evidence, procedures, and capabilities after it sees any request. Keep the context assembler as a budget and provenance layer. It may attach obvious scope, but it should not need to predict the full intent before the agent can work.
There is no settled industry standard. This recommendation is the convergence of current mechanisms and evidence, not a vendor recipe. Its thresholds must be validated against Demi traces.
The design target is an agent that can handle unknown, ambiguous, and cross-domain requests. The question for every always-on segment is whether it helps the agent discover, acquire, verify, or use the right context across that full space.
| First-call capability | General-purpose relevance | Correct representation |
|---|---|---|
| Operating model: intent, evidence, authority, approvals, recovery | 5/5 | One compact, non-duplicated core with backend enforcement. |
| Context-source manifest + acquisition protocol | 5/5 | Always show what can be known, its authority and freshness, and how to fetch it. |
| Current request + bounded recent conversation | 5/5 | Always present verbatim so the agent can form its own acquisition plan. |
| Authoritative tasks, meetings, approvals, files, memory, and integrations | 5/5 | Advertise every source; attach only explicit surface state and fetch records on demand. |
| Capability and skill discovery | 4/5 | Compact categories, descriptions, and loading handles. |
| Full tool schemas and skill bodies | 2/5 | Load only after the agent selects the capability. |
| Raw memory, context trees, transcripts, and provider artifacts | 1/5 | Keep external; retrieve bounded, source-linked slices. |
| Duplicate policy, response contracts, and repeated context | 0/5 | Remove. |
Across the sampled development data: minimum 27.6k, median 96.1k, p90 172.2k, and maximum 212.0k tokens. Prompt caching can reduce compute and latency, but the model still has to attend over the full input.
The core should teach one reusable rule: before claiming that something does not exist, query the authoritative source named in the manifest. This fixes task-status answers and transfers to meetings, approvals, files, people, and integrations.
What is waiting, running, blocked, scheduled, or awaiting approval now. Read from authoritative projections, not memory search.
The recent turns needed to resolve references, corrections, tone, and the active line of thought.
Older messages, transcripts, files, and provider results. Retrieve only when the request needs them.
Which tools can help. Start with namespace summaries; load full schemas after the relevant group is selected.
Goals, decisions, attempts, blockers, outputs, and exact artifact references for long-running work.
What Demi may do and the precise action being approved. Preserve exact typed data and enforce outside the model.
| Approach | How it works | Where it wins | Primary failure | Verdict |
|---|---|---|---|---|
| A. Large prompt + caching | Send broad history, state, and tool schemas every turn; cache the stable prefix. | Lowest implementation complexity. High recall when the model actually attends to the right item. | Cost and latency grow; stale and irrelevant facts compete with the request. Caching reduces recompute, not attention load. | Reject |
| B. Intent router + specialist prompts | Classify the request, then select one narrow prompt and tool bundle. | Cheap, predictable, and easy to reason about for clean single-domain intents. | Misroutes ambiguous or multi-domain requests; policy and cross-domain state get duplicated. | Use only as a hint |
| C. Retrieval-first / vector RAG | Embed all state and history; retrieve top-k chunks for each turn. | Scales optional historical evidence and fuzzy recall. | False negatives are silent. Semantic similarity is a poor authority model for exact, fresh, pending state. | Evidence lane only |
| D. One agent per domain | Tasks, meetings, approvals, and chat each own a prompt, memory, and tool set. | Strong isolation and domain specialization. | Handoffs and shared decisions become distributed-systems problems; state and policy diverge. | For execution, not context |
| E. Agent-led bounded acquisition | The agent gets a context map and universal retrieval primitives; it chooses what to load while the runtime enforces budgets and authority. | Preserves generality without paying the full context cost on every turn. | Requires reliable discovery, recovery behavior, typed sources, and multi-step evaluation. | Recommend |
get_context, source inspection, search, and load_tool_group return typed, bounded resultsThe assembler should return structured lanes plus an audit manifest. Rendering them into provider-specific messages is the final adapter step.
type ContextEnvelope = {
request: UserRequest
scope: {
user_id: ID
thread_id: ID
surface: "main" | "task" | "meeting" | "approval" | "proactive"
task_id?: ID
meeting_id?: ID
approval_id?: ID
}
policy_version: string
available_context: ContextSource[]
current_state: ContextItem[]
recent_history: ContextItem[]
retrieved_evidence: ContextItem[]
active_tool_groups: string[]
budget: { limit: number; used: number; by_lane: Record<string, number> }
omissions: { lane: string; reason: string; recover_with?: ToolCall }[]
}
type ContextSource = {
name: "tasks" | "meetings" | "approvals" | "memory" | "files" | "integrations"
description: string
authority: "live" | "canonical" | "derived"
freshness: Timestamp | "on_fetch"
retrieval_tool: string
estimated_cost: "small" | "medium" | "large"
}
type ContextItem = {
source_type: "projection" | "event" | "artifact" | "memory" | "message"
source_id: ID
authority: "live" | "canonical" | "derived" | "user_claim"
observed_at: Timestamp
expires_at?: Timestamp
sensitivity: "normal" | "private" | "restricted"
token_estimate: number
admission_reason: string
content: unknown
}
Allocate the policy/safety core and request envelope first. These cannot be displaced by history or tool output.
List what context and capabilities exist, their authority and freshness, and the primitive that loads each one.
Preserve recent turns verbatim and attach explicit ids or surface state that are unambiguously in scope.
Keep source inspection, exact context lookup, search, and tool-group loading available on every turn.
After reading the request, the agent chooses a source, fetches the smallest useful slice, and widens only when the result is insufficient.
The runtime enforces tenant, authorization, sensitivity, freshness, size, and provenance before returning a typed result.
The agent sees compact capability descriptions first and receives full operation schemas only after selecting a group.
Log which sources were considered, fetched, widened, unavailable, or omitted, plus tokens, latency, and outcome.
| Lane | Starting budget | Overflow behavior |
|---|---|---|
| Policy + request | 3–4k tokens | Never evict; refactor the core if it exceeds budget. |
| Tool discovery + schemas | 0.5–1k index; ≤3k loaded | Unload unused groups; fetch another group on demand. |
| Current state | 1–2k | Keep exact in-scope records; collapse unrelated lists to counts/ids. |
| Recent raw history | 4–6k | Move oldest turns behind the latest structured checkpoint. |
| Retrieved evidence | 4–8k | Rerank, diversify sources, retain stable references for reload. |
| Total initial context | 12–16k default; p95 ≤20–24k | >40k requires a recorded reason and evaluation bucket. |
| Surface | Always include | Load only when needed | Never rely on |
|---|---|---|---|
| Main chat | Request, recent raw tail, source manifest, acquisition primitives, and explicit surface ids. | Work Index, exact task record, provider evidence, personal memory, or older conversation. | A generated recap or absence from the initial prompt as proof that nothing exists. |
| Task | Task brief, current phase, pending input, checkpoint, recent raw tail, artifact ids. | Old attempts, large outputs, external provider state. | Whole main-chat history. |
| Meeting | Meeting id, participants, phase, agenda/goal, current notes, commitments. | Transcript slices and related account/task history. | A semantic match when an exact meeting id exists. |
| Approval | Exact proposed action, parameters, target, requester, expiry, policy decision, approval status. | Supporting evidence referenced by the approval. | Summaries, embeddings, or model memory for authorization. |
| Proactive run | Trigger evidence, user preferences, relevant task/routine, cooldown, last notification/outcome. | Evidence needed to validate the opportunity. | General chat history as permission or evidence of urgency. |
Do not make a rolling prose summary the durable memory. Use three layers:
Verbatim turns for local reference resolution, corrections, conversational intent, and tone.
Goal, plan, decisions, unresolved blockers, pending user input, attempted actions, failures, and artifact ids.
Full messages, transcripts, provider results, and files addressed by stable ids and reloaded only when needed.
Call get_context with a typed request. If the source is unavailable, say so; do not convert absence from the prompt into absence in the world.
Refresh records past their TTL. Live authoritative state wins, and the model receives both the new value and the superseded claim when the conflict matters.
Retry with exact keywords, metadata, and a broader time window. Then return a scoped inability, not a confident negative.
Discard reloadable tool output, duplicate evidence, then low-ranked history. Never drop the request, policy, exact approval, or mandatory current state.
Return candidates with ids and distinguishing fields. Ask only when choosing one would materially change the answer or action.
Fall back to core + request + current surface projection + recent tail + context-fetch tool. Record the degraded path.
Optimize for reliable behavior, not just fewer tokens. Build a replay set from production traces and score each competency separately.
| Scenario family | Must demonstrate | Primary measures |
|---|---|---|
| Status and pending work | Exact active/blocked/waiting state; abstain when a source is unavailable. | State accuracy, unsupported negatives, projection freshness. |
| Corrections and updates | New value supersedes old without erasing useful history. | Update accuracy, stale-claim rate, conflict disclosure. |
| Meetings and approvals | Resolve exact entity; preserve commitments and authorization parameters. | Entity accuracy, authorization regressions, evidence traceability. |
| Long tasks | Recover goal, checkpoint, blockers, artifacts, and significant failed attempts. | Compaction recovery, long-range completion, repeated-work rate. |
| Proactive runs | Use fresh trigger evidence and preferences without inventing urgency or permission. | Precision, actionability, false-positive notifications, cooldown compliance. |
| Tool discovery | Select the right group and recover when it was not preloaded. | Tool-group miss rate, recovery turns, schema tokens, latency. |
| Retrieval stress | Find exact, temporal, cross-session, and contradictory evidence; abstain on misses. | Recall@k, reranker recall, source diversity, false certainty. |
Release gate: no regression on state/approval correctness; lower p50 and p95 input tokens; bounded latency; measurable recovery from deliberate context omission. Budget numbers change only from these results.
Evidence labels matter. Peer-reviewed benchmarks are the strongest support for measured failure modes, but remain workload/model-specific. 2026 preprints are useful directional evidence, not settled practice. Vendor docs prove a mechanism is available; they do not prove it is optimal.
Evaluates accurate retrieval, test-time learning, long-range understanding, and selective forgetting. Long-context, RAG, and external-memory agents each leave material gaps. Supports evaluating competencies separately, not choosing one universal memory technique.
Tests information extraction, multi-session reasoning, temporal reasoning, knowledge updates, and abstention. Supports explicit indexing/retrieval/reading stages plus time-aware query construction.
Finds context-length-related premature termination and model-dependent mitigation results across long-horizon search. Supports bounded context and evaluation per model; does not establish a universal threshold or remedy.
Evaluates memory inside tool-use dialogue, research, coding, and computer-use agents while separating memory from reasoning and tool skill. Supports testing Demi on real agent workflows rather than isolated recall.
Explores compact indexed experience summaries that point to external full-fidelity evidence. It motivates reference-preserving compaction, but its results do not make that a production standard.
Examines structured memory such as ledgers, lists, and trees. Supports typed checkpoints and projections where generic chunk retrieval struggles.
Separates long-horizon tasks, long-context models, and long-term memory systems across a broad literature review. Useful taxonomy and very current, but too recent to treat as settled consensus.
Show deferred tool and namespace loading. Evidence that progressive schemas are implementable, not that OpenAI’s defaults are correct for Demi.
Distinguish tool search, programmatic calls, caching, clearing, and compaction. Useful implementation taxonomy; provider-specific and not independent validation.
Demonstrates separation between event history and mutable session scratchpad/state. An implementation reference, not comparative evidence.
Current evidence favors separating authoritative state, recent raw context, optional retrieval, progressive capabilities, and durable external memory. It does not justify one vector store, one compactor, one provider, or one context-size number as the answer.