Large first calls are routine
Across the sampled development data: minimum 27.6k, median 96.1k, p90 172.2k, and maximum 212.0k tokens. Prompt caching can reduce compute and latency, but the model still has to attend over the full input.
A concrete design for giving the model the smallest trustworthy working set without losing task, meeting, approval, or proactive-agent state.
Answer canonical state questions directly from authoritative projections. For open-ended reasoning and actions, compose the turn from mandatory current state, a bounded raw conversation tail, progressively loaded tool schemas, and optional retrieved evidence. Both paths use the same sources and authorization boundary.
There is no settled industry standard. This recommendation is the convergence of current mechanisms and evidence, not a vendor recipe. Its thresholds must be validated against Demi traces.
For the request “Are there any tasks pending on me?”, only the current request, the materialized Work Index, and the task-state interpretation rules are essential. The latest main prompt at 9ef2aeb8 sends a much larger general-purpose surface before the model can answer.
| First-call segment | Approximate size | Relevance to this request | Design implication |
|---|---|---|---|
| Current request + Work Index + task-state rules | Small, data-dependent | 5/5 | Make this the complete input to the canonical state path. |
| Evidence and no-bluff policy | Part of shared policy | 3/5 | Keep a compact invariant; enforce authoritative reads in code. |
| Time, routing, and response-format guidance | Several overlapping blocks | 1–2/5 | Include only fields that change the answer; use one output contract. |
| 42 tool schemas | ~27.9k tokens | 0/5 | Expose zero tools for this path. Make load_tool_group genuinely progressive elsewhere. |
| 42-item skill catalog | ~3.7k tokens | 0/5 | Keep the catalog out of routine state turns; load a skill by intent. |
| Identity, priorities, style, people files, context tree | Unbounded in several lanes | 0/5 | Retrieve personal context only when it can change the response or action. |
| Meeting context | Up to two 40k transcript copies | 0/5 | Deduplicate the two injection paths and admit transcript slices only for meeting intents. |
Across the sampled development data: minimum 27.6k, median 96.1k, p90 172.2k, and maximum 212.0k tokens. Prompt caching can reduce compute and latency, but the model still has to attend over the full input.
Compute pending = tasks where pending != "none". If empty, answer “Nothing needs you right now.” Do not pad the response with recently completed work or infer completion from chat history.
What is waiting, running, blocked, scheduled, or awaiting approval now. Read from authoritative projections, not memory search.
The recent turns needed to resolve references, corrections, tone, and the active line of thought.
Older messages, transcripts, files, and provider results. Retrieve only when the request needs them.
Which tools can help. Start with namespace summaries; load full schemas after the relevant group is selected.
Goals, decisions, attempts, blockers, outputs, and exact artifact references for long-running work.
What Demi may do and the precise action being approved. Preserve exact typed data and enforce outside the model.
| Approach | How it works | Where it wins | Primary failure | Verdict |
|---|---|---|---|---|
| A. Large prompt + caching | Send broad history, state, and tool schemas every turn; cache the stable prefix. | Lowest implementation complexity. High recall when the model actually attends to the right item. | Cost and latency grow; stale and irrelevant facts compete with the request. Caching reduces recompute, not attention load. | Reject |
| B. Intent router + specialist prompts | Classify the request, then select one narrow prompt and tool bundle. | Cheap, predictable, and easy to reason about for clean single-domain intents. | Misroutes ambiguous or multi-domain requests; policy and cross-domain state get duplicated. | Use only as a hint |
| C. Retrieval-first / vector RAG | Embed all state and history; retrieve top-k chunks for each turn. | Scales optional historical evidence and fuzzy recall. | False negatives are silent. Semantic similarity is a poor authority model for exact, fresh, pending state. | Evidence lane only |
| D. One agent per domain | Tasks, meetings, approvals, and chat each own a prompt, memory, and tool set. | Strong isolation and domain specialization. | Handoffs and shared decisions become distributed-systems problems; state and policy diverge. | For execution, not context |
| E. Hybrid bounded assembler | Typed state is deterministic; history is bounded; evidence is retrieved; tools are progressive; artifacts remain external. | Balances correctness, latency, inspectability, and cross-domain work. | Requires provenance, projections, budgets, evaluation, and observability infrastructure. | Recommend |
get_context(type, ids, query, time_range) and load_tool_group(namespace) let the model recover from deliberate under-inclusion.The assembler should return structured lanes plus an audit manifest. Rendering them into provider-specific messages is the final adapter step.
type ContextEnvelope = {
request: UserRequest
scope: {
user_id: ID
thread_id: ID
surface: "main" | "task" | "meeting" | "approval" | "proactive"
task_id?: ID
meeting_id?: ID
approval_id?: ID
}
policy_version: string
current_state: ContextItem[]
recent_history: ContextItem[]
retrieved_evidence: ContextItem[]
active_tool_groups: string[]
budget: { limit: number; used: number; by_lane: Record<string, number> }
omissions: { lane: string; reason: string; recover_with?: ToolCall }[]
}
type ContextItem = {
source_type: "projection" | "event" | "artifact" | "memory" | "message"
source_id: ID
authority: "live" | "canonical" | "derived" | "user_claim"
observed_at: Timestamp
expires_at?: Timestamp
sensitivity: "normal" | "private" | "restricted"
token_estimate: number
admission_reason: string
content: unknown
}
Allocate the policy/safety core and request envelope first. These cannot be displaced by history or tool output.
Use surface, explicit ids, reply targets, attachments, active task, meeting phase, and pending approval. Emit ambiguity instead of inventing scope.
Read exact projections for in-scope tasks, meetings, approvals, and proactive triggers. Apply freshness TTLs and authority tags.
Preserve the most recent turns verbatim. Attach the latest structured checkpoint for older execution state.
Rules pre-load mandatory groups; a compact catalog exposes other groups. Full schemas load only when selected.
Search with entity, tenant, time, source, and sensitivity filters; combine lexical and semantic scores; rerank; deduplicate against current state.
Trim within lanes rather than globally. Drop reloadable outputs and low-authority history before current state, corrections, approvals, or the user request.
Log tokens by component, admitted source ids, omissions, freshness, retrieval queries, active tools, latency, and downstream outcome.
| Lane | Starting budget | Overflow behavior |
|---|---|---|
| Policy + request | 3–4k tokens | Never evict; refactor the core if it exceeds budget. |
| Tool discovery + schemas | 0.5–1k index; ≤3k loaded | Unload unused groups; fetch another group on demand. |
| Current state | 1–2k | Keep exact in-scope records; collapse unrelated lists to counts/ids. |
| Recent raw history | 4–6k | Move oldest turns behind the latest structured checkpoint. |
| Retrieved evidence | 4–8k | Rerank, diversify sources, retain stable references for reload. |
| Total initial context | 12–16k default; p95 ≤20–24k | >40k requires a recorded reason and evaluation bucket. |
| Surface | Always include | Load only when needed | Never rely on |
|---|---|---|---|
| Main chat / status | Compact work index: active, blocked, pending input, next scheduled event; freshness timestamp. | Exact task record, provider evidence, or older conversation after reference. | A generated recap as proof that nothing is pending. |
| Task | Task brief, current phase, pending input, checkpoint, recent raw tail, artifact ids. | Old attempts, large outputs, external provider state. | Whole main-chat history. |
| Meeting | Meeting id, participants, phase, agenda/goal, current notes, commitments. | Transcript slices and related account/task history. | A semantic match when an exact meeting id exists. |
| Approval | Exact proposed action, parameters, target, requester, expiry, policy decision, approval status. | Supporting evidence referenced by the approval. | Summaries, embeddings, or model memory for authorization. |
| Proactive run | Trigger evidence, user preferences, relevant task/routine, cooldown, last notification/outcome. | Evidence needed to validate the opportunity. | General chat history as permission or evidence of urgency. |
Do not make a rolling prose summary the durable memory. Use three layers:
Verbatim turns for local reference resolution, corrections, conversational intent, and tone.
Goal, plan, decisions, unresolved blockers, pending user input, attempted actions, failures, and artifact ids.
Full messages, transcripts, provider results, and files addressed by stable ids and reloaded only when needed.
Call get_context with a typed request. If the source is unavailable, say so; do not convert absence from the prompt into absence in the world.
Refresh records past their TTL. Live authoritative state wins, and the model receives both the new value and the superseded claim when the conflict matters.
Retry with exact keywords, metadata, and a broader time window. Then return a scoped inability, not a confident negative.
Discard reloadable tool output, duplicate evidence, then low-ranked history. Never drop the request, policy, exact approval, or mandatory current state.
Return candidates with ids and distinguishing fields. Ask only when choosing one would materially change the answer or action.
Fall back to core + request + current surface projection + recent tail + context-fetch tool. Record the degraded path.
Optimize for reliable behavior, not just fewer tokens. Build a replay set from production traces and score each competency separately.
| Scenario family | Must demonstrate | Primary measures |
|---|---|---|
| Status and pending work | Exact active/blocked/waiting state; abstain when a source is unavailable. | State accuracy, unsupported negatives, projection freshness. |
| Corrections and updates | New value supersedes old without erasing useful history. | Update accuracy, stale-claim rate, conflict disclosure. |
| Meetings and approvals | Resolve exact entity; preserve commitments and authorization parameters. | Entity accuracy, authorization regressions, evidence traceability. |
| Long tasks | Recover goal, checkpoint, blockers, artifacts, and significant failed attempts. | Compaction recovery, long-range completion, repeated-work rate. |
| Proactive runs | Use fresh trigger evidence and preferences without inventing urgency or permission. | Precision, actionability, false-positive notifications, cooldown compliance. |
| Tool discovery | Select the right group and recover when it was not preloaded. | Tool-group miss rate, recovery turns, schema tokens, latency. |
| Retrieval stress | Find exact, temporal, cross-session, and contradictory evidence; abstain on misses. | Recall@k, reranker recall, source diversity, false certainty. |
Release gate: no regression on state/approval correctness; lower p50 and p95 input tokens; bounded latency; measurable recovery from deliberate context omission. Budget numbers change only from these results.
Evidence labels matter. Peer-reviewed benchmarks are the strongest support for measured failure modes, but remain workload/model-specific. 2026 preprints are useful directional evidence, not settled practice. Vendor docs prove a mechanism is available; they do not prove it is optimal.
Evaluates accurate retrieval, test-time learning, long-range understanding, and selective forgetting. Long-context, RAG, and external-memory agents each leave material gaps. Supports evaluating competencies separately, not choosing one universal memory technique.
Tests information extraction, multi-session reasoning, temporal reasoning, knowledge updates, and abstention. Supports explicit indexing/retrieval/reading stages plus time-aware query construction.
Finds context-length-related premature termination and model-dependent mitigation results across long-horizon search. Supports bounded context and evaluation per model; does not establish a universal threshold or remedy.
Evaluates memory inside tool-use dialogue, research, coding, and computer-use agents while separating memory from reasoning and tool skill. Supports testing Demi on real agent workflows rather than isolated recall.
Explores compact indexed experience summaries that point to external full-fidelity evidence. It motivates reference-preserving compaction, but its results do not make that a production standard.
Examines structured memory such as ledgers, lists, and trees. Supports typed checkpoints and projections where generic chunk retrieval struggles.
Separates long-horizon tasks, long-context models, and long-term memory systems across a broad literature review. Useful taxonomy and very current, but too recent to treat as settled consensus.
Show deferred tool and namespace loading. Evidence that progressive schemas are implementable, not that OpenAI’s defaults are correct for Demi.
Distinguish tool search, programmatic calls, caching, clearing, and compaction. Useful implementation taxonomy; provider-specific and not independent validation.
Demonstrates separation between event history and mutable session scratchpad/state. An implementation reference, not comparative evidence.
Current evidence favors separating authoritative state, recent raw context, optional retrieval, progressive capabilities, and durable external memory. It does not justify one vector store, one compactor, one provider, or one context-size number as the answer.