Shrink the always-on prompt
Keep only universal behavior, privacy, and approval invariants. Move integration, meeting, document, routine, and tool procedures behind existing on-demand mechanisms. Target: no more than 4,000 tokens.
Architecture note · Proposed
Bounded context assembly for Personal Demi
Make every model prompt a small, request-specific projection. Keep the full capability surface, but reveal tools, procedures, and state only when the turn needs them.
Decision
Treat model context as a disposable view of durable system state, not as the system of record.
User message
→
Core policy + current state + bounded recent history
→
Answer directly
or retrieve evidence
or retrieve evidence
The direction matches current guidance from Anthropic and OpenAI: keep a small stable core, defer large tool catalogs, and retrieve detail just in time.
Four changes
Make tool visibility progressive
Start normal turns with at most ten universal tools. Load integrations, meetings, communications, files, documents, web, sandbox, workflows, and Mac context only when relevant. Visibility changes; backend authorization does not.
Stop adding current state to history
Provide one ephemeral block per turn: current time, a compact Work Index of up to 30 open tasks, and meeting, approval, or view state when relevant. Fetch detailed Task Briefs only for a specific task.
Bound conversation history
Compact around 20,000 tokens, preserving unresolved requests, corrections, decisions, durable references, and a recent raw tail. Remove stale state, reloadable guidance, reasoning, and historical artifacts.
Context boundaries
| Turn | Include up front | Retrieve if needed |
|---|---|---|
| Main answer | Core, recent chat, time, compact Work Index | Task Brief or memory file |
| Integration action | Core plus integration group | Operation schema, guidance, relevant memory |
| Task continuation | Core, Task Brief, pending input, recent task tail | Provider or meeting evidence |
| Meeting question | Core, recent chat, meeting identity | Transcript passages and related tasks |
| Approval response | Exact action, decision, authorization evidence | Live provider preview |
| Proactive run | Trigger evidence and task or routine state | Relevant memory and provider reads |
Meetings, approvals, proactiveness, and general memory remain separate context sources. They should not collapse into one permanent Main prompt.
Success criteria
≤4ktokens of stable instructions
≤10universal tool schemas
≤20kinput tokens at p95 for direct no-tool turns
<10sp95 latency with warm transport
Calls above 40,000 input tokens require an explicit large-input reason. Golden evaluations must reject unrelated task mentions, unsupported completion claims, and unnecessary tool calls.
Rollout
Fix state and history first
Make turn state ephemeral, remove Task Brief summaries and historical artifacts, and lower the compaction threshold.
Progressively reveal tools
Default rare tool groups to inactive while preserving the shared authorization surface.
Shrink policy and enforce budgets
Add component budgets to tests and use Context Token Attribution telemetry to catch regressions.
Release internally, then to 10%, then 100%. Compare token use, no-tool latency, tool-group misses, compaction, status accuracy, and approval regressions.
Architecture impact
Preserve
Main and Task share one capability surface. Raw chats, traces, Task Briefs, approvals, and User Files remain durable outside the prompt. Writes and approvals stay fail-closed.
Change
Capability does not imply visibility on every call. A compact full Work Index replaces durable turn-by-turn deltas. Model context becomes disposable and never acts as a second durable store.