Jarvis — Coordination Deserves a Brain of Its Own in Agentic Interactive Models

τ-Voice · same front model

192/278

tasks solved on Venus-Audio when only the harness changes. Qwen Audio Agent solves 29.

Full-Duplex-Bench v3

69%

Pass@1 for Jarvis-Audio, ahead of GPT-Realtime at 60% and 21 points above the model it runs on.

τ-Voice public leaderboard

69.2%

macro task success, higher than every realtime speech model on the board, with an open 9B front model.

Scripted races

240/240

runs consistent under refinements, cancellations and interruptions: no stale audio, no stale tool effect, no false delivery.

Decision brain

91%

of JEV decisions are correct and confident within one second. General LLM controllers manage 0%.

Jarvis-Omni

+22.7pp

correct answers come from the screen, and advice whose visual evidence has expired is never spoken.

Overview

Coordination deserves a brain of its own

Interactive models listen and speak at once, but keeping pace with a conversation leaves them little capacity for tool use, multi-step reasoning or memory. Agentic interactive models therefore keep such a model in front and hand that work to a slow brain in the background, a larger model or a full agent harness. This split raises a coordination problem, because the conversation goes on while the work runs. By the time a result is ready, the user may have refined the request, cancelled it or moved on. Neither side can handle this alone, since the front model knows the conversation but not the tasks, while the back end knows its task but not the conversation.

Jarvis is a slow-brain harness that gives coordination a brain of its own. Each request becomes a versioned task in the Task Ledger, and each change to it becomes an operation on the ledger, such as refining version 2 of a game. The harness follows one rule, models judge, code enforces. A small adapter on the front model's control tokens judges when a turn should reach the slow brain. JEV, a decision model that sees both the conversation and the tasks, then picks the turn's operation from options the runtime builds from the ledger. Code enforces the outcome, rejecting operations and results tied to a replaced version and counting an answer as delivered only when playback receipts show the user heard it.

We build Jarvis-Audio on full-duplex speech models and Jarvis-Omni on an audio-visual model. Since what the model sees can also change while the work runs, Jarvis-Omni checks every visual fact again before speaking. With the front model and back end held fixed, replacing Qwen Audio Agent with Jarvis raises τ-Voice task success from 29 to 192 of 278 tasks. Jarvis-Audio reaches 69% Pass@1 on Full-Duplex-Bench v3, ahead of GPT-Realtime at 60%.

The rule

Models judge,
code enforces.

When. The Native Gate reads the front model's own control tokens and hands a turn to the slow brain once the request is complete.
What. JEV picks one of the operations the runtime wrote from the Task Ledger, such as REFINE:snacks:1. It never writes an instruction.
Whether. The runtime applies the choice only if that task version is still live, and accepts a result only for the version that asked for it.
Heard. A task is delivered only when playback receipts show the user heard all of it.

The problem

Six ways a voice agent goes wrong while work is running

These are not recognition errors and not reasoning errors. They happen because two processes move at different speeds and nothing keeps them in step. The first five arise with audio alone; the sixth appears once the agent can see.

Stale delivery

The first draft of a game is presented after the user has already asked for a different theme.

Prevented by Task Ledger versions

Misbinding

“Make it sixty dollars” changes the game instead of the snack plan.

Prevented by JEV over version-bound options

Ghost execution

A plan formed before the user corrected the order number is executed anyway.

Prevented by Freshness check on the plan

Lost delivery

The user cuts an answer off halfway, yet the task is recorded as delivered.

Prevented by Playback receipts and epochs

Memory drift

“Only sixty tonight” overwrites the usual budget of forty.

Prevented by Gated memory writes

Expired evidence Omni

Advice to push forward arrives after the teammate on screen has died.

Prevented by Belief State validity windows

All six share one cause: the system acts on a picture of the conversation that has stopped being true. Jarvis makes the current state explicit and checks every action against it.

How it works

Two brains, one harness

The fast brain keeps the conversation going and decides when to hand a turn over. The slow brain interprets the turn with JEV, records the work in the Task Ledger, runs it in the background and returns the answer in the fast brain's own voice, but only while the result is still valid. The two never wait for each other.

Fast talk, slow work. The amber pulse is a turn the front model answers on its own; the blue one is a delegated turn on its way through the slow brain and back to the speaker. The dashed module is used by Jarvis-Omni only. On a phone, swipe the diagram sideways.

01

Learn when, keep how

The front model already speaks well. Only the decision of when to involve the slow brain is trained, on the model's own control tokens; every weight that produces speech stays frozen.

02

Models judge, code enforces

Questions that need understanding go to a decision model, but the model answers by choosing among operations the runtime prepared. The runtime then applies the choice and checks it against the current state.

03

Nothing is assumed current

Plans, results, playback and visual facts all carry a version, an epoch or a timestamp, and each is checked again before it may act or speak.

04

Done is not delivered

A worker finishing a task and the user hearing the answer are two separate events, and the system records them separately.

Inside the harness

Five modules, one per tab

Learning when to hand off

“Build me a little game for tonight” needs the slow brain, “tell me a joke” does not, and “hold on, let me ask my roommate” is not addressed to the assistant at all. The decision has to be made from audio while the user may still be talking, and the words alone cannot tell a finished request from a pause. The front model already knows, because it is the part that decides when to speak.

So for every one-second audio unit the gate reads a three-way policy, LISTEN, DIRECT or DELEGATE, straight off the front model's native control tokens, and acts only once a branch reaches 0.8. It is a rank-8 LoRA trained with GRPO on the branch, not on a token, and it is switched on only in the routing pass, so the voice you hear comes from exactly the released weights.

98.8%first admissions at the right time and in the right branch (162 / 164)
0.07%of the model is trained: 6.2M parameters

Typed judgments, not free text

Midway through game night the user says “make it sixty dollars”. The game is on its third version, the snack plan has already been delivered and the answer to the Python follow-up has just been played. Understanding the sentence needs a model; acting on it changes real tasks, so the model must never produce an operation the runtime cannot check.

JEV never composes an instruction. The runtime writes the options from the live Task Ledger, and every option names a task and a version, so whatever JEV picks is executable by construction. Below a confidence of 0.8 the runtime does not guess: a worker takes the turn with full context and every existing task stays as it was.

0.30 smedian decision request, against 1.57 s and 3.14 s for general LLM controllers
91%correct and confident within one second; the LLM controllers manage none
252 / 256replays bind the right operation to the right task

Request

input: “For tonight, only sixty dollars is fine.”
history: last 12 turns
user_profile: [“usual snack budget is $40”, …]
question: operation Choice

Typed answer

ANSWERrespond, use tools or ask0.04
MEMORYfetch a fact from an earlier session0.01
REFINE:T5:1modify “snack plan with the usual budget” · DELIVERED0.91
CANCEL:T5:1cancel that task without replacement0.01
REFINE:T2:3modify “two-player star game” · DELIVERED0.02
STOP_SPEAKINGstop the voice, keep all work0.01

p(REFINE:T5:1) ≥ 0.8, so the runtime supersedes version 1 of the snack plan and starts version 2 with the one-night budget. The game is not touched. Probabilities are illustrative, as in the report's Figure 5.

One session, decision by decision

Game night, as the Task Ledger saw it

A recorded five-minute session from the report. In one breath the user asks for a browser game, tonight's weather in Shanghai and two technology headlines, then keeps talking while up to four tasks run. Every utterance becomes one or more typed operations, and every operation names a live task version. Hover an utterance to find it on the timeline; press play to hear it in the Game night demo.

✓ delivered, playback confirmed
? worker asks back
× cancelled
one task version
memory

Figure 2 of the report, made interactive. The game goes through four versions, the Python question gets a follow-up, a one-night budget of $60 leaves the usual $40 in memory untouched, and the cancellation removes the welcome sign and nothing else. Timeline times are seconds into the session the figure was drawn from; the Game night video is a recording of the same script, so each play button jumps to where that line is said there.

Results

The harness alone makes the agent better

Three questions. Does the harness by itself make a voice agent better, with everything else held fixed? How does it compare with the strongest published voice systems? And why does it work, mechanism by mechanism?

1

Same front model, same back end, only the harness changes

Each pair keeps the front model, the admission path and the DeepSeek-V4-Flash back end fixed and swaps Qwen Audio Agent (QAA) for Jarvis, so every difference comes from what happens to a turn after it is admitted. Jarvis never trails QAA on any benchmark, domain or task category.

97 vs 68FDB-v3 recordings that get an audible response on Venus-Audio. An admitted request is answered, asked about or reported as failed, never silently dropped.
137 vs 0of the hardest τ-Voice tasks solved, the ones that must write to the database, such as rebooking a flight. QAA solves none on any front model.
12.9 s vs 14.2 smedian task-completion time on FDB-v3. None of the gain costs time.

Full table: harnesses on the same front model (Table 2)

Front model	Harness	τ-Voice (solved)				FDB-v3		FDB-v2
Front model	Harness	Airline	Retail	Telecom	Total	Exact	Semantic	Task score
Venus-Audio	Native harness	17	6	0	23	45	35	3.09
	QAA	23	6	0	29	41	59	2.57
	Jarvis	35	77	80	192	51	63	3.36
MiniCPM-o 4.5	QAA	23	6	0	29	40	56	2.98
MiniCPM-o 4.5	Jarvis	23	6	9	38	50	63	3.28
VoiceChat-11B	QAA	1	1	0	2	26	38	2.44
VoiceChat-11B	Jarvis	23	6	4	33	34	50	2.84
Other full-duplex systems with their own runtimes
MoshiRAG	Native retrieval	0	0	0	0	0	0	3.08
Gander	Native runtime†	23	6	0	29	5	6	3.16

τ-Voice reports solved tasks out of 278, FDB-v3 passes out of 100 under exact and semantic matching, and FDB-v2 the mean task score over 300 task and pace pairs. All rows share one judge. †Released runtime with a DeepSeek-V4-Flash worker in place of its Codex back end.

2

Against leading voice systems

Jarvis-Audio reaches 69% Pass@1 on Full-Duplex-Bench v3, nine points above GPT-Realtime and 21 above the Venus-Audio model it runs on, and the margin grows with difficulty: on hard scenarios it passes 56.7% against 43.3%. On the public τ-Voice leaderboard it beats every realtime speech model. The two systems above it follow the same fast-and-slow design on proprietary frontier back ends; Jarvis gets close with an open 9B front model and a flash-tier back end.

Full-Duplex-Bench v3 Pass@1, %

τ-Voice public leaderboard macro task success, %

realtime front end with a proprietary reasoning back end

Full table: Full-Duplex-Bench v3, all metrics (Table 3)

System	Tool F1 ↑	Arg. acc. ↑	Resp. qual. ↑	Pass@1 ↑	Take-turn ↑	Interrupt ↓	Filler ↓
Commercial and cascaded systems
GPT-Realtime	0.876	0.680	0.792	0.600	96.0	13.5	16.9
Gemini Live 3.1	0.817	0.588	0.718	0.540	78.0	19.2	31.7
Gemini Live 2.5	0.786	0.593	0.554	0.490	92.0	14.1	8.9
Cascaded (Whisper, GPT-4o, TTS)	0.803	0.562	0.600	0.450	100.0	33.0	26.9
Grok	0.797	0.542	0.617	0.430	94.0	25.5	44.3
Ultravox v0.7	0.794	0.513	0.510	0.410	96.0	47.9	88.0
Open full-duplex models and agents
Venus-Audio	0.850	0.542	–	0.480	–	–	–
Venus-Omni	0.860	0.531	–	0.430	–	–	–
VoiceChat-11B	0.825	0.422	–	0.330	–	–	–
Gander	0.759	0.503	0.490	0.400	100.0	8.0	51.6
Jarvis-Audio	0.927	0.783	0.850	0.690	100.0	7.0	20.4

The first four metrics are fractions and the last three are percentages. Baseline numbers come from the benchmark release, the Realtime-Venus report and the Gander report. Bold marks the best value in each column, and – means not reported.

Full table: τ-Voice by domain (Table 4)

System	Retail ↑	Airline ↑	Telecom ↑	Macro avg. ↑
Realtime speech models
GPT-Realtime 1.0	35.96	36.00	19.30	30.42
GPT-Realtime 1.5	44.70	40.00	21.10	35.27
GPT-Realtime 2	47.37	58.00	21.93	42.43
Gemini 2.5 Flash Live	29.80	30.00	17.50	25.77
Gemini 3.1 Flash Live (thinking high)	45.61	64.00	21.93	43.85
Grok Voice Fast 1.0	38.60	36.00	40.40	38.33
Grok Voice Think Fast 1.0	62.28	66.00	73.68	67.32
Grok Voice Think Fast 2.0	59.65	56.00	71.93	62.53
Qwen3.5-Omni-Plus Realtime	46.49	54.00	60.53	53.67
Cascaded pipeline
LiveKit (Nova-3, GPT-4.1, Aura)	28.95	48.00	16.67	31.20
Realtime front end with a proprietary reasoning back end
GPT Live 1 with GPT-6 Astra	78.95	82.00	84.21	81.72
Pine Voice Preview	70.18	70.00	85.96	75.38
Jarvis-Audio (open 9B front, DeepSeek-V4-Flash)	67.54	70.00	70.18	69.24

Task success in percent on the regular tasks. Other systems are the standard voice submissions on the public leaderboard. Bold marks the best value among systems without a proprietary reasoning back end.

3

Why it works

A benchmark caller rarely changes a request while the work on it is still running, so the mechanisms are tested directly: scripted races against the real runtime, decision timing against general LLM controllers, and spoken memory.

Task Ledger checks

Each check removes a different failure

80 race scripts inject a refinement, a cancellation, a correction during planning, a task switch or an interruption during playback, three runs each against the real runtime.

Decision brain

JEV decides correctly, and in time

Same options and context for all three controllers; 192 bilingual test cases, each called three times, then replayed in the runtime.

Memory

Keeping what both speakers said

80 fixed questions about 20 ContextDialog conversations, fed as real audio, scoring the answer that was actually played (1 to 5).

Full tables: races, controllers and memory (Tables 5 to 7)

Consistency under 80 race scripts, three runs each (Table 5)
Runtime	Consistent runs ↑	Stale audio ↓	Stale tool effect ↓	False delivery ↓	Gain of full runtime (pp)
Jarvis, full runtime	240 / 240	0	0	0	–
without result validity (I2)	168 / 240	72	0	0	+30.0 [15.0, 45.0]
without plan freshness (I3)	168 / 240	72	72	0	+30.0 [15.0, 45.0]
without playback receipts (I4)	204 / 240	0	0	36	+15.0 [3.8, 30.0]

JEV against general LLM controllers (Table 6)
Controller	Threshold	Median request (s) ↓	In time, 1 s ↑	With fallback, 3.5 s ↑	Correct binding ↑	Stopped in time ↑	Replays with stale audio ↓	Unchanged tasks kept ↑
DeepSeek-V4-Flash	0.95	1.57	0 / 576	465 / 576 (80.7%)	220 / 256	77 / 224	36	32 / 32
Gemini 3.5 Flash	1.00	3.14	0 / 576	456 / 576 (79.2%)	160 / 256	9 / 224	96	32 / 32
JEV	0.43	0.30	524 / 576 (91.0%)	544 / 576 (94.4%)	252 / 256	150 / 224	4	32 / 32

Spoken conversational memory on ContextDialog (Table 7)
Harness and memory	Mean score ↑	Score ≥ 4 ↑	Evidence tokens
QAA, default Markdown memory with automatic injection	1.45	6 / 80	55
Jarvis, JEV selection	4.19	61 / 80	700
Jarvis, hybrid dense and sparse retrieval	4.41	65 / 80	1013

Jarvis-Omni

When evidence expires

In audio, the facts behind a decision are mostly in the user's words, and words do not change after they are spoken. On a screen, facts change on their own: a teammate who was alive two seconds ago is dead, and a low health bar is full again after a recall. Jarvis-Omni treats visual facts the way Jarvis-Audio treats tasks, as state with a version and a lifetime, and checks them again before anything based on them reaches the user.

0.0 s

Figure 7 of the report, animated. Claims read from the screen carry a validity window and are invalidated by events such as a death or a camera change. Advice is checked against the Belief State when it is ready, so a recommendation that rested on an expired claim is dropped and planned again from the new state. Drag the slider to scrub through the scene.

23.7% vs 1.0% correct answers with the screen and without it, over 300 questions. Two blind reviews of the played audio and the recorded screen must both accept an answer.

12,323 → 0 ms of expired audio reaching the user when the evidence expires while the answer waits in the speech queue, without and with the validity check.

75 / 96 gate decisions right on the fixed test set, with one missed action. The frozen control head of Venus-Omni gets 30 and misses 66.

5.05 → 1.78 s median time to the first spoken response in live sessions on the same user recordings, once the new scene-aware gate is in place.

≈2.2 s from the end of a tactical question to the start of the answer on the fast path (2.15 s in CS2, 2.23 s in LoL).

The answers rest on the screen

Removing only the screen frames that enter the runtime, in two independent cohorts of recorded game sessions.

First cohort, 16 uploads, 300 questions

with screen23.7%

without1.0%

Second cohort, 12 new uploads, 120 questions

with screen25.0%

without2.5%

Silent controls stay silent either way: 60 / 60 and 24 / 24.

Expired evidence stays off the speaker

Admitted answers played through the real speech scheduler and browser player, with their evidence set to expire. Expired audio in milliseconds, four replays each.

Evidence expires before playback

no check12,323

Jarvis-Omni0

Evidence expires during playback

no check9,947

Jarvis-Omni0.125

0.125 ms is three audio samples. Utterances whose evidence stays valid play in full either way.

A gate that also watches

The gate combines microphone and screen features with the dialogue so far, and adds a SCENE action to speak up unprompted. Fixed test set of 96 episodes.

Frozen Venus-Omni

30

Previous adapter

67

Contextual GRPO adapter

75

correctmissed actionacted early or in the wrong branch

In live sessions on the same user recordings, all 18 replies are heard with the new adapter, against 13 with the previous one.

The two instances side by side

	Jarvis-Audio	Jarvis-Omni
Front model	Realtime-Venus-Audio, 9B	Venus-Omni
Native Gate	Causal audio prefixes, three actions	Causal audio and screen features with dialogue context, plus a SCENE action for proactive turns
Decision brain	JEV, text	Jev-Omni, 12B, text and images
Perception state	ASR observer	ASR observer, Evidence Timeline and Belief State
Memory	Profile and task records	Adds in-match events and game knowledge
Shared: JEV controller, Task Ledger, workers, Delivery and playback control
Main evidence	τ-Voice, Full-Duplex-Bench v2 and v3, ContextDialog, race and control studies, daily use, PvZ, Luanti	Gate test set, visual and playback ablations, live CS2 and League of Legends sessions

Jarvis: Coordination Deserves a Brain of Its Own in Agentic Interactive Models

Jarvis in Action