EchoTwin

EchoTwin joins a Discord voice channel and talks back in a cloned voice. You speak, it answers — fast enough that it feels like a call rather than a request-response cycle.

The pipeline is VAD → local streaming ASR → addressee decision → LLM (with tool calls) → cloned-voice TTS, and every stage of it is on the clock.

A live run, traced stage by stage: the ASR transcript, the addressee decision and its score, tool calls resolving mid-turn, and the latency breakdown for each reply.

Latency, measured

Voice agents live or die on the gap after you stop talking. The numbers below are p50 over real production turns, on an M-series Mac over a residential connection — reproducible via the bench scripts in the repo, not vendor figures.

	p50
First voice (cached filler)	~0.6 s after end of speech
Full reply, pipeline only	~1.2 s (ASR 19 ms · LLM 971 ms · Fish TTS 174 ms · playout 40 ms)
Full reply, mouth-to-ear	~1.75 s, adding the deliberate 550 ms end-of-turn wait
Fastest logged production turn	361 ms endpoint → first audio

The useful result is where the time isn’t. TTS is 10% of the pipeline and never the bottleneck; the two dominant terms are the LLM’s first sentence (55%) and a deliberate VAD end-of-turn wait (31%) — a conversation-design choice, not a cost. What buys the rest is speculative ASR/LLM execution, pre-opened TTS sockets, and cached fillers.

Production log aggregates over-state pipeline cost, and the latency report explains why: llm_first_delta→first_audio silently includes the LLM still composing the rest of the sentence, because the chunker only hands TTS complete sentences.

Knowing when to speak

One-on-one is the mode it’s best at. The hard problem is a room: in a group channel, most of what the bot hears isn’t addressed to it. Three layers decide:

1. Reflexes — a table lookup settles the obvious cases instantly. Its name at a sentence edge, or being alone with you, always gets a reply.
2. Arbiter — ambiguous utterances go to a fast LLM (Groq qwen3-32b, ~350 ms) that reads the room’s recent transcript before deciding.
3. Heuristics — a golden-set-tested ruleset backstops failures of the layer above.

Rejected chatter isn’t discarded — it feeds a rolling ambient transcript, so when the bot does answer, it already knows what the room was talking about. Open questions to the room wait ~1.5 s and yield the floor to humans first. Utterances queued while the bot is talking merge into one reply instead of stacking up.

It ships on by default and it works, but it’s the part still under active development — expect it to occasionally read a room wrong.

The rest of it

• Barge-in — speak while it’s replying and it stops mid-sentence. A backchannel filter drops “嗯” / “ok” / “yeah” and anything under 600 ms, so a casual nod doesn’t truncate the answer.
• Hot-swappable personas — one command switches character, clearing histories and refreshing wake words, the addressee detector, and the cached fast-response audio.
• Tool calls — time, date, weather, mid-conversation.
• Cost tracking — per-turn spend with daily and monthly budget caps, queryable from Discord.
• Hot reload — kill -HUP or a slash command re-reads config and persona files; voice id, wake-word mode, whitelist and co-owners all persist across restarts.
• Localized commands — Discord renders slash commands in the client’s language (English / Simplified Chinese), and a new locale is one strings file away.

Pinned on purpose

Three dependencies are pinned to exact versions, each with the reason in the manifest. discord.py, discord-ext-voice-recv and davey get monkey-patched at the internals level, so a silent upgrade can break audio decryption outright. And sherpa-onnx is held at 1.13.2 because 1.13.4 decodes garbage on macOS arm64 — verified with a raw-API repro against clean audio, with a harness test to re-check before anyone bumps it.

Local models do the listening (VAD + streaming ASR, ~440 MB); the only outbound calls are the LLM and Fish Audio.