Building TruGen's Chime-01: How We Taught Our AI Teammates When to Speak in Group Calls

Sit in on any group video call and watch what humans actually do. We don't talk in clean turns. We trail off mid-sentence, finish each other's thoughts, raise eyebrows in lieu of words, and somehow still know, most of the time, when it's our turn to jump in. The cues are everywhere: tone, pace, eye contact, who looked at whom, the tiny pause after a question.

Now drop an AI agent into that meeting and ask it to do the same thing.

That's the problem we set out to solve at TruGen. Internally, we've been calling it Chime-01: a small, fast, fine-tuned model that lives inside our group-call agents and answers exactly one question, hundreds of times a minute: given everything that just happened in this meeting, should the AI speak right now, or stay silent?

This post is about how we built it: what we tried first, why it didn't work, and how a 0.8-billion-parameter model fine-tuned on the right data ended up doing a job that 120-billion-parameter foundation models couldn't quite do well enough.

Why "should I speak?" is harder than it looks

When customers add a TruGen agent (call her Lisa) to their Google Meet, Zoom, or Teams call, Lisa is hearing live transcripts of everyone in the room. She needs to know:

• When someone says "Lisa, can you summarize the last five minutes?", she should speak.
• When two engineers are debating a deployment back and forth, she should stay silent and not interrupt.
• When someone says "I was talking to Lisa last week about this", she should stay silent. The name came up, but she wasn't being addressed.
• When someone asks a follow-up two turns after Lisa's last response, she should speak again, recognizing the conversational thread even though her name isn't repeated.

These aren't edge cases. They're the bulk of every real meeting. Get them wrong and the agent feels broken in two opposite ways: an agent that interrupts is annoying, and an agent that goes silent when called is useless. There's no soft middle. Every chime-in decision is a yes or a no, made in real time, with the next decision coming a fraction of a second later.

Our first attempt: prompt a big model and see what happens

The natural first step was to send the rolling transcript to a strong general-purpose LLM, ask it via prompt whether the agent should speak, parse the answer, and act on it. We ran extensive experiments with foundation models including gpt-oss-120b served via Groq and Gemini Flash variants, with carefully iterated prompts and few-shot examples.

Two problems showed up almost immediately, and they turned out to be structural rather than tunable.

Latency. Just for the chime-in decision, not the actual response, these models added 300–400ms of round-trip time. Our budget for the entire turn (transcript chunk arriving → chime-in decision → LLM response → TTS audio playing in the room) was under one second. Spending nearly half of that on a binary yes/no, before we'd even started generating anything, was a non-starter.

Accuracy on the long tail. Direct name-calling worked fine. The moment we got into follow-ups, ambiguous addressees, and two-person back-and-forth, the failure modes piled up. We saw false positives where the agent tried to insert itself into private exchanges between humans. 

We saw missed chime-ins where a perfectly clear conversational follow-up didn't trigger a response. We saw the agent answer when its name was mentioned in passing ("I was telling Lisa earlier…") because the prompt couldn't reliably separate being addressed from being referenced.

We tuned prompts, swapped models, added few-shot examples, and restructured context windows. Nothing closed the gap. The honest read was that we were asking a generalist to do a niche job, and the niche was specific enough that no amount of prompting would get us where we needed to be.

Going small and specific

The decision was to fine-tune a small, dedicated model that does this one thing extremely well, runs inside our own infrastructure, and fits comfortably inside our latency budget.

We chose Qwen3.5-0.8B as the base: small enough to be fast on commodity GPUs, and large enough to handle the contextual reasoning the task actually needs.

The technology behind Chime-01

Under the hood, Chime-01 is a deliberately simple model architecture. We started with Qwen3.5-0.8B and fine-tuned it using LoRA, rather than training a model from scratch.

Instead of adding a traditional classification head, we added two tokens to the model's vocabulary:

chimein → the agent should speak

silent → the agent should stay quiet

During training, each conversational example is paired with one of these tokens as the expected output. This teaches the language model to map the current conversational state to one of two decisions.

The interesting part is how we use the model at inference time: we don't actually generate text.

A normal LLM generates a sequence of tokens autoregressively. For Chime-01, we only care about the logits produced for our two decision tokens at the next-token position:

logit<|chimein|>

logit<|silent|> 

We then apply a softmax over those two logits, which gives us a direct probability for the decision:

chimein ->  0.87

silent ->    0.13

The agent runtime then applies a configurable threshold to determine whether the main agent should be invoked.

This is an important distinction: Chime-01 isn't generating a response. It's using the language model's internal token distribution as a decision signal.

That makes the model extremely lightweight to run in a real-time pipeline:

• No parsing. The decision is a single, unambiguous signal, with no "Yes, the agent should…" preamble to strip and no off-by-one risks.
• No additional classifier network. The decision comes directly from the language model itself.
• No waiting on an autoregressive response. The moment the decision is available, the next stage of the pipeline can begin.

The result is a small model with a very narrow responsibility: decide whether the conversation has reached a point where the AI should participate.

The dataset is the product

The model architecture got us into the fight; the dataset is what won it. A foundation model fails at chime-in not because it lacks intelligence, but because it has never been shown enough of the exact distribution of meeting transcripts where this question actually matters. Building that distribution was the most consequential work in the project.

A few choices mattered more than the rest.

A blend of synthetic and real transcripts. Real meeting data captured the messiness: people talking over each other, ambient noise transcribed as filler, half-finished sentences, and ASR errors that look like content. Synthetic data let us deliberately construct rare-but-critical cases at scale: third-person mentions of the agent, ambiguous "what about you?" follow-ups, name collisions, and multi-party debates that the agent should not interrupt. Neither source on its own would have been enough.

Dynamic agent-name substitution. Customers configure their own agent names: Lisa, Joy, Aria, anything they like. The model can't be allowed to overfit on any single string. During training, we rotated through a pool of different agent names, randomly substituting them into each example. The model learned the role of the addressed agent, not any one literal name.

Strict transcript-level splits. The fundamental risk in any conversational dataset is leakage between train and test: the same speakers, topics, and vocabulary drifting across the partition and quietly inflating the numbers. We split at the level of whole transcripts, not individual examples, so that no conversation appeared on both sides of the split.

Balanced training, natural-distribution evaluation. Real-world chime-in is heavily imbalanced; an agent should stay silent far more often than it speaks. We deliberately balanced the training set 50/50 between chime-in and silent, so the model learned the decision boundary with equal pressure from both sides, then evaluated against the natural, imbalanced distribution to make sure the numbers reflected real conditions.

What the numbers look like

On a held-out test set drawn from transcripts the model had never seen, with their own speakers and topics, the fine-tuned Chime-01 model hits:

• 97.9% accuracy
• 0.975 macro F1

Forward-pass inference on our serving infrastructure runs at roughly 50ms, an order of magnitude faster than the foundation-model baselines we started with.

The number that actually matters to a customer, though, is the end-to-end one. From the moment a participant stops speaking to the moment the agent's voice is back in the room (transcript chunk arriving → Chime In decision → LLM generating a response → TTS streaming audio back into the meeting), the full loop runs under one second. The detector's tight inference budget is what makes that number possible. The decision is essentially "free" against the rest of the pipeline.

Beyond the headline metrics, the qualitative changes are what we actually shipped this for. The agent now:

• Responds confidently when directly addressed by name.
• Picks up on follow-up questions across multiple turns, even when its name isn't repeated.
• Holds back when two humans are mid-debate, instead of barging in on the first pause.
• Stays silent when its name is mentioned in passing rather than actually being addressed.

Where this fits

The chime-in detector sits between the live meeting and the main agent. The main LLM is only invoked when the detector returns <|chimein|>

The chime-in detector is the silent skill that makes our group-call agents usable in real settings: internal stand-ups where the agent answers when called and stays out of the way otherwise, customer support sessions where it jumps in to clarify product details on cue, and panel discussions where it acts more like a participant than a button. 

The agent can join any Google Meet, Zoom, or Teams call, take any name the customer chooses, and behave like someone who's actually sat through meetings before.

Conclusion

Turn-taking is the unglamorous, decisive problem in conversational AI. Fluent voices and expressive avatars are necessary; they're not sufficient. An agent that talks at the wrong time is the one that gets removed from the calendar invite next week.

The chime-in detector is our bet that the right way to solve this is small, fast, dedicated models trained on data that captures the actual texture of human conversation, not bigger general-purpose ones bent into the wrong shape.

If you're building a product where a real-time AI needs to live inside a group conversation, we'd love to talk. The chime-in detector now powers every TruGen group-call agent by default. You can try it through our portal, or reach out to our team to talk through deploying it in your own meeting flows.