FormVoiceAgentBench

A Hindi benchmark for FormBharo, a voice agent that fills a structured enrollment form over a phone call to enroll low-income, Hindi-speaking mothers in antenatal and postnatal care. It pairs 380 human-recorded Hindi audio clips with 3,760 multi-turn conversation tests across 960 simulated calls, and measures speech-to-text, data extraction accuracy, form completion accuracy, and response quality.

Leaderboard

Results for the configuration the paper selected. The Evaluation section below shows how each model changes across speech-to-text models.

Dataset

Five simulated users each have fixed personal details for the 12 form fields. The form branches, so each user takes 48 paths through it: pregnant or not (2), whether the calling number is linked to the clinic (3), whether the WhatsApp number is the same as the calling number (4), and whether the caller knows her Aadhaar digits (2). That gives 5 × 48 = 240 calls.

Every answer was recorded by native Hindi speakers under four acoustic conditions: ideal, background noise, microphone distance, and speaking pace. Each call uses one condition throughout, so the 240 calls become 240 × 4 = 960.

Each benchmark is published on Calibrate, the open-source package used to run these evaluations. Open a benchmark and click Export results to download it.

Every call fills the same maternal and child health enrollment form. Some questions are asked only after certain answers, and each question allows one or two retries before the call moves on or ends.

Evaluation

Each component is first tested in isolation, assuming its inputs are perfect. The agent is then tested end to end: the reference transcripts in the test inputs are replaced with transcripts from the speech-to-text models, so transcription errors carry into the LLMs.

A speech-to-text model transcribes the caller's speech. Each model transcribes the 380 recordings, and its transcripts are compared with the reference transcripts.

WER is commonly used to compare speech-to-text models, but it counts every surface difference as an error, such as “nine” against “9”, even when the meaning is unchanged. LLM-WER and LLM-CER are robust to these differences. The two disagree here: Nova-3 has the best WER yet the second-worst LLM-WER, partly because the reference transcripts are in Romanized Hindi while the predictions are in Devanagari. Models are therefore ranked by LLM-WER. Chirp 3 is the most accurate but costs at least twice as much as any other model, while Scribe v2 comes close at a fraction of the cost. Saaras v3, Scribe v2 and Nova-3 are carried into the tests below.

An LLM extracts the relevant form values from the transcribed text. Each test is a single turn of a call: the agent receives the conversation so far, and the values it extracts are compared with the expected values for that turn. Closed-ended values such as numbers, dates and yes or no answers are scored by exact match. Open-ended values such as names can be spelled many ways (“Lakshmi” or “Laxmi”), so an LLM judge scores them instead.

Frontier models perform almost perfectly when transcription is perfect. With real transcripts, accuracy drops: the median drop is 0.42 points with Scribe v2 and 3.14 points with Saaras v3 and Nova-3. Weaker models degrade much more: GLM-5.1 collapses by 35 points with Scribe v2 transcripts. Real transcripts also change the leaderboard. GPT-5.5 leads with perfect transcription (99.79%), while Gemini 3.5 Flash and Claude Sonnet 4.6 lead with Scribe v2 (98.94%).

The agent returns the extracted values by calling the process_user_turn tool, along with a short acknowledgement. Each test checks the values in that tool call.

Form completion accuracy

Testing each turn in isolation is not enough, because errors in one turn carry into the final form. For each call, starting from an empty form, the values extracted at each turn are written into it in order. Comparing the final form with the expected values gives the form completion accuracy: the share of form fields filled in correctly at the end of the call.

The best model end to end differs from the best model per turn. With perfect transcription, GPT-5.5 leads extraction accuracy (99.79%), whereas Gemini 3 Flash and Gemini 3.5 Flash tie for the highest form completion accuracy at 100%. The rule-based layer recovers turn-level errors: every extraction error by Gemini 3 Flash was a number returned as text, which the rules convert before storing the value. Form completion degrades more than extraction with real transcripts: the median drop is 7.67 points with Saaras v3, 7.38 with Scribe v2 and 13.56 with Nova-3, compared with 0.42 to 3.14 points per turn.

A second LLM phrases the next question, which a text-to-speech model speaks back to the caller. If every field has been answered, it ends the call instead. Along with the conversation so far, each test passes in the rule-based layer's decision about which question to ask next. Of the 1,880 tests, 920 require a reply and 960 check whether the call is ended at the right time. These results use Gemini 3.5 Flash for data extraction.

A reply has many correct phrasings, so binary LLM judges grade it instead of an exact match. A reply passes only if every judge listed for it passes.

The decision to end the call is recorded as a call to the end_call tool and checked by exact match.

Video walkthrough

The video walks through the paper's evaluations.

License

CC BY-SA 4.0

Citation

@article{dalmia2026formbharo,
  title   = {FormBharo: Designing and Evaluating a Voice Agent for Conversational Form Filling in Rural India},
  author  = {Dalmia, Aman and Midha, Sanskriti and Doshi, Jigar},
  journal = {arXiv preprint arXiv:2608.06027},
  year    = {2026},
  url     = {https://arxiv.org/abs/2608.06027}
}