The Open ASR Leaderboard Adds Its First Global South Language
2026-08-29 · Hugging Face
Voice Arena and Hugging Face have partnered to launch a significant update to the Open ASR Leaderboard, introducing new evaluation sets for Hindi and Indian English, collectively named 'Monsoon.' This marks a pivotal moment as Hindi becomes the first Indic and, more broadly, the first Global South language to be featured on the multilingual leaderboard, which previously only covered European languages.
The Limitations of Current ASR Benchmarks
Benchmarks fundamentally shape technological development. While the Open ASR Leaderboard has made strides in making its primary metric, Word Error Rate (WER), more trustworthy—through private test sets and advanced analysis—it still boils down to a single number. This aggregate score masks a well-documented problem: ASR error rates are not evenly distributed across user populations.
Prior research has shown significant racial, gender, and accent-based disparities in ASR performance. For instance, some commercial systems perform nearly twice as poorly for Black speakers as for white speakers. These biases remain invisible on leaderboards because their test sets typically lack metadata about the speakers.
The Monsoon Datasets: Designed for Diversity
To address this critical gap, the Monsoon datasets were created. They are intentionally designed to expose model failure modes by systematically varying across nine distinct axes:
- Geography: Recruiting contributors across hundreds of districts rather than in a few locations.
- Devices & Environments: Using contributors' own phones and connections in real-world indoor and outdoor settings.
- Vocabulary & Speech Rate: Prompting spontaneous conversation on everyday topics to elicit natural speech with named entities, numbers, and unrehearsed phrasing.
- Age & Gender: Recording and verifying speaker demographics.
- Multiple Valid Transcripts: For Hindi, accepting multiple correct spellings for the same audio.
In total, 12 speaker attributes are recorded for each of the 4,888 unique speakers, including occupation, education, and handset brand.
Dataset Composition
The data, sourced from unscripted spontaneous conversations, is organized into four speaker-disjoint splits:
| Set | Language | Duration | Speakers | M/F | Districts | Devices |
| :--- | :--- | :--- | :--- | :--- | :--- | :--- |
| Monsoon en-IN public | Indian English | 5.62 h | 1,444 | 50/50 | 428 | 556 |
| Monsoon en-IN private | Indian English | 5.58 h | 1,405 | 45/55 | 420 | 560 |
| Monsoon hi-IN public | Hindi | 1.33 h | 468 | 54/46 | 202 | 315 |
| Monsoon hi-IN private | Hindi | 4.47 h | 1,571 | 55/45 | 295 | 582 |
Key Features and Value
1. High Speaker Coverage
The true value of Monsoon lies in its vast number of speakers, not its total duration. The design deliberately limits the contribution of any single individual—the median audio per speaker is only about 8-12 seconds, and the top 10 speakers account for less than 3-4% of the data in the private splits. This ensures that no single voice can carry the score, forcing models to generalize across a diverse population.
2. Lattice Transcription for Hindi
Unlike English, Hindi has significant orthographic variation that cannot be resolved by a simple normalizer. To account for this, the Hindi sets use a 'lattice' transcript. For each segment of audio, it provides a list of all spellings that are considered correct, allowing for a more accurate and fair evaluation.
The introduction of the Monsoon datasets pushes the Open ASR Leaderboard beyond a single metric. It challenges the community to build ASR systems that are not just accurate on average, but are robust and equitable for diverse users worldwide.