Scaling Indic Pretraining: 2.8 Trillion Tokens and an Open Data Generation Pipeline
2026-10-18 · Hugging Face
Scaling Indic Pretraining: 2.8 Trillion Tokens and an Open Data Generation Pipeline
NVIDIA, in collaboration with IIT Madras and its AI4Bharat lab, has introduced Nemotron-pretraining-Indic-v1. This release features approximately 2.8 trillion tokens across 22 Indian languages, alongside an open, GPU-accelerated data generation pipeline used to create it.
Background and Objectives
The 22 languages included are those listed in India's Constitution, serving as the first languages for 97% of the population. Mitesh Khapra, head of AI4Bharat, identified the core hurdle facing Indian language technology: a lack of sufficient data compared to English. To bridge this gap, the teams translated curated English material—spanning mathematics, code, and question-answer content—into Indian languages. This provides model builders with substantial resources to develop models capable of explaining technical concepts and answering questions in these native tongues.
Corpus Composition
Using the Gemma-4 tokenizer, the corpus comprises 2,601.0 billion native-script tokens and 224.6 billion romanized tokens. It is organized into two groups encompassing 34 language-script configurations:
- Native-script and romanized data (12 languages): Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Nepali, Odia, Punjabi, Tamil, Telugu, and Urdu.
- Native-script data only (10 languages): Assamese, Bodo, Dogri, Kashmiri, Konkani, Maithili, Manipuri, Sanskrit, Santali, and Sindhi.
In this context, "native-script" refers to text translated from English but written in the traditional writing system. "Romanized" indicates Indic-language text written in Latin script. For the twelve dual-script languages, romanization covers a subset of the accepted native-script translations. This distinction is vital for constructing training mixtures, as native and romanized records offer different script exposures while expressing the same underlying content.
Data Generation Pipeline
To build this corpus, the team sourced documents from 15 Nemotron-CC-v2 subsets, including web text, Q&A material, mathematics, and code. The workflow is implemented using NeMo Data Designer, NVIDIA's open-source framework for synthetic data generation. The four-stage pipeline includes:
1. Document Translation: Translating English documents into the target Indic languages.
2. Quality Assessment: Evaluating the quality of the translations.
3. Targeted Repair: Repairing low-scoring outputs to improve quality.
4. Content Preservation: Ensuring equations, tables, and code blocks are preserved during conversion.
Throughout the process, each source identifier is retained, allowing developers to trace any output back to its original document. This clear provenance makes the open data practical to audit and build upon.
Open Source and Future Applications
By sharing the open generation recipe—including prompts, LLM workflows, and quality checks—the teams empower the community to reproduce the data and extend the work. Developers can use the corpus directly, adapt the prompts, or introduce their own source documents into the workflow.
The corpus has already been put into practice. During the continued pretraining of Nemotron 3.5 Super VL, 10 billion Indic tokens were drawn from a twelve-language subset of this corpus combined with natural Indic text. NVIDIA plans to build upon this foundation in future Nemotron models.