Sarvam AI, a relatively new player in the AI field, recently introduced OpenHathi-Hi-v0.1, the inaugural model in their OpenHathi series. This Hindi large language model (LLM) is based on Meta AI’s Llama2-7B architecture, known for its robust capabilities. Sarvam AI claims that OpenHathi-Hi-v0.1 performs on par with GPT-3.5 specifically for Indic languages, showcasing its potential in the multilingual AI landscape.
The AI model developed by Sarvam AI extends Llama2-7B’s tokenizer with a substantial 48,000-token addition. The training process is divided into two phases: first, an embedding alignment phase that aligns randomly initialized Hindi embeddings, followed by a bilingual language modeling phase where the model is trained to attend cross-lingually across tokens.
Sarvam AI emphasizes that their model’s effectiveness goes beyond standard Natural Language Generation (NLG) tasks, as they have evaluated its performance in real-world applications. The company collaborated with KissanAI to fine-tune the base model using conversational data collected from interactions between a GPT-powered bot and farmers in various languages.
“We show that our model works as well as, if not better than GPT-3.5 on various Hindi tasks while maintaining its English performance,” the company said in a post on X (formerly Twitter).
In their efforts to enhance Llama-2’s Hindi capabilities, Sarvam AI took steps to improve the efficiency of tokenization on Hindi text. They aimed to make both training and inferencing faster by decreasing the fertility score, which represents the average number of tokens a word is split into. This involved training a sentence-piece tokenizer from a subset of the Sangraha corpus and merging it with the Llama2 tokenizer, resulting in a new tokenizer with a 48,000-word vocabulary.
Founded in July 2023 by Vivek Raghavan and Pratyush Kumar, Sarvam AI has quickly garnered attention and support. In a funding round earlier this month, the startup secured $41 million in investment, with leading contributions from Lightspeed Ventures, and participation from Peak XV Partners and Khosla Ventures. This substantial funding reflects the industry’s confidence in Sarvam AI’s potential and the significance of their contributions to the AI landscape.
Comments