paza-whisper-large

Model Details

This model is a fine-tuned version of the openai/whisper-large-v3-turbo model finetuned for automatic speech recognition (ASR) in several Kenyan languages, including Swahili, Kalenjin, Kikuyu, Luo, Maasai and Somali. Whisper is a transformer-based encoder-decoder model that converts raw audio into text. The encoder processes audio inputs as log-Mel spectrograms, capturing acoustic and linguistic features, while the decoder generates text tokens in an autoregressive manner. This design allows the model to handle diverse languages, accents, and noise conditions with strong generalization.

Fine-tuning was performed on the entire unified multilingual ASR dataset, which includes the mentioned six languages, to encourage cross-lingual generalization. The fine-tuning process involved continued supervised training on labeled audio-text pairs, adjusting all the model’s parameters to better capture the phonetic and linguistic patterns unique to them. As a result, this model provides improved transcription accuracy for low-resource speech recognition tasks while maintaining Whisper’s robustness and efficiency.

Alignment approach

Traditional alignment measures are not applicable for this model because it does not generate new content; its primary function is to transcribe speech to text. In this context, alignment is best approximated by accuracy—the degree to which the transcription reflects the original spoken input.

Generative risks such as hallucination, harmful content are not applicable because the model does not create novel text or interpret meaning beyond transcription.

The post-training alignment process for this model focused on ensuring that transcriptions are reliable, consistent, and safe for downstream use in the specific language context. Supervised fine-tuning was performed on a multi-domain dataset of speech-text pairs in the mentioned languages, covering conversational, instructional, and broadcast speech to enhance robustness to diverse accents, noise conditions, and domains. No additional filtering for harmful or offensive content was applied during data preparation, because the focus was on capturing the full range of natural language use in a low-resource setting.

To align the model to the target ASR task, the training strategy included standard supervised learning with cross-entropy loss on the token level, followed by iterative validation monitoring using Word Error Rate (WER) and Character Error Rate (CER) metrics.

For safety considerations, refer to the Responsible AI section below. Additional performance metrics and evaluation results are provided in the Evaluation section.

Usage

Primary Use Cases

This model is designed for automatic speech recognition (ASR) across a wide range of already supported languages plus software and audio conditions and with additional support for Swahili, Kalenjin, Kikuyu, Luo, Maasai and Somali. This model is being shared with the research community to facilitate reproduction of our results and foster further research in this area.

Out-of-Scope Use Cases

It is not intended for automated decision-making, or any use cases that require understanding beyond transcription, and care should be taken to avoid applications where misinterpretation of speech could have safety or legal consequences.

This model is not specifically designed or evaluated for all downstream purposes and has not been evaluated on any other tasks besides speech recognition in the six languages mentioned above.

Developers should consider common limitations of language models and multimodal models, as well as performance difference across languages, as they select use cases, and evaluate and mitigate for accuracy, safety, and fairness before using within a specific downstream use case, particularly for high-risk scenarios. Developers should be aware of and adhere to applicable laws or regulations (including but not limited to privacy, trade compliance laws, etc.) that are relevant to their use case.

We do not recommend using this model in commercial or real-world applications without further testing and development. It is being released for research purposes.

Data overview

Training Data

The model was finetuned on the Africa Next Voices Kenya, DigiGreen Kikuyu, a proprietary Kikuyu dataset and the Swahili split of the Mozilla Common Voice dataset.

Due to the model’s maximum input length of 448 tokens, audio samples exceeding this limit were discarded during tokenization. Performance in each language correlates strongly with the amount of training data available for that language.

Data distribution by language

Figure 1: Data distribution by language

Training Procedure

The model was finetuned in a full-precision floating-point setup using a streaming data pipeline. A custom trainer was used to cycle through batches from different datasets, enabling mixed-language training (e.g., one batch Swahili, the next Kalenjin, etc.). A weighted random sampler was applied within the dataset generator to maintain language balance. Multi-GPU training was ochestrated using accelerate on 8×A100 (40GB) GPUs. Data preprocessing followed the same steps outlined in the Whisper finetuning guide.

Since the base Whisper model supports only one of the six languages, we extended the tokenizer to new language tokens and resized the input embeddings so the new tokens could be trained and used for inference.

from transformers import WhisperTokenizer, WhisperForConditionalGeneration

# Load model + tokenizer
model_name = "openai/whisper-large-v3-turbo"
tokenizer = WhisperTokenize