DiCoW_v3_2

This repository hosts the DiCoW_v3.2 model developed by BUT Speech@FIT, tailored for multi-talker automatic speech recognition (MT-ASR).

This model is available under the terms of CC BY 4.0. It incorporates an MIT-licensed base model and CC BY 4.0 licensed training data.

🔧 Key Improvements over DiCoW v1

• FDDT (Frame-Level Diarization Dependent Transformation) before positional embeddings
• Less strict suppressive initialization to ease early training dynamics
• Enhanced sequential decoding with fallback seeking
• Frozen decoder during fine-tuning to retain language modeling capabilities

🧪 Augmentations

• Random STNO noise injection
• Segment-wise random class flipping of STNO tokens
• SpecAugment
• MUSAN noise mixing

⚙️ Optimization & Inference Enhancements

• Updated learning schedule
• Improved hallucination detection & mitigation during inference

🛠️ Model Usage

from transformers import AutoModelForSpeechSeq2Seq

MODEL_NAME = "BUT-FIT/DiCoW_v3_2"
dicow = AutoModelForSpeechSeq2Seq.from_pretrained(MODEL_NAME, trust_remote_code=True)

➡️ For detailed inference pipelines, see: DiCoW GitHub (Inference)

🏆 Performance

See how DiCoW_v3.2 performs on our multi-talker ASR benchmark:

• 🔗 EMMA-MT ASR Leaderboard

📦 Model Details

• Base Model: Whisper large-v3-turbo

• Training Datasets:

  • NOTSOFAR-1
  • AMI Meeting Corpus
  • Libri2Mix

🧬 Source Repositories

• 🔧 Training Code: TS-ASR-Whisper
• 🚀 Inference

📚 Related Publications

• 📰 Journal Paper: DiCoW: Diarization-Conditioned Whisper for Target Speaker Automatic Speech Recognition Computer Speech & Language, 2026

• 📰 ICASSP 2025: Target Speaker ASR with Whisper IEEE ICASSP 2025

• 📰 CHiME-8 System Description: BUT/JHU System Description for CHiME-8 NOTSOFAR-1 Challenge CHiME 2024 Proceedings

• 📰 MLC-SLM Challenge Submission: BUT System for the MLC-SLM Challenge arXiv:2506.13414

📝 Citation

If you use this model, please cite the following works:

@article{POLOK2026101841,
    title = {DiCoW: Diarization-conditioned Whisper for target speaker automatic speech recognition},
    journal = {Computer Speech & Language},
    volume = {95},
    pages = {101841},
    year = {2026},
    issn = {0885-2308},
    doi = {https://doi.org/10.1016/j.csl.2025.101841},
    url = {https://www.sciencedirect.com/science/article/pii/S088523082500066X},
    author = {Alexander Polok and Dominik Klement and Martin Kocour and Jiangyu Han and Federico Landini and Bolaji Yusuf and Matthew Wiesner and Sanjeev Khudanpur and Jan Černocký and Lukáš Burget},
    keywords = {Diarization-conditioned Whisper, Target-speaker ASR, Speaker diarization, Long-form ASR, Whisper adaptation},
}

@INPROCEEDINGS{10887683,
    author={Polok, Alexander and Klement, Dominik and Wiesner, Matthew and Khudanpur, Sanjeev and Černocký, Jan and Burget, Lukáš},
    booktitle={ICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
    title={Target Speaker ASR with Whisper},
    year={2025},
    volume={},
    number={},
    pages={1-5},
    keywords={Transforms;Signal processing;Transformers;Acoustics;Speech processing;target-speaker ASR;diarization conditioning;multi-speaker ASR;Whisper},
    doi={10.1109/ICASSP49660.2025.10887683}
}

📬 Contact

For questions or collaboration inquiries:

📧 Email: ipoloka@fit.vut.cz

🏢 Affiliation: BUT Speech@FIT, Brno University of Technology

🔗 GitHub: BUTSpeechFIT