SheetSage2

· · ·            

SheetSage2 turns music recordings into lead sheets and timed musical annotations. Transcribe a complete song, edit its ABC or MIDI, render a piano preview, or extract embeddings and token predictions for your own tools.

Built on MERT-v2-FullSong, with adapters that merge automatically when you load the model.

🚀 Quick start

Use Python 3.10 or 3.11 and FFmpeg 6.1 with its shared libraries. Sign in with access to this repository and its MERT-v2 parent:

python -m pip install huggingface-hub==0.36.0
huggingface-cli download m-a-p/SheetSage2 --local-dir SheetSage2
cd SheetSage2
python -m pip install torch==2.8.0 torchaudio==2.8.0 --index-url https://download.pytorch.org/whl/cu126
python -m pip install -r requirements.txt

import torch
from transformers import AutoModel

model = AutoModel.from_pretrained(
    "m-a-p/SheetSage2", trust_remote_code=True,
).eval().to("cuda" if torch.cuda.is_available() else "cpu")

result = model.transcribe("song.mp3", output_dir="output")

Files are decoded, mixed to mono and resampled automatically. Long songs use overlapping windows.

Omit output_dir to keep results in memory. Pass a Tensor or NumPy waveform with its sample rate:

# waveform: [samples] or [channels, samples]
result = model.transcribe(waveform, sampling_rate=24000)
abc = result["abc"]        # str
midi = result["midi"]      # bytes
events = result["events"]  # list of timed events

Paths, encoded audio bytes and binary streams are also accepted. result["midis"] contains the separate MIDI parts; result["labs"] contains annotation text. Request export_logits=True, export_scores=True or export_embeddings=True for CPU tensors in result["tensors"], grouped by window; add output_hidden_states=True for all 24 MERT layers. With output_dir, these tensors are saved as safetensors instead. Low-level forward() returns logits and optional hidden states; generate() returns symbolic tokens.

Save a self-contained model for offline use:

model.save_pretrained("sheetsage2-local")
model = AutoModel.from_pretrained(
    "sheetsage2-local", trust_remote_code=True, local_files_only=True,
)

Melody-only ABC for covers

Set melody_only=True to retain both the Vocal and Ins melodies while omitting chord symbols from the ABC and chord accompaniment from playback/combined MIDI. Raw predicted annotations remain available; the default full transcription is unchanged.

result = model.transcribe("song.mp3", output_dir="cover-score", melody_only=True)
abc = result["abc"]  # Also saved as cover-score/score.abc.

python infer.py song.mp3 --output cover-score --melody-only

Review the transcription, then pass result["abc"] or the saved cover-score/score.abc to YuE2 as abc, with cot="melody" and your target style and lyrics. If the requested ABC cannot be produced, Python raises an error (partial results are available as error.result) and the CLI exits with a nonzero status.

Command line and rendering

python infer.py song.mp3 --output output

# Optional piano audio and printable sheet music:
python setup_render.py
python infer.py song.mp3 --output output --render-audio --render-score

# Render existing results without loading the model:
python render.py --input output --output rendered --audio --score pdf,svg,png

On minimal Linux servers, use python setup_render.py --with-deps. Audio follows the original MIDI timing. Scores preserve the vocal and instrumental staves. For separate piano previews, use --render-parts vocal,instrumental,chords with infer.py, or --parts vocal,instrumental,chords with render.py.

Python also accepts render_audio=True, render_score="pdf,svg,png". In memory mode, result["rendered"] contains audio (part → WAV bytes) and score (format → pages as PDF/PNG bytes or SVG text).

📊 Benchmarks

Scores (%) on H800 with BF16 inference; higher is better. Use preset="paper" or --preset paper with the dataset prompts in scores and settings.

Melody F1 uses pitch classes. SheetSage1 beat/downbeat results use madmom. *The madmom chord model includes Chords1217 in training. †ChordFormer uses five-fold cross-validation; SheetSage2 uses one checkpoint across all tracks.