Orpheus Music Transformer
SOTA 8k multi-instrumental music transformer trained on 2.31M+ high-quality MIDIs
Abstract
Project Los Angeles is very proud to present Orpheus Music Transformer, an efficient, SOTA transformer model for long-form, multi-instrumental music generation. At its core lies a 479 M-parameter autoregressive transformer equipped with Rotary Positional Embeddings (RoPE) and Flash Attention, enabling sequence lengths up to 8 k tokens—sufficient to capture extended musical structures. Trained for three epochs on 2.31 million high-quality MIDI tracks from the Godzilla dataset, our model employs a compact 3-token-per-note and 7-token-per-tri-chord encoding, plus a novel duration-and-velocity-last ordering to enhance expressivity. We leverage PyTorch’s bfloat16 precision and memory-efficient sparse-dense products for accelerated inference on CUDA, and provide a top-p sampling filter with adjustable temperature.
The Gradio interface empowers users to upload seed MIDI files or generate from scratch, tune prime/generation token counts, control randomness (temperature, top-p), and optionally append drums or natural “outro” tokens. Generated outputs appear in ten parallel batches with synchronized audio previews and piano-roll plots. Users can iteratively add or remove entire batches to sculpt a final composition, which is rendered back into MIDI and audio via an integrated SoundFont pipeline. Our release demonstrates a seamless blend of state-of-the-art model performance, efficient MIDI tokenization, and user-centric design, fostering rapid exploration of algorithmic composition.
Models
Presented are two models:
Orpheus Music Transformer Model
This is a base model that is capable of music generation/continuation and notes/drums inpainting
Orpheus Bridge Music Transformer Model
This is an auxiliary model that is capable of seamless bridge inpainting/infilling in any music composition
Live Hugging Face spaces demos
Orpheus Music Transformer
If you enjoyed any of the Orpheus Music Transformer demos, please star and duplicate. It helps a lot! 🤗
Training dataset code
Models were trained on select HQ MIDIs from Godzilla MIDI Dataset
Please check out Orpheus Taining Dataset Maker notebook for details
Models training code
Please check out Orpheus Music Transformer Maker code/notebook for details
Project Los Angeles
Tegridy Code 2025
Inference Providers
NEW
This model isn't deployed by any Inference Provider.
🙋
Ask for provider support