Intermediate

Music & Audio Generation

AI music generation has gone from research demos to commercially deployed tools in two years. You can now describe a song in a sentence and get a fully produced track - vocals, instruments, mixing, and mastering - in seconds. The technology is powerful, the legal questions are unresolved, and the creative applications are expanding rapidly.

The Landscape

Tool	Developer	Capability	Access
Suno	Suno AI	Full songs with vocals from text prompt; best consumer quality	SaaS / API
Udio	Udio	High-quality music, genre control, stem separation	SaaS
MusicGen	Meta	Open-weight text-to-music; melody conditioning	Open source
AudioCraft	Meta	Suite: MusicGen + AudioGen (sound FX) + EnCodec	Open source
Stable Audio	Stability AI	Music and sound effects; time-based conditioning	Open weight + SaaS
ElevenLabs Sound Effects	ElevenLabs	Text-to-sound effects (not music)	API

Autoregressive Approach - How Suno Works

Suno and similar tools use an autoregressive approach - predicting audio tokens one at a time, like an LLM predicts text tokens. The key insight: audio can be tokenized using a neural audio codec (EnCodec from Meta) that compresses audio into discrete tokens:

Text: "upbeat indie pop song about summer, female vocals"
       │
       ▼
Text Encoder → condition embeddings
       │
       ▼
Audio Token Predictor (transformer)
  → Generates audio tokens sequentially
  → Each token represents ~25ms of audio at multiple
    bitrate levels (EnCodec hierarchical tokens)
       │
       ▼
EnCodec Decoder → WAV / MP3 audio

The same transformer architecture that predicts the next word in text can be applied to predicting the next audio token. The result is musically coherent generation that respects harmonic and rhythmic structure - because music follows patterns learnable from data.

Diffusion Approach - How Stable Audio Works

Stable Audio uses diffusion on a compressed audio latent space (similar to how Stable Diffusion works for images):

Text prompt + duration conditioning
       │
       ▼
Diffusion on audio latent space
  → Diffusion model denoises a compressed
    representation of the audio
  → Time-based conditioning: model can target
    a specific duration (30s, 60s, 90s)
       │
       ▼
Autoencoder decoder → WAV audio

Stable Audio's key differentiation: timing conditioning - you can specify "generate a 47-second track" and the model generates exactly that duration with appropriate structure (intro, verse, chorus, outro).

MusicGen - Open Source Option

Meta's MusicGen is the go-to for self-hosted music generation:

Available in 300M, 1.5B, and 3.3B parameter sizes
Supports text conditioning and melody conditioning (hum a melody, continue it with generated music)
Runs locally on a GPU with 8GB+ VRAM (1.5B model)
Part of the AudioCraft suite which also includes AudioGen (sound effects) and EnCodec (the codec model)

What Works Well

Background music - for videos, podcasts, presentations, apps
Jingles and short loops - brand audio, game UI sounds
Genre-specific generation - "lofi hip hop study beats" or "epic orchestral trailer music"
Rapid prototyping - generate 20 variations of a concept in minutes
Sound effects - UI sounds, game effects, ambient audio (AudioGen/ElevenLabs)

What Still Does Not Work Well

Precise structural control - you cannot reliably specify "chorus at 0:45, key change at 1:30"
Stem control - cannot independently control specific instruments after generation (though stem separation is improving)
Long-form consistency - musical structure tends to drift and repeat after 2–3 minutes
Vocal quality - AI vocals are often detectable; expressiveness and nuance remain limited
Specific style matching - "sound exactly like this reference track" is unreliable