AudioCraft: Audio Generation

Comprehensive guide to using Meta's AudioCraft for text-to-music and text-to-audio generation with MusicGen, AudioGen, and EnCodec.

When to use AudioCraft

Use AudioCraft when:

• Need to generate music from text descriptions
• Creating sound effects and environmental audio
• Building music generation applications
• Need melody-conditioned music generation
• Want stereo audio output
• Require controllable music generation with style transfer

Key features:

• MusicGen: Text-to-music generation with melody conditioning
• AudioGen: Text-to-sound effects generation
• EnCodec: High-fidelity neural audio codec
• Multiple model sizes: Small (300M) to Large (3.3B)
• Stereo support: Full stereo audio generation
• Style conditioning: MusicGen-Style for reference-based generation

Use alternatives instead:

• Stable Audio: For longer commercial music generation
• Bark: For text-to-speech with music/sound effects
• Riffusion: For spectogram-based music generation
• OpenAI Jukebox: For raw audio generation with lyrics

Quick start

Installation

# From PyPI
pip install audiocraft
From GitHub (latest)

pip install git+https://github.com/facebookresearch/audiocraft.git
Or use HuggingFace Transformers

pip install transformers torch torchaudio

Basic text-to-music (AudioCraft)

import torchaudio
from audiocraft.models import MusicGen
Load model

model = MusicGen.get_pretrained('facebook/musicgen-small')
Set generation parameters

model.set_generation_params(
    duration=8,  # seconds
    top_k=250,
    temperature=1.0
)
Generate from text

descriptions = ["happy upbeat electronic dance music with synths"]
wav = model.generate(descriptions)
Save audio

torchaudio.save("output.wav", wav[0].cpu(), sample_rate=32000)

Using HuggingFace Transformers

from transformers import AutoProcessor, MusicgenForConditionalGeneration
import scipy
Load model and processor

processor = AutoProcessor.from_pretrained("facebook/musicgen-small")
model = MusicgenForConditionalGeneration.from_pretrained("facebook/musicgen-small")
model.to("cuda")
Generate music

inputs = processor(
    text=["80s pop track with bassy drums and synth"],
    padding=True,
    return_tensors="pt"
).to("cuda")
audio_values = model.generate(
    **inputs,
    do_sample=True,
    guidance_scale=3,
    max_new_tokens=256
)
Save

sampling_rate = model.config.audio_encoder.sampling_rate
scipy.io.wavfile.write("output.wav", rate=sampling_rate, data=audio_values[0, 0].cpu().numpy())

Text-to-sound with AudioGen

from audiocraft.models import AudioGen
Load AudioGen

model = AudioGen.get_pretrained('facebook/audiogen-medium')
model.set_generation_params(duration=5)
Generate sound effects

descriptions = ["dog barking in a park with birds chirping"]
wav = model.generate(descriptions)
torchaudio.save("sound.wav", wav[0].cpu(), sample_rate=16000)

Core concepts

Architecture overview

AudioCraft Architecture:
┌──────────────────────────────────────────────────────────────┐
│                    Text Encoder (T5)                          │
│                         │                                     │
│                    Text Embeddings                            │
└────────────────────────┬─────────────────────────────────────┘
                         │
┌────────────────────────▼─────────────────────────────────────┐
│              Transformer Decoder (LM)                         │
│     Auto-regressively generates audio tokens                  │
│     Using efficient token interleaving patterns               │
└────────────────────────┬─────────────────────────────────────┘
                         │
┌────────────────────────▼─────────────────────────────────────┐
│                EnCodec Audio Decoder                          │
│        Converts tokens back to audio waveform                 │
└──────────────────────────────────────────────────────────────┘

Model variants

| Model | Size | Description | Use Case |
|-------|------|-------------|----------|
| musicgen-small | 300M | Text-to-music | Quick generation |
| musicgen-medium | 1.5B | Text-to-music | Balanced |
| musicgen-large | 3.3B | Text-to-music | Best quality |
| musicgen-melody | 1.5B | Text + melody | Melody conditioning |
| musicgen-melody-large | 3.3B | Text + melody | Best melody |
| musicgen-stereo-* | Varies | Stereo output | Stereo generation |
| musicgen-style | 1.5B | Style transfer | Reference-based |
| audiogen-medium | 1.5B | Text-to-sound | Sound effects |

Generation parameters

| Parameter | Default | Description |
|-----------|---------|-------------|
| duration | 8.0 | Length in seconds (1-120) |
| top_k | 250 | Top-k sampling |
| top_p | 0.0 | Nucleus sampling (0 = disabled) |
| temperature | 1.0 | Sampling temperature |
| cfg_coef | 3.0 | Classifier-free guidance |

MusicGen usage

Text-to-music generation

from audiocraft.models import MusicGen
import torchaudio
model = MusicGen.get_pretrained('facebook/musicgen-medium')
Configure generation

model.set_generation_params(
    duration=30,          # Up to 30 seconds
    top_k=250,            # Sampling diversity
    top_p=0.0,            # 0 = use top_k only
    temperature=1.0,      # Creativity (higher = more varied)
    cfg_coef=3.0          # Text adherence (higher = stricter)
)
Generate multiple samples

descriptions = [
    "epic orchestral soundtrack with strings and brass",
    "chill lo-fi hip hop beat with jazzy piano",
    "energetic rock song with electric guitar"
]
Generate (returns [batch, channels, samples])

wav = model.generate(descriptions)
Save each

for i, audio in enumerate(wav):
    torchaudio.save(f"music_{i}.wav", audio.cpu(), sample_rate=32000)

Melody-conditioned generation

from audiocraft.models import MusicGen
import torchaudio
Load melody model

model = MusicGen.get_pretrained('facebook/musicgen-melody')
model.set_generation_params(duration=30)
Load melody audio

melody, sr = torchaudio.load("melody.wav")
Generate with melody conditioning

descriptions = ["acoustic guitar folk song"]
wav = model.generate_with_chroma(descriptions, melody, sr)
torchaudio.save("melody_conditioned.wav", wav[0].cpu(), sample_rate=32000)

Stereo generation

from audiocraft.models import MusicGen
Load stereo model

model = MusicGen.get_pretrained('facebook/musicgen-stereo-medium')
model.set_generation_params(duration=15)
descriptions = ["ambient electronic music with wide stereo panning"]
wav = model.generate(descriptions)
wav shape: [batch, 2, samples] for stereo

print(f"Stereo shape: {wav.shape}")  # [1, 2, 480000]
torchaudio.save("stereo.wav", wav[0].cpu(), sample_rate=32000)

Audio continuation

from transformers import AutoProcessor, MusicgenForConditionalGeneration
processor = AutoProcessor.from_pretrained("facebook/musicgen-medium")
model = MusicgenForConditionalGeneration.from_pretrained("facebook/musicgen-medium")
Load audio to continue

import torchaudio
audio, sr = torchaudio.load("intro.wav")
Process with text and audio

inputs = processor(
    audio=audio.squeeze().numpy(),
    sampling_rate=sr,
    text=["continue with a epic chorus"],
    padding=True,
    return_tensors="pt"
)
Generate continuation

audio_values = model.generate(**inputs, do_sample=True, guidance_scale=3, max_new_tokens=512)

MusicGen-Style usage

Style-conditioned generation

from audiocraft.models import MusicGen
Load style model

model = MusicGen.get_pretrained('facebook/musicgen-style')
Configure generation with style

model.set_generation_params(
    duration=30,
    cfg_coef=3.0,
    cfg_coef_beta=5.0  # Style influence
)
Configure style conditioner

model.set_style_conditioner_params(
    eval_q=3,          # RVQ quantizers (1-6)
    excerpt_length=3.0  # Style excerpt length
)
Load style reference

style_audio, sr = torchaudio.load("reference_style.wav")
Generate with text + style

descriptions = ["upbeat dance track"]
wav = model.generate_with_style(descriptions, style_audio, sr)

Style-only generation (no text)

# Generate matching style without text prompt
model.set_generation_params(
    duration=30,
    cfg_coef=3.0,
    cfg_coef_beta=None  # Disable double CFG for style-only
)
wav = model.generate_with_style([None], style_audio, sr)

AudioGen usage

Sound effect generation

from audiocraft.models import AudioGen
import torchaudio
model = AudioGen.get_pretrained('facebook/audiogen-medium')
model.set_generation_params(duration=10)
Generate various sounds

descriptions = [
    "thunderstorm with heavy rain and lightning",
    "busy city traffic with car horns",
    "ocean waves crashing on rocks",
    "crackling campfire in forest"
]
wav = model.generate(descriptions)
for i, audio in enumerate(wav):
    torchaudio.save(f"sound_{i}.wav", audio.cpu(), sample_rate=16000)

EnCodec usage

Audio compression

from audiocraft.models import CompressionModel
import torch
import torchaudio
Load EnCodec

model = CompressionModel.get_pretrained('facebook/encodec_32khz')
Load audio

wav, sr = torchaudio.load("audio.wav")
Ensure correct sample rate

if sr != 32000:
    resampler = torchaudio.transforms.Resample(sr, 32000)
    wav = resampler(wav)
Encode to tokens

with torch.no_grad():
    encoded = model.encode(wav.unsqueeze(0))
    codes = encoded[0]  # Audio codes
Decode back to audio

with torch.no_grad():
    decoded = model.decode(codes)
torchaudio.save("reconstructed.wav", decoded[0].cpu(), sample_rate=32000)

Common workflows

Workflow 1: Music generation pipeline

import torch
import torchaudio
from audiocraft.models import MusicGen
class MusicGenerator:
    def __init__(self, model_name="facebook/musicgen-medium"):
        self.model = MusicGen.get_pretrained(model_name)
        self.sample_rate = 32000
    def generate(self, prompt, duration=30, temperature=1.0, cfg=3.0):
        self.model.set_generation_params(
            duration=duration,
            top_k=250,
            temperature=temperature,
            cfg_coef=cfg
        )
        with torch.no_grad():
            wav = self.model.generate([prompt])
        return wav[0].cpu()
    def generate_batch(self, prompts, duration=30):
        self.model.set_generation_params(duration=duration)
        with torch.no_grad():
            wav = self.model.generate(prompts)
        return wav.cpu()
    def save(self, audio, path):
        torchaudio.save(path, audio, sample_rate=self.sample_rate)
Usage

generator = MusicGenerator()
audio = generator.generate(
    "epic cinematic orchestral music",
    duration=30,
    temperature=1.0
)
generator.save(audio, "epic_music.wav")

Workflow 2: Sound design batch processing

import json
from pathlib import Path
from audiocraft.models import AudioGen
import torchaudio
def batch_generate_sounds(sound_specs, output_dir):
    """
    Generate multiple sounds from specifications.
    Args:
        sound_specs: list of {"name": str, "description": str, "duration": float}
        output_dir: output directory path
    """
    model = AudioGen.get_pretrained('facebook/audiogen-medium')
    output_dir = Path(output_dir)
    output_dir.mkdir(exist_ok=True)
    results = []
    for spec in sound_specs:
        model.set_generation_params(duration=spec.get("duration", 5))
        wav = model.generate([spec["description"]])
        output_path = output_dir / f"{spec['name']}.wav"
        torchaudio.save(str(output_path), wav[0].cpu(), sample_rate=16000)
        results.append({
            "name": spec["name"],
            "path": str(output_path),
            "description": spec["description"]
        })
    return results
Usage

sounds = [
    {"name": "explosion", "description": "massive explosion with debris", "duration": 3},
    {"name": "footsteps", "description": "footsteps on wooden floor", "duration": 5},
    {"name": "door", "description": "wooden door creaking and closing", "duration": 2}
]
results = batch_generate_sounds(sounds, "sound_effects/")

Workflow 3: Gradio demo

import gradio as gr
import torch
import torchaudio
from audiocraft.models import MusicGen
model = MusicGen.get_pretrained('facebook/musicgen-small')
def generate_music(prompt, duration, temperature, cfg_coef):
    model.set_generation_params(
        duration=duration,
        temperature=temperature,
        cfg_coef=cfg_coef
    )
    with torch.no_grad():
        wav = model.generate([prompt])
    # Save to temp file
    path = "temp_output.wav"
    torchaudio.save(path, wav[0].cpu(), sample_rate=32000)
    return path
demo = gr.Interface(
    fn=generate_music,
    inputs=[
        gr.Textbox(label="Music Description", placeholder="upbeat electronic dance music"),
        gr.Slider(1, 30, value=8, label="Duration (seconds)"),
        gr.Slider(0.5, 2.0, value=1.0, label="Temperature"),
        gr.Slider(1.0, 10.0, value=3.0, label="CFG Coefficient")
    ],
    outputs=gr.Audio(label="Generated Music"),
    title="MusicGen Demo"
)
demo.launch()

Performance optimization

Memory optimization

# Use smaller model
model = MusicGen.get_pretrained('facebook/musicgen-small')
Clear cache between generations

torch.cuda.empty_cache()
Generate shorter durations

model.set_generation_params(duration=10)  # Instead of 30
Use half precision

model = model.half()

Batch processing efficiency

# Process multiple prompts at once (more efficient)
descriptions = ["prompt1", "prompt2", "prompt3", "prompt4"]
wav = model.generate(descriptions)  # Single batch
Instead of

for desc in descriptions:
    wav = model.generate([desc])  # Multiple batches (slower)

GPU memory requirements

| Model | FP32 VRAM | FP16 VRAM |
|-------|-----------|-----------|
| musicgen-small | ~4GB | ~2GB |
| musicgen-medium | ~8GB | ~4GB |
| musicgen-large | ~16GB | ~8GB |

Common issues

| Issue | Solution |
|-------|----------|
| CUDA OOM | Use smaller model, reduce duration |
| Poor quality | Increase cfg_coef, better prompts |
| Generation too short | Check max duration setting |
| Audio artifacts | Try different temperature |
| Stereo not working | Use stereo model variant |

References

• Advanced Usage - Training, fine-tuning, deployment
• Troubleshooting - Common issues and solutions

Resources

• GitHub: https://github.com/facebookresearch/audiocraft
• Paper (MusicGen): https://arxiv.org/abs/2306.05284
• Paper (AudioGen): https://arxiv.org/abs/2209.15352
• HuggingFace: https://huggingface.co/facebook/musicgen-small
• Demo: https://huggingface.co/spaces/facebook/MusicGen