Audio

RVQ codebooks

Quantizes audio into discrete tokens through several rounds of vector lookup, each round encoding the previous round’s error.

First page of SoundStream: An End-to-End Neural Audio CodecPopularized inJul 2021SoundStream: An End-to-End Neural Audio CodecZeghidour et al. · arXiv 2107.03312 ↗
RVQ: audio → discrete tokens, round by roundtarget≈ reconcodebook 1: coarse shape+ codebook 2: the residual+ codebook 3: nearly exact · a few tokens per frame

Residual vector quantization turns continuous audio features into discrete tokens a language model can predict: the first codebook approximates the vector, the second encodes the leftover error, and so on. More codebooks mean higher fidelity at more tokens per second. Neural audio codecs and most modern TTS/music models are built on this.

Adoption over time

Share of new models that have included RVQ codebooks over time.

1%202220232024202520261%20222023202420252026

See it in real models

Open any of these on hfviewer to find this block in the interactive architecture graph.

Qwen/Qwen3-TTS-12Hz-1.7B-Base architecture graphQwen/Qwen3-TTS-12Hz-1.7B-Base↓ 3.8M · ♡ 525Open in visualizer Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice architecture graphQwen/Qwen3-TTS-12Hz-1.7B-CustomVoicetext-to-speech · ↓ 2.6M · ♡ 2kOpen in visualizer facebook/wav2vec2-base architecture graphfacebook/wav2vec2-base↓ 2.5M · ♡ 127Open in visualizer facebook/musicgen-medium architecture graphfacebook/musicgen-mediumtext-to-audio · ↓ 1.9M · ♡ 169Open in visualizer ResembleAI/chatterbox architecture graphResembleAI/chatterboxtext-to-speech · ↓ 1.8M · ♡ 2kOpen in visualizer k2-fsa/OmniVoice architecture graphk2-fsa/OmniVoicetext-to-speech · ↓ 1.3M · ♡ 1kOpen in visualizer Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice architecture graphQwen/Qwen3-TTS-12Hz-0.6B-CustomVoicetext-to-speech · ↓ 1.0M · ♡ 190Open in visualizer facebook/wav2vec2-xls-r-300m architecture graphfacebook/wav2vec2-xls-r-300m↓ 735k · ♡ 138Open in visualizer Qwen/Qwen3-TTS-12Hz-0.6B-Base architecture graphQwen/Qwen3-TTS-12Hz-0.6B-Basetext-to-speech · ↓ 655k · ♡ 299Open in visualizer Comfy-Org/MiniMax-Music-3 architecture graphComfy-Org/MiniMax-Music-3↓ 468k · ♡ 235Open in visualizer audio-cpp/MiniMax-Music3-GGUF architecture graphaudio-cpp/MiniMax-Music3-GGUFtext-to-audio · ↓ 457k · ♡ 26Open in visualizer incoai/Qwen3.8-27B-DFlash2 architecture graphincoai/Qwen3.8-27B-DFlash2text-generation · ↓ 391k · ♡ 247Open in visualizer

View 100 catalog matches for RVQ codebooks →

Related concepts

Mel spectrogram front-end Convolutional subsampling Conformer block

hfviewer renders the full architecture of 6,700+ Hugging Face models as interactive graphs. Hover any block to see what it does, with this model’s real numbers.

Browse all model graphs →