Journal
IEEE-ACM TRANSACTIONS ON AUDIO SPEECH AND LANGUAGE PROCESSING
Volume 31, Issue -, Pages 2523-2533Publisher
IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
DOI: 10.1109/TASLP.2023.3288409
Keywords
Semantics; Acoustics; Training; Computational modeling; Codecs; Predictive models; Task analysis; Computer generated music; speech synthesis
Categories
Ask authors/readers for more resources
AudioLM is a framework for high-quality audio generation that maintains long-term consistency. It maps input audio to discrete tokens and treats audio generation as a language modeling task. This approach combines existing audio tokenizers to balance reconstruction quality and long-term structure, and leverages a hybrid tokenization scheme. By training on large corpora of raw audio waveforms, AudioLM learns to generate natural and coherent continuations. It can generate syntactically and semantically plausible speech continuations without any transcript or annotation, and even extend to generating coherent piano music continuations without symbolic representation.
We introduce AudioLM, a framework for high-quality audio generation with long-term consistency. AudioLM maps the input audio to a sequence of discrete tokens and casts audio generation as a language modeling task in this representation space. We show how existing audio tokenizers provide different trade-offs between reconstruction quality and long-term structure, and we propose a hybrid tokenization scheme to achieve both objectives. Namely, we leverage the discretized activations of a masked language model pre-trained on audio to capture long-term structure and the discrete codes produced by a neural audio codec to achieve high-quality synthesis. By training on large corpora of raw audio waveforms, AudioLM learns to generate natural and coherent continuations given short prompts. When trained on speech, and without any transcript or annotation, AudioLM generates syntactically and semantically plausible speech continuations while also maintaining speaker identity and prosody for unseen speakers. Furthermore, we demonstrate how our approach extends beyond speech by generating coherent piano music continuations, despite being trained without any symbolic representation of music.
Authors
I am an author on this paper
Click your name to claim this paper and add it to your profile.
Reviews
Recommended
No Data Available