PVAE-TTS: High-Quality Adaptive Text-to-Speech via Progressive Variational Autoencoder

Adaptive text-to-speech (TTS) has attracted increasing interests in order to train TTS system without tons of high quality data. Nevertheless, existing adaptive TTS systems still show low adaptation quality for novel speakers since it is hard to learn extensive speaking style with limited data at once. To address this issue, we propose progressive variational autoencoder (PVAE) which generates data in adapting to style gradually. PVAE learns progressively style-normalized representation being a key component of progressive style adaptation. Then, we extend PVAE to PVAE-TTS, a multi-speaker adaptive TTS model which generates natural speech with high adaptation quality for novel speakers. To further improve the adaptation quality, we propose dynamic style layer normalization (DSLN) by utilizing convolution operation. The experimental results demonstrate the superiority of PVAE-TTS based on PVAE on both subjective and objective metrics.