More details about wav2mask ?

[My-captain]

Hi, @jianfch .
Thanks for your awesome work! I test the repos on Mandarin songs, the alignment time points of tokens generated by the algorithm are more accurate.
But I'm having a little trouble reading the source code and understanding your algorithm, I'm stuck in the wav2mask, especially the principle of audio2loudness.
I am a beginner in the field of speech processing, please forgive my stupidity. I will very appreciate if you could give me some hints.

[jianfch]

audio2loudness converts the waveform into list of amplitude values that corresponds with the 1500 timestamps tokens in the prediction (i.e. each is 0.02s and total is 30s). Essentially tells you how loud each 0.02s chunk of the audio is. wav2mask does loudness equalization on that list of amplitude values and quantize those value (i.e. it zeros the low values). Then it converts this list of values into all mask for the 1500 timestamps tokens as a way tell the decoder which timestamp values to ignore because their relative loudness tell us that those timestamps are silent. On the other hand, vad=True generates this mask using another neural net.

[My-captain]

Thanks for your reply，It`s clear.
👍😊

[dgoryeo]

Hi @jianfch , In my normal workflow I pre-process my audios with Audacity compressor, or ffmpeg speechnorm to make quieter parts louder.

Do I understand right (from this discussion) that by using stable-ts I won't need to do any pre-processing as stable-ts already performs that on every input audio?

Thanks

[jianfch]

@dgoryeo, if you're trying to make the quieter parts louder because of model not doing as well with the quieter parts, then you should keep your normal workflow. The preprocessing in stable-ts is only for generating for a mask for timestamps, so does not alter the audio that is presented to the model (except only_voice_freq=True).

[dgoryeo]

Thanks @jianfch . On a side note: I've gotten hooked on demucs. It has improved my transcriptons a lot. Thanks for that feature.