Human speech is composed of various elements, including vowels and consonants. Vowels are sounds that are produced without any restriction in the vocal tract, while consonants are sounds that are created by partially or completely closing the vocal tract. When we break down words into their individual sounds, we refer to them as phones or phonemes. These sounds are produced through a combination of the larynx or glottis, which generates pulses (known as the source), and the vocal tract, which shapes these pulses (acting as a filter).
While both the source and the filter contribute to the production of speech, it is the filter, represented by the vocal tract, that carries the majority of the information. In the context of speech recognition, the source is considered less important compared to the filter.
Speech as a Time Series
Speech can be understood as a representation of sound waves, specifically a time series of pressure values over time. When someone speaks, their words are translated into a wave pattern. For instance, imagine a wave that corresponds to the phrase “It’s time for lunch!”
Within speech, there are various types of sounds that can be identified. These include vowels, which are periodic signals (like the “a” sound in the word “has”). Additionally, there are fricatives, which are consonants produced by forcing air through a narrow channel (such as the “ch” sound in the word “watch”). Glides are another type of sound, characterized by a smooth transition (like the transition from the “w” to the “a” sound in “watch”). Lastly, bursts are rapid transitions (like the “d” sound in the word “dime”).
To differentiate between these different types of sounds, it is helpful to analyze them in the frequency domain using techniques like the Fourier Transform. By applying this transform, we can identify the specific frequencies that make up a given speech signal.
Converting Audio Signal into 2D Representation
The audio signal is composed of a sequence of sounds, and to analyze the frequency components of these consecutive sounds, we can compute a spectrogram. The spectrogram is obtained by applying the Short-time Fourier Transform (STFT) to the signal. The STFT divides the signal into
When processing audio signals, it is common to divide them into smaller chunks for analysis. These chunks are typically around 25 milliseconds in duration and are separated by a 10-millisecond gap. However, when applying analysis techniques to these chunks, such as the Fourier Transform, border effects can occur due to abrupt changes at the edges of the chunks.
To mitigate these border effects and ensure a smoother analysis, it is common to apply a windowing function to each chunk. One commonly used windowing function is the Hamming window. The Hamming window is defined by the equation:
By multiplying each chunk by the Hamming window, the border effects can be further reduced, resulting in a more accurate analysis of the audio signal. This technique is widely used in various applications, including speech processing and audio feature extraction.
Usually, spectrograms are viewed as 2-dimensional heatmaps, where the x-axis represents time, the y-axis represents frequency, and the color intensity represents the amplitude of the frequency components: brighter colors indicate higher amplitudes. This visualization provides valuable insights into the frequency components of the audio signal over time and is commonly used in speech processing and audio analysis.
Before performing the Short-time Fourier Transform (STFT), it is recommended to apply a pre-emphasis filter to boost high frequencies. This helps to balance the spectrum by amplifying high-frequency components, which often have smaller amplitudes, to avoid numerical issues during Fourier transform calculations and potentially improves the signal-to-noise ratio (SNR):
Humans have the ability to hear frequencies ranging from 20 Hz to 20 kHz. However, dogs and cats can hear higher frequencies than us. We perceive sounds based on their relative pitch. In the case of a piano, each key increases the pitch by a fixed multiple of
The perception of notes doesn't increase additively, but rather multiplicatively. Additionally, human hearing is sensitive to a wide range of loudness levels, which is why amplitude is represented on a logarithmic scale.
Mel Spectrogram
To create a Mel spectrogram, follow these steps:
- Limit the frequency range to a maximum of 8 kHz. This ensures that only frequencies within this range are considered in the spectrogram.
- Represent frequencies on a logarithmic scale. This helps to better visualize the distribution of frequencies. The y-axis of the spectrogram increases with multiples of frequency, following the formula:
This formula maps the frequency values to a logarithmic scale, allowing for a more intuitive representation. - Represent the amplitude on a logarithmic scale. Instead of directly measuring the signal’s amplitude, it is common to measure it in decibels (dB). The decibel scale is logarithmic and provides a more perceptually relevant representation of the signal’s intensity. The formula to convert the signal’s amplitude to decibels is:
. This formula calculates the decibel value ( ) based on the amplitude value ( ) of the signal.
Speech-to-Text (Automatic Speech Recognition - ASR)
The task of converting an audio signal into text involves utilizing various techniques, such as extracting features from the Mel Spectrogram, specifically the mel-frequency cepstrum coefficients (MFCCs), and employing Hidden Markov Models (HMMs) with Gaussian Mixture Models (GMMs). In recent years, Deep Learning approaches have also been successfully applied to tackle this task.
One challenge that arises when training a Convolutional Neural Network (CNN) model to detect phonemes is that the classifier produces predictions over fixed windows. However, it is crucial to determine how many windows correspond to the same phoneme or letter in order to accurately transcribe the audio.
To address this challenge, an effective solution is to employ an Encoder-decoder model. This model is designed to handle situations where the output length is not the same as the input length. By utilizing an encoder to process the input waveform or Mel Spectrogram and a decoder to generate the corresponding text output, the Encoder-decoder model enables accurate transcription of audio signals into text.
Wav2vec (2020)
Wav2vec is an advanced and highly effective transformer-based architecture that was introduced in 2020. This cutting-edge architecture is specifically designed to work with raw audio in a time series representation. It leverages the power of a convolutional neural network to generate initial embeddings, which are then processed by a transformer. This unique combination of techniques allows Wav2vec to achieve remarkable results in various audio-related tasks, including speech recognition and speech synthesis. By utilizing the strengths of both convolutional neural networks and Transformers, Wav2vec has become a game-changer in the field of natural language processing and audio analysis. Its ability to directly process raw audio data sets it apart from traditional approaches, making it a powerful tool for researchers and practitioners alike.
Whisper (2022)
Whisper is an advanced and cutting-edge Transformer-based system that has emerged in the field of natural language processing. With its state-of-the-art performance, Whisper has garnered significant attention and acclaim. Similar to the Vision Transformer, Whisper leverages the power of the Mel spectrogram as its input representation, enabling it to effectively process and analyze audio data.
One of the key advantages of Whisper is its weakly supervised nature, which allows for the utilization of larger training sets. This enables the model to learn from a diverse range of data, leading to improved performance and generalization. Additionally, Whisper is designed to be multi-lingual and supports multi-task training, making it a versatile solution for various language-related tasks.
With its innovative approach and impressive results, Whisper has established itself as a state-of-the-art system in the field of natural language processing. Its ability to leverage the Mel spectrogram and its adaptability to different languages and tasks make it a powerful tool for researchers and practitioners alike.
Evaluating Speech-to-Text
We can assess the performance of a speech-to-text system by calculating the Word Error Rate (WER) for a given string. The WER measures the percentage of detected words that differ from the correct words based on the edit distance. The formula for calculating the WER is as follows:
In addition to the WER, we can also evaluate the system using the Sentence Error Rate (SER). The SER measures the percentage of sentences that contain at least one error. The formula for calculating the SER is as follows:
When evaluating a speech-to-text system, both the WER and SER provide valuable insights into its performance.
Personalizing Speech-to-Text
There are various methods available for personalizing speech-to-text systems. Two common approaches include vocal tract length normalization and modifying the acoustic model.
- Vocal tract length normalization involves adjusting the frequency axis of the speech power spectrum to accommodate the variations in vocal tract resonances caused by differences in physical speaker size. This technique helps improve accuracy in speech recognition. Additionally, pitch normalization based on the speaker’s age can be beneficial, especially for children.
- Modifying the acoustic model is another approach to personalize speech-to-text systems. This involves taking a pre-trained model and a small dataset from a new speaker, and then transforming the model to optimize its performance specifically for that speaker.
Handling Non-Verbal Sounds
To handle non-word sounds, a specific phone is created for each non-verbal sound. A special word is added to the lexicon for that phone, and the phones are trained using standard training techniques. The training data transcripts include labels for these new special words, and the special words must be incorporated into the language model.
Text-to-Speech (Speech Synthesis)
Text-to-speech technology has made significant advancements in recent years. Its objective is to transform a text string into an audio waveform, typically achieved through a 3-stage process:
- Converting text to phonemes
- Mapping phonemes to a Mel spectrogram
- Generating an audio signal from the Mel spectrogram
Text Normalization
During the text-to-speech process, abbreviated text needs to be expanded. For example, the phrase “They live at 224 Mission St.” should be expanded to “They live at two twenty-four Mission Street.” The expansion process depends on context, and numbers like “1750” can be verbalized in various ways:
- The economy in the year 1750 = The economy in the year seventeen fifty
- The password is 1750 = The password is one seven five zero
- It costs 1750 dollars = It costs one thousand seven hundred and fifty dollars
One possible solution is to train a seq2seq model to appropriately expand the text.
Homograph disambiguation involves determining the correct pronunciation for words that are spelled the same but pronounced differently. For example, “bass” could refer to a musical instrument (pronounced /b eys/) or a type of fish (pronounced /b ae s/). The correct pronunciation needs to be determined based on context: “I like playing my bass guitar” vs. “I’m cooking sea bass for dinner.”
Tacotron2 and WaveNet
Tacotron2 is a remarkable example of a text-to-speech architecture that was introduced in 2018. This advanced system utilizes Long Short-Term Memory (LSTM) networks to convert text into high-quality speech. One of the key features of Tacotron2 is its autoregressive dilated convolution-based generation approach, which ensures the production of natural and coherent audio signals. By leveraging the power of LSTMs and incorporating innovative techniques, Tacotron2 has made significant advancements in the field of speech synthesis. Its ability to generate realistic and expressive speech has made it a popular choice among researchers and practitioners in the natural language processing domain.
WaveNet is a powerful Vocoder utilized in Tacotron2, a state-of-the-art text-to-speech architecture. Its primary function is to convert the Mel Spectrogram, a visual representation of audio frequencies, into a high-quality and natural-sounding audio signal. WaveNet achieves this by employing an autoregressive dilated convolution-based generation approach, which ensures the production of coherent and realistic speech. With its advanced techniques and capabilities, WaveNet has become a crucial component in the field of speech synthesis, enabling the generation of lifelike and expressive voices for various applications.
Speech Synthesis
This process involves two main steps: text analysis, where the text is converted into a phonetic representation, and waveform synthesis, where the phonetic representation is transformed into an actual waveform.
There are several approaches to waveform synthesis, including formant synthesis, which uses an acoustic model and additive synthesis to create a robotic-sounding voice. Another approach is articulatory synthesis, which simulates the movements of articulators to generate speech. Lastly, there is concatenative synthesis, which involves combining pre-recorded speech units to produce natural-sounding speech.
Concatenative TTS
Concatenative synthesis is the predominant method employed in commercial text-to-speech systems.
It follows an hourglass model consisting of two stages:
- Text analysis stage:
- Text normalization: tokenization, handling non-standard words, and disambiguating homographs.
- Phonetic analysis: generating a sequence of phonemes from words. Traditionally, phonetic analysis involves converting words into a list of phonemes using different techniques:
- Dictionary-based: containing phonetic representations of words.
- Grapheme-to-phoneme (g2p) conversion: training a classifier to handle names and unknown words.
- For transparent languages (e.g., Italian), using pronunciation rules and a dictionary for irregular forms and foreign words.
- Prosodic analysis: incorporating prosodic information such as intonation and stress.
- Waveform synthesis: transforming phonemes into waveforms by concatenating pre-recorded speech units.
Prosodic Analysis
Prosody encompasses intonation, stress, and rhythm in speech. Utterances have a prosodic structure, including intonation phrases and intermediate phrases. Boundaries can be identified using classifiers. Prosodic prominence, also known as stress, emphasizes certain words through pitch, rhythm, and energy. Empathic accent highlights words for semantic reasons, while unaccented words have regular stress. Reduced accent indicates less emphasis than usual. Tune refers to the rise or fall of the base frequency (
Phoneme duration can be adjusted based on contextual rules. Klatt’s formula calculates the duration of a phone using a set of actor weights.
- Pre-pausal lengthening: Vowels in syllables before a pause are extended by
. - Non-phrase-final shortening: Segments that are not at the end of a phrase are shortened by
. - Klatt’s formula for phone duration is:
Alternatively, machine learning can be employed to address this issue.
Waveform Synthesis
There are two approaches to concatenative waveform synthesis in text-to-speech:
- Diphone synthesis
- Unit selection synthesis
Diphone Synthesis
Definitions
- A diphone is a unit that represents the transition from the middle of one phone to the middle of the next.
- Coarticulation is a phenomenon where each phone is influenced by the preceding and following phones.
To create a diphone database, a speaker records one example of each diphone, marks the boundaries of each diphone, cuts them out, and compiles them into a database. When synthesizing an utterance, the relevant sequence of diphones is retrieved from the database and concatenated, ensuring smooth transitions between them.
Unit Selection Synthesis
Definition
Unit Selection Synthesis is a technique used in text-to-speech systems where a database of speech units is created. These units can be any piece of speech that can be concatenated together to form synthesized speech. The goal is to find the best sequence of units from the database that closely matches the target description.
To achieve this, a cost function is used, which takes into account various factors such as phonetic context, F0 (pitch), stress, and phrase position. The target cost measures how closely the selected units match the desired target description. The join cost determines the best way to join neighboring units, considering factors like formants, spectral characteristics, energy, and F0.
Solving this problem can be done using algorithms like Viterbi or beam search, which help find the optimal sequence of units from the database.
Evaluating Text-to-Speech Systems
The evaluation of text-to-speech systems involves assessing both intelligibility and quality.
- Intelligibility refers to the ability of a listener to accurately understand the meaning of an utterance. This can be measured through phone discrimination tests, such as the Diagnostic Rhyme Test (DRT) and the Modified Rhyme Test (MRT).
- Quality, on the other hand, focuses on the naturalness, fluency, and clarity of the synthesized speech. It can be evaluated using methods like the Mean Opinion Score (MOS) and AB tests, where testers compare the same utterance produced by two different systems and choose the one that sounds better.
References
- 3-3 Human Voice Production
- Speech & Audio Processing & Recognition
- Speech Processing for Machine Learning: Filter banks, Mel-Frequency Cepstral Coefficients (MFCCs) and What’s In-Between | Haytham Fayek
- Audio Deep Learning Made Simple - Why Mel Spectrograms perform better | Ketan Doshi Blog
- wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
- Robust Speech Recognition via Large-Scale Weak Supervision
- Dilated Convolution (Semantic Segmentation)