In the previous post, we explored Long Short Term Memory Networks (LSTMs), a variant of Recurrent Neural Networks (RNNs) designed to handle long sequence data more effectively. Now, let’s consider a practical application: translating sentences from one language to another. For instance, translating the Spanish sentence (“Te quiero”) into the English sentence (“I love you.”)

One challenge with standard LSTM models is that they produce output sequences with the same length as the input. However, translation tasks often require generating output sequences of different lengths than the input. For example, translating a two-word Spanish sentence into a three-word English sentence is beyond the capability of basic LSTMs. This limitation can be problematic in many real-world applications where the lengths of input and output sequences vary. Fortunately, the Sequence to Sequence (Seq2Seq) model addresses this limitation.

Sequence to Sequence (Seq2Seq) Model

The Seq2Seq model is specifically designed to handle the scenario where the input and output sequences have different lengths. The model consists of two main components: an encoder and a decoder. The encoder processes the input sequence and converts it into a fixed-size context vector. The decoder then generates the output sequence using this context vector.

seq2seq

While there are several possible architectures for the encoder and decoder, we will focus on LSTM networks due to their ability to maintain information over long sequences and effectively handle the vanishing gradient problem.

Let’s dive deeper into the components of the Seq2Seq and understand how they work together.

Encoder

The encoder processes the input sequence one element at a time, converting it into a fixed-size context vector that encapsulates the entire sequence’s information.

Workflow

  1. Input sequence:
    • Consider an input sequence of words $[x_1, x_2,…,x_T]$, where $T$ is the length of the input sequence.
  2. Embedding layer:
    • Since the inputs are words, not numbers, we first convert them into numerical vectors using an embedding layer. This step transforms each word into a vector of fixed dimensions, which can then be fed into the LSTM.
  3. LSTM processing:
    • The LSTM cell processes each word vector sequentially, updating its hidden state $h_t$ and cell state $c_t$ at each time step.
  4. Context vector:
    • After the last token in the input sequence is processed, the final hidden state $h_T$ (and optionally the cell state $c_T$) serves as the context vector. This vector is a compressed representation of the entire input sequence and is passed on to the decoder.

seq2seq-encoder

In this diagram, the embedding layer is shown with just two dimensions, but in practice, hundreds or thousands of dimensions are typically used. Similarly, the LSTM can have multiple layers and a large number of cells in each layer to capture more complex patterns in the data.

Decoder

The decoder generates the output sequence one element at a time, using the context vector from the encoder as its initial state.

Workflow

  1. Initialization:
    • The decoder’s LSTM is initialized with the context vector, which includes the final hidden state $h_T$ from the encoder.
  2. Start token:
    • The decoder usually starts with a special token such as <SOS> (start of sequence) or <EOS> (end of sequence) as its first input. This token signals the beginning of the sequence generation process.
  3. Token generation:
    • At each time step, the decoder predicts the next token in the sequence based on its current hidden state and the embedding of the previously generated token.
  4. Softmax layer:
    • A softmax function is applied to the decoder’s output to produce a probability distribution over the possible next tokens, and the token with the highest probability is selected.
  5. Recurrent process:
    • The predicted token is fed back into the decoder as the input for the next time step. This process continues until a special end-of-sequence token (<EOS>) is generated or a predefined maximum length is reached.

Training

During training, the actual previous token from the target sequence is used as the input to the decoder (a technique known as teacher forcing) to improve convergence. In addition, the process is stopped once the model outputs the correct sequence length or generates the <EOS> token.
In other words, if the model predicts the tokens “I”, “love”, and “you” in sequence, and the current output is “you”, we conclude the training at that point.

seq2seq-decoder

Training the Seq2Seq Model

Training a Seq2Seq model involves feeding pairs of input and output sequences into the model. The loss (typically cross-entropy loss) is calculated based on the difference between the predicted tokens and the actual tokens in the output sequence. The model’s parameters, such as the weight matrices and biases of the LSTM cells and embedding layers, are updated to minimize this loss.

As mentioned before, to stablilize training and speed up convergence, we employ teacher forcing by feeding the actual target sequence as the input to the decoder during training. This approach helps the model learn more effectively, especially during the early stages of training.

Conclusion

The Seq2Seq model is a versatile framework for handling various sequence transformation tasks where the input and output sequences have different lengths. By using an LSTM-based encoder to capture the essence of the input sequence and an LSTM-based decoder to generate the output sequence, the Seq2Seq model overcomes the limitations of standard LSTMs.

In the next post, we’ll explore the limitations of the basic Seq2Seq model and introduce the attention mechanism, which improves the model’s performance by allowing it to focus on relevant parts of the input sequence at each step of the output generation.

Source