LLMs Part 2: Understand token position through AI

Positional encoding is a technique used in the architecture of large language models (LLMs) to provide the model with information about the position of each token in an input sequence.

In the standard sinusoidal positional encodings, each position in the input sequence is mapped to a vector of a fixed dimension, where each dimension in the vector is a function of both the position and the frequency, using sine and cosine functions of different frequencies. This allows the model to learn to attend to tokens at specific positions.

for even dimension indices

for odd dimension indices

In large language models (LLMs), the choice of wavelengths for positional encoding is designed to cover a wide range of positional information, from short-term to long-term dependencies. The wavelengths are implicitly defined by the positional encoding formulas using sine and cosine functions with different frequencies.

The sine and cosine functions used in positional encoding will repeat their values periodically due to their nature. The period of repetition, or wavelength, is determined by the functions' parameters. This repetition is intentional and ensures that the positional encoding can accommodate sequences of various lengths. The choice of using different frequencies for sine and cosine functions helps the model capture both short-term and long-term dependencies in the input sequence. By encoding positions in this way, the model can effectively learn the relative positions of tokens and understand the order of the sequence.

The more dimensions the embedding vector has, the longer the sequence before the patterns repeat.

In essence, as the dimensionality of the embedding vector increases, the wavelengths of the sine and cosine functions span a broader range, thereby extending the sequence length before the patterns start to repeat.

• Learnable Positional Embeddings Instead of using fixed sine and cosine functions, learnable positional embeddings use trainable parameters to represent positional information. Each position in the sequence has its own embedding vector, which is learned during training. This allows the model to potentially capture more complex positional relationships.
• Relative Positional Encoding Relative positional encoding represents the position of tokens relative to each other rather than absolute positions. This method allows the model to focus on the relative distance between tokens, which can be useful for handling long sequences where absolute positions may not be as important.
• Rotary Positional Embedding (RoPE) Rotary Positional Embedding incorporates rotational matrix transformations to encode position information. It modifies the token embeddings through rotations determined by their positions, preserving the model's ability to generalize across different sequence lengths and enhancing its performance in capturing contextual relationships.
• Explicit Position Indicators In some cases, models may use explicit position indicators or flags to mark the position of tokens. This can be done by concatenating position indicators to the token embeddings or using separate position-specific tokens.
• Bucketed Positional Embeddings Bucketed positional embeddings group positions into fixed-size buckets and assign the same positional embedding to all positions within a bucket. This approach is useful for handling very long sequences where precise positional information may be less critical.
• Gaussian Positional Embedding Gaussian positional embedding uses Gaussian distributions centered around each token to represent positional information. The embedding for a position is determined by a Gaussian function, providing a smooth and continuous way to encode positions.

Each of these methods has its own advantages and trade-offs, and the choice of positional encoding can depend on the specific task and model architecture.

This software was created using Vibe Coding by a Large Language Model LLM / chatbot and reworked in look & feel.

Some features had to be implemented manually and corrections and improvements had to be made.

The following Vibe Coding prompts were used on Copilot: