NLP Fundamentals: From Tokenization to Transformers

Introduction

Natural Language Processing has undergone a remarkable transformation over the past decade. What began with hand-written rules and statistical n-gram models has evolved into a landscape dominated by large language models capable of generating human-quality text, translating between hundreds of languages, and reasoning over complex documents. In 2026, NLP is no longer a niche research field — it is the backbone of products used by billions of people every day.

Yet beneath every modern NLP system lies a stack of fundamental techniques that remain essential regardless of the model architecture. Tokenization turns raw text into discrete units the model can process. Word embeddings map those units into mathematical spaces where semantic relationships are captured as geometric patterns. Sequence models learn to process temporal dependencies in text. And the Transformer architecture, introduced in 2017, has become the universal foundation for virtually every state-of-the-art system.

This guide covers the full pipeline from raw text to production-ready NLP systems. We start with tokenization and preprocessing, move through embeddings and sequence modelling, unpack the Transformer revolution, and finish with transfer learning, modern benchmarks, and production deployment considerations. Whether you are new to NLP or need a structured refresher, this guide covers the concepts that every practitioner should know in 2026.

Tokenization

Tokenization is the process of splitting raw text into smaller units — tokens — that serve as the atomic inputs to a model. The choice of tokenization method has far-reaching consequences for vocabulary size, out-of-vocabulary handling, model performance, and computational efficiency. Three broad families of tokenization exist: word-level, character-level, and subword-level.

Word-Level Tokenization

Word-level tokenization splits text on whitespace and punctuation, producing one token per word. It is intuitive and maps cleanly to linguistic units, but it produces enormous vocabularies — English alone has hundreds of thousands of words, and morphological variants (run, runs, running, ran) each require a separate token. This makes word-level tokenization impractical for modern models: a vocabulary of 500,000 tokens requires a 500,000-way softmax at the output layer, adding significant computational cost.

Character-Level Tokenization

Character-level tokenization uses individual characters or bytes as tokens. Vocabulary size drops to around 100-200 (26 letters, digits, punctuation, and special tokens), and the model can theoretically handle any text including misspellings and unknown words. However, sequence lengths increase dramatically — “hello” becomes 5 tokens instead of 1 — which makes it difficult for models to learn long-range dependencies. Character-level models also struggle to capture subword morphological patterns.

Subword Tokenization

Subword tokenization strikes a balance between word and character approaches. Common words become single tokens, while rare words are decomposed into smaller, meaningful subunits. This keeps vocabulary sizes manageable (typically 30,000-100,000 tokens) while ensuring no word is truly out-of-vocabulary. Four major subword algorithms dominate modern NLP:

BPE (Byte-Pair Encoding), introduced by Sennrich et al. in 2016, starts with a vocabulary of individual characters and iteratively merges the most frequent adjacent pair. WordPiece, developed by Google, merges pairs that maximise the likelihood of the training data. Unigram, introduced by Kudo, starts from a large vocabulary and prunes tokens with the lowest loss contribution. SentencePiece, also by Kudo, wraps BPE or Unigram to operate directly on raw text without requiring pre-tokenisation, making it language-agnostic.

from transformers import AutoTokenizer

# Word-level tokenization
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
text = "Tokenization is the first step in NLP!"
tokens = tokenizer.tokenize(text)
print(tokens)
# ["token", "##ization", "is", "the", "first", "step", "in", "nlp", "!"]

# Byte-Pair Encoding example
bpe_tokenizer = AutoTokenizer.from_pretrained("gpt2")
bpe_tokens = bpe_tokenizer.tokenize("Unbelievable!")
print(bpe_tokens)
# ["Un", "bel", "ie", "vable", "!"]

Modern models typically use vocabularies of 32,000 to 100,000 subword tokens. Llama 3 uses a 128,000-token tokenizer based on SentencePiece with BPE. The tokenizer choice directly affects model performance: a tokenizer with poor coverage of the target domain forces the model to learn from fragmented subword sequences, increasing the effective sequence length and reducing the model's capacity to capture meaning.

Text Normalization and Preprocessing

Raw text is messy. Before tokenization, text typically passes through a normalization pipeline that reduces variability while preserving meaning. The specific steps depend on the language, domain, and downstream task.

Case folding converts all text to lowercase, reducing vocabulary size at the cost of losing case-specific information (e.g., “Apple” the company vs “apple” the fruit). For most tasks, case folding improves generalisation. For named entity recognition, preserving case is important because proper nouns are typically capitalised.

Unicode normalization (NFKC or NFD) ensures that visually identical characters with different Unicode representations (e.g., precomposed é vs decomposed e + combining accent) map to the same token. This is critical for multilingual systems where Unicode inconsistencies are common.

Stemming and lemmatisation reduce words to their base forms. Stemming uses heuristic rules to chop affixes (“running” → “run”, “studies” → “studi”), while lemmatisation uses vocabulary and morphological analysis to return the dictionary form (“ran” → “run”). Modern subword tokenizers handle most morphological variation internally, making explicit stemming less important for Transformer-based models than it was for bag-of-words or n-gram classifiers.

Stop word removal — filtering out high-frequency words like “the,” “is,” “at” — was standard practice in pre-neural NLP but is rarely used with Transformers. The self-attention mechanism naturally learns to weight tokens by importance, and removing stop words can discard useful syntactic context. For embedding-based retrieval systems, however, stop word removal still improves index quality.

Regex patterns handle domain-specific cleaning: removing HTML tags, normalising URLs and email addresses, collapsing repeated whitespace, and handling social media artefacts like mentions and hashtags. Language-specific considerations include Chinese and Japanese (which lack spaces between words), Arabic and Hebrew (right-to-left text), and code-switching (mixed-language text common in multilingual communities). For a deeper treatment of how models process these representations, see our guide on vectors, tensors, and scalars.

Word Embeddings

Word embeddings map discrete tokens into continuous vector spaces where semantic relationships correspond to geometric relationships. The evolution from sparse representations to dense contextual embeddings is one of the defining arcs of modern NLP.

One-Hot Encoding and Its Limitations

The simplest representation is one-hot encoding: each token corresponds to a binary vector with a single 1 at the token's index in the vocabulary. For a vocabulary of 50,000 tokens, each vector has dimension 50,000 with exactly one non-zero entry. This representation is sparse (most entries are zero), high-dimensional, and lacks any notion of semantic similarity — “cat” and “dog” are as different as “cat” and “quantum.” Every pair of one-hot vectors has the same dot product (zero), giving the model no signal about word relationships.

Word2Vec: CBOW and Skip-gram

Word2Vec, introduced by Mikolov et al. at Google in 2013, learns dense, low-dimensional embeddings (typically 100-300 dimensions) by training a shallow neural network on a language modelling objective. The Continuous Bag of Words (CBOW) variant predicts the current word from its surrounding context. The Skip-gram variant predicts the surrounding context from the current word. Skip-gram is slower to train but produces better embeddings for rare words.

The key insight of Word2Vec is that words with similar contexts have similar embeddings. This captures surprisingly rich semantic relationships through vector arithmetic: “king” - “man” + “woman” ≈ “queen”. While this linear analogy property was celebrated, it works reliably only for well-structured domains like royalty, geography, and verb tenses — it fails for more abstract relationships.

GloVe and FastText

GloVe (Global Vectors for Word Representation), introduced by Pennington et al. at Stanford in 2014, takes a different approach: instead of predicting words from context, it factorises the global word-word co-occurrence matrix. GloVe embeddings tend to capture global corpus statistics better than Word2Vec, which is inherently local. FastText, by Facebook AI Research in 2016, extends Word2Vec by representing each word as a bag of character n-grams. This allows FastText to generate embeddings for out-of-vocabulary words by summing their constituent n-gram vectors — a significant advantage for morphologically rich languages.

Static vs Contextual Embeddings

All the embeddings above are static: each word has a single vector regardless of context. The word “bank” has the same embedding in “river bank” and “investment bank.” Contextual embeddings, introduced by ELMo in 2018 and now standard in all Transformer models, produce a unique representation for each token based on its surrounding context. This resolves polysemy and captures syntax-sensitive meaning. The shift from static to contextual embeddings was the single most impactful improvement in NLP representation quality before the Transformer revolution.

Sequence Modeling: RNNs, LSTMs, and the Attention Mechanism

Language is sequential — the meaning of a word depends on the words that came before it. Before Transformers, recurrent neural networks (RNNs) were the primary architecture for modelling sequential dependencies in text.

Recurrent Neural Networks

An RNN processes tokens one at a time, maintaining a hidden state that summarises all previous tokens. At each step t, the hidden state h_t is a function of the current token x_t and the previous hidden state h_t-1. In theory, the hidden state can capture arbitrarily long dependencies. In practice, standard RNNs suffer from the vanishing gradient problem: gradients propagated backward through many time steps shrink exponentially, making it impossible for the model to learn dependencies spanning more than 5-10 tokens.

LSTMs and GRUs

LSTMs (Long Short-Term Memory), introduced by Hochreiter and Schmidhuber in 1997, address the vanishing gradient problem through a gated cell structure. An input gate, forget gate, and output gate control the flow of information into and out of the cell state, which acts as a gradient highway. LSTMs can reliably learn dependencies spanning 100-200 tokens, which was state-of-the-art for sequence modelling until 2017.

GRUs (Gated Recurrent Units), introduced by Cho et al. in 2014, simplify the LSTM architecture by merging the input and forget gates into a single update gate. GRUs have fewer parameters than LSTMs and train faster while achieving comparable performance on most tasks. Both architectures were used in production NLP systems for translation, sentiment analysis, and text generation before Transformers rendered them largely obsolete.

Encoder-Decoder Architecture and Attention

Sequence-to-sequence models use an encoder RNN that reads the input sequence into a fixed-length context vector, and a decoder RNN that generates the output sequence from that context vector. The fixed-length context vector is a bottleneck — it must compress the entire input into a single vector, losing information about long inputs.

Attention mechanisms, introduced by Bahdanau et al. in 2015, solve this bottleneck by allowing the decoder to access the full encoder hidden state at each generation step. Instead of relying on a single context vector, attention computes a weighted sum of all encoder hidden states, where the weights depend on the current decoder state. Luong's 2015 variant simplified the attention computation and introduced global and local attention variants. This was the direct precursor to the Transformer's self-attention mechanism — the core innovation that replaced recurrence entirely.

The Transformer Revolution

The Transformer architecture, introduced by Vaswani et al. in the landmark 2017 paper “Attention Is All You Need,” replaced recurrence with a fully attention-based architecture. It is the foundation upon which every major language model in 2026 is built. For a detailed architectural walkthrough, see our guide on Transformer architecture explained.

Self-Attention

Self-attention computes a representation of a sequence by relating each position to every other position. For each token, the model computes a query vector Q, a key vector K, and a value vector V by multiplying the token's embedding with learned weight matrices. The attention score between token i and token j is the dot product Q_i · K_j, scaled by the square root of the key dimension and normalised with softmax. The output for token i is the weighted sum of all value vectors, where the weights are the attention scores.

This mechanism gives every token direct access to every other token in a single layer, eliminating the sequential bottleneck of RNNs. The computational cost is O(n²) in sequence length, which is the main practical limitation of Transformers — processing a 100,000-token document requires 10 billion attention computations per layer.

Multi-Head Attention

Instead of computing a single attention distribution, multi-head attention runs multiple attention operations in parallel (typically 8-96 heads). Each head learns to attend to different types of relationships — syntactic dependencies, coreference, semantic similarity, positional proximity. The outputs of all heads are concatenated and projected back to the model dimension.

class MultiHeadAttention(nn.Module):
    def __init__(self, d_model, num_heads):
        super().__init__()
        self.num_heads = num_heads
        self.d_k = d_model // num_heads

        self.W_q = nn.Linear(d_model, d_model)
        self.W_k = nn.Linear(d_model, d_model)
        self.W_v = nn.Linear(d_model, d_model)
        self.W_o = nn.Linear(d_model, d_model)

    def forward(self, x):
        B, T, D = x.shape
        Q = self.W_q(x).view(B, T, self.num_heads, self.d_k).transpose(1, 2)
        K = self.W_k(x).view(B, T, self.num_heads, self.d_k).transpose(1, 2)
        V = self.W_v(x).view(B, T, self.num_heads, self.d_k).transpose(1, 2)

        attn = F.softmax(
            Q @ K.transpose(-2, -1) / (self.d_k ** 0.5), dim=-1
        )
        return self.W_o(
            (attn @ V).transpose(1, 2).contiguous().view(B, T, D)
        )

Positional Encoding

Unlike RNNs, self-attention has no inherent notion of token order — the attention operation is permutation-invariant. Positional encodings inject sequence position information into the input. The original Transformer used fixed sinusoidal encodings, but modern models use learned positional embeddings (BERT, GPT-2), rotary positional embeddings (RoPE, used in Llama, Mistral, Qwen), or relative position biases (ALiBi, used in MPT). RoPE has become the dominant choice in 2026 because it encodes relative positions naturally and generalises to longer sequences than seen during training.

Encoder-Decoder and Variants

The original Transformer is an encoder-decoder architecture. The encoder uses bidirectional self-attention (each token attends to all tokens). The decoder uses masked self-attention (each token attends only to preceding tokens) and cross-attention to the encoder output. This architecture is used by T5 and BART for sequence-to-sequence tasks like translation and summarisation.

Two major architectural variants have emerged. BERT (Bidirectional Encoder Representations from Transformers) uses only the encoder with bidirectional self-attention, making it ideal for understanding tasks like classification, NER, and question answering. GPT (Generative Pre-trained Transformer) uses only the decoder with causal masking, making it ideal for generation. The distinction between encoder-only (BERT-style), decoder-only (GPT-style), and encoder-decoder (T5-style) architectures is the primary taxonomic split in modern LLMs. See our LLM comparison guide for a detailed comparison of model families based on this distinction.

Transfer Learning in NLP

The Transformer revolution enabled transfer learning at unprecedented scale. Instead of training models from scratch for each task, practitioners can now download a pre-trained model and adapt it with a fraction of the original compute.

Pre-Training Objectives

Different architectures use different pre-training objectives. BERT uses masked language modelling (MLM): 15% of input tokens are masked, and the model learns to predict the masked tokens from their bidirectional context. This forces the model to build deep bidirectional representations but creates a mismatch with fine-tuning (which does not use masks). GPT uses autoregressive language modelling: the model predicts the next token given all previous tokens, matching the inference-time behaviour exactly.

T5 uses a text-to-text framework where every NLP task is framed as text-to-text generation, with a prefix or prompt indicating the task. Prefix language modelling, used by UniLM and other models, combines MLM and autoregressive objectives by applying a mask to a prefix of the sequence and generating the remainder. Span corruption, used by T5 and Bart, masks contiguous spans of tokens and trains the model to reconstruct them.

Fine-Tuning, Adapters, and Prompting

Fine-tuning updates the pre-trained model's weights on a task-specific dataset. Full fine-tuning updates all parameters, which is effective but produces a separate copy of the model per task. Parameter-efficient methods like LoRA and Adapter modules train a small number of additional parameters while freezing the base model, enabling multi-task serving from a single base model.

Prompting, at the other extreme, does not modify model weights at all. Instead, it crafts the input text to elicit the desired behaviour. Prompt engineering has become a critical skill in the LLM era, and it often achieves surprisingly good results without any training cost. For a practical guide, see our post on prompt engineering in production.

Adapter modules, introduced by Houlsby et al. in 2019, insert small bottleneck layers between the existing layers of a frozen pre-trained model. Only the adapter parameters are updated during fine-tuning, keeping the base model intact. Adapters are less popular than LoRA in 2026 because they add inference latency (the adapter layers must be computed sequentially), while LoRA adapters can be merged into the base weights.

Modern NLP Tasks and Benchmarks

NLP encompasses a wide range of tasks, from word-level to document-level, each with established benchmarks and evaluation protocols.

Core Tasks

Text classification assigns a label to a text — sentiment analysis, topic classification, spam detection. Transformer-based models fine-tuned on 1,000-10,000 examples typically achieve 90-97% accuracy on standard benchmarks like IMDB (sentiment) and AG News (topic classification). Named Entity Recognition (NER) identifies named entities (people, organisations, locations, dates) and their types. Modern NER systems using fine-tuned BERT or Llama variants achieve F1 scores above 92% on CoNLL-2003.

Question answering comes in extractive form (finding the answer span in a passage) and abstractive form (generating a free-form answer). SQuAD 2.0 remains a standard extractive QA benchmark, while Natural Questions and TriviaQA are common for open-domain settings. Summarisation compresses documents while preserving key information, evaluated with ROUGE scores and increasingly with LLM-based quality judgments. Machine translation between language pairs uses BLEU, COMET, and chrF metrics, with large models achieving near-human quality on high-resource language pairs like English-French and English-German.

Benchmarks

GLUE (General Language Understanding Evaluation) and its harder successor SuperGLUE are collections of nine diverse NLP tasks including linguistic acceptability, sentiment analysis, textual entailment, and question answering. They were the standard benchmarks for BERT-era models but have been saturated (models exceed human performance on SuperGLUE). MMLU (Massive Multitask Language Understanding) tests knowledge across 57 subjects from STEM to humanities and is the most widely used benchmark for comparing LLM knowledge capabilities in 2026.

BIG-bench, a collaboration across hundreds of researchers, goes beyond traditional NLP metrics to test reasoning, common sense, and creativity through 204 diverse tasks. It revealed that many capabilities emerge only at specific model scales, providing crucial insights for the scaling laws that govern model development. For more on how these models work under the hood, see How LLMs work explained.

Production NLP Pipeline

Deploying an NLP model to production requires more than training a good model. The production pipeline spans text ingestion, preprocessing, inference, post-processing, and continuous monitoring. Each stage introduces latency and reliability considerations that determine whether the system meets its service-level objectives.

Text Ingestion and Cleaning

Production text arrives in many forms: API payloads, database records, streaming events, file uploads, webhook callbacks. The ingestion layer must handle encoding detection (UTF-8, ISO-8859-1, Windows-1252), malformed input, injection attacks, and size limits. Cleaning removes HTML entities, normalises line endings, strips control characters, and applies the regex-based preprocessing rules established during development. A cleaning failure — such as passing raw HTML to a model — can produce garbage output or trigger safety filter false positives.

Tokenization and Batching

Tokenization is often the preprocessing bottleneck. The tokenizer must handle the same vocabulary and algorithm used during training — mismatches cause silent degradation. Batching groups multiple inputs to maximise GPU utilisation: padding shorter sequences to the length of the longest in the batch, or using dynamic batching with attention masks. For latency-sensitive applications, batch sizes of 1-4 keep time-to-first-token low, while throughput-oriented applications use batch sizes of 16-64 with sequence lengths of 512-2048.

Inference and Post-Processing

Inference frameworks like vLLM, TensorRT-LLM, and llama.cpp optimise the forward pass through kernel fusion, KV-cache management, and continuous batching. Quantisation to 4-bit or 8-bit reduces memory and improves throughput with minimal quality loss. Post-processing includes decoding strategies (greedy, beam search, top-k, top-p sampling), output filtering via regex or safety classifiers, and response formatting to match the expected API contract.

Latency and Throughput Considerations

Production NLP systems are typically constrained by one of two factors: latency (time to first token for interactive applications) or throughput (tokens per second for batch processing). For interactive chat, target time-to-first-token is under 500 ms, which requires optimising the prefill phase of the transformer. For offline batch processing, throughput of 1,000-10,000 tokens per second per GPU is achievable with the right model size and quantisation level. The choice of model, quantisation, batch size, and hardware must be tuned together to meet both latency and throughput targets — optimising one in isolation often degrades the other.

Conclusion and References

NLP in 2026 rests on a foundation built over decades of research. Tokenization turns text into units the model can process. Embeddings map those units into meaningful vector spaces. Sequence models capture temporal dependencies. The Transformer provides the universal architecture that scales from language understanding to generation. Transfer learning makes it practical to adapt large models to specific tasks. And production pipelines bridge the gap between trained models and deployed systems that serve users reliably.

The field continues to evolve rapidly. Longer context windows (1 million+ tokens), mixture-of-experts architectures, multimodal models that process text, images, and audio together, and increasingly efficient fine-tuning methods are pushing the boundaries of what NLP systems can do. But the fundamentals covered here — how text becomes tokens, how tokens become representations, how representations become predictions — will remain essential regardless of where the field goes next.

For further reading, explore our related guides on how LLMs work, transformer architecture, vectors, tensors, and scalars, prompt engineering, and the LLM comparison guide.