Vectors, Tensors, and Scalars: The Mathematical Foundation of AI

Introduction

Every AI system, from the simplest linear classifier to the largest language model, is built on a single mathematical abstraction: the tensor. Tensors are to AI what atoms are to matter — the fundamental unit from which everything else is composed. Understanding them is not optional for anyone working seriously with machine learning.

This guide builds from the ground up. We start with the simplest mathematical object — a scalar, which is just a single number — and progress through vectors, matrices, and finally tensors of arbitrary dimension. Along the way, we connect each level of abstraction to concrete AI applications: word embeddings as vectors, neural network weights as matrices, and multi-dimensional data as tensors. By the end, you will understand the mathematics that powers every model you use.

Scalars: The Simplest Object

A scalar is a single number. It is the simplest mathematical object — a magnitude without direction. In AI, scalars appear everywhere: learning rates, loss values, regularization coefficients, accuracy scores, and individual pixel values are all scalars.

Properties.

Scalars obey the familiar rules of arithmetic: addition, subtraction, multiplication, division. They have order (3 is less than 5) and can be compared. They have zero dimensions — a scalar is a point on the number line, not a position in a space.

In practice, scalar operations in AI are heavily optimized. When you call loss.backward() in PyTorch, the resulting gradient is a scalar that propagates through the entire computation graph. The learning rate — a scalar — determines how much each parameter updates during training. These single numbers control the behavior of models with hundreds of billions of parameters, which is a remarkable fact worth pausing over.

Vectors: Magnitude and Direction

A vector is an ordered collection of scalars. It has both magnitude (length) and direction. In n-dimensional space, a vector is a position — a point defined by n coordinates.

Vectors are written as arrays: v = [v₁, v₂, ..., vₙ]. The number of elements is the vector's dimension. A 3-dimensional vector lives in 3D space. A 768-dimensional vector lives in a space we cannot visualize but can compute with.

Why vectors matter in AI.

Vectors are the native language of AI because they represent entities in a way that captures relationships. Every word, sentence, image, or user profile in a modern AI system is represented as a vector. The position of that vector in high-dimensional space encodes meaning — words with similar meanings cluster together, images of similar content appear nearby, users with similar preferences occupy overlapping regions.

This is the central insight of representation learning: find a vector space where semantic relationships correspond to geometric relationships. The rest — classification, retrieval, generation — is computation on those vectors.

Vector Operations

Dot product.

The dot product a · b = Σ aᵢ × bᵢ measures how aligned two vectors are. Two parallel vectors pointing in the same direction have a high positive dot product. Perpendicular vectors have a dot product of zero. Opposing vectors have a negative dot product. In neural networks, the dot product of an input vector with a weight vector is the fundamental operation at every neuron — the weighted sum before activation.

Norm (magnitude).

The L2 norm ||v|| = √(v · v)measures a vector's length. Normalizing a vector to unit length preserves its direction while making its magnitude 1. This is critical for cosine similarity, which divides the dot product by the product of the magnitudes.

Cosine similarity.

Cosine similarity measures the angle between two vectors, ignoring their magnitudes: cos(θ) = (a · b) / (||a|| × ||b||). The result ranges from -1 (opposite) to 0 (orthogonal) to 1 (identical direction). In semantic search, cosine similarity is the standard metric because the magnitude of an embedding vector is often meaningless — only its direction encodes semantic content.

The Syntave RAG pipeline uses cosine similarity as its primary relevance metric between query embeddings and document chunk embeddings. It is fast, well-understood, and works across embedding models.

Matrices: 2D Arrays of Numbers

A matrix is a rectangular array of scalars arranged in rows and columns. If a vector is a 1D array, a matrix is a 2D array. It has shape (m × n), where m is the number of rows and n is the number of columns.

Matrices as linear transformations.

A matrix multiplied by a vector produces a new vector. This operation — matrix-vector multiplication — is a linear transformation. It can rotate, scale, reflect, shear, or project the input vector. Every layer in a neural network applies a linear transformation through its weight matrix: output = W × input + b.

Matrix multiplication.

Matrix multiplication is the workhorse of deep learning. When you multiply an (m × n) matrix with an (n × p) matrix, you get an (m × p) result. Each element of the result is the dot product of one row from the first matrix and one column from the second.

In a transformer model, attention scores are computed as Q × Kᵀ — the matrix product of query and key matrices. This single multiplication produces a matrix of attention scores that captures how every token relates to every other token. The transformer architecture guide walks through this in detail.

Tensors: N-Dimensional Arrays

A tensor is a generalization of scalars, vectors, and matrices to arbitrary numbers of dimensions. A scalar is a 0-dimensional tensor. A vector is a 1-dimensional tensor. A matrix is a 2-dimensional tensor. A 3D tensor has depth, width, and height. A 4D tensor adds a batch dimension. And so on.

The order (or rank) of a tensor is the number of dimensions it has. A 4th-order tensor has four indices: T[i][j][k][l]. Each index ranges over the size of that dimension.

Tensors in practice.

Every piece of data in a deep learning framework is a tensor. A batch of 32 color images of size 224x224 is a tensor of shape (32, 3, 224, 224) — batch dimension, RGB channels, height, width. A batch of 64 sentences, each tokenized to 128 tokens and embedded to 768 dimensions, is a tensor of shape (64, 128, 768). A neural network with 4 layers of 512 neurons each has weight tensors of shape (512, 512) per layer.

import numpy as np

# Scalars
x = np.float32(3.14159)         # A single number: scalar
learning_rate = 1e-4            # Also a scalar

# Vectors
v1 = np.array([1.0, 2.0, 3.0])  # 3-dimensional vector
v2 = np.array([4.0, 5.0, 6.0])

# Vector operations
dot_product = np.dot(v1, v2)    # 1*4 + 2*5 + 3*6 = 32
norm_v1 = np.linalg.norm(v1)    # sqrt(1 + 4 + 9) = 3.742
cos_sim = dot_product / (norm_v1 * np.linalg.norm(v2))

# Matrices
M = np.array([[1, 2], [3, 4]])   # 2x2 matrix
N = np.array([[5, 6], [7, 8]])

# Matrix multiplication
product = M @ N                  # [[19, 22], [43, 50]]

# Tensors (3D and beyond)
batch = np.random.randn(32, 3, 224, 224)  # 32 images, 3 channels, 224x224
batch.shape                     # (32, 3, 224, 224)

Tensor Operations

Broadcasting.

Broadcasting lets you perform operations between tensors of different shapes by automatically expanding the smaller tensor. If you add a vector of shape (3,) to a matrix of shape (4, 3), NumPy broadcasts the vector to match the matrix by replicating it along the missing dimension. Broadcasting eliminates explicit loops and enables efficient batch operations.

Reshaping.

Tensor reshaping changes the dimensions without changing the underlying data. A tensor of shape (32, 768) can be reshaped to (8, 4, 768) as long as the total number of elements remains constant. Reshaping is zero-cost in terms of memory — it only changes the metadata that describes the tensor layout.

Reduction.

Reduction operations collapse one or more dimensions by applying an aggregation function. sum(), mean(), max(), and min() are the most common reductions. The dim parameter specifies which dimension to reduce. Pooling layers in convolutional neural networks are essentially local reduction operations.

Understanding these three operations — broadcasting, reshaping, and reduction — covers 90% of tensor manipulation in everyday ML programming. Master them and you can work with any tensor shape.

How Tensors Flow Through Neural Networks

A neural network is a sequence of tensor transformations. Data enters as a tensor. Each layer transforms it. The final layer outputs a tensor. Training adds a backward pass where gradients — also tensors — flow in reverse through the same transformations.

A linear layer y = Wx + b transforms an input tensor of shape (batch, in_features) to an output of shape (batch, out_features) by matrix multiplication. An attention layer computes softmax(Q × Kᵀ / √d) × V, involving three matrix multiplications and a softmax — all tensor operations. A convolution layer applies filters as sliding tensor windows over the input.

The remarkable thing is that every operation in this chain is differentiable. The gradient of the loss with respect to every parameter is a tensor of the same shape as the parameter, computed via the chain rule of calculus. This is automatic differentiation — the mechanism that makes deep learning possible — and it operates entirely on tensors.

We cover the full tensor flow through transformer models in detail in our transformer architecture guide.

Word Embeddings: From One-Hot to Dense Vectors

Before embedding models, words were represented as one-hot vectors: a vector of vocabulary size with a 1 at the word's index and 0 everywhere else. For a vocabulary of 100,000 words, each word was a 100,000-dimensional binary vector with exactly one non-zero element. These vectors were sparse, high-dimensional, and semantically meaningless — "king" and "queen" were as dissimilar as "king" and "zebra".

Word2Vec (Mikolov et al., 2013).

Word2Vec revolutionized NLP by learning dense vector representations through a simple prediction task. The skip-gram variant predicts surrounding words given a target word. The continuous bag-of-words (CBOW) variant predicts a target word from its context. In both cases, the learned weight matrix of the prediction task becomes the embedding space. Words used in similar contexts end up with similar vectors [1].

The resulting embeddings exhibit remarkable semantic structure. king - man + woman ≈ queen. paris - france + italy ≈ rome. These analogies emerge purely from distributional statistics — no explicit semantic knowledge was provided.

# Approximate Word2Vec-style embeddings via Gensim
from gensim.models import Word2Vec

sentences = [
    ["king", "queen", "man", "woman"],
    ["paris", "france", "london", "england"],
    ["walking", "walked", "swimming", "swam"],
]

model = Word2Vec(sentences, vector_size=100, window=5, min_count=1)

# Vector arithmetic: king - man + woman ≈ queen
king = model.wv["king"]
man = model.wv["man"]
woman = model.wv["woman"]
queen = model.wv["queen"]

result = king - man + woman          # In embedding space
similarity = model.wv.cosine_similarities(result, [queen])
print(f"Similarity to 'queen': {similarity[0]:.3f}")  # ~0.75-0.85

GloVe (Pennington et al., 2014).

GloVe (Global Vectors) takes a different approach. Instead of predicting context words, it factorizes the global word co-occurrence matrix. The intuition is that word-word co-occurrence statistics contain rich semantic information that prediction-based methods only capture indirectly. GloVe embeddings tend to perform better on word analogy tasks, while Word2Vec often performs better on similarity tasks [2].

Contextual embeddings (BERT and beyond).

Word2Vec and GloVe produce static embeddings — each word has one vector regardless of context. "Bank" has the same embedding in "river bank" and "investment bank." BERT and other transformer models produce contextual embeddings where the representation of each token depends on its surrounding tokens, resolving polysemy and capturing nuanced contextual meaning [3]. Modern embedding models like text-embedding-3-small and E5 produce 768-3072 dimensional contextual embeddings that power today's most accurate retrieval systems.

Embedding Spaces: Clustering, Analogies, Arithmetic

Once words, sentences, or documents are embedded as vectors, the embedding space itself becomes a representation of knowledge. The geometry of this space encodes relationships.

Clustering.

Related concepts cluster naturally in embedding space. K-means or DBSCAN on document embeddings reveals topical clusters without any supervision. In practice, this is used for document organization, topic modeling, and recommendation systems. The distance between cluster centroids measures topical similarity at a macro level.

Analogies.

As noted, vector arithmetic in embedding space reveals analogies. This is possible because the embedding space encodes relational structure directionally. The vector from "man" to "woman" (the gender direction) is approximately parallel to the vector from "king" to "queen." These directional regularities mean that adding and subtracting embeddings produces meaningful results.

Arithmetic.

Beyond analogies, embedding arithmetic enables compositional semantics. The vector for "hot" + "summer" lands near "sweltering." The vector for "AI" + "healthcare" lands near "medical AI." While this works better with some embedding models than others, the principle holds: semantic composition corresponds to vector addition in well-structured embedding spaces.

Our Syntave documentation covers how we use embedding spaces for retrieval, clustering, and semantic search in production.

Cosine Similarity and Semantic Search

Semantic search is the application of vector operations to information retrieval. Given a query, embed it into the same space as your documents, find the document embeddings with the highest cosine similarity, and return those documents.

Cosine similarity is preferred over Euclidean distance for semantic search because embedding magnitudes are often uninformative. A document and a query about the same topic may have vectors with very different lengths but pointing in the same direction. Cosine similarity captures directional alignment, which is what semantic relevance depends on.

cosine_similarity(a, b) = (a · b) / (||a|| × ||b||)

In practice, embeddings are typically L2-normalized before indexing (||v|| = 1), which simplifies cosine similarity to a dot product. This is why vector databases like Pinecone, Qdrant, and Weaviate use dot product as their default similarity metric — with normalized vectors, dot product equals cosine similarity and is significantly faster to compute.

For a deeper discussion of how semantic search integrates into RAG systems, see our RAG best practices guide.

The Curse of Dimensionality

As the dimension of a vector space increases, its geometry becomes counterintuitive. Distances between random points converge to a constant. The volume of space grows exponentially, making data sparse. Every point is equally far from every other point. This is the curse of dimensionality.

Consider a unit cube in 2D: the longest distance between two points is √2 ≈ 1.41. In 100 dimensions, the longest distance is √100 = 10. But the shortest non-zero distance (between adjacent points on a grid) remains 1. The ratio of farthest to nearest grows as √d, meaning that in high dimensions, the concept of "nearest neighbor" becomes ill-defined — the nearest point is almost as far as the farthest point.

Why it matters for AI.

Modern embedding models produce vectors in 768 to 3072 dimensions. These spaces are subject to the curse of dimensionality. However, the data in these spaces is not uniformly random — it lies on lower-dimensional manifolds within the high-dimensional space. Words and documents occupy only a small, structured region of the full embedding space. This is why similarity search works despite the dimensionality.

The curse of dimensionality also explains why dimensionality reduction techniques like PCA, t-SNE, and UMAP are widely used for visualization and why many vector databases use quantization and Product Quantization (PQ) to compress high-dimensional vectors while preserving relative distances.

PyTorch vs TensorFlow: Tensor Operations Compared

Both PyTorch and TensorFlow provide tensor computation engines with broadly similar APIs but different design philosophies. PyTorch uses a Pythonic, imperative style where tensor operations execute immediately. TensorFlow originally used a graph-based declarative style but has since added eager execution as the default.

import torch
import torch.nn.functional as F

# Create tensors
scalar = torch.tensor(3.14159)
vector = torch.tensor([1.0, 2.0, 3.0])
matrix = torch.tensor([[1, 2], [3, 4]])
tensor_3d = torch.randn(4, 3, 2)     # 4 matrices of shape 3x2

# Reshaping
x = torch.randn(2, 3, 4)
x_flat = x.view(-1)                   # Flatten to 24 elements
x_2d = x.view(6, 4)                   # Reshape to 6x4

# Broadcasting
a = torch.tensor([[1], [2], [3]])     # Shape (3, 1)
b = torch.tensor([10, 20, 30])        # Shape (3,)
c = a + b                             # Shape (3, 3) — broadcast!

# Reduction
x = torch.randn(100, 768)
mean = x.mean(dim=0)                  # Mean over batch, shape (768,)
max_vals = x.max(dim=0).values        # Max over batch

# Cosine similarity (pairwise)
embeddings = torch.randn(10, 384)     # 10 embeddings, dim 384
normed = F.normalize(embeddings, p=2, dim=1)
similarity = normed @ normed.T        # 10x10 similarity matrix

Key differences.

• Auto-diff: PyTorch uses autograd with a dynamic computation graph built on the fly. TensorFlow uses GradientTape with a similar dynamic approach. Both are functionally equivalent for most use cases.
• Device management: PyTorch uses explicit .to(device) calls. TensorFlow uses tf.device() context managers. PyTorch's approach is more flexible for multi-GPU workflows.
• Distribution: PyTorch's DistributedDataParallel is the industry standard for distributed training, used by most LLM training frameworks. TensorFlow's tf.distribute.Strategy is less widely adopted.
• Production deployment: TensorFlow's TensorRT and TFLite ecosystems provide more mature deployment options. PyTorch has been catching up with torch.compile, TorchScript, and torch.onnx export.

Both frameworks use the same fundamental tensor operations and share the same mathematical foundations. The choice between them is increasingly a matter of ecosystem preference rather than technical capability. Our guide to GPU parallel processing covers how both frameworks leverage GPU tensor cores for accelerated computation.

Practical Tensor Programming

Rule 1: Think in shapes.

Every tensor operation transforms shapes. Before writing any tensor code, know the input shape, the output shape, and how the operation transforms one to the other. The most common source of bugs is shape mismatch — a matrix multiplication where the inner dimensions disagree, a broadcast that silently creates an unintended shape.

Rule 2: Avoid explicit loops.

Tensor operations are vectorized — they operate on entire arrays at once. Python loops over tensor elements are orders of magnitude slower than native tensor operations. The reason is that native tensor operations are implemented in C/C++ and CUDA, with full parallelism on GPU. A loop that iterates over 100,000 elements in Python runs 100,000 Python bytecode operations. The same operation as a native tensor call runs one CUDA kernel launch.

Rule 3: Normalize before distance.

When computing pairwise similarities, normalize your embeddings first. L2-normalization followed by matrix multiplication is significantly faster than computing cosine similarity element by element, and the result is mathematically identical.

Rule 4: Profile before optimizing.

Tensor operations are fast, but some are faster than others. torch.bmm (batched matrix multiply) is faster than looping over individual matrix multiplies. torch.nn.functional.scaled_dot_product_attention is faster than implementing attention manually. Use flash attention when available. Use mixed precision training (torch.float16 or bfloat16) for 2x throughput on tensor core-capable GPUs.

The connection between tensor programming and GPU hardware — tensor cores, memory bandwidth, kernel fusion — is the subject of our guide to GPUs and parallel processing.

Conclusion

Scalars, vectors, matrices, and tensors form a hierarchy of abstraction that underlies every AI system. A scalar is a single number. A vector is a list of numbers that represents a point in space. A matrix is a 2D array that transforms vectors. A tensor generalizes this to any number of dimensions, providing the universal data structure for machine learning.

Understanding these mathematical foundations transforms how you work with AI. When you normalize embeddings before computing similarity, you are applying vector geometry. When you batch your data into a 4D tensor, you are leveraging tensor parallelism. When you choose between two models, you are implicitly comparing their learned representations in embedding space.

The mathematics of tensors is not a prerequisite for using AI — it is a prerequisite for understanding it. Build your intuition for these abstractions, and every AI technique becomes clearer. The vectors you compute today encode the knowledge that powers tomorrow's systems.

For further reading: 3Blue1Brown's linear algebra series provides the best geometric intuition [4]. The official PyTorch and NumPy documentation are the definitive references for tensor operations [5][6]. And our transformer architecture guide shows how all of this mathematics comes together in the models that power modern AI.