Vectors & Matrices: The Language of ML

Every ML model you'll ever use — linear regression, random forests, GPT-4 — is ultimately doing arithmetic on vectors and matrices. Learn to read that language fluently and the rest gets dramatically easier.

1. What is a vector?

A vector is an ordered list of numbers. Vectors show up everywhere in ML because almost anything we want to reason about — a user, a document, a pixel, a word — can be described by a fixed list of measurements, and once you have a list of numbers, you can do math on it.

Why vectors

An ML model takes some input and produces some output. The input could be a user row, a photograph, an email, or a sentence — wildly different things. The model can't deal with "a user" in the abstract. It needs numbers — specifically, fixed-length, fixed-order lists of numbers. That's a vector. Converting real-world objects into vectors — and choosing which numbers to include — is half the job of an ML engineer. The other half is what you do to those vectors once you have them.

A concrete example

Take a single user in your database. Pick three things you know about them:

age            = 34
lifetime_spend = 487.50
days_active    = 120

Those are three Python variables. Pack them into a list in a fixed order — and agree the order never changes — and you have a vector:

The vertical bracket notation writes "three numbers, stacked into a column." Some papers write it horizontally as $u = (34, 487.50, 120)$; they mean the same thing. In code:

import numpy as np
u = np.array([34, 487.50, 120])
u.shape   # (3,)   ← "a 1-dimensional array of length 3"

That trailing comma in (3,) trips people up. It's NumPy's way of saying this is a Python tuple with one entry — the entry being 3. NumPy always reports array shapes as tuples because shapes can have multiple numbers (you'll see (1000, 7) for a matrix soon). A one-dimensional array still gets a tuple, just with one entry. The comma is what tells Python "this is a tuple, not a parenthesized integer."

Order matters — a lot

If I swap positions to [120, 34, 487.50], I've made a different vector. In ML terms, I've changed which feature lives in which dimension. A model that was trained expecting "age" in position 0 will happily take "days_active" there and quietly produce garbage predictions. No error. No warning. Just wrong answers.

Each number inside a vector is a feature. The whole vector — all the features for one thing — is a feature vector, equivalently a sample, or a data point. All three terms appear in docs and papers interchangeably.

Two ways to picture a vector

There are two mental models for a vector. They describe the same object and emphasize different things. Code wants the first; geometric intuition lives in the second.

1. As a list. Numbers in a specific order. This is the computational picture — what's in memory, what NumPy stores, what you pass between functions. Most of the time this is the only view needed to write correct code.

2. As an arrow. In 2D, the vector [3, 1] is an arrow starting at the origin — the point (0, 0) — and ending at (3, 1). In 3D, same idea with three coordinates. Beyond 3D we can't draw the arrow, but we still talk about it: direction, length, angle. The math works identically; 2D pictures are a tool for building intuition.

The arrow view matters because operations on vectors have geometric meaning, and that meaning lets you reason about what a model is actually doing. Adding two vectors chains arrows tip-to-tail. Multiplying by 2 doubles an arrow's length. Multiplying by -1 flips it to point the other way. When the geometry looks simple, the algebra is simple too.

Play with it

Drag the orange dot to move the tip of the vector. Three things to try:

• The two numbers in the readout update in real time. They are the vector — list and arrow are the same thing.
• Drag the tip below the x-axis or left of the y-axis. Vectors can have negative components; they point the other way.
• Drag the tip exactly onto the origin. The vector [0, 0] is an arrow of zero length — the zero vector. It has no direction, and it's still a valid vector.

Self-check before moving on

2. Vector operations (and what they mean)

This section introduces four operations — addition, scalar multiplication, dot product, and norm — and explains what each one means, not just how to compute it. All four show up in every ML model, usually under the hood.

Addition: combining effects

Add vectors element-wise: position 0 to position 0, position 1 to position 1, and so on. That's the computational rule.

Geometrically, place the second arrow's tail at the first arrow's tip. The sum is the arrow from the original origin to where you end up. If the first vector takes you "3 east and 1 north" and the second takes you "1 east and 2 north," the combination takes you "4 east and 3 north." Addition chains directions.

One reason to add feature vectors: in learned word embeddings, vector arithmetic roughly captures meaning.

king_vec   - man_vec   + woman_vec   ≈   queen_vec

This doesn't hold perfectly — the pop-science version oversimplifies — but "+ woman − man" literally means "shift the meaning in the same direction that takes a male word to a female word." Addition of vectors is composition of whatever those vectors represent.

a = np.array([3, 1])
b = np.array([1, 2])
a + b  # array([4, 3])

Scalar multiplication: stretching

A scalar — just a single number, not a vector or matrix — multiplied into a vector multiplies every component. Geometrically: it stretches the arrow. 2·v doubles the length. 0.5·v halves it. -1·v flips it to point the other way. 0·v collapses it to the origin.

Scalar multiplication is everywhere in ML. Re-weighting features (multiply a feature by 100 to convert meters to centimeters). Optimizer updates during training — the gradient descent rule is w = w - learning_rate * gradient, and the learning rate is a scalar that shrinks the update step. Every time you scale a vector, you're doing this.

Dot product: measuring alignment

The dot product takes two vectors and returns a single number. It is the most important operation in this lesson, and arguably in the whole module.

The formula: multiply each corresponding pair of components, then sum the products.

Step through the calculation for a = [3, 1] and b = [2, 4]:

1. Multiply position 0 of a by position 0 of b: 3 × 2 = 6
2. Multiply position 1 of a by position 1 of b: 1 × 4 = 4
3. Add the products: 6 + 4 = 10
4. Return the single number: 10

In NumPy, there are two equivalent spellings — np.dot(a, b) and the @ operator. The @ operator is matrix multiplication; for 1D vectors it equals the dot product. Run this and confirm the answer matches the hand calculation above:

That's the computation. What it measures: the dot product is a number that tells you how much two vectors point in the same direction.

• Large positive dot product → the arrows point in similar directions.
• Zero dot product → the arrows are perpendicular. They share no directional overlap. (In ML-speak: "orthogonal.")
• Negative dot product → the arrows point in roughly opposite directions. Anti-aligned.

Why does the multiply-and-sum formula measure alignment? When two vectors point the same way, their components share signs, so every pairwise product is positive and they sum to a large positive. When they point opposite, signs disagree, products are negative, and the sum is negative. When perpendicular, the positive and negative products cancel. Run the three cases and compare:

The perpendicular case is the one worth pausing on — it's how ML decides two features (or two embedding directions) carry independent information. A zero dot product means neither vector contributes anything to the other's direction.

Norm: measuring length

The norm of a vector — usually written ||v|| — is its length. The formula above is specifically the L2 norm (also called "Euclidean norm" because it's the one from ordinary Euclidean geometry). It's literally the Pythagorean theorem generalized to any number of dimensions. For [3, 4], the norm is √(9 + 16) = √25 = 5 — same as the hypotenuse of a 3-4-5 right triangle.

Three reasons vector length matters in ML:

• Distance between points. The distance between two data points a and b is ||a - b||. This is the foundation of k-nearest neighbors, clustering (k-means), and anomaly detection.
• Normalizing vectors. Divide a vector by its norm and you get a unit vector — same direction, length 1. We call this normalizing the vector. Useful when only direction matters, not magnitude. It's the basis of cosine similarity.
• Regularization. To prevent overfitting, many models add a penalty on the norm of the weight vector — effectively telling the model "keep the weights small." This is the entire idea behind L2 regularization (aka ridge regression).

v = np.array([3, 4])
np.linalg.norm(v)           # 5.0

# Normalize to unit length
v / np.linalg.norm(v)       # array([0.6, 0.8]) — same direction, length 1

# Distance between two data points
a = np.array([1, 1])
b = np.array([4, 5])
np.linalg.norm(a - b)       # 5.0 — a is 5 units away from b

3. What is a matrix?

A matrix is a 2D grid of numbers — rows and columns, like a spreadsheet or a SQL result set. The conventions and operations attached to it are what make it useful.

This is a 2×3 matrix: 2 rows, 3 columns. "m by n" (written m × n) always means m rows by n columns, in that order. The convention never varies.

A = np.array([[1, 2, 3],
              [4, 5, 6]])
A.shape   # (2, 3) — 2 rows, 3 columns

Your dataset is a matrix

When you run SELECT age, spend, days FROM users LIMIT 1000, you get back a 1000-row, 3-column result. That's a matrix. In ML code the feature matrix is almost always named X, with shape (n_samples, n_features).

• Each row is one sample — one user, one document, one image. It's a feature vector (the thing from Section 1).
• Each column is one feature across all samples — for example, the "age" values for every user in the batch.

Other names for this exact object: data matrix, feature matrix, design matrix (the stats-paper term), or just X. They all refer to the same (n_samples, n_features) table.

Transpose: swapping rows and columns

The transpose of a matrix flips rows and columns — what was row 1 becomes column 1, what was row 2 becomes column 2, and so on. Written $A^T$ in math, A.T in NumPy.

Element by element: the entry at row 1, column 1 of A (which is 1) stays at row 1, column 1 of Aᵀ. The entry at row 1, column 2 of A (which is 2) moves to row 2, column 1 of Aᵀ. In general, A[i][j] becomes Aᵀ[j][i] — indices swap. Shape-wise, a (3, 2) matrix transposes to a (2, 3) matrix. A (1000, 7) matrix transposes to (7, 1000).

A.T       # transpose — rows become columns; shape swaps
A.T.T     # transpose of a transpose is back to the original A

Transpose exists because many matrix operations have strict shape requirements (the next section), and transposing is the standard tool for reshaping things so operations line up. The ordinary least squares formula is w = (XᵀX)⁻¹ Xᵀ y — full of transposes because they're the glue that makes the shapes work out. Understanding that formula is a later-module concern; for now, note transpose's role as shape glue.

Matrix multiplication

Matrix multiplication is where most people bounce off the math. The rule is mechanical but needs to be seen in slow motion the first time.

For A @ B where A is m × n and B is n × p, the result is an m × p matrix. The inner dimensions — A's columns and B's rows — must match. The outer dimensions become the shape of the result.

A visual for shape-matching:

    A         @        B        =     result
 (m × n)           (n × p)            (m × p)
       └─────────────┘
        must match
       (inner dims)

Each entry in the result is itself a dot product. Specifically, entry at row i, column j of the result is the dot product of row i of A with column j of B:

The formula says: for each pair (row of A, column of B), compute a dot product and place the result in that position. A worked example, step by step:

Both are 2×2. The inner dimensions match (both are 2). The result will be 2×2. Compute it entry by entry:

• Result[0][0] = row 0 of A · column 0 of B = [1, 2] · [5, 7] = 1·5 + 2·7 = 5 + 14 = 19
• Result[0][1] = row 0 of A · column 1 of B = [1, 2] · [6, 8] = 1·6 + 2·8 = 6 + 16 = 22
• Result[1][0] = row 1 of A · column 0 of B = [3, 4] · [5, 7] = 3·5 + 4·7 = 15 + 28 = 43
• Result[1][1] = row 1 of A · column 1 of B = [3, 4] · [6, 8] = 3·6 + 4·8 = 18 + 32 = 50

A = np.array([[1, 2], [3, 4]])
B = np.array([[5, 6], [7, 8]])

A @ B        # matrix multiplication — this is the one you want
# array([[19, 22],
#        [43, 50]])

A * B        # element-wise multiplication — DIFFERENT operation, be careful
# array([[ 5, 12],
#        [21, 32]])

A @ B is not the same as B @ A, even when both are defined. Sometimes only one ordering is defined at all — if A is (3, 4) and B is (4, 5), then A @ B works (result (3, 5)) but B @ A errors (inner dims 5 and 3 don't match). Matrix multiplication is non-commutative; order always matters.

4. Matrix-vector multiplication: where ML lives

A special case of matrix multiplication deserves its own section: a matrix times a vector. If A is m × n and v has length n (shape (n,)), then A @ v is a vector of length m (shape (m,)). This operation is the core of almost every model you'll train, and there are two equivalent angles on it.

Two equivalent views of this operation. Same answer; different perspectives.

View 1: as a stack of dot products

The i-th output is the dot product of row i of A with v. If A has m rows, that's m dot products, stacked into a length-m vector.

• Row 0 of A is [2, 0]. Dot with [1, 4]: 2·1 + 0·4 = 2. That's output[0].
• Row 1 of A is [0, 3]. Dot with [1, 4]: 0·1 + 3·4 = 12. That's output[1].
• Result: [2, 12].

This is the computational view — it's what NumPy does under the hood. It's also the view that makes "linear regression is a matrix-vector product" fall into place.

View 2: as a linear combination of A's columns

Equivalently: A @ v equals "v[0] times column 0 of A, plus v[1] times column 1 of A, plus …". For our example:

Same answer, different framing. The vector v says "take 1 copy of A's first column and 4 copies of A's second column, and add them up." This is a linear combination of A's columns — a weighted sum.

The second view is what lets you understand what a matrix does geometrically: a matrix acts on an input vector by returning a specific combination of its own columns. That framing is the content of Lesson 2 (matrices as transformations) and underlies eigenvalues and PCA in Lesson 4. Both views are correct; advanced ML concepts lean heavily on view 2.

Linear regression in one line

y_pred = X @ w      # shape: (n_samples,) — one prediction per row

5. Shapes are everything

The hardest part of deep learning is not the math — it's keeping track of tensor shapes as data flows through a model. A 3-line neural network function can have five implicit shape assumptions, and one wrong guess makes everything silently wrong. Engineers who develop shape fluency make fewer bugs; engineers who don't spend their afternoons in pdb.

Shape rules for everything in this lesson:

The shape-asserting habit

Production ML code is full of assert statements checking expected shapes. They're cheap to write, they fail loudly the moment something's wrong, and they document the function's contract inline.

def predict(X, w):
    """
    Given features X with shape (n_samples, n_features)
    and weights w with shape (n_features,),
    return predictions with shape (n_samples,).
    """
    assert X.ndim == 2, f"X should be 2D, got shape {X.shape}"
    assert w.ndim == 1, f"w should be 1D, got shape {w.shape}"
    assert X.shape[1] == w.shape[0], (
        f"shape mismatch: X has {X.shape[1]} features, "
        f"w has {w.shape[0]} weights"
    )
    return X @ w

The alternative is debugging at 11pm because predictions were silently wrong-shape and downstream code took the first element of a broadcast result as "the answer." The assert pattern is a small investment for substantially fewer bugs. The NumPy ndarray docs cover every attribute — .shape, .ndim, .dtype, .T — and repay a skim.

6. Broadcasting

Broadcasting is NumPy's rule for performing element-wise operations on arrays of different but compatible shapes, without a for-loop. It's how you subtract a mean vector from every row of a data matrix, add a bias vector to every sample in a batch, or multiply every column by a scaling factor. Almost every per-row or per-column transformation in NumPy/PyTorch code uses broadcasting.

Why bother?

Without broadcasting, you'd write explicit nested loops:

# The painful way
for i in range(X.shape[0]):
    for j in range(X.shape[1]):
        X[i, j] = X[i, j] - means[j]

That's slow (Python loops are interpreted), verbose, and easy to get wrong. With broadcasting, it's one line that runs in optimized C under the hood and reads almost like math:

X_centered = X - means

The rule, in detail

When shapes don't match, NumPy tries to "stretch" the smaller array along missing or length-1 dimensions so the shapes align. It only stretches when the result is unambiguous; otherwise it raises an error.

# Subtract the mean of each feature from every row
X = np.random.randn(1000, 5)    # shape (1000, 5) — 1000 samples, 5 features
means = X.mean(axis=0)           # shape (5,) — one mean per feature
X_centered = X - means           # broadcast: (1000, 5) - (5,) → (1000, 5)
#                                  means is conceptually tiled to every row

Step by step, broadcasting did the following:

1. X has shape (1000, 5). means has shape (5,).
2. NumPy aligns them from the rightmost axis. X's last axis is 5; means's last (and only) axis is 5. Match.
3. X has an extra axis on the left (1000). means has nothing there. NumPy treats means as if it were tiled to (1000, 5) — the same (5,) vector, conceptually copied 1000 times.
4. The subtraction happens element-wise on the now-aligned shapes. Result is (1000, 5).

Another common case — adding a bias to every row of a batch:

batch = np.zeros((4, 3))              # shape (4, 3) — 4 samples, 3 features
bias = np.array([1.0, 2.0, 3.0])      # shape (3,)
batch + bias                          # shape (4, 3) — bias added to every row
# array([[1., 2., 3.],
#        [1., 2., 3.],
#        [1., 2., 3.],
#        [1., 2., 3.]])

When broadcasting errors

If the shapes are incompatible — axes that are neither the same length nor length-1 — you get a ValueError:

a = np.zeros((3, 4))
b = np.zeros((3,))
a + b     # ERROR: shapes (3,4) and (3,) cannot broadcast
#           the trailing axis of b (3) must match a's trailing axis (4), and it doesn't

The fix is usually a reshape. b[:, None] inserts a new axis and turns (3,) into (3, 1), which can broadcast against (3, 4) — NumPy stretches the length-1 axis to 4. Mentally performing this reshape is a reliable sign of broadcasting fluency.

7. Self-check

Answer these without scrolling back up. Each question is checkable — wrong answers reveal the full reasoning so you know what to re-read.

For any that were fuzzy, jump back to the relevant section — or open the matrix visualizer to watch shapes transform live.