Deep Learning Foundations: How Linear Algebra Powers Neural Networks#

Introduction to Vectors#

A vector is an array of numbers arranged in a particular order. For the purposes of deep learning:

• A vector can represent an input sample (e.g., pixel intensities of an image flattened into a single, long vector).
• A vector can represent model parameters.
• A vector can represent the activation of a particular layer in a neural network.

In linear algebra terms, vectors are often treated as elements of a vector space (e.g., ℝⁿ for real vectors). What we typically do with vectors in deep learning:

1. Dot products: Combine pairs of vectors to measure similarity or compute sums weighted by some coefficients.
2. Scalar/vector addition: Adjust each dimension by the same amount or add one vector to another (like adding a bias term to a weight vector).

Linear Combinations and Span#

One of the fundamental ideas is the span of a set of vectors. If you have a set of vectors, the span is all possible linear combinations of these vectors. In neural networks, you often try to learn a weight matrix that “spans�?the space of potential transformations of the input. If your vectors are linearly independent, they can describe a larger space of variations.

Norms#

In deep learning, we often measure the magnitude (size) of vectors or how far apart two vectors are. This is done through vector norms. The L2 norm (Euclidean norm) is especially useful:

‖x‖₂ = �?x₁�?+ x₂�?+ �?+ xₙ�?

We frequently rely on L2 norms when designing regularization (like weight decay). This normalizes or constrains the weight vectors so they don’t blow up.

Definition of a Matrix#

A matrix is a 2D array of numbers. In deep learning, crucial uses of matrices include:

• Weight matrices in fully connected layers.
• Parameter matrices for transformations in recurrent networks.
• Filters in convolutional layers can be “unrolled�?into matrices.

Matrix as a Transformation#

You can see a matrix W multiplying a vector x as a linear transformation T:

1
y = T(x) = W x

Each column of W can be viewed as the contribution to the output dimension from each input dimension. That is, matrix multiplication is summing up each input vector scaled by a particular weight.

Transpose, Inverse, and Orthogonality#

• The transpose (Wᵀ) flips rows and columns; it is crucial in operations like backpropagation when we want gradients with respect to weights.
• The inverse (W⁻�? is the matrix that undoes the transformation of W, if it exists. In deep learning, we often work with non-square or singular matrices that do not have an inverse.
• Two vectors (or matrices) are orthogonal if their dot product is zero (for vectors) or if WᵀW = I for square matrices. Orthogonality has broad implications, such as preserving lengths and angles, and orthonormal transformations are widely used in certain architectures to help with stable training.

A key point is that linear algebra organizes these transformations systematically. By understanding matrix operations, we can reason more deeply about what neural network architectures are doing.

Matrix Multiplication as a Core Operation#

The classic neural network operation can be expressed as:

1
Z = W * X + b

where:

• W is a weight matrix of dimension (output_size × input_size).
• X is the input vector (or matrix in the case of a batch).
• b is a bias vector, often broadcasted across the output.

You can generalize this: each neuron’s output is a linear combination of all inputs, plus the bias. Matrix multiplication is at the heart of these combinations.

Element-wise Operations#

Neural networks also heavily use element-wise (Hadamard) operations:

• Element-wise multiplication (�?: Multiply each element in a vector or matrix by the corresponding element in another vector or matrix of the same shape.
• Element-wise activation functions: After computing Z = WX + b, you apply an activation function σ(�? to each element of Z.

For example, the ReLU function (Rectified Linear Unit) is:

1
ReLU(z) = max(0, z) (element-wise)

Broadcasting#

In many deep learning frameworks (e.g., NumPy, PyTorch, TensorFlow), operations like “add a bias vector�?to a matrix are done via broadcasting. If Z has shape (batch_size × output_size) and b has shape (output_size), the bias is automatically broadcasted along the batch dimension. This is a subtle detail but crucial for designing code that matches your mathematical intuition.

Rank and Dimensionality#

The rank of a matrix is the dimension of the space spanned by its columns (or rows). Why does rank matter in deep learning?

• If your weight matrix has low rank, then it can only capture a small range of transformations.
• In some advanced neural network compression techniques, you might use low-rank approximations of weight matrices to reduce the parameter count.

Eigenvalues and Eigenvectors#

Eigenvalues and eigenvectors of a matrix W satisfy:

1
W v = λ v

where v is an eigenvector and λ is the corresponding eigenvalue.

Eigenvalues often tell us about the “stretch�?or “shrink�?factor induced by the transformation W. In deep learning, understanding eigenvalues can sometimes help in analyzing stability or in methods like PCA (Principal Component Analysis) for dimensionality reduction and data preprocessing.

Singular Value Decomposition (SVD) and PCA#

SVD and PCA are close cousins; both break down matrices in ways that reveal hidden structure. SVD of W is:

1
W = U Σ Vᵀ

• U contains orthonormal eigenvectors for WWᵀ.
• V contains orthonormal eigenvectors for WᵀW.
• Σ is a diagonal matrix of singular values.

SVD is used for dimensionality reduction, compressing matrices, and analyzing how the transformation W distorts space. PCA is a special case of SVD for covariance matrices, commonly used in data preprocessing before training neural networks.

Orthogonal and Unitary Matrices#

If W is orthogonal (in real space) or unitary (in complex space), then WᵀW = I. Such transformations preserve vector lengths and angles. In recurrent neural networks (RNNs), orthogonal or unitary weight matrices sometimes help alleviate issues like exploding/vanishing gradients. There are specialized RNN architectures that constrain or parameterize weights to be orthogonal to enhance stability during training.

Step-by-Step Guide#

1. Initialize Weights and Biases
   Let’s say our input dimension is D_in = 2, hidden dimension is H = 3, and output dimension is D_out = 1. We can represent the parameters as:

   • W�? (2 × 3)
   • b�? (3,)
   • W�? (3 × 1)
   • b�? (1,)

2. Forward Pass

   1. Compute hidden = ReLU(XW�?+ b�?.
   2. Compute output = hiddenW�?+ b�?

3. Loss Function
   A simple mean squared error (MSE) for regression tasks: L = (1/N) �?(y_pred - y_true)²

4. Backward Pass (Gradients)
   We compute partial derivatives of L with respect to W�? b�? W�? b�?via chain rule. All these partial derivatives revolve around matrix operations.

5. Parameter Update
   W�?:= W�?- η * dW�? b�?:= b�?- η * db�? …and similarly for W�?and b�? where η is the learning rate.

Below is the simplified implementation:

1
import numpy as np
2

3
# Generate dummy data
4
np.random.seed(0)
5
X = np.random.randn(5, 2)  # 5 samples, each dimension is 2
6
Y = np.random.randn(5, 1)  # 5 target values
7

8
# Dimensions
9
D_in, H, D_out = 2, 3, 1
10

11
# Initialize parameters
12
W1 = np.random.randn(D_in, H)
13
b1 = np.zeros(H)
14
W2 = np.random.randn(H, D_out)
15
b2 = np.zeros(D_out)
16

17
# Hyperparameters
18
learning_rate = 0.01
19
num_epochs = 1000
20

21
def relu(x):
22
    return np.maximum(0, x)
23

24
for epoch in range(num_epochs):
25
    # Forward pass
26
    hidden = relu(X.dot(W1) + b1)
27
    y_pred = hidden.dot(W2) + b2
28

29
    # Loss (Mean Squared Error)
30
    loss = np.mean((y_pred - Y)**2)
31

32
    # Backpropagation
33
    # dLoss/d(y_pred) = (2/N) * (y_pred - Y)
34
    grad_y_pred = (2.0 / X.shape[0]) * (y_pred - Y)
35

36
    # dLoss/dW2 = hiddenᵀ * grad_y_pred
37
    grad_W2 = hidden.T.dot(grad_y_pred)
38
    grad_b2 = np.sum(grad_y_pred, axis=0)
39

40
    # dLoss/d(hidden) = grad_y_pred * W2ᵀ
41
    grad_hidden = grad_y_pred.dot(W2.T)
42
    grad_hidden[hidden <= 0] = 0  # derivative of ReLU
43

44
    # dLoss/dW1 = Xᵀ * grad_hidden
45
    grad_W1 = X.T.dot(grad_hidden)
46
    grad_b1 = np.sum(grad_hidden, axis=0)
47

48
    # Update parameters
49
    W2 -= learning_rate * grad_W2
50
    b2 -= learning_rate * grad_b2
51
    W1 -= learning_rate * grad_W1
52
    b1 -= learning_rate * grad_b1
53

54
    if (epoch+1) % 200 == 0:
55
        print(f"Epoch {epoch+1}, Loss = {loss:.4f}")
56

57
print("Final predicted outputs:\n", y_pred)

Batch Normalization and Linear Algebra#

When you apply batch normalization, you are computing mean and variance statistics across batches of data and then scaling and shifting (which are again linear transformations). The formula is:

1
x_norm = (x - μ) / �?σ² + ε)
2
y = γ · x_norm + β

Here, (x - μ)/�?σ² + ε) is element-wise normalization, while γ and β are trainable parameters that scale and shift, essentially performing another linear operation.

Convolution as Matrix Multiplication#

A convolutional layer can be viewed in matrix form by flattening local patches of the input and multiplying by the filter weights. In practice, this is optimized via specialized ops, but conceptually it is still a multiplication of a “patch matrix�?by a “filter matrix.�?Understanding this equivalence can help you:

• Interpret how convolutions reduce to matrix multiplications.
• Utilize techniques for compressing or accelerating convolution operations (like FFT-based methods).

Recurrent Neural Networks and Orthogonal Matrices#

Recurrent neural networks, especially vanilla RNNs, can benefit from orthogonal or unitary weight matrices for stable gradient flow. If W is orthogonal, then ‖Wx‖₂ = ‖x‖₂. This property ensures that repeated multiplications by W do not shrink or explode vector norms as quickly, which can help mitigate vanishing or exploding gradients.

Low-Rank Factorizations in Model Compression#

Deep learning models can have a huge number of parameters. Techniques like SVD-based approaches look for low-rank approximations of the weight matrices. For example, if W is approximated by:

1
W �?U_r Σ_r V_rᵀ

where r < rank(W), this reduces the number of parameters drastically (since U_r and V_r will be smaller). This technique can be used to:

• Compress a model to run on resource-constrained devices.
• Speed up inference by reducing the multiplication cost.

Attention Mechanisms and Linear Algebra#

Transformers rely on attention, often described with operations such as:

1
Attention(Q, K, V) = softmax(Q Kᵀ / √d_k) V

Here you see:

• Q, K, and V are matrices gleaned from input embeddings.
• The operation QKᵀ is a crucial matrix multiplication, whose output’s size depends on the batch and sequence length.
• A softmax is applied row-wise to yield the attention weights.
• Finally, another matrix multiplication with V merges the “values�?with the attention weights.

All these steps underscore how advanced neural architectures still revolve around fundamental matrix multiplications, dot products, and transformations.