From Concept to Code: Practical Linear Algebra in Python#

1.1 Scalars, Vectors, and Matrices#

• Scalar
  A scalar is a single number. In linear algebra, scalars usually come from real numbers �?(although they can come from complex numbers �?or other fields). For instance, 3.14 is a scalar.

• Vector
  A vector is an ordered list of numbers—an element in ℝⁿ (n-dimensional space). You can think of a vector as either a one-dimensional array or a point in space. For example, [2, 5, -1] is a vector in ℝ�?

• Matrix
  A matrix is a rectangular array of numbers arranged in rows and columns. For instance,
  [ [1, 2, 3],
  [4, 5, 6],
  [7, 8, 9]
  ]
  is a 3×3 matrix. A matrix can have any number of rows (m) and columns (n), thus often referred to as an m×n matrix.

1.2 Matrix Properties and Notation#

• Square Matrix
  A matrix with the same number of rows and columns (e.g., 3×3, 4×4). Square matrices are particularly useful because many important operations (determinants, inverses) are defined only for them.

• Diagonal Matrix
  A square matrix where all non-diagonal elements are zero. Example:
  [ [d�? 0, 0],
  [0, d�? 0],
  [0, 0, d₃]
  ]

• Identity Matrix (I)
  A diagonal matrix with ones on the diagonal. It effectively leaves a vector unchanged when multiplied by it:
  I =
  [ [1, 0, 0],
  [0, 1, 0],
  [0, 0, 1]
  ]

• Zero Matrix (0)
  A matrix where all entries are zero; acts as the additive identity in matrix arithmetic.

2.1 Vector Addition and Scalar Multiplication#

• Vector Addition: Add two vectors component-wise.
  If u = [u�? u�? u₃] and v = [v�? v�? v₃], then
  u + v = [u�?+ v�? u�?+ v�? u�?+ v₃]

• Scalar Multiplication: Multiply each component of the vector by the scalar.
  If α is a scalar and u is a vector, then
  αu = [αu�? αu�? αu₃]

2.2 Dot Product and Cross Product#

• Dot Product (Inner Product): For two vectors u and v of the same dimension,
  u · v = u₁v�?+ u₂v�?+ … + uₙv�? This operation measures the “similarity�?of two vectors and has geometric interpretations related to projection and angles.

• Cross Product: For two 3D vectors u and v,
  u × v =
  [ (u₂v�?- u₃v�?,
  (u₃v�?- u₁v�?,
  (u₁v�?- u₂v�?
  ]
  The resulting vector is orthogonal to both u and v.

2.3 Matrix Multiplication and Division#

• Matrix Multiplication: The product AB of an m×n matrix A and an n×p matrix B is an m×p matrix. The element in the i-th row and j-th column of AB is given by the dot product of the i-th row of A with the j-th column of B.

• Matrix Division: In general, if A is invertible, then dividing by A can be seen as multiplying by its inverse (A⁻�?. For Non-square or singular matrices, more advanced techniques (pseudo-inverses, for instance) are used.

2.4 Transpose, Inverse, and Determinant#

• Transpose (Aᵀ): The rows of A become the columns of Aᵀ, and the columns of A become the rows. For instance,
  A =
  [ [1, 2],
  [3, 4],
  [5, 6]
  ]
  �? Aᵀ =
  [ [1, 3, 5],
  [2, 4, 6]
  ]

• Inverse (A⁻�?: The matrix A⁻�?is the matrix that satisfies A · A⁻�?= I. A matrix must be square and nonsingular (non-zero determinant) to have an inverse.

• Determinant (det(A)): A single number that can be computed from the elements of a square matrix. It has geometric interpretations (like area or volume scaling) and tells us if A is invertible (if det(A) �?0, then A is invertible).

3.1 Creating Arrays and Basic Operations#

Install NumPy with:
pip install numpy

Then, begin your Python script or notebook:

1
import numpy as np
2

3
# Creating vectors (1D arrays)
4
u = np.array([1, 2, 3])
5
v = np.array([4, 5, 6])
6

7
# Vector addition
8
w = u + v
9
print("u + v =", w)
10

11
# Creating matrices (2D arrays)
12
A = np.array([[1, 2], [3, 4]])
13
B = np.array([[5, 6], [7, 8]])
14
C = A + B  # element-wise addition
15
print("A + B =\n", C)

3.2 Reshaping, Concatenating, and Splitting#

NumPy gives you flexibility in changing the shape of arrays and combining or splitting them:

1
x = np.array([1, 2, 3, 4, 5, 6])
2
print("Original x:", x)
3
print("Shape:", x.shape)
4

5
# Reshape into 2D array (2x3)
6
X = x.reshape(2, 3)
7
print("Reshaped X:\n", X)
8
print("New shape:", X.shape)
9

10
# Concatenate arrays
11
Y = np.array([[7, 8, 9], [10, 11, 12]])
12
Z = np.concatenate((X, Y), axis=0)  # Stack vertically
13
print("Concatenated Z:\n", Z)

3.3 Indexing and Slicing#

Pythonic slicing is extremely powerful for subsetting arrays:

1
M = np.array([[ 1,  2,  3],
2
              [ 4,  5,  6],
3
              [ 7,  8,  9],
4
              [10, 11, 12]])
5

6
# First row
7
row0 = M[0, :]
8
# Last column
9
col_last = M[:, -1]
10
# Sub-matrix of first two rows and last two columns
11
sub_mat = M[0:2, 1:3]
12

13
print("row0 =", row0)
14
print("col_last =", col_last)
15
print("sub_mat =\n", sub_mat)

3.4 Performance Considerations#

NumPy arrays are efficient because of several reasons:

1. Vectorized Operations: Element-wise array methods in NumPy are implemented in lower-level languages (C/C++/Fortran) for speed.
2. Contiguous Memory Layout: Arrays are stored contiguously in memory, leading to faster access.
3. Broadcasting: NumPy can stretch smaller dimensions to match larger ones without making unnecessary copies of data.

For large-scale problems, further performance improvements can be done through parallelization (e.g., using libraries like Dask, or GPU-accelerated frameworks like CuPy).

4.1 Linear Independence, Rank, and Span#

• Linear Independence: A set of vectors {v�? v�? …, vₖ} is linearly independent if the only solution to
  α₁v�?+ α₂v�?+ … + αₖv�?= 0
  is α�?= α�?= … = α�?= 0.

• Rank: The rank of a matrix A is the maximum number of linearly independent columns (or rows) in A. Informally, it indicates the dimensionality of the space spanned by the columns/rows of A.

• Span: The span of a set of vectors is all possible linear combinations of these vectors. In simple terms, it’s the subspace of ℝⁿ those vectors can “reach.�?

4.2 Eigenvalues and Eigenvectors#

For a square matrix A, an eigenvector x and its corresponding eigenvalue λ satisfy:
A x = λ x

This equation indicates that when A acts on x, it only scales x by λ without changing its direction. Eigenvalues and eigenvectors reveal essential properties about transformations:

• Eigenvalues can indicate scaling factors in certain directions.
• Eigenvectors correspond to principal directions of transformation.

In Python, you can use NumPy or SciPy to compute them:

1
import numpy as np
2

3
A = np.array([[2, 1],
4
              [1, 2]])
5

6
values, vectors = np.linalg.eig(A)
7
print("Eigenvalues:", values)
8
print("Eigenvectors:\n", vectors)

4.3 Decompositions (LU, QR, Cholesky, SVD)#

Matrix decompositions simplify complex matrix operations:

1. LU Decomposition: Factor a matrix A into the product of a lower triangular matrix L and an upper triangular matrix U. Useful for solving linear systems.

2. QR Decomposition: Decompose A into Q (orthonormal columns) and R (upper triangular). Often used in least squares solutions, orthonormal transformations.

3. Cholesky Decomposition: Specialized decomposition for positive definite matrices. Decompose A = L Lᵀ where L is lower triangular.

4. Singular Value Decomposition (SVD): A = U Σ Vᵀ
   Where U and V are orthonormal, and Σ is a diagonal matrix of singular values. SVD is a powerful tool in data compression, noise reduction, and dimensionality reduction.

Example with NumPy for SVD:

1
A = np.random.randn(4, 3)  # random 4x3 matrix
2
U, S, Vt = np.linalg.svd(A, full_matrices=False)
3
print("U:", U)
4
print("Singular values:", S)
5
print("Vᵀ:", Vt)
6

7
# To reconstruct A:
8
A_reconstructed = U @ np.diag(S) @ Vt

4.4 Orthogonality and Orthonormality#

• Orthogonal Vectors: Two vectors u and v are orthogonal if u · v = 0.
• Orthogonal (or Orthonormal) Matrix: A square matrix Q where QᵀQ = I. For orthonormal Q, each column is a unit vector and they are mutually orthogonal.

Orthonormal matrices simplify many linear algebra problems because inverse operations become straightforward (Q⁻�?= Qᵀ).

5.1 Systems of Linear Equations#

Linear algebra is very effective for solving large systems of linear equations:

Ax = b

where A is an m×n matrix, x is an n×1 column vector of unknowns, and b is an m×1 column vector. In Python:

1
A = np.array([[2, 1],
2
              [5, 7]])
3
b = np.array([11, 13])
4

5
# Solve Ax = b
6
x = np.linalg.solve(A, b)
7
print("Solution x =", x)

5.2 Dimensionality Reduction (PCA)#

Principal Component Analysis (PCA) is a technique to reduce the dimensionality of data while retaining most of its variance:

1. Mean-Center the Data: Subtract mean from each dimension.
2. Compute Covariance Matrix: Usually a d×d matrix for data of dimension d.
3. Eigen Decomposition: The principal axes of variance (principal components) are revealed by the eigenvectors of the covariance matrix.
4. Project Onto Principal Components: Use only the top k components for dimensionality reduction.

Example in Python (simple conceptual snippet):

1
import numpy as np
2

3
# Suppose X is shape (n_samples, n_features)
4
X = np.random.randn(100, 5)
5

6
# Step 1: Mean center
7
X_mean = np.mean(X, axis=0)
8
X_centered = X - X_mean
9

10
# Step 2: Compute covariance matrix
11
cov_mat = np.cov(X_centered, rowvar=False)
12

13
# Step 3: Eigen decomposition
14
eigenvalues, eigenvectors = np.linalg.eig(cov_mat)
15

16
# Step 4: Sort eigenvalues/eigenvectors
17
idx = np.argsort(eigenvalues)[::-1]
18
eigenvalues = eigenvalues[idx]
19
eigenvectors = eigenvectors[:, idx]
20

21
# Choose top k components
22
k = 2
23
principal_components = eigenvectors[:, :k]
24

25
# Project data
26
X_pca = X_centered @ principal_components
27
print("Reduced dimensionality shape:", X_pca.shape)

5.3 Least Squares Regression#

Linear regression attempts to fit a line (or hyperplane in higher dimensions) to minimize the sum of squared differences between predicted and actual values. The classical solution using matrix calculus is:

( XᵀX )⁻�?Xᵀ y

where X is the design matrix of input features (with an initial column of 1s for the intercept) and y is the response vector.

A quick example with NumPy:

1
# Suppose you have data X (shape: n_samples x n_features) and y (shape: n_samples)
2
n_samples = 50
3
X = np.random.rand(n_samples, 1)  # single feature
4
y = 3 * X.flatten() + 2 + 0.5 * np.random.randn(n_samples)  # perfect line + noise
5

6
# Add a column of ones to X
7
X_ones = np.hstack([np.ones((n_samples, 1)), X])
8

9
# Compute (X^T X)^(-1) X^T y
10
theta = np.linalg.inv(X_ones.T @ X_ones) @ X_ones.T @ y
11
print("Estimated parameters:", theta)

Here, theta[0] should be close to 2 (intercept) and theta[1] close to 3 (slope).

5.4 Advanced Machine Learning Perspectives#

• Neural Networks: Matrix multiplication forms the backbone of forward and backward passes through neural networks. Weight matrices multiply input (or hidden layer) vectors, and updates rely heavily on linear algebra operations for gradient computations.

• Support Vector Machines (SVM): The solution to the SVM optimization problem involves solving a system of linear equations once the optimization is formulated in dual form.

• Recommendation Systems: Matrix factorization for collaborative filtering (e.g., SVD or other decompositions) underpins many recommendation algorithms, using user-item rating matrices.