"Clar" is a root word of Latin origin, meaning "clear."
CLAR is inspired by the excellent Linear Algebra Review and Reference by Kolter and Do, but is intended as a more compact alternative.
- Fundamental Equation: \(Ax = b\)
- \(A \in \mathbb{R}^{m \times n}\) is a matrix with \(m\) rows and \(n\) columns
- \(m\) is the number of equations (rows), while \(n\) is the number of unknowns (columns)
- \(x \in \mathbb{R}^{n}\) is a (column) vector with \(n\) entries/unknowns (or an \(n \times 1\) matrix)
- \(b \in \mathbb{R}^{m}\) is a (column) vector with \(m\) entries/outputs (or an \(m \times 1\) matrix)
- \(A \in \mathbb{R}^{m \times n}\) is a matrix with \(m\) rows and \(n\) columns
- Span: \(\text{span}(\{x_1, \dots, x_n\}) = \Big(v : v = \sum_{i = 1}^n \alpha_ix_i, \alpha_i \in \mathbb{R}\Big)\)
- Interpretation: set of all vectors that can be expressed as a linear combination of vectors \(x_1, \dots, x_n\)
- Matrix Rank: \(\text{rank}(A)\) is number of linearly independent rows or columns
- Note that Row Rank = Column Rank, and hence we refer to both simply as rank
- \(A\) is full rank iff \(\text{rank}(A) = m = n\)
- Matrix Spaces:
- Range (or Columnspace): \(\mathcal{R}(A) = \{v \in \mathbb{R}^m : v = Ax, x \in \mathbb{R}^n\} = \text{span}(\text{Col}(A))\)
- Nullspace: \(\mathcal{N}(A) = \{x \in \mathbb{R} : Ax = 0\}\)
- Transpose: matrix \(A^T\) formed by swapping the rows and columns of \(A\)
- \((A^T)^T = A\)
- \((AB)^T = B^TA^T\)
- \((A + B)^T = A^T + B^T\)
- \(\forall a \in \mathbb{R}, a^T = a\)
- Vector Dot (or Inner) Product: If \(x, y \in \mathbb{R}^n\), then \(x^Ty = y^Tx = \sum_{i=1}^n x_iy_i\)
- \(x\) and \(y\) are orthogonal iff \(x^Ty = y^Tx = 0\) (geometrically, these two vectors will be perpendicular to each other)
- Matrix Multiplication (MatMul): if \(A \in \mathbb{R}^{m \times n}\) and \(B \in \mathbb{R}^{n \times p}\), then \(C = AB \in \mathbb{R}^{m \times p}\), with \(C_{ij} = \sum_{k=1}^n A_{ik}B_{kj}\)
- MatMul is Associative: \((AB)C = A(BC)\)
- MatMul is Distributive: \(A(B + C) = AB + AC\)
- MatMul is NOT Commutative: \(AB \neq BA\)
- In fact, MatMul does not exist if \(m \neq p\)
- Determinant: \(\det : \mathbb{R}^{n \times n} \rightarrow \mathbb{R}\)
- Interpretation: the signed \(n\)-volume of the parallelotope formed by the matrix's column vectors
- For \(A \in \mathbb{R}^{2 \times 2} = \langle a, b; c, d \rangle\), we have \(\det A = ad - bc\)
- If \(\det A = 0\), matrix \(A\) is singular (non-invertible), otherwise it is non-singular (invertible)
- Trace: \(\mathrm{tr} A = \sum_{i=1}^n A_{ii}\), which is the sum of diagonal elements
- \(\mathrm{tr}\ A = \mathrm{tr}\ A^T\)
- For \(A, B \in \mathbb{R}^{n \times n}, \mathrm{tr}\ (A + B) = \mathrm{tr}\ A + \mathrm{tr}\ B\)
- Vector \(L_p\) Norm: \(\lvert\lvert x\rvert\rvert_p = \big(\sum_{i=1}^n \lvert x_i \rvert^p\big)^{1/p}\)
- \(\text{L}_1\) Norm: \(\lvert\lvert x\rvert\rvert_1 = \sum_{i=1}^n x_i\)
- \(\text{L}_2\) Norm: \(\lvert\lvert x\rvert\rvert_2 = \sqrt{\sum_{i=1}^n x_i^2}\)
- \(\text{L}_\infty\) Norm: \(\lvert\lvert x\rvert\rvert _\infty = \max_i \lvert x_i\rvert\)
- Matrix Frobenius Norm: \(\lvert\lvert A\rvert\rvert_{\text{F}} = \sqrt{\sum_{i=1}^m\sum_{j=1}^n A_{ij}^2} = \sqrt{\text{tr}(A^TA)}\)
- Identity Matrix: \(I \in \mathbb{R}^{n \times n}\) s.t. \(AI = A = IA\) with \(I_{ij} = 1\) iff \(i = j\) and \(0\) otherwise
- Inverse Matrix: \(A^{-1}A = I = AA^{-1}\)
- \(A\) is invertible (non-singular) and inverse \(A^{-1}\) exists if \(\det A \neq 0\), and it is non-invertible (singular) if \(\det A = 0\)
- \((A^{-1})^{-1} = A\)
- \((AB)^{-1} = B^{-1}A^{-1}\)
- \((A^{-1})^T = (A^T)^{-1} = A^{-T}\)
- Symmetric Matrix: square matrix \(A^{n \times n}\) is symmetric iff \(A = A^T\)
- Matrix \(A + A^T\) is symmetric
- Anti-symmetric Matrix: square matrix \(A^{n \times n}\) is symmetric iff \(A = -A^T\)
- Matrix \(A - A^T\) is anti-symmetric
- Matrix Quadratic Form: for a symmetric matrix \(A \in \mathbb{n \times n}\), we have \(Q(x) = x^TAx\)
- Positive Definite (PD) Matrix: \(\forall x \in \mathbb{R}^n \neq 0, Q(x) > 0 \implies A \succ 0 \text{ (or just } A > 0)\)
- Positive Semidefinite (PSD) Matrix: \(\forall x \in \mathbb{R}^n \neq 0, Q(x) \geq 0 \implies A \succeq 0 \text{ (or just } A \geq 0)\)
- Negative Definite (ND) Matrix: \(\forall x \in \mathbb{R}^n \neq 0, Q(x) < 0 \implies A \prec 0 \text{ (or just } A < 0)\)
- Negative Semidefinite (NSD) Matrix: \(\forall x \in \mathbb{R}^n \neq 0, Q(x) \leq 0 \implies A \preceq 0 \text{ (or just } A \leq 0)\)
- Orthogonal Matrix: a square matrix \(U \in \mathbb{R}^{n \times n}\) is orthogonal if all of its columns are orthogonal to each other and are normalized
- The columns of \(U\) are referred to as orthonormal
- It follows that \(U^TU = I = UU^T\)
- Eigenvalue and eigenvector: given \(A \in \mathbb{R}^{n \times n}\), a scalar \(\lambda \in \mathbb{C}\) is an eigenvalue with eigenvector \(x \in \mathbb{C}^n \setminus{\{0\}}\) if \(Ax = \lambda x\)
- Reformulation: \((\lambda, x)\) is an eigenvalue-eigenvector pair of A iff \((A - \lambda I)x = 0\)
- Interpretation: an eigenvector is a direction a matrix scales without changing direction, and the eigenvalue is how much it scales
- Trace equals to the sum of eigenvalues: \(\mathrm{tr} A = \sum_{i=1}^n \lambda_i\)
- Determinant equals to the product of eigenvalues: \(\det A = \prod_{i=1}^n \lambda_i\)
- Rank is equal to the number of non-zero eigenvalues: \(\text{rank}(A) = \left| \{ \lambda_i \mid \lambda_i \neq 0 \} \right|\)
- If \(A\) is non-singular, \(A^{-1}x_i = (1/\lambda_i)x_i\) (i.e., \(1/\lambda_i\) is the eigenvalue and \(x_i\) is the eigenvector)
- Eigenvalues of a diagonal matrix \(D = \text{diag}(d_1,\dots,d_n)\) are the diagonal entries \(d_1,\dots,d_n\) themselves
- In PCA, eigenvalues represent the variance explained by each principal component, while eigenvectors define their directions
- Diagonalizable Matrix: a square matrix \(A \in \mathbb{R}^{n \times n}\) is diagonalizable iff its eigenvector matrix is full rank
- Eigenvalue Decomposition (EVD): \(A = PDP^{-1}\) where \(P\) is the eigenvector matrix and \(D\) is a diagonal matrix of eigenvalues
- LU Decomposition: \(A = L \cdot U\)
- Efficiently solves systems of equations \(Ax = b_1, \dots, Ax = b_n\) by factoring \(A = LU\) once, then using substitution for each \(b_i\)
- To perform LU decomposition, solve two equations: \(Ly = b\) for \(y\) and then solve \(Ux = y\) for \(x\)
- \(L\) and \(U\) are lower and upper triangular matrices, respectively, with \(L\) having 1s on the diagonal
- Gram Matrix: \(G = A^TA\)