ml(Math-1)

Linear Algebra

Vector Algebra

Colab

Matrix Algebra

Types of Matrix

Diagonal Matrix

• A square matrix whose off-diagonal elements are all zero.

$$\left(\begin{array}{ccc}2& 0& 0\\ 0& 3& 0\\ 0& 0& 1\end{array}\right)$$

Triangular Matrix

• A square matrix whose elements above or below the main diagonal are all zero.

1. Upper Triangular Matrix

$$\left(\begin{array}{ccc}5& 8& 7\\ 0& 3& 8\\ 0& 0& 1\end{array}\right)$$

2. Lower Triangular Matrix

$$\left(\begin{array}{ccc}4& 0& 0\\ 3& 2& 0\\ 5& 0& 3\end{array}\right)$$

Identity Matrix

• A diagonal matrix whose all diagonal entries are all 1.

$$\left(\begin{array}{ccc}1& 0& 0\\ 0& 1& 0\\ 0& 0& 1\end{array}\right)$$

Matrix Algebra

• Two matrices are equal if their dimensions are equal and all corresponding elements are equal.

• The addition or subtraction of two matrices, A and B of the same size yields a matrix C of the same size.

$$c_{ij}=a_{ij}+b_{ij}foralliandj.$$

1. Commutative Law: A + B = B + A
2. Associative Law: A + (B + C) = (A + B) + C = A + B + C

• Matrices of different sizes cannot be added or subtracted.

Scalar multiplication in matrices

$$\left(\begin{array}{ccc}{a}_{11}& a_{12}& a_{13}\\ a_{21}& a_{22}& a_{23}\\ a_{31}& a_{32}& a_{33}\end{array}\right)$$ $$\alpha \in \mathbb{R}$$ $$\alpha A=\left(\begin{array}{ccc}\alpha a_{11}& \alpha a_{12}& \alpha a_{13}\\ \alpha a_{21}& \alpha a_{22}& \alpha a_{23}\\ \alpha a_{31}& \alpha a_{32}& \alpha a_{33}\end{array}\right)$$

Matrix Multiplication

• The product of two matrices is another matrix.
• The necessary condition for multiplication of two matrices A and B is that the number of columns of A must equal the number of rows of B.

$$A_{m\times n}\times B_{n\times p}=C_{m\times p}$$ $$\left(\begin{array}{ccc}{a}_{11}& a_{12}& a_{13}\\ a_{21}& a_{22}& a_{23}\end{array}\right)\times \left(\begin{array}{cc}{b}_{11}& b_{12}\\ b_{21}& b_{22}\\ b_{31}& b_{32}\end{array}\right)=\left(\begin{array}{cc}{a}_{11}{b}_{11}+a_{12}{b}_{21}+a_{13}{b}_{31}& a_{11}{b}_{12}+a_{12}{b}_{22}+a_{13}{b}_{32}\\ a_{21}{b}_{11}+a_{22}{b}_{21}+a_{23}{b}_{31}& a_{21}{b}_{12}+a_{22}{b}_{22}+a_{23}{b}_{32}\end{array}\right)$$

Properties of Matrix Multiplication

1. Associative Law: $A(BC)=(AB)C$
2. Distributive Law: $A(B+C)=AB+AC$
3. Identity Matrix: $AI=IA=A$

Note: Matrix multiplication is not commutative in general.i.e. AB != BA Note: AB=0 does not imply A=0 or B=0. Note: AB=AC does not imply B=C.

Transpose of a Matrix

• The transpose of a matrix A is denoted by A^T.
• The transpose of a matrix is obtained by interchanging the rows and columns of the matrix.
• If A is an m x n matrix, then A^T is an n x m matrix.

$$A=\left(\begin{array}{ccc}{a}_{11}& a_{12}& a_{13}\\ a_{21}& a_{22}& a_{23}\end{array}\right)$$ $$A^T=\left(\begin{array}{cc}{a}_{11}& a_{21}\\ a_{12}& a_{22}\\ a_{13}& a_{23}\end{array}\right)$$

Properties of Transpose

1. $(A^T{)}^T=A$
2. $(A+B{)}^T=A^T+B^T$
3. $(AB{)}^T=B^T{A}^T$
4. $(kA{)}^T=k{A}^T$

Determinant of a Matrix

• Every square matrix has a determinant.
• The determinant of a matrix is a number.
• The determinant of a matrix is denoted by |A| or det(A).
• The determinant of a 2x2 matrix is given by: $\left|\begin{array}{cc}a& b\\ c& d\end{array}\right|=ad-bc$
• The determinant of a 3x3 matrix is given by: $\left|\begin{array}{ccc}a& b& c\\ d& e& f\\ g& h& i\end{array}\right|=a(ei-fh)-b(di-fg)+c(dh-eg)$

Inverse of a Matrix

• The inverse of a matrix A is denoted by $A^{-1}$ .

• If A is a square matrix, then A has an inverse if and only if the determinant of A is not equal to zero.

Properties of Inverse

1. $(A^{-1}{)}^{-1}=A$
2. $(AB{)}^{-1}=B^{-1}{A}^{-1}$
3. $(A^T{)}^{-1}=(A^{-1}{)}^T$
4. $(kA{)}^{-1}=\frac{1}{k}{A}^{-1}$
5. $A{A}^{-1}=A^{-1}A=I$

• A square matrix that has an inverse is called a non-singular matrix.
• A square matrix that does not have an inverse is called a singular matrix.
• Square matrices that are singular are also called degenerate matrices.
• When the determinant of a matrix is zero, the matrix is singular and does not have an inverse.

Colab

Vector Space

• A vector space is a set of vectors that are closed under two operations: vector addition and scalar multiplication.

Properties of Vector Space

1. Closure under Addition: If u and v are vectors in V, then u + v is also in V.
2. Associative Law: u + (v + w) = (u + v) + w
3. Commutative Law: u + v = v + u
4. (V,+) forms an abelian group.
5. The operation (.) is defined between scalars and vectors such that if u is a vector in V and a is a scalar, then au is also in V.
6. Distributive Law: a(u + v) = au + av
7. Distributive Law: (ab)u = a(bu)
8. Unitary law: 1u = u

Note: (V,+,.) is a vector space if it satisfies the above properties.

Geometric Interpretation of Vector Space

• For n=1,2,or 3 the vector space is called 1D, 2D, or 3D space.

• If V1, V2 and V3 are linearly independent vectors in a vector space.

  • V1 and V2 are not collinear, and V3 does not lie in the plane of V1 and V2.

Vector Subspace

• If ${\mathbb{R}}^n$ is a vector space, then any subset of ${\mathbb{R}}^n$ that satisfies the properties of a vector space is called a subspace of ${\mathbb{R}}^n$.

Theorem-1

• Let W be a subset of ${\mathbb{R}}^n$. Then W is a subspace of ${\mathbb{R}}^n$ if and only if W satisfies the following three properties:
  1. W is non-empty.
  2. For any $a,b\in \mathbb{R}$ and any $\vec{u},\vec{v}\in W$, $a\vec{u}+b\vec{v}\in W$.

Example

• If we have a subspace of ${\mathbb{R}}^2$. Then,

  1. The origin is a subspace.
  2. A line passing through the origin is a subspace.

• If we have a subspace of ${\mathbb{R}}^3$. Then,

  1. The origin is a subspace.
  2. A line passing through the origin is a subspace.
  3. A plane passing through the origin is a subspace.

Theorem-2

• The intersection of any non-empty collection of subspaces of ${\mathbb{R}}^n$ is a subspace of ${\mathbb{R}}^n$.

Theorem-3

• The union of two subspaces of ${\mathbb{R}}^n$ is a subspace of ${\mathbb{R}}^n$ if one of them is contained in the other.

Example

$S_1=(x_1,x_2,x_3)\in {\mathbb{R}}^2|x_1+x_2+x_3=0$ $S_2=(x_1,x_2,x_3)\in {\mathbb{R}}^2|x_1=x_2=x_3$ $S_1\cup S_2$ = {(0,0,0)}$

Linear Span

• Let $V(F)$ be a vector space.
• Let $S=v_1,v_2,...,v_n$ be a non-empty subset of V.Then the set $L(S)=c_1{v}_1+c_2{v}_2+...+c_n{v}_n|c_i\in \mathbb{R}|1\le i\le n$ is called the linear span of S.

Example

• Let $V={\mathbb{R}}^2$ and $S=(1,0),(0,1)$ and $S\subset V$. $L(S)=c_1(1,0)+c_2(0,1)|c_1,c_2\in \mathbb{R}$ $L(S)=(c_1,c_2)|c_1,c_2\in \mathbb{R}={\mathbb{R}}^2$

  $S=(1,0,0),(0,1,0),(0,0,1)$ $L(S)=(c_1,c_2,c_3)|c_1,c_2,c_3\in \mathbb{R}={\mathbb{R}}^3$

Theorem-1

• Let $V(F)$ be a vector space and $S=v_1,v_2,...,v_n$ be a non-empty subset of V.Then $L(S)$ is a subspace of V.

Theorem-2

• Let S be a non-empty subset of a vector space V. Then $L(S)$ is the smallest subspace of $V(F)$ that contains S.

Some important subspaces

Let,

$$A_{m\times n}=\left(\begin{array}{cccc}{a}_{11}& a_{12}& \dots & a_{1n}\\ a_{21}& a_{22}& \dots & a_{2n}\\ ⋮& ⋮& \ddots & ⋮\\ a_{m1}& a_{m2}& \dots & a_{mn}\end{array}\right)$$

where $a_{ij}\in \mathbb{R}$.Then

1. The row space of A is given by $L((a_{11},a_{12},\dots ,a_{1n}),(a_{21},a_{22},\dots ,a_{2n}),\dots ,(a_{m1},a_{m2},\dots ,a_{mn}))$ is a subspace of ${\mathbb{R}}^n$.
2. The column space of A is given by $L((a_{11},a_{21},\dots ,a_{m1}),(a_{12},a_{22},\dots ,a_{m2}),\dots ,(a_{1n},a_{2n},\dots ,a_{mn}))$ is a subspace of ${\mathbb{R}}^m(\mathbb{R})$.
3. The set $N(A)=x\in {\mathbb{R}}^n|Ax=0$ is said to be nullspace of A.
4. The set $R(A)=b\in {\mathbb{R}}^m|Ax=b$ is said to be range of A. In general, column space $A=R(A)$.

Relevance with ML

In many of the ML algorithms:

• We need to represent feature vectors as a linear combination of set of linearly independent vectors or orthogonal vectors.
• We need to represent all the vectors of the feature space of our dataset as the linear combination of these vectors.

Basis

Definition: Let $V(F)$ be a vector space. A basis for V is a set of linearly independent vectors that span V.

• The space V is finite-dimensional if it has a finite basis.
• A vector in basis is called basis vector.

Example

1. Let $V={\mathbb{R}}^2$ and $B=\{(1,0),(0,1)\}$.
   • B is a basis for ${\mathbb{R}}^2$.
2. Let $V={\mathbb{R}}^3$ and $B=\{(1,0,0),(0,1,0),(0,0,1)\}$.
   • B is a basis for ${\mathbb{R}}^3$.
3. $V=M_{2\times 2}|\text{set of all 2}\times 2\text{matrices}$
   • $V(\mathbb{R})$ is a vector space. Now, $$\left\{\left(\begin{array}{cc}1& 0\\ 0& 0\end{array}\right),\left(\begin{array}{cc}0& 1\\ 0& 0\end{array}\right),\left(\begin{array}{cc}0& 0\\ 1& 0\end{array}\right),\left(\begin{array}{cc}0& 0\\ 0& 1\end{array}\right)\right\}$$
4. $V=M_{2\times 2}|\text{set of all 2}\times 2\text{real matrices}$ $$\left\{\left(\begin{array}{cc}1& 0\\ 0& 0\end{array}\right),\left(\begin{array}{cc}0& 1\\ 1& 0\end{array}\right),\left(\begin{array}{cc}0& 0\\ 0& 1\end{array}\right)\right\}$$

Theorem 1

• Let $v_1,v_2,...,v_n$ be a basis for a vector space $V(F)$. If S be a subset of vectors having more than $n$ vectors, then the set S is linearly dependent.

Theorem 2

• Let $v_1,v_2,...,v_n$ be a linearly independent subset of a n-dimensional vector space $V(F)$, where $k<n$. Then S can be extended to a basis of $V(F)$.

Theorem 3

• Let $V(F)$ be a finite-dimensional vector space. Then any two bases of V have the same number of vectors.

Example

${\mathbb{R}}^3(\mathbb{R})$ $B_1=(1,0,0),(0,1,0),(0,0,1)$ $B_2=(1,1,0),(0,1,1),(1,0,1)$ $B_3=(1,0,0),(1,1,0),(1,1,1)$

Theorem 4

• Let $V(F)$ be a finite-dimensional vector space and let $S_1$ and $S_2$ be two bases of V. Then $dim(S_1)+dim(S_2)=dim(S_1+S_2)+dim(S_1\cap S_2)$

Example

1. Find a basis and dimension of the subspaces S of ${\mathbb{R}}^3$ given as

• $S=(x_1,x_2,x_3)\in {\mathbb{R}}^3|x_1+x_2-x_3=0$ $=(x_1,x_2,x_1+x_2)$ $=x_1(1,0,1)+x_2(0,1,1)$

  Hence, the basis is {(1,0,1),(0,1,1)} and the dimension is 2. $dim(S)=2$

2. $W=(x,y,z)|x_1=x_2=x_3$ Hence, the basis is {(1,1,1)} and the dimension is 1. $dim(W)=1$

Basis of $S\cap W$ $S\cap W=(x_1,x_2,x_3)\in \mathbb{R}|x_1+x_2-x_3=0\&x_1=x_2=x_3$ $=(x_1,x_2,x_1+x_2)|x_1=x_2=x_3$ $=(x_1,x_2,2x_1)|x_1=x_2=x_3$ $B_{S\cap W}=(0,0,0)$