ml(Math-4)

PCA (Principle, Component, Analysis)

• PCA is a method used to reduce the number of variables in a dataset while preserving as much information as possible.

• PCA is an unsupervised learning algorithm.

• PCA is used to reduce the dimensionality of a dataset.

• PCA is used to identify patterns in data and detect the correlation between variables.

• PCA is used to identify the most important features in a dataset.

• PCA is used to reduce the number of features in a dataset.

Definition we need to know

• Mean: The average value of a dataset.

• Variance: The spread of a dataset.

• Covariance: The relationship between two variables.

• Standard deviation: The square root of the variance.

• Correlation: The relationship between two variables.

• Eigenvector: A vector that does not change its direction when a linear transformation is applied.

• Eigenvalue: A scalar that represents the magnitude of an eigenvector.

• Projection: The transformation of a dataset onto a lower-dimensional subspace.

• Orthogonal: Two vectors are orthogonal if their dot product is zero.

• Linear transformation: A transformation that preserves the linearity of a dataset.

• Covariance matrix: A matrix that contains the covariance between each pair of variables in a dataset.

• The principle components are the eigenvectors of the covariance matrix of the data.

  • First principle component is the eigenvector with the highest eigenvalue of the covariance matrix.
  • If we want to transform a n-dimensional dataset into a k-dimensional dataset, we can select the top k eigenvectors of the covariance matrix.
  • The eigenvectors of the covariance matrix are the directions of maximum variance in the data.

Steps to perform PCA

1. Standardize the data.
2. Calculate the covariance matrix.
3. Calculate the eigenvectors and eigenvalues of the covariance matrix.
4. Sort the eigenvectors by decreasing eigenvalues and choose the top k eigenvectors.
5. Construct the projection matrix W from the selected k eigenvectors.
6. Transform the original dataset X via W to obtain a k-dimensional feature subspace Y.
7. The new dataset Y can be used to train a machine learning model.
8. The PCA algorithm can be used to reduce the number of features in a dataset.

Tools to implement PCA

• sklearn.decompositon can be used to directly apply PCA algorithm.

# Direct Implementation
from sklearn.decomposition import PCA
pca=PCA(n_components=1)
pca.fit(X_meaned)
Xr = pca.transform(X_meaned)
print(Xr)

Example

LDA (Linear Discriminant Analysis)

• LDA is a supervised learning algorithm.

• LDA is used to reduce the dimensionality of a dataset.

• Main idea: Find the projection to a line such that samples from different classes are well separated.

Steps to perform LDA

1. Standardize the data.
2. Calculate the mean vectors for each class.
3. Calculate the within-class scatter matrix.
4. Calculate the between-class scatter matrix.
5. Calculate the eigenvectors and eigenvalues of the matrix $S_w^{-1}{S}_b$.
6. Sort the eigenvectors by decreasing eigenvalues and choose the top k eigenvectors.
7. Construct the projection matrix W from the selected k eigenvectors.
8. Transform the original dataset X via W to obtain a k-dimensional feature subspace Y.
9. The new dataset Y can be used to train a machine learning model.
10. The LDA algorithm can be used to reduce the number of features in a dataset.

Difference between PCA and LDA