Change of Basis and Its Applications

The concept of changing the basis in a vector space is fundamental in linear algebra and has profound implications in data science, particularly in dimensionality reduction, data transformation, and simplifying complex problems. This article explores the concept of change of basis, the mathematical process behind it, and its practical applications.

---

1. What is a Basis?

1.1 Definition of a Basis

A basis of a vector space is a set of linearly independent vectors that span the entire space. Any vector in the space can be uniquely expressed as a linear combination of the basis vectors.

Example:

In $\mathbb{R}^2$, the standard basis is:

$$\mathbf{e}_1 = \begin{pmatrix} 1 \\ 0 \end{pmatrix}, \quad \mathbf{e}_2 = \begin{pmatrix} 0 \\ 1 \end{pmatrix}$$

Any vector $\mathbf{v}$ in $\mathbb{R}^2$ can be written as:

$$\mathbf{v} = x_1\mathbf{e}_1 + x_2\mathbf{e}_2 = \begin{pmatrix} x_1 \\ x_2 \end{pmatrix}$$

1.2 Why Change the Basis?

Changing the basis allows us to express vectors in a new coordinate system that might simplify calculations, reveal hidden structures, or align with the problem's geometry. For example, in data science, changing the basis can align data with principal components (as in PCA), simplifying analysis and reducing dimensionality.

---

2. Mathematical Process of Changing the Basis

2.1 Transition Matrix

To change the basis from one set of basis vectors $\{\mathbf{b}_1, \mathbf{b}_2, \dots, \mathbf{b}_n\}$ to another set $\{\mathbf{c}_1, \mathbf{c}_2, \dots, \mathbf{c}_n\}$, we use a transition matrix $\mathbf{P}$.

If $\mathbf{P}$ is the matrix whose columns are the vectors of the new basis expressed in terms of the old basis, then for any vector $\mathbf{v}$:

$$\mathbf{v}_{\text{new}} = \mathbf{P}^{-1} \mathbf{v}_{\text{old}}$$

2.2 Example of Changing Basis

Consider changing the basis in $\mathbb{R}^2$ from the standard basis $\{\mathbf{e}_1, \mathbf{e}_2\}$ to a new basis $\{\mathbf{b}_1, \mathbf{b}_2\}$ where:

$$\mathbf{b}_1 = \begin{pmatrix} 2 \\ 1 \end{pmatrix}, \quad \mathbf{b}_2 = \begin{pmatrix} 1 \\ 3 \end{pmatrix}$$

The transition matrix $\mathbf{P}$ is:

$$\mathbf{P} = \begin{pmatrix} 2 & 1 \\ 1 & 3 \end{pmatrix}$$

To express a vector $\mathbf{v} = \begin{pmatrix} 5 \\ 7 \end{pmatrix}$ in the new basis, we calculate:

$$\mathbf{v}_{\text{new}} = \mathbf{P}^{-1} \mathbf{v}$$

where $\mathbf{P}^{-1}$ is the inverse of the transition matrix $\mathbf{P}$.

Since $\det(\mathbf{P})=5$, the inverse exists and the coordinates are:

$$\mathbf{v}_{\text{new}} = \frac{1}{5}\begin{pmatrix}3 & -1 \\ -1 & 2\end{pmatrix} \begin{pmatrix}5 \\ 7\end{pmatrix} = \begin{pmatrix}8/5 \\ 9/5\end{pmatrix}$$

Indeed, $\frac{8}{5}\mathbf{b}_1+\frac{9}{5}\mathbf{b}_2=(5,7)^T$, so the coordinates describe the same vector in the new basis.

---

3. Applications in Data Science

3.1 Principal Component Analysis (PCA)

In PCA, centered data are transformed to orthonormal principal directions ordered by projected variance. Keeping all principal directions is a change of orthonormal basis; keeping only the first few is additionally a projection onto a lower-dimensional subspace and therefore is not an invertible change of basis for the original space.

Example:

After performing PCA on a dataset, the full set of principal directions forms an orthonormal basis (up to non-uniqueness when eigenvalues tie). The centered data can be projected onto a selected leading subset, reducing the number of dimensions while retaining the corresponding fraction of sample variance.

3.2 Data Transformation

In many machine learning algorithms, data is transformed to a new basis to improve the algorithm's performance or interpretability. For example, in linear discriminant analysis (LDA), data is projected onto a new basis that maximizes class separability.

3.3 Simplifying Linear Systems

Changing the basis can simplify solving linear systems, particularly when the new basis aligns with the system's inherent structure. For instance, in diagonalization, changing the basis to the eigenvectors of a matrix simplifies matrix operations, making them easier to compute and understand.

---

4. Practical Considerations

4.1 Choosing a Good Basis

Choosing an appropriate basis is crucial. The new basis should align with the problem's structure or simplify the calculations. For example, in PCA, the principal components are chosen because they capture the most variance in the data.

4.2 Computational Efficiency

Changing the basis, particularly in high-dimensional spaces, can be computationally expensive. Efficient algorithms and numerical methods are necessary to compute the transition matrices and transform the data effectively.

4.3 Interpreting Results

When changing the basis, it's important to interpret the results correctly in the new coordinate system. For instance, after performing PCA, the transformed data represents projections on the principal components, which may require rethinking the original data's interpretation.

---

5. Connection to Other Linear Algebra Concepts

5.1 Eigenvalues and Eigenvectors

When a matrix is diagonalizable, a basis of eigenvectors turns its representation into a diagonal matrix. Real symmetric matrices are especially convenient because they always admit an orthonormal eigenbasis. A defective matrix does not have enough linearly independent eigenvectors to form such a basis.

5.2 Orthogonality and Orthonormal Bases

Changing to an orthonormal basis, where the basis vectors are orthogonal and of unit length, can greatly simplify many computations. This is particularly important in optimization and data science applications.

---

Conclusion

The concept of changing the basis in vector spaces is a powerful tool in linear algebra, with numerous applications in data science. Whether you are performing dimensionality reduction, transforming data for machine learning, or simplifying linear systems, understanding how and when to change the basis is essential for effective problem-solving and data analysis.