1 SVD

Weight = torch.randn(2,2).to("cpu")
#Weight := tensor([[ 0.3439,  0.9503], [-0.2960,  0.1503]])
Output = points @ Weight.t()
print(Output.shape)
#SVD
U, S, V = torch.svd(Weight)
#>U := tensor([[ 0.9990,  0.0449], [ 0.0449, -0.9990]])
#>S := tensor([1.0115, 0.3292])
#>V := tensor([[ 0.3265,  0.9452], [ 0.9452, -0.3265]])
plt.scatter(Output[:,0],Output[:,1],c=colors)
mbases = mbases @ Weight.t()
#> mbases := tensor([[ 0.0000,  0.0000],[ 0.0000,  0.0000],[ 0.3439, -0.2960],[ 0.9503,  0.1503]])

assert Weight.allclose(U@torch.diag(S)@V.t())
## Weight.allclose() is effectively testing equality because pure equality w/ floating points is f***ing dangerous

plot_bases(mbases)
plt.show()

Notice how the yellow dots rotated to the bottom

\[Weight = \begin{bmatrix} 0.3439 \\ 0.9503 \end{bmatrix} \begin{bmatrix} -0.2960 \\ 0.1503 \end{bmatrix}\] \[U = Rotation = \begin{bmatrix} 0.9990 & 0.0449 \\ 0.0449 & -0.9990 \end{bmatrix}\] \[S = Symmetry = \begin{bmatrix} 1.0115 & 0 \\ 0 & 0.3292 \end{bmatrix}\] \[V = Reflection = \begin{bmatrix} 0.3265 & 0.9452 \\ 0.9452 & -0.3265 \end{bmatrix}\]

\[SVD(Weight) = Rotation \times Symmetric\ Matrix \times Reflection^{T}\] \[W = U \begin{bmatrix} s_1 & 0 \\ 0 & s_2 \end{bmatrix} V^T\]

We can decompose the Weight matrix into 3 easily understood operations

• U = Rotation
  • Each column represents a eigenbasis or a fundamental component of your overall dataset.
• S = Symmetry
  • Diagonal matrix
  • each column has a value that represents how important the associated column in U are in forming the data point.
    • each the first value in the diagonal is largest so it tells us the 1st eigenbasis is most important aka the first fundamental component is the most important in creating your datapoint.
• Vt = Reflection
  • Each column of the transposed matrix Tells us what mixture of each eigenbasis is needed to form the associated input.
    • eg. first column tell us we need a certain mixture of each column of eigenbasis to form the first data point.

Pytorch makes the above linear transformation easy

model = nn.Sequential(
        nn.Linear(2, 2, bias=False)
)
model.to("cpu")
with torch.no_grad():
    Output = model(dataset)

2 Symmetric matrices

• Symmetrices metrices always have orthogonal eigenvectors