Matrix, Vector

Matrix

My math knowledge from freshman calculus and linear algebra has mostly faded. Refreshing it here alongside numpy notation.

Annotation

scalar calculation

In numpy, + and - work directly.

scalar product

Hadamard Product: element-wise product of identically shaped vectors. X · Y

1
X * Y

Norm

 Distance from the origin to the vector. L1 norm = sum of absolute values of changes L2 norm = Euclidean distance

Angle between vectors

 Using the law of cosines, we can compute the angle between two vectors.

1
def angle(x, y):
2
    v = np.inner(x, y) / (l2_norm(x) * l2_norm(y))
3
    theta = np.arccos(v)
4
    return theta

multiplication

XY

1
X @ Y

Through matrix multiplication, a matrix can be understood as an operator in vector space. Because matrix multiplication can send a vector to a different dimensional space. In other words, it can be used for pattern extraction and data compression.

inner product



inner in numpy

np.inner computes the inner product between vectors. To express the inner product of vectors in matrix form, a transpose is typically used. 

1
np.inner(X, Y)

Inverse matrix

1
np.linalg.inv(X)

Pseudo-inverse, Moore-Penrose matrix

• Unlike the regular inverse, the number of rows and columns doesn’t need to match.
• Still performs a role similar to the inverse.  n = rows, m = columns

1
np.linalg.pinv(X)

Solving systems of equations


Linear regression


Considering the distribution of data, doing linear regression as a system of equations is not possible. Therefore, finding a solution that minimizes the L2 norm of y is the general approach.

1
# using sklearn for linear regression
2
from sklearn.linear_model import LinearRegression
3
model = LinearRegression()
4
model.fit(X, y)
5
y_test = model.predict(x_test)
6

7
# Moore-Penrose inverse matrix
8
X_ = np.array([np.append(x, [1]) for x in X]) # add y-intercept
9
beta = np.linalg.pinv(X_) @ y
10
y_test = np.append(x_test) @ beta

sklearn automatically estimates the y-intercept when performing linear regression. When doing linear regression via the Moore-Penrose inverse, you need to manually add the y-intercept to construct X.