Vectors and Vector Operations
What vectors are, how to add and scale them, dot and cross products, and why they're the natural language for space and direction.
What a Vector Is
A vector is a quantity with both magnitude and direction. A scalar has only magnitude.
- Temperature is a scalar: 25°C. No direction.
- Velocity is a vector: 60 km/h north. Both magnitude and direction.
Geometrically, a vector is an arrow. Algebraically, it’s an ordered list of numbers — its components in each dimension.
2D vector: v = (3, 4) or v = [3, 4]ᵀ
3D vector: v = (1, −2, 5)
The column notation (ᵀ denotes transpose) is standard in linear algebra. The numbers are the coordinates of the tip, assuming the tail is at the origin.
Magnitude
The magnitude (length) of a vector is the distance from tail to tip:
|v| = √(v₁² + v₂² + ... + vₙ²)
For v = (3, 4): |v| = √(9 + 16) = √25 = 5.
This is Pythagoras, generalised to n dimensions.
Unit vector: a vector with magnitude 1. To normalise v (make it a unit vector):
v̂ = v / |v|
Unit vectors indicate direction only. The standard basis unit vectors:
î = (1, 0, 0) (x direction)
ĵ = (0, 1, 0) (y direction)
k̂ = (0, 0, 1) (z direction)
Any vector can be written as a combination: v = v₁î + v₂ĵ + v₃k̂.
Vector Addition and Scalar Multiplication
Addition: add component by component.
u + v = (u₁ + v₁, u₂ + v₂, ...)
(1, 3) + (4, −1) = (5, 2)
Geometrically: place vectors tip to tail; the sum goes from the first tail to the last tip (parallelogram rule).
Scalar multiplication: scale every component.
c · v = (cv₁, cv₂, ...)
3 · (2, −1) = (6, −3)
Positive c: stretches (or shrinks if 0 < c < 1). Negative c: reverses direction. c = −1: flips.
Subtraction: u − v = u + (−v).
(5, 2) − (3, 4) = (2, −2)
Geometrically: the vector from the tip of v to the tip of u.
Properties
u + v = v + u (commutativity)
(u + v) + w = u + (v + w) (associativity)
c(u + v) = cu + cv (distributivity)
The Dot Product
The dot product of two vectors gives a scalar:
u · v = u₁v₁ + u₂v₂ + ... + uₙvₙ
(2, 3) · (4, −1) = 8 − 3 = 5
Geometric interpretation:
u · v = |u| |v| cos θ
where θ is the angle between the vectors.
This means:
- u · v = 0: vectors are perpendicular (cos 90° = 0). Orthogonal.
- u · v > 0: angle less than 90°, vectors point roughly the same way
- u · v < 0: angle greater than 90°, vectors point roughly opposite ways
Key uses:
- Finding the angle between vectors: cos θ = (u · v) / (|u| |v|)
- Projection of u onto v: proj_v(u) = (u · v / |v|²) v
- Testing orthogonality: u · v = 0
Note: |v|² = v · v — the dot product of a vector with itself is its squared magnitude.
The Cross Product (3D only)
The cross product of two 3D vectors gives a new vector:
u × v = (u₂v₃ − u₃v₂, u₃v₁ − u₁v₃, u₁v₂ − u₂v₁)
Geometric interpretation:
- Direction: perpendicular to both u and v (right-hand rule)
- Magnitude: |u × v| = |u| |v| sin θ = area of the parallelogram formed by u and v
Key properties:
u × v = −(v × u) (anti-commutative)
u × u = 0 (parallel vectors have zero cross product)
î × ĵ = k̂, ĵ × k̂ = î, k̂ × î = ĵ
Uses: finding a vector perpendicular to two given vectors (normal to a plane), torque in physics, surface normals in 3D graphics.
Linear Combinations and Span
A linear combination of vectors v₁, v₂, …, vₖ is:
c₁v₁ + c₂v₂ + ... + cₖvₖ
for any scalars c₁, c₂, …, cₖ.
The span of a set of vectors is all possible linear combinations — every vector reachable by scaling and adding those vectors.
Two non-parallel 2D vectors span all of ℝ² — you can reach any point. Three 3D vectors span ℝ³ if none is a linear combination of the others.
Linear Independence
Vectors are linearly independent if no one of them can be written as a linear combination of the others. Equivalently, the only solution to:
c₁v₁ + c₂v₂ + ... + cₖvₖ = 0
is c₁ = c₂ = … = cₖ = 0.
If vectors are linearly dependent, one is “redundant” — it adds no new direction.
Intuition: in 2D, two vectors are linearly independent if they’re not parallel. In 3D, three vectors are linearly independent if they don’t all lie in the same plane.
Vectors in Higher Dimensions
Nothing changes in n dimensions — the formulas are identical, only the number of components grows. This is why linear algebra applies to machine learning: data points are vectors in high-dimensional space, and all the geometric intuitions (distance, angle, projection, orthogonality) carry over directly.
- A dataset of 1000 images, each 28×28 pixels = 1000 vectors in ℝ⁷⁸⁴
- The dot product measures similarity between data points
- Projection finds the component of one vector in the direction of another
- Orthogonality means two features are uncorrelated
The geometry of vectors in 2D and 3D is the same geometry that underlies all of data science.