To understand transformers, you must understand attention.
Many explanations focus on the how rather than the why, making terms like queries, keys, and the underlying math feel arbitrary. You can memorize the attention formula and still have no idea why it actually works. But once you grasp the core intuition behind each step, attention becomes easy to remember and reason about.
In this article, we’ll explore both the how and the why.
This article discusses the derivative of the softmax function, providing an introductory example suitable for readers with a basic understanding of single-variable calculus. It outlines the mathematical principles necessary for computing this derivative and presents a concise expression for the softmax function’s Jacobian matrix, showcasing its application in probability distributions.
Is linear regression a statistical model or a machine learning algorithm? While often used as the “Hello World” of deep learning, the method is deeply rooted in 18th-century statistics. The confusion arises because both fields tackle the same problem from different perspectives, using different terms for the same concepts. This article serves as a bridge between these two worlds. We will break down why certain terms are favored in one domain over the other, giving you a ‘translation layer’ that makes reading both statistical papers and machine learning textbooks much more intuitive.
In this brain-friendly deep dive, we’re stripping Principle Component Analysis down to its first principles. You won’t just learn how to use it; you’ll master the underlying logic and the math that makes it work—no hand-waving or vague metaphors required. By pairing theory with practical code, we’ll move beyond simply calling libraries to achieving a true, foundational understanding of the ‘why’ behind the ‘how.
The Singular Value Decomposition (SVD) is “a highlight of linear algebra” to quote Prof. Strang ( [1] p. 371). However, I must confess that when I studied it I had a difficult time understanding it and this was due to how it was presented. The SVD is often introduced as a given formula which is then shown to just work. But it always felt very unsatisfying to me not knowing why. So – here is the SVD explained the way I wish I had been taught, which is deriving it from first principles.
Why do orthogonal matrices rotate while diagonal matrices stretch?
In this deep dive into linear algebra, we prove the geometric intuition behind the Singular Value Decomposition (SVD). By examining how inner products are preserved and how basis vectors are scaled, I demonstrate why linear transformations are restricted to these specific movements.
