AI & Machine Learning Formulas

Machine learning looks like magic from the outside. Inside it is a short list of formulas used over and over: a weighted sum, a squashing function, a way to measure how wrong a guess was, and a rule for getting a little less wrong each time. Read them in this order and the whole field gets a lot smaller.

8 formulas, in the order that makes them easiest to learn:

Artificial Neuron (Weighted Sum)

$$z = \sum_{i=1}^{n} w_i x_i + b, \quad a = f(z)$$

An artificial neuron multiplies each input by a weight, adds them up with a bias, and then decides how strongly to fire.

Sigmoid Function

$$\sigma(z) = \frac{1}{1 + e^{-z}}$$

The sigmoid takes any number, however big or small, and squeezes it into a value between 0 and 1 that can be read as a chance.

Softmax

$$\operatorname{softmax}(z)_i = \frac{e^{z_i}}{\sum_{j=1}^{K} e^{z_j}}$$

Softmax takes a list of scores, any numbers at all, and turns them into percentages that add up to 100%.

Mean Squared Error (MSE)

$$\mathrm{MSE} = \frac{1}{n}\sum_{i=1}^{n} (y_i - \hat{y}_i)^2$$

Mean squared error measures how wrong a model’s guesses are by squaring each miss, so big misses count much more, and then averaging them.

Cross-Entropy Loss

$$L = -\sum_{i=1}^{K} y_i \ln(\hat{y}_i)$$

Cross-entropy loss gives a small score when the model was confident in the right answer and a big score when it was not.

Gradient Descent

$$\theta_{\text{new}} = \theta_{\text{old}} - \eta \, \nabla L(\theta_{\text{old}})$$

Gradient descent nudges a model’s settings a small step in the direction that lowers the error, then repeats until the error stops falling.

Bayes' Theorem

$$P(A \mid B) = \frac{P(B \mid A)\, P(A)}{P(B)}$$

Bayes’ theorem tells you how much to change your belief in something after you see new evidence.

Scaled Dot-Product Attention

$$\operatorname{Attention}(Q, K, V) = \operatorname{softmax}\left(\frac{Q K^{\top}}{\sqrt{d_k}}\right) V$$

Attention lets a model look at every item in a list and decide how much each one matters for the item it is working on.

Other subjects: Computer Science | Electricity | Quantum Mechanics. Plain-text copy: ai-ml.md.