# AI & Machine Learning Formulas

Machine learning looks like magic from the outside. Inside it is a list of formulas used over and over: a weighted sum, a squashing function, a way to measure how wrong a guess was, a rule for getting a little less wrong each time, and a few scores that say how good a model is. Read them in this order and the whole field gets a lot smaller.

Page: https://formulabro.yunusemrevurgun.com/ai-ml/  
Updated: 2026-10-22

## [Artificial Neuron (Weighted Sum)](https://formulabro.yunusemrevurgun.com/ai-ml/artificial-neuron/)

$$
z = \sum_{i=1}^{n} w_i x_i + b, \quad a = f(z)
$$

An artificial neuron multiplies each input by a weight, adds them up with a bias, and then decides how strongly to fire.

Markdown: https://formulabro.yunusemrevurgun.com/ai-ml/artificial-neuron.md

## [Sigmoid Function](https://formulabro.yunusemrevurgun.com/ai-ml/sigmoid-function/)

$$
\sigma(z) = \frac{1}{1 + e^{-z}}
$$

The sigmoid takes any number, however big or small, and squeezes it into a value between 0 and 1 that can be read as a chance.

Markdown: https://formulabro.yunusemrevurgun.com/ai-ml/sigmoid-function.md

## [Softmax](https://formulabro.yunusemrevurgun.com/ai-ml/softmax/)

$$
\operatorname{softmax}(z)_i = \frac{e^{z_i}}{\sum_{j=1}^{K} e^{z_j}}
$$

Softmax takes a list of scores, any numbers at all, and turns them into percentages that add up to 100%.

Markdown: https://formulabro.yunusemrevurgun.com/ai-ml/softmax.md

## [Mean Squared Error (MSE)](https://formulabro.yunusemrevurgun.com/ai-ml/mean-squared-error/)

$$
\mathrm{MSE} = \frac{1}{n}\sum_{i=1}^{n} (y_i - \hat{y}_i)^2
$$

Mean squared error measures how wrong a model's guesses are by squaring each miss, so big misses count much more, and then averaging them.

Markdown: https://formulabro.yunusemrevurgun.com/ai-ml/mean-squared-error.md

## [Cross-Entropy Loss](https://formulabro.yunusemrevurgun.com/ai-ml/cross-entropy-loss/)

$$
L = -\sum_{i=1}^{K} y_i \ln(\hat{y}_i)
$$

Cross-entropy loss gives a small score when the model was confident in the right answer and a big score when it was not.

Markdown: https://formulabro.yunusemrevurgun.com/ai-ml/cross-entropy-loss.md

## [Gradient Descent](https://formulabro.yunusemrevurgun.com/ai-ml/gradient-descent/)

$$
\theta_{\text{new}} = \theta_{\text{old}} - \eta \, \nabla L(\theta_{\text{old}})
$$

Gradient descent nudges a model's settings a small step in the direction that lowers the error, then repeats until the error stops falling.

Markdown: https://formulabro.yunusemrevurgun.com/ai-ml/gradient-descent.md

## [Bayes' Theorem](https://formulabro.yunusemrevurgun.com/ai-ml/bayes-theorem/)

$$
P(A \mid B) = \frac{P(B \mid A)\, P(A)}{P(B)}
$$

Bayes' theorem tells you how much to change your belief in something after you see new evidence.

Markdown: https://formulabro.yunusemrevurgun.com/ai-ml/bayes-theorem.md

## [Scaled Dot-Product Attention](https://formulabro.yunusemrevurgun.com/ai-ml/scaled-dot-product-attention/)

$$
\operatorname{Attention}(Q, K, V) = \operatorname{softmax}\left(\frac{Q K^{\top}}{\sqrt{d_k}}\right) V
$$

Attention lets a model look at every item in a list and decide how much each one matters for the item it is working on.

Markdown: https://formulabro.yunusemrevurgun.com/ai-ml/scaled-dot-product-attention.md

## [Linear Regression](https://formulabro.yunusemrevurgun.com/ai-ml/linear-regression/)

$$
\hat{y} = w x + b
$$

Linear regression predicts a number by drawing the straight line that sits closest to the data points, so you can read off a guess for any new input.

Markdown: https://formulabro.yunusemrevurgun.com/ai-ml/linear-regression.md

## [Logistic Regression](https://formulabro.yunusemrevurgun.com/ai-ml/logistic-regression/)

$$
p = \sigma(w x + b) = \frac{1}{1 + e^{-(w x + b)}}
$$

Logistic regression takes the same weighted sum as a straight line, then squeezes it through an S-shaped curve so the result is a probability between 0 and 1.

Markdown: https://formulabro.yunusemrevurgun.com/ai-ml/logistic-regression.md

## [ReLU Activation](https://formulabro.yunusemrevurgun.com/ai-ml/relu-activation/)

$$
\text{ReLU}(z) = \max(0, z)
$$

ReLU passes positive numbers through unchanged and replaces every negative number with zero, like a switch that is on for positive input and off otherwise.

Markdown: https://formulabro.yunusemrevurgun.com/ai-ml/relu-activation.md

## [Tanh Activation](https://formulabro.yunusemrevurgun.com/ai-ml/tanh-activation/)

$$
\tanh(z) = \frac{e^{z} - e^{-z}}{e^{z} + e^{-z}}
$$

Tanh squashes any input into a number between minus 1 and plus 1, with zero at the center, so a negative input gives a negative output.

Markdown: https://formulabro.yunusemrevurgun.com/ai-ml/tanh-activation.md

## [Mean Absolute Error (MAE)](https://formulabro.yunusemrevurgun.com/ai-ml/mean-absolute-error/)

$$
\mathrm{MAE} = \frac{1}{n}\sum_{i=1}^{n} |y_i - \hat{y}_i|
$$

Mean absolute error measures how far off a model's guesses are on average, counting each miss by its size only, so one wild miss does not dominate the score.

Markdown: https://formulabro.yunusemrevurgun.com/ai-ml/mean-absolute-error.md

## [Huber Loss](https://formulabro.yunusemrevurgun.com/ai-ml/huber-loss/)

$$
L_{\delta}(a) = \begin{cases} \frac{1}{2} a^{2} & \text{if } |a| \le \delta \\ \delta \left(|a| - \frac{1}{2}\delta\right) & \text{otherwise} \end{cases}
$$

Huber loss scores small misses by squaring them, as mean squared error does, and large misses with a straight-line penalty, so outliers cannot take over the score.

Markdown: https://formulabro.yunusemrevurgun.com/ai-ml/huber-loss.md

## [Hinge Loss](https://formulabro.yunusemrevurgun.com/ai-ml/hinge-loss/)

$$
L = \max(0,\ 1 - y\, f(x))
$$

Hinge loss gives no penalty to an answer that is correct with room to spare, and a growing penalty as an answer gets wrong or too close to the boundary.

Markdown: https://formulabro.yunusemrevurgun.com/ai-ml/hinge-loss.md

## [L2 Regularization](https://formulabro.yunusemrevurgun.com/ai-ml/l2-regularization/)

$$
J(w) = L(w) + \lambda \sum_{j} w_j^{2}
$$

L2 regularization adds a penalty for large weights, so the model prefers small, gentle weights that tend to generalize better to new data.

Markdown: https://formulabro.yunusemrevurgun.com/ai-ml/l2-regularization.md

## [L1 Regularization](https://formulabro.yunusemrevurgun.com/ai-ml/l1-regularization/)

$$
J(w) = L(w) + \lambda \sum_{j} |w_j|
$$

L1 regularization adds a penalty proportional to each weight's size, which encourages the model to drop unneeded inputs by setting their weights to exactly zero.

Markdown: https://formulabro.yunusemrevurgun.com/ai-ml/l1-regularization.md

## [Momentum Optimizer](https://formulabro.yunusemrevurgun.com/ai-ml/momentum-optimizer/)

$$
v_t = \beta \, v_{t-1} + \nabla L(\theta_{t-1}) \qquad \theta_t = \theta_{t-1} - \eta \, v_t
$$

Momentum lets gradient descent remember its recent steps, so it speeds up where the slope keeps pointing the same way and wobbles less where it flips.

Markdown: https://formulabro.yunusemrevurgun.com/ai-ml/momentum-optimizer.md

## [Adam Optimizer](https://formulabro.yunusemrevurgun.com/ai-ml/adam-optimizer/)

$$
\theta_t = \theta_{t-1} - \eta \, \frac{\hat m_t}{\sqrt{\hat v_t} + \epsilon}
$$

Adam keeps a running average of each setting's slope and of how big those slopes have been, then uses both to choose a step size for every setting.

Markdown: https://formulabro.yunusemrevurgun.com/ai-ml/adam-optimizer.md

## [Backpropagation and the Chain Rule](https://formulabro.yunusemrevurgun.com/ai-ml/backpropagation-chain-rule/)

$$
\frac{\partial L}{\partial w} = \frac{\partial L}{\partial \hat y} \, \frac{\partial \hat y}{\partial z} \, \frac{\partial z}{\partial w}
$$

Backpropagation finds how much each weight affects the error by multiplying the small slopes along the chain from the weight to the loss.

Markdown: https://formulabro.yunusemrevurgun.com/ai-ml/backpropagation-chain-rule.md

## [Batch Normalization](https://formulabro.yunusemrevurgun.com/ai-ml/batch-normalization/)

$$
\hat x_i = \frac{x_i - \mu_B}{\sqrt{\sigma_B^2 + \epsilon}} \qquad y_i = \gamma \, \hat x_i + \beta
$$

Batch normalization rescales the values in a group of examples so they are centered near zero with a steady spread, then lets the network choose its own scale and offset.

Markdown: https://formulabro.yunusemrevurgun.com/ai-ml/batch-normalization.md

## [Layer Normalization](https://formulabro.yunusemrevurgun.com/ai-ml/layer-normalization/)

$$
\hat x_i = \frac{x_i - \mu}{\sqrt{\sigma^2 + \epsilon}} \qquad y_i = \gamma_i \, \hat x_i + \beta_i
$$

Layer normalization centers and scales the numbers that describe one example, so its values stay in a steady range no matter what other examples are in the batch.

Markdown: https://formulabro.yunusemrevurgun.com/ai-ml/layer-normalization.md

## [Dropout](https://formulabro.yunusemrevurgun.com/ai-ml/dropout/)

$$
\tilde h_i = \frac{m_i \, h_i}{1 - p} \qquad m_i \sim \text{Bernoulli}(1 - p)
$$

Dropout randomly switches off some neurons on each training pass, and scales up the ones that stay on, so the network learns patterns that do not rely on any single neuron.

Markdown: https://formulabro.yunusemrevurgun.com/ai-ml/dropout.md

## [Precision and Recall](https://formulabro.yunusemrevurgun.com/ai-ml/precision-and-recall/)

$$
\text{Precision} = \frac{TP}{TP + FP} \qquad \text{Recall} = \frac{TP}{TP + FN}
$$

Precision asks how many of the items a model flagged were right, and recall asks how many of the real cases it managed to find.

Markdown: https://formulabro.yunusemrevurgun.com/ai-ml/precision-and-recall.md

## [F1 Score](https://formulabro.yunusemrevurgun.com/ai-ml/f1-score/)

$$
F_1 = \frac{2 \cdot P \cdot R}{P + R}
$$

The F1 score is one number from 0 to 1 that stays low unless both precision and recall are reasonably high.

Markdown: https://formulabro.yunusemrevurgun.com/ai-ml/f1-score.md

## [Classification Accuracy](https://formulabro.yunusemrevurgun.com/ai-ml/classification-accuracy/)

$$
\text{Accuracy} = \frac{TP + TN}{TP + TN + FP + FN}
$$

Accuracy is the share of a model's guesses that were right, out of all the guesses it made.

Markdown: https://formulabro.yunusemrevurgun.com/ai-ml/classification-accuracy.md

## [Cosine Similarity](https://formulabro.yunusemrevurgun.com/ai-ml/cosine-similarity/)

$$
\cos\theta = \frac{\mathbf{A}\cdot\mathbf{B}}{\|\mathbf{A}\|\,\|\mathbf{B}\|}
$$

Cosine similarity scores how closely two lists of numbers point the same way, from -1 for opposite directions to 1 for the same direction.

Markdown: https://formulabro.yunusemrevurgun.com/ai-ml/cosine-similarity.md

## [Euclidean Distance](https://formulabro.yunusemrevurgun.com/ai-ml/euclidean-distance/)

$$
d(\mathbf{p}, \mathbf{q}) = \sqrt{\sum_{i=1}^{n} (p_i - q_i)^2}
$$

Euclidean distance is the straight-line distance between two points, found by squaring the gaps, adding them up and taking the square root.

Markdown: https://formulabro.yunusemrevurgun.com/ai-ml/euclidean-distance.md

## [k-Means Clustering](https://formulabro.yunusemrevurgun.com/ai-ml/k-means-clustering/)

$$
J = \sum_{k=1}^{K} \sum_{\mathbf{x} \in C_k} \|\mathbf{x} - \boldsymbol{\mu}_k\|^2
$$

k-means clustering sorts points into K groups by moving each group's center to its middle and reassigning every point to its nearest center.

Markdown: https://formulabro.yunusemrevurgun.com/ai-ml/k-means-clustering.md

## [Kullback-Leibler Divergence](https://formulabro.yunusemrevurgun.com/ai-ml/kl-divergence/)

$$
D_{\mathrm{KL}}(P \,\|\, Q) = \sum_x P(x) \log \frac{P(x)}{Q(x)}
$$

KL divergence measures how badly a set of guessed probabilities fits the real probabilities.

Markdown: https://formulabro.yunusemrevurgun.com/ai-ml/kl-divergence.md

## [Naive Bayes Classifier](https://formulabro.yunusemrevurgun.com/ai-ml/naive-bayes-classifier/)

$$
P(y \mid x_1, \dots, x_n) \propto P(y) \prod_{i=1}^{n} P(x_i \mid y)
$$

Naive Bayes guesses the most likely class by multiplying how common the class is by how well each clue fits it.

Markdown: https://formulabro.yunusemrevurgun.com/ai-ml/naive-bayes-classifier.md

## [Gini Impurity](https://formulabro.yunusemrevurgun.com/ai-ml/gini-impurity/)

$$
G = 1 - \sum_{k=1}^{K} p_k^2
$$

Gini impurity is a score for how mixed the classes are in a group: 0 means every item in the group belongs to the same class.

Markdown: https://formulabro.yunusemrevurgun.com/ai-ml/gini-impurity.md

## [Information Gain](https://formulabro.yunusemrevurgun.com/ai-ml/information-gain/)

$$
\mathrm{IG} = H(\text{parent}) - \sum_{k} \frac{n_k}{n}\, H(\text{child}_k)
$$

Information gain tells you how many bits of guessing a split removes, so a tree can ask the question that sorts the examples best.

Markdown: https://formulabro.yunusemrevurgun.com/ai-ml/information-gain.md

## [Softmax Temperature](https://formulabro.yunusemrevurgun.com/ai-ml/softmax-temperature/)

$$
p_i = \frac{e^{z_i / T}}{\sum_{j=1}^{K} e^{z_j / T}}
$$

Temperature is a dial that makes a model's choices more focused when it is low and more varied when it is high.

Markdown: https://formulabro.yunusemrevurgun.com/ai-ml/softmax-temperature.md

## [Perplexity](https://formulabro.yunusemrevurgun.com/ai-ml/perplexity/)

$$
\text{PPL} = \exp\left(-\frac{1}{N}\sum_{i=1}^{N} \ln p(w_i \mid w_{<i})\right)
$$

Perplexity counts how many words a language model is, on average, torn between each time it guesses the next word. Lower is better.

Markdown: https://formulabro.yunusemrevurgun.com/ai-ml/perplexity.md

## [Positional Encoding](https://formulabro.yunusemrevurgun.com/ai-ml/positional-encoding/)

$$
\text{PE}_{(pos,\,2i)} = \sin\left(\frac{pos}{10000^{2i/d}}\right) \qquad \text{PE}_{(pos,\,2i+1)} = \cos\left(\frac{pos}{10000^{2i/d}}\right)
$$

Positional encoding stamps each word with a pattern of waves that says where it sits in the sentence.

Markdown: https://formulabro.yunusemrevurgun.com/ai-ml/positional-encoding.md

## [Convolution Output Size](https://formulabro.yunusemrevurgun.com/ai-ml/convolution-output-size/)

$$
O = \left\lfloor \frac{W - K + 2P}{S} \right\rfloor + 1
$$

This formula counts how many spots a small sliding window can land on across an image, so you know how big a convolution layer's output will be.

Markdown: https://formulabro.yunusemrevurgun.com/ai-ml/convolution-output-size.md

## [R-Squared](https://formulabro.yunusemrevurgun.com/ai-ml/r-squared/)

$$
R^2 = 1 - \frac{SS_{res}}{SS_{tot}}
$$

R-squared is a score that says how much of the ups and downs in the data a model predicts correctly. A perfect model scores 1.

Markdown: https://formulabro.yunusemrevurgun.com/ai-ml/r-squared.md

## [Q-Learning Update](https://formulabro.yunusemrevurgun.com/ai-ml/q-learning-update/)

$$
Q(s,a) \leftarrow Q(s,a) + \alpha\left[r + \gamma \max_{a'} Q(s',a') - Q(s,a)\right]
$$

Q-learning lets an agent improve its guess about how good an action is, a little at a time, using the reward it received on the latest step.

Markdown: https://formulabro.yunusemrevurgun.com/ai-ml/q-learning-update.md

## [Discounted Return](https://formulabro.yunusemrevurgun.com/ai-ml/discounted-return/)

$$
G_t = \sum_{k=0}^{\infty} \gamma^{k} r_{t+k+1}
$$

The discounted return is the total reward an agent can expect from now on, where rewards that arrive later count for a little less.

Markdown: https://formulabro.yunusemrevurgun.com/ai-ml/discounted-return.md

## [Z-Score Standardization](https://formulabro.yunusemrevurgun.com/ai-ml/z-score-standardization/)

$$
z = \frac{x - \mu}{\sigma}
$$

A z-score says how far a number is from the average, measured in typical steps of spread, so different features can be compared fairly.

Markdown: https://formulabro.yunusemrevurgun.com/ai-ml/z-score-standardization.md