A cheat sheet of 10 regression and classification loss functions.

> Regression: 
1) Mean bias error: captures the average bias in the prediction but is rarely used in training as negative and positive errors can cancel each other.

2) Mean absolute error: Measures the average absolute difference between predicted and actual value. One caveat is that small errors are as important as big ones. Thus, the magnitude of the gradient is independent of error size.

3) Mean squared error: Larger errors contribute more significantly than smaller errors. But this may also be a caveat as it is sensitive to outliers.

4) Root mean squared error: Used to ensure that loss and the dependent variable (y) have the same units.

5) Huber loss: A combination of MAE and MSE. For smaller errors, mean squared error is used. For large errors, mean absolute error is used. One caveat is that it is parameterized — adding another hyperparameter to the list.

6) Log cosh loss: A non-parametric alternative to Huber loss which is a bit computationally expensive.

> Classification: 
1) BCE: Used for binary classification tasks. Measures the dissimilarity between predicted probabilities and true binary labels, through the logarithmic loss.

2) Hinge loss: Penalizes both wrong and right (but less confident) predictions). It is based on the concept of margin, which represents the distance between a data point and the decision boundary. Particularly used to train support vector machines (SVMs).

3) Cross-entropy loss: An extension of BCE loss to multi-class classification tasks.

4) KL Divergence: Measure information lost when one distribution is approximated using another distribution. For classification, however, using KL divergence is the same as minimizing cross entropy. Thus, it is recommended to use cross-entropy loss directly. It is used in t-SNE and knowledge distillation for model compression. 