scikit-learn : Bias-variance tradeoff

The bias-variance tradeoff is a central problem in supervised learning.
Ideally, one wants to choose a model that both accurately captures the regularities in its training data, but also generalizes well to unseen data.
Unfortunately, it is typically impossible to do both simultaneously.
High-variance learning methods may be able to represent their training set well, but are at risk of overfitting to noisy or unrepresentative training data.
In contrast, algorithms with high bias typically produce simpler models that don't tend to overfit, but may underfit their training data, failing to capture important regularities.

Source : Bias-variance tradeoff

picture source: Python Machine Learning - Sebastian Raschka

picture source: Understanding the Bias-Variance Tradeoff

Suppose we want to approximate a $\sin {\pi x}$ function with two points ($N=2$).
We're going to use two sets of hypothesis , $\mathcal H_0$ and $\mathcal H_1$.
$\mathcal H_0$ is a hypothesis of using a const ($y=C$) while $\mathcal H_1$ using a line ($y=ax+b$).

The left figure shows outputs from const hypothesis($\mathcal H_0$), and each line is approximation corresponding each data set. The picture on the right side shows the the averaged target hypothesis ($\bar g(x)$) we get and the gray strip tells us the variance. Note that for each learning, we get one of the lines in the picture below, and the green avareged hypothesis $\bar g(x)$ is actually the best approximation we get.

Now, we want to play with $\mathcal H_1$ with the same data set we used in the previous picture. The output is shown in the figure below:

It appears we now have better approximation in terms of averaged hypothesis ($\bar g(x)$) while we got greater variance.

Let's compare the two side by side:

Picture source: Caltech : Lecture 08 - Bias-Variance Tradeoff

At its root, dealing with bias and variance is really about dealing with over- and under-fitting.
Bias is reduced and variance is increased in relation to model complexity. As more and more parameters are added to a model, the complexity of the model rises and variance becomes our primary concern while bias steadily falls.
For example, as more polynomial terms are added to a linear regression, the greater the resulting model's complexity will be.



Understanding bias and variance is critical for understanding the behavior of prediction models, but in general what we really care about is overall error, not the specific decomposition.
The sweet spot for any model is the level of complexity at which the decrease in bias is equivalent to the increase in variance. - ref. Understanding the Bias-Variance Tradeoff

Machine Learning with scikit-learn

Machine learning algorithms and concepts

Artificial Neural Networks (ANN)