When we try to build a prediction algorithm, it is a useful practice to first try out a baseline algorithm and see how they perform. Here we will describe a case of random prediction.

(Inspiration and examples are from Jason Brownlee's Machine Learning Algorithms from Scratch book.)

import pandas as pd
import altair as alt
import numpy as np

from vega_datasets import data
np.random.seed(42)

For the examples we will use the vega 'volcano' dataset. The width and height values are constant, so we work only with the values column.

volcano = data('volcano')
volcano.head()
width height values
0 87 61 103
1 87 61 104
2 87 61 104
3 87 61 105
4 87 61 105

The random prediction algorithm

  1. takes a training and a test set
  2. generates the prediction by selecting random elements from the training set

We split the dataset to training and test sets, by taking the first 2/3 and last 1/3 of the data respectively.

train, test = volcano.iloc[: volcano.shape[0] * 2 // 3, :], volcano.iloc[volcano.shape[0] * 2 // 3 :, :]

A small check that the split did not leave out an entry or did not result in an overlap.

assert train.index[-1] + 1 == test.index[0]

We plot the training and the test sets, respectively.

alt.Chart(train.reset_index()).mark_line().encode(alt.Y('values'), alt.X('index:Q'), alt.Tooltip('values')).interactive().properties(title='Train data')
alt.Chart(test.reset_index()).mark_line().encode(alt.Y('values'), alt.X('index:Q'), alt.Tooltip('values')).interactive().properties(title='Test data')

First, we generate the random predictions with replacement. That is, the same values can occur more than once.

predictions = np.random.choice(train['values'], size=test.shape[0], replace=True)
predictions
array([166, 179,  96, ..., 105, 157,  95])

We calculate the error terms

errors = test['values'] - predictions
errors
3538   -16
3539   -29
3540    54
3541    54
3542    56
        ..
5302    -8
5303     2
5304    -7
5305   -60
5306     2
Name: values, Length: 1769, dtype: int64

We calculate the root mean squared error of the predictions.

def calculate_rmse(observed, predicted):
    return np.sqrt(sum((observed - predicted) ** 2)/len(observed))

The RMSE is around 34.59 which stands for about 1.34 STD.

rmse = calculate_rmse(test['values'], predictions)
rmse
34.93380641982711

For reference, if we would simply use the simple mean, we would get an RMSE of 18.42,

calculate_rmse(test['values'], test['values'].mean())
18.420942507684327

Let's plot the results

to_compare = pd.concat(
    [
        test['values'].rename('observed').reset_index(drop=True),
        pd.Series(predictions).rename('predictions').reset_index(drop=True)
    ], axis=1
).stack().reset_index().rename(columns={'level_1': 'status', 'level_0': 'index', 0: 'values'})

to_compare
index status values
0 0 observed 150
1 0 predictions 166
2 1 observed 150
3 1 predictions 179
4 2 observed 150
... ... ... ...
3533 1766 predictions 105
3534 1767 observed 97
3535 1767 predictions 157
3536 1768 observed 97
3537 1768 predictions 95

3538 rows × 3 columns

alt.Chart(to_compare).mark_line().encode(
    alt.X('index:Q'), alt.Y('values'), alt.Color('status'), alt.OpacityValue(0.7)
).properties(width=1200, title='Predictions with replacement')

We put the steps into a function and rerun the prediction without replacement.

def predict_randomly(train, test, replace=True):
    predictions = np.random.choice(train, size=test.shape[0], replace=True)
    errors = test - predictions
    rmse = calculate_rmse(test, predictions)

    return predictions, rmse
predictions, rmse = predict_randomly(train['values'], test['values'], replace=False)
rmse
35.90392934559341

Finally, we also put the plotting steps into a function

def plot_predictions(observed, predicted):

    to_compare = pd.concat(
        [observed.rename('observed').reset_index(drop=True), pd.Series(predicted).rename('predictions').reset_index(drop=True)], axis=1
    ).stack().reset_index().rename(columns={'level_1': 'status', 'level_0': 'index', 0: 'values'})

    chart = alt.Chart(to_compare).mark_line().encode(
        alt.X('index:Q'), alt.Y('values'), alt.Color('status'), alt.OpacityValue(0.7)
    ).properties(width=1200)

    chart.display()

The predictions without replacement

plot_predictions(test['values'], predictions)