Neural Networks are Hypothesis Testers

Consider the following two time series.

[Plot: Two time series plots that visually appear as random noise.]

If we associate labels to the samples and if a neural network, after being shown many examples of these noisy time series paired with their labels, manages to consistently reduce its loss during training, something is being learned. The network is, in effect, discovering a subtle correlation, a hidden pattern invisible to the naked eye. It is testing the hypothesis that the noise isn't entirely random, that there's a discernible relationship between the time series and the labels. The decreasing loss[^1] is the evidence that supports this hypothesis.

Similarly, consider the task of next bit prediction.

[Series of random 0's and 1's.]

If the sequence is truly random, there's nothing to predict. So lets train a neural network to auto-regressively predict the next bit, and observe the loss. If it decreases, however infinitesimally, a non-trivial inference can be drawn: there is structure.

This leads to a fundamental observation: for any learning system and any amount of data, a decrease in the loss function signifies the acceptance (or at least, non-rejection) a hypothesis. The hypothesis is embedded within the network's architecture and the specific configuration of its weights. The training process is the experiment, and the loss function is the metric by which the hypothesis is evaluated.

Extending beyond the direct realm of bits and pixels, consider having a controversial statement and you want to gauge public reaction to subtle variations. Formulate variations of the statement, tweaking aspects you are uncertain about, and feed them into a prominent LLM. The language model, trained on vast amounts of text representing human expression, acts as a proxy for average human sentiment, offering a potentially insightful A/B test of public opinion[^2].

What does the absence of color look like?

Based on the general consensus, the absence of color is either black (cite) or white (cite). Lets ask a model just to be sure. Revisiting an idea from around 2019: Imagine we train a simple generative neural network to predict the pixel value (i.e., color) of an image based on a one-hot encoded input representing a specific color. We feed it millions of examples of different colors.

[Plot: A few examples of one-hot vectors with corresponding color inputs.]

What does the network, trained on millions of color examples, predict when presented with this "absence of color" input? Keep in mind, the zero-vector is not in the training distribution, we never explicitly trained on the concept of "no color," but we still get an output. And its the following:

[Plot: Visual representation of the input (all zeros) and the network's output (predicted color).]

Now, you might say that this is due to random parameter initialization. Well lets try the experiment for 100 different models and average the result.

[Plot: Visual representation of the input (all zeros) and the average output as a density plot over all colors.]

And like that, another hard day at the hypothesis factory is over.

[^1]: You can assume test loss for now. [^2]: With the problem being that these models are trained to be agreeable, to avoid controversy, and to provide solutions that are generally acceptable. In essence, they offer average solutions for average problems.