Majority Vote Flipped 33% of Ambiguous Digit Labels, Only 3% on MNIST

This article introduces a re-labelled, controlleddatasetdesigned for uncertainty exploration. We briefly cover the following:

For more details, please check out the paper itself.

Yet another digit dataset, in this day and age?

Deep learning research keeps scaling, but massive datasets can introduce confounding factors that make understanding fundamental principles of deep learning rather challenging. While modern neural networks do really well on clear, unambiguous examples, their behaviour on examples that humans themselves can’t agree upon is still poorly understood. When a datapoint is genuinely ambiguous (e.g. a digit that could be either a 4 or a 9), how should a model learn and evaluate such data?

A model predicting a 50/50 split could look like indecision, but it might just be the model confidently reporting that the answer is ambiguous. Keeping human uncertainty in labels has been shown to make classification more robust. To understand the benefits of such data better we introduce two carefully-controlled testbeds for exploring the impact of difficult and ambiguous data. Rather than replacing large-scale research, these testbeds allow the science of deep learning community to iterate quickly and cheaply to help with understanding machine learning behaviour at a more fundamental level. 

So why re-label digits?Ground truth labels can diverge from collective human perception: a digit created under the instruction to write a 4 is a 4 by the writer’s intent, but others observing that same digit may mostly see a 9. A common approach to resolve this is to take a majority vote over multiple annotations (e.g. 3 out of 5 people see a 9, so the label is one-hot encoded as a 9). See image below. Another option would be to keep all annotations, which results in soft-labels (e.g., 2/5 for 4 and 3/5 for 9). This gives us an image-level soft-label that was created from each annotator’s top choice, but we don’t really have a measure of uncertainty beyond the differences in top choice. Another way of creating such soft-labels is to collect individual annotator uncertainty by letting them choose more than one label (e.g. see right most part of the image below). 

This last option is the approach we take here: extracting multiple-responses for an image per annotator to get a sense of individual annotator uncertainty. Before we go into annotation details, however, let’s first look at our high level pipeline.

Brief Dataset Pipeline & Annotation

We first narrowed down our dataset candidates to the classical MNIST digits and a synthetically created variant from Mukhoti, which specifically focused on generating ambiguous digits (we’ll refer to this dataset as Mukhoti). The first step in processing was to remove any duplicate images from the dataset. We then sampled a subset of each dataset (see paper for more details) and had humans re-label the digits.

For every image the annotators were instructed to either select yes, no or unsure for each potential digit option (see paper for reasoning). In the example below, the person clearly sees a 9, thus selects yes for the option: 9, and no for every other digit option.

In the case of the next example below the annotator is less certain about the label and selects unsure for both option 5 and 6, while discounting the other options with a no. This results in a label distribution rather than a one-hot target as in the case above. We now have a 50/50 soft-label for the image below.

Such annotations give us annotator-level soft-labels for each digit, which we then aggregate to an image-level soft-label, by averaging the distributions from individual annotators over a given image:  

This gives us data that looks as follows:

On the left side, all annotators choose the option yes for 8 and no for all other options. None of them select unsure as the digit is rather clear. On the right hand, most annotators choose an 8 again, but a portion of annotators also see a 5 or 3 or 9. The image is less clear cut compared to the left one, and so several annotators select the unsure option on this image. 

Brief Dataset Overview

The MNIST subset contains 3,046 labelled images, the Mukhoti subset has 2,484. Each image has between 3 and 10 annotators per image; on average there are about 6 annotators per image.

We compare the original labels with the newly-relabelled data and find that the original label changes when we take a majority vote over the collection of new labels. For instance, say someone got the instruction to write down a 7, follows the guideline and so the label says 7. But the majority of observers (annotators) sees a 1, not a 7, so this image gets relabelled as a 1 (changing its mode). In MNIST only about 3% of the data changed the original label in this way, while in Mukhoti’s synthetic variant about 33% of the labels changed.

The above considers the traditional view of only having one label. If we now look at human label variation, so the soft-labels, we get the following picture: 

The MNIST subset has mostly clear/easy digits where annotators agree on the ground truth label. Around 40% have some label variation. Mukhoti has considerably more difficult/ambiguous images. Only around 20% of the data has no human label variation.  

*_{Note: we intentionally don’t clean the annotations (we recruit high-quality annotators but some noise/sloppy annotations may still exist).}

Data available on Hugging Face

The dataset is released on Hugging Face as image-level-soft-label-digits. These are some of the available columns: 

• Images (images):the grayscale images as a list of size [1, 28, 28]

• Original labels(original_labels):the original labels associated with the this is the synthetically generated soft-label:

• Mean uncertainty(human_uncert_mean):this is a number that gives us the average magnitude of doubt across all annotators for an image. Say we have three annotators in total for an image, each has to select yes/no/unsure for each digit option. So if the first annotator chooses unsure four times, that person’s uncertainty for that image is 4/10 = 0.4. We then take the average uncertainty over all three, see below: 

• Unsure proportion (pct_ann_unsure):this is a number that tells usthe fraction of annotators who expressed uncertainty at least once for a given image. So for the same example as above this would mean 2 out of 3 annotators were unsure at least once, so we have ⅔. 

• Soft-labels(soft_label_yes_unc_equal& soft_label):we take two simple approaches to converting the annotator choices to annotator-level soft-labels, which we then average to get the final image-level soft-labels. To illustrate this more simply, imagine we only have 3 options: 0, 1 or 7 (not all 10 digits); see image below.

  
  

  • In option 1 (soft_label_yes_unc_equal) both yes and unsure selections receive a weight of 1 to form each annotator’s individual probability distribution. So in the case of the very first annotator, having selected yes and unsure, we get a distribution that gives 0.5 to 1 and 0.5 to 7. 

  • In option 2(soft_label), however, unsure gets a half the weight to yes, so we have 1 for yes and 0.5 for unsure which results in a label that gives 0.67 to 1 and 0.33 to 7. 

  These are in no way the only ways to aggregate these labels to a distribution, we leave other options for future exploration. 

• New majority label(soft_label_argmax): New majority voted labels (maj_n). A one-hot encoded label representing the class that received the highest consensus from the human annotators, derived directly from the image-level soft-labels:

How to access a sample

import jsonimport matplotlib.pyplot as pltimport pandas as pddf = pd.read_parquet("hf://datasets/majapavlo/image-level-soft-label-digits/image_level.parquet")row = df.iloc[5113] # sample image indeximage = row["images"].reshape(28, 28)soft_label = json.loads(row["soft_label"])header = " ".join(f"{d:>4}" for d in range(len(soft_label)))values = " ".join(f"{p:>4.2f}" for p in soft_label)plt.imshow(image, cmap="gray")plt.axis("off")plt.subplots_adjust(bottom=0.4)plt.figtext(0.5, 0.06, f"{header}n{values}nnsource: {row['source']}   annotators unsure: {row['pct_ann_unsure']:.0%}", ha="center", family="monospace", fontsize=14)plt.show()

That should result in an output that plots the following:

For more details have a look at:

• Hugging Face: https://huggingface.co/datasets/majapavlo/image-level-soft-label-digits

• Paper to appear in NeurIPS 2026 Evaluations and Datasets Track: https://arxiv.org/pdf/2605.18648

Following the data collection described above, the dataset was recently expanded using a voice-to-text annotation process.  An annotator viewed the images and classified them aloud, which was then processed into the following:

• Unfiltered thought process(asstring): instead of forcing a single final label, the speech was transcribed word-for-word, intentionally keeping hesitation and hedging such as “umm” or “maybe”

• Duration(in seconds):the time it took to label the digit was also included in the add-on data

  
  

Note: This data is saved as a separate file (data_multimodal_addon.csv) on a differentHugging Face repository(image-level-soft-label-digits-addon) as an independent extension and was collected separately from the original paper’s dataset.

Licensing

The soft-digits dataset is released under the CC-BY-SA 4.0 license. This dataset is a combination of a…

• Re-annotated subset of the MNIST dataset, originally created by Yann LeCun and Corinna Cortes. (released under CC-BY-SA 3.0)

• Re-annotated subset of the distributional Ambiguous-MNIST dataset, originally created by Jishnu Mukhoti, Andreas Kirsch, Joost van Amersfoort, Philip H.S. Torr, and Yarin Gal. (released under MIT license)