There is a simple and logical way to approach machine learning model selection.
You start with many models. Then you remove the weaker candidates, keep a smaller group of promising ones, and spend your main compute budget on those. That is usually the right thing to do.
But there is one risk: What if the shortlist itself is too narrow?
A practical example from our financial machine learning research illustrates why this question matters.
After our internal selection process, we already had a core set of models for the ML Evaluator, the machine-learning layer that evaluates trading signals. But we decided to leave one additional slot for an external challenger: a model that had not participated in the original internal competition, but had enough independent evidence behind it to justify a serious test.
That model was RealMLP.
Later, RealMLP became part of the final ensembles in both of our ML Evaluator tracks: Intra and Swing.
That experience changed how we think about model selection itself.
We started with a normal internal selection process
We did not begin with the idea, “Let’s try RealMLP.”
At first, we had a broad predefined list of models, and RealMLP was not on it. The list included gradient-boosted trees, neural networks for tabular data, time-series architectures, and several more specialized approaches.
The purpose of the early stages was not to fully train everything. It was to identify which model families justified further investment.
After several rounds of comparison, we had an initial shortlist.
- LightGBM;
- TimesNet;
- XGBoost as a baseline.
- XGBoost;
- CatBoost;
- LSTM;
- MR-Hydra;
- LightGBM as a baseline.
We could have stopped there.
We didn’t.
Why we added a challenger
The reason was fairly simple.
The idea appeared almost by accident: what if we tried a model that had not taken part in our original internal competition at all?
That led us to look outside the project: research papers, benchmarks, and published comparisons of models that might be relevant to our problem. In 2024, David Holzmüller, Léo Grinsztajn, and Ingo Steinwart introduced RealMLP in Better by Default: Strong Pre-Tuned MLPs and Boosted Trees on Tabular Data.
The authors meta-tuned RealMLP and GBDT methods on 118 datasets and then evaluated them on a separate meta-test set of 90 additional datasets. RealMLP was competitive with gradient-boosted trees on medium-to-large tabular datasets. The paper also showed that combining RealMLP with GBDTs could produce particularly strong results.
Of course, none of this proved that RealMLP would work well on our financial data.
But it was enough to ask a useful question: Could we have missed a valuable model family simply because it did not go through our original internal selection path?
We decided to test that.
RealMLP became a separate challenger.
Not a winner. Not a favorite.
Just a candidate that had earned a fair test.
Why you cannot keep adding models forever
There is an obvious danger here.
A bad process would run the main model competition, inspect the validation result, dislike it, add another model, inspect the result again, and repeat until something looks good. At that point, the selection process itself starts to overfit the evaluation data. In other words, you are no longer testing whether a model generalizes to genuinely new data. You are gradually adapting your selection process to one particular evaluation set.
This problem has been known for a long time.
In 2010, Cawley and Talbot showed that overfitting can happen not only during model training, but also during model selection itself. They noted that the resulting selection bias can, in some cases, be comparable to the performance differences between learning algorithms.
That is why a challenger should be introduced before the protected final evaluation, not after seeing the answer. That is how we handled RealMLP.
There was another important development.
We later moved to a new data foundation designed to represent market state without information leaking in from the future. In practical terms, this included correcting part of the data-leakage problems in the dataset. Once that changed, we could no longer assume that the old model ranking was still valid.
So before full training, we expanded the internal roster.
- LightGBM;
- XGBoost;
- CatBoost;
- MR-Hydra;
- TimesNet.
- LightGBM;
- XGBoost;
- CatBoost;
- MR-Hydra;
- LSTM.
Each of these became a full-training candidate. RealMLP remained separate as the external challenger for both tracks.
The structure looked like this:
|
Model families in the initial broad landscape |
||
|
Additional external challenger |
After full training and comparison, RealMLP eventually became part of the final stacking solution.
What did RealMLP achieve on its own?
This is where it is important not to oversell the result. RealMLP’s standalone metrics, meaning its results when evaluated independently rather than as part of an ensemble, were moderate rather than outstanding.
A quick clarification on the stages:
- P3 one-shot OOT means a one-time evaluation on a separate time period that had not been used for training or previous model selection.
- P2 family gate was an earlier governed evaluation stage used to decide whether a model family could continue in the process.
The Intra and Swing numbers should not be compared directly with each other or treated as production-accepted performance figures. They come from different tasks and different evaluation stages.
We include them for transparency, so that anyone looking back at the project history can see how RealMLP performed on its own before it became part of the final ensemble.
In Intra, both sides passed the required quality and temporal-stability checks.
In Swing, the standalone result was weaker. Our internal independent audit noted that the LONG advantage over the causal baseline, a comparison model that only uses information available at the relevant point in time, was small: about +0.0043 AUC.
Performance across different time periods was also uneven. So RealMLP did not look like an obvious standalone champion.
A challenger does not have to be the best standalone model
If the final system uses an ensemble, the question “Which model has the best standalone score?” is not enough. Two strong models may make almost the same mistakes. If they do, adding the second one may contribute very little.
A somewhat weaker standalone model can sometimes be more useful if it contributes information that the other models do not.
This is one reason modern tabular ML research increasingly looks not only at individual algorithms, but also at combinations of model families.
For example, a large NeurIPS 2023 study compared 19 algorithms across 176 datasets.
One practical conclusion was that the simple framing of “neural networks versus boosted trees” is often too crude. On many tasks, the difference between model families was relatively small, while proper model tuning could matter more than the choice of family itself. NeurIPS 2023 study
An earlier NeurIPS 2022 study also showed that tree-based models remain very strong on typical tabular datasets and benefit from useful inductive biases that neural approaches may not naturally have. NeurIPS 2022 study
Then, in 2025, TabArena showed another important point: cross-model ensembles can outperform individual models.
At the same time, the authors warned that even ensemble composition itself can be over-optimized on validation data, which can make the result transfer worse to unseen data. TabArena, NeurIPS 2025
So the useful principle is not:
It is: Evaluate different models under the same rules and measure whether they actually add something to the system.
What happened in Intra
After full training and comparison, RealMLP was not declared the best standalone model.
Instead, it became one of the candidates for the next stage: building an ensemble that combines outputs from multiple models.
The final Intra ensemble was then evaluated separately under predefined rules.
In that evaluation, the ensemble performed better than each of its individual constituent models, and also better than a simple equal-weight combination.
The comparison used two key criteria:
- discrimination between successful and unsuccessful cases;
- improvement among the top 10% of selected predictions.
RealMLP was one of the models inside that ensemble.
We are not claiming that the entire gain of the final ensemble came from RealMLP. The available evidence does not support that conclusion.
But the model that was not an obvious standalone winner turned out to be useful enough to survive the next selection stage and remain part of the final decision.
What happened in Swing
The Swing story was different.
RealMLP did not look like a strong standalone winner there either, but it successfully passed the evaluation stage required for its family and remained eligible for later ensemble comparison.
The final historical Swing solution was also an ensemble, and RealMLP became one of its constituent models.
So we ended up with two separate ML tracks, Intra and Swing, with different selection cycles and different evaluation stages.
And in both cases, the external challenger that had not been part of the original internal shortlist eventually became part of the final ensemble.
RealMLP did not prove that it was better than LightGBM or LSTM.
It proved useful inside the ensemble.
That is a very different conclusion.
The rule we now consider reasonable
The main compute budget should still go to the shortlist that has the strongest internal evidence.
But after that, there can be value in reserving one limited challenger slot, provided that:
- there is strong independent external evidence for testing it;
- the model is meaningfully different from the main roster;
- it is evaluated under the same rules as the other candidates.
A challenger slot is not an invitation to build another model zoo.
It is a small safeguard against closing the search space too early.
The main lesson
A good model-selection process should be able to question not only the models, but also its own shortlist.
The purpose of a shortlist is to reduce the search space and avoid spending resources on everything.
But sometimes one carefully justified slot for an outsider can reveal something that the internal selection process missed.
The idea appeared almost by accident.
In this case, the challenger earned a place in the final ensembles—not because it was the strongest standalone model, but because it remained useful within the broader evaluation process.
