My Machine Learning Pipeline Cheated, and Cross-Validation Never Noticed

The Pipeline We All Write

If you have used scikit-learn for a while, you have probably written this pipeline. Numeric columns get imputed and scaled. Everything else gets imputed and one-hot encoded. Columns are routed by dtype, so it works on almost any table without you looking at it.

def auto_pipeline(model):num = make_column_selector(dtype_include="number")cat = make_column_selector(dtype_exclude="number")pre = ColumnTransformer([("num", make_pipeline(SimpleImputer(strategy="median"), StandardScaler()), num),("cat", make_pipeline(SimpleImputer(strategy="most_frequent"),OneHotEncoder(handle_unknown="ignore")), cat)])return make_pipeline(pre, model)

It is short, reusable, and easy to trust. That is the problem. I wanted to find out what it silently gets wrong, so I tried to break it on purpose.

Four Traps I Planted

Data. Everything here is synthetic. I generated it myself with a Python script: a customer-churn table with 8,000 rows per dataset (6,000 for training, 2,000 as a clean holdout of “new customers”). I generated 5 independent datasets (seeds 0 to 4) and report the mean across them.

I planted four traps that a dtype-based pipeline cannot see:

1. Region as a number.region_code holds the integers 1 to 12, but each region affects churn differently and not in order. The pipeline scales it like a measurement.

2. Informative missing values.support_calls is missing for about 45% of churners and 8% of non-churners. Median imputation erases that signal.

3. A date stored as text.signup_date is a “YYYY-MM-DD” string, so it becomes a one-hot column per unique date (1,931 unique values in 6,000 rows).

4. An ID that leaks the label. In the training data, customer_id was assigned roughly in order of outcome, a stand-in for the way export order or timing can correlate with outcomes in real systems. In the clean holdout, IDs are unrelated to churn.

I tested the naive pipeline, the naive pipeline plus one fix at a time, and all four fixes together. I used two models: logistic regression (cross-validation AUC and holdout AUC) and gradient boosting (holdout AUC only, to keep runtime down).

The Score That Lied

The naive pipeline scored 0.979 in 5-fold cross-validation and 0.564 on new customers. Nothing crashed, no warning appeared, and every number looked like a strong model.

The ID Column Knew the Answer

I looked inside the logistic regression for seed 0. Its single strongest feature, out of 1,940, was customer_id (|coefficient| 5.05). The next strongest were individual signup-date columns, which are one-hot columns for specific days.

So the model did two things. It read the label off the ID, and it memorized per-date noise. The date problem is not about unseen dates: 96.7% of holdout rows had a date that also appeared in training. The model had simply memorized what happened on those specific days.

One Fix Wasn’t Enough

Look at the “+ drop the ID column” row. Cross-validation became honest (0.640 versus 0.632 on new data), which is what you want from a validation number. But holdout AUC was only 0.632, because the other three problems were still there.

The missing-value indicator gave the largest single gain for logistic regression (0.564 to 0.670). Casting region to a category helped (to 0.624). Parsing the date helped a little (to 0.585). Only all four together reached 0.826, which is where the honest cross-validation score (0.833) and the holdout score finally agree.

Can a Better Model Rescue It?

Partly. Gradient boosting scored 0.614 on the naive pipeline, a bit above logistic regression’s 0.564, but still poor. Dropping the ID helped it the most (0.739), and all four fixes reached 0.809.

Two caveats. For gradient boosting, the region fix (0.607) is within noise of the naive result, and the missing-indicator variant has a large spread (std 0.084 across datasets). I would not read anything into those two rows individually. The consistent finding is the ID effect and the all-fixes result.

Teaching the Pipeline to Check Itself

If you build <a href="https://myappsplus.com/c-h-robinson-expands-ai-automation-to-ocean-freight-air-cargo-and-customs-brokerage/” title=”C.H. Robinson Expands AI Automation to Ocean Freight, Air Cargo and Customs Brokerage”>automation for people who don’t code, “remember to check for leaks” is not a plan. The checks have to run by themselves. Every trap above leaves a statistical fingerprint, so I wrote one function that scans a table before modeling and prints plain-English warnings:

import numpy as np, pandas as pdfrom sklearn.metrics import roc_auc_scoredef audit_table(df, target):"""Plain-English warnings for the traps a dtype-based pipeline cannot see."""y = df[target].valueswarns = []for col in df.columns.drop(target):s = df[col]n_unique = s.nunique()# 1. ID-like or order-leaking columnif pd.api.types.is_numeric_dtype(s) and n_unique > 0.95 * s.notna().sum():m = s.notna().valuesauc = roc_auc_score(y[m], s[m])if abs(auc - 0.5) > 0.15:warns.append(f"'{col}' is almost all-unique and predicts the target on its own "f"(AUC {auc:.2f} from row order). Likely a leaking ID: drop it.")else:warns.append(f"'{col}' is almost all-unique. Looks like an ID: consider dropping it.")# 2. numeric column that behaves like a categoryelif pd.api.types.is_numeric_dtype(s) and n_unique <= 20:m = s.notna().valuesx, yy = s[m].values, y[m]r2_lin = np.corrcoef(x, yy)[0, 1] ** 2means = pd.Series(yy).groupby(x).transform("mean").valuesr2_cat = 1 - ((yy - means) ** 2).sum() / ((yy - yy.mean()) ** 2).sum()if r2_cat - r2_lin > 0.02:warns.append(f"'{col}' has {n_unique} distinct numeric values and its effect is not a straight line "f"(per-value means explain {r2_cat:.1%} of the target vs {r2_lin:.1%} as a number). "f"Try treating it as a category.")# 3. informative missingnessif s.isna().any():miss = s.isna().valuesgap = abs(y[miss].mean() - y[~miss].mean())if gap > 0.10:warns.append(f"'{col}' is missing {miss.mean():.0%} of the time, and the target rate differs by "f"{gap:.0%} when it is missing. Add a missing-value indicator.")# 4. high-cardinality textif not pd.api.types.is_numeric_dtype(s) and n_unique > 50:warns.append(f"'{col}' is text with {n_unique:,} unique values. One-hot encoding will memorize noise. "f"Parse it (e.g. dates to year) or drop it.")return warnsdef gap_warning(cv_score, holdout_score, tol=0.05):if cv_score - holdout_score > tol:return (f"Cross-validation ({cv_score:.2f}) is much higher than holdout ({holdout_score:.2f}). "f"Suspect leakage or memorization.")return "Cross-validation and holdout agree."

I ran it on the training table of all 5 datasets. It flagged exactly the four planted columns every time (customer_id, region_code, support_calls, signup_date) and left tenure_months, monthly_spend and plan_type alone. For seed 0 it printed:

The second function, gap_warning(0.979, 0.564), catches the headline problem directly: “Cross-validation (0.98) is much higher than holdout (0.56). Suspect leakage or memorization.”

The point is not that these thresholds are perfect. It is that a non-coder never has to know what a leaked ID looks like, because the tool tells them in a sentence.

Where This Approach Falls Short

• One synthetic dataction of the effects should carry over, but the sizes will differ on real data. Fix: run the same variants on a real table you know well

• The ID leak is planted and exaggerated. I built the ID to correlate strongly with the outcome so the failure would be visible. Real leaks are usually weaker. It stands in for the real pattern of identifiers correlated with outcome through export order or time. Fix: check every ID-like or timestamp-like column for its relationship with the target before modeling.

• Only two model families. Other models may react differently. Fix: repeat with your own model, using a holdout built the way your future data will arrive.

• Spread in some results. Gradient boosting’s missing-indicator and region numbers are noisy. Fix: use more seeds or repeated CV before drawing conclusions about a single change.

• The checker was built and tested on the same synthetic data. The four traps are strong and clean, so detection was easy, and I did not test false-alarm rates on real tables. Real data will trigger warnings that need a human decision (a flagged ID may be harmless, a skewed category may be real). Fix: treat warnings as prompts to look, not as automatic fixes, and tune the thresholds on your own data.

Five Questions Before You Trust Automation

1. Is there a holdout that looks like future data? Cross-validation on training data cannot catch a leak that exists in the training data. Build the holdout the way new data will actually arrive (by time, or from a separate export).

2. Which column does the model lean on most? Look at the top coefficients or permutation importances. If an ID, row number or timestamp is on top, stop.

3. Is a numeric column really a category? Codes, zip codes and region IDs are numbers that are not quantities.

4. Are the missing values random? Compare the target rate where a column is missing against where it is present. If they differ, add a missing indicator.

5. Are there text columns with thousands of unique values? Dates, names and free text need their own handling, not one-hot encoding.

Automation Isn’t Verification

An 8-line pipeline is a good starting point and a poor stopping point. In this experiment it produced a 0.98 that fell to 0.56 on new data, and it took four separate fixes, not one, to reach an honest 0.83. Automation saved time, but it did not do the checking, so I added the checking to the automation. If you build tools for people who can’t spot these problems themselves, that is the part to build in.