I Trained Six Models for Fraud Detection, and the Best One Isn’t in Production

Somewhere in the middle of my final-year project, I trained six different models using the same fraud dataset, logged every training run, and picked a winner. Right now, the model that powers the current deployment is not the winning model.

NairaShield is an AI-based fraud detection system that I built as my final year project for banking transactions. It started as a straightforward classification problem and ended up as something closer to a decision-support pipeline, partly because of that mismatch, and partly because my supervisor wasn’t satisfied with the straightforward version.

He really liked the project, but there was one little thing; he had seen an anti-money laundering project by one of my classmates and felt that the two were too similar.

I tried so hard to explain that the modelling problem was different, but he still wasn’t convinced.

We had quite a few arguments over this matter before he told me what I needed to do to solve the problem at hand. And that single note is the reason the system ended up with a full regulatory review workflow and a second model family it wouldn’t otherwise have had. It’s also most of what this piece is actually about.

This is the first thing I’ve properly written since I left school a few weeks ago, capping off a two-month stretch away from writing while coursework and this project ate up everything else, so it felt right to come back with the project that has the most to actually show.

Merging two datasets that don’t agree with each other

I trained on two public datasets that have no common schema. PaySim is a simulation of mobile money transactions and has attributes like oldbalanceOrg and newbalanceOrig. On the other hand, the IEEE-CIS fraud detection dataset covers card transactions and has none of that; instead, it has only ProductCD, card1, card2, and a long tail of other anonymized features. Before any of that merging actually happens, each dataset gets validated against its own schema first, so a malformed row fails loudly instead of quietly poisoning the training set:

class PaySimSchema(BaseModel):step: inttype: stramount: float = Field(ge=0)nameOrig: stroldbalanceOrg: floatnewbalanceOrig: floatnameDest: stroldbalanceDest: floatnewbalanceDest: floatisFraud: int = Field(ge=0, le=1)isFlaggedFraud: intclass IeeeCisSchema(BaseModel):TransactionID: intisFraud: int = Field(ge=0, le=1)TransactionDT: intTransactionAmt: float = Field(ge=0)ProductCD: strcard1: Optional[float] = Nonecard2: Optional[float] = Noneaddr1: Optional[float] = NoneP_emaildomain: Optional[str] = None

Once both sides pass validation, mapping them into one shared structure looks like this:

df_paysim_aligned["transaction_id"] = "PAYSIM_" + df_paysim["step"].astype(str) + "_" + df_paysim.index.astype(str)df_paysim_aligned["channel"] = df_paysim["type"].astype(str)product_map = {"W": "CARD_WEB", "H": "CARD_HOST", "C": "CARD_PHONE", "S": "CARD_STORE", "R": "CARD_RECURRING"}df_ieee_aligned["channel"] = df_ieee["ProductCD"].map(product_map).fillna("CARD_OTHER")

What took me way too long to realize was that having a unified schema meant the chosen features would have to be ones both datasets shared.

The paySim balances didn’t make the cut because IEEE-CIS doesn’t have a similar field. The actual deployed model is trained using three things: the amount of transaction, one-hot encoding of the channel, and one-hot encoding of the

I’m flagging that now because it matters a lot later in this piece, and ignoring it would undercut a very good thing I am trying to say here.

Balancing a dataset where fraud barely shows up

Since there is very little fraud in both data sources, a model will be able to achieve high accuracy simply by predicting “no fraud” a lot of times while not predicting many fraud instances.

I used SMOTE to oversample the minority class on the training split only, with a fallback to a hand-written interpolation version in case imbalanced-learn wasn’t available in the deployment environment:

k_neigh = min(2, sum(y_train == 1) - 1)if k_neigh >= 1:smote = SMOTE(random_state=42, k_neighbors=k_neigh)X_train_res, y_train_res = smote.fit_resample(X_train, y_train)else:print("[Warning] Insufficient minority class samples to perform SMOTE. Skipping resampling.")X_train_res, y_train_res = X_train, y_train

The zero dependency fallback uses interpolation between a sample and its closest neighbors through Euclidean distance to produce synthetic minority samples, which is essentially a simpler version of how SMOTE works internally but left in the code for the sake of preventing failure when in a minimalistic setting.

Six models and a spreadsheet that argues with itself

I trained Random Forest, Logistic Regression, XGBoost (both baseline and optimized versions), and LightGBM (again, both baseline and optimized). Every one of them ran through the same evaluation helper, so the numbers would actually be comparable instead of six slightly different measurement approaches wearing one result:

def fit_eval_and_log(model, name, filename, params, fit_kwargs=None):print(f"n--- Training Model: {name} ---")start_time = time.time()model.fit(X_train, y_train, **(fit_kwargs or {}))print(f"[Training Info] Model training completed in {time.time() - start_time:.3f} seconds.")preds = model.predict(X_test)probs = model.predict_proba(X_test)[:, 1] if hasattr(model, "predict_proba") else preds.astype(float)metrics = {"Accuracy": round(accuracy_score(y_test, preds), 6),"Precision": round(precision_score(y_test, preds, zero_division=0), 6),"Recall": round(recall_score(y_test, preds, zero_division=0), 6),"AUC-ROC": round(roc_auc_score(y_test, probs), 6),"AUC-PR": round(average_precision_score(y_test, probs), 6),}print(f"{name} Test AUC-PR:", metrics["AUC-PR"])joblib.dump(model, filename)log_experiment_run(name, params, metrics)return metrics# Baseline vs. tuned, run back to back so the comparison is apples to applesxgb_base = xgb.XGBClassifier(use_label_encoder=False, eval_metric="logloss", random_state=42)fit_eval_and_log(xgb_base, "XGBoost (Baseline)", "xgboost_model.joblib", xgb_base.get_params(),fit_kwargs={"eval_set": [(X_train, y_train), (X_test, y_test)], "verbose": 10})xgb_tuned_params = tuned_params.get("xgboost", {"n_estimators": 150, "max_depth": 6, "learning_rate": 0.1,"subsample": 0.8, "colsample_bytree": 0.8, "random_state": 42})xgb_tuned = xgb.XGBClassifier(**xgb_tuned_params, use_label_encoder=False, eval_metric="logloss")fit_eval_and_log(xgb_tuned, "XGBoost (Tuned)", "xgboost_model_tuned.joblib", xgb_tuned_params)lgb_base = lgb.LGBMClassifier(random_state=42, verbose=-1)fit_eval_and_log(lgb_base, "LightGBM (Baseline)", "lightgbm_model.joblib", lgb_base.get_params())

Every run, tuned or not, gets logged to experiment_runs.json with its own hyperparameters and metrics, which is the only reason it was possible to notice, months later, that the model in production wasn’t the one at the top of that log:

A few things that are worth considering rather than skipping over. Logistic Regression shows the highest value of recall among all models at 0.95.

But don’t just look at that; it also proves to be the least usable model of the six, as its low precision value of 0.14 means that most of its identified cases of fraud are false positives.

Tuning XGBoost improved AUC-PR slightly but actually made precision worse than the baseline, which is a reminder that hyperparameter search optimizes whatever metric you point it at, not necessarily the one that actually matters once the model is making real decisions.

Finally, the baseline settings for LightGBM proved to be more efficient in terms of AUC-PR than optimized ones and were the best among all other models in the test.

You can always look back up to the diagram above to see what I’m talking about.

The production API still uses XGBoost (Tuned) as an active model, as this choice was made before the comparison of LightGBM models was completed.

Which, when you look at it now, is honestly worth revisiting.

SHAP is what makes any of these numbers accountable to a human. Every flagged prediction comes with a list of features of what pushed it toward fraud, since “the model said so” isn’t an answer a bank or a regulator should accept:

explainer = shap.TreeExplainer(model)shap_values = explainer.shap_values(X_instance)

What my supervisor actually wanted

The anti-money laundering project by a colleague of mine from the beginning of this piece is worth returning to, because the similarity my supervisor saw wasn’t imagined. On the surface, his work and mine were doing the same basic job, both taking transaction data and deciding whether it looked suspicious, and my supervisor wasn’t wrong to notice that.

What he was actually pushing me to build wasn’t another algorithm.

He was pointing me toward the step that comes after a transaction gets flagged, the part where something actually happens next. From what I found digging into other academic AML work, most of it stops right at classification. His comment was what got me thinking about what follows after the flag.

This led to the creation of a Regulatory Notification Center; a review layer built around role types modeled on Nigeria’s Central Bank, the EFCC, and NDIC, each having a separate login and their own queue of flagged transactions.

A regulatory reviewer can approve, ask for OTP verification, or block it outright, and each decision will be documented in the audit log with who acted and when. Building that workflow is also what forced me to include LightGBM in the comparison.

Where the confidence gate actually comes from

The API doesn’t treat a fraud prediction as a single yes-or-no. The base classification threshold sits at 0.50, but what happens after that is split into bands:

if response.get("prediction") == 1:fp = response.get("fraud_probability", 0.0)alert_payload = {"alert_id": f"API-ALT-{_random.randint(100000, 999999)}","transaction_id": data.get("transaction_id", f"TXN-{_random.randint(10000, 99999)}"),"amount": float(data.get("amount", 0.0)),"timestamp": datetime.datetime.now().isoformat(),"model_probability": fp,"triggered_rules": ["MODEL_HIGH_FRAUD_RISK"] if fp >= 0.85 else ["MODEL_SUSPICIOUS_RISK"],"status": "BLOCKED" if fp >= 0.80 else "PENDING_OTP","rag_context": response.get("rag_context", []),"sender_bank": data.get("sender_bank"),"regulatory_body": data.get("regulatory_body")}# Dispatch to SMS/Email notifier if it is critical (>= 85%)if _api_notifier is not None and fp >= 0.85:threading.Thread(target=_api_notifier.dispatch, args=(alert_payload,)).start()

Anything that scores less than 0.50 passes without any alarm whatsoever going off. If the score falls between 0.50 and 0.80, then the transaction becomes flagged but is not stopped; it’s routed to PENDING_OTP, which basically means that a human or a secondary verification step gets involved before anything final happens.

Scores from 0.80 and above block the transaction from happening, while at 0.85 an immediate alarm is sent eitherhout waiting for any request-response cycle

Defense day

The panel started with the usual, or would I say expected, set of questions. Which models I used, what hyperparameters I tuned and how I arrived at them, where the datasets came from, and whether they were representative enough of transaction behavior specifically.

I told them XGBoost (Tuned), because that’s what was actually in production. What I left out, mostly because I hadn’t sat with it properly myself yet, was that LightGBM’s baseline had already beaten it on the metric that mattered most, days before I stood in that room. Nobody followed up on it. I’m the one following up on it now.

Then it moved to the one I had not really prepared for: what is special about this fraud detection solution compared to all the others that already exist, and what problems with other solutions does it solve?

The scenario questions came next.

If the CEO of a big company decides one random afternoon to move a million dollars in a single transaction, would it get flagged?

And if a student attempted the same transaction, what would happen?

I told them the system studies patterns and behavior, meaning it would treat those two transactions differently depending on who was making them.

Writing this months later, with the code in front of me instead of a panel across the table, I want to give the more accurate answer instead of the more comfortable one.

With the system as it is actually built now, those two transactions would receive an identical prediction. There’s no account identity or transaction history in the thirteen features on which the model learns to look at how much money changed hands, through what channel, and how similar the transaction pattern is to either of the two

The honest answer is that my system currently treats a million dollars the same regardless of whose account it left. It sees a million dollars, and that’s all, and it does nothing with that knowledge except treat it like any other transaction.

The thing it does, though, is that it doesn’t rely on that number alone to make a decision. In my system, if a million-dollar transaction is predicted to be fraud at a probability of 0.85, it is flagged, escalated, and in the regulatory workflow, is sent to a human for review. That human will have more context to work with than the model.

The confidence gate is doing the job that account-level reasoning would do if it existed yet. It’s a real answer, just a more modest one than the answer I gave that day.

Final thoughts and what I’d actually do differently

That leads us straight to the next step, which is the one the defense question revealed: adding behavioral features per account, averaging transactions, and deviation from historical user amounts. With time, the model should then be able to make the distinction I spoke of, rather than approximating it through the confidence gate.

Additionally, I would revisit defaulting to XGBoost (Tuned) as the production model given LightGBM’s baseline configuration scored better on the metric that matters most for this problem.

Both of those come from the same mistake: I locked in a production model before the comparison was even finished, and only really noticed while writing this down.

If there’s one thing worth taking from all of it, it’s that the best score on a spreadsheet isn’t the same as the right production decision, and a model was never going to be the whole system by itself. The confidence gate, the audit log, the human reading a PENDING_OTP alert, that’s doing as much of the real fraud-catching as the model is.

I’m back to writing regularly after two months away, and this felt like the right project to be honest about first. There’s a lot I haven’t covered here: the RBAC setup, the audit logging, the regulatory notification pipeline itself; that’s probably its own piece once this one’s out in the world.

Before you go!

I write about the real, messy engineering behind AI systems, where abstractions help, where they hurt, and what it takes to build reliably.

You can subscribe to my newsletter if you’d like more of that.