This article discusses my recent participation in a Hugging Face competition where I held position 7 out of the roughly 300 that participated and spent less than 80 dollars while at it.
I should have written this way earlier while in the competition, but if you have realized, I am quite lazy and also do not respect the algorithm enough to appeal to its style of ranking – rather, I hope that my content, no matter how random, reaches the right eyes, and fetches the right audience in time.
The competiton had a very simple task built out of neccessity: There is way more papers out there now in each conference than ever before, mostly due to AI. As such, rather than banning calculators and televisions, we must approach it pragmatically, and in doing so, they believed so be it, we shall have agents reproduce all the papers, and all the reproductions will be cointaned within a space on a hugging face with a logbook that basically talks about why a certain claim was inconclusive, verified, falsified, toy-verified. Each of these verdicts had points to be gained, with verified and falsified sitting at 2 points, toy-verified at 1, and inconclusive at 0.
Thanks for reading Navigating Noise! Subscribe for free to receive new posts and support my work.
There was one more catch. You can only push 20 spaces a day – It means if you started out later than when the competition debuted, you cannot possibly ever reach the top ranks. I started about 3 days later when there were roughly 70 or so participants.
When I first started, it was simply picking a few papers that I wanted to see it work on – with my ChatGPT subscription on Codex – and used up all of my limit in just about 5 reproductions. Ouch. What prompt to send in to set it out for this bold and courageous task? Fret not, Hugging Face had that bit sorted.
Add the paper name, fetch the guide and paste it in, the rest your agent should be able to do end to end. But!
If you read the guide, it tells the agent to publish the logbook as soon as it is done. This is not right. You don’t want this to become part of the subagent, but that is not wrong from Hugging Face’s part. This is more of a customisation.Once I had spent my Codex limit on about 5 or so reproductions, I needed to explore more ways – cheaper ways – to do this. I stumbled on OpenCode, ArtificialAnalysis, and more to optimise my methodology. After all, I wanted to test out my approach beyond the prompt they had given.
First Iteration:
The field of AI moves rapidly. At the time, the Artificial Analysis did not have Qwen 3.8 Max, nor did it have DeepSeek 0813 Pro, or Grok 4.6 for that matter. But now it does. At the time, DeepSeek did not even have the DeepSeek 0731 Flash model.
I set up GLM 5.2 via OpenRouter as an orchestration model, it is quite popular if you want multiple models API, has a free model router which automatically moves you from one model to another if you exhaust the limits of one free model. But it was not decent enough for the long term goal of surviving in this competition as the models would often die out midway and then the orchestrator would have to restart it only to find the subagent die again and too many corruptions within the artificats or logbooks would appear as you would scan. This is also why allowing for direct publish of the logbook was a bad idea, as once published, you exhaust from the 20 spaces a day limit. For the sub-agent models, if not already obvious, I initially set up free model router.
To run OpenRouter, I used OpenCode, probably thebest harness for most models out there. (proof attached below, not for all though)
Later, I bettered the approach. I used GLM 5.2 still for orchestration, but moved towards DeepSeek V4 Flash (not 0731 as it was not there yet). It was dirt cheap. You could essentially run it endlessly without ever straining your budget of even 10 dollars. But GLM 5.2 even in orchestration would still eat up a lot of that budget.
Break it down.
First, decide what papers you want to do. To do this, you must find what papers others exclusive of you have done and done well. Then, you must find a subset of that which can run on your resources, ideally on just a CPU. But soon enough, you will realise, this too will saturate as you pick up pace. How? You are not going to do just 20 and stop, you want to do as many as possible, and pick the best 20 locally, and then push it out.
Best? how are we deciding what is the best? Good question. You want to replicate their judges behaviour locally, and then have it rank the reproductions locally.
Then, I would repeat this process. Quite tiring – and could improve quite a lot by minimising my role as much as possible without degrading the output.
The wipeout scare
I had guests over the weekend while this competition was running, I could not really leave it as I had not automated it end-to-end, and for reference, my opencode.json was very basic and listed what sub-agents name was, what model to use, and what it was meant to do, that was all.
On the Sunday that they left, I decided after having entertained them over the past 3 days to take a deep sleep and let the agent run the reproduction as this part rarely required looking after – the only bits where I was fully involved was just passing the paths to the next prompt so it would do internal judging, and pushing or scheduling if pushing fails all through prompts but in seperate chat sessions and all of them followed the same structure, planning and designing by the orchestrator and the grunt work by deepseek v4 flash.
At 9PM, I woke up, sat down with a cup of tea and headed to Discord to play the game me and my friends and siblings were playing – Valheim – and Steam said the game does not exist. I waved it off and said eh might be some kind of a bug I will just install it again and play as this anomaly never occured – curse of a fast internet I suppose, you stop caring about a lot of things like download times :D.
Then, mid game, I get a pop up, my last days scheduled task had failed, I cross it out, head out back into the game, and then it pops up again for a different repro, and then again. I got suspicious. I was like wait that never happens. Mind you, all this time, the reproduction orchestration was still running.
I said to myself, hold up, whats going on here, why is it failing, and my worst nightmare had occured. What you only read about on subreddits or some other dark corner of the internet – isolated from all emotions and just another story on the internet from a stranger happening to a stranger. But no, this was me. How could this have happened? I checked my disk drive X, it was all empty. All my wedding photos were gone, I let out a scream. Wife comes out of the bedroom to see what the fuss is about. I tell her. She is shocked, tells me to get them back. I never had a backup. All of the projects are gone too, and so are the video games. But all of that could be brought back to life fairly easily – most of my personal hobby project work is synced with Github and games I could just redownload.
But wedding photos? Oh no.
I switch off TRIM right away – its a thing that basically tells the SSD that a block is empty and could be written on. Remember I said the reproduction was still working? Yes. It could have very well be overwriting on blocks that belonged to something else entirely. I stop it. I ask what the fuck happened here? It too is shocked.
After this, I spend the next 5 hours using one method after another to try and recover, but to no avail, most of what was recovered was corrupted, and I had lost a day or two within the competition as well.
I then asked the wedding photographers to please fetch them for me if they had copies, and luckily they did. I then said no more free roaming for the sub agents. We must control a bit more. This is important as far as OpenCode is concerned as it does not Sandbox like Codex or Claude Code.
I then rewrote the opencode.json to this:
{
"$schema": "https://opencode.ai/config.json",
"share": "disabled",
"snapshot": false,
"default_agent": "build",
"agent": {
"reconstruction-worker": {
"description": "Reconstructs exactly one assigned ICML reproduction workspace under D:\hf_challenge without deletion, publication, or nested agents.",
"mode": "all",
"model": "opencode-go/deepseek-v4-flash",
"prompt": "Work only in the absolute workspace assigned by the orchestrator. Read only that workspace and the explicitly named files under D:\hf_challenge\_shared. Never access C:, X:, or any path outside D:\hf_challenge. Never delete, move, clean, reset, truncate, or replace files or directories. Never invoke another agent. Never publish. Use additive files and targeted edits, run all six claim procedures independently, and return literal validator output.",
"permission": {
"read": "allow",
"edit": "allow",
"glob": "allow",
"grep": "allow",
"list": "allow",
"bash": {
"*": "allow",
"rm": "deny",
"rm *": "deny",
"rmdir": "deny",
"rmdir *": "deny",
"rd": "deny",
"rd *": "deny",
"del": "deny",
"del *": "deny",
"erase": "deny",
"erase *": "deny",
"Remove-Item *": "deny",
"Move-Item *": "deny",
"mv *": "deny",
"move *": "deny",
"git clean *": "deny",
"git reset *": "deny",
"git checkout *": "deny",
"cmd *": "deny",
"powershell *": "deny",
"pwsh *": "deny",
"python -c *": "deny",
"py -c *": "deny"
},
"task": "deny",
"external_directory": {
"*": "deny",
"D:\hf_challenge\_shared": "allow",
"D:\hf_challenge\_shared\*": "allow"
},
"todowrite": "allow",
"question": "deny",
"webfetch": "deny",
"websearch": "deny",
"skill": "deny"
}
},
"general-worker": {
"description": "Does what is commanded, not a fibre or inch more or less. Does not assume, asks the main agent if confused or vagueness felt",
"mode": "subagent",
"model": "opencode-go/deepseek-v4-flash",
"prompt": "Work only in the absolute workspace assigned by the orchestrator. Read only that workspace and the explicitly named files. Never access C:, X:, or any path outside D:\hf_challenge. Never delete, move, clean, reset, truncate, or replace files or directories. Never invoke another agent. Never publish. ",
"permission": {
"read": "allow",
"edit": "allow",
"glob": "allow",
"grep": "allow",
"list": "allow",
"bash": {
"*": "allow",
"rm": "deny",
"rm *": "deny",
"rmdir": "deny",
"rmdir *": "deny",
"rd": "deny",
"rd *": "deny",
"del": "deny",
"del *": "deny",
"erase": "deny",
"erase *": "deny",
"Remove-Item *": "deny",
"Move-Item *": "deny",
"mv *": "deny",
"move *": "deny",
"git clean *": "deny",
"git reset *": "deny",
"git checkout *": "deny",
"cmd *": "deny",
"powershell *": "deny",
"pwsh *": "deny",
"python -c *": "deny",
"py -c *": "deny"
},
"task": "deny",
"external_directory": {
"*": "deny",
"D:\hf_challenge\_shared": "allow",
"D:\hf_challenge\_shared\*": "allow"
},
"todowrite": "allow",
"question": "deny",
"webfetch": "deny",
"websearch": "deny",
"skill": "deny"
}
},
"discovery-worker": {
"description": "Audits exactly one assigned <=100-paper ICML discovery shard and writes one isolated candidate result JSON.",
"mode": "subagent",
"model": "opencode-go/deepseek-v4-flash",
"prompt": "Read D:\hf_challenge\discovery\discovery shard prompt.txt, then process only the absolute shard input and output paths assigned by the coordinator. Work only under D:\hf_challenge\discovery. Account for every input ORID exactly once. Write only the assigned shard result file. Never edit candidate_pool.json, shard_manifest.json, another shard, reproduction workspaces, shared controls, or Spaces. Never invoke another agent, run reproduction experiments, schedule jobs, publish, delete, or use a write token.",
"permission": {
"read": "allow",
"edit": "allow",
"glob": "allow",
"grep": "allow",
"list": "allow",
"bash": {
"*": "allow",
"rm *": "deny",
"rmdir *": "deny",
"del *": "deny",
"erase *": "deny",
"Remove-Item *": "deny",
"Move-Item *": "deny",
"mv *": "deny",
"git clean *": "deny",
"git reset *": "deny",
"git checkout *": "deny"
},
"task": "deny",
"external_directory": {
"*": "deny",
"D:\hf_challenge\discovery": "allow",
"D:\hf_challenge\discovery\*": "allow"
},
"todowrite": "allow",
"question": "deny",
"webfetch": "allow",
"websearch": "allow",
"skill": "deny"
}
},
"reconstruction-correction-worker-2": {
"description": "Audits exactly one assigned <=100-paper ICML discovery shard and writes one isolated candidate result JSON.",
"mode": "subagent",
"model": "openai/gpt-5.6-luna",
"prompt": "Never edit candidate_pool.json, shard_manifest.json, another shard, reproduction workspaces, shared controls, or Spaces. Never invoke another agent, schedule jobs, publish, delete, or use a write token.",
"permission": {
"read": "allow",
"edit": "allow",
"glob": "allow",
"grep": "allow",
"list": "allow",
"bash": {
"*": "allow",
"rm *": "deny",
"rmdir *": "deny",
"del *": "deny",
"erase *": "deny",
"Remove-Item *": "deny",
"Move-Item *": "deny",
"mv *": "deny",
"git clean *": "deny",
"git reset *": "deny",
"git checkout *": "deny"
},
"task": "deny",
"external_directory": {
"*": "deny",
"D:\hf_challenge\discovery": "allow",
"D:\hf_challenge\discovery\*": "allow"
},
"todowrite": "allow",
"question": "deny",
"webfetch": "allow",
"websearch": "allow",
"skill": "deny"
}
},
"reconstruction-correction-worker": {
"description": "Runs one fresh DeepSeek V4 Flash correction pass on exactly one already-authored ICML reproduction workspace.",
"mode": "all",
"model": "opencode-go/deepseek-v4-flash",
"prompt": "Correct exactly one absolute ICML reproduction workspace assigned by the orchestrator after its reconstruction-worker has terminated. Read that workspace, the explicitly supplied validator and evidence findings, and the named controls under D:\hf_challenge\_shared. Never work on a second ORID. Never resume or message another agent. Preserve raw evidence and downgrade unsupported verdicts; never fabricate, replace, or promote results. Make only targeted or additive corrections, rerun the exact validators, return literal output, then terminate. Never delete, publish, use a token, or invoke another agent.",
"permission": {
"read": "allow",
"edit": "allow",
"glob": "allow",
"grep": "allow",
"list": "allow",
"bash": {
"*": "allow",
"rm": "deny",
"rm *": "deny",
"rmdir": "deny",
"rmdir *": "deny",
"rd": "deny",
"rd *": "deny",
"del": "deny",
"del *": "deny",
"erase": "deny",
"erase *": "deny",
"Remove-Item *": "deny",
"Move-Item *": "deny",
"mv *": "deny",
"move *": "deny",
"git clean *": "deny",
"git reset *": "deny",
"git checkout *": "deny",
"cmd *": "deny",
"powershell *": "deny",
"pwsh *": "deny"
},
"task": "deny",
"external_directory": {
"*": "deny",
"D:\hf_challenge\_shared": "allow",
"D:\hf_challenge\_shared\*": "allow"
},
"todowrite": "allow",
"question": "deny",
"webfetch": "deny",
"websearch": "deny",
"skill": "deny"
}
},
"build": {
"description": "Built-in build agent",
"mode": "primary",
"disable": false
},
"plan": {
"description": "Built-in plan agent",
"mode": "primary",
"disable": false
}
}
}After this, no more scares happened. Phew. This basically takes away the ability to delete anything.
Now we move to the second iteration.
Second Iteration:
Before I continue any further, please head out and read Hugging Face’s own article on this reproduction challenge and it is a valuable read.
At this point, I had gotten tired of manually copy pasting prompts that I had sitting over to trigger one step after the other, and desired a more natural flow. First, break it down into steps:
- Discovery
- Batch Execution (multiple allowed in parallel, without duplicate reproductions)
- Publish or Schedule
This is essentially it.
my internal board for end-to-end repro challenge
It is fairly simple, if unclaimed candidates fall to 0, run discovery. Run batches as many as you wish in parallel, and after a regular interval of time, you will see eligible repros. Eligibility is decided by the internal judge.
This here shows what our internal judge scored, what the actual judge scored, and the difference (delta) between the two. Whenever the delta was higher, I would inquire and improve the internal judge to the point where the difference was at max 2 points away.
You would see the batches and you could go inside them to see what papers they took on and the internal judge score:
inside batch: paper view alongside internal judge delta
I also did not want to go OpenCode again and again to see progress, so each running batch would show what the logs were.
OpenCode logs within the dashboard
And that was it. Once this was perfected, I easily climbed and made gains to then continue battling out at the end of days at around position 7.
Had I started on day 1, and not had the wipeout scare, it would have been a mathematical possibility to reach the top 3, but I felt quite good doing this. And kudos to HuggingFace for pulling this off!
In their own words, extracted from this article they posted:
Questions about how reproducible AI research really is are older than the current AI wave. But these questions are exacerbated by scale. ICML 2026 received 23,918 submissions and accepted 6,352 papers, roughly double the previous year, continuing an exponential trend that is at least partly driven by AI agents making it faster to run experiments and write them up.
Reviewing capacity has not doubled along with it. Reviewers at most conferences are volunteers who may not have the time or expertise to fully review a paper. Here is a review of one accepted ICML 2026 spotlight paper, in the reviewer’s own words
There was another guy ‘agharsallah’ who wrote about their approach and because it was costing them too much, they resigned mid-competition where they were last seen at position 6.
Were all the agents always perfect in their work? absolutely not. Did you have to read logbooks every now and then to ensure it was not a joke? Yes, sometimes it would just botch it and do all toy-verifications and pretend that it was the best it could do.
I am also sharing my github for this UI and these prompts in case it helps someone reproduce some other competition or some other conference. Also, OpenResearch is a great harness. If I were to redo my participation, I would solely build around OpenResearch.
