Written and maintained byCASRAI Editorial Board
Target audience and objective: This guide is for machine learning researchers, computational scientists, and research software engineers seeking to locate open- benchmark data, and navigate state-of-the-art model evaluations
What Papers with Code Is
Papers with Code (commonly styled as paperswithcode) is a semi-automated indexing service designed to link scholarly machine learning publications with their corresponding source code repositories, datasets, and empirical evaluation tables. In scientific fields driven by machine learning, peer-reviewed articles frequently describe complex neural architectures without embedding runnable scripts or training routines. Papers with Code addresses this reproducibility gap by connecting preprints—predominantly hosted on arXiv—to public repositories hosted on GitHub, GitLab, and other version-control platforms.
Beyond simple repository linkage, the platform organized machine learning research into an interconnected graph of tasks, methods, and evaluation leaderboards. When researchers evaluate models in machine learning and artificial intelligence, identifying which model achieves state-of-the-art (SOTA) performance across standard benchmark suites requires tracking metrics across dozens of competing papers. Papers with Code formalized this tracking by curating benchmark tables that paired specific tasks (such as image classification, semantic segmentation, or natural language inference) with published metric scores, dataset splits, and verified code repositories.
Who Runs It and What It Covers
Papers with Code was founded in July 2018 by Robert Stojnic and Ross Taylor through Atlas ML (Deeptide Ltd.), an independent startup based in London. In December 2019, Facebook AI (subsequently Meta AI) acquired Atlas ML and assumed stewardship of the platform. Under Meta, Papers with Code remained a freely accessible public utility for the global research community, expanding its catalog to cover hundreds of thousands of papers, tens of thousands of code repositories, and thousands of benchmark evaluation leaderboards.
On July 24, 2025, Meta retired the standalone web portal and sunsetted the active service. The primary domain (paperswithcode.com) was configured with an HTTP 302 redirect routing incoming web traffic directly to the Trending Papers service on Hugging Face (https://huggingface.co/papers/trending). Despite the discontinuation of the live submission interface, Meta and the original development team preserved the entire structured knowledge graph as an open-access archival asset.
The coverage of the platform spans several core research artefacts, which remain publicly distributed under the Creative Commons Attribution-ShareAlike 4.0 International (CC-BY-SA 4.0) license
- Papers and Abstracts: Bibliographic records detailing title, authors, abstracts, publication dates, and persistent identifiers (primarily arXiv IDs).
- Paper-to-Code Links: Directed mappings between academic publications and verified open-source software repositories, tracking repository stars and framework implementations (such as PyTorch, TensorFlow, and JAX).
- Evaluation Tables: Structured benchmark results pairing models, datasets, metric definitions (such as BLEU, Top-1 Accuracy, or F1 Score), and observed numerical values.
- Methods Ontology: Hierarchical classifications of machine learning mechanisms, including attention mechanisms, convolutional layers, optimizers, and loss functions.
- Datasets: Curated catalogs of benchmark datasets, documentation links, and usage tasks.
How to Search or Use It
Because the original web interface no longer accepts interactive user queries, researchers working with Papers with Code utilize its open data dumps and successor endpoints. To inspect historical machine learning benchmarks and code links, researchers follow these documented procedural steps:
- Clone or Download the Archival Repository: Access the official dataset repository at
https://github.com/paperswithcode/paperswithcode-dataor the Hugging Face community mirror (pwc-archive). - Locate Code Link Records: Uncompress
links-between-paper-and-code.json. Each JSON object contains the fieldspaper_title,paper_url,paper_arxiv_id, andrepo_urlto locate official and community code implementations. - Extract Structured Evaluation Metrics: Inspect
evaluation-tables.jsonto access historical performance tables. Records are structured using thesota-extractorontology, detailing task names, dataset identifiers, evaluation metrics, and verified scores. - Query Methods and Architectural Taxonomies: Ingest
methods.jsonto trace how specific neural network architectures or mathematical functions relate to parent concepts and pioneer papers. - Incorporate Formal Software Metadata: When developing machine learning packages based on discovered implementations, adopt open scholarly software standards such as Citation File Format (CITATION.cff) or structured schema descriptions like CodeMeta to ensure proper attribution.
- Consult Open-Access Software Journals: When validating research software engineering pipelines, consult dedicated publication venues such as the Journal of Open Source Software (JOSS) for formal peer review of software artefacts.
- Navigate Live Successor Feeds: For newly published papers, query active community tracking platforms such as Hugging Face Trending Papers to discover code checkpoints, community comments, and linked model weights.
Strengths and Limits
Understanding the capabilities and structural boundaries of Papers with Code is critical for researchers building bibliometric pipelines or designing reproducible computational workflows.
Key Strengths
- Advancement of FAIR Principles: By standardizing paper-code pairings, the platform significantly enhanced the Findability, Accessibility, and Reusability of machine learning software artefacts.
- Permissive Open Licensing: Releasing the complete structured dataset under CC-BY-SA 4.0 ensures that academic institutions, independent researchers, and third-party developers can reuse the catalog without commercial restrictions.
- Ontological Structure: Organizing models around specific evaluation tasks provided machine-readable benchmark comparisons that surpassed unstructured PDF tables.
Technical and Operational Limitations
- Platform Deprecation: Following Meta’s July 2025 shutdown, the live platform no longer accepts automated pull requests, community submissions, or real-time leaderboard updates on its original domain.
- Taxonomic Overlap: In academic evaluations of the platform (such as studies published in the CEUR Workshop Proceedings), researchers identified substantial semantic overlap between task categories. Broad or ambiguous category boundaries occasionally reduce retrieval precision in multi-task benchmarks.
- Self-Reporting and Verification Gaps: While automated scrapers extracted scores, many leaderboard entries relied on author-reported performance metrics without centralized re-computation or containerized execution checks.
- Repository Drift and Bitrot: Code links point to external GitHub repositories that may suffer from unpinned dependencies, deprecated CUDA toolchains, or deleted codebases over time.
Alternatives
Researchers seeking active state-of-the-art tracking, model leaderboards, and code indexing can select from several modern alternatives designed for contemporary machine learning workflows:
| Platform | Primary Focus | Code Linking | Benchmark Tracking | Governance and Access |
|---|---|---|---|---|
| Hugging Face Papers | Trending AI literature, model weights, and datasets | Direct integration with GitHub, Hub repositories, and Spaces | Integrated community leaderboards (e.g., Open LLM Leaderboard, MTEB) | Commercial platform with open-access community features |
| CodeSOTA | Task-oriented capability taxonomies and verified benchmarks | Automated repository mapping | Verified benchmark metrics with strict date and task tracking | Community-maintained open-science initiative |
| OpenCodePapers | Git-native paper and code aggregation | Direct repository integration | Collaborative benchmark submission via pull requests | Open-source community project |
| LMSYS Chatbot Arena | Crowdsourced human preference evaluation for LLMs | Secondary (links to model weights and API endpoints) | Elo-rated head-to-head pairwise comparative leaderboards | Academic consortium (UC Berkeley, UCSD, CMU) |
| LiveBench | Contamination-free automated evaluation suites | Public evaluation harnesses on GitHub | Continuous testing on unreleased reasoning and coding tasks | Academic research collaboration |
Frequently Asked Questions
What happened to paperswithcode.com?
On July 24, 2025, Meta retired the standalone Papers with Code website and implemented an HTTP 302 redirect directing domain visitors to the Hugging Face Trending Papers feed. The historical dataset remains accessible
Where can researchers download the Papers with Code dataset?
The complete historical data dumps are hosted in the public GitHub repository paperswithcode/paperswithcode-data and mirrored on the Hugging Face organization pwc-archive. Available files include papers, abstracts, code repository links, methods, and evaluation tables.
What license governs the Papers with Code data dumps?
All data dumps published by Papers with Code are released under the Creative Commons Attribution-ShareAlike 4.0 International license (CC-BY-SA 4.0). Users are free to share and adapt the material for any purpose, provided appropriate attribution is given and derivatives are distributed under the same license.
What is the most effective Papers with Code alternative for current research?
For daily literature tracking and model checkpoint integration, Hugging Face Papers serves as the primary modern ecosystem. For standardized LLM evaluations, platforms like the LMSYS Chatbot Arena and LiveBench provide contemporary benchmark leaderboards.
Sources
- GitHub Repository (read 2026-11-07): Papers with Code Datasets and Documentation. Available at:
https://github.com/paperswithcode/paperswithcode-data - CEUR Workshop Proceedings (read 2026-11-07): Jenifer Tabita Ciuciu-Kiss and Daniel Garijo. A Study of the Categories used in ‘Papers with Code’. CEUR-WS.org, Vol-3453. Available at:
http://ceur-ws.org/Vol-3453/paper7.pdf - Hugging Face (read 2026-11-07): Hugging Face Trending Papers and Redirect Service. Available at:
https://huggingface.co/papers/trending
Research-administration guidance, standards updates and independent tool reviews.
