It's that time of year again for many in the machine learning community – the call for submissions to major conferences. And for those of us deeply invested in the foundational elements of ML research, the NeurIPS Datasets and Benchmarks track is a particularly exciting prospect. This isn't just about presenting a novel algorithm; it's about showcasing the very building blocks that enable that algorithm to shine: the data and the benchmarks that rigorously test its mettle.
I remember the early days of ML, where finding high-quality, well-documented datasets felt like a treasure hunt. The Datasets and Benchmarks track, now a well-established part of NeurIPS, directly addresses this. It's a dedicated venue for high-quality publications, talks, and posters that highlight valuable machine learning datasets and benchmarks. More than that, it fosters crucial discussions on how we can collectively improve the way we develop and share these essential resources.
What makes this track so important? Well, datasets and benchmarks are the lifeblood of ML progress. They allow us to compare methods fairly, understand limitations, and push the boundaries of what's possible. But, as anyone who's worked with them knows, they come with their own unique set of challenges, especially when it comes to reviewing. Unlike traditional papers, datasets often can't be fully anonymized for double-blind review. This is where the track's specific guidelines come into play, ensuring that reviewers can properly assess the data's origins, potential biases, and long-term accessibility.
The NeurIPS Datasets and Benchmarks track is also a strong proponent of the open-source movement. They actively encourage submissions of libraries and tools that make ML research more accessible and efficient. It’s inspiring to see how much progress has been made, and you can get a sense of the caliber of work by looking at the accepted papers and best paper award winners from previous years – 2021, 2022, and 2023 all have fantastic examples.
So, what are they looking for? The review process aims for the same rigor as the main conference, but tailored for datasets and benchmarks. A key criterion, and rightly so, is accessibility. Datasets need to be readily available – no hoops to jump through to get them – and any accompanying code should be open source. They even encourage the use of the Croissant format, which is a fantastic way to document datasets in a machine-readable way, making them easier to discover and use. Beyond the scientific paper, authors are expected to provide detailed supplementary materials covering data collection, ethical considerations, and maintenance plans. It’s a holistic approach to ensuring the longevity and responsible use of these valuable resources.
Submissions to this track are integrated into the main NeurIPS conference, meaning your work will be presented alongside other cutting-edge research and officially published in the proceedings. This year, there's a single deadline for submissions, so mark your calendars! While you can still submit to the main conference, remember that dual submission to both tracks isn't allowed. Authors have the flexibility to choose between single-blind or double-blind submission, depending on whether true anonymity is feasible for review. Importantly, papers won't be visible during the review period, and only accepted papers will be made public afterward. The review process itself is also kept private until after decisions are made.
The scope of this track is wonderfully broad, encompassing all aspects of data-centric ML research. This includes not only entirely new datasets but also thoughtfully curated collections of existing data, data generators, and reinforcement learning environments. They welcome work on data-centric AI methods and tools that improve data quality or utility, as well as studies that offer novel insights into data-centric AI. Even advanced data collection and curation practices that are generally applicable, even if the data itself can't be shared, are encouraged. Frameworks for responsible dataset development, audits of existing datasets, and identifying their problems are also within scope. And of course, benchmarks on new or existing datasets, along with benchmarking tools, are central. The track also looks for in-depth analyses of ML challenges and competitions that yield significant new insights, and systematic analyses of existing systems on novel datasets.
It’s a comprehensive approach, recognizing that the future of machine learning is as much about the data as it is about the algorithms. If you've been working on a valuable dataset, a robust benchmark, or innovative tools for data management, this track offers a prime opportunity to share your contributions with the global ML community.
