New for SC21
The SC Best Reproducibility Advancement Award aims to recognize outstanding efforts in improving transparency and reproducibility of methods for high performance computing, storage, networking and analysis. This award will recognize the paper or a research object of a paper that most significantly advances the state of art in transparency and reproducibility.
In this context, a research object refers to a collection of computational artifacts that extend beyond the submitted paper, such as software, datasets, environment configuration, mechanized proofs, benchmarks, or test suites with scripts.
Goals
This award aims to encourage transparency and reproducibility and to reward outstanding efforts that broadly fall under the following scope:
- Improvements in transparency and reproducibility of research in HPC, e.g., via tools or methods
- Efforts to build transparent and reproducible research objects
- Experimental design methodologies for verifying and future-proofing scientific claims and conclusions
- Generation and publication of open datasets/software of relevance to the SC community
- General advancements in transparency and reproducibility
Criteria
The award will be given based on the advancements in transparency and reproducibility and/or extraordinary exemplars of reproducible research in a paper or research object. Typical examples include:
- Paper
- Developing novel tools and methodologies to assist transparency and reproducibility
- Demonstrating and studying transparency and reproducibility at SC
- Enabling transparency and reproducibility in a subfield at SC with barriers for transparency and reproducibility
- Research Objects of a Paper
- Showcasing best practices
- Open sourcing research objects
- Packaging research objects in a novel/particularly helpful way
- Building research objects tailored to education
- Automating reproduction
- Adapting research objects for alternative experiment environments
- Including original measurements and information about the experiments
Nomination Process
All SC papers and research objects are eligible for this award and can be nominated by the SC Technical Paper Committee or SC Artifact Description/Evaluation Committee as part of the usual review process. Each submission will be given the opportunity to highlight their advancement in transparency and reproducibility as part of their submission form. This information would then be considered during the selection process.
Selection Process
The Award Selection Committee includes representatives from both the SC Reproducibility Initiative and the SC Technical Paper Committee. The selection committee only considers accepted papers and research objects that have been nominated for the award (see Nomination Process above) and bases its evaluations on the award criteria stated above. In principle, nominations can include papers that did not apply for badges but that nevertheless represent an important resource enabling reproducibility.