More coming soon…
At submission time
TSimTools will enforce a strict reproducibility policy: all software used for the publication, including scripts for the production of graphics and the data sets will have to be submitted along with the paper or made available online on a public open access site (eg. sourceforge, Google Code, GitHub, …). In addition, permission to copy, archive, and redistribute (without profit) the submission material must be granted to the journal.
To be accepted for publication, a simulation software tool must meet the following requirements:
- Be available for download, fully open source, and at no charge for use in the production of published scientific results (1)
- Be redistributable (without profit) by the journal with no limitation
- Include a fully automated, self-contained installation procedure (no network connection required)
- Be available and ready for use on common execution platforms (2)
(1) Any restriction may still apply for the production of confidential results. (2) Software requiring a particular platform in terms of hardware or operating system will be reviewed using a specific protocol yet to be defined.
Long term reproducibility
Ensuring long term reproducibility is a technical challenge for which the journal has no revolutionary solution. However, we do intend to support long term reproducibility as much as possible as follows:
- Yearly check: the published material will be rechecked regularly, at least every year.
- Yearly snapshot: the authors willing to do so will be allowed to provide a new version of their software every year including bug fixes, additions, improvements or even new versions. However, all the previous versions will be kept available along with the new version and rechecked regularly as long as as possible.
Corner cases and allowance
Even when using fully open software, it may be extremely difficult or even impossible to reach a 100% reproducibility level. So far we have identified two main sources of difficulty:
- The software uses volatile resources, such as a web service or a real-time input
- The software submitted by the authors fails partially to pass reproducibility tests for some unexplained reason.
For such cases corner cases we must have some flexibility in order to find a reasonable trade-off. Here is our suggested allowance policy: