Providing Performance Information about Tools and Resources
Since its inception, NITRC has provided answers to basic questions to help with the selection of neuroimaging software: What does this tool do? What platforms does it run on? What license is it released under? NITRC now takes a step further by providing performance information about tools: How long does it take to run? How well does it perform a given function? Exposing this information before software is downloaded and installed, let alone run, will save time when selecting a tool for a given analysis and will provide insights on the best tool for a task.
NITRC now reports performance data for image segmentation. For segmentation tools, we report run time, accuracy compared to Internet Brain Segmentation Repository data, and test-retest stability using Brain Segmentation Testing Protocol data. Sample images of segmentation results are also provided for qualitative analysis of results.
While much of this data can be used to compare tools, care should be taken not to jump from narrowly defined metrics to broad judgements of quality. Indeed, most metrics, even if they can be expressed quantitatively, do not map simply to quality. By the same token, these metrics express nuanced functional performance that can help distinguish tools and select the best option for a given analysis.
Find performance data by following the Performance Data menu item at the left on a tool's main page. (See, for example, FreeSurfer.) Stay tuned as we expand performance data to other functions in the future and send any feedback you have to moderator@nitrc.org.
— Christian Haselgrove, NITRC Co-PI
