Home    Details    Workshop    Register    Download    Submit    Results    Organizers 

The MRBrainS challenge

Many algorithms for segmenting brain structures in MRI scans have been proposed over the years. Especially in such a well-established research area, there is a tremendous need for fair comparison of these methods with respect to accuracy and robustness. Publicly available databases like the ‘Alzheimer’s Disease Neuroimaging Initiative’ (ADNI) and the ‘Internet Brain Segmentation Repository’ (IBSR) are great initiatives and promote the use of publicly available data for automatic segmentation algorithm comparison, but either lack full manual segmentations (ADNI) or provide low-field (1.5T) single sequence (T1-weighted) MRI data (IBSR). Although there is an increasing awareness of the importance of comparing different algorithms on the same data, many methods are still compared to previous versions of the same type of algorithm on privately held data. This complicates the choice for a certain brain segmentation method among a wide variety of available methods. Furthermore, high-field (3T) multi-sequence MRI data is increasingly available in the clinic, while some available methods were developed for 1.5T single sequence T1-weighted MRI data. Comparing these methods to multi-sequence approaches on high-field (3T) data is of great interest to the medical image processing community.

The aim of this challenge is to compare (semi-)automatic algorithms for segmentation of gray matter, white matter and cerebrospinal fluid on 3T MRI scans of the brain. Since participants to the challenge apply their own methods to the data, there is no bias toward one particular method, and parameters are optimally tuned to achieve the best possible performance. For the challenge, twenty fully annotated multi-sequence (T1-weighted, T1-weighted inversion recovery and FLAIR) 3T MRI brain scans are available. More information on the data can be found here. We encourage both authors of new methods as well as authors of already published brain segmentation methods to participate in the challenge and tune their algorithms in the best possible way on the training data, to enable a fair comparison of the methods. We welcome both multi- and single-sequence (only T1-weighted scan) approaches.

If you wish to participate in the MRBrainS challenge, read the details on this page and follow the steps below:

1) Register on this website, sign the MRBrainS13 confidentiality agreement and e-mail it to mrbrains13@isi.uu.nl. After you have received a confirmation e-mail, you can download the data using your team name and password.
2) Download the 5 training datasets and 15 test datasets on the download page.
3) Use the training data to fine-tune your segmentation algorithm.
4) Apply your segmentation algorithm to the test datasets.
5) Submit your results in the format described here together with a description of your method, your evaluation results will be sent to you by e-mail, and will be published on this website in the results section.

Segmentation Task and Data

We kept the segmentation task as simple as possible, to allow more algorithms to participate. Participants should segment gray matter, white matter and cerebrospinal fluid. Algorithms that are able to segment substructures, such as cortical gray matter and basal ganglia, should merge these results and label them as gray matter. This is described in further detail in the Test data section below. In the manual segmentations provided for training, the substructures are labeled separately to allow algorithms that segment substructures to use these labels. Authors with algorithms that only use labels for gray matter, white matter and cerebrospinal fluid, should merge these labels themselves. This is described in further detail in the Training data section below. Both multi-sequence as well as single-sequence (using only the T1-weighted scan) approaches are welcome to participate. We will distinguish between these approaches in the results section.

Twenty fully annotated multi-sequence (T1-weighted, T1-weighted inversion recovery and FLAIR) 3T MRI brain scans are available. These scans have been acquired at the UMC Utrecht (the Netherlands) of patients with diabetes and matched controls (with increased cardiovascular risk) with varying degrees of atrophy and white matter lesions (age > 50).

For each patient, the following sequences are provided:

  1. T1_1mm: 3D T1-weighted scan (voxel size: 1.0mm x 1.0mm x 1.0mm)
  2. T1: 3D T1-weighted scan registered to the T2 FLAIR (voxel size: 0.958mm x 0.958mm x 3.0mm)
  3. T1_IR: Multi-slice T1-weighted inversion recovery scan registered to the T2 FLAIR (voxel size: 0.958mm x 0.958mm x 3.0mm)
  4. T2_FLAIR: Multi-slice FLAIR scan (voxel size: 0.958mm x 0.958mm x 3.0mm)
All scans are bias corrected, and scans 2-4 are aligned. This is done to eliminate the influence of different registration and bias correction algorithms on the segmentation results. The manual labeling is done on scans 2-4. The 3D T1-weighted scan is provided for algorithms that prefer to use 3D high-resolution data. However, the final results of your segmentation method should be aligned with scan 2-4, since the ground truth is only available for these scans! The face of the patient is cut out of the 3D T1-weighted scan for the purpose of anonymization. Some of the T1-weighted IR images contain artifacts at the bottom of the scan. These artifacts occur often in clinical scans, and it is therefore interesting to know how automatic segmentation methods that use these scans perform under these circumstances.

Training data

Five datasets are provided with manual segmentations to use as training data to tune your segmentation algorithm. Manual segmentations (ground truth) were drawn on the thick-slice scans (3mm slice thickness), using an in-house developed manual segmentation tool based on the contour segmentation objects (CSO) tool available in Mevislab. A freehand spline drawing technique was used to segment all structures in the brain. The outline of each structure was delineated, starting with the innermost structures. By iteratively subtracting delineations to create holes, binary images were created for each structure. Segmentations were performed in a darkened room with optimal viewing conditions. All segmentations were inspected for correctness by an expert not involved in the segmentation procedure and corrections were made if needed. A third expert approved all final segmentations.

The following structures are manually segmented and will be available for training:

  1. Cortical gray matter
  2. Basal ganglia
  3. White matter
  4. White matter lesions
  5. Cerebrospinal fluid in the extracerebral space
  6. Ventricles
  7. Cerebellum
  8. Brainstem
The numbers in front of the structures indicate the labels in the ground truth image. Background will be labeled as 0. When your algorithm only uses gray matter, white matter and cerebrospinal fluid labels, you should merge labels 1 and 2, 3 and 4, and 5 and 6 yourself.

Notes on the manual segmentations

  • White matter lesions were segmented on the FLAIR scan.
  • The outer border of the CSF was segmented using both the T1-weighted scan and the T1-weighted IR scan.
  • All other structures were segmented on the T1-weighted scan (0.958mm x 0.958mm x 3.0mm).
  • Vessels were not segmented separately. The CSF segmentation therefore also includes the superior sagittal sinus and transverse sinuses.
  • The cerebral falx is also included in the CSF segmentation.

Test data

The remaining 15 scans are provided as test data. Only the scans are provided, not the manual segmentations. Authors can submit the segmentation results of their algorithms, after which the evaluation results will be sent to them by e-mail. The results will be published on this website in the results section. Segmented tissue should be labeled as follows:
  1. Background (everything outside the brain)
  2. Cerebrospinal fluid (including ventricles)
  3. Gray matter (cortical gray matter and basal ganglia)
  4. White matter (including white matter lesions)

Your submission should be one image with the voxels labeled as described above. If your algorithm segments substructures (e.g. basal ganglia or ventricles), you should assign them to one of the above classes (1-3). The brainstem and cerebellum will be excluded from the evaluation. Your submission should be written in the ITK MetaImage format (mhd and raw files) or Nifti format (*.nii for matlab users). The ITK MetaImage format consists of a text file with "mhd" extension that contains the ASCII header information and a binary file with "raw" extension that contains the voxel data.

Please note that your result file should have the thick slice resolution (voxel size: 0.958mm x 0.958mm x 3.0mm), since the ground truth is only available for the thick slice data!

Evaluation

Submitted segmentation results will be compared to the manually obtained reference standard. For each tissue type (gray matter, white matter and cerebrospinal fluid), the Dice coefficient (DC), the 95th-percentile of the Hausdorff distance (HD-95) and the absolute volume difference (AVD) will be calculated. The final ranking is based on the evaluation results of all 15 test datasets and is determined as follows: For each evaluation measure (DC, HD-95, AVD), the mean value over all 15 datasets is determined for white matter (WM), gray matter (GM) and cerebrospinal fluid (CSF). Each team receives a rank (1=best) for each tissue type (GM, WM, CSF) and each evaluation measure (DC, HD-95, AVD) based on the mean value of the evaluation measures over all 15 datasets. The final score is determined by adding the ranks of all tissue types and evaluation measures for each team. The team with the lowest score will be ranked number 1. In case two teams would have an equal score, the team with the lowest standard deviation over the tissue types will be ranked number 1. Results will be presented on the results page.

Terms of Participation

The collection of the data, the organization of the MRBrainS challenge and the maintenance of this website require a large effort. In the spirit of cooperative scientific progress, we are committed to maintaining this site as a public repository of benchmark results for segmentation of gray matter, white matter and cerebrospinal fluid in MR brain images. However, we ask everyone who uses this site to respect the "Terms of Participation" described below.

These terms amount to a simple tit for tat: we actively encourage anyone to use the data for testing brain segmentation algorithms. In return, we ask you to submit the results of your method and send us a document that describes your method. The score of your algorithm and your description will be made publicly available in the results section of this website.

We do not claim any ownership or rights to the algorithms or uploaded documents, and do not want to create any obstacles for publishing methods that use the MRBrainS data. However, by participating in the MRBrainS13 challenge (registering a team and downloading the data), you acknowledge that you have read, understood, and agree to be bound by the following "Terms of Participation":

  • The downloaded data sets or any data derived from these data sets, may not be given or redistributed under any circumstances to persons not belonging to the registered team.
  • All information entered when registering a team, including the name of the contact person, the affiliation (institute, organization or company the team's contact person works for) and the e-mail address must be complete and correct. In other words, anonymous registration is not allowed. If you wish to submit anonymously, for example because you want to submit your results to a conference that requires anonymous submission, please contact mrbrains13@isi.uu.nl first.
  • Data downloaded from this website may only be used for the purpose of preparing an entry to be submitted on the MRBrainS website. The data may not be used for other purposes in scientific studies and may not be used to train or develop other algorithms, including but not limited to algorithms used in commercial products.
  • Results of your submission will only be published on the website when a document describing the method is provided.
  • If a commercial system is evaluated no method description is necessary, but the system has to be publicly available and the exact name and version number have to be provided.
  • The organizers of the challenge will check the method description before publishing the results on the website.
  • Evaluation of the results uploaded to this website will be made publicly available on this site (see the Results section), and by submitting results, you grant us permission to publish our evaluation. Participating teams maintain full ownership and rights to their method.
  • If the results of algorithms in this challenge are to be used in scientific publications (journal publications, conference papers, technical reports, presentations at conferences and meetings) you must make an appropriate citation. Currently, this citation will refer to the MRBrainS webpage (http://mrbrains13.isi.uu.nl), and later to the publication that will describe the results of the MICCAI MRBrainS13 challenge workshop.
  • Teams must notify the organizers of MRBrainS13 about any publication that is (partly) based on the results data published on this site, in order for us to maintain a list of publications associated with the challenge.

© 2013 Image Sciences Institute.