Validation Tool Manual

Validation Tool Manual

Crash course to get started and exemplary reproduce results from the paper.

The Validation Tool is implemented as a plugin within the Medical Imaging and Interaction Toolkit1[1,2]. It allows you to validate your own stereo reconstruction algorithm using the data sets acquired in a recent comparative validation study [3]. The following paragraphs describe how to use the data sets with your own reconstruction algorithm, how to save the results to allow performance analysis with the provided validation tool and how to use the validation tool.

Applying your stereo algorithm

After downloading the open data zip files here, you will find folders Stereo_SD_d_ [all|distance|angle|smoke|blood] whose names indicate which validation criterion they represent (cf. [3]) . Each of these folders contains subfolders, one for each stereo pair ordered by an ID, as explained in detail here. The data sets relevant for your reconstruction are the following:

  1. Stereo-SD_<ds>-<nb>_IMG-left.bmp and Stereo_SD_<ds>_<nb>_IMG-right.bmp: the original stereo image pair

  2. Stereo_SD_<ds>_<nb>_IMG_REC_left.bmp and Stereo_SD_<ds>_<nb>_IMG_REC_right.bmp: the stereo image pair like in (a), but rectified, if your stereo algorithm works on rectified images

  3. Stereo_SD_<ds>_<nb>_MASK_left.png and Stereo-SD_<ds>_<nb>_MASK-right.png: binary masks that indicate the regions to be excluded during correspondence search (mainly the colored markers)

  4. Stereo-SD_<ds>_<nb>_MASK_REC_left.png and Stereo_SD_<ds>_<nb>_MASK-REC_right.png: masks like in (c), but recitified, if your stereo algorithm works on rectified images

  5. Stereo_SD_d_<criteria>_<nb>_Calibration.txt: contains the calibration parameters for every dataset in a format described here.

 

Saving your stereo results

When applying your stereo algorithm on the datasets, the resulting 3D reconstruction should be saved in the following format:

  • Each 3D point, which can be reconstructed from a 2D-pixel, should be saved as one line in a textfile: 2D_Pixel_x 2D-Pixel_y 3D-coordinate_x 3D-coordinate_y 3D-coordinate_z

  • If a pixel cannot be reconstructed, just leave it out.

  • For a comparable result, there should be no interpolation, upsampling, etc.

  • The 2D-coordinates can either be in the original image coordinate system or in the rectified image coordinate system (en-/disable checkbox, see Fig. 2 and 3).

  • Each reconstruction should then be saved as text file as Stereo_SD_<ds>_<nb>_result.xyz.

  • It can either be stored in the respective subfolder that contains all single files or in a separate folder specified by the user (see Fig. 3).

Example reconstructions can be downloaded here.
 


Figure 1: Icon of the Reconstruction Validation Tool

Figure 2: Screenshot of validation tool implemented in MITK [1,2]

 

Using the validation tool

The validation tool is started by clicking on the MBIApp executable. To start a validation the user needs to open the reconstruction validation plugin (Fig. 2) by clicking on the icon of the plugin (Fig. 1). After opening the plugin the user has to select a folder, where the reference data is stored (first “Choose Folder” button, see Fig. 3) as well as where the reconstructed stereo results are stored (second “Choose Folder” button, see Fig. 3).  The folder with reference data  is defined to be one of the downloaded data sets  Stereo_SD_d_ [all|distance|angle|smoke|blood] with all its subfolders and files as described here. Also, it is possible to select a parent directory containing an arbitrary number of these data sets. The folder with the reconstruction results has to contain the *.xyz files, named as described above. Make sure, that the validation criterion (all, angle, blood, etc.) of reference and reconstruction data is the same. In case each stereo reconstruction result was stored in the corresponding reference folder, the second input box can be left out.

If you want to save your reconstructions as surfaces in stl-format, check the first check box (see Fig. 3). The surface files will be stored in the respective subfolders as Stereo-SD_<ds>-<nb>_surface.stl. The second check box indicates, if the 2D coordinates refer to the original or rectified image coordinate system (see above).

Once the "Run Validation" button is pressed, a progress bar on the bottom of the program indicates the current state of the validation process. Also have a look at the console window in the background to see the progress and outputs of the validation algorithm. Depending on the number of data sets to be processed this might take a while. As soon as the progress bar and the hourglass mouse pointer disappear the validation is completed and the results are stored in Stereo_SD_<ds>.cvs in the reference folder. The *.csv files can be opened by any text editor or spreadsheet program.

Figure 3: Reconstruction Validation Tool plugin and options.

Interpreting the results

The reconstruction validation tool creates a *.csv file for each validated data set in the working directory containing the following metrics for each 3D reconstruction (represented by one line):

  • Mean

  • Standard deviation

  • RMS

  • Median

  • Maximum error

  • Minimum error

  • Number of reconstructed points

  • Surface coverage with radius 3 mm as defined in [3]

Each *.csv file also provides descriptive statistics for the Median and RMS error as well as for the number of points reconstructed and the surface coverage.

Downloads

Validation Tool for Windows

Validation Tool for Linux

References

[1] Wolf, Ivo, et al. The medical imaging interaction toolkit. Med Imag Anal, 2005, 9. Jg., Nr. 6, S. 594-604.

[2] Nolden, Marco, et al. The medical imaging interaction toolkit: challenges and advances. Int J Cars, 2013, 8. Jg., Nr. 4, S. 607-620.

[3] Maier-Hein L, Groch A, Bartoli A, Bodenstedt S, Boissonnat G, Chang PL, Clancy NT, Elson DS, Haase S, Heim E, Hornegger J, Jannin P, Kenngott H, Kilgus T, Müller-Stich B, Oladokun D, Röhl S, Dos Santos TR, Schlemmer HP, Seitel A, Speidel S, Wagner M, Stoyanov D. Comparative Validation of Single-shot Optical Techniques for Laparoscopic 3D Surface Reconstruction. IEEE T Med Imag (in press), 2014