ITK/Roadmap 2005 2006

Introduction

The future developement of ITK relies heavily on feedback from the community of users and developers as well as guidelines from the clinical and medical research community.

This page is intended to gather feedback regarding the future direction of ITK and in particular to list specific features and functionalities that would make the toolkit more useful for the medical imaging community.

Users/Developers Support

This section relates to the services that will help beginners to start with ITK as well as material that will help experienced users to take better advantage of the functionalities available in the Insight toolkit.

Tutorials

Conferences

Tutorials have been typically presented in conferences related to Medical Imaging

• MICCAI
• SPIE Medical Imaging
• IEEE Visualization
• ISMRM
• ISBI

Advanced Tutorials

The tutorials so far have been focused on Introductory material. The typical introductory tutorial is presented in 8 hours and does not allow for hands-on experience with the segmentation and registration methods. Now that the toolkit has a considerable user-base it seems appropriate to develop tutorials in more advanced topics and present those tutorials at conferences. Advanced tutorial could cover topics such as

• How to write an ITK filter
  • How to deal with Traits and Multi-platform
  • How to enjoy Multi-threading
• Advanced registration
  • Fine tuning parameters
  • Profiling metrics
  • Improving performance
  • Specific clinical uses
• Advanced segmentation
  • Fine tuning parameters
  • Specific clinical uses

• Statistics

• Spatial Objects
  • How to use them
  • How to extend them

• Level Sets
  • creating new level set methods

• Registration
  • How to write new metrics
  • How to write new transforms
  • How to write new optimizers

Workshops

Workshops allow for a more interactive experience than tutorials. In the context of a workshop, attendants are expected to contribute material and share it with colleagues. The framework of a workshop could provide a more effective atmosphere for discussing the addition of new classes to the toolkit and for improving the implementation of existing ones.

Documentation

Documentation for ITK is provided at several levels. It includes the user's manual level provided by the ITK Software Guide, the reference manual level as provided by Doxygen and the community forum as provided by the Wiki.

Doxygen

• How to make Doxygen a more effective tool for assisting users.
• Identify classes with poor documentation level and take correction action.
• Increase the crosslinking of Doxygen with the software guide.
• Improve description of parameters
• Clean up documentation as part of the release process.
• namespaces: Functors/Function

Software Guide

• Adding more 3D Examples
• Adding some 4D Examples
• Adding more links to papers
• Add a section on the inner-workings of the pipeline.
• Add the neural nets
• Transforms IO
• more DICOM examples
  • How to generate a valid DICOM
  • Read/Write/Anonymize DICOM
• Add the Archetype reader
• How to combine ITK and VTK (related to ISC efforts)
• Section on how to integrate ITK with other applications.
  • Matlab
  • Analyze
  • SCIRun
  • VolView

• How to submit to the Insight Journal

WIKI

• Add a Google search button (like NAMIC)
• How to make the wiki more effective tool for the community
• Flag particular sections of the Wiki that may have become outdated and correct them
• Reduce the duplication of information (e.g. ITK installation is explained in the tutorials material, the software guide, the wiki and the HTML pages).
• Use stories of ITK (news?)
• Gallery of ITK results.
• List of publications that cite ITK.

The Insight Journal

The Insight Journal will come up online very soon.

We anticipate it to be the standard mechanism for

• Bringing new algorithms into ITK.
• Evaluating existing algorithms.
• Sharing experiences and "how-to" knowledge among the community.

In particular, the Insight Journal will become a repository on specific recipies on how to process specific types of data sets. This will bring ITK a layer above of the plain image processing and into the more domain specific area of Medical Image Analysis.

Extending to new Wrapping approach

• Check on Matlab integration
• Check on C++ non-templated layer

ITK-Pyhton

• Creating Python wrapper files for existing and future ITK filters
• Implementation of ITK filters in Python
• Embedding Python in applications for calling ITK classes

Features and Functionalities

Software Engineering

Categorize expectations on PixelType API from every filter

Filters should be classified according to the pixel types that they can manage

• Scalar
• Vector
• Tensor

The functionalities of the pixels should include, for example, being "additive", supporting "product by scalar", and so on.

This categorization should encompass two aspects

1. Documentation
2. Verification at compilation time, using concept-checking-like error messages

Making VNL Thread-Safe:

• Maybe by removing Static Variables from vnl

Explicit Template Instantiations

Most of ITK is provided as class templates. User code must instantiate these templates in order to use them. As a result very little of ITK is actually compiled when ITK itself is built. Only when testing, wrapping, or application code is compiled do the ITK class templates actually get instantiated and compiled. This leads to excessive compilation times for this code because every source that is compiled includes the implementation code of ITK for every class template it uses and everything those templates need to compile. It also leads to translation units that are too large to compile on certain compilers.

The motivation for using templates in the original design was to allow users to work with any kind of data. This functionality has been successfully provided, but most users' needs can be met by a few common instantiations of ITK's class templates. We plan to provide common instantiations in the ITK libraries themselves. It will then be possible for user code to use these instantiations by including only the interface (.h) files and not the implementation (.txx) files.

If we provide an explicit instantiation for every template currently instantiated by testing, wrapping, and common application code the following benefits will be reaped:

• Reduced compilation times for sources outside ITK libraries
• Reduced wrapper module sizes
• Reduced executable sizes when using shared libraries
• Increased testing of compilation of ITK templates

and the following drawbacks will be met:

• Increased compilation time of ITK libraries themselves (though total time for ITK+application code will still be reduced)
• Increased complexity in ITK sources to specify instantiations (similar to current wrapping specification)

Typeless layer

A C++ typeless layer will provide users with a toolkit where the pixel type and dimension of the image do not have to be decided in advance. This layer will look very close to what VTK uses for representing images and imaging filters.

Increasing Code Coverage

Code coverage is low for a significan number of files. It is usually correlated with the less used classes in the toolkit

Add support to CTest for reducing redundant tests

CTest could provide coverage of classes per test in order to determine if a particular class is being covered in multiple tests. From this analysis, developers could remove redundant testing in the toolkit.

Explore other tools for code coverage in CTest

Currently we use gcov but this tool have some limitations. It will be desirable to have an alternative way of measuring code coverage.

CTest will have to parse the output of these alternative tools and format it according to what Dart/Dart2 expects as input.

Potential Candidates

• Valgrind

 Valgrind is a good candidate since it has so many different tools/skin. It allow profile on memory usage, do profiling of code.. (see addrcheck, cachegrind, corecheck, lackey, massif, memcheck )

• Visual Studio
• http://c2.com/cgi/wiki?CodeCoverageTools
• http://www.testing.com/writings/coverage-terminology.html

Increasing Valgrind/Purify Testing

Testing for run-time problems has received reduced attention. As ITK move more and more into applications, we should pay closer attention to the run-time correctness of the code. We should setup more machines for running Valgrind/Purify testing

Since these builds take longer, we could take advantage here of Dart2 new alternative periodicity. For example, run these tests on a weekly or montly basis and then enter in sessions for cleaning up the code.

Add support to CTest for runtime profiling

Improving run-timeperformance of ITK algorithem requires to analyze the amount of time that each piece of code take for execution.

Potential candidate tools

• Visual Studio
• gprof
• Valgrind/Calltree

Add Support for parallel testing in CTest

This will make possible to launch multiple test simultaneously from CTest. In this way, users with multiprocessor machines will be able to reduce the time needed for running the test suite.

Add Support to CTest for generating reports locally

This functionality will make possible to analyze the results of the tests suite in a local machine, before it is submitted to the Dart/Dart2 server.

Adopting Dart2

Dart2 will bring a large number of interesting features that will improve the quality control system for ITK. We should plan the transition from Dart to Dart2 in a progressive and organized way.

Deprecation Policy

A committee has been named for designing and enforcing the implementation of a compassionate and user-friendly deprecation policy.

Technical solutions to allow compile time deprecation warnings is to use vendor specifc flags:

• __attribute__((deprecated)) for gcc
• __declspec(deprecated) for Visual Studio compiler.

For more info see VTK_LEGACY in vtkSetGet.h

• How to measure the popularity of a class
  • How useful it is to users
  • Should a class be removed/replaced

Creating a separate IO Toolkit

• Factorizing IO Classes with VTK

The ITK IO classes in ITK are very similar to the VTK ones. In many cases they were simply copy/pasted from VTK

• A particular attention should be done for jpeg since ITK and VTK does not use exactly the same one.
• Question: Does this mean that VTK/ITK should be synchronized on let say the exact same png library ?

• Include the .vti format from VTK (XML compressed images)
• HDF5 (check on VTK)
• Statistics IO
  • Neural Networks
  • Meshes (VTK style ?, ITK style ?)
  • Samples (Statistics)
• Shared Memory
  • passing data between processes using the IO classes.

Refactor FEM

• Update the FEM library from UPenn's version
• Integrate with a new ITK Mesh

Refactor ITK Mesh

• Create a simpler Mesh

Refactor ITK Optimizers

• Revisit the API of ITK optimizers
  • Make it more uniform
• Find a mechanism for specifying the Parameter Scaling from the transforms.
  • Maybe a default or a recomended scaling
  • Is there a way of avoiding the scaling altogether ?

• Standarize the order of parameters in the Transform.
• Some optimizers need to know about the Transforms.

• Add more Transform Initializer classes
  • Maybe some of them could be trainable.

ITK Complete

Package a release of ITK that contains

• Insight
• InsightApplications
• VTK
• FLTK

Along with a common CMakeLists.txt file that will allow to configure the entire package with consistent options and minimal effort.

• These distribution will factorize common resources such as KWSYS/PNG/JPEG/TIFF/expat/gdcm/

• CPack may solve this issue.

Experimental repository with Subversion (SVN)

Subversion is an alternative to CVS and provides functionalites that are superior to CVS. Since this is quite a significant change, what we probably want to do is to start maintaining a parallel Subversion repository in order to gain experience with it for a project of the size of ITK.

• This may be the same Sandbox used by the Insight Journal.

Image Modalities

A variety of image modalities are now well supported in ITK at this point. New image modalities may become important for the medical research community and later on for the clinical users. This section list the upcoming image modalities, their relevance and what would it take to support them in ITK. Note that the categories are not independent and that some of them will overlap.

Color Images

Basic support exists now in ITK for dealing with RGB images. A reduced number of filters and segmenation techniques are ready-to-use in the toolkit. No support for registration of color images is currently available. Although several users have explored this domains, have written ITK classes for it and are willing to contribute their code to the toolkit.

The challenges are

• Color model conversion (HSV, RGB)
• Image IO
• Segmentation
  • Statistical classifications
  • Region growing
  • Levelsets

Multi-Spectral Images

Multi-spectral images are used for modalities such as Microscopy, MRI and optical tomograpy. Not to mention that ITK has been used for astronomy and for satellite imagery. The issues here are very similar to the ones of color images, with the distinction that here we are not subject to the particularities of the human visual system.

The challenges are

• Enhancement and Noise reduction
• Principla components analysis
• Registration for multiple-component images
• Callibration
• Segmentation
  • Statistical classification
  • Region growing
  • Multicomponents level sets
• Pattern recognition
• Image registration
• File formats

4D Images (time series)

Time series are becoming more and more commong in clinical and medical research applications. ITK has the advantage of providing support for N-D images and algorithms. However few of it has been exercised in real cases, and few 4D specific studies have been performed. This issue is relevant for

• Functional MRI
• Perfusion images
• Cardiac 4D
  • MRI
  • CT
• 3D Ultrasound + time
• PET

Tensor Images

Diffusion tensor images are probably the best known variant of tensor images. However, a large number of other physical properties of tissue are represented with Tensors, for example

• Electrical conductivity
• Refraction index
• Mechanical resistance

It is conceivable that images measuring such parameters will become availabe in the near future.

ITK currently has expanded its support for DTI (Diffusion Tensor Images), but much remains to be done in order to provide all the functionalities needed for helping users to develop medical applications.

The challenges are

• File formats
• Image size
• Tensor operations
• Tractography
• Tensor based Segmentation
• Tensor based Registration
  • Tensor to Tensor images
  • Tensor to Vector images
  • Tensor to Scalar images
• Feature detection

Microscopy Images

The specific features of microscopy images are

• 2D images
• Large images (1K x 1K) up to (4K to 4K)
• Mosaics
• Need for annotations
• Movement and tracking of particles
• Multi-spectral images
• Polarization
• File formats (the microscopy community uses a large set of fileformats)
• Deconvolution

Confocal Microscopy Images

The specific features of Confocal Microscopy images are

• 3D images
• Strong anisotropy (Z spacing >> X,Y spacing)
• Attenuation in Z
• Large images
• Multi-spectral images
• Need for registration
• 3D Mosaics
• File formats (the microscopy community uses a large set of fileformats)

Transmision Electron Microscopy (TEM)

The specific features of Transmission Electron Microscopy images are

• 2D images
• Small signal to noise ratio
• Large images
• Need for model based segmentation techniques
• Image features are close to the image resolution

Tomography from Transmision Electron Microscopy (TEM)

The specific features of TEM Tomography images are

• 3D images
• Small signal to noise ratio
• Large images
• Need for model based segmentation techniques
• Image features are close to the image resolution

Endoscopic Images

The specific features of Endoscopic Microscopy images are

• Video images
• Color
• Reflections in tissue
• Continues movement
• Shape from shading

Endoscopic images are more on the domain of computer vision but they are quite important in clinical applications.

Fluoroscopy Images

Fluoroscopic images are very important for image guided surgery, and for computer assisted diagnosis. Since fluoroscopy images are 2D projections of a 3D structure, they have specific challenges.

• Time varying images (video)
• Geometric distorsion
• Need for detection and tracking of instruments
• 3D / 2D registration with 3D preoperative images

Mamography Images

Very few is availble in ITK in order to support Mamograms. This is an area of significant public health relevance and deserves more attention.

Particular issues are

• Large images
• 12 bits / 16 bits
• Contrast enhancement
• Calcification detection
• Significance for Biopsies (assist localization)

Perfusion Images

Perfusion images are quite important for evaluating the vascularization of tissue and for diagnosis and treatement of cardiovascular disorders.

The challenges with this images are

• Intensity normalization
• Time series
• Vascular modeling

Slice contiguous images

ITK uses a contiguous image memory model. All the pixels in an image/volume are in a single contiguous memory block. Many existing medical image analysis systems use a slice contiguous memory model. Here, all the pixels in a slice are in a contiguous memory block but the slices may not be contiguous in memory. A slice contiguous memory model has benefits over a volume contiguous memory model in memory management and cache coherency. ITK should consider adding an ability to operate on slice contiguous volumes. This will require modification to the image iterators and anywhere Image::GetBufferPointer() is called.

SpatialObjects

Spatial objects are being used by a variety of projects in foreseen and unforeseen ways, and their API must be reviewed/expanded to support the unforeseen applications. Additionally, SpatialObjects remain relatively unconnected to the rest of ITK - they are easy to use as a product of ITK processing, but there is limited support for having them drive ITK processing. Specifically,

1. Enable SpatialObjects to be passed along ITK's pipeline. This requires the definition of an itkSpatialObjectSource class as well as derived classes such as itkImageToSpatialObjectFilter
2. Implement SpatialObject traits. SpatialObjects use the itkTree data structure; however, methods for querying that tree are very limited. Currently spatialObject type definition is maintained by a class-specific string accessed via GetTypeName(). A standard naming structure or other form of type designation (e.g., traits) is needed so that filters can be tuned by spatialObject type. For example, certain filters can work on Tube, DTITube, and VesselTube spatialObjects (these classes have a common derivation), but the current GetTypeName designations do not expose their common derivation.

Neural Networks

Neural network methods will be added to ITK. These include multilayered neural networks with linear, sigmoid, and radial basis transfer functions. Training strategies such as backpropagation and quickprop. Also Kohonen self-organizing feature maps will be added. Helper-classes will simplify the implementation and use of traditional 3-layer backprop, non-linear component analysis, radial basis function, and feature map networks.

DICOM Query and Retrieve

A file-dialog-style FLTK-based tool will be provided for dicom query and retrieve. This tool will be require the use of DCMTK. Using this tool, studies on a remote dicom server can be searched, retrieved, and loaded by an ITK application with minimal programming.

What to include in ITK 2.2 (Scheduled for September 2005)

Tensor Images

• Symmetric Second Rank Pixel
• Diffusion Tensor Pixel
• Eigen Values Calculator
• Image IO for Tensor Images
• Support for registration of Tensor Images

FITS File Format

This file format is mostly used in Astronomy, but has the among others, the following interesting features

• Supporting streaming. That is, we can load block of data from the files.
• Supports N-Dimensional data blocks
• Supports Multiple components per pixel

This file format seems to be suitable for managing Tensor images, Microscopy images, and very large datasets.

Support for Logging

Logging classes are useful for debugging purposes and for facilitating to track the behaviour of applications at run time. Classes have been already developed by the ISIS center at Georgtown University and are planned to be integrated into ITK.

What to include in ITK 2.4 (Scheduled for November 30 2006)

• new wrapping

What to include in ITK 2.6 (Scheduled for February 2006)

• New Mesh class
  • Simpler triangular Mesh (2D manifold in 3D space).