[vtk-developers] Pay attention, grab a beer: VTK 5.0

[Will Schroeder]

Hi Folks-

This email is worth paying attention to as it addresses the future of VTK
(namely VTK 5.0). We have had many discussions regarding these issues here at
Kitware and we'd like to hear what some of the other developers have to
say. Some of these changes are pretty big, many are small, but we've worked
hard to balance the needs of the future with legacy requirements from the
past. Our assumptions moving towards this major release are that we may break
some backward compatibility (mainly in the internals, little in the public
API), and that we must provide an upgrade path for users wishing to move to
VTK 5.0. Right now we are anticipating patching VTK 4.2.3 by adding some of
the features/recent additions to create VTK 4.2.4. This will serve as the
starting point for moving ahead. This will be followed by several interim
releases (at least two) which will ultimately culminate in VTK 5.0. We expect
to keep VTK 4.2 around for a long time so users can take the necessary time
to adapt to VTK5.0 as needed. This process may take a year or more to unfold.
We are very interested in feedback; this is a difficult issue and there are
likely ideas out there that we haven't thought of yet.

Before we get into the details I'd like to add some personal perspective. I
do not like the idea of breaking an API. Here at Kitware we end up spending a
lot of our time and resources rewriting books, applications, etc. and it
hurts (since we probably use VTK more than most folks). On the other hand, I
have seen the cost that failing to move forward with new ideas and excessive
attachment to backward compatibility has on software, which can and has
caused the failure of many systems. Many of the proposed changes here are
being driven by users who are dealing with large data, meaning large in
coordinate space (astrophysics, geology) or size of the data. Several of
these folks are committing significant resources towards VTK, both in terms
of funding and development. I am personally taking this very seriously and am
committed to helping the VTK community with these changes as best I can, and
doing a better job of communicating this time around. Anyway, I hope I have
your attention with this long segway :-) And you might want to grab a beer
before continuing...

Here's an itemized summary list of the changes. I will follow these with an
explanation and rationale for why and what we think the outcome will be. I've
roughly sorted the changes from smaller impact to larger.

0. Ongoing local changes

1. Implementing data arrays with a single, templated class

2. Reference counting observers

3. Higher-order cell support

4. AMR/composite dataset support

5. Use iterators to traverse collections

6. Volume rendering changes (support unstructured volume rendering)

7. Fast and slow path filter support

8. Remove old-style callbacks (i.e., SetStartMethod, etc.)

9. Blanking support for structured data

10. Pipeline execution reorganization

11. Using double rather than float in API (and internal calculations)

12. vtkMesh class


Still with me? Take another sip of beer... Here goes some explanation.


0. Ongoing local changes

Obviously there are many additions in the form of new classes, bug fixes,
documentation enhancements, etc. but these do not affect the global VTK
architecture. For example, Jean Favre has contributed a super LIC filter, and
we are on the path to develop new filters and other localized additions for
parallel processing, rendering, 3D widgets, etc.


1. Implementing data arrays with a single, templated class

Lots of data arrays, lots of duplicated code. Bad. We can make this change
with no impact on the public API....it's unlikely to affect developers either.


2. Reference counting observers

Although vtkCommand inherits from vtkObjectBase, we do not use the reference
counting facilities. As a result we have found problems when using complex 3D
widgets. This is a relatively obscure change that will not affect much of
anything.


3. Higher-order cell support

A lot of work to support (finite element) cells of arbitrary complexity. This
is an extension to VTK and should not affect anything.


4. AMR/composite dataset support

Adaptive Mesh Refinement datasets consist of hierarchies of structured data.
Many other data sets (i.e., PLOT3D files with multiple grids) form composite
structures. We belive that this functionality can be added without too much
impact since it is new, but it does depend on item #10 in order to manage the
execution of an algorithm over each of the datasets forming the composite
dataset.

Find more detail at
http://public.kitware.com/pipermail/vtk-developers/2003-October/002597.html


5. Use iterators to traverse collections

VTK missed the boat with iterators. It's nice to support simultaneous
iteration over a collection and to bring it more in line with modern C++.


6. Volume rendering changes (support unstructured volume rendering)

We (with Sandia Natl Lab) are adding volume rendering support for unstructured
grids. There are some changes to the volume rendering classes (new classes,
reorganization of old classes). We believe that we can do this without
impacting the existing class hierarchy.


7. Fast and slow path filter support

To a large extent we are thinking about vtkImageData here. In VTK image data
and associated filters has been implemented with pointer access and do not
handle point/cell attributes other than scalar data. However, to support this
means slowing these filters way down. So we are thinking about reworking
filters like this with support for fast and slow paths depending on how
users configure the filter. I don't see too much impact on users or
developers; maybe some derivation issues. This will likely not be completed
by VTK 5.0 and can be done as we go.


8. Remove old-style callbacks (i.e., SetStartMethod, etc.)

Command/observers good. Old style callbacks bad. Lots of obsolete code
to support the old style stuff. Time to get rid of it.


9. Blanking support for structured data

A few years ago I added blanking to the structured grid dataset. Now that we
are supporting AMR and composite data sets this functionality is becoming
more important. Moreover, we'd like to support point and cell blanking, and
methods to blank data other than just 0/1 on off arrays (e.g., specify
blanking with "all cells in this extent are off"). This will introduce minor
API changes to users and developers who used blanking (which I suspect are
few in number).


10. Pipeline execution reorganization

A bigger one. Anybody who has delved into the execution process in VTK is
probably dazed and confused. I personally suspect there are just plain
wrong implementations here and there. A big problem is the complexity that
the execution model brings to the filters and data objects. Moreover,
VTK users who are doing parallel/large data processing have been constrained
by the execution model that the current implementation enforces (e.g.,
demand driven versus data driven; how do requests for data in a particular
region or at a time step get propagated; how to manage filters that
require iterative execution or that iterate over composite data sets;
how to adapt VTK to different parallel processing models, etc.)

The goal of this work is to isolate future pipeline enhancements to the
execution model (like prioritized streaming and incremental updates).  After
the reorganization, new pipelines can be implemented externally without
changing filters and data objects. We have been working very hard on a paper
that explains our design (we'll share it with the VTK community in the near
future). We believe that we can maintain compatibility with the public API
and even with derived filters using the current pipeline mechanism. It may be
that some complex filters will require rework.


11. Using double rather than float in API (and internal calculations)

This is a severe deficiency in VTK, we need proper double support.  As you
know the current VTK API is float oriented. We have found this to be a
problem not only in filters, but also in the camera model and in
widgets. Many people cannot use VTK because their data is represented with
numbers outside the float range.  And even if the data is scaled to float
range, there are issues because a lot of operations do things like square or
^4 data, meaning the effective float range is really much smaller. The good
news is that anywhere a float value is expected old code will work fine, and
interpreted code should not be affected much at all. It is where C++ pointers
or arrays are used that the problems set in. Note that this change does not
mean float data has to be represented with doubles; it means calculations are
done with double. To do this I believe the right thing to do is to change
float->double in all cases, but there are other options like using a typedef
to select float/double at compile time. Another is to have a dual float/double
interface (similar to what many of you have already started to add).  Both
these ideas are problematic because of the complexity of carrying a dual
float/double API. And using a compile-time typedef introduces problems of its
own when implementing computation, IO, etc.


12. vtkMesh class

The idea behind this is to create a new data object vtkMesh that combines the
best qualities of both vtkPolyData and vtkUnstructuredGrid. Ultimately we may
wish to either reimplement vtkPolyData and vtkUGrid with vtkMesh, or
eliminate these classes altogether. Obviously this needs more thought for the
long term.

Motivation: there are some design problems with vtkPolyData. Namely, the use
of four arrays (verts, lines, polys, strips) and the assumed ordering of
associated cell ids causes problems (inserting a polygon first and then a
line wreaks havoc because the traversal later on assumes lower order
primitives first effectively changing the cell ids). Also, vtkPolyData and
vtkUGrid are essentially the same thing, do we really have to have two
representations?  We also have discovered ways to reduce the total storage,
improve neighborhood operation speed, etc. and otherwise improve the
performance of the classes.



Okay, if you're still here I'd like to reiterate that we are going to do our
best to minimize the impact on both developers and users. And if change is
necessary we will communicate and help provide a way to migrate to the newer
code base. Comments?