VisIt Database Bridge

From KitwarePublic
Jump to navigationJump to search

Introduction

This article describes the Visit Database Bridge ParaView plugin. The motivation for our plugin is to allow ParaView to make use of VisIt's IO components, and to explore the re-usable capabilities of VisIt and its underlying pipeline library, avt. Like ParaView, VisIt is a scientific data visualization application based on VTK and Qt. There are a number of subtle differences in the internal workings of the two applications. However, in general there are more similarities than differences. Hence porting functionality from one to the other is fairly straight forward and potentially very beneficial as both applications have large, publicly available, thoroughly tested code bases.

VisIt IO components are called database plugins. The VisIt developers have provided a consistent interface both for database plugin authors, who are typically scientists, via the avtFileFormat interface and for VisIt developers via the avtGenericDatabase interface. We thus had two options when designing a bridge between the applications. The file format interface provides low level access that could lead to a minimalistic approach where the essential code needed to read data from disk is easily isolated. However, if this path were taken we'd have to forgo quite a lot of VisIt's functionality and would end up re-inventing a large body of code in order to provide only basic functionality. This is why we chose the second option, namely to use the avtGenericDatabase interface which turns out to be quite a bit more efficient from a coding point of view as we can make use of all of VisIt's avt pipeline objects and filters. Because in this approach we are under the covers building an avt pipeline we can provide those already familiar with VisIt expected functionality such as expression evaluation, material interface reconstruction (MIR), and sub-setting operations.

In our implementation we have a single VTK class called the vtkVisItDatabase that exposes all of VisIt's 84 database plugins in a general way. At run time a database name is passed into the vtkVisItDatabase. Internally the vtkVisItDatabase owns an avt DatabasePluginManager object which loads the named VisIt database plugin. A file name can then be set and the data file opened. Once a specific file has been opened the DatabasePluginManager will create an avtGenericDatabase object from which meta data can be extracted and an avtSource object can be created. This avt pipeline source object can then be used to construct pipelines from which we can extract VTK datasets. The vtkVisItDatabase class is layered inside of a VTK algorithm subclass named vtkVisItDatabaseBridge which is where all of the VTK pipeline code is. This class is used like any other VTK pipeline object.

To expose the bridge as a reader in ParaView we generate a server manager XML source proxy tag for each of VisIt's database plugins. The tags name the specific database plugin to use with the given list of file extensions and all point toward the bridge class. Due to VisIt's long list of external library dependencies, there can be quite a bit of variability in the available database plugins between builds. We handle this by probing the VisIt installation and generating XML for only those database plugins that are found as the bridge is compiled.

VisIt's database concept is very general, and as such very powerful. What the database abstraction does is make any file readable by one of the 84 available database plugins appear the same as any other to the application. The database thus represents some arbitrary collection of datasets and provides a means to query and access what ever is available. There are only a few restrictions on what datasets can be provided. Further, the database concept implies little about how the provided data is organized. For that purpose in addition to a couple of meta data objects VisIt uses the avtSILSet object. SIL stands for subset inclusion lattice, and was introduced to facilitate sub-setting operations. The avtSILSet can be thought of as a directed graph that describes a collections of and collections within VTK datasets. Entities called sets, blocks in and AMR dataset for instance, are given unique SILSetIds and are hierarchically organized. This scheme allows for collections or groupings of data to be simply constructed. For example mechanical assemblies can be handled by creating a SIL set with edges to a number of domains. Sub-setting operations are performed by following the graphs edges. Beyond facilitating sub-setting, the avtSILSet provides a consistent and flexible presentation of arbitrary collections of VTK data.

An important use of SIL sets are in VisIt's avt pipeline inside of avtDataRequest objects which travel through the pipeline. An avt source provides a data request initialized with everything it can provide. Starting at the end of a pipeline avt filters then can modify the data request and pass it upstream. In our case, eventually the data request reaches the generic database object which processes it by manipulating the currently loaded database plugin. Data is read into memory and passed back down the pipeline in an avtDataSet object which is very similar to a vtkCompositeDataSet. In the bridge we have exposed the avtDataRequest to the ParaView client via a custom Qt panel. Each of the generated ParaView reader source proxies are associated with this panel, thus all of the 84 available VisIt database plugins have the same user interface.

Upgrading VisIt should be as simple as installing the new version and reconfiguring the bridge. Unfortunately a number aspects of VisIt's design complicate things. The first and largest complication is that VisIt has forked VTK at release 5.0 2006. In order to use VisIt from within ParaView we have to use the same version of VTK in both. Unfortunately, VTK has changed substantially in between version 5.0 and the current cvs trunk, 5.5.0 and the VisIt developers have modified their fork. Not all of VisIt's new or modified VTK code is compatible with the current revision of VTK. Some of the major incompatibilities include VisIt's rendering, actor and mapper additions to VTK or are a result of VTK's interfaces changing to make use of vtkIdType. To work around these issues we ended up patching VisIt's build system. Additionally VisIt is not a seamlessly cross platform application, so there are two sets of patches one for Windows and one for Linux/Mac. Fortunately for our purposes only the build system configuration files needed to be modified. No modifications to the VisIt sources were required. In the next major release of VisIt, VisIt 2.0, these issues will likely be resolved, as VisIt developers are actively cleaning up their code base with its use as a library in mind and switching to CMake.

In conclusion the VisIt database bridge, developed as a plugin for ParaView 3.6, successfully exposes 84 VisIt database plugins to ParaView users in a consistent way. ParaView will likely only use 67 of these due to overlap in existing functionality. For the 3.6 release we have not included expression evaluation or MIR filters, nor addressed the translation of ghost cell or blanking arrays. However, we have shown that the potential benefits of reusing VisIt code are realizable and that at least some of the technical challenges remaining will be mitigated as VisIt 2.0 is released next year.

Figure 1
vtkVisItDatabaseBridge schematic, showing how the plugin's five primary components, vtkVisItDatabaseBridge, vtkVisItDatabase, DatabasePluginManager, VisIt's 84 database plugins, and pqVtkVisItDatabaseBridgePanel, work in ParaView, and demonstrates their use on a CGNS data file which contains two VTK datasets.
VisItSlideScreenShot.png

Classes

The structure of our bridge is as follows:

  • vtkVisItDatabase -- VTK object that encapsulates all avt code, providing clients with an API for manipulating database plugins, the meta-data produced from files they can load, configuration of data requests and conversion the data returned from actual read operations into VTK data objects. Contains a DatabasePluginManager.
  • vtkVisItDatabaseBridge -- A VTK algorithm that implements the VTK pipline required of a ParaView reader. Contains a vtkVisItDatabase.
  • pqVisItDatabaseBridgePanel -- A custom QT panel for ParaView that can read the directed graph describing the data available in any given file.

Which of VisIt's databases are available?

This depends largely on how VisIt is built because databases only are built if their dependencies are found. The following list reflects what is available in a full VisIt 1.10.0 build. A number of these however are intentionally skipped since ParaView already has support for them, the current count is 67 new formats available to ParaView.

  • ANALYZE
  • ANSYS
  • AUXFile
  • AugDecomp
  • BOV
  • BOW
  • Boxlib2D
  • Boxlib3D
  • CEAucd
  • CGNS
  • CMAT
  • CTRL
  • Cale
  • Chombo
  • Claw
  • Cosmos
  • CosmosPP
  • Curve2D
  • DDCMD
  • Dune
  • Dyna3D
  • EnSight
  • Enzo
  • ExtrudedVol
  • FITS
  • FLASH
  • Fluent
  • GDAL
  • GGCM
  • GTC
  • H5Nimrod
  • H5Part
  • Hex
  • Image
  • KullLite
  • Lines
  • M3D
  • MFIX
  • MM5
  • Miranda
  • NASTRAN
  • NETCDF
  • Nek3D
  • OVERFLOW
  • OpenFOAM
  • PATRAN
  • PDB
  • PFLOTRAN
  • PLOT2D
  • PLOT3D
  • Pixie
  • PlainText
  • Point3D
  • ProteinDataBank
  • RAW
  • Rect
  • S3D
  • SAMI
  • SAMRAI
  • SAR
  • SAS
  • STL
  • Shapefile
  • Silo
  • SimV1
  • SimV1Writer
  • Spheral
  • StreamGhostTest
  • TFT
  • TSurf
  • Tecplot
  • Tetrad
  • UNIC
  • VASP
  • VLI
  • Vis5D
  • Vista
  • WavefrontOBJ
  • XDMF
  • XYZ
  • Xmdv
  • ZeusMP
  • ZipWrapper

Building ParaView

There are two special considerations with respect to building ParaView for use with the bridge. First it must be linked to the same HDF5 that is used with VisIt. Second, ParaView must be compiled with 64 bit support disabled. Here is the relevant CMake configuration:

CMake Variable Description
PARAVIEW_USE_SYSTEM_HDF5 ON
HDF5_LIBRARY /opt/hdf5-1.6.8_ser/lib/libhdf5.a;/opt/szip-2.1/lib/libsz.a
VTK_USE_64BIT_IDS OFF

Note the inclusion of the szip library.

The correct order to build things is as follows:

  1. Szip
  2. Hdf5
  3. ParaView
  4. VisIt's dependencies
  5. VisIt
  6. ParaView Bridge Plugin

Building the ParaView Plugin

Building our ParaView plugin is straight forward as it makes use of CMake. One unconventional aspect of the the plugin build is a bootstrap step that install's VisIt for the plugin. This bootstrap step should be done behind the scenes, however for technical reasons it wasn't possible. Once our patched version of VisIt has been built building the plugin can be achieved as follows: 

#!/bin/bash
cd ./ParaView3/Plugins/VisItDatabaseBridge      # Switch into the bridge plugin source directory.
mkdir bin                                       # Make a directory for our out of source build.
cd bin                                          # Switch into the out of source build dir.
ccmake ../                                      # Set BOOTSTRAP=ON, and whatever else makes sense ;) .
make                                            # Bootstrap, you should see a status report.
ccmake ../                                      # Set BOOTSTRAP=OFF.
make                                            # Build the Paraview plugin.
sudo make install                               # Optional.

I have shown the command line procedure for Linux, however, the same steps, translated into that context, are needed on Windows with Visual Studio as well. Below is a table with explanation of the CMake variables that configure the bootstrap and plugin build.

CMake Variable Description
BOOTSTRAP ON will install VisIt and generate ParaView servermanager XML, Pq XML, and custom panel associations CMake file. OFF will configure the build the ParaView plugin.
BOOTSTRAP_CONFIGURE_EXLUDES Path to a file with the names of the VisIt database plugins to skip. The file format is one VisIt database plugin per line. The default value should be suitable for most purposes.
CMAKE_INSTALL_PREFIX Use only for install build. Where to install the plugin.
ParaView_DIR Point to a ParaView out of source build.
PLUGIN_BUILD_TYPE Build type for windows builds set to either "Release" or "Debug" to match your ParaView and VisIt build type. Windows only!!
VISIT_BASE Point to VisIt build/source tree.
VISIT_RUN_TIME Use only for install build. Relative path from the ParaView plugin's final location to VisIt databases.
VISIT_WITH_MPI Set if you want to use VisIt's MPI libs. This option has not been used.

NOTE: The CMakeLists.txt file is well documented see that for more information.

Build VisIt and dependencies

Linux

VisIt uses a site-config file, which is nothing more than a bash script that it sources, to configure itself. Below find my site-config used for Linux, namely vtkVisItDatabaseBridge.conf. There are also notes on building each of VisIt's dependencies. It looks a lot like the "build_visit" script. If you get stuck, the build_visit script is the ultimate reference. 

#==============================================================================
#
# VisIt site-config for the VTK--VisIt database bridge.
#
#==============================================================================(BUILD NOTES)
#
# This file is a site-config file that I used to build VisIt as a library for use with
# ParaView. To do so VisIt's makefiles had to be patched so that its GUI is excluded and
# ParaView's VTK can be used. The top section of the file is a set of notes as to how I
# configured VisIt's dependencies and VisIt itself. The bottom of this file is a bash
# script that VisIt uses to configure itself. You will have to modify the variables to
# reflect locations of the dependencies on your system. If you are only interested in a
# specific database plugin then you can skip many of the dependencies. You can't skip szip,
# silo, or hdf5. Be sure to point ParaView to the version of HDF5 you are using here when
# you build ParaView.
#
# Notes:
#
# VisIt suggests to use static libraries for its dependencies, I have gone with that even
# when it would make sense to do otherwise.
#
# The same hdf5 libraries must be used by both VisIt and PV. Basically this implies the 
# following procedure:
# 
# 1) Build szip, see below.
# 2) Build hdf5, see below.
# 3) Build ParaView link against hdf5 from step 2.
# 4) Build the remianing VisIt dependencies, see below.
# 5) Build VisIt, see below.
# 5) Bootstrap the database bridge PV plugin, see its cmake lists.
# 6) Build the database bridge PV plugin, see its cmake lists.
# 
# This documentation found here in this file pertains to steps 1,2,4 and 5. 
#
# Windows has a completely different build process because VisIt used visual studio. Mac is
# as of yet uncharted territory.
#
# In theory one can build all of VisIt's dependencies using the "build_visit" script available
# from VisIt's web site. If you go that route, then you still should use this file to configure
# the patched VisIt with the ./configure line below, then scroll down to where all the variables
# are.
#
# You'll need to have Qt 3 development packge around somewhere because VisIt's ./configure
# script has a Qt 3 dependency.
#
# It's best to try ot use the versions of the libraries I have used here, if not then
# second best is to use the ones recomended by VisIt. If you do niether you're on
# your own, but things will probably be OK as long as you stick to the recomended
# configuration.
#
#==============================================================================(BUILD NOTES)
#
# szip
#======
# LIBS=-lm CFLAGS=-fPIC CXXFLAGS=-fPIC ./configure --prefix=/opt/szip-2.1 --disable-shared
# make -j 8
# sudo make install
#
# HDF4
#======
# LIBS=-lm CFLAGS=-fPIC CXXFLAGS=-fPIC ./configure --prefix=/opt/hdf4-4.2r4 --disable-fortran --with-szlib=/opt/szip-2.1/ 
# make -j 8
# sudo make install
#
# HDF5
#======
# for gcc 4.3 you'll have to edit perform/zip_perf.c change line 549 to "output = open(filename, O_RDWR | O_CREAT, S_IRWXU);"
#
# When you configure your ParaView build do this:
# USE_SYSTEM_HDF5   ON
# HDF5_LIBRARY      /opt/hdf5-1.6.8_ser/lib/libhdf5.a;/opt/szip-2.1/lib/libsz.a
#
# CFLAGS=-fPIC CXXFLAGS=-fPIC ./configure --prefix=/opt/hdf5-1.6.8_ser --disable-shared --disable-fortran --disable-parallel --with-szlib=/opt/szip-2.1 
# make -j 8
# sudo make install
# sudo ln -s /opt/szip-2.1/lib/libsz.a /opt/hdf5-1.6.8_ser/lib/libsz.a
#
# BoxLib
#=======
# For gcc-4.3 add #include <cstdlib> to ParallelDescriptor.cpp
# Usiing gfortran instead of g77 works well.
#
#
# cd into boxlib directory in the CCSEApps
# edit GNUMakefile, set USE_MPI=false, COMP=g++
# mv std std.old
# chmod 644 *.H
# CXXFLAGS=-fPIC CFLAGS=-fPIC FFLAGS=-fPIC make -j 8
# sudo  mkdir -p /opt/boxlib/{include/2D,include/3D,lib}
# sudo cp libbox3d.Linux.g++.f77.DEBUG.a /opt/boxlib/lib/libbox3D.a
# sudo cp *.H /opt/boxlib/include/3D/
# edit GNUMakefile,set DIM=2
# CXXFLAGS=-fPIC CFLAGS=-fPIC FFLAGS=-fPIC make -j 8
# sudo cp libbox2d.Linux.g++.f77.DEBUG.a /opt/boxlib/lib/libbox2D.a
# sudo cp *.H /opt/boxlib/include/2D/
#
# NetCDF
#========
# For gcc-4.3 add #include <cstring> to ./cxx/ncvalues.cpp
#
# CXXFLAGS=-fPIC CFLAGS=-fPIC ./configure --prefix=/opt/netcdf-3.6.0-p1
# make -j 8
# sudo mkdir /opt/netcdf-3.6.0-p1
# sudo make install 
#
# Silo
#======
# 4.6.2 doesn't work with VisIt and gcc 4.3 as of this writing(2009-02-25). Some sort of link issue, may be libtool.?
#
# ./configure --prefix=/opt/silo-4.6.2 --without-readline --with-hdf5=/opt/hdf5-1.6.8_ser/include/,/opt/hdf5-1.6.8_ser/lib/ --without-exodus --with-szlib=/opt/szip-2.1 --disable-fortran --disable-browser --disable-shared --disable-silex
# make -j 8
# sudo make install
# sudo ln -s /opt/silo-4.6.2/lib/libsiloh5.a /opt/silo-4.6.2/lib/libsilo.a
#
# CGNS
#======
# Only use hdf5 if you need it. & you don't! --with-hdf5=/opt/hdf5-1.6.8_ser/
# CXXFLAGS=-fPIC CFLAGS=-fPIC ./configure --prefix=/opt/cgns-2.4 --with-szip=/opt/szip-2.1/lib/libsz.a 
# make -j 8
# sudo mkdir -p /opt/cgns-2.4/{include,lib}
# sudo make install
# 
# CFITSIO
#=========
# ./configure --prefix=/opt/cfitsio
# make -j 8
# sudo make install
#
# H5Part
#========
# CFLAGS=-fPIC ./configure --prefix=/opt/h5part-1.3.3 --with-hdf5path=/opt/hdf5-1.6.8_ser/
# make -j 8
# sudo make install
#
# CCMIO
#=======
# Didn't work & looks like it uses qmake ?!?. If some one complains we'll get it working.
#
# GDAL
#=======
# For gcc-4.3 download gdal-1.6.0, the following config works with both version
#
# CFLAGS=-fPIC CXXFLAGS=-fPIC ./configure --prefix=/opt/gdal-1.6.0 --enable-static --disable-shared --with-libtiff=internal --with-gif=internal --with-png=internal --with-jpeg=internal --with-libz=internal --with-netcdf=no --without-jasper --without-python
# make -j 8
# sudo make install
#
# Qt 3
#=======
# We don't have to link against Qt but VisIt plugin system requires Qt to build.
# export QTDIR=`pwd`
# export LD_LIBRARY_PATH=$QTDIR/lib
# ./configure --prefix=/opt/qt-3.3.8
# make -j 8
# sudo make install
#
#
# VisIt
#=======
# NOTE for now set VTK_USE_64BIT_IDS to OFF
# NOTE force the use of static libs when both shared and static are available.? -Bstatic 
#
# NOTE Don't patch. use the pre-patched tarball!
# From the top level VisIt directory, one up from src.
# patch -p6 < /home/burlen/ext/kitware_cvs/VisitPluginTool/VisItPV3Build.in.patch
#
# ./configure --prefix=/opt/VisIt-1.10.0 --with-config=/home/burlen/ext/kitware_cvs/VisitPluginTool/building_visit/vtkVisitDatabaseBridge.conf --with-hdf5=/opt/hdf5-1.6.8_ser/include,/opt/hdf5-1.6.8_ser/lib --enable-parallel --disable-scripting --disable-visitmodule --disable-viewer-mesa-stub --disable-icet --disable-bilib --disable-glew --disable-bzip2 --with-dbs=all
#
#
#==============================================================================(Configuration)
VTK=$VTK_BUILD/VTK 
VTK_SOURCE=/home/burlen/ext/ParaView3/VTK
VTK_BUILD=/home/burlen/ext/PV3/
VISIT_VTK_CPPFLAGS="\
      -I$VTK_SOURCE/Parallel\
      -I$VTK_SOURCE/GenericFiltering\
      -I$VTK_SOURCE/Views\
      -I$VTK_SOURCE/Imaging\
      -I$VTK_SOURCE/GUISupport\
      -I$VTK_SOURCE/Infovis\
      -I$VTK_SOURCE/Hybrid\
      -I$VTK_SOURCE/VolumeRendering\
      -I$VTK_SOURCE/Examples\
      -I$VTK_SOURCE/Wrapping\
      -I$VTK_SOURCE/IO\
      -I$VTK_SOURCE/Filtering\
      -I$VTK_SOURCE/Common\
      -I$VTK_SOURCE/Widgets\
      -I$VTK_SOURCE/Rendering\
      -I$VTK_SOURCE/Rendering/Testing/Cxx\
      -I$VTK_SOURCE/Patented\
      -I$VTK_SOURCE/Graphics\
      -I$VTK_SOURCE/Utilities"
VISIT_VTK_CPPFLAGS="$VISIT_VTK_CPPFLAGS -I$VTK_BUILD/VTK -I$VTK_BUILD/VTK/Utilities"
VISIT_VTK_LDFLAGS="\
     -rdynamic\
     -L$VTK_BUILD/bin\
     -L/opt/mpich2-1.0.8/lib\
     -lvtkFiltering\
     -lvtkHybrid\
     -lvtkParallel\
     -lvtkGraphics\
     -lvtkImaging\
     -lvtkRendering\
     -lvtkGraphics\
     -lvtkImaging\
     -lvtkftgl\
     -lvtkfreetype\
     -lGL -lXt -lSM -lICE -lX11 -lXext\
     -lvtkIO\
     -lvtkDICOMParser\
     -lvtkmetaio\
     -lvtksqlite\
     -lvtkpng\
     -lvtktiff\
     -lvtkzlib\
     -lvtkjpeg\
     -lvtkexpat\
     -lvtkexoIIc\
     -lvtkNetCDF\
     -lvtkverdict\
     -lvtkFiltering\
     -lvtkCommon\
     -lvtksys\
     -ldl\
     -lm\
     -Wl,-rpath,$VTK_BUILD/bin:/opt/mpich2-1.0.8/lib\
     -lmpichcxx -lmpich -lpthread -lrt"

VTK_INCLUDE=$VISIT_VTK_CPPFLAGS

# MESA=/opt/Mesa-7.2
# MESA_INCLUDE=/opt/Mesa-7.2/include
# MESA_LIBS=-L/opt/Mesa-7.2/lib

# QT, is required by the build system eg. xmlToMakefile
QT_BIN=/opt/qt-3.3.8/bin
QT_INCLUDE=/opt/qt-3.3.8/include
QT_LIB=/opt/qt-3.3.8/lib

# # NOTE Also have to modify configure.in add version test for 4.4.3 
# # other issues: include structure.
# QT_BIN=/usr/bin
# QT_INCLUDE=/usr/include/qt4/Qt
# QT_LIB=/usr/lib

# Compiler flags.
CC="gcc"
CXX="g++"
CFLAGS="-g -Wno-deprecated"
CXXFLAGS="-fPIC -g -Wno-deprecated -DMPICH_IGNORE_CXX_SEEK -I/opt/mpich2-1.0.8/include $CXXFLAGS"
CPPFLAGS="-fPIC $VISIT_VTK_CPPFLAGS -g -Wno-deprecated $CPPFLAGS"
MPI_LIBS="-L/opt/mpich2-1.0.9/lib -Wl,-rpath -Wl,/opt/mpich2-1.0.8/lib -lmpichcxx -lmpich -lpthread -lrt "

# Database reader plugin support libraries
DEFAULT_SZIP_INCLUDE=/opt/szip-2.1/include
DEFAULT_SZIP_LIB=/opt/szip-2.1/lib
DEFAULT_HDF4_FLAGS=/opt/hdf4-4.2r4/include
DEFAULT_HDF4_LIBS=/opt/hdf4-4.2r4/lib
DEFAULT_HDF5_INCLUDE=/opt/hdf5-1.6.8_ser/include
DEFAULT_HDF5_LIB=/opt/hdf5-1.6.8_ser/lib
DEFAULT_NETCDF_INCLUDE=/opt/netcdf-3.6.0-p1/include
DEFAULT_NETCDF_LIB=/opt/netcdf-3.6.0-p1/lib
DEFAULT_SILO_INCLUDES=/opt/silo-4.6.1/include
DEFAULT_SILO_LIBRARY=/opt/silo-4.6.1/lib
DEFAULT_BOXLIB2D_INCLUDE=/opt/boxlib/include/2D
DEFAULT_BOXLIB2D_LIBS=/opt/boxlib/lib
DEFAULT_BOXLIB3D_INCLUDE=/opt/boxlib/include/3D
DEFAULT_BOXLIB3D_LIBS=/opt/boxlib/lib
DEFAULT_CFITSIO_INCLUDE=/opt/cfitsio/include
DEFAULT_CFITSIO_LIB=/opt/cfitsio/lib
DEFAULT_H5PART_INCLUDE=/opt/h5part-1.3.3/include
DEFAULT_H5PART_LIB=/opt/h5part-1.3.3/lib
DEFAULT_CGNS_INCLUDE=/opt/cgns-2.4/include
DEFAULT_CGNS_LIB=/opt/cgns-2.4/lib
DEFAULT_GDAL_INCLUDE=/opt/gdal-1.6.0/include
DEFAULT_GDAL_LIB=/opt/gdal-1.6.0/lib
# DEFAULT_CCMIO_INCLUDE=
# DEFAULT_CCMIO_LIB=
# BV_MILI_DIR=
# DEFAULT_VISUS_INCLUDE=
# DEFAULT_VISUS_LIB=
# DEFAULT_ITAPS_INCLUDE=
# DEFAULT_ITAPS_LIB=
# DEFAULT_EXODUS_INCLUDES=
# DEFAULT_EXODUS_LIBRARY=

Windows

The windows build is somewhat easier. You need to use our patched solutions. These are in the MSVC8 directory. You only need to build two solutions. First VisIt, then the databases. ParView has to link against VisIt's HDF5 library. Here are some technical details.

  1. Un-tar the VisIt sources that include our modified Visual Studio solution files.
  2. Build ParaView and link against the VisIt HDF5 library in "C:\Path\To\Your\VisIt\UnTar\VisIt-1.10.0.X-all\windowsbuild\lib\MSVC8.Net\ThirdParty"
  3. In the Windows control panel set an environment variable as follows, "VISITDEVDIR=C:\Path\To\Your\VisIt\UnTar"
  4. Copy all dll and libs from: "C:\Path\To\Your\ParaViewBuild\bin\Release\{*.lib,*.dll}" to: "C:\Path\To\Your\VisIt\UnTar\VisIt-1.10.0.X-all\windowsbuild\MSVC8.Net\ThirdParty"
  5. In \windowsbuild\projects-MSVC8.Net modify "VisItLibPaths.vsprops" replace: "C:\PV3\Debug" with "C:\Path\to\your\PV3Build\Release"
  6. In \windowsbuild\projects-MSVC8.Net modify "VisItIncludePaths.vsprops" replace: "C:\ParaView3\VTK" with "C:\Path\To\Your\Source\ParaView3\VTK"
  7. In your VisIt windows build directory find and build: VisIt.sln using MSVC. - The VisIt.exe will fail to build but that is fine.
  8. In your VisIt windows build directory find and build: databases.sln using MSVC.
  9. Build the ParaView plugin as described above. CMake will handle the plugin build in a seamlessly cross platform build.

Notes for reconstructing the Visual Studio solution files follow,these are not needed to build the modified solutions as described above! Read on if you need to reconstruct the modified solutions from scratch. To get our modified build set up on Windows using MSVC Express 2008 I had go through the patch that we used for the linux build line by line and identify the coresponding edit in the corresponding Visual Studio solution and project. Most of this was simply eliminating projects or individual source files. Additionally I had to add 3 property pages, "VisItLibPaths.vsprops", "VisItIncludePaths.vsprops", and "vtkLibs.vsprops", to the two solutions involved, "VisIt.sln", and "databases.sln".

Mac

Uncharted territory. There have been posts to the VisIt list detailing issues with the OS X build (April 2009). There is extensive notes on building dependencies for OS X in the top level source directory. The build_visit script should be used as a guide. There are a number of patches in line. Don't attempt he mac build without reading the build_visit script!

General Information and Notes

On Linux and Mac VisIt makes use of autotools, while on Windows VisIt makes use of Visual Studio solutions.

An additional complication when developing our bridge plugin was that VisIt builds only those plugins whose dependencies can be resolved at compile time so that a static build configuration was impractical. We have handled this by writing a bootsrap configuration utility that probes the named VisIt install loading database plugins one by one, extracting name, and file extensions as we go and generating the servermanager XML. The one complication of the bootstrap is that ParaView plugins which require a custom panel have to name that panel in the CMake file. We don't know which plugin's are going to be required until we are not able to modify the CMake files while CMake is running. To work around this we build the plugin in two steps, a bootstrap step and a build step.

This file is a site-config file that I used to build VisIt as a library for use with ParaView. To do so VisIt's makefiles had to be patched so that its GUI is excluded and ParaView's VTK can be used. The top section of the file is a set of notes as to how I configured VisIt's dependencies and VisIt itself. The bottom of this file is a bash script that VisIt uses to configure itself. You will have to modify the variables to reflect locations of the dependencies on your system. If you are only interested in a specific database plugin then you can skip many of the dependencies. You can't skip szip, silo, or hdf5. Be sure to point ParaView to the version of HDF5 you are using here when you build ParaView.

Notes:

VisIt suggests to use static libraries for its dependencies. This isn't always a good idea. HDF5 for instance should be shared when using with ParaView. Theses notes need to be updated to reflect this. TODO

Windows has a completely different build process because VisIt used visual studio. Mac is as of yet uncharted territory.

In theory one can build all of VisIt's dependencies using the "build_visit" script available from VisIt's web site. If you go that route, then you still should use this file to configure the patched VisIt with the ./configure line below.

You'll need to have Qt 3 development packge around somewhere because VisIt's ./configure script has a Qt 3 dependency.

It's best to try ot use the versions of the libraries I have used here, if not then second best is to use the ones recomended by VisIt. If you do niether you're on your own, but things will probably be OK as long as you stick to the recomended configuration.

An additional complication when developing our bridge plugin was that VisIt builds only those plugins whose dependencies can be resolved at compile time so that a static build configuration was impractical. We have handled this by writing a bootsrap configuration utility that probes the named VisIt install loading database plugins one by one, extracting name, and file extensions as we go and generating the servermanager XML. The one complication of the bootstrap is that ParaView plugins which require a custom panel have to name that panel in the CMake file. We don't know which plugin's are going to be required until we are not able to modify the CMake files while CMake is running. To work around this we build the plugin in two steps, a bootstrap step and a build step.

When a user attempts to open a file that can be read using one of the VisIt database plugins it will return a true value from it's CanReadFile method. When ParaView executes the request information phase of a pipeline update the bridge, by manipulating the database object, queries database plugin. The newly discovered SIL is displayed in the ParaView client. The user then interacts with the SIL before initiating the request data pipeline update phase during which the user's selections are translated into an avtDataRequest and fed into an avt pipeline. At this point the avt pipeline is executed and the resulting avt data set is translated into a VTK data object where it is shallow copied into the ParaView pipeline. From that point on ParaView operates as usual.

The plugin build has two distinct parts, building our patched version of VisIt and building the plugin itself.Building VisIt is a complicated process due to a large number of external library dependencies. VisIt has handled the issue by providing a the "build_visit" bash script to automate the process for Linuxes and Mac. Unfortunately

Furthermore VisIt was not designed as a library but rather as a stand alone application so that on some platforms there are many unnamed interdependencies between the compiled objects leading to a situation where dynamic loaders don't know how to resolve undefined symbols.

What's Not Done

  1. Install issues on Linux need to be resolved. rpath is stripped leading to some loading issues. Windows is OK.
  2. PV gives up if the first plugin to try to load the file fails, it should keep trying until all have been tried. For example many databases use a .h5 extension, all need to be tried not just the first one.
  3. We need to map VisIt ghost cells to VTK ghost cells, not doing so is going to introduce block-block artifacts.
  4. Finish MIR. Part of the MIR code has been written (it's commented out). Basically we need to inquire with the VisIt devs about what is the right way to do this.
  5. Finish Experssions.Part of the expression code has been written (it's commented out). Expressions will be expected by VisIt users.They can be trivially added. See the avtExpresssionEvaluationFilter.
  6. Leaks! Run the plugin through valgrind and you'll find all kinds of leaks. These are certainly coming from within VisIt. There is a Wiki page on VisIt memory management strategy and it may have useful pointers.
  7. AMR data. AMR data has been partially implemented. For the 3.6 release I made it possible to open AMR files and browse their contents via the PV U.I. but no data is loaded, The main issue to resolve is the mapping from VisIt ghost arrays to P.V. blanking arrays.
Retrieved from "https://public.kitware.com/Wiki/index.php?title=VisIt_Database_Bridge&oldid=21675"