View Issue Details Jump to Notes ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0012375VTK(No Category)public2011-07-18 10:352016-08-12 09:55
ReporterPietro Cerutti 
Assigned ToDave DeMarle 
PrioritynormalSeverityminorReproducibilityhave not tried
StatusclosedResolutionmoved 
PlatformOSOS Version
Product Version 
Target VersionFixed in Version 
Summary0012375: [patch] buffer vtkImageReader2
DescriptionThe attached patch adds buffering to the vtkImageReader2 class. This greatly improves performance when reading over a network, e.g., from a CIFS mounted file system.
TagsNo tags attached.
ProjectKitware
Typeperformance
Attached Filesdiff file icon patch-vtkImageReader2.diff [^] (1,369 bytes) 2011-07-18 10:35 [Show Content]
? file icon patch-IO_vtkImageReader2 [^] (2,656 bytes) 2013-05-14 10:45 [Show Content]
? file icon patch-IO_vtkImageReader2-20130516 [^] (2,790 bytes) 2013-05-16 05:30 [Show Content]

 Relationships

  Notes
(0027012)
David Gobbi (developer)
2011-07-18 10:55

This is a bad idea. Using a buffer that is the full size of the file will double the amount of memory that VTK uses when reading the whole file.

You should be able to get the same performance boost by calling UpdateWholeExtent() on the reader.
(0030249)
Pietro Cerutti (reporter)
2013-01-25 12:06

David,

this is not what I have experienced.

The sample program below reads a directory (containing DICOM files) via vtkImageReader2, using SetDataExtent() and UpdateWholeExtent(), and prints the time taken and the resource usage.

This is the result of loading a directory containing 145 DICOM files (CT scan), for a total of 74 MB.

With my patch:

ImageReader2 : 14.745842 sec
Memory usage : 91.6367 MB

Without my patch:

ImageReader2 : 27.874251 sec
Memory usage : 91.1953 MB

This shows that my patch improves the throughput by ~ 80% and does not affect the memory footprint.

=================================================================================================

#include <cstdio>
#include <sys/types.h>
#include <sys/time.h>
#include <sys/resource.h>

#include <vtkSmartPointer.h>
#include <vtkGlobFileNames.h>
#include <vtkImageReader2.h>
#include <vtkStringArray.h>

#define vtkNew(type, name) vtkSmartPointer<type> name = vtkSmartPointer<type>::New()

int
main(int argc, char **argv)
{
    if (argc != 2) {
        std::cerr << "Usage: " << argv[0] << " <path>" << std::endl;
        return (1);
    }

    /* time */
    struct timeval beg, end, delta;

    /* files list */
    vtkNew (vtkGlobFileNames, glob);
    glob->SetDirectory (argv[1]);
    glob->AddFileNames ("*");
    vtkIdType nofFiles = glob->GetNumberOfFileNames ();

    /* load all files with vtkImageReader2 */
    gettimeofday (&beg, NULL);

    vtkNew (vtkImageReader2, read);
    read->SetFileNames (glob->GetFileNames());
    read->SetDataExtent (0, 511, 0, 511, 0, nofFiles);
    read->UpdateWholeExtent ();

    gettimeofday (&end, NULL);
    timersub (&end, &beg, &delta);
    std::cout << "ImageReader2 : " << delta.tv_sec << "." << delta.tv_usec << " sec" << std::endl;

    /* get current resource footprint */

    struct rusage r;
    getrusage (RUSAGE_SELF, &r);
    std::cout << "Memory usage : " << r.ru_maxrss / 1024. << " MB" << std::endl;

    return (0);
}
(0030251)
David Gobbi (developer)
2013-01-25 13:40

The extra memory usage is a temporary and per-file. So if you are reading the image as a whole bunch of slices, you are not going to see much effect.

But let's say someone is loading a single 10GB file (i.e. not a whole bunch of slices, but a single very large file). Then memory usage will temporarily spike at 20GB before dropping down to 10GB.

If you want this patch to be considered, you should add some methods to the class so that people can choose whether or not they want this feature.
(0030761)
Pietro Cerutti (reporter)
2013-05-14 10:45

Ok, I figured out that the double amount of memory is due to the fact that the algorithm copies from the allocated buffer into its datastructure.

How would you see this reworked patch? It adds a boolean macro (BufferWholeFile) to enable / disable this feature. In the doxy comments I have made explicit that twice the ram is used, and that this could be of benefit especially over CIFS shares.
(0030784)
Pietro Cerutti (reporter)
2013-05-16 05:28

I have a revised patch, which more conform to the VTK style.
(0030899)
Jean-Christophe Fillion-Robin (manager)
2013-06-05 13:24

Hi Folks,

Is the problem is still present in VTK 6.

Would be great if you could submit a patch using Gerrit. For more details see http://www.vtk.org/Wiki/VTK/Git/Develop [^]

Thanks
Jc
(0031191)
Dave DeMarle (administrator)
2013-07-22 17:46

Please submit the patch via gerrit so that it can be more easily reviewed.
(0037239)
Kitware Robot (administrator)
2016-08-12 09:55

Resolving issue as `moved`.

This issue tracker is no longer used. Further discussion of this issue may take place in the current VTK Issues page linked in the banner at the top of this page.

 Issue History
Date Modified Username Field Change
2011-07-18 10:35 Pietro Cerutti New Issue
2011-07-18 10:35 Pietro Cerutti File Added: patch-vtkImageReader2.diff
2011-07-18 10:55 David Gobbi Note Added: 0027012
2013-01-25 12:06 Pietro Cerutti Note Added: 0030249
2013-01-25 13:40 David Gobbi Note Added: 0030251
2013-05-14 10:45 Pietro Cerutti Note Added: 0030761
2013-05-14 10:45 Pietro Cerutti File Added: patch-IO_vtkImageReader2
2013-05-16 05:28 Pietro Cerutti Note Added: 0030784
2013-05-16 05:30 Pietro Cerutti File Added: patch-IO_vtkImageReader2-20130516
2013-06-05 13:24 Jean-Christophe Fillion-Robin Note Added: 0030899
2013-07-22 17:46 Dave DeMarle Note Added: 0031191
2013-07-22 17:46 Dave DeMarle Status backlog => expired
2013-07-22 17:46 Dave DeMarle Assigned To => Dave DeMarle
2016-08-12 09:55 Kitware Robot Note Added: 0037239
2016-08-12 09:55 Kitware Robot Status expired => closed
2016-08-12 09:55 Kitware Robot Resolution open => moved


Copyright © 2000 - 2018 MantisBT Team