View Issue Details [ Jump to Notes ] | [ Print ] | ||||||||
ID | Project | Category | View Status | Date Submitted | Last Update | ||||
0012375 | VTK | (No Category) | public | 2011-07-18 10:35 | 2016-08-12 09:55 | ||||
Reporter | Pietro Cerutti | ||||||||
Assigned To | Dave DeMarle | ||||||||
Priority | normal | Severity | minor | Reproducibility | have not tried | ||||
Status | closed | Resolution | moved | ||||||
Platform | OS | OS Version | |||||||
Product Version | |||||||||
Target Version | Fixed in Version | ||||||||
Summary | 0012375: [patch] buffer vtkImageReader2 | ||||||||
Description | The attached patch adds buffering to the vtkImageReader2 class. This greatly improves performance when reading over a network, e.g., from a CIFS mounted file system. | ||||||||
Tags | No tags attached. | ||||||||
Project | Kitware | ||||||||
Type | performance | ||||||||
Attached Files | ![]() ![]() ![]() | ||||||||
Relationships | |
Relationships |
Notes | |
(0027012) David Gobbi (developer) 2011-07-18 10:55 |
This is a bad idea. Using a buffer that is the full size of the file will double the amount of memory that VTK uses when reading the whole file. You should be able to get the same performance boost by calling UpdateWholeExtent() on the reader. |
(0030249) Pietro Cerutti (reporter) 2013-01-25 12:06 |
David, this is not what I have experienced. The sample program below reads a directory (containing DICOM files) via vtkImageReader2, using SetDataExtent() and UpdateWholeExtent(), and prints the time taken and the resource usage. This is the result of loading a directory containing 145 DICOM files (CT scan), for a total of 74 MB. With my patch: ImageReader2 : 14.745842 sec Memory usage : 91.6367 MB Without my patch: ImageReader2 : 27.874251 sec Memory usage : 91.1953 MB This shows that my patch improves the throughput by ~ 80% and does not affect the memory footprint. ================================================================================================= #include <cstdio> #include <sys/types.h> #include <sys/time.h> #include <sys/resource.h> #include <vtkSmartPointer.h> #include <vtkGlobFileNames.h> #include <vtkImageReader2.h> #include <vtkStringArray.h> #define vtkNew(type, name) vtkSmartPointer<type> name = vtkSmartPointer<type>::New() int main(int argc, char **argv) { if (argc != 2) { std::cerr << "Usage: " << argv[0] << " <path>" << std::endl; return (1); } /* time */ struct timeval beg, end, delta; /* files list */ vtkNew (vtkGlobFileNames, glob); glob->SetDirectory (argv[1]); glob->AddFileNames ("*"); vtkIdType nofFiles = glob->GetNumberOfFileNames (); /* load all files with vtkImageReader2 */ gettimeofday (&beg, NULL); vtkNew (vtkImageReader2, read); read->SetFileNames (glob->GetFileNames()); read->SetDataExtent (0, 511, 0, 511, 0, nofFiles); read->UpdateWholeExtent (); gettimeofday (&end, NULL); timersub (&end, &beg, &delta); std::cout << "ImageReader2 : " << delta.tv_sec << "." << delta.tv_usec << " sec" << std::endl; /* get current resource footprint */ struct rusage r; getrusage (RUSAGE_SELF, &r); std::cout << "Memory usage : " << r.ru_maxrss / 1024. << " MB" << std::endl; return (0); } |
(0030251) David Gobbi (developer) 2013-01-25 13:40 |
The extra memory usage is a temporary and per-file. So if you are reading the image as a whole bunch of slices, you are not going to see much effect. But let's say someone is loading a single 10GB file (i.e. not a whole bunch of slices, but a single very large file). Then memory usage will temporarily spike at 20GB before dropping down to 10GB. If you want this patch to be considered, you should add some methods to the class so that people can choose whether or not they want this feature. |
(0030761) Pietro Cerutti (reporter) 2013-05-14 10:45 |
Ok, I figured out that the double amount of memory is due to the fact that the algorithm copies from the allocated buffer into its datastructure. How would you see this reworked patch? It adds a boolean macro (BufferWholeFile) to enable / disable this feature. In the doxy comments I have made explicit that twice the ram is used, and that this could be of benefit especially over CIFS shares. |
(0030784) Pietro Cerutti (reporter) 2013-05-16 05:28 |
I have a revised patch, which more conform to the VTK style. |
(0030899) Jean-Christophe Fillion-Robin (manager) 2013-06-05 13:24 |
Hi Folks, Is the problem is still present in VTK 6. Would be great if you could submit a patch using Gerrit. For more details see http://www.vtk.org/Wiki/VTK/Git/Develop [^] Thanks Jc |
(0031191) Dave DeMarle (administrator) 2013-07-22 17:46 |
Please submit the patch via gerrit so that it can be more easily reviewed. |
(0037239) Kitware Robot (administrator) 2016-08-12 09:55 |
Resolving issue as `moved`. This issue tracker is no longer used. Further discussion of this issue may take place in the current VTK Issues page linked in the banner at the top of this page. |
Notes |
Issue History | |||
Date Modified | Username | Field | Change |
2011-07-18 10:35 | Pietro Cerutti | New Issue | |
2011-07-18 10:35 | Pietro Cerutti | File Added: patch-vtkImageReader2.diff | |
2011-07-18 10:55 | David Gobbi | Note Added: 0027012 | |
2013-01-25 12:06 | Pietro Cerutti | Note Added: 0030249 | |
2013-01-25 13:40 | David Gobbi | Note Added: 0030251 | |
2013-05-14 10:45 | Pietro Cerutti | Note Added: 0030761 | |
2013-05-14 10:45 | Pietro Cerutti | File Added: patch-IO_vtkImageReader2 | |
2013-05-16 05:28 | Pietro Cerutti | Note Added: 0030784 | |
2013-05-16 05:30 | Pietro Cerutti | File Added: patch-IO_vtkImageReader2-20130516 | |
2013-06-05 13:24 | Jean-Christophe Fillion-Robin | Note Added: 0030899 | |
2013-07-22 17:46 | Dave DeMarle | Note Added: 0031191 | |
2013-07-22 17:46 | Dave DeMarle | Status | backlog => expired |
2013-07-22 17:46 | Dave DeMarle | Assigned To | => Dave DeMarle |
2016-08-12 09:55 | Kitware Robot | Note Added: 0037239 | |
2016-08-12 09:55 | Kitware Robot | Status | expired => closed |
2016-08-12 09:55 | Kitware Robot | Resolution | open => moved |
Issue History |
Copyright © 2000 - 2018 MantisBT Team |