MantisBT - ParaView
View Issue Details
IDProjectCategoryView StatusDate SubmittedLast Update
0010178ParaViewFeaturepublic2010-01-22 13:242010-03-09 22:08
ReporterKen Moreland 
Assigned ToKen Moreland 
PriorityimmediateSeverityminorReproducibilityhave not tried
StatusclosedResolutionfixed 
PlatformOSOS Version
Product Version 
Target Version3.8Fixed in Version3.8 
Project
Topic Name
Type
Summary0010178: Efficient raw image reads on parallel file systems
DescriptionWhen reading raw image data from a file, the current image reader does uncoordinated reads of the data. When many processes are each trying to read a 3D subblock from a file, each process requests many small strips of data from the file. When a parallel file system gets saturated with lots of these small requests, the devices get swamped trying to satisfy all the independent requests.

In contrast, reading from a parallel file system can be very efficient when reading large blocks of data. If the reads requests from the independent process are consolidated, they can be satisfied with one or a few large reads. MPI version 2 has a set of IO commands to allow you to do just this. I propose creating a subset of the image reader that takes advantage of the MPI IO calls to make parallel reads efficient.
Steps To Reproduce
Additional Information
TagsNo tags attached.
Relationships
Attached Filespatch MPIImageReader.patch (65,498) 2010-01-22 13:27
https://www.vtk.org/Bug/file/7797/MPIImageReader.patch
Issue History
Date ModifiedUsernameFieldChange
2010-01-22 13:24Ken MorelandNew Issue
2010-01-22 13:24Ken MorelandStatusbacklog => tabled
2010-01-22 13:24Ken MorelandAssigned To => Ken Moreland
2010-01-22 13:27Ken MorelandFile Added: MPIImageReader.patch
2010-01-22 13:34Ken MorelandNote Added: 0019273
2010-01-22 13:34Ken MorelandAssigned ToKen Moreland => Utkarsh Ayachit
2010-02-04 13:04Utkarsh AyachitPrioritynormal => immediate
2010-02-10 09:34Utkarsh AyachitAssigned ToUtkarsh Ayachit => Berk Geveci
2010-03-01 09:11Utkarsh AyachitAssigned ToBerk Geveci => Ken Moreland
2010-03-05 12:30Ken MorelandNote Added: 0019729
2010-03-05 12:30Ken MorelandStatustabled => @80@
2010-03-05 12:30Ken MorelandFixed in Version => 3.8
2010-03-05 12:30Ken MorelandResolutionopen => fixed
2010-03-09 22:08Alan ScottNote Added: 0019792
2010-03-09 22:08Alan ScottStatus@80@ => closed
2011-06-16 13:10Zack GalbreathCategoryFeature Request => Feature

Notes
(0019273)
Ken Moreland   
2010-01-22 13:34   
The issue is fixed. Because this fix touches some of the base IO and parallel communication files, someone from Kitware should look at these changes and sign off before I check it in. I have assigned the bug to Utkarsh so that he knows he needs to look at the change. He should assign the bug back to me once he reviews the code.

The change is attached as a patch to this report. I have also posted the change to a branch of my github repository. The repository can be cloned from the url <git://github.com/kmorel/ParaView.git> [^] and is in the <MPIImageReader> branch. The branch is also summarized at this github.com web site.

http://github.com/kmorel/ParaView/tree/MPIImageReader [^]
(0019729)
Ken Moreland   
2010-03-05 12:30   
/cvsroot/VTKData/VTKData/Baseline/Parallel/ParallelIso.7proc.png,v <-- ParallelIso.7proc.png
initial revision: 1.1
/cvsroot/VTKData/VTKData/Data/headsq/quarter.nhdr,v <-- quarter.nhdr
initial revision: 1.1
/cvsroot/ParaView3/ParaView3/Applications/ParaView/ParaViewReaders.xml,v <-- Applications/ParaView/ParaViewReaders.xml
new revision: 1.6; previous revision: 1.5
/cvsroot/ParaView3/ParaView3/Servers/ServerManager/Resources/readers.xml,v <-- Servers/ServerManager/Resources/readers.xml
new revision: 1.197; previous revision: 1.196
/cvsroot/ParaView3/ParaView3/VTK/IO/vtkImageReader.cxx,v <-- VTK/IO/vtkImageReader.cxx
new revision: 1.123; previous revision: 1.122
/cvsroot/ParaView3/ParaView3/VTK/IO/vtkImageReader.h,v <-- VTK/IO/vtkImageReader.h
new revision: 1.74; previous revision: 1.73
/cvsroot/ParaView3/ParaView3/VTK/Parallel/CMakeLists.txt,v <-- VTK/Parallel/CMakeLists.txt
new revision: 1.204; previous revision: 1.203
/cvsroot/ParaView3/ParaView3/VTK/Parallel/vtkMPI.h,v <-- VTK/Parallel/vtkMPI.h
new revision: 1.7; previous revision: 1.6
/cvsroot/ParaView3/ParaView3/VTK/Parallel/vtkMPICommunicator.cxx,v <-- VTK/Parallel/vtkMPICommunicator.cxx
new revision: 1.56; previous revision: 1.55
/cvsroot/ParaView3/ParaView3/VTK/Parallel/vtkMPICommunicator.h,v <-- VTK/Parallel/vtkMPICommunicator.h
new revision: 1.43; previous revision: 1.42
/cvsroot/ParaView3/ParaView3/VTK/Parallel/vtkMPIController.cxx,v <-- VTK/Parallel/vtkMPIController.cxx
new revision: 1.29; previous revision: 1.28
/cvsroot/ParaView3/ParaView3/VTK/Parallel/vtkMPIController.h,v <-- VTK/Parallel/vtkMPIController.h
new revision: 1.26; previous revision: 1.25
/cvsroot/ParaView3/ParaView3/VTK/Parallel/vtkMPIImageReader.cxx,v <-- VTK/Parallel/vtkMPIImageReader.cxx
initial revision: 1.1
/cvsroot/ParaView3/ParaView3/VTK/Parallel/vtkMPIImageReader.h,v <-- VTK/Parallel/vtkMPIImageReader.h
initial revision: 1.1
/cvsroot/ParaView3/ParaView3/VTK/Parallel/vtkMultiProcessController.cxx,v <-- VTK/Parallel/vtkMultiProcessController.cxx
new revision: 1.39; previous revision: 1.38
/cvsroot/ParaView3/ParaView3/VTK/Parallel/vtkMultiProcessController.h,v <-- VTK/Parallel/vtkMultiProcessController.h
new revision: 1.45; previous revision: 1.44
/cvsroot/ParaView3/ParaView3/VTK/Parallel/vtkPNrrdReader.cxx,v <-- VTK/Parallel/vtkPNrrdReader.cxx
initial revision: 1.1
/cvsroot/ParaView3/ParaView3/VTK/Parallel/vtkPNrrdReader.h,v <-- VTK/Parallel/vtkPNrrdReader.h
initial revision: 1.1
/cvsroot/ParaView3/ParaView3/VTK/Parallel/Testing/Cxx/CMakeLists.txt,v <-- VTK/Parallel/Testing/Cxx/CMakeLists.txt
new revision: 1.93; previous revision: 1.92
/cvsroot/ParaView3/ParaView3/VTK/Parallel/Testing/Cxx/ExerciseMultiProcessController.cxx,v <-- VTK/Parallel/Testing/Cxx/ExerciseMultiProcessController.cxx
new revision: 1.11; previous revision: 1.10
/cvsroot/ParaView3/ParaView3/VTK/Parallel/Testing/Cxx/ParallelIso.cxx,v <-- VTK/Parallel/Testing/Cxx/ParallelIso.cxx
new revision: 1.28; previous revision: 1.27
cvs update: warning: `vtkMPIImageReader.h' was lost
===================================================================
Checking out vtkMPIImageReader.h
RCS: /cvsroot/ParaView3/ParaView3/VTK/Parallel/vtkMPIImageReader.h,v
VERS: 1.1
***************
cvs update: warning: `vtkPNrrdReader.h' was lost
===================================================================
Checking out vtkPNrrdReader.h
RCS: /cvsroot/ParaView3/ParaView3/VTK/Parallel/vtkPNrrdReader.h,v
VERS: 1.1
***************
cvs update: warning: `vtkPNrrdReader.cxx' was lost
===================================================================
Checking out vtkPNrrdReader.cxx
RCS: /cvsroot/ParaView3/ParaView3/VTK/Parallel/vtkPNrrdReader.cxx,v
VERS: 1.1
***************
cvs update: warning: `vtkMPIImageReader.cxx' was lost
===================================================================
Checking out vtkMPIImageReader.cxx
RCS: /cvsroot/ParaView3/ParaView3/VTK/Parallel/vtkMPIImageReader.cxx,v
VERS: 1.1
***************
(0019792)
Alan Scott   
2010-03-09 22:08   
I tried this back about a week ago on the small supernova, and it worked fine. Remote servers, Trunk.