Hello list,<br><br>Can someone look at this simple pipeline and explain if I'm using the StreamingPipeline correctly? It's operating on a set of 6k x 6k images, but any large image set would work. I would expect that the streamer would help reduce the memory usage, but I'm blowing the memory on my mac no matter how I set it,<br>
<br>Note: the image reading code is not the most efficient. We were having some issues reading these large images and were experimenting with different techniques. Suggestions for a more efficient pipeline is appreciated. Also, note that in the real application were reading in two different 6k x 6k image stacks and overlaying them.<br>
<br>Thanks,<br><br>Mark<br><br>File:<br># ------------------------- imports -------------------------<br>import os<br>import vtk<br><br><br># ------------------------- constants -------------------------<br><br>graydir = "/Users/bolstadm/Desktop/6k_6k_150"<br>
grayfilepattern = "image.v0.%05d.png"<br><br># sp maps instead:<br># graydir = os.path.join(basedir, "superpixel_maps")<br># grayfilepattern = "sp_map.v0.%05d.png"<br><br># there are 30 numbered files in above dir<br>
nfiles = 14<br><br>def printEvent(a, b):<br> print 'event called for piece', a.GetExtentTranslator().GetPiece(), <br> print 'out of', a.GetExtentTranslator().GetNumberOfPieces()<br> print 'with extent ', a.GetExtentTranslator().GetExtent()<br>
<br><br>def printExecEvent(a, b):<br> print 'event called for piece', a<br> <br><br># ------------------------- script starts here -------------------------<br>if __name__ == "__main__":<br>
# try to reproduce our failure<br> <br> print "working with %d files" % nfiles<br> print "creating factory, first reader"<br> firstfile = os.path.join(graydir, grayfilepattern % 1) <br>
factory = vtk.vtkImageReader2Factory()<br> reader = factory.CreateImageReader2(firstfile)<br> <br> <br> # switch between two reader schemes:<br> if True:<br> # use a file appender to read multiple images in <br>
# multiple readers<br> print "using multiple readers and ImageAppend"<br> <br> # grab the image size:<br> reader.SetFileName(firstfile)<br> reader.Update()<br> extent = reader.GetDataExtent()<br>
print "extent =", extent<br> <br> # this will be our data structure; axis 2 is z:<br> grayreader0 = vtk.vtkImageAppend()<br> the_exec = vtk.vtkStreamingDemandDrivenPipeline()<br>
grayreader0.SetExecutive( the_exec )<br> grayreader0.AddObserver("EndEvent", printExecEvent)<br> grayreader0.SetAppendAxis(2)<br> <br> # add first plane: will end up in 0-th plane<br>
grayreader0.AddInput(reader.GetOutput())<br> <br> # don't reuse readers; create one per file:<br> grayreaderlist = {}<br> tilelist = [os.path.join(graydir, grayfilepattern % i)<br> for i in range(1, nfiles + 1)]<br>
print "creating readers"<br> for tilepath in tilelist:<br> print tilepath<br> reader = factory.CreateImageReader2(tilepath)<br> reader.SetFileName(tilepath)<br> grayreaderlist[tilepath] = vtk.vtkMemoryLimitImageDataStreamer()<br>
grayreaderlist[tilepath].SetInputConnection(reader.GetOutputPort())<br> grayreaderlist[tilepath].SetNumberOfStreamDivisions(16)<br> grayreaderlist[tilepath].SetMemoryLimit( 256 * 1024 )<br>
# make sure we get the correct translator.<br> #self.imageStreamer.UpdateInformation()<br> grayreaderlist[tilepath].GetExtentTranslator().SetSplitModeToBlock()<br> grayreaderlist[tilepath].AddObserver("EndEvent", printEvent)<br>
grayreader0.AddInput(grayreaderlist[tilepath].GetOutput())<br> <br> # data gets read from disk here<br> print "updating ImageAppend"<br> grayreader0.Update()<br> grayreader0.UpdateWholeExtent()<br>
<br> else:<br> # read in one reader with a file pattern<br> print "using a single reader" <br> <br> # hardcode for now--diff. prefix/pattern than above<br> reader.SetFilePrefix(os.path.join(basedir, graydir) + "/image.v0.")<br>
reader.SetFilePattern("%s%05d.png")<br> reader.SetFileNameSliceOffset(1)<br> reader.Update()<br> extent = reader.GetDataExtent()<br> print "extent =", extent<br> <br>
# no ImageAppend, so just pass this reader along:<br> grayreader0 = reader<br> <br> print 'Setting a streaming executive'<br> the_exec.SetUpdateNumberOfPieces(grayreader0.GetExecutive().GetOutputInformation(0), 16);<br>
the_exec.GetExtentTranslator( grayreader0.GetExecutive().GetOutputInformation(0) ).SetSplitModeToBlock()<br><br> # debug<br> # import pdb; pdb.set_trace()<br> <br> <br> print "creating color tables"<br>
grayreader = vtk.vtkImageMapToColors()<br> <br> grayreader.SetInputConnection(grayreader0.GetOutputPort())<br> graylut = vtk.vtkLookupTable()<br> graylut.SetTableRange(0, 256)<br> # can replace these values with those from spmap, doesn't matter<br>
graylut.SetHueRange(0, 0)<br> graylut.SetSaturationRange(0, 0)<br> graylut.SetValueRange(0, 1)<br> graylut.Build()<br> grayreader.SetLookupTable(graylut)<br> <br> # this will fail for 14+ planes, with imageappend<br>
print "updating ImageMapToColors"<br> grayreader.Update()<br> <br> print "done!"<br><br>----------------------------------------------------------------------------<br>Output:<br><br>[bolstadm-lm1:~/test/VTK] bolstadm% python stress-v1.py<br>
working with 14 files<br>creating factory, first reader<br>using multiple readers and ImageAppend<br>extent = (0, 6033, 0, 6209, 0, 0)<br>creating readers<br>/Users/bolstadm/Desktop/6k_6k_150/image.v0.00001.png<br>/Users/bolstadm/Desktop/6k_6k_150/image.v0.00002.png<br>
/Users/bolstadm/Desktop/6k_6k_150/image.v0.00003.png<br>/Users/bolstadm/Desktop/6k_6k_150/image.v0.00004.png<br>/Users/bolstadm/Desktop/6k_6k_150/image.v0.00005.png<br>/Users/bolstadm/Desktop/6k_6k_150/image.v0.00006.png<br>
/Users/bolstadm/Desktop/6k_6k_150/image.v0.00007.png<br>/Users/bolstadm/Desktop/6k_6k_150/image.v0.00008.png<br>/Users/bolstadm/Desktop/6k_6k_150/image.v0.00009.png<br>/Users/bolstadm/Desktop/6k_6k_150/image.v0.00010.png<br>
/Users/bolstadm/Desktop/6k_6k_150/image.v0.00011.png<br>/Users/bolstadm/Desktop/6k_6k_150/image.v0.00012.png<br>/Users/bolstadm/Desktop/6k_6k_150/image.v0.00013.png<br>/Users/bolstadm/Desktop/6k_6k_150/image.v0.00014.png<br>
updating ImageAppend<br>event called for piece 0 out of 1<br>with extent (0, 6033, 0, 6209, 0, 0)<br>event called for piece 0 out of 1<br>with extent (0, 6033, 0, 6209, 0, 0)<br>event called for piece 0 out of 1<br>with extent (0, 6033, 0, 6209, 0, 0)<br>
event called for piece 0 out of 1<br>with extent (0, 6033, 0, 6209, 0, 0)<br>event called for piece 0 out of 1<br>with extent (0, 6033, 0, 6209, 0, 0)<br>event called for piece 0 out of 1<br>with extent (0, 6033, 0, 6209, 0, 0)<br>
event called for piece 0 out of 1<br>with extent (0, 6033, 0, 6209, 0, 0)<br>event called for piece 0 out of 1<br>with extent (0, 6033, 0, 6209, 0, 0)<br>event called for piece 0 out of 1<br>with extent (0, 6033, 0, 6209, 0, 0)<br>
event called for piece 0 out of 1<br>with extent (0, 6033, 0, 6209, 0, 0)<br>event called for piece 0 out of 1<br>with extent (0, 6033, 0, 6209, 0, 0)<br>event called for piece 0 out of 1<br>with extent (0, 6033, 0, 6209, 0, 0)<br>
event called for piece 0 out of 1<br>with extent (0, 6033, 0, 6209, 0, 0)<br>event called for piece 0 out of 1<br>with extent (0, 6033, 0, 6209, 0, 0)<br>event called for piece vtkImageAppend (0x47eaf0)<br> Debug: Off<br>
Modified Time: 4336<br> Reference Count: 2<br> Registered Events: <br> Registered Observers:<br> vtkObserver (0x481320)<br> Event: 4<br> EventName: EndEvent<br> Command: 0x4812c0<br> Priority: 0<br>
Tag: 1<br> Executive: 0x481260<br> ErrorCode: Unknown error: 0<br> Information: 0x47e430<br> AbortExecute: Off<br> Progress: 1<br> Progress Text: (None)<br> NumberOfThreads: 2<br> AppendAxis: 2<br> PreserveExtents: 0<br>
<br><br>Setting a streaming executive<br>creating color tables<br>updating ImageMapToColors<br>Python(33833,0xa079a720) malloc: *** mmap(size=2248269824) failed (error code=12)<br>*** error: can't allocate region<br>*** set a breakpoint in malloc_error_break to debug<br>
ERROR: In /Users/bolstadm/Projects/SCE/build/src/VTK5.2.0/Common/vtkDataArrayTemplate.txx, line 137<br>vtkUnsignedCharArray (0x4c1b40): Unable to allocate 2248268400 elements of size 1 bytes. <br><br>ERROR: In /Users/bolstadm/Projects/SCE/build/src/VTK5.2.0/Filtering/vtkImageData.cxx, line 2117<br>
vtkImageData (0x4c0300): Coordinate (0, 0, 0) out side of array (max = -1<br><br>ERROR: In /Users/bolstadm/Projects/SCE/build/src/VTK5.2.0/Filtering/vtkImageData.cxx, line 2117<br>vtkImageData (0x4c0300): Coordinate (0, 0, 8) out side of array (max = -1<br>
<br>Bus error<br><br>