<div dir="ltr">Thanks alot, using Windows swap actually works pretty good.<div>I have another question concerning File I/O. I stumbled upon the file Format ".pvtp"  looking at VTK File Formats (<a href="https://www.vtk.org/wp-content/uploads/2015/04/file-formats.pdf">https://www.vtk.org/wp-content/uploads/2015/04/file-formats.pdf</a>). I would like to write parallel readable pvtp files by timestep. Let's say I have a simulation with 12 timesteps and I want to distribute the reading to 4 cores, letting each processor read 3 steps.</div><div>Unfortunatly I did not find alot of information about it. There was only a similar question (<a href="https://www.vtk.org/pipermail/vtkusers/2009-June/052257.html">https://www.vtk.org/pipermail/vtkusers/2009-June/052257.html</a>) which has seemingly not been answered and one about spatial partitioning (<a href="https://www.vtk.org/pipermail/vtkusers/2017-January/097642.html">https://www.vtk.org/pipermail/vtkusers/2017-January/097642.html</a>). As I would like to read/write parallelly over time, I am stuck at this point. Plus, do I need to build VTK with MPI/ OpenMP packages for using those pvtp-files?</div><div><br></div><div>Thanks alot in advance!</div></div><div class="gmail_extra"><br><div class="gmail_quote">2018-05-03 15:04 GMT+02:00 Andras Lasso <span dir="ltr"><<a href="mailto:lasso@queensu.ca" target="_blank">lasso@queensu.ca</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">> it will be hard to get around streaming data (because of RAM limitations).<br>
<br>
</span>You may increase virtual memory size setting in your system and let the operating system worry about how to save/restore physical memory content to/from disk. You have less control over what is written to disk and when, but everything is done at a very low level and you don't need file parsing, so it may be simpler and faster than re-loading hdf files.<br>
<br>
Andras<br>
<div><div class="h5"><br>
-----Original Message-----<br>
From: vtkusers <<a href="mailto:vtkusers-bounces@vtk.org">vtkusers-bounces@vtk.org</a>> On Behalf Of David E DeMarle<br>
Sent: Thursday, May 3, 2018 8:57 AM<br>
To: Jamil Goettlich <<a href="mailto:jamil.goettlich@gmail.com">jamil.goettlich@gmail.com</a>><br>
Cc: vtk <<a href="mailto:vtkusers@vtk.org">vtkusers@vtk.org</a>><br>
Subject: Re: [vtkusers] Fast File I/O for VTK<br>
<br>
Sorry I can only answer one of your questions at this time.<br>
<br>
On Thu, May 3, 2018 at 6:10 AM, Jamil Goettlich <<a href="mailto:jamil.goettlich@gmail.com">jamil.goettlich@gmail.com</a>> wrote:<br>
> Hello vtkUsers,<br>
><br>
> I am planning on writing some code for visualizing time sequential <br>
> data and figured out that it will be hard to get around streaming data <br>
> (because of RAM limitations). I am currently using vtp files for <br>
> storing. They are loaded into multiblocks of vtkPolyData to be <br>
> visualized further on. Since the data can be pretty big, its taking a <br>
> while to import them, hence streaming data is alot about fast file I/O (please correct me if I'm wrong).<br>
> I am currently reading vtp files (only Point Data) with about six <br>
> attributes (velocity, acceleration, pressure, ...) and with the <br>
> vtkDataArraySelectionClass I am able to read in only selected arrays, <br>
> which makes everything faster. I am happy with that, but as I do not <br>
> know anything about File I/O, I guess there is alot of optimization possible.<br>
><br>
> My questions are the following and I'm happy about every answer on one <br>
> of<br>
> them:<br>
> - I realized that for reading XML Data, there is only one CPU core <br>
> being used. Is there a (c++) implementation to read parallelly?<br>
> - When I stored all timestep files in vtkMultiBlockData (collection of <br>
> vtp) and imported those ones with XMLMultiblockDataReader, the import <br>
> took even longer than manually importing each timestep into polyData <br>
> and putting them into a multiblock afterwards. Is it possible to <br>
> parallelly read files of the multiblock, so reading each block is <br>
> distributed on different cores? (Sorry if the question seems dumb, I <br>
> have no idea of parallellism)<br>
> - Has anyone got experience using the HDF5 format for storing and <br>
> reading polydata?And is it faster than reading XML data/ are multiple <br>
> cores being used? If so, how much effort does it take to write a <br>
> custom HDF5 reader/writer (as I did not find any vtk-c++ implementation on the web)?<br>
<br>
You could look at the source code of IO/Truchas. That reader took me perhaps a week of effort to get to where it is now.<br>
<br>
It makes unstructured grids of volumetric cells types. For somple points your code will be less complex.<br>
<br>
> - Or is there any better/ faster/ more flexible/ parallel file format <br>
> than HDF5/vtp for this purpose?<br>
> - Is there some good (vtk related) literature concerning that topic?<br>
><br>
> I am sorry for asking many general questions at once but I thought it <br>
> would help many users that struggle with the same problems to get an <br>
> overview. I would be happy for any answers/experiences.<br>
><br>
> Thanks alot<br>
><br>
> Jamil<br>
><br>
><br>
><br>
> ______________________________<wbr>_________________<br>
> Powered by <br>
</div></div>> <a href="https://na01.safelinks.protection.outlook.com/?url=www.kitware.com&dat" rel="noreferrer" target="_blank">https://na01.safelinks.<wbr>protection.outlook.com/?url=<wbr>www.kitware.com&dat</a><br>
> a=02%7C01%7Classo%<a href="http://40queensu.ca" rel="noreferrer" target="_blank">40queensu.ca</a><wbr>%<wbr>7C5d51857638d640abfce308d5b0f5<wbr>757a%7Cd6<br>
> 1ecb3b38b142d582c4efb2838b925c<wbr>%7C1%7C0%7C636609490646480690&<wbr>sdata=ZlZe<br>
> F0bsckE9EbKlv9jspraeoJNwfP0Oze<wbr>V2MTyDeyQ%3D&reserved=0<br>
<span class="">><br>
> Visit other Kitware open-source projects at<br>
</span>> <a href="https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.ki" rel="noreferrer" target="_blank">https://na01.safelinks.<wbr>protection.outlook.com/?url=<wbr>http%3A%2F%2Fwww.ki</a><br>
> <a href="http://tware.com" rel="noreferrer" target="_blank">tware.com</a>%2Fopensource%<wbr>2Fopensource.html&data=02%<wbr>7C01%7Classo%40queens<br>
> <a href="http://u.ca" rel="noreferrer" target="_blank">u.ca</a>%<wbr>7C5d51857638d640abfce308d5b0f5<wbr>757a%<wbr>7Cd61ecb3b38b142d582c4efb2838b<br>
> 925c%7C1%7C0%<wbr>7C636609490646480690&sdata=<wbr>Yc6YnPgFfYMzrRIIX8FdjWNQe40UbF<br>
> ojCPyQ7CDzckY%3D&reserved=0<br>
<span class="">><br>
> Please keep messages on-topic and check the VTK FAQ at:<br>
</span>> <a href="https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.vt" rel="noreferrer" target="_blank">https://na01.safelinks.<wbr>protection.outlook.com/?url=<wbr>http%3A%2F%2Fwww.vt</a><br>
> <a href="http://k.org" rel="noreferrer" target="_blank">k.org</a>%2FWiki%2FVTK_FAQ&data=<wbr>02%7C01%7Classo%<a href="http://40queensu.ca" rel="noreferrer" target="_blank">40queensu.ca</a>%<wbr>7C5d51857638d<br>
> 640abfce308d5b0f5757a%<wbr>7Cd61ecb3b38b142d582c4efb2838b<wbr>925c%7C1%7C0%7C636<br>
> 609490646480690&sdata=<wbr>w8ravzj8oob5nwoV6WNIf3ITYJdJAk<wbr>DmfINUy3wzR54%3D&r<br>
> eserved=0<br>
<span class="">><br>
> Search the list archives at: <br>
</span>> <a href="https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmarkma" rel="noreferrer" target="_blank">https://na01.safelinks.<wbr>protection.outlook.com/?url=<wbr>http%3A%2F%2Fmarkma</a><br>
> <a href="http://il.org" rel="noreferrer" target="_blank">il.org</a>%2Fsearch%2F%3Fq%<wbr>3Dvtkusers&data=02%7C01%<wbr>7Classo%<a href="http://40queensu.ca" rel="noreferrer" target="_blank">40queensu.ca</a>%7C<br>
> 5d51857638d640abfce308d5b0f575<wbr>7a%<wbr>7Cd61ecb3b38b142d582c4efb2838b<wbr>925c%7C<br>
> 1%7C0%7C636609490646480690&<wbr>sdata=hDaOmlmp%<wbr>2FMsuC2E463IMIAfdHqRZRI4rtAX<br>
> 1x%2BuEDdo%3D&reserved=0<br>
<span class="">><br>
> Follow this link to subscribe/unsubscribe:<br>
</span>> <a href="https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fvtk.o" rel="noreferrer" target="_blank">https://na01.safelinks.<wbr>protection.outlook.com/?url=<wbr>https%3A%2F%2Fvtk.o</a><br>
> rg%2Fmailman%2Flistinfo%<wbr>2Fvtkusers&data=02%7C01%<wbr>7Classo%<a href="http://40queensu.ca" rel="noreferrer" target="_blank">40queensu.ca</a>%7<br>
> C5d51857638d640abfce308d5b0f57<wbr>57a%<wbr>7Cd61ecb3b38b142d582c4efb2838b<wbr>925c%7<br>
> C1%7C0%7C636609490646480690&<wbr>sdata=<wbr>tZytuI3tWINvEA9vLmYV8q52awE56K<wbr>pUGpaw<br>
> yKsn9JM%3D&reserved=0<br>
><br>
______________________________<wbr>_________________<br>
Powered by <a href="https://na01.safelinks.protection.outlook.com/?url=www.kitware.com&data=02%7C01%7Classo%40queensu.ca%7C5d51857638d640abfce308d5b0f5757a%7Cd61ecb3b38b142d582c4efb2838b925c%7C1%7C0%7C636609490646480690&sdata=ZlZeF0bsckE9EbKlv9jspraeoJNwfP0OzeV2MTyDeyQ%3D&reserved=0" rel="noreferrer" target="_blank">https://na01.safelinks.<wbr>protection.outlook.com/?url=<wbr>www.kitware.com&data=02%7C01%<wbr>7Classo%40queensu.ca%<wbr>7C5d51857638d640abfce308d5b0f5<wbr>757a%<wbr>7Cd61ecb3b38b142d582c4efb2838b<wbr>925c%7C1%7C0%<wbr>7C636609490646480690&sdata=<wbr>ZlZeF0bsckE9EbKlv9jspraeoJNwfP<wbr>0OzeV2MTyDeyQ%3D&reserved=0</a><br>
<br>
Visit other Kitware open-source projects at <a href="https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.kitware.com%2Fopensource%2Fopensource.html&data=02%7C01%7Classo%40queensu.ca%7C5d51857638d640abfce308d5b0f5757a%7Cd61ecb3b38b142d582c4efb2838b925c%7C1%7C0%7C636609490646480690&sdata=Yc6YnPgFfYMzrRIIX8FdjWNQe40UbFojCPyQ7CDzckY%3D&reserved=0" rel="noreferrer" target="_blank">https://na01.safelinks.<wbr>protection.outlook.com/?url=<wbr>http%3A%2F%2Fwww.kitware.com%<wbr>2Fopensource%2Fopensource.<wbr>html&data=02%7C01%7Classo%<wbr>40queensu.ca%<wbr>7C5d51857638d640abfce308d5b0f5<wbr>757a%<wbr>7Cd61ecb3b38b142d582c4efb2838b<wbr>925c%7C1%7C0%<wbr>7C636609490646480690&sdata=<wbr>Yc6YnPgFfYMzrRIIX8FdjWNQe40UbF<wbr>ojCPyQ7CDzckY%3D&reserved=0</a><br>
<br>
Please keep messages on-topic and check the VTK FAQ at: <a href="https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.vtk.org%2FWiki%2FVTK_FAQ&data=02%7C01%7Classo%40queensu.ca%7C5d51857638d640abfce308d5b0f5757a%7Cd61ecb3b38b142d582c4efb2838b925c%7C1%7C0%7C636609490646480690&sdata=w8ravzj8oob5nwoV6WNIf3ITYJdJAkDmfINUy3wzR54%3D&reserved=0" rel="noreferrer" target="_blank">https://na01.safelinks.<wbr>protection.outlook.com/?url=<wbr>http%3A%2F%2Fwww.vtk.org%<wbr>2FWiki%2FVTK_FAQ&data=02%7C01%<wbr>7Classo%40queensu.ca%<wbr>7C5d51857638d640abfce308d5b0f5<wbr>757a%<wbr>7Cd61ecb3b38b142d582c4efb2838b<wbr>925c%7C1%7C0%<wbr>7C636609490646480690&sdata=<wbr>w8ravzj8oob5nwoV6WNIf3ITYJdJAk<wbr>DmfINUy3wzR54%3D&reserved=0</a><br>
<br>
Search the list archives at: <a href="https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmarkmail.org%2Fsearch%2F%3Fq%3Dvtkusers&data=02%7C01%7Classo%40queensu.ca%7C5d51857638d640abfce308d5b0f5757a%7Cd61ecb3b38b142d582c4efb2838b925c%7C1%7C0%7C636609490646480690&sdata=hDaOmlmp%2FMsuC2E463IMIAfdHqRZRI4rtAX1x%2BuEDdo%3D&reserved=0" rel="noreferrer" target="_blank">https://na01.safelinks.<wbr>protection.outlook.com/?url=<wbr>http%3A%2F%2Fmarkmail.org%<wbr>2Fsearch%2F%3Fq%3Dvtkusers&<wbr>data=02%7C01%7Classo%<wbr>40queensu.ca%<wbr>7C5d51857638d640abfce308d5b0f5<wbr>757a%<wbr>7Cd61ecb3b38b142d582c4efb2838b<wbr>925c%7C1%7C0%<wbr>7C636609490646480690&sdata=<wbr>hDaOmlmp%<wbr>2FMsuC2E463IMIAfdHqRZRI4rtAX1x<wbr>%2BuEDdo%3D&reserved=0</a><br>
<span class=""><br>
Follow this link to subscribe/unsubscribe:<br>
</span><a href="https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fvtk.org%2Fmailman%2Flistinfo%2Fvtkusers&data=02%7C01%7Classo%40queensu.ca%7C5d51857638d640abfce308d5b0f5757a%7Cd61ecb3b38b142d582c4efb2838b925c%7C1%7C0%7C636609490646480690&sdata=tZytuI3tWINvEA9vLmYV8q52awE56KpUGpawyKsn9JM%3D&reserved=0" rel="noreferrer" target="_blank">https://na01.safelinks.<wbr>protection.outlook.com/?url=<wbr>https%3A%2F%2Fvtk.org%<wbr>2Fmailman%2Flistinfo%<wbr>2Fvtkusers&data=02%7C01%<wbr>7Classo%40queensu.ca%<wbr>7C5d51857638d640abfce308d5b0f5<wbr>757a%<wbr>7Cd61ecb3b38b142d582c4efb2838b<wbr>925c%7C1%7C0%<wbr>7C636609490646480690&sdata=<wbr>tZytuI3tWINvEA9vLmYV8q52awE56K<wbr>pUGpawyKsn9JM%3D&reserved=0</a><br>
</blockquote></div><br></div>