<br><br><div class="gmail_quote">2008/12/2 David E DeMarle <span dir="ltr"><<a href="mailto:dave.demarle@kitware.com">dave.demarle@kitware.com</a>></span><br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div class="Ih2E3d">> One major goal of mine is to allow data to be processed in the background,<br>
> and for it to "appear" on the screen when it is available.<br>
><br>
> To cut a long story short (that long story is below if you keep reading), I<br>
> want to know if VTK's pipeline can be part-push and part-pull ?<br>
<br>
</div>Theoretically one could modify vtkDemandDrivenPipeline (DDP) to<br>
support push. In practice writing a new Executive class would be far<br>
easier because DDP and its decendendents still need to do all the<br>
things there code is fairly complex,<br>
<div class="Ih2E3d"></div></blockquote><div><br>I've noticed a lot of the filters seem to be built to directly support DDP, especially to support streamed or partial processing... so the work may be a lot broader than just a new Executive class.<br>
</div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div class="Ih2E3d"><br>
<br>
><br>
> longer story...<br>
><br>
> I already have this working with a different 3D system... I have a data-push<br>
> system that farms out jobs to a thread pool, and I had thought about<br>
> reimplementing vtkDataDrivenPipeline so that certain jobs could be done in<br>
> the background and when the job is complete, the data becomes available to<br>
> the pipeline.<br>
><br>
> But, I see a few problems with that...<br>
><br>
> 1) I get the impression that when an actor or filter demands data, it<br>
> expects to have the data after the call has completed.<br>
<br>
</div>This is true.<br>
<div class="Ih2E3d"></div></blockquote><div><br>another reason why Push would take a lot more work (i think) because the callers who are demanding data would have to be changed to either<br><br>a) filters expect async data in the middle of operations ... means they would have to be able to postpone their operations until later - difficult/impossible/slow if its in the middle of a lot of GL calls<br>
<br>b) know what it needs first, send the requests upstream, and only when the data is available, then process. forecasting requirements may be impossible in some filters, so it may just degenerate into a non-streaming/partial solution.<br>
<br>If you remove the need to support streaming, then all the filters could just execute and calculate their whole extents and then push that data up the chain... but you would need to ensure that all a filter's sources have finished updating before executing otherwise it would be either wasted computation or worse, executing on data sources that do not match up with each other.<br>
<br><br></div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div class="Ih2E3d"><br>
> This can't be done in my thread-job world... the data is ready when its ready, and its pushed<br>
> through the outputs, not pulled.<br>
><br>
> 2) I am afraid that I'll lose the ability to use MPI to distribute load via<br>
> the streaming system. In my system, the job is processed fully before the<br>
> data is passed on (as a read-only reference or as a copy). Although that<br>
> might be possible if I choose a filter at the end to put in a threaded job,<br>
> and assign all its inputs into MPI executives. Then it'll execute in a<br>
> thread but stream the data from its (MPI) inputs. That will benefit from<br>
> task-level parallelism and better memory usage.<br>
> Note that memory usage is not currently a problem, but things change...<br>
><br>
> 3) This is doing things in threads. As we have said, its bad. I could<br>
> potentially farm out the jobs to an MPI processor pool, which seems to be<br>
> what VTK would do, but I am still stuck with a pull-based pipeline which<br>
> freezes the GUI while it waits for processing to complete.<br>
><br>
><br>
> What could I do?<br>
<br>
</div>I have done this before in ParaView by building a reader which has a<br>
method which can safely determine if the asynchronous process has new<br>
data ready (via asynchronous messages or file status) when it is<br>
ready, the reader calls Modified() on itself. Idle events at the<br>
source end of the pipeline (RenderWindowInteractor for example)<br>
periodically call this method and when necessary update the pipeline<br>
(pull) by calling render. In ParaView it is a bit more complicated<br>
than that because of the dataserver/renderserver/client breakup.<br>
</blockquote><div><br>when the reader calls Modified(), at which point is the data loaded into the pipeline? at the moment i must construct and copy the data into the pipeline while in the main thread, as I cannot build vtkDataObjects in a separate thread.<br>
<br></div></div>thanks,<br>Paul<br><br>