[vtk-developers] vtkImageConvolve crashes when malloc fails

[Mathieu Coursolle]

On 4-Jun-08, at 3:57 PM, Berk Geveci wrote:
>
> It is surprising to me that you are experiencing this with a 100Mb
> dataset (which not that big). Do you have a lot of 100Mb data sets
> maybe? Assuming that you are compiling 32 bit, it would take quite a
> few 100Mb objects to fragment the virtual memory (2 Gb)  enough to
> cause problems. Also, this may be caused by a misbehaving algorithm,
> if it grows an array slowly to get to 100Mb causing a lot of reallocs.

I could reproduce the problem without VTK with a simple App:

bool ok = true;
unsigned long idx = 0;
do
{
	char* x = malloc(100000000);  // 100 MBytes
	if (x)
	{
		free(x);
                 idx++;
	}
	else
	{
		ok = false;
	}
}
while (ok);

At the end, the loop is executed less than 50 times.

I could reproduce the same behavior by applying vtkImageConvolve to a  
100 MBytes image data in the loop.

>>
>
> I agree. VTK's handling of memory allocation failures is far from
> optimal. How would you address it?

When an algorithm is executed, is there a return value indicating it  
did not succeeded, or should we have to inspect
the returning value? vtkAlgorithm update method returns void.

Thanks!

Mathieu



>
>
> -berk
>
>
>>> The parallel computing experts should address this, but basically  
>>> the
>>> solution to large data is to stream the data in pieces. While this  
>>> is not
>>> always possible, VTK provides good support for this especially for  
>>> image
>>> data. This means that your source object must be able to respond  
>>> to data
>>> requests that are less than the whole extent of the image.
>>>
>>> Also, make sure that you are setting the release data flags, etc. to
>>> discard
>>> memory in the pipeline when it is no longer needed.
>>>
>>> W
>>>
>>> On Tue, Jun 3, 2008 at 4:37 PM, Berk Geveci  
>>> <berk.geveci at kitware.com>
>>> wrote:
>>>
>>>> If memory allocation fails, you will run into a crash sooner or  
>>>> later
>>>> unfortunately. Even if you were to add such a check, somewhere down
>>>> the pipeline some other class is going to die. IMHO, the only
>>>> "graceful" way of handling this is to throw an exception that the
>>>> application catches. Then it would have to report an error and shut
>>>> down. Adding bunch of if statements for an exceptional case is
>>>> probably not a good idea.
>>>>
>>>> -berk
>>>>
>>>> 2008/6/3 Mathieu Coursolle <mcoursolle at rogue-research.com>:
>>>>> Hi VTK developers,
>>>>> I am performing a series of filters, where some of them are
>>>>> vtkImageConvolve.
>>>>> My input vtkImageData is usually close to 100Mbytes, and the  
>>>>> execution
>>>> of
>>>> my
>>>>> pipeline
>>>>> seems to cause some memory fragmentation. Therefore, malloc will
>>>> eventually
>>>>> fail, causing
>>>>> a crash in vtkImageConvolve.
>>>>> If malloc fails during the execution of a pipeline, then the
>>>> scalarPointer
>>>>> of the resulting vtkImageData is NULL. If a connected  
>>>>> vtkImageConvolve
>>>>> filter receives such a vtkImageData
>>>>> as its input, it tries to access the scalar pointer (which is  
>>>>> NULL),
>>>> causing
>>>>> a crash.
>>>>> A solution would be to check the pointers:
>>>>>  // ADDED CODE BEGIN--------
>>>>>  // Make sure the scalar pointers are valid.
>>>>>  if (!inData[0][0]->GetScalarPointer() ||
>>>> !outData[0]->GetScalarPointer())
>>>>>    {
>>>>> vtkErrorMacro(<< "Execute: input and/or output image does not  
>>>>> have any
>>>>> scalar value.");
>>>>>    return;
>>>>>    }
>>>>>  // ADDED CODE END--------
>>>>>
>>>>>
>>>>>
>>>>>  switch (inData[0][0]->GetScalarType())
>>>>>    {
>>>>>    vtkTemplateMacro(
>>>>>      vtkImageConvolveExecute(this, inData[0][0],
>>>>>                              static_cast<VTK_TT *>(inPtr),
>>>> outData[0],
>>>>>                              static_cast<VTK_TT *>(outPtr),
>>>>>                              outExt, id, inInfo));
>>>>>    default:
>>>>>      vtkErrorMacro(<< "Execute: Unknown ScalarType");
>>>>>      return;
>>>>>    }
>>>>> So I was wondering,
>>>>> 1) Is that the right way to avoid such crashes?
>>>>> 2) I guess that problem might occur with other filters as well...
>>>>> Thanks.
>>>>> Mathieu
>>>>>
>>>>> _______________________________________________
>>>>> vtk-developers mailing list
>>>>> vtk-developers at vtk.org
>>>>> http://www.vtk.org/mailman/listinfo/vtk-developers
>>>>>
>>>>>
>>>> _______________________________________________
>>>> vtk-developers mailing list
>>>> vtk-developers at vtk.org
>>>> http://www.vtk.org/mailman/listinfo/vtk-developers
>>>>
>>>
>>>
>>>
>>> --
>>> William J. Schroeder, PhD
>>> Kitware, Inc.
>>> 28 Corporate Drive
>>> Clifton Park, NY 12065
>>> will.schroeder at kitware.com
>>> http://www.kitware.com
>>> 518-371-3971 (phone and fax)
>>>
>>
>>
>