1. main@rasterio.groups.io

Re: tempfile.NamedTemporaryFile behaving as /vsimem and eating all the machine memory

Sean Gillies
 

Hi Vincent,

This is expected (if not well-documented) behavior. tempfile.NamedTemporaryFile() returns an open Python file object, not a filename. GDAL can't use a Python file object, so in that case rasterio.open reads all the bytes from the file object, copies them to the vsimem filesystem, and opens that vsimem file.

I think what you want do do is pass the name of the temp file object to GDAL. Like this:

with tempfile.NamedTemporaryFile() as temp:
    with rasterio.open(temp.name) as dataset:
        print(dataset)

No copy in the vsimem filesystem will be made.

On Tue, Apr 16, 2019 at 6:55 AM <vincent.sarago@...> wrote:
While working on https://github.com/cogeotiff/rio-cogeo/pull/75 we noticed strange behaviors with `vsimem` driver (this could be a GDAL but TBH). 

1. When using `tempfile.NamedTemporaryFile()` Rasterio uses `vsimem` driver

with tempfile.NamedTemporaryFile() as tmpfile:

   print(tmpfile)

   with rasterio.open(tmpfile, "w", **meta) as tmp_dst:

       print(tmp_dst)

<tempfile._TemporaryFileWrapper object at 0x1151bb2e8>

<open DatasetWriter name='/vsimem/d26d4650-3010-4d39-8b0c-3d947b94f1d5.' mode='w+'>

Here I was expecting Rasterio/GDAL to behave as `tempfile` was a regular file.

2. When closing a `vsimem`  (`MemoryFile` or `tempfile`) we observe a huge memory surge when working with big images.

code: https://github.com/cogeotiff/rio-cogeo/pull/75#issuecomment-482745580



Tested on Mac OS and linux with python 3.7 (gdal 2.4 and 2.3) 

Thanks 



--
Sean Gillies
Join main@rasterio.groups.io to automatically receive all group messages.
More
Home Messages Hashtags Subgroups About Features Pricing Updates Terms Help Twitter