Re: tempfile.NamedTemporaryFile behaving as /vsimem and eating all the machine memory

Sean Gillies

Hi Vincent,

This is expected (if not well-documented) behavior. tempfile.NamedTemporaryFile() returns an open Python file object, not a filename. GDAL can't use a Python file object, so in that case reads all the bytes from the file object, copies them to the vsimem filesystem, and opens that vsimem file.

I think what you want do do is pass the name of the temp file object to GDAL. Like this:

with tempfile.NamedTemporaryFile() as temp:
    with as dataset:

No copy in the vsimem filesystem will be made.

On Tue, Apr 16, 2019 at 6:55 AM <vincent.sarago@...> wrote:
While working on we noticed strange behaviors with `vsimem` driver (this could be a GDAL but TBH). 

1. When using `
tempfile.NamedTemporaryFile()` Rasterio uses `vsimem` driver

with tempfile.NamedTemporaryFile() as tmpfile:


   with, "w", **meta) as tmp_dst:


<tempfile._TemporaryFileWrapper object at 0x1151bb2e8>

<open DatasetWriter name='/vsimem/d26d4650-3010-4d39-8b0c-3d947b94f1d5.' mode='w+'>

Here I was expecting Rasterio/GDAL to behave as `tempfile` was a regular file.

2. When closing
 a `vsimem`  (`MemoryFile` or `tempfile`) we observe a huge memory surge when working with big images.


Tested on Mac OS and linux with python 3.7 (gdal 2.4 and 2.3) 


Sean Gillies

Join to automatically receive all group messages.