Memory error, rasterio merge


Simon
 

Hello everyone,

I tried to merge/stitch my 1000 geotif images, each of around 4gb using rasterio.

files = glob.glob('*.tif')
src_files = []
for fi in files:
    src = rasterio.open(fi)
    src_files.append(src)
mosaic, out_trans = merge(src_files)
out_meta = src.meta.copy()
out_meta.update({"driver": "GTiff",
                  "height": mosaic.shape[1],
                  "width": mosaic.shape[2],
                  "transform": out_trans,
                  "crs": src.crs})

with rasterio.open(fo, "w", **out_meta) as dest:
     dest.write(mosaic)

Unfortunately, I could not proceed with the Memory error.
I could merge/stitch them using gdal translate, though it was very slow. So, physical memory should not be problem. To speed up, I  tried rasterio, but did not work. Any help to make it possible would be appreciated.

Thank you.
Simon


Sean Gillies
 

Hi Simon,

On Wed, Jan 29, 2020 at 7:29 AM Simon via Groups.Io <twinbirds=protonmail.com@groups.io> wrote:
Hello everyone,

I tried to merge/stitch my 1000 geotif images, each of around 4gb using rasterio.

files = glob.glob('*.tif')
src_files = []
for fi in files:
    src = rasterio.open(fi)
    src_files.append(src)
mosaic, out_trans = merge(src_files)
out_meta = src.meta.copy()
out_meta.update({"driver": "GTiff",
                  "height": mosaic.shape[1],
                  "width": mosaic.shape[2],
                  "transform": out_trans,
                  "crs": src.crs})

with rasterio.open(fo, "w", **out_meta) as dest:
     dest.write(mosaic)

Unfortunately, I could not proceed with the Memory error.
I could merge/stitch them using gdal translate, though it was very slow. So, physical memory should not be problem. To speed up, I  tried rasterio, but did not work. Any help to make it possible would be appreciated.

Thank you.
Simon

Rasterio's merge function is not well optimized with respect to memory. An output array is allocated such that it covers all the inputs. If your inputs don't overlap, this could be 4TB or more, which probably exceeds the available memory on your computer.

Best wishes,

--
Sean Gillies