Hi Angus,

On Mon, Oct 26, 2020 at 6:04 PM Angus Dickey <angus@...> wrote:

Does GDAL_CACHEMAX have to be set in bytes when using rasterio? I see in the docs there is an example using MBs but it seems to be causing rasterio to set a very small cache size when I use it. For example, accessing a COG in S3 using rasterio 1.1.8:

# No problem here
with rasterio.Env() as env:
    # Prints 851132006 (5% of my system RAM in bytes)
    print(get_gdal_config('GDAL_CACHEMAX'))
    with rasterio.open('s3://path/to/cog') as src:
        # Do stuff with the COG

# No problem here either
with rasterio.Env(GDAL_CACHEMAX=536870912) as env:

    # Prints  536870912
    print(get_gdal_config('GDAL_CACHEMAX'))
    with rasterio.open('s3://path/to/cog') as src:

        # Do stuff with the COG

# Really slow
with rasterio.Env(GDAL_CACHEMAX=512) as env:
    # Prints 512 (in bytes?)
    print(get_gdal_config('GDAL_CACHEMAX'))
    with rasterio.open('s3://path/to/cog') as src:
        # Do stuff with the COG

It seems like rasterio is setting the GDAL raster block cache to 512 bytes and this is causing the slow read. I don't really understand the internals of rasterio but it looks to be using GDALSetCacheMax() (which only accepts bytes) and is passing it 512. I might be misunderstanding the problem though, it could be something else slowing things down but that is the only change I am making.

Any input is appreciated.

Thanks,

Angus

Yes, GDAL_CACHEMAX passed to `Env()` must be in bytes (since https://github.com/mapbox/rasterio/pull/1042/files).

--

Sean Gillies