Re: concurrent.futures.ThreadPoolExecutor for s3 COG reads fails - CPLReleaseMutex: Error

Sean Gillies

Hi Darren,

I've seen this error message before, and opened an issue at I haven't tried to reproduce the error with GDAL 3.3 yet.

On Fri, May 7, 2021 at 2:51 PM <> wrote:
This could be similar to

In an AWS-Lambda python 3.7 runtime, there is an error like:
    CPLReleaseMutex: Error = 1 (Operation not permitted)
    [WARNING] 2021-05-07T17:38:33.315Z 3e8b4b55-9066-426c-9319-540e39e17709
    CPLE_AppDefined in HTTP response code on 0
    2021-05-07 17:38:33,366 | ERROR | '/vsis3/bucket-xxx-us-west-2/prefix/geotiff.tif' not recognized as a supported file format.

The python concurrency that wraps the rasterio/gdal reader is a common pattern like:

    dataset = []
    with concurrent.futures.ThreadPoolExecutor() as executor:
        future_to_task = {}
        for task in peril_queries:
            future = executor.submit(extract_geotiff_data, task)
            future_to_task[future] = task
    for future in concurrent.futures.as_completed(future_to_task):
            data = future.result()
        except Exception as exc:
            task = future_to_task[future]
            LOGGER.error("Failed Task: %s", task)

Where the 'extract_geotiff_data' function is basically given a task to read an s3-COG,
and that function encapsulates everything to setup a rasterio.Env with gdal env-var
settings, opening the s3 object and reading data from it.

What is causing the `CPLReleaseMutex: Error` and how is it debugged and avoided?


Sean Gillies

Join to automatically receive all group messages.