Re: Asyncio + Rasterio for slow network requests?
Hi Kyle, Dion:
On Mon, Mar 30, 2020 at 1:41 PM <kylebarron2@...> wrote:
Sorry for the slow response. As Vincent noted, just moving back to GDAL 2.4 made the process ~8x faster, from 1.7s to read to ~200ms to read each source tile.
Thank you for the details.
Dion, can you say a little more about reads not being thread-safe?
It's intended that we can call GDAL's RasterIO functions in different threads concurrently as long as we don't share dataset handles between threads. If we observe otherwise, then there is a GDAL bug that we can fix.
There is an additional consideration for VRTs explained in https://gdal.org/drivers/raster/vrt.html#multi-threading-issues. If we have multiple VRTs, used in different threads, pointing to the same URLs, we need to take an extra step to prevent GDAL from accidentally sharing those non-VRT dataset handles between the threads.