Re: Asyncio + Rasterio for slow network requests?

Home Messages Hashtags Subgroups

Sean Gillies #480 Hi Vincent, On Mon, Mar 30, 2020 at 7:12 AM <vincent.sarago@...> wrote: Hi All, I'll answer for Kyle but he can jump back if needed. The problem Kyle was facing was due to GDAL3 (running on AWS Lambda, CentOS) being extremely slow for image reprojection. We faced this in https://github.com/RemotePixel/amazonlinux/issues/16 and though it was fixed when we updated sqlite lib (https://github.com/RemotePixel/amazonlinux/pull/17) but while this made things a bit faster, it seems there is still a `huge` difference between gdal2/proj5 and gdal3/proj6. We still went through some testing with async but because kyle uses AWS Lambda and https://github.com/vincentsarago/lambda-proxy which is not async compatible we just switched to gdal2 and to threading. FYI, I've updated another tiling project using async but I need to run benchmarks https://github.com/developmentseed/titiler/blob/master/titiler/api/api_v1/endpoints/tiles.py#L26 Vincent Thanks for the update. This situation points out a downside of using the warped VRT: it abstracts everything (network, reprojection, caching) and makes diagnosing problems difficult. -- Sean Gillies
More All Messages By This Member

Join main@rasterio.groups.io to automatically receive all group messages.