Hi All, 
I'll answer for Kyle but he can jump back if needed. 

The problem Kyle was facing was due to GDAL3 (running on AWS Lambda, CentOS) being extremely slow for image reprojection. 
We faced this in https://github.com/RemotePixel/amazonlinux/issues/16 and though it was fixed when we updated sqlite lib (https://github.com/RemotePixel/amazonlinux/pull/17) but while this made things a bit faster, it seems there is still a `huge` difference between gdal2/proj5 and gdal3/proj6.

We still went through some testing with async but because kyle uses AWS Lambda and https://github.com/vincentsarago/lambda-proxy which is not async compatible we just switched to gdal2 and to threading. 

FYI, I've updated another tiling project using async but I need to run benchmarks https://github.com/developmentseed/titiler/blob/master/titiler/api/api_v1/endpoints/tiles.py#L26 

Vincent

I'm trying to improve performance of dynamic satellite imagery tiling, using

[`cogeo-mosaic-tiler`](https://github.com/developmentseed/cogeo-mosaic-tiler)/[`rio-tiler`](https://github.com/cogeotiff/rio-tiler),

which combines source Cloud-Optimized GeoTIFFs into a web mercator tile on the

fly. I'm using AWS Landsat and NAIP imagery stored in S3 buckets, and running

code on AWS Lambda in the same region.

 

Since NAIP imagery doesn't overlap cleanly with web mercator tiles, at zoom 12 I

have to load on average [6 assets to create one mercator

tile](https://user-images.githubusercontent.com/15164633/77286861-cfc7df00-6c99-11ea-84e9-8ed584b030c0.png).

While profiling the AWS Lambda instance using AWS X-Ray, I found that the

biggest bottleneck was the [base

call](https://github.com/cogeotiff/rio-tiler/blob/6b0d4df0b6aa1454c50312e8d352ed57f0a4e3cb/rio_tiler/utils.py#L449-L455)

to `WarpedVRT.read()`. That call always takes [between 1.7 and 2.0

seconds](https://user-images.githubusercontent.com/15164633/77289999-c5f5aa00-6ca0-11ea-816a-5aaf248a782c.png)

for each tile, regardless of the amount of overlap with the mercator tile.

 

When testing tile load times on an EC2 t2.nano in the same region, for the first

tile load, CPU time is 120 ms but wall time is 1.1 seconds. That leads me to

believe that the bottleneck is S3 latency.

 

If the code running on Lambda shares the same 90% proportion spent on latency

for each asset, that would imply that 9 seconds total are spent waiting on

latency.

 

Using multithreading with a `ThreadPoolExecutor` takes longer than running

single-threaded. Given the situation, it would seem ideal to use `asyncio` for

the COG network requests to improve performance.

 

Has this been attempted ever with Rasterio? I saw a [Rasterio example of using

async](https://github.com/mapbox/rasterio/blob/master/examples/async-rasterio.py)

to improve performance on a CPU bound function, and plan to try that out, but

I'm pessimistic about that approach directly because I'd think that the `async`

calls would need to be applied on the core fetch calls directly.

Hi all,

As of GDAL 3 I've started seeing warnings written to stderr on open, such as:

proj_create_from_database: datum not found

It's possible that these datasets have malformed projection information, but in cases where I don't care I'd like to be able to silence these types of warnings, as they can easily be emitted hundreds of times.  However I can't find a way to do so.

Using an Env() context with CPL_DEBUG set to os.devnull has no effect, nor does redirecting sys.stderr itself.  It also doesn't look like this is being emitted through the logging module.  So it seems like the underlying library itself may be writing to stderr?  I don't know if this is a rasterio issue, a gdal issue, or a proj issue.

Any pointers are appreciated.

Greg

ERROR 1: PROJ: proj_create_from_database: datum not found

I am attempting to use rasterio to merge 3 bands from Sentinel-2 data, sourced as .JP2 files, into a georeferenced GeoTiff.  However, I am running into 2 problems doing this.

First of all, here is my code for the merge:

    #Setup the bands
    b4 = rio.open(path + Bands.B04.value[0])
    b3 = rio.open(path + Bands.B03.value[0])
    b2 = rio.open(path + Bands.B02.value[0])
   
    #Setup the path for the constructed RGB image
    tif = path + "RGB.tif"
   
    #Create the RGB image
    with rio.open(tif, 'w', driver='Gtiff', width=b4.width, height=b4.height, count=3, crs=b4.crs, tranform=b4.transform, dtype=b4.dtypes[0], photometric="RGB", tile=True, compress='lzw', bigtiff=True) as rgb:
        rgb.write(b2.read(1), 1)
        rgb.write(b3.read(1), 2)
        rgb.write(b4.read(1), 3)
        rgb.close()

Now, how do I copy the source projection information and bounds in the new GeoTiff I am creating?  Second, why is it showing up as all black when viewing the image in most common image browsers, but just fine in mapping software such as ArcMap or ArcGIS Pro?

Any ideas ?