1. posterid:1231769
  2. Occasional "not recognized as a supported file format" errors when reading from S3

Re: Occasional "not recognized as a supported file format" errors when reading from S3

Sean Gillies
 

I found the VRT multi-threading guidance at https://gdal.org/drivers/raster/vrt.html#multi-threading-issues. It moved during the site migration.

This smells like a GDAL bug to me, perhaps a manifestation of https://github.com/OSGeo/gdal/issues/1244 or https://github.com/OSGeo/gdal/issues/1031.


On Mon, May 27, 2019 at 2:50 AM Dion Häfner <dion.haefner@...> wrote:

Hey Sean,

thank you very much for the detailed assessment. I think this already sheds some light on some of the problems.

To answer your questions:

- The /vsis3/ identifiers are correct and work most of the time, we only see these errors from time to time.

- This is just a default read from a private S3 bucket. I don't know which protocol boto (or GDAL?) uses under the hood.

- In the first case, there is indeed multithreading involved, so this is a hot lead. I will have a look at the documentation as soon as I can find it somewhere :) It working most of the time would of course suggest some sort of race condition.

In the second case, the setup is actually more complicated. We experimented with "cloud-optimized" VRT files, that we created by

1. gdalbuildvrt on many small-ish COG

2. gdaladdo on the resulting VRT into an external .vrt.ovr file

We could then serve these COG mosaics up as a single dataset via Terracotta, but apparently the read from the VRT would fail when we zoomed in too far.

Thank you for your time!

Dion

On 24/05/2019 20.49, Sean Gillies via Groups.Io wrote:
Hi Dion,

That error comes from these lines in GDALOpenEx when the function fails to return successfully: https://github.com/OSGeo/gdal/blob/fe0b5dd644abb4ac0c9869db28e9cf977181fbce/gdal/gcore/gdaldataset.cpp#L3455. The way the errors are re-raised obscures the problem.

First, are the /vsis3/ identifiers for the datasets correct? Is Rasterio mangling them? Can you access the data successfully any time at all?

Are you using any advanced HTTP features? HTTP/2? The Rasterio wheels use a slightly old version of curl and that project is ever fixing bugs.

Is multithreading involved? I recommend that you consult https://gdal.org/gdal_vrttut.html#gdal_vrttut_mt as soon as it is back online. Multithreaded access to VRTs is complicated.

On Fri, May 24, 2019 at 8:13 AM Dion Häfner <dion.haefner@...> wrote:

We've only tried rasterio wheels.

On 23/05/2019 21.23, vincent.sarago via Groups.Io wrote:
This is not something I've been seen for now. does this happens with Rasterio wheels and rasterio build on gdal source ? 



--
Sean Gillies
Join main@rasterio.groups.io to automatically receive all group messages.