rasterio vsicurl issues with docker-compose


Madhav Desetty
 

I am having a strange rasterio issue where if I build rasterio docker image from osgeo/gdal:ubuntu-small-latest as the base image and run it as docker-compose up, the rio fails to open any http based COG files. It works fine when the same TIF file is copied locally and accessed from a mounted drive. However, gdalinfo succeeds for COG file on S3 via HTTP url. When I run the docker image without docker-compose and use the docker commands (docker build && docker run), rio works for COG file on S3 via HTTP url. It seems like some environment is not being set or initialized when run via docker-compose and I am missing something obvious. Anyone ran into this issue before and any debugging tips to debug why the FileOpen is failing via Docker Compose? I set gdal CPL_DEBUG and CPL_LOG_ERRORS env to ON and logs are still minimal.

    

$ rio --version

OGRCT: PROJ >= 4.8.0 features enabled

OGRCT: Using locale-safe proj version

1.1.1

$ rio info -v http://oin-hotosm.s3.amazonaws.com/59c66c5223c8440011d7b1e4/0/7ad397c0-bba2-4f98-a08a-931ec3a6e943.tif

OGRCT: PROJ >= 4.8.0 features enabled

OGRCT: Using locale-safe proj version

Traceback (most recent call last):

  File "rasterio/_base.pyx", line 216, in rasterio._base.DatasetBase.__init__

  File "rasterio/_shim.pyx", line 67, in rasterio._shim.open_dataset

  File "rasterio/_err.pyx", line 205, in rasterio._err.exc_wrap_pointer

rasterio._err.CPLE_OpenFailedError: '/vsicurl/http://oin-hotosm.s3.amazonaws.com/59c66c5223c8440011d7b1e4/0/7ad397c0-bba2-4f98-a08a-931ec3a6e943.tif' does not exist in the file system, and is not recognized as a supported dataset name.

 

During handling of the above exception, another exception occurred:

 

Traceback (most recent call last):

  File "/usr/local/bin/rio", line 11, in <module>

    sys.exit(main_group())

  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 764, in __call__

    return self.main(*args, **kwargs)

  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 717, in main

    rv = self.invoke(ctx)

  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 1137, in invoke

    return _process_result(sub_ctx.command.invoke(sub_ctx))

  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 956, in invoke

    return ctx.invoke(self.callback, **ctx.params)

  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 555, in invoke

    return callback(*args, **kwargs)

  File "/usr/local/lib/python3.6/dist-packages/click/decorators.py", line 17, in new_func

    return f(get_current_context(), *args, **kwargs)

  File "/usr/local/lib/python3.6/dist-packages/rasterio/rio/info.py", line 66, in info

    with ctx.obj['env'], rasterio.open(input) as src:

  File "/usr/local/lib/python3.6/dist-packages/rasterio/env.py", line 445, in wrapper

    return f(*args, **kwds)

  File "/usr/local/lib/python3.6/dist-packages/rasterio/__init__.py", line 219, in open

    s = DatasetReader(path, driver=driver, sharing=sharing, **kwargs)

  File "rasterio/_base.pyx", line 218, in rasterio._base.DatasetBase.__init__

rasterio.errors.RasterioIOError: '/vsicurl/http://oin-hotosm.s3.amazonaws.com/59c66c5223c8440011d7b1e4/0/7ad397c0-bba2-4f98-a08a-931ec3a6e943.tif' does not exist in the file system, and is not recognized as a supported dataset name.

$ gdalinfo http://oin-hotosm.s3.amazonaws.com/59c66c5223c8440011d7b1e4/0/7ad397c0-bba2-4f98-a08a-931ec3a6e943.tif

HTTP: Fetch(http://oin-hotosm.s3.amazonaws.com/59c66c5223c8440011d7b1e4/0/7ad397c0-bba2-4f98-a08a-931ec3a6e943.tif)

HTTP: libcurl/7.58.0 GnuTLS/3.5.18 zlib/1.2.11 libidn2/2.0.4 libpsl/0.19.1 (+libidn2/2.0.4) nghttp2/1.30.0 librtmp/2.3

GDAL: GDALOpen(/vsimem/http_1/7ad397c0-bba2-4f98-a08a-931ec3a6e943.tif, this=0x55e917f3afa0) succeeds as GTiff.

GDAL: GDALOpen(http://oin-hotosm.s3.amazonaws.com/59c66c5223c8440011d7b1e4/0/7ad397c0-bba2-4f98-a08a-931ec3a6e943.tif, this=0x55e917f3afa0) succeeds as HTTP.

Driver: GTiff/GeoTIFF

GDAL: GDALDefaultOverviews::OverviewScan()

MDReaderPleiades: Not a Pleiades product

MDReaderPleiades: Not a Pleiades product

Files: none associated

Size is 49402, 28398

Coordinate System is:

PROJCRS["WGS 84 / UTM zone 20N",

    BASEGEOGCRS["WGS 84",

        DATUM["World Geodetic System 1984",

            ELLIPSOID["WGS 84",6378137,298.257223563,

                LENGTHUNIT["metre",1]]],

        PRIMEM["Greenwich",0,

            ANGLEUNIT["degree",0.0174532925199433]],

        ID["EPSG",4326]],

    CONVERSION["UTM zone 20N",

        METHOD["Transverse Mercator",

            ID["EPSG",9807]],

        PARAMETER["Latitude of natural origin",0,

            ANGLEUNIT["degree",0.0174532925199433],

            ID["EPSG",8801]],

        PARAMETER["Longitude of natural origin",-63,

            ANGLEUNIT["degree",0.0174532925199433],

            ID["EPSG",8802]],

        PARAMETER["Scale factor at natural origin",0.9996,

            SCALEUNIT["unity",1],

            ID["EPSG",8805]],

        PARAMETER["False easting",500000,

            LENGTHUNIT["metre",1],

            ID["EPSG",8806]],

        PARAMETER["False northing",0,

            LENGTHUNIT["metre",1],

            ID["EPSG",8807]]],

    CS[Cartesian,2],

        AXIS["(E)",east,

            ORDER[1],

            LENGTHUNIT["metre",1]],

        AXIS["(N)",north,

            ORDER[2],

            LENGTHUNIT["metre",1]],

    USAGE[

        SCOPE["unknown"],

        AREA["World - N hemisphere - 66°W to 60°W - by country"],

        BBOX[0,-66,84,-60]],

    ID["EPSG",32620]]

Data axis to CRS axis mapping: 1,2

Origin = (494088.931940000038594,1993386.902620000066236)

Pixel Size = (0.027070000000000,-0.027070000000000)

Metadata:

  AREA_OR_POINT=Area

  TIFFTAG_SOFTWARE=pix4dmapper

Image Structure Metadata:

  COMPRESSION=YCbCr JPEG

  INTERLEAVE=PIXEL

  SOURCE_COLOR_SPACE=YCbCr

GTiff: ScanDirectories()

GTiff: Opened 24701x14199 overview.

GTiff: Opened 12351x7100 overview.

GTiff: Opened 6176x3550 overview.

GTiff: Opened 3088x1775 overview.

GTiff: Opened 1544x888 overview.

GTiff: Opened 772x444 overview.

GTiff: Opened 386x222 overview.

Corner Coordinates:

Upper Left  (  494088.932, 1993386.903) ( 63d 3'21.05"W, 18d 1'44.14"N)

Lower Left  (  494088.932, 1992618.169) ( 63d 3'21.04"W, 18d 1'19.13"N)

Upper Right (  495426.244, 1993386.903) ( 63d 2'35.56"W, 18d 1'44.15"N)

Lower Right (  495426.244, 1992618.169) ( 63d 2'35.56"W, 18d 1'19.14"N)

Center      (  494757.588, 1993002.536) ( 63d 2'58.30"W, 18d 1'31.64"N)

Band 1 Block=512x512 Type=Byte, ColorInterp=Red

  NoData Value=-10000

  Overviews: 24701x14199, 12351x7100, 6176x3550, 3088x1775, 1544x888, 772x444, 386x222

Band 2 Block=512x512 Type=Byte, ColorInterp=Green

  NoData Value=-10000

  Overviews: 24701x14199, 12351x7100, 6176x3550, 3088x1775, 1544x888, 772x444, 386x222

Band 3 Block=512x512 Type=Byte, ColorInterp=Blue

  NoData Value=-10000

  Overviews: 24701x14199, 12351x7100, 6176x3550, 3088x1775, 1544x888, 772x444, 386x222

GDAL: GDALClose(http://oin-hotosm.s3.amazonaws.com/59c66c5223c8440011d7b1e4/0/7ad397c0-bba2-4f98-a08a-931ec3a6e943.tif, this=0x55e917f3afa0)

GDAL: In GDALDestroy - unloading GDAL shared library.

$ 

Join main@rasterio.groups.io to automatically receive all group messages.