Re: rasterio vsicurl issues with docker-compose


Sean Gillies
 

GDAL's verbose HTTP "logs" are printed directly to stderr and won't be captured by the Python logger. Can you try to record them somehow? I think we'll see some clues there. I don't want to speculate, but maybe docker compose's networking system lacks or disables features we need?

On Tue, Dec 3, 2019 at 1:52 PM Madhav Desetty <madhav@...> wrote:
This the log I getting when the rio command with verbose curl. Not seeing specific http connection errors.

$ CPL_CURL_VERBOSE=YES rio -vvv info http://oin-hotosm.s3.amazonaws.com/59c66c5223c8440011d7b1e4/0/7ad397c0-bba2-4f98-a08a-931ec3a6e943.tif

OGRCT: PROJ >= 4.8.0 features enabled

OGRCT: Using locale-safe proj version

DEBUG:botocore.hooks:Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane

DEBUG:botocore.hooks:Changing event name from before-call.apigateway to before-call.api-gateway

DEBUG:botocore.hooks:Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict

DEBUG:botocore.hooks:Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration

DEBUG:botocore.hooks:Changing event name from before-parameter-build.route53 to before-parameter-build.route-53

DEBUG:botocore.hooks:Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search

DEBUG:botocore.hooks:Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section

DEBUG:botocore.hooks:Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask

DEBUG:botocore.hooks:Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section

DEBUG:botocore.hooks:Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search

DEBUG:botocore.hooks:Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section

DEBUG:botocore.credentials:Looking for credentials via: env

INFO:botocore.credentials:Found credentials in environment variables.

DEBUG:rasterio.env:Entering env context: <rasterio.env.Env object at 0x7f1a7cf62208>

DEBUG:rasterio.env:Starting outermost env

DEBUG:rasterio.env:No GDAL environment exists

DEBUG:rasterio.env:New GDAL environment <rasterio._env.GDALEnv object at 0x7f1a7d6ad048> created

DEBUG:rasterio._env:GDAL_DATA found in environment: '/usr/local/lib/python3.6/dist-packages/rasterio/gdal_data'.

DEBUG:rasterio._env:PROJ data files are available at built-in paths

DEBUG:rasterio._env:Started GDALEnv <rasterio._env.GDALEnv object at 0x7f1a7d6ad048>.

DEBUG:rasterio.env:Entered env context: <rasterio.env.Env object at 0x7f1a7cf62208>

DEBUG:rasterio.env:Got a copy of environment <rasterio._env.GDALEnv object at 0x7f1a7d6ad048> options

DEBUG:rasterio.env:Entering env context: <rasterio.env.Env object at 0x7f1a7ccf2320>

DEBUG:rasterio.env:Got a copy of environment <rasterio._env.GDALEnv object at 0x7f1a7d6ad048> options

DEBUG:rasterio.env:Entered env context: <rasterio.env.Env object at 0x7f1a7ccf2320>

DEBUG:rasterio._base:Sharing flag: 0

DEBUG:rasterio._env:CPLE_None in CPLError: `/vsicurl/http://oin-hotosm.s3.amazonaws.com/59c66c5223c8440011d7b1e4/0/7ad397c0-bba2-4f98-a08a-931ec3a6e943.tif' does not exist in the file system, and is not recognized as a supported dataset name.

DEBUG:rasterio.env:Exiting env context: <rasterio.env.Env object at 0x7f1a7ccf2320>

DEBUG:rasterio.env:Cleared existing <rasterio._env.GDALEnv object at 0x7f1a7d6ad048> options

DEBUG:rasterio._env:Stopped GDALEnv <rasterio._env.GDALEnv object at 0x7f1a7d6ad048>.

DEBUG:rasterio.env:No GDAL environment exists

DEBUG:rasterio.env:New GDAL environment <rasterio._env.GDALEnv object at 0x7f1a7d6ad048> created

DEBUG:rasterio._env:GDAL_DATA found in environment: '/usr/local/lib/python3.6/dist-packages/rasterio/gdal_data'.

DEBUG:rasterio._env:PROJ data files are available at built-in paths

DEBUG:rasterio._env:Started GDALEnv <rasterio._env.GDALEnv object at 0x7f1a7d6ad048>.

DEBUG:rasterio.env:Exited env context: <rasterio.env.Env object at 0x7f1a7ccf2320>

DEBUG:rasterio.env:Exiting env context: <rasterio.env.Env object at 0x7f1a7cf62208>

DEBUG:rasterio.env:Cleared existing <rasterio._env.GDALEnv object at 0x7f1a7d6ad048> options

DEBUG:rasterio._env:Stopped GDALEnv <rasterio._env.GDALEnv object at 0x7f1a7d6ad048>.

DEBUG:rasterio.env:Exiting outermost env

DEBUG:rasterio.env:Exited env context: <rasterio.env.Env object at 0x7f1a7cf62208>

Traceback (most recent call last):

  File "rasterio/_base.pyx", line 216, in rasterio._base.DatasetBase.__init__

  File "rasterio/_shim.pyx", line 67, in rasterio._shim.open_dataset

  File "rasterio/_err.pyx", line 205, in rasterio._err.exc_wrap_pointer

rasterio._err.CPLE_OpenFailedError: '/vsicurl/http://oin-hotosm.s3.amazonaws.com/59c66c5223c8440011d7b1e4/0/7ad397c0-bba2-4f98-a08a-931ec3a6e943.tif' does not exist in the file system, and is not recognized as a supported dataset name.

 

During handling of the above exception, another exception occurred:

 

Traceback (most recent call last):

  File "/usr/local/bin/rio", line 11, in <module>

    sys.exit(main_group())

  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 764, in __call__

    return self.main(*args, **kwargs)

  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 717, in main

    rv = self.invoke(ctx)

  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 1137, in invoke

    return _process_result(sub_ctx.command.invoke(sub_ctx))

  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 956, in invoke

    return ctx.invoke(self.callback, **ctx.params)

  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 555, in invoke

    return callback(*args, **kwargs)

  File "/usr/local/lib/python3.6/dist-packages/click/decorators.py", line 17, in new_func

    return f(get_current_context(), *args, **kwargs)

  File "/usr/local/lib/python3.6/dist-packages/rasterio/rio/info.py", line 66, in info

    with ctx.obj['env'], rasterio.open(input) as src:

  File "/usr/local/lib/python3.6/dist-packages/rasterio/env.py", line 445, in wrapper

    return f(*args, **kwds)

  File "/usr/local/lib/python3.6/dist-packages/rasterio/__init__.py", line 219, in open

    s = DatasetReader(path, driver=driver, sharing=sharing, **kwargs)

  File "rasterio/_base.pyx", line 218, in rasterio._base.DatasetBase.__init__

rasterio.errors.RasterioIOError: '/vsicurl/http://oin-hotosm.s3.amazonaws.com/59c66c5223c8440011d7b1e4/0/7ad397c0-bba2-4f98-a08a-931ec3a6e943.tif' does not exist in the file system, and is not recognized as a supported dataset name.

--
Sean Gillies

Join main@rasterio.groups.io to automatically receive all group messages.