Date   

rio-hist 1.0.0

Sean Gillies
 

Hi all,

Version 1.0.0 for rio-hist is on PyPI now. Note that there is no wheel for Python 2.7. I expect the package to work with 2.7, and the sdist will serve, but the project doesn't officially support Python 2.7 any longer.

--
Sean Gillies


Re: rio-hist 1.0b1 release

Sean Gillies
 

Thanks for test driving, Matt. It's a nice package and I'm glad to see it back in good condition.

I'm reminded that we've talked about aggregating the plugins in mapbox org repos into a single project. It's an idea that keeps getting better, but I still haven't found the space to work on it. 😬

On Tue, Dec 3, 2019, 5:29 PM Matthew Perry <perrygeo@...> wrote:
Thanks Sean!

I can confirm the `rio hist` CLI is working well with Rasterio 1.1.1, py37. Thanks for keeping it afloat.


Re: rio-hist 1.0b1 release

Matthew Perry
 

Thanks Sean!

I can confirm the `rio hist` CLI is working well with Rasterio 1.1.1, py37. Thanks for keeping it afloat.


rio-hist 1.0b1 release

Sean Gillies
 

Hi all,

I realized today that the rio-hist CLI plugin was nearly derelict. It didn't work with rasterio versions ~= 1.0 and hadn't been built in a long time. I gave it an overhaul, put a 1.0b1 release on PyPI, and have made a plan to release 1.0.0 next Monday.

If anyone would be willing to `python -m pip install rio-hist==1.0b1` and give it a try, I would be very grateful.

Thanks,

--
Sean Gillies


Re: rasterio vsicurl issues with docker-compose

Sean Gillies
 

GDAL's verbose HTTP "logs" are printed directly to stderr and won't be captured by the Python logger. Can you try to record them somehow? I think we'll see some clues there. I don't want to speculate, but maybe docker compose's networking system lacks or disables features we need?

On Tue, Dec 3, 2019 at 1:52 PM Madhav Desetty <madhav@...> wrote:
This the log I getting when the rio command with verbose curl. Not seeing specific http connection errors.

$ CPL_CURL_VERBOSE=YES rio -vvv info http://oin-hotosm.s3.amazonaws.com/59c66c5223c8440011d7b1e4/0/7ad397c0-bba2-4f98-a08a-931ec3a6e943.tif

OGRCT: PROJ >= 4.8.0 features enabled

OGRCT: Using locale-safe proj version

DEBUG:botocore.hooks:Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane

DEBUG:botocore.hooks:Changing event name from before-call.apigateway to before-call.api-gateway

DEBUG:botocore.hooks:Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict

DEBUG:botocore.hooks:Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration

DEBUG:botocore.hooks:Changing event name from before-parameter-build.route53 to before-parameter-build.route-53

DEBUG:botocore.hooks:Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search

DEBUG:botocore.hooks:Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section

DEBUG:botocore.hooks:Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask

DEBUG:botocore.hooks:Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section

DEBUG:botocore.hooks:Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search

DEBUG:botocore.hooks:Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section

DEBUG:botocore.credentials:Looking for credentials via: env

INFO:botocore.credentials:Found credentials in environment variables.

DEBUG:rasterio.env:Entering env context: <rasterio.env.Env object at 0x7f1a7cf62208>

DEBUG:rasterio.env:Starting outermost env

DEBUG:rasterio.env:No GDAL environment exists

DEBUG:rasterio.env:New GDAL environment <rasterio._env.GDALEnv object at 0x7f1a7d6ad048> created

DEBUG:rasterio._env:GDAL_DATA found in environment: '/usr/local/lib/python3.6/dist-packages/rasterio/gdal_data'.

DEBUG:rasterio._env:PROJ data files are available at built-in paths

DEBUG:rasterio._env:Started GDALEnv <rasterio._env.GDALEnv object at 0x7f1a7d6ad048>.

DEBUG:rasterio.env:Entered env context: <rasterio.env.Env object at 0x7f1a7cf62208>

DEBUG:rasterio.env:Got a copy of environment <rasterio._env.GDALEnv object at 0x7f1a7d6ad048> options

DEBUG:rasterio.env:Entering env context: <rasterio.env.Env object at 0x7f1a7ccf2320>

DEBUG:rasterio.env:Got a copy of environment <rasterio._env.GDALEnv object at 0x7f1a7d6ad048> options

DEBUG:rasterio.env:Entered env context: <rasterio.env.Env object at 0x7f1a7ccf2320>

DEBUG:rasterio._base:Sharing flag: 0

DEBUG:rasterio._env:CPLE_None in CPLError: `/vsicurl/http://oin-hotosm.s3.amazonaws.com/59c66c5223c8440011d7b1e4/0/7ad397c0-bba2-4f98-a08a-931ec3a6e943.tif' does not exist in the file system, and is not recognized as a supported dataset name.

DEBUG:rasterio.env:Exiting env context: <rasterio.env.Env object at 0x7f1a7ccf2320>

DEBUG:rasterio.env:Cleared existing <rasterio._env.GDALEnv object at 0x7f1a7d6ad048> options

DEBUG:rasterio._env:Stopped GDALEnv <rasterio._env.GDALEnv object at 0x7f1a7d6ad048>.

DEBUG:rasterio.env:No GDAL environment exists

DEBUG:rasterio.env:New GDAL environment <rasterio._env.GDALEnv object at 0x7f1a7d6ad048> created

DEBUG:rasterio._env:GDAL_DATA found in environment: '/usr/local/lib/python3.6/dist-packages/rasterio/gdal_data'.

DEBUG:rasterio._env:PROJ data files are available at built-in paths

DEBUG:rasterio._env:Started GDALEnv <rasterio._env.GDALEnv object at 0x7f1a7d6ad048>.

DEBUG:rasterio.env:Exited env context: <rasterio.env.Env object at 0x7f1a7ccf2320>

DEBUG:rasterio.env:Exiting env context: <rasterio.env.Env object at 0x7f1a7cf62208>

DEBUG:rasterio.env:Cleared existing <rasterio._env.GDALEnv object at 0x7f1a7d6ad048> options

DEBUG:rasterio._env:Stopped GDALEnv <rasterio._env.GDALEnv object at 0x7f1a7d6ad048>.

DEBUG:rasterio.env:Exiting outermost env

DEBUG:rasterio.env:Exited env context: <rasterio.env.Env object at 0x7f1a7cf62208>

Traceback (most recent call last):

  File "rasterio/_base.pyx", line 216, in rasterio._base.DatasetBase.__init__

  File "rasterio/_shim.pyx", line 67, in rasterio._shim.open_dataset

  File "rasterio/_err.pyx", line 205, in rasterio._err.exc_wrap_pointer

rasterio._err.CPLE_OpenFailedError: '/vsicurl/http://oin-hotosm.s3.amazonaws.com/59c66c5223c8440011d7b1e4/0/7ad397c0-bba2-4f98-a08a-931ec3a6e943.tif' does not exist in the file system, and is not recognized as a supported dataset name.

 

During handling of the above exception, another exception occurred:

 

Traceback (most recent call last):

  File "/usr/local/bin/rio", line 11, in <module>

    sys.exit(main_group())

  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 764, in __call__

    return self.main(*args, **kwargs)

  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 717, in main

    rv = self.invoke(ctx)

  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 1137, in invoke

    return _process_result(sub_ctx.command.invoke(sub_ctx))

  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 956, in invoke

    return ctx.invoke(self.callback, **ctx.params)

  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 555, in invoke

    return callback(*args, **kwargs)

  File "/usr/local/lib/python3.6/dist-packages/click/decorators.py", line 17, in new_func

    return f(get_current_context(), *args, **kwargs)

  File "/usr/local/lib/python3.6/dist-packages/rasterio/rio/info.py", line 66, in info

    with ctx.obj['env'], rasterio.open(input) as src:

  File "/usr/local/lib/python3.6/dist-packages/rasterio/env.py", line 445, in wrapper

    return f(*args, **kwds)

  File "/usr/local/lib/python3.6/dist-packages/rasterio/__init__.py", line 219, in open

    s = DatasetReader(path, driver=driver, sharing=sharing, **kwargs)

  File "rasterio/_base.pyx", line 218, in rasterio._base.DatasetBase.__init__

rasterio.errors.RasterioIOError: '/vsicurl/http://oin-hotosm.s3.amazonaws.com/59c66c5223c8440011d7b1e4/0/7ad397c0-bba2-4f98-a08a-931ec3a6e943.tif' does not exist in the file system, and is not recognized as a supported dataset name.

--
Sean Gillies


Re: rasterio vsicurl issues with docker-compose

Madhav Desetty
 

This the log I getting when the rio command with verbose curl. Not seeing specific http connection errors.

$ CPL_CURL_VERBOSE=YES rio -vvv info http://oin-hotosm.s3.amazonaws.com/59c66c5223c8440011d7b1e4/0/7ad397c0-bba2-4f98-a08a-931ec3a6e943.tif

OGRCT: PROJ >= 4.8.0 features enabled

OGRCT: Using locale-safe proj version

DEBUG:botocore.hooks:Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane

DEBUG:botocore.hooks:Changing event name from before-call.apigateway to before-call.api-gateway

DEBUG:botocore.hooks:Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict

DEBUG:botocore.hooks:Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration

DEBUG:botocore.hooks:Changing event name from before-parameter-build.route53 to before-parameter-build.route-53

DEBUG:botocore.hooks:Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search

DEBUG:botocore.hooks:Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section

DEBUG:botocore.hooks:Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask

DEBUG:botocore.hooks:Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section

DEBUG:botocore.hooks:Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search

DEBUG:botocore.hooks:Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section

DEBUG:botocore.credentials:Looking for credentials via: env

INFO:botocore.credentials:Found credentials in environment variables.

DEBUG:rasterio.env:Entering env context: <rasterio.env.Env object at 0x7f1a7cf62208>

DEBUG:rasterio.env:Starting outermost env

DEBUG:rasterio.env:No GDAL environment exists

DEBUG:rasterio.env:New GDAL environment <rasterio._env.GDALEnv object at 0x7f1a7d6ad048> created

DEBUG:rasterio._env:GDAL_DATA found in environment: '/usr/local/lib/python3.6/dist-packages/rasterio/gdal_data'.

DEBUG:rasterio._env:PROJ data files are available at built-in paths

DEBUG:rasterio._env:Started GDALEnv <rasterio._env.GDALEnv object at 0x7f1a7d6ad048>.

DEBUG:rasterio.env:Entered env context: <rasterio.env.Env object at 0x7f1a7cf62208>

DEBUG:rasterio.env:Got a copy of environment <rasterio._env.GDALEnv object at 0x7f1a7d6ad048> options

DEBUG:rasterio.env:Entering env context: <rasterio.env.Env object at 0x7f1a7ccf2320>

DEBUG:rasterio.env:Got a copy of environment <rasterio._env.GDALEnv object at 0x7f1a7d6ad048> options

DEBUG:rasterio.env:Entered env context: <rasterio.env.Env object at 0x7f1a7ccf2320>

DEBUG:rasterio._base:Sharing flag: 0

DEBUG:rasterio._env:CPLE_None in CPLError: `/vsicurl/http://oin-hotosm.s3.amazonaws.com/59c66c5223c8440011d7b1e4/0/7ad397c0-bba2-4f98-a08a-931ec3a6e943.tif' does not exist in the file system, and is not recognized as a supported dataset name.

DEBUG:rasterio.env:Exiting env context: <rasterio.env.Env object at 0x7f1a7ccf2320>

DEBUG:rasterio.env:Cleared existing <rasterio._env.GDALEnv object at 0x7f1a7d6ad048> options

DEBUG:rasterio._env:Stopped GDALEnv <rasterio._env.GDALEnv object at 0x7f1a7d6ad048>.

DEBUG:rasterio.env:No GDAL environment exists

DEBUG:rasterio.env:New GDAL environment <rasterio._env.GDALEnv object at 0x7f1a7d6ad048> created

DEBUG:rasterio._env:GDAL_DATA found in environment: '/usr/local/lib/python3.6/dist-packages/rasterio/gdal_data'.

DEBUG:rasterio._env:PROJ data files are available at built-in paths

DEBUG:rasterio._env:Started GDALEnv <rasterio._env.GDALEnv object at 0x7f1a7d6ad048>.

DEBUG:rasterio.env:Exited env context: <rasterio.env.Env object at 0x7f1a7ccf2320>

DEBUG:rasterio.env:Exiting env context: <rasterio.env.Env object at 0x7f1a7cf62208>

DEBUG:rasterio.env:Cleared existing <rasterio._env.GDALEnv object at 0x7f1a7d6ad048> options

DEBUG:rasterio._env:Stopped GDALEnv <rasterio._env.GDALEnv object at 0x7f1a7d6ad048>.

DEBUG:rasterio.env:Exiting outermost env

DEBUG:rasterio.env:Exited env context: <rasterio.env.Env object at 0x7f1a7cf62208>

Traceback (most recent call last):

  File "rasterio/_base.pyx", line 216, in rasterio._base.DatasetBase.__init__

  File "rasterio/_shim.pyx", line 67, in rasterio._shim.open_dataset

  File "rasterio/_err.pyx", line 205, in rasterio._err.exc_wrap_pointer

rasterio._err.CPLE_OpenFailedError: '/vsicurl/http://oin-hotosm.s3.amazonaws.com/59c66c5223c8440011d7b1e4/0/7ad397c0-bba2-4f98-a08a-931ec3a6e943.tif' does not exist in the file system, and is not recognized as a supported dataset name.

 

During handling of the above exception, another exception occurred:

 

Traceback (most recent call last):

  File "/usr/local/bin/rio", line 11, in <module>

    sys.exit(main_group())

  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 764, in __call__

    return self.main(*args, **kwargs)

  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 717, in main

    rv = self.invoke(ctx)

  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 1137, in invoke

    return _process_result(sub_ctx.command.invoke(sub_ctx))

  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 956, in invoke

    return ctx.invoke(self.callback, **ctx.params)

  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 555, in invoke

    return callback(*args, **kwargs)

  File "/usr/local/lib/python3.6/dist-packages/click/decorators.py", line 17, in new_func

    return f(get_current_context(), *args, **kwargs)

  File "/usr/local/lib/python3.6/dist-packages/rasterio/rio/info.py", line 66, in info

    with ctx.obj['env'], rasterio.open(input) as src:

  File "/usr/local/lib/python3.6/dist-packages/rasterio/env.py", line 445, in wrapper

    return f(*args, **kwds)

  File "/usr/local/lib/python3.6/dist-packages/rasterio/__init__.py", line 219, in open

    s = DatasetReader(path, driver=driver, sharing=sharing, **kwargs)

  File "rasterio/_base.pyx", line 218, in rasterio._base.DatasetBase.__init__

rasterio.errors.RasterioIOError: '/vsicurl/http://oin-hotosm.s3.amazonaws.com/59c66c5223c8440011d7b1e4/0/7ad397c0-bba2-4f98-a08a-931ec3a6e943.tif' does not exist in the file system, and is not recognized as a supported dataset name.


Re: problems installing GDAL and Rasterio in virtual environment in Windows

Eyal Saiet
 

I think it was hard to read the PrintScreen from last night so I am reposting:
image.png

On Mon, Dec 2, 2019 at 11:14 PM Eyal Saiet via Groups.Io <ejsaiet=alaska.edu@groups.io> wrote:
Hello,
I was successful in installing GDAL and Rasterio in my Windows10 machine using pip. But for some reason, I am failing to do so in the virtual environment, getting the limited error message in red. I did try to install Numpy with no difficulty. Both GDAL and Rasterio wheel files were downloaded and installed successfully using pip on the system. But I am failing to instal the same files in the virtual environment.
Thanks for any help,
image.png

--


Eyal Saiet

The mind is not a vessel to be filled, but a fire to be kindled. Plutarch



--


Eyal Saiet


The mind is not a vessel to be filled, but a fire to be kindled. Plutarch


Re: rasterio vsicurl issues with docker-compose

Sean Gillies
 

Hi,

There's a big, but not obvious, difference between


and


In the first case, rasterio dispatches https (or http) URLs to vsicurl. In the second case, gdalinfo does not. Rather, it downloads the entire file to a temp location and opens it locally, not using vsicurl. I've forgotten where this is documented on the GDAL site.

To compare rasterio and gdalinfo, you must use `gdalinfo /vsicurl/https://example.com/file.tif`.

I think we can get to the bottom of this, starting with running the following command:


Can you do that and see if any HTTP connection problems are revealed?


On Tue, Dec 3, 2019 at 9:27 AM Madhav Desetty <madhav@...> wrote:

I am having a strange rasterio issue where if I build rasterio docker image from osgeo/gdal:ubuntu-small-latest as the base image and run it as docker-compose up, the rio fails to open any http based COG files. It works fine when the same TIF file is copied locally and accessed from a mounted drive. However, gdalinfo succeeds for COG file on S3 via HTTP url. When I run the docker image without docker-compose and use the docker commands (docker build && docker run), rio works for COG file on S3 via HTTP url. It seems like some environment is not being set or initialized when run via docker-compose and I am missing something obvious. Anyone ran into this issue before and any debugging tips to debug why the FileOpen is failing via Docker Compose? I set gdal CPL_DEBUG and CPL_LOG_ERRORS env to ON and logs are still minimal.

    

$ rio --version

OGRCT: PROJ >= 4.8.0 features enabled

OGRCT: Using locale-safe proj version

1.1.1

$ rio info -v http://oin-hotosm.s3.amazonaws.com/59c66c5223c8440011d7b1e4/0/7ad397c0-bba2-4f98-a08a-931ec3a6e943.tif

OGRCT: PROJ >= 4.8.0 features enabled

OGRCT: Using locale-safe proj version

Traceback (most recent call last):

  File "rasterio/_base.pyx", line 216, in rasterio._base.DatasetBase.__init__

  File "rasterio/_shim.pyx", line 67, in rasterio._shim.open_dataset

  File "rasterio/_err.pyx", line 205, in rasterio._err.exc_wrap_pointer

rasterio._err.CPLE_OpenFailedError: '/vsicurl/http://oin-hotosm.s3.amazonaws.com/59c66c5223c8440011d7b1e4/0/7ad397c0-bba2-4f98-a08a-931ec3a6e943.tif' does not exist in the file system, and is not recognized as a supported dataset name.

 

During handling of the above exception, another exception occurred:

 

Traceback (most recent call last):

  File "/usr/local/bin/rio", line 11, in <module>

    sys.exit(main_group())

  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 764, in __call__

    return self.main(*args, **kwargs)

  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 717, in main

    rv = self.invoke(ctx)

  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 1137, in invoke

    return _process_result(sub_ctx.command.invoke(sub_ctx))

  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 956, in invoke

    return ctx.invoke(self.callback, **ctx.params)

  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 555, in invoke

    return callback(*args, **kwargs)

  File "/usr/local/lib/python3.6/dist-packages/click/decorators.py", line 17, in new_func

    return f(get_current_context(), *args, **kwargs)

  File "/usr/local/lib/python3.6/dist-packages/rasterio/rio/info.py", line 66, in info

    with ctx.obj['env'], rasterio.open(input) as src:

  File "/usr/local/lib/python3.6/dist-packages/rasterio/env.py", line 445, in wrapper

    return f(*args, **kwds)

  File "/usr/local/lib/python3.6/dist-packages/rasterio/__init__.py", line 219, in open

    s = DatasetReader(path, driver=driver, sharing=sharing, **kwargs)

  File "rasterio/_base.pyx", line 218, in rasterio._base.DatasetBase.__init__

rasterio.errors.RasterioIOError: '/vsicurl/http://oin-hotosm.s3.amazonaws.com/59c66c5223c8440011d7b1e4/0/7ad397c0-bba2-4f98-a08a-931ec3a6e943.tif' does not exist in the file system, and is not recognized as a supported dataset name.

$ gdalinfo http://oin-hotosm.s3.amazonaws.com/59c66c5223c8440011d7b1e4/0/7ad397c0-bba2-4f98-a08a-931ec3a6e943.tif

HTTP: Fetch(http://oin-hotosm.s3.amazonaws.com/59c66c5223c8440011d7b1e4/0/7ad397c0-bba2-4f98-a08a-931ec3a6e943.tif)

HTTP: libcurl/7.58.0 GnuTLS/3.5.18 zlib/1.2.11 libidn2/2.0.4 libpsl/0.19.1 (+libidn2/2.0.4) nghttp2/1.30.0 librtmp/2.3

GDAL: GDALOpen(/vsimem/http_1/7ad397c0-bba2-4f98-a08a-931ec3a6e943.tif, this=0x55e917f3afa0) succeeds as GTiff.

GDAL: GDALOpen(http://oin-hotosm.s3.amazonaws.com/59c66c5223c8440011d7b1e4/0/7ad397c0-bba2-4f98-a08a-931ec3a6e943.tif, this=0x55e917f3afa0) succeeds as HTTP.

Driver: GTiff/GeoTIFF

GDAL: GDALDefaultOverviews::OverviewScan()

MDReaderPleiades: Not a Pleiades product

MDReaderPleiades: Not a Pleiades product

Files: none associated

Size is 49402, 28398

Coordinate System is:

PROJCRS["WGS 84 / UTM zone 20N",

    BASEGEOGCRS["WGS 84",

        DATUM["World Geodetic System 1984",

            ELLIPSOID["WGS 84",6378137,298.257223563,

                LENGTHUNIT["metre",1]]],

        PRIMEM["Greenwich",0,

            ANGLEUNIT["degree",0.0174532925199433]],

        ID["EPSG",4326]],

    CONVERSION["UTM zone 20N",

        METHOD["Transverse Mercator",

            ID["EPSG",9807]],

        PARAMETER["Latitude of natural origin",0,

            ANGLEUNIT["degree",0.0174532925199433],

            ID["EPSG",8801]],

        PARAMETER["Longitude of natural origin",-63,

            ANGLEUNIT["degree",0.0174532925199433],

            ID["EPSG",8802]],

        PARAMETER["Scale factor at natural origin",0.9996,

            SCALEUNIT["unity",1],

            ID["EPSG",8805]],

        PARAMETER["False easting",500000,

            LENGTHUNIT["metre",1],

            ID["EPSG",8806]],

        PARAMETER["False northing",0,

            LENGTHUNIT["metre",1],

            ID["EPSG",8807]]],

    CS[Cartesian,2],

        AXIS["(E)",east,

            ORDER[1],

            LENGTHUNIT["metre",1]],

        AXIS["(N)",north,

            ORDER[2],

            LENGTHUNIT["metre",1]],

    USAGE[

        SCOPE["unknown"],

        AREA["World - N hemisphere - 66°W to 60°W - by country"],

        BBOX[0,-66,84,-60]],

    ID["EPSG",32620]]

Data axis to CRS axis mapping: 1,2

Origin = (494088.931940000038594,1993386.902620000066236)

Pixel Size = (0.027070000000000,-0.027070000000000)

Metadata:

  AREA_OR_POINT=Area

  TIFFTAG_SOFTWARE=pix4dmapper

Image Structure Metadata:

  COMPRESSION=YCbCr JPEG

  INTERLEAVE=PIXEL

  SOURCE_COLOR_SPACE=YCbCr

GTiff: ScanDirectories()

GTiff: Opened 24701x14199 overview.

GTiff: Opened 12351x7100 overview.

GTiff: Opened 6176x3550 overview.

GTiff: Opened 3088x1775 overview.

GTiff: Opened 1544x888 overview.

GTiff: Opened 772x444 overview.

GTiff: Opened 386x222 overview.

Corner Coordinates:

Upper Left  (  494088.932, 1993386.903) ( 63d 3'21.05"W, 18d 1'44.14"N)

Lower Left  (  494088.932, 1992618.169) ( 63d 3'21.04"W, 18d 1'19.13"N)

Upper Right (  495426.244, 1993386.903) ( 63d 2'35.56"W, 18d 1'44.15"N)

Lower Right (  495426.244, 1992618.169) ( 63d 2'35.56"W, 18d 1'19.14"N)

Center      (  494757.588, 1993002.536) ( 63d 2'58.30"W, 18d 1'31.64"N)

Band 1 Block=512x512 Type=Byte, ColorInterp=Red

  NoData Value=-10000

  Overviews: 24701x14199, 12351x7100, 6176x3550, 3088x1775, 1544x888, 772x444, 386x222

Band 2 Block=512x512 Type=Byte, ColorInterp=Green

  NoData Value=-10000

  Overviews: 24701x14199, 12351x7100, 6176x3550, 3088x1775, 1544x888, 772x444, 386x222

Band 3 Block=512x512 Type=Byte, ColorInterp=Blue

  NoData Value=-10000

  Overviews: 24701x14199, 12351x7100, 6176x3550, 3088x1775, 1544x888, 772x444, 386x222

GDAL: GDALClose(http://oin-hotosm.s3.amazonaws.com/59c66c5223c8440011d7b1e4/0/7ad397c0-bba2-4f98-a08a-931ec3a6e943.tif, this=0x55e917f3afa0)

GDAL: In GDALDestroy - unloading GDAL shared library.


--
Sean Gillies


rasterio vsicurl issues with docker-compose

Madhav Desetty
 

I am having a strange rasterio issue where if I build rasterio docker image from osgeo/gdal:ubuntu-small-latest as the base image and run it as docker-compose up, the rio fails to open any http based COG files. It works fine when the same TIF file is copied locally and accessed from a mounted drive. However, gdalinfo succeeds for COG file on S3 via HTTP url. When I run the docker image without docker-compose and use the docker commands (docker build && docker run), rio works for COG file on S3 via HTTP url. It seems like some environment is not being set or initialized when run via docker-compose and I am missing something obvious. Anyone ran into this issue before and any debugging tips to debug why the FileOpen is failing via Docker Compose? I set gdal CPL_DEBUG and CPL_LOG_ERRORS env to ON and logs are still minimal.

    

$ rio --version

OGRCT: PROJ >= 4.8.0 features enabled

OGRCT: Using locale-safe proj version

1.1.1

$ rio info -v http://oin-hotosm.s3.amazonaws.com/59c66c5223c8440011d7b1e4/0/7ad397c0-bba2-4f98-a08a-931ec3a6e943.tif

OGRCT: PROJ >= 4.8.0 features enabled

OGRCT: Using locale-safe proj version

Traceback (most recent call last):

  File "rasterio/_base.pyx", line 216, in rasterio._base.DatasetBase.__init__

  File "rasterio/_shim.pyx", line 67, in rasterio._shim.open_dataset

  File "rasterio/_err.pyx", line 205, in rasterio._err.exc_wrap_pointer

rasterio._err.CPLE_OpenFailedError: '/vsicurl/http://oin-hotosm.s3.amazonaws.com/59c66c5223c8440011d7b1e4/0/7ad397c0-bba2-4f98-a08a-931ec3a6e943.tif' does not exist in the file system, and is not recognized as a supported dataset name.

 

During handling of the above exception, another exception occurred:

 

Traceback (most recent call last):

  File "/usr/local/bin/rio", line 11, in <module>

    sys.exit(main_group())

  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 764, in __call__

    return self.main(*args, **kwargs)

  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 717, in main

    rv = self.invoke(ctx)

  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 1137, in invoke

    return _process_result(sub_ctx.command.invoke(sub_ctx))

  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 956, in invoke

    return ctx.invoke(self.callback, **ctx.params)

  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 555, in invoke

    return callback(*args, **kwargs)

  File "/usr/local/lib/python3.6/dist-packages/click/decorators.py", line 17, in new_func

    return f(get_current_context(), *args, **kwargs)

  File "/usr/local/lib/python3.6/dist-packages/rasterio/rio/info.py", line 66, in info

    with ctx.obj['env'], rasterio.open(input) as src:

  File "/usr/local/lib/python3.6/dist-packages/rasterio/env.py", line 445, in wrapper

    return f(*args, **kwds)

  File "/usr/local/lib/python3.6/dist-packages/rasterio/__init__.py", line 219, in open

    s = DatasetReader(path, driver=driver, sharing=sharing, **kwargs)

  File "rasterio/_base.pyx", line 218, in rasterio._base.DatasetBase.__init__

rasterio.errors.RasterioIOError: '/vsicurl/http://oin-hotosm.s3.amazonaws.com/59c66c5223c8440011d7b1e4/0/7ad397c0-bba2-4f98-a08a-931ec3a6e943.tif' does not exist in the file system, and is not recognized as a supported dataset name.

$ gdalinfo http://oin-hotosm.s3.amazonaws.com/59c66c5223c8440011d7b1e4/0/7ad397c0-bba2-4f98-a08a-931ec3a6e943.tif

HTTP: Fetch(http://oin-hotosm.s3.amazonaws.com/59c66c5223c8440011d7b1e4/0/7ad397c0-bba2-4f98-a08a-931ec3a6e943.tif)

HTTP: libcurl/7.58.0 GnuTLS/3.5.18 zlib/1.2.11 libidn2/2.0.4 libpsl/0.19.1 (+libidn2/2.0.4) nghttp2/1.30.0 librtmp/2.3

GDAL: GDALOpen(/vsimem/http_1/7ad397c0-bba2-4f98-a08a-931ec3a6e943.tif, this=0x55e917f3afa0) succeeds as GTiff.

GDAL: GDALOpen(http://oin-hotosm.s3.amazonaws.com/59c66c5223c8440011d7b1e4/0/7ad397c0-bba2-4f98-a08a-931ec3a6e943.tif, this=0x55e917f3afa0) succeeds as HTTP.

Driver: GTiff/GeoTIFF

GDAL: GDALDefaultOverviews::OverviewScan()

MDReaderPleiades: Not a Pleiades product

MDReaderPleiades: Not a Pleiades product

Files: none associated

Size is 49402, 28398

Coordinate System is:

PROJCRS["WGS 84 / UTM zone 20N",

    BASEGEOGCRS["WGS 84",

        DATUM["World Geodetic System 1984",

            ELLIPSOID["WGS 84",6378137,298.257223563,

                LENGTHUNIT["metre",1]]],

        PRIMEM["Greenwich",0,

            ANGLEUNIT["degree",0.0174532925199433]],

        ID["EPSG",4326]],

    CONVERSION["UTM zone 20N",

        METHOD["Transverse Mercator",

            ID["EPSG",9807]],

        PARAMETER["Latitude of natural origin",0,

            ANGLEUNIT["degree",0.0174532925199433],

            ID["EPSG",8801]],

        PARAMETER["Longitude of natural origin",-63,

            ANGLEUNIT["degree",0.0174532925199433],

            ID["EPSG",8802]],

        PARAMETER["Scale factor at natural origin",0.9996,

            SCALEUNIT["unity",1],

            ID["EPSG",8805]],

        PARAMETER["False easting",500000,

            LENGTHUNIT["metre",1],

            ID["EPSG",8806]],

        PARAMETER["False northing",0,

            LENGTHUNIT["metre",1],

            ID["EPSG",8807]]],

    CS[Cartesian,2],

        AXIS["(E)",east,

            ORDER[1],

            LENGTHUNIT["metre",1]],

        AXIS["(N)",north,

            ORDER[2],

            LENGTHUNIT["metre",1]],

    USAGE[

        SCOPE["unknown"],

        AREA["World - N hemisphere - 66°W to 60°W - by country"],

        BBOX[0,-66,84,-60]],

    ID["EPSG",32620]]

Data axis to CRS axis mapping: 1,2

Origin = (494088.931940000038594,1993386.902620000066236)

Pixel Size = (0.027070000000000,-0.027070000000000)

Metadata:

  AREA_OR_POINT=Area

  TIFFTAG_SOFTWARE=pix4dmapper

Image Structure Metadata:

  COMPRESSION=YCbCr JPEG

  INTERLEAVE=PIXEL

  SOURCE_COLOR_SPACE=YCbCr

GTiff: ScanDirectories()

GTiff: Opened 24701x14199 overview.

GTiff: Opened 12351x7100 overview.

GTiff: Opened 6176x3550 overview.

GTiff: Opened 3088x1775 overview.

GTiff: Opened 1544x888 overview.

GTiff: Opened 772x444 overview.

GTiff: Opened 386x222 overview.

Corner Coordinates:

Upper Left  (  494088.932, 1993386.903) ( 63d 3'21.05"W, 18d 1'44.14"N)

Lower Left  (  494088.932, 1992618.169) ( 63d 3'21.04"W, 18d 1'19.13"N)

Upper Right (  495426.244, 1993386.903) ( 63d 2'35.56"W, 18d 1'44.15"N)

Lower Right (  495426.244, 1992618.169) ( 63d 2'35.56"W, 18d 1'19.14"N)

Center      (  494757.588, 1993002.536) ( 63d 2'58.30"W, 18d 1'31.64"N)

Band 1 Block=512x512 Type=Byte, ColorInterp=Red

  NoData Value=-10000

  Overviews: 24701x14199, 12351x7100, 6176x3550, 3088x1775, 1544x888, 772x444, 386x222

Band 2 Block=512x512 Type=Byte, ColorInterp=Green

  NoData Value=-10000

  Overviews: 24701x14199, 12351x7100, 6176x3550, 3088x1775, 1544x888, 772x444, 386x222

Band 3 Block=512x512 Type=Byte, ColorInterp=Blue

  NoData Value=-10000

  Overviews: 24701x14199, 12351x7100, 6176x3550, 3088x1775, 1544x888, 772x444, 386x222

GDAL: GDALClose(http://oin-hotosm.s3.amazonaws.com/59c66c5223c8440011d7b1e4/0/7ad397c0-bba2-4f98-a08a-931ec3a6e943.tif, this=0x55e917f3afa0)

GDAL: In GDALDestroy - unloading GDAL shared library.

$ 


Re: Representing many raster files as one big file

rasterio@...
 

Hi Dion,

Thanks a lot, that worked!

Cheers,


Sean

On 2019-12-03 15:31, Dion Häfner wrote:
Hey Sean,
as far as I know, this is not possible with "just" rasterio. But there is one thing you can do.
GDAL (the library rasterio is built upon) supports building virtual datasets (https://gdal.org/programs/gdalbuildvrt.html). So you should be able to do
$ gdalbuildvrt dem.vrt *.tif
and then read this VRT file with rasterio.open (which should work out of the box).
Good luck!
Dion
On 03/12/2019 15.09, rasterio via Groups.Io wrote:
Hi all,

I'm new to rasterio and it's so far working very nicely for reading some GeoTIFF files (specifically, the EU DEM elevation data set). I have a question I hope someone can answer: it's about whether it's possible to take the 27 contiguous individual TIFF files that form this data set and represent them as if they were one big file.

Currently I have to load each file in a loop to perform my analysis, but ideally I would like to be able to give all of the file names and their coordinates to some function that then lets me fetch data from anywhere in the whole dataset using a single `dataset.read` with my window settings. This would avoid me having to deal with the edges where the window I'm looking at would overlap with other tiles. This would ideally work without requiring the whole dataset to be in memory, since it's huge (10s of GB).

Is this possible already in rasterio?

Cheers,


Sean



Re: Representing many raster files as one big file

Dion Häfner
 

Hey Sean,

as far as I know, this is not possible with "just" rasterio. But there is one thing you can do.

GDAL (the library rasterio is built upon) supports building virtual datasets (https://gdal.org/programs/gdalbuildvrt.html). So you should be able to do

$ gdalbuildvrt dem.vrt *.tif

and then read this VRT file with rasterio.open (which should work out of the box).

Good luck!
Dion

On 03/12/2019 15.09, rasterio via Groups.Io wrote:
Hi all,
I'm new to rasterio and it's so far working very nicely for reading some GeoTIFF files (specifically, the EU DEM elevation data set). I have a question I hope someone can answer: it's about whether it's possible to take the 27 contiguous individual TIFF files that form this data set and represent them as if they were one big file.
Currently I have to load each file in a loop to perform my analysis, but ideally I would like to be able to give all of the file names and their coordinates to some function that then lets me fetch data from anywhere in the whole dataset using a single `dataset.read` with my window settings. This would avoid me having to deal with the edges where the window I'm looking at would overlap with other tiles. This would ideally work without requiring the whole dataset to be in memory, since it's huge (10s of GB).
Is this possible already in rasterio?
Cheers,
Sean


Representing many raster files as one big file

rasterio@...
 

Hi all,

I'm new to rasterio and it's so far working very nicely for reading some GeoTIFF files (specifically, the EU DEM elevation data set). I have a question I hope someone can answer: it's about whether it's possible to take the 27 contiguous individual TIFF files that form this data set and represent them as if they were one big file.

Currently I have to load each file in a loop to perform my analysis, but ideally I would like to be able to give all of the file names and their coordinates to some function that then lets me fetch data from anywhere in the whole dataset using a single `dataset.read` with my window settings. This would avoid me having to deal with the edges where the window I'm looking at would overlap with other tiles. This would ideally work without requiring the whole dataset to be in memory, since it's huge (10s of GB).

Is this possible already in rasterio?

Cheers,


Sean


problems installing GDAL and Rasterio in virtual environment in Windows

Eyal Saiet
 

Hello,
I was successful in installing GDAL and Rasterio in my Windows10 machine using pip. But for some reason, I am failing to do so in the virtual environment, getting the limited error message in red. I did try to install Numpy with no difficulty. Both GDAL and Rasterio wheel files were downloaded and installed successfully using pip on the system. But I am failing to instal the same files in the virtual environment.
Thanks for any help,
image.png

--


Eyal Saiet

The mind is not a vessel to be filled, but a fire to be kindled. Plutarch


Re: Problem with AAIGrid overwrite

Sean Gillies
 

I suspect the error is coming from the gdal module function, not from rasterio. If it was coming from rasterio, you'd see a log message instead.


On Mon, Dec 2, 2019 at 3:32 AM Amine Aboufirass <amine.aboufirass@...> wrote:
The strange thing is that it actually updates the file but still throws the error. 

On Mon, Dec 2, 2019 at 11:17 AM Amine Aboufirass via Groups.Io <amine.aboufirass=gmail.com@groups.io> wrote:
Dear Sean,

I tried your example under an administrator command prompt and I got no problems. So I then tried running my own example under the same prompt and got the error again. 

I have attached a zip file which contains my example. Would you care to take a look? I really can't pinpoint why this behavior occurs.

Thanks in advance for your help,

Regards,

Amine
 
...

--
Sean Gillies


Re: Problem with AAIGrid overwrite

Amine Aboufirass <amine.aboufirass@...>
 

The strange thing is that it actually updates the file but still throws the error. 

On Mon, Dec 2, 2019 at 11:17 AM Amine Aboufirass via Groups.Io <amine.aboufirass=gmail.com@groups.io> wrote:
Dear Sean,

I tried your example under an administrator command prompt and I got no problems. So I then tried running my own example under the same prompt and got the error again. 

I have attached a zip file which contains my example. Would you care to take a look? I really can't pinpoint why this behavior occurs.

Thanks in advance for your help,

Regards,

Amine

On Fri, Nov 29, 2019 at 8:14 PM Sean Gillies <sean.gillies@...> wrote:
Amine,

I'm unable to reproduce this with rasterio 1.1.0. The following script can be executed over and over again and the shade.asc file is overwritten with no error and no messages.

import rasterio

with rasterio.open("/Users/seang/code/rasterio/tests/data/shade.tif") as tif:
    profile = tif.profile
    profile["driver"] = "AAIGrid"
    with rasterio.open("/tmp/shade.asc", "w", **profile) as asc:
        asc.write(tif.read())

It might actually be a permissions issue on your computer.


On Fri, Nov 29, 2019 at 7:38 AM Amine Aboufirass <amine.aboufirass@...> wrote:
Dear All,

For one project I am reading and writing these test raster datasets as ASCII grids. I have found that overwriting these grids is impossible and I get the following error:

ERROR 1: Deleting test_data/out.asc failed:
Permission denied

This is quite annoying because for my testing I need to overwrite and delete files repeatedly. I have to keep deleting them again before I run my application each time and this is quite time consuming.

I have the impression this is related to the GDAL creation options but I am not sure. Could someone please enlighten me? Thanks.

Regards,

Amine



--
Sean Gillies


Re: Problem with AAIGrid overwrite

Amine Aboufirass <amine.aboufirass@...>
 

Dear Sean,

I tried your example under an administrator command prompt and I got no problems. So I then tried running my own example under the same prompt and got the error again. 

I have attached a zip file which contains my example. Would you care to take a look? I really can't pinpoint why this behavior occurs.

Thanks in advance for your help,

Regards,

Amine

On Fri, Nov 29, 2019 at 8:14 PM Sean Gillies <sean.gillies@...> wrote:
Amine,

I'm unable to reproduce this with rasterio 1.1.0. The following script can be executed over and over again and the shade.asc file is overwritten with no error and no messages.

import rasterio

with rasterio.open("/Users/seang/code/rasterio/tests/data/shade.tif") as tif:
    profile = tif.profile
    profile["driver"] = "AAIGrid"
    with rasterio.open("/tmp/shade.asc", "w", **profile) as asc:
        asc.write(tif.read())

It might actually be a permissions issue on your computer.


On Fri, Nov 29, 2019 at 7:38 AM Amine Aboufirass <amine.aboufirass@...> wrote:
Dear All,

For one project I am reading and writing these test raster datasets as ASCII grids. I have found that overwriting these grids is impossible and I get the following error:

ERROR 1: Deleting test_data/out.asc failed:
Permission denied

This is quite annoying because for my testing I need to overwrite and delete files repeatedly. I have to keep deleting them again before I run my application each time and this is quite time consuming.

I have the impression this is related to the GDAL creation options but I am not sure. Could someone please enlighten me? Thanks.

Regards,

Amine



--
Sean Gillies


Re: Geodatabase, FileGeodatabase, and Esri Geodatabase

Alan Snow
 

Hello,

It appears the GeoPackage format can support vector and raster data in GDAL similar to the FIleGDB format.

https://gdal.org/drivers/raster/gpkg.html
https://gdal.org/drivers/vector/gpkg.html

Not sure how to do both in the same file as I haven't tried to do it myself. But, sounds worth a shot. Hope that helps.

Best,
Alan


Re: Problem with AAIGrid overwrite

Sean Gillies
 

Amine,

I'm unable to reproduce this with rasterio 1.1.0. The following script can be executed over and over again and the shade.asc file is overwritten with no error and no messages.

import rasterio

with rasterio.open("/Users/seang/code/rasterio/tests/data/shade.tif") as tif:
    profile = tif.profile
    profile["driver"] = "AAIGrid"
    with rasterio.open("/tmp/shade.asc", "w", **profile) as asc:
        asc.write(tif.read())

It might actually be a permissions issue on your computer.


On Fri, Nov 29, 2019 at 7:38 AM Amine Aboufirass <amine.aboufirass@...> wrote:
Dear All,

For one project I am reading and writing these test raster datasets as ASCII grids. I have found that overwriting these grids is impossible and I get the following error:

ERROR 1: Deleting test_data/out.asc failed:
Permission denied

This is quite annoying because for my testing I need to overwrite and delete files repeatedly. I have to keep deleting them again before I run my application each time and this is quite time consuming.

I have the impression this is related to the GDAL creation options but I am not sure. Could someone please enlighten me? Thanks.

Regards,

Amine



--
Sean Gillies


Re: metadata and profile: What's the difference?

Amine Aboufirass <amine.aboufirass@...>
 

Thanks for your reply. I guess the profile is more comprehensive then. Is that a good argument for dropping metadata then? 

On Fri, Nov 29, 2019 at 4:11 PM Sean Gillies <sean.gillies@...> wrote:
Indeed they are largely redundant. The meta property came first. Then we introduced a profile property to better support the pattern of using one dataset's profile as keyword arguments for creating new similar datasets. Note that the profile may contain items that don't appear in the meta property. 

On Fri, Nov 29, 2019, 9:57 AM Amine Aboufirass <amine.aboufirass@...> wrote:
Also, what is the difference between a dataset's metadata and profile? I have the impression one of these values is redundant. Please see example below:

>>> dataset.profile
{'driver': 'AAIGrid', 'dtype': 'int32', 'nodata': -9999.0, 'width': 8, 'height': 8, 'count': 1, 'crs': None, 'transform': Affine(1.0, 0.0, 0.0,
       0.0, -1.0, 8.0), 'tiled': False}
>>> dataset.meta
{'driver': 'AAIGrid', 'dtype': 'int32', 'nodata': -9999.0, 'width': 8, 'height': 8, 'count': 1, 'crs': None, 'transform': Affine(1.0, 0.0, 0.0,
       0.0, -1.0, 8.0)}


Re: metadata and profile: What's the difference?

Sean Gillies
 

Indeed they are largely redundant. The meta property came first. Then we introduced a profile property to better support the pattern of using one dataset's profile as keyword arguments for creating new similar datasets. Note that the profile may contain items that don't appear in the meta property. 


On Fri, Nov 29, 2019, 9:57 AM Amine Aboufirass <amine.aboufirass@...> wrote:
Also, what is the difference between a dataset's metadata and profile? I have the impression one of these values is redundant. Please see example below:

>>> dataset.profile
{'driver': 'AAIGrid', 'dtype': 'int32', 'nodata': -9999.0, 'width': 8, 'height': 8, 'count': 1, 'crs': None, 'transform': Affine(1.0, 0.0, 0.0,
       0.0, -1.0, 8.0), 'tiled': False}
>>> dataset.meta
{'driver': 'AAIGrid', 'dtype': 'int32', 'nodata': -9999.0, 'width': 8, 'height': 8, 'count': 1, 'crs': None, 'transform': Affine(1.0, 0.0, 0.0,
       0.0, -1.0, 8.0)}