Date   

Python 2/3 compatibility management in rasterio

adrienoyono@...
 

Hello everyone,

First of all, this is my occasion to say thanks for the rasterio project. It has been very helpful simplifying my life last year. I write a sofware that depends on rasterio, should ensure compatibility with Python 2 and Python 3, and should be deployed behind a web server (Apache with mod_wsgi). This last requirement triggers a problem caused by the way rasterio ensures Python 2/3 compatibility as I understand it after reading the "compat" module and the "setup.py".

Below is the stacktrace I have from Apache when trying to load my app:

Traceback (most recent call last):
File "/opt/eodag/venv/lib/python2.7/site-packages/eodag/rest/server.wsgi", line 3, in <module>
  from eodag.rest.server import app as application
File "/opt/eodag/venv/lib/python2.7/site-packages/eodag/__init__.py", line 18, in <module>
  from .api.core import EODataAccessGateway      # noqa
File "/opt/eodag/venv/lib/python2.7/site-packages/eodag/api/core.py", line 27, in <module>
  from eodag.api.search_result import SearchResult
File "/opt/eodag/venv/lib/python2.7/site-packages/eodag/api/search_result.py", line 26, in <module>
  from eodag.api.product import EOProduct
File "/opt/eodag/venv/lib/python2.7/site-packages/eodag/api/product/__init__.py", line 18, in <module>
  from ._product import EOProduct  # noqa
File "/opt/eodag/venv/lib/python2.7/site-packages/eodag/api/product/_product.py", line 25, in <module>
  import rasterio
File "/opt/eodag/venv/lib/python2.7/site-packages/rasterio/__init__.py", line 22, in <module>
  from rasterio._base import gdal_version
File "rasterio/_base.pyx", line 20, in init rasterio._base
File "/opt/eodag/venv/lib/python2.7/site-packages/rasterio/compat.py", line 26, in <module>
  import mock
ImportError: No module named mock
I guess "mock" is used in rasterio for testing purpose, I do use it for that myself. If that's the case, it is weird to have rasterio brake if we are in Python 2 without the "mock" module being installed, which is clearly the case here. To have mock installed at the same time as rasterio, one must do: "pip install rasterio[dev]". However, we always just do "pip install rasterio" (which fails to install "mock").

I think compatibility for testing purposes should be decoupled from compatibility for the core functionality. Is it possible to have it done in rasterio ? That would help avoid this kind of problem.

Thank you in advance for your time on this issue.


Re: Output to grib2?

Shane Mill - NOAA Affiliate
 

On Fri, Mar 15, 2019 at 11:02 AM, Sean Gillies wrote:
rio convert -f GRIB ~/code/rasterio/tests/data/RGB.byte.tif /tmp/rgb.grb2


Sean, thanks for the quick response! You are correct, that does work. I think my issue may have been that I was using gdalinfo to check the resulting grib2 file, and my gdal version was older. When I check with rio info it seems to be correct.

Thanks again!

-Shane


Re: Output to grib2?

Sean Gillies
 

Hi Shane,

Rasterio's rio-convert command can convert from GeoTIFF to GRIB, so it should be possible

$ rio convert -f GRIB ~/code/rasterio/tests/data/RGB.byte.tif /tmp/rgb.grb2
$ rio info /tmp/rgb.grb2
{"bounds": [101985.0, 2611486.28, 339316.64, 2826915.0], "colorinterp": ["undefined", "undefined", "undefined"], "count": 3, "crs": "PROJCS[\"unnamed\",GEOGCS[\"Coordinate System imported from GRIB file\",DATUM[\"WGS_1984\",SPHEROID[\"WGS 84\",6378137,298.257223563]],PRIMEM[\"Greenwich\",0],UNIT[\"degree\",0.0174532925199433]],PROJECTION[\"Transverse_Mercator\"],PARAMETER[\"latitude_of_origin\",0],PARAMETER[\"central_meridian\",-2072.483648],PARAMETER[\"scale_factor\",0.9996],PARAMETER[\"false_easting\",500000],PARAMETER[\"false_northing\",0],UNIT[\"Metre\",1]]", "descriptions": ["0[-] RESERVED(0) (Reserved)", "0[-] RESERVED(0) (Reserved)", "0[-] RESERVED(0) (Reserved)"], "driver": "GRIB", "dtype": "float64", "height": 718, "indexes": [1, 2, 3], "lnglat": [84.75845370919801, 24.561589206961006], "mask_flags": [["nodata"], ["nodata"], ["nodata"]], "nodata": 0.0, "res": [300.04, 300.04], "shape": [718, 791], "tiled": false, "transform": [300.04, 0.0, 101985.0, 0.0, -300.04, 2826915.0, 0.0, 0.0, 1.0], "units": [null, null, null], "width": 791}

I'm not a regular GRIB user, but it looks to me that GDAL produces a GRIB2 file when you write using the GRIB format driver, and that the format driver reads either version without a need to specify. GRIB2 writing requires GDAL 2.3.0 or newer. The latest rasterio wheels on PyPI include GDAL 2.4.0, and should be sufficient.

On Fri, Mar 15, 2019 at 11:53 AM Shane Mill - NOAA Affiliate via Groups.Io <shane.mill=noaa.gov@groups.io> wrote:
Is there anyway to output to grib2 after performing map algebra? For example, If you are performing a calculation between two grib2 bands, which rasterio does quite well, are you able to transform the numpy array into a grib2 file? I have been able to do this with geotiff but not grib2.



--
Sean Gillies


Output to grib2?

Shane Mill - NOAA Affiliate
 

Is there anyway to output to grib2 after performing map algebra? For example, If you are performing a calculation between two grib2 bands, which rasterio does quite well, are you able to transform the numpy array into a grib2 file? I have been able to do this with geotiff but not grib2.


Rasterio 1.0.20

Sean Gillies
 

Hi all,

Rasterio 1.0.20 is on PyPI today. I hope you'll like the changes: https://github.com/mapbox/rasterio/blob/master/CHANGES.txt#L4-L24.

I discovered that 1.0.19 was defective while building linux wheels to distribute. No distributions of 1.0.19 were uploaded.

--
Sean Gillies


Re: Implementing -scale functionality of gdal_translate

linden.kyle@...
 

That worked perfectly. Thanks for all your help.


Re: Implementing -scale functionality of gdal_translate

Sean Gillies
 

Hi,

On Wed, Feb 6, 2019 at 2:32 PM <linden.kyle@...> wrote:
So I'm seeing some odd behavior now. I ended up trying the masked_array solution, but I'm seeing breaks in my data at the tile boundaries.

Here's what the data looks like if I use reprojection with just the source band (code: https://pastebin.com/dZVRVKi2): https://imgur.com/WpFKfij

Here's what the data looks like I read into a masked array and use an ndarray source (code: https://pastebin.com/z1xd8mXu): https://imgur.com/2QkQQZf
Ironically enough I get the exact same result with my solution that doesn't use a masked array, but instead modifies the transform: https://pastebin.com/7ph1VYWX

Your code looks okay. I recommend padding your read window by a few pixels on each side so that tile_window, tile_data, and tile_transform are slight dilated. I think this is very likely to eliminate what looks like an off-by-one error in the reprojection, an error that you don't see when you use the entire source dataset as the reprojection source. Does that make sense?

--
Sean Gillies


Rasterio 1.0.18

Sean Gillies
 

Hi all,

We've made three new releases this week to fix regressions introduced in version 1.0.14. Please upgrade to 1.0.18 to take advantage of the changes in 1.0.14, but without suffering from the bugs. See https://github.com/mapbox/rasterio/blob/master/CHANGES.txt for more details.

--
Sean Gillies


Re: Reading an image by overlapped blocks of chosen depth

mail2abhisek.maiti@...
 

I was not aware of the boundless reading. Fantastic implementation. Thanks.


Re: Reading an image by overlapped blocks of chosen depth

Luke
 

I've put a couple of methods in my answer to your SO question, by expanding the block windows and also for arbitrary windows (unrelated to internal blocks/tiles).


Re: Implementing -scale functionality of gdal_translate

linden.kyle@...
 

So I'm seeing some odd behavior now. I ended up trying the masked_array solution, but I'm seeing breaks in my data at the tile boundaries.

Here's what the data looks like if I use reprojection with just the source band (code: https://pastebin.com/dZVRVKi2): https://imgur.com/WpFKfij

Here's what the data looks like I read into a masked array and use an ndarray source (code: https://pastebin.com/z1xd8mXu): https://imgur.com/2QkQQZf
Ironically enough I get the exact same result with my solution that doesn't use a masked array, but instead modifies the transform: https://pastebin.com/7ph1VYWX


Re: Implementing -scale functionality of gdal_translate

linden.kyle@...
 

Sean, no problem. Thanks for writing back. I'll keep that in mind for the future. What I ended up doing instead was modifying the tile window transform (which is passed to the reproject method) in the case where either the left or top coordinate is outside the bounds of the geotiff. In that case I change the translation of the transform to the left and/or top bound of the geotiff respectively. Not sure if this is the "right" way to do it or not, but it seems to be working.


Re: Implementing -scale functionality of gdal_translate

Sean Gillies
 

Hi,

I apologize for being slow to reply, I've been occupied with fixing bugs in the last couple of releases.

In theory it is possible to add an option to the read method which will cast the data because GDAL already supports this well in versions 2 and up. I haven't pursued this because converting unsigned int to float is trivial, if not cheap, with numpy:

   data = dataset.read().astype('float64')

And if you pass masked=True to read, you'll get a masked array and then can apply a fill_value of numpy.nan under the invalid data mask. Is this useful for you or have you already been down this path?

GDAL's data type casting would not produce an extra copy of the data and so would be optimal for some applications.


On Tue, Feb 5, 2019 at 11:09 AM <linden.kyle@...> wrote:
Ok, so it appears the issue was that I was attempting to do a windowed read outside the bounds of the geotiff (for TMS edge tiles) and `src.read()` was only returning the data inside the bounds since I wasn't using the `boundless=True` option. However, the next problem I am running into is that I'd like to use `fill_value=numpy.nan` to fill in the edge data with NaN values. However, my data is UInt8. Is there any way to specify an option to cast the data read by the call to `read()` into float32 so that I can support this?



--
Sean Gillies


Re: Implementing -scale functionality of gdal_translate

linden.kyle@...
 

Ok, so it appears the issue was that I was attempting to do a windowed read outside the bounds of the geotiff (for TMS edge tiles) and `src.read()` was only returning the data inside the bounds since I wasn't using the `boundless=True` option. However, the next problem I am running into is that I'd like to use `fill_value=numpy.nan` to fill in the edge data with NaN values. However, my data is UInt8. Is there any way to specify an option to cast the data read by the call to `read()` into float32 so that I can support this?


Reading an image by overlapped blocks of chosen depth

mail2abhisek.maiti@...
 

In rasterio, reading image array in the unit of blocks is pretty convenient and useful. However, Overlapping blocks could be very useful for certain applications which requires kernel wise processing (such as simple smoothing filter).  So far there is no provision to make a block overlap with its neighboring blocks as per requirement of the application. It would be immensely handy if this feature gets implemented in rasterio. Related Stackoverflow Question.


Re: Implementing -scale functionality of gdal_translate

linden.kyle@...
 

Hi Sean, I'm having some trouble with the implementation of this and was wondering if you might be able to give me a hand. Here's the code I am working with: https://pastebin.com/kecx0D2b. What I'm trying to do is take a geotiff and convert it to a set of TMS-formatted tiles (including reprojecting to web mercator). In this example `tile_bounds` are the bounds of the tile in web mercator. I originally had this code sample working until I attempted to add in the code to do the pixel scaling. Here's the original working code: https://pastebin.com/dZVRVKi2

Here's what the output looks like with the broken code: https://imgur.com/xLtWrGZ
And here's the correct version (but no support for pixel scaling): https://imgur.com/aoNCyIp


Re: New rasterio versions don't handle blank SRS gracefully

Sean Gillies
 

Hi Nicholas,

Thanks for the explanation. We have at least one more related regression in 1.0.15 and I intend to fix both of these in 1.0.16.


On Fri, Feb 1, 2019 at 10:36 AM <nicholas.maxwell@...> wrote:
Short version:

The following causes errors, in rasterio 1.0.15,  but is fine in rasterio 1.0.9. 

from rasterio.crs import CRS
print(CRS())
The following are the only tests in rasterio to check blank CRSs.

def test_null_crs_equality():
assert CRS() == CRS()
a = CRS()
assert a == a
assert not a != a


def test_null_and_valid_crs_equality():
assert (CRS() == CRS(init='epsg:4326')) is False

I feel like print(CRS()) should not cause errors - CRS().__str__ should just return an empty string, for example.

Longer version:

We work with pre-orthorectified sat imagery, which have blank geotransforms (but may have GCPs set), eg,

gdalinfo /data/16APR27183709-P1BS-502376654080_01_P009.tif 
Driver: GTiff/GeoTIFF
Files: /data/16APR27183709-P1BS-502376654080_01_P009.tif
Size is 3201, 3281
Coordinate System is `'
GCP Projection = 
GEOGCS["WGS 84",
    DATUM["WGS_1984",
        SPHEROID["WGS 84",6378137,298.257223563,
...

This causes problems on newer rasterio versions (presumably due to the switch to WKT representation for CRS). Example:

import rasterio
with rasterio.open('/data/16APR27183709-P1BS-502376654080_01_P009.tif') as src:
print("CRS:", src.crs)

On rasterio 1.0.9, this is fine, src.crs evaluates to None:

python ~/test.py
/home/nick/venvs/ras9/lib/python3.6/site-packages/rasterio-1.0.9-py3.6-linux-x86_64.egg/rasterio/__init__.py:217: NotGeoreferencedWarning: Dataset has no geotransform set. The identity matrix may be returned.
  s = DatasetReader(path, driver=driver, **kwargs)
CRS: None

The NotGeoreferencedWarning is fine, we explicitly ignore it. The point is that src.crs is None.
On 1.0.15, however,

python ~/test.py
/home/nick/venvs/ras15/lib/python3.6/site-packages/rasterio/__init__.py:216: NotGeoreferencedWarning: Dataset has no geotransform set. The identity matrix may be returned.
  s = DatasetReader(path, driver=driver, **kwargs)
CRS: Traceback (most recent call last):
  File "rasterio/_crs.pyx", line 274, in rasterio._crs._CRS.to_dict
  File "rasterio/_err.pyx", line 194, in rasterio._err.exc_wrap_ogrerr
rasterio._err.CPLE_BaseError: OGR Error code 7

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/nick/test.py", line 3, in <module>
    print("CRS:", src.crs)
  File "/home/nick/venvs/ras15/lib/python3.6/site-packages/rasterio/crs.py", line 245, in to_string
    return self.to_wkt() or self.to_proj4()
  File "/home/nick/venvs/ras15/lib/python3.6/site-packages/rasterio/crs.py", line 106, in to_proj4
    return ' '.join(['+{}={}'.format(key, val) for key, val in self.data.items()])
  File "/home/nick/venvs/ras15/lib/python3.6/site-packages/rasterio/crs.py", line 170, in data
    self._data = self.to_dict()
  File "/home/nick/venvs/ras15/lib/python3.6/site-packages/rasterio/crs.py", line 164, in to_dict
    return self._crs.to_dict()
  File "rasterio/_crs.pyx", line 277, in rasterio._crs._CRS.to_dict
rasterio.errors.CRSError: The WKT could not be parsed. OGR Error code 7

Interestingly, the GDAL Python API handles this in a different way:

from osgeo import gdal
ds = gdal.Open('/data/16APR27183709-P1BS-502376654080_01_P009.tif')
prj = ds.GetProjection()
print(prj)

which prints

GEOGCS["WGS 84",DATUM["WGS_1984",SPHEROID["WGS 84",6378137,298.257223563,AUTHORITY["EPSG","7030"]],AUTHORITY["EPSG","6326"]],PRIMEM["Greenwich",0],UNIT["degree",0.0174532925199433],AUTHORITY["EPSG","4326"]]
I guess GDAL assumes the GCP projection, in case of blank geotransform.

In any case, I think the src.crs is None behavior is good, since we can just look at src.gcps for the gcps.

I will add a workaround on our side, since
print(CRS().is_valid)
works, (that is, CRS().is_valid evaluates to False, without throwing exceptions.



--
Sean Gillies


New rasterio versions don't handle blank SRS gracefully

nicholas.maxwell@...
 

Short version:

The following causes errors, in rasterio 1.0.15,  but is fine in rasterio 1.0.9. 

from rasterio.crs import CRS
print(CRS())
The following are the only tests in rasterio to check blank CRSs.

def test_null_crs_equality():
assert CRS() == CRS()
a = CRS()
assert a == a
assert not a != a


def test_null_and_valid_crs_equality():
assert (CRS() == CRS(init='epsg:4326')) is False

I feel like print(CRS()) should not cause errors - CRS().__str__ should just return an empty string, for example.

Longer version:

We work with pre-orthorectified sat imagery, which have blank geotransforms (but may have GCPs set), eg,

gdalinfo /data/16APR27183709-P1BS-502376654080_01_P009.tif 
Driver: GTiff/GeoTIFF
Files: /data/16APR27183709-P1BS-502376654080_01_P009.tif
Size is 3201, 3281
Coordinate System is `'
GCP Projection = 
GEOGCS["WGS 84",
    DATUM["WGS_1984",
        SPHEROID["WGS 84",6378137,298.257223563,
...

This causes problems on newer rasterio versions (presumably due to the switch to WKT representation for CRS). Example:

import rasterio
with rasterio.open('/data/16APR27183709-P1BS-502376654080_01_P009.tif') as src:
print("CRS:", src.crs)

On rasterio 1.0.9, this is fine, src.crs evaluates to None:

python ~/test.py
/home/nick/venvs/ras9/lib/python3.6/site-packages/rasterio-1.0.9-py3.6-linux-x86_64.egg/rasterio/__init__.py:217: NotGeoreferencedWarning: Dataset has no geotransform set. The identity matrix may be returned.
  s = DatasetReader(path, driver=driver, **kwargs)
CRS: None

The NotGeoreferencedWarning is fine, we explicitly ignore it. The point is that src.crs is None.
On 1.0.15, however,

python ~/test.py
/home/nick/venvs/ras15/lib/python3.6/site-packages/rasterio/__init__.py:216: NotGeoreferencedWarning: Dataset has no geotransform set. The identity matrix may be returned.
  s = DatasetReader(path, driver=driver, **kwargs)
CRS: Traceback (most recent call last):
  File "rasterio/_crs.pyx", line 274, in rasterio._crs._CRS.to_dict
  File "rasterio/_err.pyx", line 194, in rasterio._err.exc_wrap_ogrerr
rasterio._err.CPLE_BaseError: OGR Error code 7

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/nick/test.py", line 3, in <module>
    print("CRS:", src.crs)
  File "/home/nick/venvs/ras15/lib/python3.6/site-packages/rasterio/crs.py", line 245, in to_string
    return self.to_wkt() or self.to_proj4()
  File "/home/nick/venvs/ras15/lib/python3.6/site-packages/rasterio/crs.py", line 106, in to_proj4
    return ' '.join(['+{}={}'.format(key, val) for key, val in self.data.items()])
  File "/home/nick/venvs/ras15/lib/python3.6/site-packages/rasterio/crs.py", line 170, in data
    self._data = self.to_dict()
  File "/home/nick/venvs/ras15/lib/python3.6/site-packages/rasterio/crs.py", line 164, in to_dict
    return self._crs.to_dict()
  File "rasterio/_crs.pyx", line 277, in rasterio._crs._CRS.to_dict
rasterio.errors.CRSError: The WKT could not be parsed. OGR Error code 7

Interestingly, the GDAL Python API handles this in a different way:

from osgeo import gdal
ds = gdal.Open('/data/16APR27183709-P1BS-502376654080_01_P009.tif')
prj = ds.GetProjection()
print(prj)

which prints

GEOGCS["WGS 84",DATUM["WGS_1984",SPHEROID["WGS 84",6378137,298.257223563,AUTHORITY["EPSG","7030"]],AUTHORITY["EPSG","6326"]],PRIMEM["Greenwich",0],UNIT["degree",0.0174532925199433],AUTHORITY["EPSG","4326"]]
I guess GDAL assumes the GCP projection, in case of blank geotransform.

In any case, I think the src.crs is None behavior is good, since we can just look at src.gcps for the gcps.

I will add a workaround on our side, since
print(CRS().is_valid)
works, (that is, CRS().is_valid evaluates to False, without throwing exceptions.


Re: Rasterio 1.0.15

Sean Gillies
 

Correct, there is no need to specify the build tag. There is also no way to override the build tag other than to specify the entire URL of the wheel, but that's a good thing.

On Tue, Jan 29, 2019 at 9:18 AM <vincent.sarago@...> wrote:
Hi @Madhav, 
On linux it should work with `rasterio==1.0.15` 



--
Sean Gillies


Re: Rasterio 1.0.15

vincent.sarago@...
 

Hi @Madhav, 
On linux it should work with `rasterio==1.0.15`