Date   
Re: Implementing -scale functionality of gdal_translate

linden.kyle@...
 

Hi Sean, I'm having some trouble with the implementation of this and was wondering if you might be able to give me a hand. Here's the code I am working with: https://pastebin.com/kecx0D2b. What I'm trying to do is take a geotiff and convert it to a set of TMS-formatted tiles (including reprojecting to web mercator). In this example `tile_bounds` are the bounds of the tile in web mercator. I originally had this code sample working until I attempted to add in the code to do the pixel scaling. Here's the original working code: https://pastebin.com/dZVRVKi2

Here's what the output looks like with the broken code: https://imgur.com/xLtWrGZ
And here's the correct version (but no support for pixel scaling): https://imgur.com/aoNCyIp

Reading an image by overlapped blocks of chosen depth

mail2abhisek.maiti@...
 

In rasterio, reading image array in the unit of blocks is pretty convenient and useful. However, Overlapping blocks could be very useful for certain applications which requires kernel wise processing (such as simple smoothing filter).  So far there is no provision to make a block overlap with its neighboring blocks as per requirement of the application. It would be immensely handy if this feature gets implemented in rasterio. Related Stackoverflow Question.

Re: Implementing -scale functionality of gdal_translate

linden.kyle@...
 

Ok, so it appears the issue was that I was attempting to do a windowed read outside the bounds of the geotiff (for TMS edge tiles) and `src.read()` was only returning the data inside the bounds since I wasn't using the `boundless=True` option. However, the next problem I am running into is that I'd like to use `fill_value=numpy.nan` to fill in the edge data with NaN values. However, my data is UInt8. Is there any way to specify an option to cast the data read by the call to `read()` into float32 so that I can support this?

Re: Implementing -scale functionality of gdal_translate

Sean Gillies
 

Hi,

I apologize for being slow to reply, I've been occupied with fixing bugs in the last couple of releases.

In theory it is possible to add an option to the read method which will cast the data because GDAL already supports this well in versions 2 and up. I haven't pursued this because converting unsigned int to float is trivial, if not cheap, with numpy:

   data = dataset.read().astype('float64')

And if you pass masked=True to read, you'll get a masked array and then can apply a fill_value of numpy.nan under the invalid data mask. Is this useful for you or have you already been down this path?

GDAL's data type casting would not produce an extra copy of the data and so would be optimal for some applications.


On Tue, Feb 5, 2019 at 11:09 AM <linden.kyle@...> wrote:
Ok, so it appears the issue was that I was attempting to do a windowed read outside the bounds of the geotiff (for TMS edge tiles) and `src.read()` was only returning the data inside the bounds since I wasn't using the `boundless=True` option. However, the next problem I am running into is that I'd like to use `fill_value=numpy.nan` to fill in the edge data with NaN values. However, my data is UInt8. Is there any way to specify an option to cast the data read by the call to `read()` into float32 so that I can support this?



--
Sean Gillies

Re: Implementing -scale functionality of gdal_translate

linden.kyle@...
 

Sean, no problem. Thanks for writing back. I'll keep that in mind for the future. What I ended up doing instead was modifying the tile window transform (which is passed to the reproject method) in the case where either the left or top coordinate is outside the bounds of the geotiff. In that case I change the translation of the transform to the left and/or top bound of the geotiff respectively. Not sure if this is the "right" way to do it or not, but it seems to be working.

Re: Implementing -scale functionality of gdal_translate

linden.kyle@...
 

So I'm seeing some odd behavior now. I ended up trying the masked_array solution, but I'm seeing breaks in my data at the tile boundaries.

Here's what the data looks like if I use reprojection with just the source band (code: https://pastebin.com/dZVRVKi2): https://imgur.com/WpFKfij

Here's what the data looks like I read into a masked array and use an ndarray source (code: https://pastebin.com/z1xd8mXu): https://imgur.com/2QkQQZf
Ironically enough I get the exact same result with my solution that doesn't use a masked array, but instead modifies the transform: https://pastebin.com/7ph1VYWX

Re: Reading an image by overlapped blocks of chosen depth

Luke Pinner
 

I've put a couple of methods in my answer to your SO question, by expanding the block windows and also for arbitrary windows (unrelated to internal blocks/tiles).

Re: Reading an image by overlapped blocks of chosen depth

mail2abhisek.maiti@...
 

I was not aware of the boundless reading. Fantastic implementation. Thanks.

Rasterio 1.0.18

Sean Gillies
 

Hi all,

We've made three new releases this week to fix regressions introduced in version 1.0.14. Please upgrade to 1.0.18 to take advantage of the changes in 1.0.14, but without suffering from the bugs. See https://github.com/mapbox/rasterio/blob/master/CHANGES.txt for more details.

--
Sean Gillies

Re: Implementing -scale functionality of gdal_translate

Sean Gillies
 

Hi,

On Wed, Feb 6, 2019 at 2:32 PM <linden.kyle@...> wrote:
So I'm seeing some odd behavior now. I ended up trying the masked_array solution, but I'm seeing breaks in my data at the tile boundaries.

Here's what the data looks like if I use reprojection with just the source band (code: https://pastebin.com/dZVRVKi2): https://imgur.com/WpFKfij

Here's what the data looks like I read into a masked array and use an ndarray source (code: https://pastebin.com/z1xd8mXu): https://imgur.com/2QkQQZf
Ironically enough I get the exact same result with my solution that doesn't use a masked array, but instead modifies the transform: https://pastebin.com/7ph1VYWX

Your code looks okay. I recommend padding your read window by a few pixels on each side so that tile_window, tile_data, and tile_transform are slight dilated. I think this is very likely to eliminate what looks like an off-by-one error in the reprojection, an error that you don't see when you use the entire source dataset as the reprojection source. Does that make sense?

--
Sean Gillies

Re: Implementing -scale functionality of gdal_translate

linden.kyle@...
 

That worked perfectly. Thanks for all your help.

Rasterio 1.0.20

Sean Gillies
 

Hi all,

Rasterio 1.0.20 is on PyPI today. I hope you'll like the changes: https://github.com/mapbox/rasterio/blob/master/CHANGES.txt#L4-L24.

I discovered that 1.0.19 was defective while building linux wheels to distribute. No distributions of 1.0.19 were uploaded.

--
Sean Gillies

Output to grib2?

Shane Mill - NOAA Affiliate
 

Is there anyway to output to grib2 after performing map algebra? For example, If you are performing a calculation between two grib2 bands, which rasterio does quite well, are you able to transform the numpy array into a grib2 file? I have been able to do this with geotiff but not grib2.

Re: Output to grib2?

Sean Gillies
 

Hi Shane,

Rasterio's rio-convert command can convert from GeoTIFF to GRIB, so it should be possible

$ rio convert -f GRIB ~/code/rasterio/tests/data/RGB.byte.tif /tmp/rgb.grb2
$ rio info /tmp/rgb.grb2
{"bounds": [101985.0, 2611486.28, 339316.64, 2826915.0], "colorinterp": ["undefined", "undefined", "undefined"], "count": 3, "crs": "PROJCS[\"unnamed\",GEOGCS[\"Coordinate System imported from GRIB file\",DATUM[\"WGS_1984\",SPHEROID[\"WGS 84\",6378137,298.257223563]],PRIMEM[\"Greenwich\",0],UNIT[\"degree\",0.0174532925199433]],PROJECTION[\"Transverse_Mercator\"],PARAMETER[\"latitude_of_origin\",0],PARAMETER[\"central_meridian\",-2072.483648],PARAMETER[\"scale_factor\",0.9996],PARAMETER[\"false_easting\",500000],PARAMETER[\"false_northing\",0],UNIT[\"Metre\",1]]", "descriptions": ["0[-] RESERVED(0) (Reserved)", "0[-] RESERVED(0) (Reserved)", "0[-] RESERVED(0) (Reserved)"], "driver": "GRIB", "dtype": "float64", "height": 718, "indexes": [1, 2, 3], "lnglat": [84.75845370919801, 24.561589206961006], "mask_flags": [["nodata"], ["nodata"], ["nodata"]], "nodata": 0.0, "res": [300.04, 300.04], "shape": [718, 791], "tiled": false, "transform": [300.04, 0.0, 101985.0, 0.0, -300.04, 2826915.0, 0.0, 0.0, 1.0], "units": [null, null, null], "width": 791}

I'm not a regular GRIB user, but it looks to me that GDAL produces a GRIB2 file when you write using the GRIB format driver, and that the format driver reads either version without a need to specify. GRIB2 writing requires GDAL 2.3.0 or newer. The latest rasterio wheels on PyPI include GDAL 2.4.0, and should be sufficient.

On Fri, Mar 15, 2019 at 11:53 AM Shane Mill - NOAA Affiliate via Groups.Io <shane.mill=noaa.gov@groups.io> wrote:
Is there anyway to output to grib2 after performing map algebra? For example, If you are performing a calculation between two grib2 bands, which rasterio does quite well, are you able to transform the numpy array into a grib2 file? I have been able to do this with geotiff but not grib2.



--
Sean Gillies

Re: Output to grib2?

Shane Mill - NOAA Affiliate
 

On Fri, Mar 15, 2019 at 11:02 AM, Sean Gillies wrote:
rio convert -f GRIB ~/code/rasterio/tests/data/RGB.byte.tif /tmp/rgb.grb2


Sean, thanks for the quick response! You are correct, that does work. I think my issue may have been that I was using gdalinfo to check the resulting grib2 file, and my gdal version was older. When I check with rio info it seems to be correct.

Thanks again!

-Shane

Python 2/3 compatibility management in rasterio

adrienoyono@...
 

Hello everyone,

First of all, this is my occasion to say thanks for the rasterio project. It has been very helpful simplifying my life last year. I write a sofware that depends on rasterio, should ensure compatibility with Python 2 and Python 3, and should be deployed behind a web server (Apache with mod_wsgi). This last requirement triggers a problem caused by the way rasterio ensures Python 2/3 compatibility as I understand it after reading the "compat" module and the "setup.py".

Below is the stacktrace I have from Apache when trying to load my app:

Traceback (most recent call last):
File "/opt/eodag/venv/lib/python2.7/site-packages/eodag/rest/server.wsgi", line 3, in <module>
  from eodag.rest.server import app as application
File "/opt/eodag/venv/lib/python2.7/site-packages/eodag/__init__.py", line 18, in <module>
  from .api.core import EODataAccessGateway      # noqa
File "/opt/eodag/venv/lib/python2.7/site-packages/eodag/api/core.py", line 27, in <module>
  from eodag.api.search_result import SearchResult
File "/opt/eodag/venv/lib/python2.7/site-packages/eodag/api/search_result.py", line 26, in <module>
  from eodag.api.product import EOProduct
File "/opt/eodag/venv/lib/python2.7/site-packages/eodag/api/product/__init__.py", line 18, in <module>
  from ._product import EOProduct  # noqa
File "/opt/eodag/venv/lib/python2.7/site-packages/eodag/api/product/_product.py", line 25, in <module>
  import rasterio
File "/opt/eodag/venv/lib/python2.7/site-packages/rasterio/__init__.py", line 22, in <module>
  from rasterio._base import gdal_version
File "rasterio/_base.pyx", line 20, in init rasterio._base
File "/opt/eodag/venv/lib/python2.7/site-packages/rasterio/compat.py", line 26, in <module>
  import mock
ImportError: No module named mock
I guess "mock" is used in rasterio for testing purpose, I do use it for that myself. If that's the case, it is weird to have rasterio brake if we are in Python 2 without the "mock" module being installed, which is clearly the case here. To have mock installed at the same time as rasterio, one must do: "pip install rasterio[dev]". However, we always just do "pip install rasterio" (which fails to install "mock").

I think compatibility for testing purposes should be decoupled from compatibility for the core functionality. Is it possible to have it done in rasterio ? That would help avoid this kind of problem.

Thank you in advance for your time on this issue.

Re: Python 2/3 compatibility management in rasterio

Sean Gillies
 

Hi,

We've always intended to keeping testing dependencies separate and the recent inclusion of mock in runtime dependencies was a mistake. It's been corrected, but it will be a few more days until we have a 1.0.22 release. I'm sorry about the inconvenience!


On Tue, Mar 19, 2019 at 8:21 AM <adrienoyono@...> wrote:
Hello everyone,

First of all, this is my occasion to say thanks for the rasterio project. It has been very helpful simplifying my life last year. I write a sofware that depends on rasterio, should ensure compatibility with Python 2 and Python 3, and should be deployed behind a web server (Apache with mod_wsgi). This last requirement triggers a problem caused by the way rasterio ensures Python 2/3 compatibility as I understand it after reading the "compat" module and the "setup.py".

Below is the stacktrace I have from Apache when trying to load my app:

Traceback (most recent call last):
File "/opt/eodag/venv/lib/python2.7/site-packages/eodag/rest/server.wsgi", line 3, in <module>
  from eodag.rest.server import app as application
File "/opt/eodag/venv/lib/python2.7/site-packages/eodag/__init__.py", line 18, in <module>
  from .api.core import EODataAccessGateway      # noqa
File "/opt/eodag/venv/lib/python2.7/site-packages/eodag/api/core.py", line 27, in <module>
  from eodag.api.search_result import SearchResult
File "/opt/eodag/venv/lib/python2.7/site-packages/eodag/api/search_result.py", line 26, in <module>
  from eodag.api.product import EOProduct
File "/opt/eodag/venv/lib/python2.7/site-packages/eodag/api/product/__init__.py", line 18, in <module>
  from ._product import EOProduct  # noqa
File "/opt/eodag/venv/lib/python2.7/site-packages/eodag/api/product/_product.py", line 25, in <module>
  import rasterio
File "/opt/eodag/venv/lib/python2.7/site-packages/rasterio/__init__.py", line 22, in <module>
  from rasterio._base import gdal_version
File "rasterio/_base.pyx", line 20, in init rasterio._base
File "/opt/eodag/venv/lib/python2.7/site-packages/rasterio/compat.py", line 26, in <module>
  import mock
ImportError: No module named mock
I guess "mock" is used in rasterio for testing purpose, I do use it for that myself. If that's the case, it is weird to have rasterio brake if we are in Python 2 without the "mock" module being installed, which is clearly the case here. To have mock installed at the same time as rasterio, one must do: "pip install rasterio[dev]". However, we always just do "pip install rasterio" (which fails to install "mock").

I think compatibility for testing purposes should be decoupled from compatibility for the core functionality. Is it possible to have it done in rasterio ? That would help avoid this kind of problem.

Thank you in advance for your time on this issue.



--
Sean Gillies

Re: Output to grib2?

Shane Mill - NOAA Affiliate
 

Hey Sean,

I know that you were saying that you don't use grib2 often, but I wanted to follow up with the previous question.

I am able to output the numpy array to GRIB. Say that you have one band, you can read the tags of that one band with src.tags(1). These tags will be the defaults that come from:
https://gdal.org/frmt_grib.html (if you scroll down to the section "GRIB2 Write Support"). The problem with this is that for the grib2 to be readable, you need to be able to update these tags.

I'm wondering how to change the tags either when originally creating the GRIB or after the fact when using "update_tags" explained here: https://github.com/mapbox/rasterio/blob/master/docs/topics/tags.rst
It appears that update_tags works fine for GTiff but not for GRIB. 

Let me know if you have any thoughts on this. Thanks as always!

-Shane

On Fri, Mar 15, 2019 at 2:16 PM Shane Mill - NOAA Affiliate via Groups.Io <shane.mill=noaa.gov@groups.io> wrote:
On Fri, Mar 15, 2019 at 11:02 AM, Sean Gillies wrote:
rio convert -f GRIB ~/code/rasterio/tests/data/RGB.byte.tif /tmp/rgb.grb2


Sean, thanks for the quick response! You are correct, that does work. I think my issue may have been that I was using gdalinfo to check the resulting grib2 file, and my gdal version was older. When I check with rio info it seems to be correct.

Thanks again!

-Shane



--
Shane Mill

Meteorological Application Developer, AceInfo Solutions

Meteorological Development Laboratory (NWS)

Phone: 301-427-9452

Re: Python 2/3 compatibility management in rasterio

adrienoyono@...
 

Awesome ! Many thanks !! I will stay tuned to the release of 1.0.22 then

Rasterio 1.0.22

Sean Gillies
 

Hi all,

Rasterio wheels and sdist for 1.0.22 are on PyPI today. This release fixes two bugs reported in 1.0.21.

Changes
=======

1.0.22 (2019-03-20)
-------------------

- Add JPEG2000 to enums.Compression so that the compression of JP2 files can be
  reported (#1654).
- Remove mock import from compat and move to test code (#1651).

--
Sean Gillies