Date   

read masked and overviews when no-data value is defined

Guy Doulberg
 

Hi all,

We are trying to read masked array using `DatasetReaderBase.read`, reading the documentation it looks that if we pass masked=True we will get what we wanted. 

I was testing the read on raster with pre-built overviews and I used the overviews in read. From functionality point of view, I always got the right result, but I was tesing the data block that are being used, and I noticed this behaviors:


1. When using band-mask, there was a read only from the overview to construct the mask -- as excpected
2. When using no-data value there were reads the data of the original resolution and not from the nearest overview? when mask=False it reads data only from the overview,  why is that?

A workaround for that is to read with no mask and to construct the mask myself using the no-data value.

Thanks.


Re: Mask to alpha band

Sean Gillies
 

Hi Denis,

There's no way to do it in Rasterio without creating a new dataset and copying the band and mask arrays from the original.

On Mon, Jul 23, 2018 at 2:08 PM, Denis Rykov <rykovd@...> wrote:
Hi, folks!

Is it possible to convert internal mask to alpha band? 




--
Sean Gillies


Rasterio 1.0.1

Sean Gillies
 

Hi all,

Rasterio 1.0.1 is on PyPI now. It fixes 4 bugs found since 1.0.0:

- Bounding envelopes are densified at higher precision in transform_bounds to
  fix #1411.
- Add LERC compression to enums.Compression (#1412).
- The reproject() function now passes dst_alpha properly to _reproject(), which
  unlocks materialization of warp destination alpha band masks (#1417).
- The --dimensions and --src-bounds options of rio-warp can be used together
  as expected (#1418).

--
Sean Gillies


Re: Reproject with mask preserving

Sean Gillies
 

Hi Denis,

As you discovered in your post to gdal-dev, GDAL's warper doesn't output internal (or sidecar) mask bands. You must materialize these into an alpha band and then copy the data to produce a new internal mask band.

Here's a test that demonstrates how to create an internal mask band:


While putting together an example of calling reproject() with alpha band creation, I found a bug


When Rasterio 1.0.1 is published to PyPI, you will be able to call reproject() and materialize source's mask band into the destination alpha band specified by the dst_alpha keyword argument as shown in this test:



On Sat, Jul 21, 2018 at 4:45 PM, Denis Rykov <rykovd@...> wrote:
Hi there!

I have GeoTIFF files with internal mask and I need to reproject them and not lose mask.

I've found that direct generating warped mask band is not supported by GDAL: https://github.com/OSGeo/gdal/issues/689

rasterio.warp.reproject() also doesn't preserve mask.

How I can use rasterio for achieving what I need? I'm looking for something similar described in this issue but with rasterio.




--
Sean Gillies


Mask to alpha band

Denis Rykov
 

Hi, folks!

Is it possible to convert internal mask to alpha band? 


Re: Copying Rasterio crs from input to output dataset

Luke
 

My GDAL_DATA env var wasn't set.

Explanation:
I was running the code within the PyCharm IDE that supposedly supports conda envs but doesn't actually *activate* them properly.  There are init scripts in CONDA_PREFIX/etc/conda/activate.d  that get run when you activate/source activate a conda env that do some additional environment setup and there's a gdal one that should've been run but wasn't.

Apologies for the noise. 

Luke


Reproject with mask preserving

Denis Rykov
 

Hi there!

I have GeoTIFF files with internal mask and I need to reproject them and not lose mask.

I've found that direct generating warped mask band is not supported by GDAL: https://github.com/OSGeo/gdal/issues/689

rasterio.warp.reproject() also doesn't preserve mask.

How I can use rasterio for achieving what I need? I'm looking for something similar described in this issue but with rasterio.


Re: Copying Rasterio crs from input to output dataset

Sean Gillies
 

Hi Luke,

Unfortunately, I cannot reproduce your results on my computer.

$ python
Python 3.6.6 (v3.6.6:4cf1f54eb7, Jun 26 2018, 19:50:54)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import rasterio
>>> with rasterio.open('tests/data/RGB.byte.tif') as src:
...     with rasterio.open('/tmp/foo.tif', 'w', **src.meta) as dst:
...         dst.write(src.read())
...
>>> with rasterio.open('/tmp/foo.tif') as src:
...     print(src.meta)
...     print(src.profile)
...
{'driver': 'GTiff', 'dtype': 'uint8', 'nodata': 0.0, 'width': 791, 'height': 718, 'count': 3, 'crs': CRS({'init': 'epsg:32618'}), 'transform': Affine(300.0379266750948, 0.0, 101985.0,
       0.0, -300.041782729805, 2826915.0)}
{'driver': 'GTiff', 'dtype': 'uint8', 'nodata': 0.0, 'width': 791, 'height': 718, 'count': 3, 'crs': CRS({'init': 'epsg:32618'}), 'transform': Affine(300.0379266750948, 0.0, 101985.0,
       0.0, -300.041782729805, 2826915.0), 'tiled': False, 'interleave': 'pixel'}

I wonder if you may have a problem with unresolved Proj data. To debug this, turn on Python's logging facility and set the level to DEBUG at the top of your code like so:

import logging
logging.basicConfig(level=logging.DEBUG)

On Thu, Jul 19, 2018 at 11:24 PM, Luke Pinner <lukepinnerau@...> wrote:
Hi all.  Congrats on the 1.0.0 release!  

Using rasterio 1.0.0 (installed from conda-forge in a fresh conda env). I'm having trouble copying the CRS from on dataset to another.  The output dataset CRS is None.

Simplified example demonstrating:

import rasterio as rio

inpath = r"test.tif"
outpath = r"testout.tif"

with rio.open(inpath) as inds:
    print(repr(inds.crs), inds.meta['crs'])
    with rio.open(outpath, 'w', **inds.meta) as outds:
        outds.write(inds.read())

with rio.open(outpath) as outds:
    print(repr(outds.crs), outds.meta['crs'])

The output of the above is:

CRS({'init': 'epsg:4326'}) +init=epsg:4326
None None

Dataset info:

rio info test.tif
{"blockxsize": 128, "blockysize": 128, "bounds": [143.22468138068513, -34.72000041269073, 146.20968138068514, -31.952500412690732], "colorinterp": ["grey", "undefined", "undefined", "undefined", "undefined", "undefined", "undefined", "undefined", "undefined", "undefined", "undefined"], "count": 11, "crs": "EPSG:4326", "descriptions": [null, null, null, null, null, null, null, null, null, null, null], "driver": "GTiff", "dtype": "float32", "height": 1107, "indexes": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11], "interleave": "pixel", "lnglat": [144.71718138068513, -33.33625041269073], "mask_flags": [["nodata"], ["nodata"], ["nodata"], ["nodata"], ["nodata"], ["nodata"], ["nodata"], ["nodata"], ["nodata"], ["nodata"], ["nodata"]], "nodata": -9999.0, "res": [0.0025000000000000113, 0.0024999999999999983], "shape": [1107, 1194], "tiled": true, "transform": [0.0025000000000000113, 0.0, 143.22468138068513, 0.0, -0.0024999999999999983, -31.952500412690732, 0.0, 0.0, 1.0], "units": [null, null, null, null, null, null, null, null, null, null, null], "width": 1194}

Is this user error (do I need to pass a crs string in?) or perhaps a bug?

Regards
Luke






--
Sean Gillies


Re: LERC Compression Error

Sean Gillies
 

Hi Caleb,

Sure, a PR would be welcome. This compression algorithm isn't available in a GDAL release yet, but it doesn't do any harm to have it enumerated in Rasterio. Behavior for Rasterio 1.0.0 and GDAL 2.2.4 is this:

$ rio convert tests/data/RGB.byte.tif /tmp/lerc.tif --co compress=LERC
WARNING:rasterio._gdal:CPLE_NotSupported in 'LERC' is an unexpected value for COMPRESS creation option of type string-select.
WARNING:rasterio._gdal:CPLE_IllegalArg in COMPRESS=LERC value not recognised, ignoring.

On Wed, Jul 18, 2018 at 7:45 PM, <calebrob6@...> wrote:
Hello all,

I have a raster that uses LERC compression. Opening/reading from the raster works fine, however I get the following error when trying to access the `.profile` property:
ValueError: 'LERC' is not a valid Compression

I've found that simply adding `lerc = 'LERC'` to the Compression enum in rasterio/enums.py solves this problem. Is this the correct fix and if so, would you like me to open a pull request for it?

Thanks,
Caleb




--
Sean Gillies


Copying Rasterio crs from input to output dataset

Luke
 

Hi all.  Congrats on the 1.0.0 release!  

Using rasterio 1.0.0 (installed from conda-forge in a fresh conda env). I'm having trouble copying the CRS from on dataset to another.  The output dataset CRS is None.

Simplified example demonstrating:

import rasterio as rio

inpath = r"test.tif"
outpath = r"testout.tif"

with rio.open(inpath) as inds:
    print(repr(inds.crs), inds.meta['crs'])
    with rio.open(outpath, 'w', **inds.meta) as outds:
        outds.write(inds.read())

with rio.open(outpath) as outds:
    print(repr(outds.crs), outds.meta['crs'])

The output of the above is:

CRS({'init': 'epsg:4326'}) +init=epsg:4326
None None

Dataset info:

rio info test.tif
{"blockxsize": 128, "blockysize": 128, "bounds": [143.22468138068513, -34.72000041269073, 146.20968138068514, -31.952500412690732], "colorinterp": ["grey", "undefined", "undefined", "undefined", "undefined", "undefined", "undefined", "undefined", "undefined", "undefined", "undefined"], "count": 11, "crs": "EPSG:4326", "descriptions": [null, null, null, null, null, null, null, null, null, null, null], "driver": "GTiff", "dtype": "float32", "height": 1107, "indexes": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11], "interleave": "pixel", "lnglat": [144.71718138068513, -33.33625041269073], "mask_flags": [["nodata"], ["nodata"], ["nodata"], ["nodata"], ["nodata"], ["nodata"], ["nodata"], ["nodata"], ["nodata"], ["nodata"], ["nodata"]], "nodata": -9999.0, "res": [0.0025000000000000113, 0.0024999999999999983], "shape": [1107, 1194], "tiled": true, "transform": [0.0025000000000000113, 0.0, 143.22468138068513, 0.0, -0.0024999999999999983, -31.952500412690732, 0.0, 0.0, 1.0], "units": [null, null, null, null, null, null, null, null, null, null, null], "width": 1194}

Is this user error (do I need to pass a crs string in?) or perhaps a bug?

Regards
Luke




LERC Compression Error

calebrob6@...
 

Hello all,

I have a raster that uses LERC compression. Opening/reading from the raster works fine, however I get the following error when trying to access the `.profile` property:
ValueError: 'LERC' is not a valid Compression

I've found that simply adding `lerc = 'LERC'` to the Compression enum in rasterio/enums.py solves this problem. Is this the correct fix and if so, would you like me to open a pull request for it?

Thanks,
Caleb


Re: Calculate transformation using output dimensions

Denis Rykov
 

Thanks for reply.

I could not find what I need in rasterio and have created PR: https://github.com/mapbox/rasterio/pull/1409


Re: Calculate transformation using output dimensions

Alan Snow
 

This may be what you are looking for as well:
rasterio.transform.from_bounds


Re: Calculate transformation using output dimensions

Alan Snow
 

Looks like you are most of the way there.

If you have a dataset open with `rasterio.open`, then you should be able to get the information you need for the input:

with rasterio.open('/path/do/dataset.tif') as rds:
    left, bottom, right, top = rds.bounds
    src_crs = rds.crs
    width = rds.width
    height = rds.height


Calculate transformation using output dimensions

Denis Rykov
 

In gdalwarp we have at least two options for managing of output size of a raster:

  • -tr xres yres: set output file resolution (in target georeferenced units)
  • -ts width height: set output file size in pixels and lines
In rasterio we have the following function:
  • rasterio.warp.calculate_default_transform(src_crs, dst_crs, width, height, left=None, bottom=None, right=None, top=None, gcps=None, resolution=None)
If I understood correctly resolution argument has the same meaning as tr in gdalwarp.
But how I can calculate transformation (destination affine first of all) if I know output dimensions (width and height)?

In other words I'm looking for how I can achieve the same result with rasterio as with gdalwarp ts option.

I hope what I have requested is possible.


Rasterio 1.0.0

Sean Gillies
 

Hi, all.

We, the authors, are pleased to announce the release of Rasterio 1.0.0.

What is the significance of 1.0.0? Stability. After more than a year of changes, there is at last a stable base for applications. You can pin `rasterio ~= 1.0`  in your project’s requirements and enjoy nothing but bug fixes for as long as the project supplies them.

Highlights

Many new features have been added since the last stable release (0.36). Especially notable are the following.

- All new documentation at https://rasterio.readthedocs.io/en/stable/.
- Binary wheels with GDAL 2.x included for the macOS and manylinux1 platforms are available on pypi.org.
- A new Window class with floating point origin and offsets has been added to help with windowed dataset operations.
- BytesIO-like MemoryFile and ZipMemoryFile classes that support access to in-memory datasets.
- A WarpedVRT class exposes GDAL’s warp-on-demand VRT features.
- Support for georeferencing by ground control points has been added.
- The rasterio.shutil module provides many of the same features as Python’s shutil, but also knows about sidecar files (masks, overviews, metadata).

Upgrading and compatibility

Rasterio is compatible with GDAL versions 1.11-2.3 and Python versions 2.7, and 3.4-3.7.

We have deprecated a number of features since 0.36. Features have been removed, after some warning, at 1.0a1, 1.0a10, 1.0b1, and 1.0rc2. We recommend migrating stepwise through those tags to get from 0.36 to 1.0.0 if you're feeling extra cautious.

Support

The primary channel for installation and usage support is the Rasterio user discussion group at  https://rasterio.groups.io/g/main. Please see https://github.com/mapbox/rasterio/blob/master/CONTRIBUTING.rst for guidance on reporting bugs or proposing new features.

Acknowledgements

Rasterio 1.0.0 is the work of 68 authors. The individuals are listed in https://github.com/mapbox/rasterio/blob/master/AUTHORS.txt.

This number does not include people who have created and commented on issues in the project’s issue tracker, but their contributions are also very important. We’ve received some of the best bug reports ever written.

Rasterio benefits from a userbase in many different areas of work and study. Advocates across companies, classrooms, and projects have helped make Rasterio what it is. We particularly thank people who help distribute it, and people who help teach others to use it. Among those are: Christoph Gohlke, the conda-forge and Debian GIS teams, Howard Butler, Sara Safavi, Dana Bauer, Leah Wasser, Chris Holdgraf, and everyone else who has taught Rasterio in a workshop or course, or presented Rasterio to a conference or meetup audience.

Rasterio has benefited from helpful folks on the GIS StackExchange. Luke Pinner, Loïc Dutrieux, Martin Laloux, and Kersten Clauss are foremost among them.

Everything we know about building and distributing binary wheels comes from the SciPy community. Thank you, Matthew Brett, Nathaniel Smith, et al., for writing the delocate and auditwheel tools and supporting them.

Even Rouault, GDAL’s maintainer, has been our patient guru and guide during many difficult passages.

Early adoption by engineering teams in the satellite imagery business has also been key to Rasterio’s success. The project is indebted to developers who tried it and managers who supported them.

Thank you all!

--
Sean Gillies


Plugin for common color maps?

me@...
 

Hey there,

Thanks for a great library - I've just been getting into it via the rio-tiler library where I'm building an app for showing imagery in combination with stream gauge data (see below).

What I'd like to do is apply a suitable colormap to my tile to match the Water Index that I'm displaying which I'm sure this is doable with rasterio (he says without having tried it).

Would it make sense to perhaps package this sort of thing as a rio plugin, eg a plugin containing standard color ramps for commonly used indices? 

Have I missed anything existing anywhere and would this make sense and be useful?

Cheers
Rowan

 


New Member

Armstrong Manuvakola Ezequias Ngolo
 

Hello everyone,

 

I’m a new member of rasterio group willing to know more about rasterio and satellite image processing. I’m also new in programming, so, don’t be surprised if I make weird questions!!!

 

P.S: non native English speaker!!!!

 

Pleasure to be in this group and willing to hear from you.

 

Armstrong Ngolo

 

Sent from Mail for Windows 10

 


Rasterio 1.0rc5

Sean Gillies
 

Hi all,

The Rasterio 1.0rc5 source distribution and wheels for macos 10.9 (or greater) and manylinux1 platforms are on PyPI now: https://pypi.org/project/rasterio/1.0rc5/.

Please remember that

  pip install -U rasterio

will install Rasterio 0.36, the last "stable" release, and that you need to execute either

  pip install rasterio==1.0rc5

or

  pip install -U --pre rasterio

to get the latest pre-release. Packages for the Anaconda platform (Linux, OS X, and Windows) should be available in the conda-forge channel (under the "dev" label) very soon.


The final 1.0.0 release is still scheduled for Thursday, July 12. I accidentally wrote Tuesday in the 1.0rc4 announcement.

--
Sean Gillies


Rasterio 1.0rc4

Sean Gillies
 

Hi all,

The Rasterio 1.0rc4 source distribution and wheels for macos 10.9 (or greater) and manylinux1 platforms are on PyPI now: https://pypi.org/project/rasterio/1.0rc4/.

Please remember that

  pip install -U rasterio

will install Rasterio 0.36, the last "stable" release, and that you need to execute either

  pip install rasterio==1.0rc4

or

  pip install -U --pre rasterio

to get the latest pre-release. Packages for the Anaconda platform (Linux, OS X, and Windows) should be available in the conda-forge channel (under the "dev" label) very soon.


The final 1.0.0 release is now tentatively scheduled for Tuesday, July 12.

--
Sean Gillies

881 - 900 of 905