Date   

Rasterio's general discussion group is open for business

Sean Gillies
 

Greetings,

Questions about installation, distribution, and usage of Rasterio are welcome here. Issues opened in Rasterio's GitHub repo that fall into one these categories may be perfunctorily closed.

Note well: the Rasterio project's code of conduct, https://github.com/mapbox/rasterio/blob/master/CODE_OF_CONDUCT.txt, applies to this group.

Yours,

--
Sean Gillies


Rasterio 1.0rc3

Sean Gillies
 

Hi all,

I almost forgot to announce here before the holiday!

The Rasterio 1.0rc3 source distribution and wheels for macos 10.9 (or greater) and manylinux1 platforms are on PyPI now: https://pypi.org/project/rasterio/1.0rc3/.

Please remember that

  pip install -U rasterio

will install Rasterio 0.36, the last "stable" release, and that you need to execute either

  pip install rasterio==1.0rc3

or

  pip install -U --pre rasterio

to get the latest pre-release. Packages for the Anaconda platform (Linux, OS X, and Windows) should be available in the conda-forge channel (under the "dev" label) very soon.


The final 1.0.0 release is tentatively scheduled for Tuesday, July 10.

--
Sean Gillies


Does rasterio load data into memory while reprojecting?

Denis Rykov
 

Hi there!

I have a general question about reprojection. Consider the following example:

with rasterio.open(fname_src) as src:
    with rasterio.open(fname_dst, 'w', **kwargs) as dst:
        for i in range(1, src.count + 1):
            rasterio.warp.reproject(
                source=rasterio.band(src, i),
                destination=rasterio.band(dst, i),
                src_transform=src.transform,
                src_crs=src.crs,
                dst_transform=dest_affine,
                dst_crs=dst_crs,
                resampling=resampling)

As I can see from source code rasterio reads each band as an N-d array.
Does it mean that rasterio loads each band into memory?

My main question: can I use this code for warping rasters of arbitrary size (or I'm limited by memory size)?

Thanks.


Re: Does rasterio load data into memory while reprojecting?

Sean Gillies
 

Hi Denis,

On Wed, Jul 4, 2018 at 4:09 AM, Denis Rykov <rykovd@...> wrote:
Hi there!

I have a general question about reprojection. Consider the following example:

with rasterio.open(fname_src) as src:
    with rasterio.open(fname_dst, 'w', **kwargs) as dst:
        for i in range(1, src.count + 1):
            rasterio.warp.reproject(
                source=rasterio.band(src, i),
                destination=rasterio.band(dst, i),
                src_transform=src.transform,
                src_crs=src.crs,
                dst_transform=dest_affine,
                dst_crs=dst_crs,
                resampling=resampling)

As I can see from source code rasterio reads each band as an N-d array.
Does it mean that rasterio loads each band into memory?

My main question: can I use this code for warping rasters of arbitrary size (or I'm limited by memory size)?

Thanks.

Yes, you can warp arbitrarily large rasters when you use rasterio.band(). The warper chunks the work so that its working arrays are no more than 64 MB in size. You can increase this limit with reproject's warp_mem_limit keyword argument.

The source dataset's bands are incrementally loaded into memory as the warper works, but only up to the limit of GDAL's block cache (Rasterio uses the GDAL C library). If you profile your program and see the memory allocation increase, this is due to GDAL's caching. You can see also the notes at http://trac.osgeo.org/gdal/wiki/UserDocs/GdalWarp#WarpandCacheMemory:TechnicalDetails, which apply to Rasterio.

Please note that you can warp multiple imagery bands in one reproject() call by passing sequences of band indexes.

  reproject(source=rasterio.band(src, src.indexes), destination=rasterio.band(src, dst.indexes), ...)

This is faster than looping over the bands.

--
Sean Gillies


Re: Does rasterio load data into memory while reprojecting?

Denis Rykov
 

Thank you very much, Sean, for such detailed reply!


Rasterio 1.0rc4

Sean Gillies
 

Hi all,

The Rasterio 1.0rc4 source distribution and wheels for macos 10.9 (or greater) and manylinux1 platforms are on PyPI now: https://pypi.org/project/rasterio/1.0rc4/.

Please remember that

  pip install -U rasterio

will install Rasterio 0.36, the last "stable" release, and that you need to execute either

  pip install rasterio==1.0rc4

or

  pip install -U --pre rasterio

to get the latest pre-release. Packages for the Anaconda platform (Linux, OS X, and Windows) should be available in the conda-forge channel (under the "dev" label) very soon.


The final 1.0.0 release is now tentatively scheduled for Tuesday, July 12.

--
Sean Gillies


Rasterio 1.0rc5

Sean Gillies
 

Hi all,

The Rasterio 1.0rc5 source distribution and wheels for macos 10.9 (or greater) and manylinux1 platforms are on PyPI now: https://pypi.org/project/rasterio/1.0rc5/.

Please remember that

  pip install -U rasterio

will install Rasterio 0.36, the last "stable" release, and that you need to execute either

  pip install rasterio==1.0rc5

or

  pip install -U --pre rasterio

to get the latest pre-release. Packages for the Anaconda platform (Linux, OS X, and Windows) should be available in the conda-forge channel (under the "dev" label) very soon.


The final 1.0.0 release is still scheduled for Thursday, July 12. I accidentally wrote Tuesday in the 1.0rc4 announcement.

--
Sean Gillies


New Member

Armstrong Manuvakola Ezequias Ngolo
 

Hello everyone,

 

I’m a new member of rasterio group willing to know more about rasterio and satellite image processing. I’m also new in programming, so, don’t be surprised if I make weird questions!!!

 

P.S: non native English speaker!!!!

 

Pleasure to be in this group and willing to hear from you.

 

Armstrong Ngolo

 

Sent from Mail for Windows 10

 


Plugin for common color maps?

me@...
 

Hey there,

Thanks for a great library - I've just been getting into it via the rio-tiler library where I'm building an app for showing imagery in combination with stream gauge data (see below).

What I'd like to do is apply a suitable colormap to my tile to match the Water Index that I'm displaying which I'm sure this is doable with rasterio (he says without having tried it).

Would it make sense to perhaps package this sort of thing as a rio plugin, eg a plugin containing standard color ramps for commonly used indices? 

Have I missed anything existing anywhere and would this make sense and be useful?

Cheers
Rowan

 


Rasterio 1.0.0

Sean Gillies
 

Hi, all.

We, the authors, are pleased to announce the release of Rasterio 1.0.0.

What is the significance of 1.0.0? Stability. After more than a year of changes, there is at last a stable base for applications. You can pin `rasterio ~= 1.0`  in your project’s requirements and enjoy nothing but bug fixes for as long as the project supplies them.

Highlights

Many new features have been added since the last stable release (0.36). Especially notable are the following.

- All new documentation at https://rasterio.readthedocs.io/en/stable/.
- Binary wheels with GDAL 2.x included for the macOS and manylinux1 platforms are available on pypi.org.
- A new Window class with floating point origin and offsets has been added to help with windowed dataset operations.
- BytesIO-like MemoryFile and ZipMemoryFile classes that support access to in-memory datasets.
- A WarpedVRT class exposes GDAL’s warp-on-demand VRT features.
- Support for georeferencing by ground control points has been added.
- The rasterio.shutil module provides many of the same features as Python’s shutil, but also knows about sidecar files (masks, overviews, metadata).

Upgrading and compatibility

Rasterio is compatible with GDAL versions 1.11-2.3 and Python versions 2.7, and 3.4-3.7.

We have deprecated a number of features since 0.36. Features have been removed, after some warning, at 1.0a1, 1.0a10, 1.0b1, and 1.0rc2. We recommend migrating stepwise through those tags to get from 0.36 to 1.0.0 if you're feeling extra cautious.

Support

The primary channel for installation and usage support is the Rasterio user discussion group at  https://rasterio.groups.io/g/main. Please see https://github.com/mapbox/rasterio/blob/master/CONTRIBUTING.rst for guidance on reporting bugs or proposing new features.

Acknowledgements

Rasterio 1.0.0 is the work of 68 authors. The individuals are listed in https://github.com/mapbox/rasterio/blob/master/AUTHORS.txt.

This number does not include people who have created and commented on issues in the project’s issue tracker, but their contributions are also very important. We’ve received some of the best bug reports ever written.

Rasterio benefits from a userbase in many different areas of work and study. Advocates across companies, classrooms, and projects have helped make Rasterio what it is. We particularly thank people who help distribute it, and people who help teach others to use it. Among those are: Christoph Gohlke, the conda-forge and Debian GIS teams, Howard Butler, Sara Safavi, Dana Bauer, Leah Wasser, Chris Holdgraf, and everyone else who has taught Rasterio in a workshop or course, or presented Rasterio to a conference or meetup audience.

Rasterio has benefited from helpful folks on the GIS StackExchange. Luke Pinner, Loïc Dutrieux, Martin Laloux, and Kersten Clauss are foremost among them.

Everything we know about building and distributing binary wheels comes from the SciPy community. Thank you, Matthew Brett, Nathaniel Smith, et al., for writing the delocate and auditwheel tools and supporting them.

Even Rouault, GDAL’s maintainer, has been our patient guru and guide during many difficult passages.

Early adoption by engineering teams in the satellite imagery business has also been key to Rasterio’s success. The project is indebted to developers who tried it and managers who supported them.

Thank you all!

--
Sean Gillies


Calculate transformation using output dimensions

Denis Rykov
 

In gdalwarp we have at least two options for managing of output size of a raster:

  • -tr xres yres: set output file resolution (in target georeferenced units)
  • -ts width height: set output file size in pixels and lines
In rasterio we have the following function:
  • rasterio.warp.calculate_default_transform(src_crs, dst_crs, width, height, left=None, bottom=None, right=None, top=None, gcps=None, resolution=None)
If I understood correctly resolution argument has the same meaning as tr in gdalwarp.
But how I can calculate transformation (destination affine first of all) if I know output dimensions (width and height)?

In other words I'm looking for how I can achieve the same result with rasterio as with gdalwarp ts option.

I hope what I have requested is possible.


Re: Calculate transformation using output dimensions

Alan Snow
 

Looks like you are most of the way there.

If you have a dataset open with `rasterio.open`, then you should be able to get the information you need for the input:

with rasterio.open('/path/do/dataset.tif') as rds:
    left, bottom, right, top = rds.bounds
    src_crs = rds.crs
    width = rds.width
    height = rds.height


Re: Calculate transformation using output dimensions

Alan Snow
 

This may be what you are looking for as well:
rasterio.transform.from_bounds


Re: Calculate transformation using output dimensions

Denis Rykov
 

Thanks for reply.

I could not find what I need in rasterio and have created PR: https://github.com/mapbox/rasterio/pull/1409


LERC Compression Error

calebrob6@...
 

Hello all,

I have a raster that uses LERC compression. Opening/reading from the raster works fine, however I get the following error when trying to access the `.profile` property:
ValueError: 'LERC' is not a valid Compression

I've found that simply adding `lerc = 'LERC'` to the Compression enum in rasterio/enums.py solves this problem. Is this the correct fix and if so, would you like me to open a pull request for it?

Thanks,
Caleb


Copying Rasterio crs from input to output dataset

Luke
 

Hi all.  Congrats on the 1.0.0 release!  

Using rasterio 1.0.0 (installed from conda-forge in a fresh conda env). I'm having trouble copying the CRS from on dataset to another.  The output dataset CRS is None.

Simplified example demonstrating:

import rasterio as rio

inpath = r"test.tif"
outpath = r"testout.tif"

with rio.open(inpath) as inds:
    print(repr(inds.crs), inds.meta['crs'])
    with rio.open(outpath, 'w', **inds.meta) as outds:
        outds.write(inds.read())

with rio.open(outpath) as outds:
    print(repr(outds.crs), outds.meta['crs'])

The output of the above is:

CRS({'init': 'epsg:4326'}) +init=epsg:4326
None None

Dataset info:

rio info test.tif
{"blockxsize": 128, "blockysize": 128, "bounds": [143.22468138068513, -34.72000041269073, 146.20968138068514, -31.952500412690732], "colorinterp": ["grey", "undefined", "undefined", "undefined", "undefined", "undefined", "undefined", "undefined", "undefined", "undefined", "undefined"], "count": 11, "crs": "EPSG:4326", "descriptions": [null, null, null, null, null, null, null, null, null, null, null], "driver": "GTiff", "dtype": "float32", "height": 1107, "indexes": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11], "interleave": "pixel", "lnglat": [144.71718138068513, -33.33625041269073], "mask_flags": [["nodata"], ["nodata"], ["nodata"], ["nodata"], ["nodata"], ["nodata"], ["nodata"], ["nodata"], ["nodata"], ["nodata"], ["nodata"]], "nodata": -9999.0, "res": [0.0025000000000000113, 0.0024999999999999983], "shape": [1107, 1194], "tiled": true, "transform": [0.0025000000000000113, 0.0, 143.22468138068513, 0.0, -0.0024999999999999983, -31.952500412690732, 0.0, 0.0, 1.0], "units": [null, null, null, null, null, null, null, null, null, null, null], "width": 1194}

Is this user error (do I need to pass a crs string in?) or perhaps a bug?

Regards
Luke




Re: LERC Compression Error

Sean Gillies
 

Hi Caleb,

Sure, a PR would be welcome. This compression algorithm isn't available in a GDAL release yet, but it doesn't do any harm to have it enumerated in Rasterio. Behavior for Rasterio 1.0.0 and GDAL 2.2.4 is this:

$ rio convert tests/data/RGB.byte.tif /tmp/lerc.tif --co compress=LERC
WARNING:rasterio._gdal:CPLE_NotSupported in 'LERC' is an unexpected value for COMPRESS creation option of type string-select.
WARNING:rasterio._gdal:CPLE_IllegalArg in COMPRESS=LERC value not recognised, ignoring.

On Wed, Jul 18, 2018 at 7:45 PM, <calebrob6@...> wrote:
Hello all,

I have a raster that uses LERC compression. Opening/reading from the raster works fine, however I get the following error when trying to access the `.profile` property:
ValueError: 'LERC' is not a valid Compression

I've found that simply adding `lerc = 'LERC'` to the Compression enum in rasterio/enums.py solves this problem. Is this the correct fix and if so, would you like me to open a pull request for it?

Thanks,
Caleb




--
Sean Gillies


Re: Copying Rasterio crs from input to output dataset

Sean Gillies
 

Hi Luke,

Unfortunately, I cannot reproduce your results on my computer.

$ python
Python 3.6.6 (v3.6.6:4cf1f54eb7, Jun 26 2018, 19:50:54)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import rasterio
>>> with rasterio.open('tests/data/RGB.byte.tif') as src:
...     with rasterio.open('/tmp/foo.tif', 'w', **src.meta) as dst:
...         dst.write(src.read())
...
>>> with rasterio.open('/tmp/foo.tif') as src:
...     print(src.meta)
...     print(src.profile)
...
{'driver': 'GTiff', 'dtype': 'uint8', 'nodata': 0.0, 'width': 791, 'height': 718, 'count': 3, 'crs': CRS({'init': 'epsg:32618'}), 'transform': Affine(300.0379266750948, 0.0, 101985.0,
       0.0, -300.041782729805, 2826915.0)}
{'driver': 'GTiff', 'dtype': 'uint8', 'nodata': 0.0, 'width': 791, 'height': 718, 'count': 3, 'crs': CRS({'init': 'epsg:32618'}), 'transform': Affine(300.0379266750948, 0.0, 101985.0,
       0.0, -300.041782729805, 2826915.0), 'tiled': False, 'interleave': 'pixel'}

I wonder if you may have a problem with unresolved Proj data. To debug this, turn on Python's logging facility and set the level to DEBUG at the top of your code like so:

import logging
logging.basicConfig(level=logging.DEBUG)

On Thu, Jul 19, 2018 at 11:24 PM, Luke Pinner <lukepinnerau@...> wrote:
Hi all.  Congrats on the 1.0.0 release!  

Using rasterio 1.0.0 (installed from conda-forge in a fresh conda env). I'm having trouble copying the CRS from on dataset to another.  The output dataset CRS is None.

Simplified example demonstrating:

import rasterio as rio

inpath = r"test.tif"
outpath = r"testout.tif"

with rio.open(inpath) as inds:
    print(repr(inds.crs), inds.meta['crs'])
    with rio.open(outpath, 'w', **inds.meta) as outds:
        outds.write(inds.read())

with rio.open(outpath) as outds:
    print(repr(outds.crs), outds.meta['crs'])

The output of the above is:

CRS({'init': 'epsg:4326'}) +init=epsg:4326
None None

Dataset info:

rio info test.tif
{"blockxsize": 128, "blockysize": 128, "bounds": [143.22468138068513, -34.72000041269073, 146.20968138068514, -31.952500412690732], "colorinterp": ["grey", "undefined", "undefined", "undefined", "undefined", "undefined", "undefined", "undefined", "undefined", "undefined", "undefined"], "count": 11, "crs": "EPSG:4326", "descriptions": [null, null, null, null, null, null, null, null, null, null, null], "driver": "GTiff", "dtype": "float32", "height": 1107, "indexes": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11], "interleave": "pixel", "lnglat": [144.71718138068513, -33.33625041269073], "mask_flags": [["nodata"], ["nodata"], ["nodata"], ["nodata"], ["nodata"], ["nodata"], ["nodata"], ["nodata"], ["nodata"], ["nodata"], ["nodata"]], "nodata": -9999.0, "res": [0.0025000000000000113, 0.0024999999999999983], "shape": [1107, 1194], "tiled": true, "transform": [0.0025000000000000113, 0.0, 143.22468138068513, 0.0, -0.0024999999999999983, -31.952500412690732, 0.0, 0.0, 1.0], "units": [null, null, null, null, null, null, null, null, null, null, null], "width": 1194}

Is this user error (do I need to pass a crs string in?) or perhaps a bug?

Regards
Luke






--
Sean Gillies


Reproject with mask preserving

Denis Rykov
 

Hi there!

I have GeoTIFF files with internal mask and I need to reproject them and not lose mask.

I've found that direct generating warped mask band is not supported by GDAL: https://github.com/OSGeo/gdal/issues/689

rasterio.warp.reproject() also doesn't preserve mask.

How I can use rasterio for achieving what I need? I'm looking for something similar described in this issue but with rasterio.


Re: Copying Rasterio crs from input to output dataset

Luke
 

My GDAL_DATA env var wasn't set.

Explanation:
I was running the code within the PyCharm IDE that supposedly supports conda envs but doesn't actually *activate* them properly.  There are init scripts in CONDA_PREFIX/etc/conda/activate.d  that get run when you activate/source activate a conda env that do some additional environment setup and there's a gdal one that should've been run but wasn't.

Apologies for the noise. 

Luke