Date   
Re: GDAL version policy

Sean Gillies
 

Hi Denis,

On Wed, Oct 3, 2018 at 1:50 AM Denis Rykov <rykovd@...> wrote:
Hello folks,

What is the policy of the project about GDAL version?
The latest stable release of GDAL is 2.3.2 but recent wheels of rasterio  are built onto 2.3.1

Good question! My intent for the wheels is to make it easy for people to try Rasterio and to be able to easily provision CI servers for testing Python software that requires Rasterio. There isn't really any policy yet except that I do keep the GDAL and other libraries versions under revision control so that they don't regress and try to keep them not too far out of date.

I will happily accept PRs at https://github.com/sgillies/rasterio-wheels to update libraries to the latest stable versions. As to format drivers, I would like to minimize drivers to keep the wheels under 20MB and keep wheel build times to under 20 mins.

--
Sean Gillies

GDAL version policy

Denis Rykov
 

Hello folks,

What is the policy of the project about GDAL version?
The latest stable release of GDAL is 2.3.2 but recent wheels of rasterio  are built onto 2.3.1.

Rasterio 1.0.8

Sean Gillies
 

Hi all,

Rasterio 1.0.8 has been uploaded to PyPI. Thanks to Denis Rykov, we have Python 3.4 wheels for manylinux1 again. And datasets stored in MemoryFile bytes are now opened in r+ or w+ mode as requested by Amine Aboufirass earlier today.

--
Sean Gillies

Re: Open a raster file with read and write permissions in rasterio

Sean Gillies
 

Hi,

On Tue, Oct 2, 2018 at 8:14 AM Amine Aboufirass <amine.aboufirass@...> wrote:
I have already asked this question in GIS stack exchange  and stack overflow but received no response:
 
I am trying to write a function which will help me construct a `rasterio` memory file and return the dataset reader object so I can then play around with it as needed. I am trying to avoid `with` statements because I would like to allow the user to read and write as they deem necessary. This is what I have so far:
 
 
    from rasterio import mask, features, warp
    from rasterio.io import MemoryFile
    import rasterio
    import numpy as np
 
    def create_memory_file(data, west_bound, north_bound, cellsize, driver='AAIGrid'):
        #data is a numpy array
        dtype = data.dtype
        shape = data.shape
        transform = rasterio.transform.from_origin(west_bound, north_bound, cellsize, cellsize)
        memfile = MemoryFile()
        dataset = memfile.open(driver=driver, width= shape[1], height = shape[0], transform=transform, count=1, dtype = dtype)
        dataset.write(data,1)
    
        return dataset
    def close_memory_file(memfile):
        memfile.close()
    
    
    data = np.array([[1,2,3], [4,5,6], [7,8,9]])
    memfile = create_memory_file(data, 0, 2, 0.5)
    memfile.read(1)
 
The last line throws an error:
 
    Traceback (most recent call last):
      File "D:/11202750-002_RA2CE/Basis/common.py", line 214, in <module>
        memfile.read(1)
      File "rasterio\_io.pyx", line 209, in rasterio._io.DatasetReaderBase.read
    rasterio.errors.UnsupportedOperation: not readable
 
The memfile apparently only has write permissions because of the following:
 
    >>> memfile
    <open BufferedDatasetWriter name='/vsimem/3c0572dd-a295-4f2b-a65d-22a404dd5d0f.' mode='w'>
    >>> memfile.mode
    'w'
 
I have tried various combinations of adding `mode=r+`, `permissions=r+` to my `memfile.open()` call but it won't allow this.
 
How can I add the read/write mode at the level of the `open` statement?
 
 

Currently, the datasets in a MemoryFile can only be opened in read or write modes. Instances created with no initial bytes have their mode fixed to "w" and instances created with initial bytes have their mode fixed to "r". I could explore changing these to "w+" and "r+" instead for the next version of Rasterio.

That said, few data formats can be used in this way. The ascii grid format in your example, for one, cannot be updated in place by GDAL or Rasterio. It has a "no" in the "Creation" column of the table in https://www.gdal.org/formats_list.html and like other such formats is read-only or write-on-copy only.

--
Sean Gillies

Open a raster file with read and write permissions in rasterio

Amine Aboufirass
 

I have already asked this question in GIS stack exchange  and stack overflow but received no response:
 
I am trying to write a function which will help me construct a `rasterio` memory file and return the dataset reader object so I can then play around with it as needed. I am trying to avoid `with` statements because I would like to allow the user to read and write as they deem necessary. This is what I have so far:
 
 
    from rasterio import mask, features, warp
    from rasterio.io import MemoryFile
    import rasterio
    import numpy as np
 
    def create_memory_file(data, west_bound, north_bound, cellsize, driver='AAIGrid'):
        #data is a numpy array
        dtype = data.dtype
        shape = data.shape
        transform = rasterio.transform.from_origin(west_bound, north_bound, cellsize, cellsize)
        memfile = MemoryFile()
        dataset = memfile.open(driver=driver, width= shape[1], height = shape[0], transform=transform, count=1, dtype = dtype)
        dataset.write(data,1)
    
        return dataset
    def close_memory_file(memfile):
        memfile.close()
    
    
    data = np.array([[1,2,3], [4,5,6], [7,8,9]])
    memfile = create_memory_file(data, 0, 2, 0.5)
    memfile.read(1)
 
The last line throws an error:
 
    Traceback (most recent call last):
      File "D:/11202750-002_RA2CE/Basis/common.py", line 214, in <module>
        memfile.read(1)
      File "rasterio\_io.pyx", line 209, in rasterio._io.DatasetReaderBase.read
    rasterio.errors.UnsupportedOperation: not readable
 
The memfile apparently only has write permissions because of the following:
 
    >>> memfile
    <open BufferedDatasetWriter name='/vsimem/3c0572dd-a295-4f2b-a65d-22a404dd5d0f.' mode='w'>
    >>> memfile.mode
    'w'
 
I have tried various combinations of adding `mode=r+`, `permissions=r+` to my `memfile.open()` call but it won't allow this.
 
How can I add the read/write mode at the level of the `open` statement?
 
 
  [1]: https://gis.stackexchange.com/questions/297562/open-a-raster-file-with-read-and-write-permissions-in-rasterio

Re: Issue when using rasterio dataset inside Class with multiprocessing

Matthew Perry
 

An open Rasterio dataset object should not be passed between multiple processes or threads; the underlying GDALDataset is not thread safe. Additionally, the dataset's lifecycle should be made explicit - either by explicitly calling .close or opening as a context manager (recommended).

It's not clear what your intention is with the `worker` function but I can see two ways to approach it, depending on your goal

if each process simply needs access to the array of data, I would read all of the data in __init__ and close out the dataset before invoking any parallel workers. Then you're just sharing a numpy array instead of a stateful dataset object.

    def __init__(self):
        with rasterio.open('/Users/mperry/work/rasterio/tests/data/RGB.byte.tif') as src:
            data = src.read()
 

if you need to read different parts of the dataset from each process, you should pass the dataset path and open/close the a new dataset within each thread. You can't share a dataset object between threads/procs but you can create multiple datasets pointing to the same resource.

Hope this helps.

Re: Float precision on writing grid

Matthew Perry
 

You can control the precision of the AAIGrid output using the GDAL creation options DECIMAL_PRECISION (or SIGNIFICANT_DIGITS). You can specify these as case-insensitive kwargs like `rasterio.open(..., decimal_precision=3)`

Float precision on writing grid

Betman, Maarten
 

I'm trying to transform xyz data to a raster grid using rasterio

I've loaded the elevation data into a numpy array, the precision of the z values is 2 decimals and the array has dtype float64.

after loading the xyz data and putting elevation data in a grid the array zi has the following properties and example value in the array:

[in]  zi.shape

[out] (3796, 3557)

 

[in]  zi.dtype

[out] dtype('float64')

 

[in]  zi[1000,1000]

[out] -5.27

I'm trying to write the gridded data to an ESRI Ascii file using rasterio and the AAIGrid driver.

import rasterio

from rasterio.transform import from_origin

from fiona.crs import from_epsg

 

transform = from_origin(xmin, ymax, 0.5, 0.5)

 

new_dataset = rasterio.open('test1.asc', 'w', driver='AAIGrid',

                        height = zi.shape[0], width = zi.shape[1],

                        count=1, dtype=str(zi.dtype)

                        crs=from_epsg(32750),

                        transform=transform,

                        nodata = -9999)

 

new_dataset.write(zi, 1)

new_dataset.close()

The z grid is written to the ESRI Ascii file successfully, only the "precision" of the values is now 19 decimals. This makes the files unnecessarily large and slows down programs using the gridded files.

Header ESRI Ascii file and first 2 values:

ncols        3557

nrows        3796

xllcorner    765005.500000000000

yllcorner    9430016.000000000000

cellsize     0.500000000000

NODATA_value -9999

-0.029999999329447746277 -0.029999999329447746277

I've tried changing the datatype to float32 and added argument "precision = 2" to rasterio.open. Is their a way I can get the number of decimals down?

Issue when using rasterio dataset inside Class with multiprocessing

luoyntech@...
 

I have a simple class which has a member variable of rasterio dataset. Inside the class, there is also a function wrapped by a python multiprocessing call. See below



This code is supposed to print 'Worker 0', 'Worker 1' and 'Worker 2'. However, when I actually ran this code, it printed nothing but exited normally.




I then tried to comment out the line reading the tif image using rasterio, which looks like this


This time it printed out text as I expected.




Is there any possible reason that causes this issue? Thanks!

Re: Rasterio 1.0.3 and Python 3.4

Sean Gillies
 

Hi Denis,

Since we switched over to making  wheels using the multibuild project we're blocked from releasing *well tested* Python 3.4 linux wheels by (my hypothesis) bugs in trusty's python 3.4.3. The issue is described here: https://github.com/sgillies/rasterio-wheels/issues/5.

I'm short on time to solve this problem, but there are 2 solid workarounds:

1) build and install rasterio from source `GDAL_CONFIG=path/to/gdal-config pip install rasterio` (requires build-essential, gdal-dev, other system packages).
2) switch to Python 3.5 or 3.6 from the deadsnakes PPA and then `pip install rasterio` to get a wheel for one of those versions.


On Fri, Sep 28, 2018 at 9:38 AM Denis Rykov <rykovd@...> wrote:
There is no Linux wheel for Python 3.4 here: https://pypi.org/project/rasterio/#files



--
Sean Gillies

Re: Rasterio 1.0.3 and Python 3.4

Denis Rykov
 

There is no Linux wheel for Python 3.4 here: https://pypi.org/project/rasterio/#files

Rasterio 1.0.3 and Python 3.4

Denis Rykov
 
Edited

Hello folks,

I have problems with installation rasterio==1.0.3 and higher on Python 3.4. There are no problems with 3.5 and 3.6.

Is it an expected behavior?

$ sudo docker run --name python34 -it python:3.4 bash
root@e306b109d4c0:/# pip install numpy
Collecting numpy
  Downloading https://files.pythonhosted.org/packages/14/1c/546724245c8b3aad39d807a0bed14a37b39943860c6b34456a363076c65b/numpy-1.15.2-cp34-cp34m-manylinux1_x86_64.whl (13.8MB)
    100% |████████████████████████████████| 13.8MB 3.1MB/s 
Installing collected packages: numpy
Successfully installed numpy-1.15.2
root@e306b109d4c0:/# pip install rasterio
Collecting rasterio
  Downloading https://files.pythonhosted.org/packages/ce/e3/86ef71d178887f2e365763607f974d5a70be9fbd422e4378bd8c16b8c306/rasterio-1.0.7.tar.gz (1.8MB)
    100% |████████████████████████████████| 1.8MB 8.7MB/s 
    Complete output from command python setup.py egg_info:
    WARNING:root:Failed to get options via gdal-config: [Errno 2] No such file or directory: 'gdal-config'
    ERROR: A GDAL API version must be specified. Provide a path to gdal-config using a GDAL_CONFIG environment variable or use a GDAL_VERSION environment variable.

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-bv7euygw/rasterio/

Rasterio 1.0.7

Sean Gillies
 

Hi all,

Rasterio 1.0.7 is compatible with click versions >= 4 and < 8 (not released yet, of course) including the new click version 7.0, and fixes a couple other small bugs as well. Please upgrade when you can.

--
Sean Gillies

Please upgrade cligj to 0.5

Sean Gillies
 

Hi all,

A rasterio dependency, cligj 0.4.0, is not compatible with click 7 (released yesterday) and if click has been upgraded in your environment you will notice that the rio commands are broken. Upgrading cligj to 0.5.0 will solve the problem and maintain compatibility with older versions of click as well.


The next rasterio release (1.0.7) will require cligj 0.5.0.

--
Sean Gillies

Re: Click 7.0 breaks Rasterio

Sean Gillies
 

It turns out that the culprit is the cligj package. I'm uploading a new release to PyPI now.

On Wed, Sep 26, 2018 at 9:31 AM Alan Snow <alansnow21@...> wrote:
I bet the subcommand remame from underscore to dash will break a lot.



--
Sean Gillies

Re: Click 7.0 breaks Rasterio

Alan Snow
 

I bet the subcommand remame from underscore to dash will break a lot.

Click 7.0 breaks Rasterio

Sean Gillies
 

Hi all,

Heads up: click 7.0 was released to PyPI yesterday and rasterio is not compatible with it. If you pin `click==6.7` in your applications, you'll be fine. The change log entry for 7 is looong and I don't immediately see the problem: https://github.com/pallets/click/blob/master/CHANGES.rst#version-70.

--
Sean Gillies

Rasterio 1.0.6

Sean Gillies
 

Hi all,

Rasterio 1.0.6 is on PyPI now with 3 significant bug fixes:
- If the build_overviews method of a dataset is passed a list of factors that specify more than one 1x1 pixel overview (#1333), Rasterio raises an exception.
- Calling calculate_default_transform for large extents should no longer result in the out of memory error reported in #1131. The rio-warp command should also now run more quickly and with a smaller memory footprint.
- We have a more general fix for the problem of filling the empty regions of boundless reads (#1471).
Thanks for the excellent issue reports and debugging help!

--
Sean Gillies

Rasterio 1.0.5

Sean Gillies
 

Hi all,

Rasterio 1.0.5 is on PyPI now. It fixes 3 bugs and provides a source distribution that will be compatible with Python 3.7. From the change log:
Bug fixes:

- The fill value for boundless reads was ignored in Rasterio versions 1-1.0.4
  but now applies (#1471).
- An invalid shortcut has been eliminated and Rasterio now produces a proper
  mask in the boundless masked read case (#1449).
- Loss of a row or column in geometry_window() and mask() has been fixed
  (#1472).
--
Sean Gillies

Re: In-memory multi dataset object

Guy Doulberg
 

Thanks, 

בתאריך יום ה׳, 13 בספט׳ 2018, 18:59, מאת Sean Gillies ‏<sean.gillies@...>:

Hi Guy,

Rasterio doesn't have first-class support for VRTs. I'm not rushing to do it because there are some VRT features that I frankly think are nuts, like embedding Python code.

However, we can use Python's XML libraries to generate XML texts and Rasterio will happily open the XML as a dataset. See the example at https://github.com/mapbox/rasterio/blob/master/tests/test_vrt.py#L9-L11. I'm doing this internally to implement Rasterio's boundless raster reads.


On Thu, Sep 13, 2018 at 7:02 AM Guy Doulberg <guyd@...> wrote:
Hi
I wonder if I can create in-memory multi-dataset dataset, something like `gdalbuildvrt` but in memory

The use case I have in mind, is that I have a list of urls some of them construct the same band and some construct different band and I want to treat them as a single dataset

Theoretically, I an produce such dataset by calling gdalbuildvrt with those urls and them open the vrt created.  Is there another way of doing that?

If no, in environment that has only rasterio installed not gdal, how can I use  gdalbuildvrt?

Thanks Guy


--
Sean Gillies