Date   

Rasterio 1.1.6

Sean Gillies
 

Hi all,

Wheels for 1.1.6 are on PyPI now. There are a number of significant bug fixes: https://github.com/mapbox/rasterio/blob/master/CHANGES.txt#L10.

The manylinux1 and macosx wheels include GDAL 2.4.4, still, but with two new patches. I'm waiting for more comments in another thread before switching to GDAL 3.1 and PROJ 7. We are not yet building wheels for Python 3.9 or 3.10, sorry. That depends on some updates to the build infrastructure that I have no time for at the moment.

Thanks for all the great bug reports!

--
Sean Gillies


Re: Request for comment: rasterio wheels on PyPI and GDAL 3.1

Howard Butler
 



On Sep 13, 2020, at 4:21 PM, Sean Gillies <sean.gillies@...> wrote:

Hi all,

The rasterio 1.1.5 wheels on PyPI include GDAL 2.4.4 and a patched PROJ 4.9.3. Neither of these versions are supported anymore. Recently I have heard from users and contributors would like to see GDAL 3.1.x in the PyPI wheels soon, to benefit from new features, and also I have heard from those who would rather stick with 2.4.4 for a while yet due to increased latency noticed with the new dependency on proj.db in PROJ versions 6 and greater. I'm sympathetic to both arguments. Rasterio depends on some new GDAL bug fixes to really shine and be its best. On the other hand, I really don't know how best to distribute GDAL 3.1 and PROJ 7.1, particularly when it comes to including PROJ data in the wheels (as we have) or relying on the PROJ CDN.

I'd love to hear comments on whether rasterio wheels should switch over to GDAL 3.1 and PROJ 7.1, especially those based on experience with the latest PROJ in production in distributed and "serverless" systems.

As an instigator for the CDN and frequent grid shifter due to our use in the geodetic and lidar domain, I've been very happy with PROJ's CDN approach in Lambda scenarios. You can find my Dockerfile at https://github.com/PDAL/lambda and my latest public layer with GDAL 3.1.3 and PROJ 7.1.1 is at arn:aws:lambda:us-east-1:163178234892:layer:pdal:35

As a library packager, I agree the PROJ data is a burden. The wheel approach of Python even makes that burden more pronounced. I like Alan's pyproj approach of pointing the user at how to fetch the data if they need it, or flip on the network bit if they're ok with that. It seems like a good compromise and a packaging approach that is not that much different from the pre-6.0 days when most people ignored the grids entirely.

The latency complaint? Are you doing 100s of lookups per second (MapServer)? There may be more performance to get there, but someone is going to have to spend *a lot* of time to get it. I think the proj.db situation (one actual database across all the projects) is gobs better than before (at least three csv "databases" with different-but-same information in them and indeterminate  code paths for calculation of a definition). 

Howard


Re: Request for comment: rasterio wheels on PyPI and GDAL 3.1

Alan Snow
 

My first thought is that if you do make the switch, I think it would make sense to do it in the 1.2 release.

> I have heard from those who would rather stick with 2.4.4 for a while yet due to increased latency noticed with the new dependency on proj.db in PROJ versions 6 and greater.

There will always be the --no-binary option and they can install older versions of GDAL/PROJ. It is what we do at work.

> I really don't know how best to distribute GDAL 3.1 and PROJ 7.1, particularly when it comes to including PROJ data in the wheels (as we have) or relying on the PROJ CDN.

Feel free to use whatever you need in the transition from here: https://github.com/pyproj4/pyproj-wheels.

pyproj 3 has decided to not include any datum/transformation grids in the wheels and has a page for users to reference for how to download grids themselves: https://pyproj4.github.io/pyproj/latest/transformation_grids.html. This allows for a smaller wheel to download and allows users to only download the grids they need (useful for serverless applications).

I would also recommend waiting until PROJ 7.2 for wheels due to the CA Bundle path support needed for wheels (https://github.com/pyproj4/pyproj/issues/689).

On another note, I was unable to get manylinux1 wheels to build with the curl setup required, so manylinux2010 will be the new minimum. There is probably a winning combination, but I was unable to find it.


Request for comment: rasterio wheels on PyPI and GDAL 3.1

Sean Gillies
 

Hi all,

The rasterio 1.1.5 wheels on PyPI include GDAL 2.4.4 and a patched PROJ 4.9.3. Neither of these versions are supported anymore. Recently I have heard from users and contributors would like to see GDAL 3.1.x in the PyPI wheels soon, to benefit from new features, and also I have heard from those who would rather stick with 2.4.4 for a while yet due to increased latency noticed with the new dependency on proj.db in PROJ versions 6 and greater. I'm sympathetic to both arguments. Rasterio depends on some new GDAL bug fixes to really shine and be its best. On the other hand, I really don't know how best to distribute GDAL 3.1 and PROJ 7.1, particularly when it comes to including PROJ data in the wheels (as we have) or relying on the PROJ CDN.

I'd love to hear comments on whether rasterio wheels should switch over to GDAL 3.1 and PROJ 7.1, especially those based on experience with the latest PROJ in production in distributed and "serverless" systems.

Thanks!

--
Sean Gillies


Re: Different values when I use a window

adrianocorbelinoii@...
 

I asked the same question on stack exchange and now I understand how to obtain the results that once I expected.
But I think the answer for that is a little bit in the dark, should we improve the documentation to have an example that explain this?


Re: Different values when I use a window

adrianocorbelinoii@...
 

Hello, this is the example:

import rasterio
#data link: https://earthexplorer.usgs.gov/download/external/options/SENTINEL_2A/3022286/INVSVC/
 
pathToImgFolder = ""
pathData = pathToImgFolder + "L1C_T21LXC_A001666_20170701T140052\S2B_MSIL1C_20170701T140049_N0205_R067_T21LXC_20170701T140052.SAFE\GRANULE\L1C_T21LXC_A001666_20170701T140052\IMG_DATA\T21LXC_20170701T140049_B01.jp2" 
boudingBoxCoordinates = (606859.0750363453, 8241169.269917269, 607219.0750363453, 8241529.269917269)
points = [(607014.0750363453,8241374.26991727),(607024.0750363444,8241374.26991727),\
          (607034.0750363453,8241374.26991727),(607044.0750363453,8241374.26991727),\
          (607054.0750363442,8241374.26991727)]
 
with rasterio.open(pathData) as img:
    bandWindow = rasterio.windows.from_bounds(*boudingBoxCoordinates, img.transform)
    winTransform = rasterio.windows.transform(bandWindow,img.transform)
    bandData = img.read(1, window = bandWindow)
    allData = img.read(1)
    for point in points:
        rBand,cBand = rasterio.transform.rowcol(winTransform,point[0],point[1])
        rFull,cFull = img.index(point[0],point[1])
        bandVal = bandData[rBand,cBand]
        fullVal = allData[rFull,cFull]
        print(f"{fullVal} {'=' if bandVal == fullVal else '!='} {bandVal}")

When I execute this example i get this result:

1202 = 1202
1330 != 1202
1330 != 1202
1330 = 1330
1330 = 1330
I want to understand what I am doing wrong.


Re: GDAL doesn't include file gcs.csv anymore

Pierrick Rambaud
 

Thank you for your quick answer,

I'm working on a production cloud company service so I don't have control on the way libs are installed. I can just ask the people in charge and they reply that they installed it with apt. 


in some notebook if I del the os.environ['GDAL_DATA'] at the start it mange to find back the gcs.csv file. Problem is: if I then use a gdal function it's the other way around. 
Is there a way to make them both compatible with gdal 3.0.4 as suggested in the README ? 


Re: GDAL doesn't include file gcs.csv anymore

Sean Gillies
 

Hi,

On Mon, Sep 7, 2020 at 8:18 AM <pierrick.rambaud49@...> wrote:

description 

 
I'm using [geemap](https://github.com/giswqs/geemap) to display a raster on a created map. 
This lib use `xarray_leaflet` to display the raster and this lib will end up using `rasterio` to manipulate the .tif file. 
 
When I launch my display :

m = geemap.Map()
m.add_raster(clip_map, colormap='terrain', layer_name='gfc')
 
I get the following error :
> CRSError: Unable to open EPSG support file gcs.csv.  Try setting the GDAL_DATA environment variable to point to the directory containing EPSG csv files.
 
This error is everywhere on SO so I tried to verify if my GDAL_DATA env variable was coorectly set : 
 
import os
import stat
gdal_data = os.environ['GDAL_DATA']
print('is dir: ' + str(os.path.isdir(gdal_data)))
gcs_csv = os.path.join(gdal_data, 'gcs.csv')
print('is file: ' + str(os.path.isfile(gcs_csv)))
st = os.stat(gcs_csv)
print('is readable: ' + str(bool(st.st_mode & stat.S_IRGRP)))
 
# out 
# is dir: True
#is file: False
#FileNotFoundError: [Errno 2] No such file or directory: '/usr/share/gdal/gcs.csv'
 
going to the glad distrib inb the `NEWS` file, I read that they removed lots of file in 3.0 including `gcs.csv`. So it's no longer included in my folder. 
 
Is there a workaround ? 


setup 

rasterio==1.1.5
python==3.6
geemap==0.7.9
gdal==3.0.4

How did you install rasterio? If you got it with `pip install rasterio` you will have installed a wheel file that includes GDAL 2.4.4 and data files and which will be incompatible with the GDAL 3.0.4 installed on your system. If that is the case, you should unset GDAL_DATA. Rasterio in the wheel will detect its own data files and does not need GDAL_DATA to be set.

--
Sean Gillies


Re: Status of GDAL 3.1 Support?

Sean Gillies
 

Hi Daryl,

On Mon, Sep 7, 2020 at 6:01 AM Herzmann, Daryl E [AGRON] <akrherz@...> wrote:
Howdy,

I am curious about rasterio's current GDAL version support?  Presently, my conda environment is pinned down to GDAL < 3.1

https://github.com/conda-forge/rasterio-feedstock/pull/171#issuecomment-678631972

As I search around on rasterio's github, I find this PR which seems to pass tests against newer GDALs?

https://github.com/mapbox/rasterio/pull/1987

Is there some GH issue tracking GDAL 3.1 support or should 1.1.5 rasterio work with GDAL 3.1?

thanks!
daryl

We're testing rasterio's trunk against GDAL 3.1.2 here: https://travis-ci.org/github/mapbox/rasterio. We are not currently checking the 1.1.x branch, but in my experience there are no problems. I have rasterio 1.1.5, GDAL 3.1.2, and PROJ 7.1.0 in production.

--
Sean Gillies


GDAL doesn't include file gcs.csv anymore

Pierrick Rambaud
 

description 

 
I'm using [geemap](https://github.com/giswqs/geemap) to display a raster on a created map. 
This lib use `xarray_leaflet` to display the raster and this lib will end up using `rasterio` to manipulate the .tif file. 
 
When I launch my display :

m = geemap.Map()
m.add_raster(clip_map, colormap='terrain', layer_name='gfc')
 
I get the following error :
> CRSError: Unable to open EPSG support file gcs.csv.  Try setting the GDAL_DATA environment variable to point to the directory containing EPSG csv files.
 
This error is everywhere on SO so I tried to verify if my GDAL_DATA env variable was coorectly set : 
 
import os
import stat
gdal_data = os.environ['GDAL_DATA']
print('is dir: ' + str(os.path.isdir(gdal_data)))
gcs_csv = os.path.join(gdal_data, 'gcs.csv')
print('is file: ' + str(os.path.isfile(gcs_csv)))
st = os.stat(gcs_csv)
print('is readable: ' + str(bool(st.st_mode & stat.S_IRGRP)))
 
# out 
# is dir: True
#is file: False
#FileNotFoundError: [Errno 2] No such file or directory: '/usr/share/gdal/gcs.csv'
 
going to the glad distrib inb the `NEWS` file, I read that they removed lots of file in 3.0 including `gcs.csv`. So it's no longer included in my folder. 
 
Is there a workaround ? 


setup 

rasterio==1.1.5
python==3.6
geemap==0.7.9
gdal==3.0.4
 


Status of GDAL 3.1 Support?

Herzmann, Daryl E [AGRON]
 

Howdy,

I am curious about rasterio's current GDAL version support? Presently, my conda environment is pinned down to GDAL < 3.1

https://github.com/conda-forge/rasterio-feedstock/pull/171#issuecomment-678631972

As I search around on rasterio's github, I find this PR which seems to pass tests against newer GDALs?

https://github.com/mapbox/rasterio/pull/1987

Is there some GH issue tracking GDAL 3.1 support or should 1.1.5 rasterio work with GDAL 3.1?

thanks!
daryl


Re: Black border after running rio mask

ts@...
 

Thanks for your help Sean,

I can confirm the black artifacts come from jpeg compression.
I also found this answered which explains the problem in detail:
https://gis.stackexchange.com/a/114453/52871

best regards,

Toni


Re: Black border after running rio mask

Sean Gillies
 

Hi,

On Sat, Aug 29, 2020 at 8:52 AM <ts@...> wrote:
Dear Rasterio developers and useers,

I' masking geotiff files with geojson polygons:

rio mask --overwrite --crop in.tif out.tif --geojson-mask mask.geojson 

Unfortunately after doing so black artificats showing up around the cropped area.



Is there a way to avoid this artifacts?


It looks like your input.tif may have lossy compression applied, and the output will have the same compression by default. You might try

rio mask --overwrite --crop in.tif out.tif --geojson-mask mask.geojson --co COMPRESS=NONE

--
Sean Gillies


Black border after running rio mask

ts@...
 
Edited

Dear Rasterio developers and users,

I' m masking geotiff files with geojson polygons:

rio mask --overwrite --crop in.tif out.tif --geojson-mask mask.geojson 

Unfortunately after doing so black artificats showing up around the cropped area.



Is there a way to avoid this artifacts?


Re: Different values when I use a window

adrianocorbelinoii@...
 

Hello Sean,

Sorry for abandoning the post.
I installed rasterio via pip, and recently i had to reinstall my OS, and also of course rasterio, and the problem still persists.
I will prepare a demo and a link to a sample data that am I using, soon as I can.


Re: How sum two raster with different shapes?

Anderson Roberto da Silva
 

Thank's Amine. I create a function to resample the raster keeping the extents and resolution to calc with other rasters.


Re: rasterio opens file from AWS S3 bucket on local machine, but can't find file when deployed to Google App Engine

Sean Gillies
 

Hi Judson,

On Fri, Aug 21, 2020 at 9:16 AM Judson Buescher <judson.buescher@...> wrote:

Sean,

Sorry, I’ll try to be a bit more explicit. I’m getting the following error after setting the keys in my app.yaml file. I’m entirely sure what it means but wonder if it has something to do with trying to run it on GAE.

Traceback (most recent call last): File "rasterio/_base.pyx", line 216, in rasterio._base.DatasetBase.__init__ File "rasterio/_shim.pyx", line 67, in rasterio._shim.open_dataset File "rasterio/_err.pyx", line 205, in rasterio._err.exc_wrap_pointer rasterio._err.CPLE_HttpResponseError: CURL error: error setting certificate verify locations: CAfile: /etc/pki/tls/certs/ca-bundle.crt CApath: none



--
Sean Gillies


Re: rasterio opens file from AWS S3 bucket on local machine, but can't find file when deployed to Google App Engine

Judson Buescher <judson.buescher@...>
 

 

Hi Sean,

I’ve tried using os.environ and also env_variables in my app.yaml. Is there a better way that you know of to set the aws access keys?

 

 

From: <main@rasterio.groups.io> on behalf of "Sean Gillies via groups.io" <sean@...>
Reply-To: "main@rasterio.groups.io" <main@rasterio.groups.io>
Date: Friday, August 14, 2020 at 7:03 PM
To: "main@rasterio.groups.io" <main@rasterio.groups.io>
Subject: Re: [rasterio] rasterio opens file from AWS S3 bucket on local machine, but can't find file when deployed to Google App Engine

 

Hi Judson,

 

On Wed, Aug 12, 2020 at 9:42 AM Judson Buescher <judson.buescher@...> wrote:

Hi Sean,

The only way that I’m installing the package is through my requirements.txt which I include in my deploy folder. I have

 

rasterio==1.1.5

 

in the requirements.txt. For the credentials I’ve tried a couple different ways. I’ve tried setting them using

 

os.environ['GS_SECRET_ACCESS_KEY'] = secret_access_key

os.environ['GS_ACCESS_KEY_ID'] = access_key

 

I’ve also tried

 

session = boto3.Session(aws_access_key_id=access_key,

                        aws_secret_access_key=secret_access_key)

 

and then using that session object when I open the raster file. Was there any other info you’d like?

Thanks a ton for the help,
Judson

 

If the source data for your App Engine app is on AWS, you'll need to make sure that you're providing AWS keys to access it.  You'll want

 

    AWS_SECRET_ACCESS_KEY = secret_access_key

    etc.

 

in your environment, not GS_SECRET_ACCESS_KEY (etc.). I've never used boto3 on App Engine and wouldn't recommend that approach of configuring AWS for GDAL and rasterio.

 

--

Sean Gillies


Re: rasterio opens file from AWS S3 bucket on local machine, but can't find file when deployed to Google App Engine

Judson Buescher <judson.buescher@...>
 

Sean,

Sorry, I’ll try to be a bit more explicit. I’m getting the following error after setting the keys in my app.yaml file. I’m entirely sure what it means but wonder if it has something to do with trying to run it on GAE.

Traceback (most recent call last): File "rasterio/_base.pyx", line 216, in rasterio._base.DatasetBase.__init__ File "rasterio/_shim.pyx", line 67, in rasterio._shim.open_dataset File "rasterio/_err.pyx", line 205, in rasterio._err.exc_wrap_pointer rasterio._err.CPLE_HttpResponseError: CURL error: error setting certificate verify locations: CAfile: /etc/pki/tls/certs/ca-bundle.crt CApath: none

 

 

From: <main@rasterio.groups.io> on behalf of "Sean Gillies via groups.io" <sean@...>
Reply-To: "main@rasterio.groups.io" <main@rasterio.groups.io>
Date: Friday, August 14, 2020 at 7:03 PM
To: "main@rasterio.groups.io" <main@rasterio.groups.io>
Subject: Re: [rasterio] rasterio opens file from AWS S3 bucket on local machine, but can't find file when deployed to Google App Engine

 

Hi Judson,

 

On Wed, Aug 12, 2020 at 9:42 AM Judson Buescher <judson.buescher@...> wrote:

Hi Sean,

The only way that I’m installing the package is through my requirements.txt which I include in my deploy folder. I have

 

rasterio==1.1.5

 

in the requirements.txt. For the credentials I’ve tried a couple different ways. I’ve tried setting them using

 

os.environ['GS_SECRET_ACCESS_KEY'] = secret_access_key

os.environ['GS_ACCESS_KEY_ID'] = access_key

 

I’ve also tried

 

session = boto3.Session(aws_access_key_id=access_key,

                        aws_secret_access_key=secret_access_key)

 

and then using that session object when I open the raster file. Was there any other info you’d like?

Thanks a ton for the help,
Judson

 

If the source data for your App Engine app is on AWS, you'll need to make sure that you're providing AWS keys to access it.  You'll want

 

    AWS_SECRET_ACCESS_KEY = secret_access_key

    etc.

 

in your environment, not GS_SECRET_ACCESS_KEY (etc.). I've never used boto3 on App Engine and wouldn't recommend that approach of configuring AWS for GDAL and rasterio.

 

--

Sean Gillies


Re: Iterate over elements in Rasterio window and obtain coordinates using transform.xy

Sean Gillies
 

Hi,

On Wed, Aug 19, 2020 at 6:24 PM <whytefish1@...> wrote:

[Edited Message Follows]

Worked it out using a previous thread, doing it using an affine transform on the window.

Excellent!
 
I would however like to know if the rasterio.transform.xy method can be used when passing col, row from a dataset window, as I would like to get the coordinate for the center of the cell rather than the ul which results from the affine window transform.

Yes. The important detail is that the transform matrix passed to xy() must apply to the window. You can't use the matrix from the dataset's .transform attribute unless your window has its origin at the upper left corner of the dataset. Dataset objects have a window_transform() method that can help compute the new transforms: https://rasterio.readthedocs.io/en/latest/api/rasterio.io.html?highlight=window_transform#rasterio.io.DatasetReader.window_transform.

--
Sean Gillies

341 - 360 of 945