1. main@rasterio.groups.io
   Date   
Re: GDAL doesn't include file gcs.csv anymore

Sean Gillies
 

Hi,

On Mon, Sep 7, 2020 at 8:18 AM <pierrick.rambaud49@...> wrote:

description 

 
I'm using [geemap](https://github.com/giswqs/geemap) to display a raster on a created map. 
This lib use `xarray_leaflet` to display the raster and this lib will end up using `rasterio` to manipulate the .tif file. 
 
When I launch my display :

m = geemap.Map()
m.add_raster(clip_map, colormap='terrain', layer_name='gfc')
 
I get the following error :
> CRSError: Unable to open EPSG support file gcs.csv.  Try setting the GDAL_DATA environment variable to point to the directory containing EPSG csv files.
 
This error is everywhere on SO so I tried to verify if my GDAL_DATA env variable was coorectly set : 
 
import os
import stat
gdal_data = os.environ['GDAL_DATA']
print('is dir: ' + str(os.path.isdir(gdal_data)))
gcs_csv = os.path.join(gdal_data, 'gcs.csv')
print('is file: ' + str(os.path.isfile(gcs_csv)))
st = os.stat(gcs_csv)
print('is readable: ' + str(bool(st.st_mode & stat.S_IRGRP)))
 
# out 
# is dir: True
#is file: False
#FileNotFoundError: [Errno 2] No such file or directory: '/usr/share/gdal/gcs.csv'
 
going to the glad distrib inb the `NEWS` file, I read that they removed lots of file in 3.0 including `gcs.csv`. So it's no longer included in my folder. 
 
Is there a workaround ? 


setup 

rasterio==1.1.5
python==3.6
geemap==0.7.9
gdal==3.0.4

How did you install rasterio? If you got it with `pip install rasterio` you will have installed a wheel file that includes GDAL 2.4.4 and data files and which will be incompatible with the GDAL 3.0.4 installed on your system. If that is the case, you should unset GDAL_DATA. Rasterio in the wheel will detect its own data files and does not need GDAL_DATA to be set.

--
Sean Gillies
Re: Status of GDAL 3.1 Support?

Sean Gillies
 

Hi Daryl,

On Mon, Sep 7, 2020 at 6:01 AM Herzmann, Daryl E [AGRON] <akrherz@...> wrote:
Howdy,

I am curious about rasterio's current GDAL version support?  Presently, my conda environment is pinned down to GDAL < 3.1

https://github.com/conda-forge/rasterio-feedstock/pull/171#issuecomment-678631972

As I search around on rasterio's github, I find this PR which seems to pass tests against newer GDALs?

https://github.com/mapbox/rasterio/pull/1987

Is there some GH issue tracking GDAL 3.1 support or should 1.1.5 rasterio work with GDAL 3.1?

thanks!
daryl

We're testing rasterio's trunk against GDAL 3.1.2 here: https://travis-ci.org/github/mapbox/rasterio. We are not currently checking the 1.1.x branch, but in my experience there are no problems. I have rasterio 1.1.5, GDAL 3.1.2, and PROJ 7.1.0 in production.

--
Sean Gillies
GDAL doesn't include file gcs.csv anymore

pierrick.rambaud49@...
 

description 

 
I'm using [geemap](https://github.com/giswqs/geemap) to display a raster on a created map. 
This lib use `xarray_leaflet` to display the raster and this lib will end up using `rasterio` to manipulate the .tif file. 
 
When I launch my display :

m = geemap.Map()
m.add_raster(clip_map, colormap='terrain', layer_name='gfc')
 
I get the following error :
> CRSError: Unable to open EPSG support file gcs.csv.  Try setting the GDAL_DATA environment variable to point to the directory containing EPSG csv files.
 
This error is everywhere on SO so I tried to verify if my GDAL_DATA env variable was coorectly set : 
 
import os
import stat
gdal_data = os.environ['GDAL_DATA']
print('is dir: ' + str(os.path.isdir(gdal_data)))
gcs_csv = os.path.join(gdal_data, 'gcs.csv')
print('is file: ' + str(os.path.isfile(gcs_csv)))
st = os.stat(gcs_csv)
print('is readable: ' + str(bool(st.st_mode & stat.S_IRGRP)))
 
# out 
# is dir: True
#is file: False
#FileNotFoundError: [Errno 2] No such file or directory: '/usr/share/gdal/gcs.csv'
 
going to the glad distrib inb the `NEWS` file, I read that they removed lots of file in 3.0 including `gcs.csv`. So it's no longer included in my folder. 
 
Is there a workaround ? 


setup 

rasterio==1.1.5
python==3.6
geemap==0.7.9
gdal==3.0.4
 
Status of GDAL 3.1 Support?

Herzmann, Daryl E [AGRON]
 

Howdy,

I am curious about rasterio's current GDAL version support? Presently, my conda environment is pinned down to GDAL < 3.1

https://github.com/conda-forge/rasterio-feedstock/pull/171#issuecomment-678631972

As I search around on rasterio's github, I find this PR which seems to pass tests against newer GDALs?

https://github.com/mapbox/rasterio/pull/1987

Is there some GH issue tracking GDAL 3.1 support or should 1.1.5 rasterio work with GDAL 3.1?

thanks!
daryl
Re: Black border after running rio mask

ts@...
 

Thanks for your help Sean,

I can confirm the black artifacts come from jpeg compression.
I also found this answered which explains the problem in detail:
https://gis.stackexchange.com/a/114453/52871

best regards,

Toni
Re: Black border after running rio mask

Sean Gillies
 

Hi,

On Sat, Aug 29, 2020 at 8:52 AM <ts@...> wrote:
Dear Rasterio developers and useers,

I' masking geotiff files with geojson polygons:

rio mask --overwrite --crop in.tif out.tif --geojson-mask mask.geojson 

Unfortunately after doing so black artificats showing up around the cropped area.



Is there a way to avoid this artifacts?


It looks like your input.tif may have lossy compression applied, and the output will have the same compression by default. You might try

rio mask --overwrite --crop in.tif out.tif --geojson-mask mask.geojson --co COMPRESS=NONE

--
Sean Gillies
Black border after running rio mask

ts@...
 
Edited

Dear Rasterio developers and users,

I' m masking geotiff files with geojson polygons:

rio mask --overwrite --crop in.tif out.tif --geojson-mask mask.geojson 

Unfortunately after doing so black artificats showing up around the cropped area.



Is there a way to avoid this artifacts?
Re: Different values when I use a window

adrianocorbelinoii@...
 

Hello Sean,

Sorry for abandoning the post.
I installed rasterio via pip, and recently i had to reinstall my OS, and also of course rasterio, and the problem still persists.
I will prepare a demo and a link to a sample data that am I using, soon as I can.
Re: How sum two raster with different shapes?

Anderson Roberto da Silva
 

Thank's Amine. I create a function to resample the raster keeping the extents and resolution to calc with other rasters.
Re: rasterio opens file from AWS S3 bucket on local machine, but can't find file when deployed to Google App Engine

Sean Gillies
 

Hi Judson,

On Fri, Aug 21, 2020 at 9:16 AM Judson Buescher <judson.buescher@...> wrote:

Sean,

Sorry, I’ll try to be a bit more explicit. I’m getting the following error after setting the keys in my app.yaml file. I’m entirely sure what it means but wonder if it has something to do with trying to run it on GAE.

Traceback (most recent call last): File "rasterio/_base.pyx", line 216, in rasterio._base.DatasetBase.__init__ File "rasterio/_shim.pyx", line 67, in rasterio._shim.open_dataset File "rasterio/_err.pyx", line 205, in rasterio._err.exc_wrap_pointer rasterio._err.CPLE_HttpResponseError: CURL error: error setting certificate verify locations: CAfile: /etc/pki/tls/certs/ca-bundle.crt CApath: none



--
Sean Gillies
Re: rasterio opens file from AWS S3 bucket on local machine, but can't find file when deployed to Google App Engine

Judson Buescher <judson.buescher@...>
 

 

Hi Sean,

I’ve tried using os.environ and also env_variables in my app.yaml. Is there a better way that you know of to set the aws access keys?

 

 

From: <main@rasterio.groups.io> on behalf of "Sean Gillies via groups.io" <sean@...>
Reply-To: "main@rasterio.groups.io" <main@rasterio.groups.io>
Date: Friday, August 14, 2020 at 7:03 PM
To: "main@rasterio.groups.io" <main@rasterio.groups.io>
Subject: Re: [rasterio] rasterio opens file from AWS S3 bucket on local machine, but can't find file when deployed to Google App Engine

 

Hi Judson,

 

On Wed, Aug 12, 2020 at 9:42 AM Judson Buescher <judson.buescher@...> wrote:

Hi Sean,

The only way that I’m installing the package is through my requirements.txt which I include in my deploy folder. I have

 

rasterio==1.1.5

 

in the requirements.txt. For the credentials I’ve tried a couple different ways. I’ve tried setting them using

 

os.environ['GS_SECRET_ACCESS_KEY'] = secret_access_key

os.environ['GS_ACCESS_KEY_ID'] = access_key

 

I’ve also tried

 

session = boto3.Session(aws_access_key_id=access_key,

                        aws_secret_access_key=secret_access_key)

 

and then using that session object when I open the raster file. Was there any other info you’d like?

Thanks a ton for the help,
Judson

 

If the source data for your App Engine app is on AWS, you'll need to make sure that you're providing AWS keys to access it.  You'll want

 

    AWS_SECRET_ACCESS_KEY = secret_access_key

    etc.

 

in your environment, not GS_SECRET_ACCESS_KEY (etc.). I've never used boto3 on App Engine and wouldn't recommend that approach of configuring AWS for GDAL and rasterio.

 

--

Sean Gillies
Re: rasterio opens file from AWS S3 bucket on local machine, but can't find file when deployed to Google App Engine

Judson Buescher <judson.buescher@...>
 

Sean,

Sorry, I’ll try to be a bit more explicit. I’m getting the following error after setting the keys in my app.yaml file. I’m entirely sure what it means but wonder if it has something to do with trying to run it on GAE.

Traceback (most recent call last): File "rasterio/_base.pyx", line 216, in rasterio._base.DatasetBase.__init__ File "rasterio/_shim.pyx", line 67, in rasterio._shim.open_dataset File "rasterio/_err.pyx", line 205, in rasterio._err.exc_wrap_pointer rasterio._err.CPLE_HttpResponseError: CURL error: error setting certificate verify locations: CAfile: /etc/pki/tls/certs/ca-bundle.crt CApath: none

 

 

From: <main@rasterio.groups.io> on behalf of "Sean Gillies via groups.io" <sean@...>
Reply-To: "main@rasterio.groups.io" <main@rasterio.groups.io>
Date: Friday, August 14, 2020 at 7:03 PM
To: "main@rasterio.groups.io" <main@rasterio.groups.io>
Subject: Re: [rasterio] rasterio opens file from AWS S3 bucket on local machine, but can't find file when deployed to Google App Engine

 

Hi Judson,

 

On Wed, Aug 12, 2020 at 9:42 AM Judson Buescher <judson.buescher@...> wrote:

Hi Sean,

The only way that I’m installing the package is through my requirements.txt which I include in my deploy folder. I have

 

rasterio==1.1.5

 

in the requirements.txt. For the credentials I’ve tried a couple different ways. I’ve tried setting them using

 

os.environ['GS_SECRET_ACCESS_KEY'] = secret_access_key

os.environ['GS_ACCESS_KEY_ID'] = access_key

 

I’ve also tried

 

session = boto3.Session(aws_access_key_id=access_key,

                        aws_secret_access_key=secret_access_key)

 

and then using that session object when I open the raster file. Was there any other info you’d like?

Thanks a ton for the help,
Judson

 

If the source data for your App Engine app is on AWS, you'll need to make sure that you're providing AWS keys to access it.  You'll want

 

    AWS_SECRET_ACCESS_KEY = secret_access_key

    etc.

 

in your environment, not GS_SECRET_ACCESS_KEY (etc.). I've never used boto3 on App Engine and wouldn't recommend that approach of configuring AWS for GDAL and rasterio.

 

--

Sean Gillies
Re: Iterate over elements in Rasterio window and obtain coordinates using transform.xy

Sean Gillies
 

Hi,

On Wed, Aug 19, 2020 at 6:24 PM <whytefish1@...> wrote:

[Edited Message Follows]

Worked it out using a previous thread, doing it using an affine transform on the window.

Excellent!
 
I would however like to know if the rasterio.transform.xy method can be used when passing col, row from a dataset window, as I would like to get the coordinate for the center of the cell rather than the ul which results from the affine window transform.

Yes. The important detail is that the transform matrix passed to xy() must apply to the window. You can't use the matrix from the dataset's .transform attribute unless your window has its origin at the upper left corner of the dataset. Dataset objects have a window_transform() method that can help compute the new transforms: https://rasterio.readthedocs.io/en/latest/api/rasterio.io.html?highlight=window_transform#rasterio.io.DatasetReader.window_transform.

--
Sean Gillies
Re: Iterate over elements in Rasterio window and obtain coordinates using transform.xy

whytefish1@...
 
Edited

Worked it out using a previous thread, doing it using an affine transform on the window. 
I would however like to know if the rasterio.transform.xy method can be used when passing col, row from a dataset window, as I would like to get the coordinate for the center of the cell rather than the ul which results from the affine window transform.
Iterate over elements in Rasterio window and obtain coordinates using transform.xy

whytefish1@...
 

Does a window generated by rasterio maintain the dataset origin and absolute col/row indices?

I'd like to be able to iterate over a window of a dem dataset, find the highest value in the window and return it's lat long coordinates using rasterio.transform.xy or similar.

Obviously the below isn't suitable (the arr is invalid), it illustrates what I'm trying to achieve. Window is generated using slices and row/col high and lows which are provided by a valid function called extract_block.

with rasterio.open(inputraster) as dataset:
        for (x,y) in zip(*nonzero):
            row_low, row_high, col_low, col_high = extract_block(arr,[x,y],windowsize)
            rasteriowindowarr = dataset.read(1,window=Window.from_slices(slice(row_low, row_high), slice(col_low, col_high)))
            with rasterio.open(rasteriowindowarr) as windowarr:
                for i,j in np.ndindex(rasteriowindowarr.shape):
                    coordinatex,coordinatey = rasterio.transform.xy(windowarr.transform, i,j,offset='center')
                    print ("long of result is " + str(coordinatex))
                    print ("lat of result is " + str(coordinatey))
Re: How sum two raster with different shapes?

Amine Aboufirass <amine.aboufirass@...>
 

You can't sum rasters with different shapes. Resample your rasters to a common resolution and align them so that they can be read as arrays with the same shape, then add the two arrays together and write to a new raster.


On Tue, 18 Aug 2020, 19:03 , <byander@...> wrote:

Hi, I'm developing a system and I need sum two raster with different shapes.
I've a DEM raster with a resolution of 30 meters and another raster with a resolution of 10 meters, both clipped in the same area by a polygon.

When I trying sum the rasters I get the error: ValueError: operands could not be broadcast together with shapes.

I know that rasterio read raster as numpy array, but If I use the Raster Calculator from QGIS software, I can sum the rasters without problem.

I would like sum the rasters without use command line, e.g. rio calc or gdal_calc.py, but I not found a example to implement this.

Thank's for any help.
How sum two raster with different shapes?

Anderson Roberto da Silva
 

Hi, I'm developing a system and I need sum two raster with different shapes.
I've a DEM raster with a resolution of 30 meters and another raster with a resolution of 10 meters, both clipped in the same area by a polygon.

When I trying sum the rasters I get the error: ValueError: operands could not be broadcast together with shapes.

I know that rasterio read raster as numpy array, but If I use the Raster Calculator from QGIS software, I can sum the rasters without problem.

I would like sum the rasters without use command line, e.g. rio calc or gdal_calc.py, but I not found a example to implement this.

Thank's for any help.
Re: Automate process of creating a mask

Jaime Lopez
 

Hi, 

You can also use skimage package, it has a filters module with threshold_otsu.
Hope it helps,

JL

On Mon, Aug 17, 2020 at 3:47 AM <ts@...> wrote:
hi Jaime,

thanks for your reply. Do you know if there is Otsu segmentation in rasterio? Or should I use f.e. OpenCV?



--
Jaime Lopez Carvajal
Re: Automate process of creating a mask

ts@...
 

hi Jaime,

thanks for your reply. Do you know if there is Otsu segmentation in rasterio? Or should I use f.e. OpenCV?
Re: Automate process of creating a mask

Jaime Lopez
 

Hi, 

It seems to me that you can apply Otsu segmentation first to every image, to detect the threshold.
Then you can use the threshold as a mask's parameter.
Sorry, if I could misunderstood the point.

JL

On Sat, Aug 15, 2020 at 8:53 AM <ts@...> wrote:
Dear Rasterios,

beeing quite new to rasterio I'm looking for help regarding the following:

I've a large amount of 1 Channel Satellite imaginary. While the data contains no-data information around the actual motif nearly black areas do exist which I'd like to cleanup.


Following the docs masking by shapefile  It perfectly works to crop my image to the desired image area:



The red shape is quite inaccurate drawn just for the purpose of testing!

My question now is, how can I automate the creation of a shapefile to mask with based on the grey value or a defined border within my data? Having a closer look  on the edge of my desired image area my problem should get clear. The border is fuzzy, further the nearly black pixel value could exists in the desired motif.



I'm thankful if you could show me a direction to automate the creation of a shape mask (or geojson which should work as well from what I read at docs)!

– Toni



--
Jaime Lopez Carvajal
101 - 120 of 698  
More
Home Messages Hashtags Subgroups About Features Pricing Updates Terms Help Twitter