1. main@rasterio.groups.io
   Date   
Padding rasterio windows

ashnair0007@...
 

I need to tile a large geotiff into equally sized tiles (with/without overlap depending on user). While I've been able to do that, a problem arises in the edge windows wherein the edge windows have a shorter length/width. I understand why this happens but I would like the edge windows to have the same size as well. 

One way to do this might be get all the edge windows, pad them and reproject them so that padding doesn't mess with the geo-referencing.  Is this an appropriate approach?
cligj 0.6b1

Sean Gillies
 

Hi all,

Another rasterio-related announcement: I'm working on releasing version 0.6.0 of cligj next week. Here's the ticket about it https://github.com/mapbox/cligj/issues/28. This version fixes a bug and also announces a future change for the API. See https://github.com/mapbox/cligj/blob/master/CHANGES.txt for details.

Regards,

--
Sean Gillies
rio-mbtiles 1.5b2

Sean Gillies
 

Hi all,

I'm working on releasing version 1.5.0 of rio-mbtiles by the end of October and published a second beta today. See https://github.com/mapbox/rio-mbtiles/issues/61 and https://github.com/mapbox/rio-mbtiles/blob/master/CHANGES.txt if you're interested in this rasterio CLI plugin. Your feedback will be much appreciated!

Regards,

--
Sean Gillies
Re: Issues Setting tags for GRIB messsages when outputting to GRIB

Shane Mill - NOAA Affiliate
 

That was the issue. For anyone who outputs data to GRIB. Make sure you set the:

GRIB_PDS_TEMPLATE_NUMBERS
GRIB_IDS
GRIB_DISCIPLINE
Re: Issues Setting tags for GRIB messsages when outputting to GRIB

Shane Mill - NOAA Affiliate
 

Noticing that I may need to set the GRIB_DISCIPLINE. If I am able to do that through rasterio, I will be good to go. I will update this message momentarily
Issues Setting tags for GRIB messsages when outputting to GRIB

Shane Mill - NOAA Affiliate
 

Hello,

When writing data to GRIB files, I've noticed sometimes an inconsistency with setting the GRIB_PDS_TEMPLATE_NUMBERS. This parameter drives the parameter name for the GRIB message.

Below is a snippet of the code:

with rasterio.open('/mnt/data-api/output.grb', 'r+', driver='GRIB',DATA_ENCODING='SIMPLE_PACKING') as ds:
    for idx,t in enumerate(tags,start=0):
       ds.update_tags(idx+1,GRIB_PDS_TEMPLATE_NUMBERS=t['GRIB_PDS_TEMPLATE_NUMBERS'])
       ds.update_tags(idx+1,GRIB_IDS=t['GRIB_IDS'])

tags is a prepopulated list containing the "tags" for each grib message. I am simply updating the tags in the new GRIB file to contain the correct GRIB_PDS_TEMPLATE_NUMBERS and GRIB_IDS.

The problem- Sometimes, the tag is not properly set. 

An example: For the Grib parameterName "Soil Temperature", this is the corresponding tag:

{'GRIB_COMMENT': 'Soil temperature [C]', 'GRIB_DISCIPLINE': '2(Land_Surface)', 'GRIB_ELEMENT': 'TSOIL', 'GRIB_FORECAST_SECONDS': '0 sec', 'GRIB_IDS': 'CENTER=7(US-NCEP) SUBCENTER=0 MASTER_TABLE=2 LOCAL_TABLE=1 SIGNF_REF_TIME=1(Start_of_Forecast) REF_TIME=2020-09-29T00:00:00Z PROD_STATUS=0(Operational) TYPE=1(Forecast)', 'GRIB_PDS_PDTN': '0', 'GRIB_PDS_TEMPLATE_ASSEMBLED_VALUES': '0 2 2 0 81 0 0 1 0 106 0 1 106 2 200', 'GRIB_PDS_TEMPLATE_NUMBERS': '0 2 2 0 81 0 0 0 1 0 0 0 0 106 0 0 0 0 1 106 2 0 0 0 200', 'GRIB_REF_TIME': '1601337600 sec UTC', 'GRIB_SHORT_NAME': '1-2-DBLL', 'GRIB_UNIT': '[C]', 'GRIB_VALID_TIME': '1601337600 sec UTC'}

However, the outputted GRIB file has:

{'GRIB_COMMENT': 'Potential temperature [C]', 'GRIB_DISCIPLINE': '0(Meteorological)', 'GRIB_ELEMENT': 'POT', 'GRIB_FORECAST_SECONDS': '0 sec', 'GRIB_IDS': 'CENTER=7(US-NCEP) SUBCENTER=0 MASTER_TABLE=2 LOCAL_TABLE=0 SIGNF_REF_TIME=1(Start_of_Forecast) REF_TIME=2020-09-29T00:00:00Z PROD_STATUS=0(Operational) TYPE=1(Forecast)', 'GRIB_PDS_PDTN': '0', 'GRIB_PDS_TEMPLATE_ASSEMBLED_VALUES': '0 2 2 0 81 0 0 1 0 106 0 1 106 2 200', 'GRIB_PDS_TEMPLATE_NUMBERS': '0 2 2 0 81 0 0 0 1 0 0 0 0 106 0 0 0 0 1 106 2 0 0 0 200', 'GRIB_REF_TIME': '1601337600 sec UTC', 'GRIB_SHORT_NAME': '1-2-DBLL', 'GRIB_UNIT': '[C]', 'GRIB_VALID_TIME': '1601337600 sec UTC'}

Notice, it was intended to set to "Soil Temperature", but the output GRIB has "Potential Temperature". Do you know of a way to overcome this?
Rasterio 1.1.7

Sean Gillies
 

Hi all,

Version 1.1.7 is on PyPI now. Here are the changes: https://github.com/mapbox/rasterio/blob/master/CHANGES.txt#L10-L23. Thank you, Alan and Vincent!

--
Sean Gillies
Rasterio 1.1.6

Sean Gillies
 

Hi all,

Wheels for 1.1.6 are on PyPI now. There are a number of significant bug fixes: https://github.com/mapbox/rasterio/blob/master/CHANGES.txt#L10.

The manylinux1 and macosx wheels include GDAL 2.4.4, still, but with two new patches. I'm waiting for more comments in another thread before switching to GDAL 3.1 and PROJ 7. We are not yet building wheels for Python 3.9 or 3.10, sorry. That depends on some updates to the build infrastructure that I have no time for at the moment.

Thanks for all the great bug reports!

--
Sean Gillies
Re: Request for comment: rasterio wheels on PyPI and GDAL 3.1

Howard Butler
 



On Sep 13, 2020, at 4:21 PM, Sean Gillies <sean.gillies@...> wrote:

Hi all,

The rasterio 1.1.5 wheels on PyPI include GDAL 2.4.4 and a patched PROJ 4.9.3. Neither of these versions are supported anymore. Recently I have heard from users and contributors would like to see GDAL 3.1.x in the PyPI wheels soon, to benefit from new features, and also I have heard from those who would rather stick with 2.4.4 for a while yet due to increased latency noticed with the new dependency on proj.db in PROJ versions 6 and greater. I'm sympathetic to both arguments. Rasterio depends on some new GDAL bug fixes to really shine and be its best. On the other hand, I really don't know how best to distribute GDAL 3.1 and PROJ 7.1, particularly when it comes to including PROJ data in the wheels (as we have) or relying on the PROJ CDN.

I'd love to hear comments on whether rasterio wheels should switch over to GDAL 3.1 and PROJ 7.1, especially those based on experience with the latest PROJ in production in distributed and "serverless" systems.

As an instigator for the CDN and frequent grid shifter due to our use in the geodetic and lidar domain, I've been very happy with PROJ's CDN approach in Lambda scenarios. You can find my Dockerfile at https://github.com/PDAL/lambda and my latest public layer with GDAL 3.1.3 and PROJ 7.1.1 is at arn:aws:lambda:us-east-1:163178234892:layer:pdal:35

As a library packager, I agree the PROJ data is a burden. The wheel approach of Python even makes that burden more pronounced. I like Alan's pyproj approach of pointing the user at how to fetch the data if they need it, or flip on the network bit if they're ok with that. It seems like a good compromise and a packaging approach that is not that much different from the pre-6.0 days when most people ignored the grids entirely.

The latency complaint? Are you doing 100s of lookups per second (MapServer)? There may be more performance to get there, but someone is going to have to spend *a lot* of time to get it. I think the proj.db situation (one actual database across all the projects) is gobs better than before (at least three csv "databases" with different-but-same information in them and indeterminate  code paths for calculation of a definition). 

Howard
Re: Request for comment: rasterio wheels on PyPI and GDAL 3.1

Alan Snow
 

My first thought is that if you do make the switch, I think it would make sense to do it in the 1.2 release.

> I have heard from those who would rather stick with 2.4.4 for a while yet due to increased latency noticed with the new dependency on proj.db in PROJ versions 6 and greater.

There will always be the --no-binary option and they can install older versions of GDAL/PROJ. It is what we do at work.

> I really don't know how best to distribute GDAL 3.1 and PROJ 7.1, particularly when it comes to including PROJ data in the wheels (as we have) or relying on the PROJ CDN.

Feel free to use whatever you need in the transition from here: https://github.com/pyproj4/pyproj-wheels.

pyproj 3 has decided to not include any datum/transformation grids in the wheels and has a page for users to reference for how to download grids themselves: https://pyproj4.github.io/pyproj/latest/transformation_grids.html. This allows for a smaller wheel to download and allows users to only download the grids they need (useful for serverless applications).

I would also recommend waiting until PROJ 7.2 for wheels due to the CA Bundle path support needed for wheels (https://github.com/pyproj4/pyproj/issues/689).

On another note, I was unable to get manylinux1 wheels to build with the curl setup required, so manylinux2010 will be the new minimum. There is probably a winning combination, but I was unable to find it.
Request for comment: rasterio wheels on PyPI and GDAL 3.1

Sean Gillies
 

Hi all,

The rasterio 1.1.5 wheels on PyPI include GDAL 2.4.4 and a patched PROJ 4.9.3. Neither of these versions are supported anymore. Recently I have heard from users and contributors would like to see GDAL 3.1.x in the PyPI wheels soon, to benefit from new features, and also I have heard from those who would rather stick with 2.4.4 for a while yet due to increased latency noticed with the new dependency on proj.db in PROJ versions 6 and greater. I'm sympathetic to both arguments. Rasterio depends on some new GDAL bug fixes to really shine and be its best. On the other hand, I really don't know how best to distribute GDAL 3.1 and PROJ 7.1, particularly when it comes to including PROJ data in the wheels (as we have) or relying on the PROJ CDN.

I'd love to hear comments on whether rasterio wheels should switch over to GDAL 3.1 and PROJ 7.1, especially those based on experience with the latest PROJ in production in distributed and "serverless" systems.

Thanks!

--
Sean Gillies
Re: Different values when I use a window

adrianocorbelinoii@...
 

I asked the same question on stack exchange and now I understand how to obtain the results that once I expected.
But I think the answer for that is a little bit in the dark, should we improve the documentation to have an example that explain this?
Re: Different values when I use a window

adrianocorbelinoii@...
 

Hello, this is the example:

import rasterio
#data link: https://earthexplorer.usgs.gov/download/external/options/SENTINEL_2A/3022286/INVSVC/
 
pathToImgFolder = ""
pathData = pathToImgFolder + "L1C_T21LXC_A001666_20170701T140052\S2B_MSIL1C_20170701T140049_N0205_R067_T21LXC_20170701T140052.SAFE\GRANULE\L1C_T21LXC_A001666_20170701T140052\IMG_DATA\T21LXC_20170701T140049_B01.jp2" 
boudingBoxCoordinates = (606859.0750363453, 8241169.269917269, 607219.0750363453, 8241529.269917269)
points = [(607014.0750363453,8241374.26991727),(607024.0750363444,8241374.26991727),\
          (607034.0750363453,8241374.26991727),(607044.0750363453,8241374.26991727),\
          (607054.0750363442,8241374.26991727)]
 
with rasterio.open(pathData) as img:
    bandWindow = rasterio.windows.from_bounds(*boudingBoxCoordinates, img.transform)
    winTransform = rasterio.windows.transform(bandWindow,img.transform)
    bandData = img.read(1, window = bandWindow)
    allData = img.read(1)
    for point in points:
        rBand,cBand = rasterio.transform.rowcol(winTransform,point[0],point[1])
        rFull,cFull = img.index(point[0],point[1])
        bandVal = bandData[rBand,cBand]
        fullVal = allData[rFull,cFull]
        print(f"{fullVal} {'=' if bandVal == fullVal else '!='} {bandVal}")

When I execute this example i get this result:

1202 = 1202
1330 != 1202
1330 != 1202
1330 = 1330
1330 = 1330
I want to understand what I am doing wrong.
Re: GDAL doesn't include file gcs.csv anymore

pierrick.rambaud49@...
 

Thank you for your quick answer,

I'm working on a production cloud company service so I don't have control on the way libs are installed. I can just ask the people in charge and they reply that they installed it with apt. 


in some notebook if I del the os.environ['GDAL_DATA'] at the start it mange to find back the gcs.csv file. Problem is: if I then use a gdal function it's the other way around. 
Is there a way to make them both compatible with gdal 3.0.4 as suggested in the README ? 
Re: GDAL doesn't include file gcs.csv anymore

Sean Gillies
 

Hi,

On Mon, Sep 7, 2020 at 8:18 AM <pierrick.rambaud49@...> wrote:

description 

 
I'm using [geemap](https://github.com/giswqs/geemap) to display a raster on a created map. 
This lib use `xarray_leaflet` to display the raster and this lib will end up using `rasterio` to manipulate the .tif file. 
 
When I launch my display :

m = geemap.Map()
m.add_raster(clip_map, colormap='terrain', layer_name='gfc')
 
I get the following error :
> CRSError: Unable to open EPSG support file gcs.csv.  Try setting the GDAL_DATA environment variable to point to the directory containing EPSG csv files.
 
This error is everywhere on SO so I tried to verify if my GDAL_DATA env variable was coorectly set : 
 
import os
import stat
gdal_data = os.environ['GDAL_DATA']
print('is dir: ' + str(os.path.isdir(gdal_data)))
gcs_csv = os.path.join(gdal_data, 'gcs.csv')
print('is file: ' + str(os.path.isfile(gcs_csv)))
st = os.stat(gcs_csv)
print('is readable: ' + str(bool(st.st_mode & stat.S_IRGRP)))
 
# out 
# is dir: True
#is file: False
#FileNotFoundError: [Errno 2] No such file or directory: '/usr/share/gdal/gcs.csv'
 
going to the glad distrib inb the `NEWS` file, I read that they removed lots of file in 3.0 including `gcs.csv`. So it's no longer included in my folder. 
 
Is there a workaround ? 


setup 

rasterio==1.1.5
python==3.6
geemap==0.7.9
gdal==3.0.4

How did you install rasterio? If you got it with `pip install rasterio` you will have installed a wheel file that includes GDAL 2.4.4 and data files and which will be incompatible with the GDAL 3.0.4 installed on your system. If that is the case, you should unset GDAL_DATA. Rasterio in the wheel will detect its own data files and does not need GDAL_DATA to be set.

--
Sean Gillies
Re: Status of GDAL 3.1 Support?

Sean Gillies
 

Hi Daryl,

On Mon, Sep 7, 2020 at 6:01 AM Herzmann, Daryl E [AGRON] <akrherz@...> wrote:
Howdy,

I am curious about rasterio's current GDAL version support?  Presently, my conda environment is pinned down to GDAL < 3.1

https://github.com/conda-forge/rasterio-feedstock/pull/171#issuecomment-678631972

As I search around on rasterio's github, I find this PR which seems to pass tests against newer GDALs?

https://github.com/mapbox/rasterio/pull/1987

Is there some GH issue tracking GDAL 3.1 support or should 1.1.5 rasterio work with GDAL 3.1?

thanks!
daryl

We're testing rasterio's trunk against GDAL 3.1.2 here: https://travis-ci.org/github/mapbox/rasterio. We are not currently checking the 1.1.x branch, but in my experience there are no problems. I have rasterio 1.1.5, GDAL 3.1.2, and PROJ 7.1.0 in production.

--
Sean Gillies
GDAL doesn't include file gcs.csv anymore

pierrick.rambaud49@...
 

description 

 
I'm using [geemap](https://github.com/giswqs/geemap) to display a raster on a created map. 
This lib use `xarray_leaflet` to display the raster and this lib will end up using `rasterio` to manipulate the .tif file. 
 
When I launch my display :

m = geemap.Map()
m.add_raster(clip_map, colormap='terrain', layer_name='gfc')
 
I get the following error :
> CRSError: Unable to open EPSG support file gcs.csv.  Try setting the GDAL_DATA environment variable to point to the directory containing EPSG csv files.
 
This error is everywhere on SO so I tried to verify if my GDAL_DATA env variable was coorectly set : 
 
import os
import stat
gdal_data = os.environ['GDAL_DATA']
print('is dir: ' + str(os.path.isdir(gdal_data)))
gcs_csv = os.path.join(gdal_data, 'gcs.csv')
print('is file: ' + str(os.path.isfile(gcs_csv)))
st = os.stat(gcs_csv)
print('is readable: ' + str(bool(st.st_mode & stat.S_IRGRP)))
 
# out 
# is dir: True
#is file: False
#FileNotFoundError: [Errno 2] No such file or directory: '/usr/share/gdal/gcs.csv'
 
going to the glad distrib inb the `NEWS` file, I read that they removed lots of file in 3.0 including `gcs.csv`. So it's no longer included in my folder. 
 
Is there a workaround ? 


setup 

rasterio==1.1.5
python==3.6
geemap==0.7.9
gdal==3.0.4
 
Status of GDAL 3.1 Support?

Herzmann, Daryl E [AGRON]
 

Howdy,

I am curious about rasterio's current GDAL version support? Presently, my conda environment is pinned down to GDAL < 3.1

https://github.com/conda-forge/rasterio-feedstock/pull/171#issuecomment-678631972

As I search around on rasterio's github, I find this PR which seems to pass tests against newer GDALs?

https://github.com/mapbox/rasterio/pull/1987

Is there some GH issue tracking GDAL 3.1 support or should 1.1.5 rasterio work with GDAL 3.1?

thanks!
daryl
Re: Black border after running rio mask

ts@...
 

Thanks for your help Sean,

I can confirm the black artifacts come from jpeg compression.
I also found this answered which explains the problem in detail:
https://gis.stackexchange.com/a/114453/52871

best regards,

Toni
Re: Black border after running rio mask

Sean Gillies
 

Hi,

On Sat, Aug 29, 2020 at 8:52 AM <ts@...> wrote:
Dear Rasterio developers and useers,

I' masking geotiff files with geojson polygons:

rio mask --overwrite --crop in.tif out.tif --geojson-mask mask.geojson 

Unfortunately after doing so black artificats showing up around the cropped area.



Is there a way to avoid this artifacts?


It looks like your input.tif may have lossy compression applied, and the output will have the same compression by default. You might try

rio mask --overwrite --crop in.tif out.tif --geojson-mask mask.geojson --co COMPRESS=NONE

--
Sean Gillies