Date   
Re: Output to grib2?

Shane Mill - NOAA Affiliate
 

Okay so I think that I can update the GRIB specific options using rio warp

ie: rio warp 1000_500_thick.grb 1000_500_thick_new2.grb --format GRIB --profile BAND_1_PDS_PDTN=

with BAND_1_PDS_PDTN being a GRIB specific option. I'm going to explore this more today.

Thanks,
Shane


On Tue, Mar 19, 2019 at 3:50 PM Shane Mill - NOAA Affiliate via Groups.Io <shane.mill=noaa.gov@groups.io> wrote:
Hey Sean,

I know that you were saying that you don't use grib2 often, but I wanted to follow up with the previous question.

I am able to output the numpy array to GRIB. Say that you have one band, you can read the tags of that one band with src.tags(1). These tags will be the defaults that come from:
https://gdal.org/frmt_grib.html (if you scroll down to the section "GRIB2 Write Support"). The problem with this is that for the grib2 to be readable, you need to be able to update these tags.

I'm wondering how to change the tags either when originally creating the GRIB or after the fact when using "update_tags" explained here: https://github.com/mapbox/rasterio/blob/master/docs/topics/tags.rst
It appears that update_tags works fine for GTiff but not for GRIB. 

Let me know if you have any thoughts on this. Thanks as always!

-Shane

On Fri, Mar 15, 2019 at 2:16 PM Shane Mill - NOAA Affiliate via Groups.Io <shane.mill=noaa.gov@groups.io> wrote:
On Fri, Mar 15, 2019 at 11:02 AM, Sean Gillies wrote:
rio convert -f GRIB ~/code/rasterio/tests/data/RGB.byte.tif /tmp/rgb.grb2


Sean, thanks for the quick response! You are correct, that does work. I think my issue may have been that I was using gdalinfo to check the resulting grib2 file, and my gdal version was older. When I check with rio info it seems to be correct.

Thanks again!

-Shane



--
Shane Mill

Meteorological Application Developer, AceInfo Solutions

Meteorological Development Laboratory (NWS)

Phone: 301-427-9452



--
Shane Mill

Meteorological Application Developer, AceInfo Solutions

Meteorological Development Laboratory (NWS)

Phone: 301-427-9452

issue with opening/closing datasets

Amine Aboufirass
 

Hi All, 

I just listed an issue on the main github log. https://github.com/mapbox/rasterio/issues/1659

If anyone could take a look I would be very grateful.

Kind Regards,

Amine

Re: Output to grib2?

Sean Gillies
 

Hi Shane,

I think I understand now. BAND_1_PDS_PDTN isn't a tag, it's a "creation option". You can apply these at the time you open a dataset for writing like

with rasterio.open("file.grb2", "w", driver="GRIB", ...., BAND_1_PDS_PDTN=something) as dataset:
    dataset.write(data)

On Fri, Mar 22, 2019 at 9:06 AM Shane Mill - NOAA Affiliate via Groups.Io <shane.mill=noaa.gov@groups.io> wrote:
Okay so I think that I can update the GRIB specific options using rio warp

ie: rio warp 1000_500_thick.grb 1000_500_thick_new2.grb --format GRIB --profile BAND_1_PDS_PDTN=

with BAND_1_PDS_PDTN being a GRIB specific option. I'm going to explore this more today.

Thanks,
Shane

On Tue, Mar 19, 2019 at 3:50 PM Shane Mill - NOAA Affiliate via Groups.Io <shane.mill=noaa.gov@groups.io> wrote:
Hey Sean,

I know that you were saying that you don't use grib2 often, but I wanted to follow up with the previous question.

I am able to output the numpy array to GRIB. Say that you have one band, you can read the tags of that one band with src.tags(1). These tags will be the defaults that come from:
https://gdal.org/frmt_grib.html (if you scroll down to the section "GRIB2 Write Support"). The problem with this is that for the grib2 to be readable, you need to be able to update these tags.

I'm wondering how to change the tags either when originally creating the GRIB or after the fact when using "update_tags" explained here: https://github.com/mapbox/rasterio/blob/master/docs/topics/tags.rst
It appears that update_tags works fine for GTiff but not for GRIB. 

Let me know if you have any thoughts on this. Thanks as always!

-Shane

On Fri, Mar 15, 2019 at 2:16 PM Shane Mill - NOAA Affiliate via Groups.Io <shane.mill=noaa.gov@groups.io> wrote:
On Fri, Mar 15, 2019 at 11:02 AM, Sean Gillies wrote:
rio convert -f GRIB ~/code/rasterio/tests/data/RGB.byte.tif /tmp/rgb.grb2


Sean, thanks for the quick response! You are correct, that does work. I think my issue may have been that I was using gdalinfo to check the resulting grib2 file, and my gdal version was older. When I check with rio info it seems to be correct.

Thanks again!

-Shane



--
Shane Mill

Meteorological Application Developer, AceInfo Solutions

Meteorological Development Laboratory (NWS)

Phone: 301-427-9452



--
Shane Mill

Meteorological Application Developer, AceInfo Solutions

Meteorological Development Laboratory (NWS)

Phone: 301-427-9452



--
Sean Gillies

Re: Output to grib2?

Shane Mill - NOAA Affiliate
 

Ah I see, I was confused then just because these come up when you "rio insp" and then "src.tags(1)", these creation options show up under the tags for band1.

That's very helpful so thanks for the response!

Shane

On Fri, Mar 22, 2019 at 12:09 PM Sean Gillies <sean.gillies@...> wrote:
Hi Shane,

I think I understand now. BAND_1_PDS_PDTN isn't a tag, it's a "creation option". You can apply these at the time you open a dataset for writing like

with rasterio.open("file.grb2", "w", driver="GRIB", ...., BAND_1_PDS_PDTN=something) as dataset:
    dataset.write(data)

On Fri, Mar 22, 2019 at 9:06 AM Shane Mill - NOAA Affiliate via Groups.Io <shane.mill=noaa.gov@groups.io> wrote:
Okay so I think that I can update the GRIB specific options using rio warp

ie: rio warp 1000_500_thick.grb 1000_500_thick_new2.grb --format GRIB --profile BAND_1_PDS_PDTN=

with BAND_1_PDS_PDTN being a GRIB specific option. I'm going to explore this more today.

Thanks,
Shane

On Tue, Mar 19, 2019 at 3:50 PM Shane Mill - NOAA Affiliate via Groups.Io <shane.mill=noaa.gov@groups.io> wrote:
Hey Sean,

I know that you were saying that you don't use grib2 often, but I wanted to follow up with the previous question.

I am able to output the numpy array to GRIB. Say that you have one band, you can read the tags of that one band with src.tags(1). These tags will be the defaults that come from:
https://gdal.org/frmt_grib.html (if you scroll down to the section "GRIB2 Write Support"). The problem with this is that for the grib2 to be readable, you need to be able to update these tags.

I'm wondering how to change the tags either when originally creating the GRIB or after the fact when using "update_tags" explained here: https://github.com/mapbox/rasterio/blob/master/docs/topics/tags.rst
It appears that update_tags works fine for GTiff but not for GRIB. 

Let me know if you have any thoughts on this. Thanks as always!

-Shane

On Fri, Mar 15, 2019 at 2:16 PM Shane Mill - NOAA Affiliate via Groups.Io <shane.mill=noaa.gov@groups.io> wrote:
On Fri, Mar 15, 2019 at 11:02 AM, Sean Gillies wrote:
rio convert -f GRIB ~/code/rasterio/tests/data/RGB.byte.tif /tmp/rgb.grb2


Sean, thanks for the quick response! You are correct, that does work. I think my issue may have been that I was using gdalinfo to check the resulting grib2 file, and my gdal version was older. When I check with rio info it seems to be correct.

Thanks again!

-Shane



--
Shane Mill

Meteorological Application Developer, AceInfo Solutions

Meteorological Development Laboratory (NWS)

Phone: 301-427-9452



--
Shane Mill

Meteorological Application Developer, AceInfo Solutions

Meteorological Development Laboratory (NWS)

Phone: 301-427-9452



--
Sean Gillies



--
Shane Mill

Meteorological Application Developer, AceInfo Solutions

Meteorological Development Laboratory (NWS)

Phone: 301-427-9452

Re: issue with opening/closing datasets

Sean Gillies
 

Hi Amine,

I think you have made in error in pasting code into the GitHub issue. The code you've given will fail at dataset = memfile.open because you haven't assigned memfile yet.

The message you see printed comes straight from the GDAL library. You haven't configured any GDAL error or log message handler and so the messages go directly to your terminal. Message handlers are configured if you run your code within a `with rasterio.Env()` block.

    import rasterio

    with rasterio.Env():
        # your code here

Also if you do

    with memfile.open(...) as dataset:

you won't see this message.


On Fri, Mar 22, 2019 at 9:25 AM Amine Aboufirass <amine.aboufirass@...> wrote:
Hi All, 

I just listed an issue on the main github log. https://github.com/mapbox/rasterio/issues/1659

If anyone could take a look I would be very grateful.

Kind Regards,

Amine



--
Sean Gillies

Cloud-optimized GDAL VRT

mail@...
 

We were discussing whether it would be possible to use `gdalbuildvrt` on several cloud-optimized GeoTIFF and obtain something that behaves like one big COG (which would be awesome to serve up large collections of GeoTIFFS, e.g. via Terracotta). `rasterio` seems to be able to read GDAL VRTs just fine, but it does not seem to have access to the overviews of the datasets. Is there anything obvious we could try? Is this missing functionality in GDAL or Rasterio?

See also https://github.com/DHI-GRAS/terracotta/issues/10

Re: Cloud-optimized GDAL VRT

vincent.sarago@...
 

Hi Dion,
I encountered the same result couple month ago and couldn't get it to work. I asked Evan about it and see his answer here https://lists.osgeo.org/pipermail/gdal-dev/2018-October/049200.html

Vincent 

Re: Cloud-optimized GDAL VRT

mail@...
 

Huh, interesting. So only birds-eye overviews of VRTs should be expensive, which is to expected. Did you observe that in your tests? Did you ever go further and experiment with external overviews for the VRT?

Best,
Dion

Re: issue with opening/closing datasets

Amine Aboufirass
 

Hi Sean, the issue is that I am writing functions where the output is often a rasterio dataset. I don't know if this can be accomplished using a with statement:
function open_raster(filename):
    rasterio_dataset_object = rasterio.open(filename)
    return rasterio_dataset_object

function do_stuff_with_raster(rasterio_dataset_object):
    ###do stuff with raster
    return rasterio_dataset_object

dataset = open_raster('raster.tif')
new_raster = do_stuff_with_raster(dataset)
new_raster.close() 

Thanks,
Amine

On Fri, Mar 22, 2019 at 5:25 PM Sean Gillies <sean.gillies@...> wrote:
Hi Amine,

I think you have made in error in pasting code into the GitHub issue. The code you've given will fail at dataset = memfile.open because you haven't assigned memfile yet.

The message you see printed comes straight from the GDAL library. You haven't configured any GDAL error or log message handler and so the messages go directly to your terminal. Message handlers are configured if you run your code within a `with rasterio.Env()` block.

    import rasterio

    with rasterio.Env():
        # your code here

Also if you do

    with memfile.open(...) as dataset:

you won't see this message.


On Fri, Mar 22, 2019 at 9:25 AM Amine Aboufirass <amine.aboufirass@...> wrote:
Hi All, 

I just listed an issue on the main github log. https://github.com/mapbox/rasterio/issues/1659

If anyone could take a look I would be very grateful.

Kind Regards,

Amine



--
Sean Gillies

cannot find API reference anymore

Amine Aboufirass
 

Dear All, 

I am quite sure that there used to be extensive online documentation in the following website:

https://rasterio.readthedocs.io/en/stable/api/rasterio.transform.html

What happened to it? Why has it been deleted and will it be brought back at some point? 

Regards,

Amine

Re: issue with opening/closing datasets

Sean Gillies
 

Hi Amine,

On Tue, Mar 26, 2019 at 5:10 AM Amine Aboufirass <amine.aboufirass@...> wrote:
Hi Sean, the issue is that I am writing functions where the output is often a rasterio dataset. I don't know if this can be accomplished using a with statement:
function open_raster(filename):
    rasterio_dataset_object = rasterio.open(filename)
    return rasterio_dataset_object

function do_stuff_with_raster(rasterio_dataset_object):
    ###do stuff with raster
    return rasterio_dataset_object

dataset = open_raster('raster.tif')
new_raster = do_stuff_with_raster(dataset)
new_raster.close() 

Thanks,
Amine

The following will work if you are calling the two functions in the same module.

with open_raster('raster.tif') as dataset:  # this gives you an implicit Env around the contained statements.
    new_raster = do_stuff_with_raster(dataset)
    new_raster.close()

 
On Fri, Mar 22, 2019 at 5:25 PM Sean Gillies <sean.gillies@...> wrote:
Hi Amine,

I think you have made in error in pasting code into the GitHub issue. The code you've given will fail at dataset = memfile.open because you haven't assigned memfile yet.

The message you see printed comes straight from the GDAL library. You haven't configured any GDAL error or log message handler and so the messages go directly to your terminal. Message handlers are configured if you run your code within a `with rasterio.Env()` block.

    import rasterio

    with rasterio.Env():
        # your code here

Also if you do

    with memfile.open(...) as dataset:

you won't see this message.


On Fri, Mar 22, 2019 at 9:25 AM Amine Aboufirass <amine.aboufirass@...> wrote:
Hi All, 

I just listed an issue on the main github log. https://github.com/mapbox/rasterio/issues/1659

If anyone could take a look I would be very grateful.

Kind Regards,

Amine


_._,_._

--
Sean Gillies

Re: issue with opening/closing datasets

Amine Aboufirass
 

Dear Sean,

I am slightly confused. You state the following:

The following will work if you are calling the two functions in the same module.
with open_raster('raster.tif') as dataset:  # this gives you an implicit Env around the contained statements.
    new_raster = do_stuff_with_raster(dataset)
    new_raster.close()

I have two issues with the above statement:
  •  Why are you closing the file if it is inside the with block? I thought one of the advantages of using the with  block is that files are closed implicitly?
  • This doesn't completely answer my question, since I would like to return  the file read object and pass it from function to function so that I can pry it open in each function and read/write the necessary values to and from it.
To be more explicit, can I do something like this?:

def open_dataset(filename):
    with rasterio.open(filename) as dataset:
        return dataset

def do_stuff_to dataset(dataset):
    dataset.write()
    return modified_dataset

def get_info_from_dataset(dataset):
    dataset.information
    return information

If so, then what are the disadvantages of using the with block inside a function and returning the object to be used outside? Is this good practice? If not what is the recommended way to write functions which use the rasterio library? This also extends to with blocks containing rasterio.Env(). Should I nest the with statement inside the function as stated above?

I ask because I would like to avoid writing a dataset to physical file until I am done modifying it.  For instance, geopandas uses the GeoDataFrame construct which is stored in memory, and not attached to any physical file. Memory files in rasterio come close, but they are still attached to a temporary file.

It seems that in rasterio defining a raster object must be via 3 disjoint entities (numpy array, affine transform and CRS ). It would be nice to have one object which groups all these entities and is somehow detached from physical/temporary files. A sort of glorified numpy array with metadata (a "GeoNumpy" array), just like geopandas glorifies the pandas Dataframe with metadata.

Of course this is just a naive (but hopefully constructive) suggestion, and perhaps also due to the fact that I do not completely understand how the library works :). 

Regards,

Amine




On Tue, Mar 26, 2019 at 2:56 PM Sean Gillies <sean.gillies@...> wrote:
Hi Amine,

On Tue, Mar 26, 2019 at 5:10 AM Amine Aboufirass <amine.aboufirass@...> wrote:
Hi Sean, the issue is that I am writing functions where the output is often a rasterio dataset. I don't know if this can be accomplished using a with statement:
function open_raster(filename):
    rasterio_dataset_object = rasterio.open(filename)
    return rasterio_dataset_object

function do_stuff_with_raster(rasterio_dataset_object):
    ###do stuff with raster
    return rasterio_dataset_object

dataset = open_raster('raster.tif')
new_raster = do_stuff_with_raster(dataset)
new_raster.close() 

Thanks,
Amine

The following will work if you are calling the two functions in the same module.

with open_raster('raster.tif') as dataset:  # this gives you an implicit Env around the contained statements.
    new_raster = do_stuff_with_raster(dataset)
    new_raster.close()

 
On Fri, Mar 22, 2019 at 5:25 PM Sean Gillies <sean.gillies@...> wrote:
Hi Amine,

I think you have made in error in pasting code into the GitHub issue. The code you've given will fail at dataset = memfile.open because you haven't assigned memfile yet.

The message you see printed comes straight from the GDAL library. You haven't configured any GDAL error or log message handler and so the messages go directly to your terminal. Message handlers are configured if you run your code within a `with rasterio.Env()` block.

    import rasterio

    with rasterio.Env():
        # your code here

Also if you do

    with memfile.open(...) as dataset:

you won't see this message.


On Fri, Mar 22, 2019 at 9:25 AM Amine Aboufirass <amine.aboufirass@...> wrote:
Hi All, 

I just listed an issue on the main github log. https://github.com/mapbox/rasterio/issues/1659

If anyone could take a look I would be very grateful.

Kind Regards,

Amine


_._,_._

--
Sean Gillies

Re: issue with opening/closing datasets

Sean Gillies
 

Amine,

On Wed, Mar 27, 2019 at 11:24 AM Amine Aboufirass <amine.aboufirass@...> wrote:
Dear Sean,

I am slightly confused. You state the following:

The following will work if you are calling the two functions in the same module.
with open_raster('raster.tif') as dataset:  # this gives you an implicit Env around the contained statements.
    new_raster = do_stuff_with_raster(dataset)
    new_raster.close()

I have two issues with the above statement:
  •  Why are you closing the file if it is inside the with block? I thought one of the advantages of using the with  block is that files are closed implicitly?
  • This doesn't completely answer my question, since I would like to return  the file read object and pass it from function to function so that I can pry it open in each function and read/write the necessary values to and from it.
To be more explicit, can I do something like this?:

def open_dataset(filename):
    with rasterio.open(filename) as dataset:
        return dataset

def do_stuff_to dataset(dataset):
    dataset.write()
    return modified_dataset

def get_info_from_dataset(dataset):
    dataset.information
    return information

If so, then what are the disadvantages of using the with block inside a function and returning the object to be used outside? Is this good practice? If not what is the recommended way to write functions which use the rasterio library? This also extends to with blocks containing rasterio.Env(). Should I nest the with statement inside the function as stated above?

I'm sorry about the confusion. I would rather not comment on the structure of your application. Your original question was about the warnings being printed in your shell, yes? I pointed out that in the absence of a custom error/warning handler, GDAL prints these directly to your shell. Rasterio does not register any custom handlers when you import it because I want to avoid import side effects that complicate testing of rasterio's modules.

The rasterio.env.Env class does register a custom error/warning handler when its __enter__() method is called. Within a `with Env():` block you should not see anything printed to the shell: messages will go to Python's logger instead, or be turned into Python exceptions.

I recommend ensuring that there is an activated Env within your function. You could do this by putting `with rasterio.env.Env():` at the top of the functions, or by using a decorator. There are examples of each in the rasterio code. The rasterio.open function itself is so decorated and you could reuse that decorator, it is part of the public API.
 

I ask because I would like to avoid writing a dataset to physical file until I am done modifying it.  For instance, geopandas uses the GeoDataFrame construct which is stored in memory, and not attached to any physical file. Memory files in rasterio come close, but they are still attached to a temporary file.

It seems that in rasterio defining a raster object must be via 3 disjoint entities (numpy array, affine transform and CRS ). It would be nice to have one object which groups all these entities and is somehow detached from physical/temporary files. A sort of glorified numpy array with metadata (a "GeoNumpy" array), just like geopandas glorifies the pandas Dataframe with metadata.

Of course this is just a naive (but hopefully constructive) suggestion, and perhaps also due to the fact that I do not completely understand how the library works :). 

Regards,

Amine

There is a file in MemoryFile, yes, but it is a formatted file in memory, not on disk. See https://www.gdal.org/gdal_virtual_file_systems.html#gdal_virtual_file_systems_vsimem for a brief explanation.

Other programmers have shown interest in a "GeoNumpy" class, and I've seen at least one project like this on GitHub. Georaster, I think. Rasterio doesn't provide such a class and I'm not ready to add one at this time. I believe t's better for Rasterio to focus on reading and writing formatted datasets and leave application-specific classes up to application developers.

--
Sean Gillies

Re: cannot find API reference anymore

Sean Gillies
 

This issue has been resolved. Check the project issue tracker's recently closed issues for details.

Rewriting uint16 headers with rasterio / applying rio color makes them unreadable by Preview, Photoshop

Edward Boyda
 


Hi all, I'm new at this, more of a computer vision person than a developer, so please bear with me....

I see the behavior I'm about to describe running pip-installed rasterio (1.0.22) on my Mac (OSX Mojave; homebrewed python and gdal) and also running rasterio (1.0.22) in a dockerized Ubuntu platform, on images from a variety of sources (DigitalGlobe, Planet, Landsat). 

Example 1: 
$ rio color -j 1 uint16_image.tif uint16_brightened.tif gamma RGB 1.5

When I try to open the output file, uint16_brightened.tif, in Preview or Photoshop, I get a message like "Could not complete your request because of a problem parsing the TIFF file." (That's from Photoshop; Preview is equivalent.) 

Example 2:  
$ rio color -j -1 uint16_image.tif uint16_brightened_v2.tif gamma RGB 1.5

The number of cores has changed from the first example. Now the output, uint16_brightened_v2.tif, is readable by Photoshop but has had its color interpretation changed to (reading from rasterio):

(<ColorInterp.gray: 1>, <ColorInterp.undefined: 0>, <ColorInterp.undefined: 0>)

When I open the file with Preview or view the thumbnail with Mac Finder, there are dark vertical lines interspersed with the actual pixels, and about a third of the original pixels have been pushed out of the frame. See screenshot attached.

Example 3:
I take a file that has a (gray, undefined, undefined) color interpretation and try to change that to RGB, now in the interpreter:
>>> with rasterio.open('uint16_noCI.tif', 'r+') as f:
    f.colorinterp = (ColorInterp.red, ColorInterp.green, ColorInterp.blue)

Again the edited file is unreadable by Preview and Photoshop. 

A couple of caveats:

1) I can read the data from files output from any of the above examples with rasterio or skimage, and the resulting numpy array is uncorrputed. I can resave it with skimage, show it with matplolib, etc., and the image looks as expected.
2) With any of the above outputs, I can read and then rewrite a new_image.tif, using rasterio, and the resulting files open as expected with Photoshop and Preview. This is my current (obviously inefficient) workaround:

with rasterio.open('uint16_brightened.tif') as f:
    prof = f.profile
    img = f.read()

with rasterio.open('new_image.tif', 'w', photometric='rgb', **prof) as f:
    f.write(img)

As far as I know these failures happen only with uint16 images (at least not with uint8), and it would seem to have to do with the way color interpretation is written into the headers via the different write mechanisms.  Has anyone come across similar behavior? I've been reproducing and beating my head against this for months and would really appreciate a sanity check.

Since the Docker container is likely cleaner than what I've installed on my Mac, I've run the tests for the attached output there. Here is the Dockerfile and some possibly relevant release details: 

FROM ubuntu:latest

RUN apt-get update && apt-get install -y software-properties-common

RUN apt-get install -y python3-pip python3-dev build-essential

RUN pip3 install --upgrade pip

RUN apt-get install -y gdal-bin libgdal-dev python3-gdal

RUN apt-get install -y libssl-dev libffi-dev libcurl4-openssl-dev

ENV DEBIAN_FRONTEND noninteractive

RUN apt-get install -y python3-tk


ADD ./requirements.txt /tmp/requirements.txt

RUN pip install -r /tmp/requirements.txt


---

DISTRIB_ID=Ubuntu

DISTRIB_RELEASE=18.04

python 3.6.6

pip 18.1 from /usr/local/lib/python3.6/dist-packages/pip (python 3.6)


affine==2.2.2

gbdx-auth==0.4.0

gbdxtools==0.16.0

GDAL==2.2.2

rasterio==1.0.22

rio-color==1.0.0

rio-mucho==1.0.0

Shapely==1.6.4

tifffile==2018.11.28



Thanks everyone!


Ed

Rasterio and PROJ.6 ?

vincent.sarago@...
 

Dear Rasterio contributors ;-) 

I installed the latest version of GDAL 2.4.1 with PROJ4 version 6 and I'm seeing this message "proj_create: init=epsg:/init=IGNF: syntax not supported in non-PROJ4 emulation mode" when ever I use rasterio.
- Is this expected
- should I revert to GDAL 2.4.0 ?
- Does rasterio support PROJ version 6 ? 

$ rio --version
proj_create: init=epsg:/init=IGNF: syntax not supported in non-PROJ4 emulation mode
1.0.22
$ gdalinfo --version
GDAL 2.4.1, released 2019/03/15

$ proj --version
Rel. 6.0.0, March 1st, 2019

Thanks 
 

Re: Rasterio and PROJ.6 ?

vincent.sarago@...
 

I reinstalled rasterio and it fixed itself. I think it was caused by environment variables

Re: Rasterio and PROJ.6 ?

Sean Gillies
 

Vincent,

We haven't been testing with PROJ 6. It might work.

This commit https://github.com/mapbox/rasterio/commit/c5b0d911994ade823a7b5a969071c2c1bd860a11 should address the error message printed to your terminal.


On Fri, Mar 29, 2019 at 2:30 PM <vincent.sarago@...> wrote:
Dear Rasterio contributors ;-) 

I installed the latest version of GDAL 2.4.1 with PROJ4 version 6 and I'm seeing this message "proj_create: init=epsg:/init=IGNF: syntax not supported in non-PROJ4 emulation mode" when ever I use rasterio.
- Is this expected
- should I revert to GDAL 2.4.0 ?
- Does rasterio support PROJ version 6 ? 

$ rio --version
proj_create: init=epsg:/init=IGNF: syntax not supported in non-PROJ4 emulation mode
1.0.22
$ gdalinfo --version
GDAL 2.4.1, released 2019/03/15

$ proj --version
Rel. 6.0.0, March 1st, 2019

Thanks 
 



--
Sean Gillies

Adding internal API reference links to the narrative docs

Sean Gillies
 

Hi all,

In the Rasterio quickstart guide I've added some links to API documentation. For example, see the "open()" link under the 2nd code block in https://rasterio.readthedocs.io/en/latest/quickstart.html. I did this with the following ReST markup: :func:`~rasterio.open`. Sphinx has already indexed where rasterio.open is defined in the docs and completes the link. I hope you'll find this useful. Any such contributions that anybody would like to make would be gratefully received!

Note that the changes won't appear in the stable docs until we have another release.

--
Sean Gillies

Re: Rewriting uint16 headers with rasterio / applying rio color makes them unreadable by Preview, Photoshop

Sean Gillies
 

Hi Ed,

Can you try the following variation on your first command?

$ rio color -j 1 uint16_image.tif uint16_brightened.tif gamma RGB 1.5 --co photometric=RGB

Note the addition of "--co photometric=RGB". GDAL automatically sets the photometric tag (which other apps need) to RGB when the created image data type is uint8 (see https://www.gdal.org/frmt_gtiff.html) but does not do the same for other data types including uint16.


On Thu, Mar 28, 2019 at 5:48 PM Edward Boyda <boyda@...> wrote:

Hi all, I'm new at this, more of a computer vision person than a developer, so please bear with me....

I see the behavior I'm about to describe running pip-installed rasterio (1.0.22) on my Mac (OSX Mojave; homebrewed python and gdal) and also running rasterio (1.0.22) in a dockerized Ubuntu platform, on images from a variety of sources (DigitalGlobe, Planet, Landsat). 

Example 1: 
$ rio color -j 1 uint16_image.tif uint16_brightened.tif gamma RGB 1.5

When I try to open the output file, uint16_brightened.tif, in Preview or Photoshop, I get a message like "Could not complete your request because of a problem parsing the TIFF file." (That's from Photoshop; Preview is equivalent.) 

Example 2:  
$ rio color -j -1 uint16_image.tif uint16_brightened_v2.tif gamma RGB 1.5

The number of cores has changed from the first example. Now the output, uint16_brightened_v2.tif, is readable by Photoshop but has had its color interpretation changed to (reading from rasterio):

(<ColorInterp.gray: 1>, <ColorInterp.undefined: 0>, <ColorInterp.undefined: 0>)

When I open the file with Preview or view the thumbnail with Mac Finder, there are dark vertical lines interspersed with the actual pixels, and about a third of the original pixels have been pushed out of the frame. See screenshot attached.

Example 3:
I take a file that has a (gray, undefined, undefined) color interpretation and try to change that to RGB, now in the interpreter:
>>> with rasterio.open('uint16_noCI.tif', 'r+') as f:
    f.colorinterp = (ColorInterp.red, ColorInterp.green, ColorInterp.blue)

Again the edited file is unreadable by Preview and Photoshop. 

A couple of caveats:

1) I can read the data from files output from any of the above examples with rasterio or skimage, and the resulting numpy array is uncorrputed. I can resave it with skimage, show it with matplolib, etc., and the image looks as expected.
2) With any of the above outputs, I can read and then rewrite a new_image.tif, using rasterio, and the resulting files open as expected with Photoshop and Preview. This is my current (obviously inefficient) workaround:

with rasterio.open('uint16_brightened.tif') as f:
    prof = f.profile
    img = f.read()

with rasterio.open('new_image.tif', 'w', photometric='rgb', **prof) as f:
    f.write(img)

As far as I know these failures happen only with uint16 images (at least not with uint8), and it would seem to have to do with the way color interpretation is written into the headers via the different write mechanisms.  Has anyone come across similar behavior? I've been reproducing and beating my head against this for months and would really appreciate a sanity check.

Since the Docker container is likely cleaner than what I've installed on my Mac, I've run the tests for the attached output there. Here is the Dockerfile and some possibly relevant release details: 

FROM ubuntu:latest

RUN apt-get update && apt-get install -y software-properties-common

RUN apt-get install -y python3-pip python3-dev build-essential

RUN pip3 install --upgrade pip

RUN apt-get install -y gdal-bin libgdal-dev python3-gdal

RUN apt-get install -y libssl-dev libffi-dev libcurl4-openssl-dev

ENV DEBIAN_FRONTEND noninteractive

RUN apt-get install -y python3-tk


ADD ./requirements.txt /tmp/requirements.txt

RUN pip install -r /tmp/requirements.txt


---

DISTRIB_ID=Ubuntu

DISTRIB_RELEASE=18.04

python 3.6.6

pip 18.1 from /usr/local/lib/python3.6/dist-packages/pip (python 3.6)


affine==2.2.2

gbdx-auth==0.4.0

gbdxtools==0.16.0

GDAL==2.2.2

rasterio==1.0.22

rio-color==1.0.0

rio-mucho==1.0.0

Shapely==1.6.4

tifffile==2018.11.28



Thanks everyone!


Ed



--
Sean Gillies