Hi all, I'm new at this, more of a computer vision person than a developer, so please bear with me....

I see the behavior I'm about to describe running pip-installed rasterio (1.0.22) on my Mac (OSX Mojave; homebrewed python and gdal) and also running rasterio (1.0.22) in a dockerized Ubuntu platform, on images from a variety of sources (DigitalGlobe, Planet, Landsat). 

Example 1: 

$ rio color -j 1 uint16_image.tif uint16_brightened.tif gamma RGB 1.5

When I try to open the output file, uint16_brightened.tif, in Preview or Photoshop, I get a message like "Could not complete your request because of a problem parsing the TIFF file." (That's from Photoshop; Preview is equivalent.) 

Example 2:  

$ rio color -j -1 uint16_image.tif uint16_brightened_v2.tif gamma RGB 1.5

The number of cores has changed from the first example. Now the output, uint16_brightened_v2.tif, is readable by Photoshop but has had its color interpretation changed to (reading from rasterio):

(<ColorInterp.gray: 1>, <ColorInterp.undefined: 0>, <ColorInterp.undefined: 0>)

When I open the file with Preview or view the thumbnail with Mac Finder, there are dark vertical lines interspersed with the actual pixels, and about a third of the original pixels have been pushed out of the frame. See screenshot attached.

Example 3:

I take a file that has a (gray, undefined, undefined) color interpretation and try to change that to RGB, now in the interpreter:

>>> with rasterio.open('uint16_noCI.tif', 'r+') as f:

    f.colorinterp = (ColorInterp.red, ColorInterp.green, ColorInterp.blue)

Again the edited file is unreadable by Preview and Photoshop. 

A couple of caveats:

1) I can read the data from files output from any of the above examples with rasterio or skimage, and the resulting numpy array is uncorrputed. I can resave it with skimage, show it with matplolib, etc., and the image looks as expected.

2) With any of the above outputs, I can read and then rewrite a new_image.tif, using rasterio, and the resulting files open as expected with Photoshop and Preview. This is my current (obviously inefficient) workaround:

with rasterio.open('uint16_brightened.tif') as f:

    prof = f.profile

    img = f.read()

with rasterio.open('new_image.tif', 'w', photometric='rgb', **prof) as f:

    f.write(img)

As far as I know these failures happen only with uint16 images (at least not with uint8), and it would seem to have to do with the way color interpretation is written into the headers via the different write mechanisms.  Has anyone come across similar behavior? I've been reproducing and beating my head against this for months and would really appreciate a sanity check.

Since the Docker container is likely cleaner than what I've installed on my Mac, I've run the tests for the attached output there. Here is the Dockerfile and some possibly relevant release details: 

FROM ubuntu:latest

RUN apt-get update && apt-get install -y software-properties-common

RUN apt-get install -y python3-pip python3-dev build-essential

RUN pip3 install --upgrade pip

RUN apt-get install -y gdal-bin libgdal-dev python3-gdal

RUN apt-get install -y libssl-dev libffi-dev libcurl4-openssl-dev

ENV DEBIAN_FRONTEND noninteractive

RUN apt-get install -y python3-tk

ADD ./requirements.txt /tmp/requirements.txt

RUN pip install -r /tmp/requirements.txt

---

DISTRIB_ID=Ubuntu

DISTRIB_RELEASE=18.04

python 3.6.6

pip 18.1 from /usr/local/lib/python3.6/dist-packages/pip (python 3.6)

affine==2.2.2

gbdx-auth==0.4.0

gbdxtools==0.16.0

GDAL==2.2.2

rasterio==1.0.22

rio-color==1.0.0

rio-mucho==1.0.0

Shapely==1.6.4

tifffile==2018.11.28

Thanks everyone!

Ed

Hi Ed,

Can you try the following variation on your first command?

$ rio color -j 1 uint16_image.tif uint16_brightened.tif gamma RGB 1.5 --co photometric=RGB

Note the addition of "--co photometric=RGB". GDAL automatically sets the photometric tag (which other apps need) to RGB when the created image data type is uint8 (see https://www.gdal.org/frmt_gtiff.html) but does not do the same for other data types including uint16.

On Thu, Mar 28, 2019 at 5:48 PM Edward Boyda <boyda@...> wrote:

Hi all, I'm new at this, more of a computer vision person than a developer, so please bear with me....

I see the behavior I'm about to describe running pip-installed rasterio (1.0.22) on my Mac (OSX Mojave; homebrewed python and gdal) and also running rasterio (1.0.22) in a dockerized Ubuntu platform, on images from a variety of sources (DigitalGlobe, Planet, Landsat). 

Example 1: 

$ rio color -j 1 uint16_image.tif uint16_brightened.tif gamma RGB 1.5

When I try to open the output file, uint16_brightened.tif, in Preview or Photoshop, I get a message like "Could not complete your request because of a problem parsing the TIFF file." (That's from Photoshop; Preview is equivalent.) 

Example 2:  

$ rio color -j -1 uint16_image.tif uint16_brightened_v2.tif gamma RGB 1.5

The number of cores has changed from the first example. Now the output, uint16_brightened_v2.tif, is readable by Photoshop but has had its color interpretation changed to (reading from rasterio):

(<ColorInterp.gray: 1>, <ColorInterp.undefined: 0>, <ColorInterp.undefined: 0>)

When I open the file with Preview or view the thumbnail with Mac Finder, there are dark vertical lines interspersed with the actual pixels, and about a third of the original pixels have been pushed out of the frame. See screenshot attached.

Example 3:

I take a file that has a (gray, undefined, undefined) color interpretation and try to change that to RGB, now in the interpreter:

>>> with rasterio.open('uint16_noCI.tif', 'r+') as f:

    f.colorinterp = (ColorInterp.red, ColorInterp.green, ColorInterp.blue)

Again the edited file is unreadable by Preview and Photoshop. 

A couple of caveats:

1) I can read the data from files output from any of the above examples with rasterio or skimage, and the resulting numpy array is uncorrputed. I can resave it with skimage, show it with matplolib, etc., and the image looks as expected.

2) With any of the above outputs, I can read and then rewrite a new_image.tif, using rasterio, and the resulting files open as expected with Photoshop and Preview. This is my current (obviously inefficient) workaround:

with rasterio.open('uint16_brightened.tif') as f:

    prof = f.profile

    img = f.read()

with rasterio.open('new_image.tif', 'w', photometric='rgb', **prof) as f:

    f.write(img)

As far as I know these failures happen only with uint16 images (at least not with uint8), and it would seem to have to do with the way color interpretation is written into the headers via the different write mechanisms.  Has anyone come across similar behavior? I've been reproducing and beating my head against this for months and would really appreciate a sanity check.

Since the Docker container is likely cleaner than what I've installed on my Mac, I've run the tests for the attached output there. Here is the Dockerfile and some possibly relevant release details: 

FROM ubuntu:latest

RUN apt-get update && apt-get install -y software-properties-common

RUN apt-get install -y python3-pip python3-dev build-essential

RUN pip3 install --upgrade pip

RUN apt-get install -y gdal-bin libgdal-dev python3-gdal

RUN apt-get install -y libssl-dev libffi-dev libcurl4-openssl-dev

ENV DEBIAN_FRONTEND noninteractive

RUN apt-get install -y python3-tk

ADD ./requirements.txt /tmp/requirements.txt

RUN pip install -r /tmp/requirements.txt

---

DISTRIB_ID=Ubuntu

DISTRIB_RELEASE=18.04

python 3.6.6

pip 18.1 from /usr/local/lib/python3.6/dist-packages/pip (python 3.6)

affine==2.2.2

gbdx-auth==0.4.0

gbdxtools==0.16.0

GDAL==2.2.2

rasterio==1.0.22

rio-color==1.0.0

rio-mucho==1.0.0

Shapely==1.6.4

tifffile==2018.11.28

Thanks everyone!

Ed

--

Sean Gillies

Thanks, Sean, that works for rio color.  I use the photometric creation option with gdal_translate but didn't connect the dots to rio color. 

I can see why, given all the possible creation options, this isn't mentioned explicitly in the docs. But maybe photometric interpretation makes a worthy special case - it's counterintuitive that an operation to adjust the color of an image doesn't by default preserve its header structure. 

Is there a similarly simple fix for setting colorinterp manually in update mode (my third example)? 

Much appreciated.

Ed

On Tue, Apr 2, 2019 at 1:55 PM Sean Gillies <sean.gillies@...> wrote:

Hi Ed,

Can you try the following variation on your first command?

$ rio color -j 1 uint16_image.tif uint16_brightened.tif gamma RGB 1.5 --co photometric=RGB

Note the addition of "--co photometric=RGB". GDAL automatically sets the photometric tag (which other apps need) to RGB when the created image data type is uint8 (see https://www.gdal.org/frmt_gtiff.html) but does not do the same for other data types including uint16.

On Thu, Mar 28, 2019 at 5:48 PM Edward Boyda <boyda@...> wrote:

Hi all, I'm new at this, more of a computer vision person than a developer, so please bear with me....

I see the behavior I'm about to describe running pip-installed rasterio (1.0.22) on my Mac (OSX Mojave; homebrewed python and gdal) and also running rasterio (1.0.22) in a dockerized Ubuntu platform, on images from a variety of sources (DigitalGlobe, Planet, Landsat). 

Example 1: 

$ rio color -j 1 uint16_image.tif uint16_brightened.tif gamma RGB 1.5

When I try to open the output file, uint16_brightened.tif, in Preview or Photoshop, I get a message like "Could not complete your request because of a problem parsing the TIFF file." (That's from Photoshop; Preview is equivalent.) 

Example 2:  

$ rio color -j -1 uint16_image.tif uint16_brightened_v2.tif gamma RGB 1.5

The number of cores has changed from the first example. Now the output, uint16_brightened_v2.tif, is readable by Photoshop but has had its color interpretation changed to (reading from rasterio):

(<ColorInterp.gray: 1>, <ColorInterp.undefined: 0>, <ColorInterp.undefined: 0>)

When I open the file with Preview or view the thumbnail with Mac Finder, there are dark vertical lines interspersed with the actual pixels, and about a third of the original pixels have been pushed out of the frame. See screenshot attached.

Example 3:

I take a file that has a (gray, undefined, undefined) color interpretation and try to change that to RGB, now in the interpreter:

>>> with rasterio.open('uint16_noCI.tif', 'r+') as f:

    f.colorinterp = (ColorInterp.red, ColorInterp.green, ColorInterp.blue)

Again the edited file is unreadable by Preview and Photoshop. 

A couple of caveats:

1) I can read the data from files output from any of the above examples with rasterio or skimage, and the resulting numpy array is uncorrputed. I can resave it with skimage, show it with matplolib, etc., and the image looks as expected.

2) With any of the above outputs, I can read and then rewrite a new_image.tif, using rasterio, and the resulting files open as expected with Photoshop and Preview. This is my current (obviously inefficient) workaround:

with rasterio.open('uint16_brightened.tif') as f:

    prof = f.profile

    img = f.read()

with rasterio.open('new_image.tif', 'w', photometric='rgb', **prof) as f:

    f.write(img)

As far as I know these failures happen only with uint16 images (at least not with uint8), and it would seem to have to do with the way color interpretation is written into the headers via the different write mechanisms.  Has anyone come across similar behavior? I've been reproducing and beating my head against this for months and would really appreciate a sanity check.

Since the Docker container is likely cleaner than what I've installed on my Mac, I've run the tests for the attached output there. Here is the Dockerfile and some possibly relevant release details: 

FROM ubuntu:latest

RUN apt-get update && apt-get install -y software-properties-common

RUN apt-get install -y python3-pip python3-dev build-essential

RUN pip3 install --upgrade pip

RUN apt-get install -y gdal-bin libgdal-dev python3-gdal

RUN apt-get install -y libssl-dev libffi-dev libcurl4-openssl-dev

ENV DEBIAN_FRONTEND noninteractive

RUN apt-get install -y python3-tk

ADD ./requirements.txt /tmp/requirements.txt

RUN pip install -r /tmp/requirements.txt

---

DISTRIB_ID=Ubuntu

DISTRIB_RELEASE=18.04

python 3.6.6

pip 18.1 from /usr/local/lib/python3.6/dist-packages/pip (python 3.6)

affine==2.2.2

gbdx-auth==0.4.0

gbdxtools==0.16.0

GDAL==2.2.2

rasterio==1.0.22

rio-color==1.0.0

rio-mucho==1.0.0

Shapely==1.6.4

tifffile==2018.11.28

Thanks everyone!

Ed

--

Sean Gillies

For GeoTIFF (at least), the photometric creation option is different from the color interpretation that we can get/set using the GDAL API. The file's layout and compression strategy is influenced by the photometric creation option, so it is needed up front and has a permanent impact on the file. Setting the color interpretation through the rasterio API (and thereby GDAL API) will change file metadata but will not change the file's structure. It's confusing, to be sure. Even Rouault (the expert) provides a bit more about it in this email to gdal-dev:

https://lists.osgeo.org/pipermail/gdal-dev/2013-November/037627.html

On Wed, Apr 3, 2019 at 10:31 AM Edward Boyda <boyda@...> wrote:

Thanks, Sean, that works for rio color.  I use the photometric creation option with gdal_translate but didn't connect the dots to rio color. 

I can see why, given all the possible creation options, this isn't mentioned explicitly in the docs. But maybe photometric interpretation makes a worthy special case - it's counterintuitive that an operation to adjust the color of an image doesn't by default preserve its header structure. 

Is there a similarly simple fix for setting colorinterp manually in update mode (my third example)? 

Much appreciated.

Ed

On Tue, Apr 2, 2019 at 1:55 PM Sean Gillies <sean.gillies@...> wrote:

Hi Ed,

Can you try the following variation on your first command?

$ rio color -j 1 uint16_image.tif uint16_brightened.tif gamma RGB 1.5 --co photometric=RGB

Note the addition of "--co photometric=RGB". GDAL automatically sets the photometric tag (which other apps need) to RGB when the created image data type is uint8 (see https://www.gdal.org/frmt_gtiff.html) but does not do the same for other data types including uint16.

On Thu, Mar 28, 2019 at 5:48 PM Edward Boyda <boyda@...> wrote:

Hi all, I'm new at this, more of a computer vision person than a developer, so please bear with me....

I see the behavior I'm about to describe running pip-installed rasterio (1.0.22) on my Mac (OSX Mojave; homebrewed python and gdal) and also running rasterio (1.0.22) in a dockerized Ubuntu platform, on images from a variety of sources (DigitalGlobe, Planet, Landsat). 

Example 1: 

$ rio color -j 1 uint16_image.tif uint16_brightened.tif gamma RGB 1.5

When I try to open the output file, uint16_brightened.tif, in Preview or Photoshop, I get a message like "Could not complete your request because of a problem parsing the TIFF file." (That's from Photoshop; Preview is equivalent.) 

Example 2:  

$ rio color -j -1 uint16_image.tif uint16_brightened_v2.tif gamma RGB 1.5

The number of cores has changed from the first example. Now the output, uint16_brightened_v2.tif, is readable by Photoshop but has had its color interpretation changed to (reading from rasterio):

(<ColorInterp.gray: 1>, <ColorInterp.undefined: 0>, <ColorInterp.undefined: 0>)

When I open the file with Preview or view the thumbnail with Mac Finder, there are dark vertical lines interspersed with the actual pixels, and about a third of the original pixels have been pushed out of the frame. See screenshot attached.

Example 3:

I take a file that has a (gray, undefined, undefined) color interpretation and try to change that to RGB, now in the interpreter:

>>> with rasterio.open('uint16_noCI.tif', 'r+') as f:

    f.colorinterp = (ColorInterp.red, ColorInterp.green, ColorInterp.blue)

Again the edited file is unreadable by Preview and Photoshop. 

A couple of caveats:

1) I can read the data from files output from any of the above examples with rasterio or skimage, and the resulting numpy array is uncorrputed. I can resave it with skimage, show it with matplolib, etc., and the image looks as expected.

2) With any of the above outputs, I can read and then rewrite a new_image.tif, using rasterio, and the resulting files open as expected with Photoshop and Preview. This is my current (obviously inefficient) workaround:

with rasterio.open('uint16_brightened.tif') as f:

    prof = f.profile

    img = f.read()

with rasterio.open('new_image.tif', 'w', photometric='rgb', **prof) as f:

    f.write(img)

As far as I know these failures happen only with uint16 images (at least not with uint8), and it would seem to have to do with the way color interpretation is written into the headers via the different write mechanisms.  Has anyone come across similar behavior? I've been reproducing and beating my head against this for months and would really appreciate a sanity check.

Since the Docker container is likely cleaner than what I've installed on my Mac, I've run the tests for the attached output there. Here is the Dockerfile and some possibly relevant release details: 

FROM ubuntu:latest

RUN apt-get update && apt-get install -y software-properties-common

RUN apt-get install -y python3-pip python3-dev build-essential

RUN pip3 install --upgrade pip

RUN apt-get install -y gdal-bin libgdal-dev python3-gdal

RUN apt-get install -y libssl-dev libffi-dev libcurl4-openssl-dev

ENV DEBIAN_FRONTEND noninteractive

RUN apt-get install -y python3-tk

ADD ./requirements.txt /tmp/requirements.txt

RUN pip install -r /tmp/requirements.txt

---

DISTRIB_ID=Ubuntu

DISTRIB_RELEASE=18.04

python 3.6.6

pip 18.1 from /usr/local/lib/python3.6/dist-packages/pip (python 3.6)

affine==2.2.2

gbdx-auth==0.4.0

gbdxtools==0.16.0

GDAL==2.2.2

rasterio==1.0.22

rio-color==1.0.0

rio-mucho==1.0.0

Shapely==1.6.4

tifffile==2018.11.28

Thanks everyone!

Ed

--

Sean Gillies

--

Sean Gillies