Date   

Re: Applying Color Ramp to Single-Band Grayscale Images

Brandon Victor
 

Hello Eyat,

I just realised I didn't respond. Definitely, if it's not a once-off, a programmatic solution is the best way. Option #2 can be used with negative integers; the gotchya is that matplotlib interprets floating point numbers as scaling between 0-1, and integers as scaling between 0-256. You'll also have to set the dtype to int16 if your DEMs are stored in uint8 or int16 and you are doing a subtraction. You just have to set your range to include the negative values. Something like this (you can also replace np.int16 with np.float32 if you need):

```python
data = np.random.uniform(-256, 256, (100, 100)).astype(np.int16)
cmap_dict = {n: np.array(cmap(int((n+256)/2)))*256 for n in range(-256, 256, 2)}
with rasterio.open('scraps/mapped.tif', 'w', width=100, height=100, count=1, dtype=np.int16) as dst:
    dst.write(data, 1)
    dst.write_colormap(1, cmap_dict)
```

There are at least two reasons option #3 is not recommended:
  1. You bake the colours into the image, meaning that for downstream users, there's no practical way to recover the variable of interest from the image. That might be you, or it might be other people.
  2. You are multiplying your data storage requirements. This may or may not be relevant to you. Smaller files are easier to deal with.
Of course, the benefits of option #3 are that you have a more portable file; it could even be opened by a non-GIS image viewer. I assumed that processing utility was paramount; perhaps I shouldn't have.

Perhaps it's too untimely a response for you, but maybe it will help anyone else who stumbles on this page.


Re: rasterio ImportError: version `GLIBCXX_3.4.30' not found

alec.glen.ostrander@...
 

Installed via conda; `conda list rasterio` does show it coming from conda-forge.

The OS is Ubuntu 20.04.4 LTS running in Docker. It should be reproducible with the following commands:
```

docker run -dit --name jbase jupyter/base-notebook

docker exec jbase conda install rasterio

docker exec jbase python -c "import rasterio"

```


Re: rasterio ImportError: version `GLIBCXX_3.4.30' not found

Herzmann, Daryl E [AGRON]
 

Greetings,

How did you install rasterio ? Via conda or pip ? Does `conda list rasterio` show it coming from conda-forge? Which version of Linux is this?

daryl

________________________________________
From: main@rasterio.groups.io <main@rasterio.groups.io> on behalf of alec.glen.ostrander via groups.io <alec.glen.ostrander@...>
Sent: Thursday, May 19, 2022 3:08 PM
To: main@rasterio.groups.io
Subject: [rasterio] rasterio ImportError: version `GLIBCXX_3.4.30' not found

Hello, I'm struggling to figure out an issue with my installation of rasterio, any help is greatly appreciated!

I've had rasterio installed and working cleanly for a month, but today when I ran my code, I received the following error:

Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/opt/conda/lib/python3.9/site-packages/rasterio/__init__.py", line 9, in <module>
from rasterio._base import gdal_version
ImportError: /opt/conda/lib/python3.9/site-packages/rasterio/../../.././libstdc++.so.6: version `GLIBCXX_3.4.30' not found (required by /opt/conda/lib/python3.9/site-packages/rasterio/../../.././libtiledb.so.2.8)

I've tried reinstalling a few times without luck, and confirmed that GLIBCXX_3.4.30 does not exist on my machine (I have versions 3.4.1-29). But I'm a bit beyond my depth trying to figure out how to fix it.


Re: rasterio ImportError: version `GLIBCXX_3.4.30' not found

alec.glen.ostrander@...
 

Seems like it might be a bug, related to https://forum.manjaro.org/t/latest-firefox-has-possible-dependency-issue-with-missing-glibcxx-3-4-30/111179?


rasterio ImportError: version `GLIBCXX_3.4.30' not found

alec.glen.ostrander@...
 

Hello, I'm struggling to figure out an issue with my installation of rasterio, any help is greatly appreciated!

I've had rasterio installed and working cleanly for a month, but today when I ran my code, I received the following error:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/opt/conda/lib/python3.9/site-packages/rasterio/__init__.py", line 9, in <module>
    from rasterio._base import gdal_version
ImportError: /opt/conda/lib/python3.9/site-packages/rasterio/../../.././libstdc++.so.6: version `GLIBCXX_3.4.30' not found (required by /opt/conda/lib/python3.9/site-packages/rasterio/../../.././libtiledb.so.2.8)

I've tried reinstalling a few times without luck, and confirmed that GLIBCXX_3.4.30 does not exist on my machine (I have versions 3.4.1-29). But I'm a bit beyond my depth trying to figure out how to fix it.


Rasterio 1.3b1

Sean Gillies
 

Hi all,

Rasterio 1.3b1 is on PyPI now. We're not accepting new features for 1.3.0 and are only working on major runtime and installation bugs. If you would be willing to try out this pre-release and provide feedback on GitHub, or in responses to this email, the 1.3.0 release will be better and will arrive sooner.

There's one new feature since 1.3a4: support for writing both band data and band mask when passing a Numpy MaskedArray to a dataset's write method. Perhaps more interesting is that the binary wheels on PyPI include GDAL 3.5.0rc4 and PROJ 9.0.0, the latest versions of each.

Our set of binary wheels is small right now, but we'll keep working on that, and will send updates when we've added support for more platforms.

Big thanks to everyone who contributed since 1.3a4, particularly Alan Snow, who cleaned up a lot of obsolete code that was supporting older versions of GDAL.

--
Sean Gillies


Inconsistent results between gdalWarp (with cutline) and rasterio.mask

bbuzz318@...
 

Hi,
Please see my stack exchange post:
https://gis.stackexchange.com/questions/430544/inconsistent-results-between-gdalwarp-with-cutline-and-rasterio-mask
and this post: https://github.com/rasterio/rasterio/issues/2355 which seems to have noticed this same issue.

On closer inspection, rasterio adds about an extra row of pixels (see attachment), which would seem to be incorrect.
Purple shows the vector I'm cropping to; the black is the result of the gdalWarp and the red is the extra from Rasterio.

Thanks!


Re: a_scale parameter for writing GTiff file

calba@...
 

Ok thanks, I will try it !


Re: a_scale parameter for writing GTiff file

Alan Snow
 

The scales and offsets properties on the DatasetWriter object should allow you to update the raster.
I recommend masking your data before scaling your data as the mask value is for the raw data.

Reference for how rioxarray does it:
  • Writing the scales/offsets to file: code
  • Reading with scales and offsets: code


Re: a_scale parameter for writing GTiff file

calba@...
 

Hello, 

Firtsly, thanks for your response ! Even if there is no option to scale on opening or on reading, do you know if it's possible to write metadatas with rasterio ?

Thank you in advance :)


Re: Applying Color Ramp to Single-Band Grayscale Images

Eyal Saiet
 

Hello Brandon,
Thanks very much for the detailed response. Unforetonetly I cannot use a GUI because my project is a software that continuous (not a one-time)  analysis change (basically subtracting DEMs), sometimes the difference will include negative values. Thus I don't think I can use uint8 #option 2. Thus seems I will need to go with option #3. Why is option #3 not recommended?
Apart from options #2 and #3 how do people create spatial figures (.tif) with a consistent color scheme to reflect the quantity of a change?


On Fri, Apr 8, 2022 at 7:22 PM Brandon Victor <interdimensional.cabbage@...> wrote:
Here's my two cents,

The way I see it, you have three options to convert a greyscale image into a colour-mapped version.

  1. You can just import your greyscale data into ArcGIS/QGIS and tell it how to render that using the colourmap you want. In QGIS, this option is called "Singleband Pseudocolor".
  2. You can create your own colourmap dictionary from matplotlib (or another library) and add that to the tif with `write_colormap()`
  3. You can modify the greyscale image to just BE the colours.


``` python
import rasterio
import numpy as np
import matplotlib.cm

cmap = matplotlib.cm.get_cmap('viridis')
data = np.random.uniform(0, 256, (100, 100)).astype(np.uint8)

# Option 1: Just use greyscale, and then interpret it in GIS software
with rasterio.open('greyscale.tif', 'w', width=100, height=100, count=1, dtype=np.uint8) as dst:
    dst.write(data, 1)

# Option 2: Explicitly add a cmap obtained from somewhere else
cmap_dict = {n: np.array(cmap(n))*256 for n in range(256)}
with rasterio.open('mapped.tif', 'w', width=100, height=100, count=1, dtype=np.uint8) as dst:
    dst.write(data, 1)
    dst.write_colormap(1, cmap_dict)

# Option 3: Save the greyscale as a coloured image (not recommended)
colour = (cmap(data)*256).astype(np.uint8) # shaped [100, 100, {RGBA}]
with rasterio.open('coloured.tif', 'w', width=100, height=100, count=3, dtype=np.uint8) as dst:
    for x in range(3):
        dst.write(colour[..., x], x+1)
```

I think you are asking about option 2. The answer, then, is that you can just use matplotlib's colourmap library to generate your colormap for you.

Most GIS software allow you to do Option 1 very easily, anyway. So if it's a one-off, it's probably easier to create/select your colormap using a GUI.

Good luck.



--


Eyal Saiet

Project manager
Remote sensing and in-situ measurements

Geophysical Institute 
University of Alaska Fairbanks
Fairbanks, AK 99775
(907) 750 6555 (cell)



The mind is not a vessel to be filled, but a fire to be kindled. Plutarch


Re: a_scale parameter for writing GTiff file

Sean Gillies
 

Hi,

On Fri, Apr 8, 2022 at 6:58 AM calba via groups.io <calba=lillemetropole.fr@groups.io> wrote:
Hi, 

I'm trying to transform the dtype of my input raster from float32 to int16 (I'm using dask, dask-rasterio and rasterio). So I multiplied the data in my matrix by 100 and convert the dtype of my dask array to rasterio.int16. 
def multiply(array):
multipliedArray = np.where(array >= 14, 100*array, -9999)
multipliedDaskArray = da.from_array(multipliedArray, chunks=(1, 5, 365))
multipliedDaskArray.dtype = rasterio.int16
return multipliedDaskArray

Now, I want to write a new GTiff file with scale 0.01 using rasterio.open and the equivalent in rasterio of the a_scale parameter from gdal_translate function. But I don't find this equivalent yet.

After numerous tests and researches, I still haven't found the a_scale option in rasterio... So can someone explain to me if this is possible ? And if so, how ?

Thank you in advance :)

Rasterio has no option to scale on opening or on reading. It's my understanding that gdal_translate's -a_scale sets a scaling metadata value but that I/O methods like https://gdal.org/api/gdaldataset_cpp.html#_CPPv4N11GDALDataset8RasterIOE10GDALRWFlagiiiiPvii12GDALDataTypeiPi8GSpacing8GSpacing8GSpacingP20GDALRasterIOExtraArg do not apply the scaling automatically. It is up to the caller to perform the scaling.

--
Sean Gillies


Re: Applying Color Ramp to Single-Band Grayscale Images

Brandon Victor
 

Here's my two cents,

The way I see it, you have three options to convert a greyscale image into a colour-mapped version.

  1. You can just import your greyscale data into ArcGIS/QGIS and tell it how to render that using the colourmap you want. In QGIS, this option is called "Singleband Pseudocolor".
  2. You can create your own colourmap dictionary from matplotlib (or another library) and add that to the tif with `write_colormap()`
  3. You can modify the greyscale image to just BE the colours.


``` python
import rasterio
import numpy as np
import matplotlib.cm

cmap = matplotlib.cm.get_cmap('viridis')
data = np.random.uniform(0, 256, (100, 100)).astype(np.uint8)

# Option 1: Just use greyscale, and then interpret it in GIS software
with rasterio.open('greyscale.tif', 'w', width=100, height=100, count=1, dtype=np.uint8) as dst:
    dst.write(data, 1)

# Option 2: Explicitly add a cmap obtained from somewhere else
cmap_dict = {n: np.array(cmap(n))*256 for n in range(256)}
with rasterio.open('mapped.tif', 'w', width=100, height=100, count=1, dtype=np.uint8) as dst:
    dst.write(data, 1)
    dst.write_colormap(1, cmap_dict)

# Option 3: Save the greyscale as a coloured image (not recommended)
colour = (cmap(data)*256).astype(np.uint8) # shaped [100, 100, {RGBA}]
with rasterio.open('coloured.tif', 'w', width=100, height=100, count=3, dtype=np.uint8) as dst:
    for x in range(3):
        dst.write(colour[..., x], x+1)
```

I think you are asking about option 2. The answer, then, is that you can just use matplotlib's colourmap library to generate your colormap for you.

Most GIS software allow you to do Option 1 very easily, anyway. So if it's a one-off, it's probably easier to create/select your colormap using a GUI.

Good luck.


a_scale parameter for writing GTiff file

calba@...
 

Hi, 

I'm trying to transform the dtype of my input raster from float32 to int16 (I'm using dask, dask-rasterio and rasterio). So I multiplied the data in my matrix by 100 and convert the dtype of my dask array to rasterio.int16. 
def multiply(array):
multipliedArray = np.where(array >= 14, 100*array, -9999)
multipliedDaskArray = da.from_array(multipliedArray, chunks=(1, 5, 365))
multipliedDaskArray.dtype = rasterio.int16
return multipliedDaskArray

Now, I want to write a new GTiff file with scale 0.01 using rasterio.open and the equivalent in rasterio of the a_scale parameter from gdal_translate function. But I don't find this equivalent yet.

After numerous tests and researches, I still haven't found the a_scale option in rasterio... So can someone explain to me if this is possible ? And if so, how ?

Thank you in advance :)


Re: Applying Color Ramp to Single-Band Grayscale Images

Eyal Saiet
 

Hello Sean,
I am at the same place Nathan was a few years ago, and have not seen more development in rasterio on that font. If I would like to use a coloramp like "spectrum" in the matplolib library or something similar in ArcGIS/QGIS what is the format of a file to read a coloramp (.csv,.json,.txt, etc')? Do I need to generate the color ramp myself (dininiary) for each pixel value from 0-256 (8 bit)? The example you referred, to, and the only thing I found in the Rasterio Documentation is a classified raster with only a few colors.
dst.write_colormap(
            1, {
                0: (255, 0, 0, 255),
                255: (0, 0, 255, 255) })
https://rasterio.readthedocs.io/en/latest/topics/color.html

Does Rasterio not have a coloramp library likematplotlib has (https://matplotlib.org/stable/gallery/color/colormap_reference.html)?
Thanks

On Wed, Mar 25, 2020 at 10:04 AM Sean Gillies <sean.gillies@...> wrote:
Hi,

Have you seen https://rasterio.readthedocs.io/en/latest/topics/color.html#writing-colormaps ? It shows how to write a colormap (or color "ramp") to a new file. You can also open an existing file in "r+" mode and call the dataset's write_colormap() method.

On Tue, Mar 24, 2020 at 4:12 PM <nathan.raley@...> wrote:
Does anyone have any methods of applying a color ramp to a grayscale image? 

I have a NDVI calculated image I am trying to apply color ramp for the decimal based value ranges, but have no idea how to accomplish this.  I was wandering if this was possible via any of the rasterio libraries, and, if so, how would one go about accomplishing this?  I have other band based calculations I'd like to attempt, but until I can figure out how to apply the color ramps, I am at a stand still.

Thanks,

--
Sean Gillies



--


Eyal Saiet

Project manager
Remote sensing and in-situ measurements

Geophysical Institute 
University of Alaska Fairbanks
Fairbanks, AK 99775
(907) 750 6555 (cell)



The mind is not a vessel to be filled, but a fire to be kindled. Plutarch


Changing Rasterio's governance

Sean Gillies
 

Dear Rasterio Contributors and Users,

Since 2013 I have been the sole arbiter of the Rasterio project. I consulted the project’s contributors on changes but had the final say over the scope, schedule, and features of the software. Eight years later, geospatial computing and software has changed and grown and expanded greatly in scope. Being a good enough “benevolent dictator” is an increasingly hard job. I don’t feel like one person can have enough domain experience to make all the right calls anymore and that the old way of running things puts the project at risk. Therefore, Rasterio will adapt and become a fully community-owned project with decisions made by community consensus. The new approach will look a lot like NumPy’s and be not radically different from those of GDAL and Python.

Community consensus means that project contributors seek and provide review of changes and that all contributors have the right to veto changes. NumPy’s governance document summarizes important points of consensus and I will quote it here.

In this context, consensus does not require:

* that we wait to solicit everybody’s opinion on every change,
* that we ever hold a vote on anything,
* or that everybody is happy or agrees with every decision.

To help the consensus process work there will be a Project Steering Council (PSC) made up of active contributors with a history of significant work. The PSC’s responsibility is to make transparency, institutional neutrality, and consensus work, and to keep the project on track. It may override contributor vetoes if necessary.

The initial members of the PSC will be Alan Snow, Sean Gillies, and Vincent Sarago. This is effective immediately. Others may be added in time.

Consensus democracy can, in theory, slow down development of a project. We’re going to be spending more time creating consensus. But Rasterio’s development has slowed already. We may be able to go faster in the end. Let’s find out.

A more detailed description of Rasterio’s governance will be added to the project repository. Watch https://github.com/rasterio/rasterio/pulls for changes to the CONTRIBUTING doc or a new GOVERNANCE doc.

Sincerely,

--
Sean Gillies


Re: The us-west-2 location constraint is incompatible for the region specific endpoint this request was sent to.

Pritimoy Podder
 

Exactly, thanks, Sean.


Re: Unable to read COG via /vsicurl using rasterio (but can access via GDAL)

henry@...
 
Edited

Thank you for your reply, Vincent! I installed rasterio==1.3a3 from pypi and set a few GDAL environment variables and now everything is working as expected in my non-conda installation.

Here are the GDAL environment variables that I need to set to read the file:

export CPL_VSIL_CURL_USE_HEAD=YES

export GDAL_DISABLE_READDIR_ON_OPEN=OPEN_DIR

export GDAL_HTTP_COOKIEJAR=/tmp/cookies.txt

export GDAL_HTTP_COOKIEFILE=/tmp/cookies.txt

export CPL_VSIL_CURL_ALLOWED_EXTENSIONS=TIF


Re: The us-west-2 location constraint is incompatible for the region specific endpoint this request was sent to.

Sean Gillies
 

Thanks for following up! You can also unset AWS_END_POINT since s3.amazonaws.com is the default.

On Thu, Feb 24, 2022 at 6:48 AM <pritimoypodder@...> wrote:
This is now fixed. I had an environment variable for "AWS_END_POINT" which is now updated to "s3.amazonaws.com" to make it work.



--
Sean Gillies


Re: The us-west-2 location constraint is incompatible for the region specific endpoint this request was sent to.

Pritimoy Podder
 

This is now fixed. I had an environment variable for "AWS_END_POINT" which is now updated to "s3.amazonaws.com" to make it work.

1 - 20 of 945