Hi there!

I have GeoTIFF files with internal mask and I need to reproject them and not lose mask.

I've found that direct generating warped mask band is not supported by GDAL: https://github.com/OSGeo/gdal/issues/689

rasterio.warp.reproject() also doesn't preserve mask.

How I can use rasterio for achieving what I need? I'm looking for something similar described in this issue but with rasterio.

Hi, folks!

Is it possible to convert internal mask to alpha band? 

I am having the same problem on Windows 7 and Mac OS X High Sierra:

1) Windows 7 (64 bits)

Using python 2.7 (32 bits).

I have installed GDAL Core and its Python bindings using the executables here (resp. gdal-202-1500-core.msi and GDAL-2.2.3.win32-py2.7.msi). This worked well and I could successfully use gdal in python scripts.

Then I tried to install rasterio using rasterio‑1.0.3‑cp27‑cp27m‑win32.whl:
py -2.7 -m pip install rasterio‑1.0.3‑cp27‑cp27m‑win32.whl

This went through successfully.

However when starting python and invoking import rasterio I get the following error message:

Traceback (most recent call last):
  File "test1.py", line 1, in <module>
    import rasterio
  File "C:\Python27\lib\site-packages\rasterio\__init__.py", line 31, in <module>
    from rasterio._base import gdal_version
  File "rasterio\_base.pyx", line 1, in init rasterio._base
  File "__init__.pxd", line 164, in init rasterio._shim
ValueError: numpy.dtype has the wrong size, try recompiling. Expected 52, got 56

2) Mac OS X High Sierra (10.13.6)

Interestingly, I am having exactly the same error on Mac OS X High Sierra after installing GDAL Complete 2.1 from here, and rasterio using:
pip2.7 install rasterio.

The installation steps are successful, but I get the same error as above when executing import rasterio in python 2.7:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Python/2.7/site-packages/rasterio/__init__.py", line 23, in <module>
    from rasterio._base import gdal_version
  File "rasterio/_base.pyx", line 1, in init rasterio._base
  File "__init__.pxd", line 164, in init rasterio._shim
ValueError: numpy.dtype has the wrong size, try recompiling. Expected 88, got 96

I have read posts advising to upgrade numpy, however it is installed as distutils project (on both systems) and therefore I cannot upgrade with pip. So I am not sure what is the best way forward.

Any help would be greatly appreciated.

Hi all,

We are trying to read masked array using `DatasetReaderBase.read`, reading the documentation it looks that if we pass masked=True we will get what we wanted. 

I was testing the read on raster with pre-built overviews and I used the overviews in read. From functionality point of view, I always got the right result, but I was tesing the data block that are being used, and I noticed this behaviors:


1. When using band-mask, there was a read only from the overview to construct the mask -- as excpected
2. When using no-data value there were reads the data of the original resolution and not from the nearest overview? when mask=False it reads data only from the overview,  why is that?

A workaround for that is to read with no mask and to construct the mask myself using the no-data value.

Thanks.

Hi Guy,

A month ago we resolved an issue in Rasterio that has some commonalities with yours. There was a report that overviews were being missed when reading masked arrays and it turned out that the overviews themselves were in error.

https://github.com/mapbox/rasterio/issues/1373#issuecomment-403520050

In Rasterio, we will skip reading a mask from the dataset if the dataset has no mask, but otherwise we don't call GDAL functions differently in the band mask vs nodata cases.

Can you tell us what nodata value is implicated in this problem and what software produced the data files?

On Mon, Aug 6, 2018 at 8:52 AM <guyd@...> wrote:

Hi all,

We are trying to read masked array using `DatasetReaderBase.read`, reading the documentation it looks that if we pass masked=True we will get what we wanted. 

I was testing the read on raster with pre-built overviews and I used the overviews in read. From functionality point of view, I always got the right result, but I was tesing the data block that are being used, and I noticed this behaviors:


1. When using band-mask, there was a read only from the overview to construct the mask -- as excpected
2. When using no-data value there were reads the data of the original resolution and not from the nearest overview? when mask=False it reads data only from the overview,  why is that?

A workaround for that is to read with no mask and to construct the mask myself using the no-data value.

Thanks.

--

Sean Gillies

He Sean,

We are creating  cog files using our code that takes as input any raster, tiles it add overviews and rearrange the tif to be a cog.

For doing so we are using rasetio shutils copy and rasterio build_overview.

The source rasters that we are working on have nodata

We did a variant of code in which we prior for creating the overviews we also created Band Mask

1. when we had no-data values, and read using the overviews it looked like it was constructing the overview from the original size

On Wed, Aug 8, 2018 at 12:14 AM, Sean Gillies <sean.gillies@...> wrote:

Hi Guy,

A month ago we resolved an issue in Rasterio that has some commonalities with yours. There was a report that overviews were being missed when reading masked arrays and it turned out that the overviews themselves were in error.

https://github.com/mapbox/rasterio/issues/1373#issuecomment-403520050

In Rasterio, we will skip reading a mask from the dataset if the dataset has no mask, but otherwise we don't call GDAL functions differently in the band mask vs nodata cases.

Can you tell us what nodata value is implicated in this problem and what software produced the data files?

On Mon, Aug 6, 2018 at 8:52 AM <guyd@...> wrote:

Hi all,

We are trying to read masked array using `DatasetReaderBase.read`, reading the documentation it looks that if we pass masked=True we will get what we wanted. 

I was testing the read on raster with pre-built overviews and I used the overviews in read. From functionality point of view, I always got the right result, but I was tesing the data block that are being used, and I noticed this behaviors:


1. When using band-mask, there was a read only from the overview to construct the mask -- as excpected
2. When using no-data value there were reads the data of the original resolution and not from the nearest overview? when mask=False it reads data only from the overview,  why is that?

A workaround for that is to read with no mask and to construct the mask myself using the no-data value.

Thanks.

--

Sean Gillies

I stacked some singleband rasters into a multiband raster using `rio stack` with a compress=deflate creation option.  The compressed output filesize was approx 1.4x the sum of the uncompressed input filesizes.  Specifying --co interleave=band when running `rio stack` got the output filesize down to 0.25x the uncompressed input (per issue #70).   Running `gdalbuildvrt` to stack them, then `rio convert` or `gdal_translate` on the VRT with the same creation options as the original rio stack command also results in an output raster of around 0.25x the uncompressed input, but with default pixel interleaving.

I realise you're planning to document issues with compression (#77) but in the meantime, do you have any idea why compression is so poor with default pixel interleaving using `rio stack` but quite ok when using `rio convert` (with default pixel interleaving) on an already stacked VRT?

Luke

Hi Sean!

After updating rasterio I get the following error:

In [1]: from rasterio import open

 

In [2]: open('tests/data/raster/rgb.jp2')

---------------------------------------------------------------------------

CPLE_OpenFailedError                      Traceback (most recent call last)

rasterio/_base.pyx in rasterio._base.DatasetBase.__init__()

 

rasterio/_shim.pyx in rasterio._shim.open_dataset()

 

rasterio/_err.pyx in rasterio._err.exc_wrap_pointer()

 

CPLE_OpenFailedError: 'tests/data/raster/rgb.jp2' not recognized as a supported file format.

 

During handling of the above exception, another exception occurred:

 

RasterioIOError                           Traceback (most recent call last)

<ipython-input-2-19e7b16f33e4> in <module>()

----> 1 open('tests/data/raster/rgb.jp2')

 

~/sandbox/telluric/env/lib/python3.5/site-packages/rasterio/env.py in wrapper(*args, **kwds)

    401             else:

    402                 pass

--> 403             return f(*args, **kwds)

    404 

    405     return wrapper

 

~/sandbox/telluric/env/lib/python3.5/site-packages/rasterio/__init__.py in open(fp, mode, driver, width, height, count, crs, transform, dtype, nodata, sharing, **kwargs)

    215         # None.

    216         if mode == 'r':

--> 217             s = DatasetReader(path, driver=driver, **kwargs)

    218         elif mode == 'r+':

    219             s = get_writer_for_path(path)(path, mode, driver=driver, **kwargs)

 

rasterio/_base.pyx in rasterio._base.DatasetBase.__init__()

 

RasterioIOError: 'tests/data/raster/rgb.jp2' not recognized as a supported file format.

All works without any problems with rasterio 1.0.2:

In [1]: from rasterio import open

 

In [2]: open('tests/data/raster/rgb.jp2')

Out[2]: <open DatasetReader name='tests/data/raster/rgb.jp2' mode='r'>

Hi Philipp,

How are you looping over each value to create a new raster?  Are you able to share a gist of what you are doing?