Topics

Is it possible to ignore existing overview when performing decimated read?


Loïc Dutrieux
 

Hi everyone,

I'm trying to perform a decimated read of a dataset that already contains overviews. Regardless of which value I pass to the resampling= argument of the read method, it seems that the already existing overview is used. Is there any way to ignore it? See the reproducible example below.

Cheers,
Loïc

---

import tempfile
import os

import numpy as np
import rasterio
from rasterio.enums import Resampling


filename = os.path.join(tempfile.gettempdir(), 'overview_test.tif')

# Create random int array
shape = (100, 100)
arr = np.random.randint(0, 9999, size=shape, dtype=np.uint16)

meta = {'height': 100,
        'width': 100,
        'driver': 'GTiff',
        'dtype': np.uint16,
        'count': 1}

# Write random array to file and compute first overview
with rasterio.open(filename, 'w', **meta) as dst:
    dst.write(arr, 1)
    dst.build_overviews([2], Resampling.nearest)

# Read with downsample
with rasterio.open(filename) as src:
    arr_avg = src.read(1, out_shape=(1,50,50), resampling=Resampling.average,
                        out_dtype=np.float)
    arr_nrt = src.read(1, out_shape=(1,50,50), resampling=Resampling.nearest)

print(np.max(arr_avg - arr_nrt))
# when the source file does not contain overviews, the max of the difference array
# is > 0


Denis Rykov
 

You can find an example how to do that here: https://github.com/mapbox/rasterio/issues/1929.


On Tue, Nov 10, 2020, 8:12 PM Loïc Dutrieux <loic.dutrieux@...> wrote:
Hi everyone,

I'm trying to perform a decimated read of a dataset that already contains overviews. Regardless of which value I pass to the resampling= argument of the read method, it seems that the already existing overview is used. Is there any way to ignore it? See the reproducible example below.

Cheers,
Loïc

---

import tempfile
import os

import numpy as np
import rasterio
from rasterio.enums import Resampling


filename = os.path.join(tempfile.gettempdir(), 'overview_test.tif')

# Create random int array
shape = (100, 100)
arr = np.random.randint(0, 9999, size=shape, dtype=np.uint16)

meta = {'height': 100,
        'width': 100,
        'driver': 'GTiff',
        'dtype': np.uint16,
        'count': 1}

# Write random array to file and compute first overview
with rasterio.open(filename, 'w', **meta) as dst:
    dst.write(arr, 1)
    dst.build_overviews([2], Resampling.nearest)

# Read with downsample
with rasterio.open(filename) as src:
    arr_avg = src.read(1, out_shape=(1,50,50), resampling=Resampling.average,
                                  out_dtype=np.float)
    arr_nrt = src.read(1, out_shape=(1,50,50), resampling=Resampling.nearest)

print(np.max(arr_avg - arr_nrt))
# when the source file does not contain overviews, the max of the difference array
# is > 0









Loïc Dutrieux
 

Hi Denis,

Thanks for pointing to that issue. I may have misunderstood it but I don't think it helps answering my question.

Cheers,
Loïc
________________________________________
From: main@rasterio.groups.io <main@rasterio.groups.io> on behalf of Denis Rykov <rykovd@...>
Sent: 10 November 2020 21:36:13
To: main@rasterio.groups.io
Subject: Re: [rasterio] Is it possible to ignore existing overview when performing decimated read?

You can find an example how to do that here: https://github.com/mapbox/rasterio/issues/1929<https://urldefense.com/v3/__https://github.com/mapbox/rasterio/issues/1929__;!!DOxrgLBm!XaInc3_J0VCLXQo6gYXo0rBWHs4b0p4ZkMuimU2g9HtJy0eRb55BN-J7JDPt-MB3Dhg5K3E$>.

On Tue, Nov 10, 2020, 8:12 PM Loïc Dutrieux <@loicdtx<mailto:@loicdtx>> wrote:
Hi everyone,

I'm trying to perform a decimated read of a dataset that already contains overviews. Regardless of which value I pass to the resampling= argument of the read method, it seems that the already existing overview is used. Is there any way to ignore it? See the reproducible example below.

Cheers,
Loïc

---

import tempfile
import os

import numpy as np
import rasterio
from rasterio.enums import Resampling


filename = os.path.join(tempfile.gettempdir(), 'overview_test.tif')

# Create random int array
shape = (100, 100)
arr = np.random.randint(0, 9999, size=shape, dtype=np.uint16)

meta = {'height': 100,
'width': 100,
'driver': 'GTiff',
'dtype': np.uint16,
'count': 1}

# Write random array to file and compute first overview
with rasterio.open(filename, 'w', **meta) as dst:
dst.write(arr, 1)
dst.build_overviews([2], Resampling.nearest)

# Read with downsample
with rasterio.open(filename) as src:
arr_avg = src.read(1, out_shape=(1,50,50), resampling=Resampling.average,
out_dtype=np.float)
arr_nrt = src.read(1, out_shape=(1,50,50), resampling=Resampling.nearest)

print(np.max(arr_avg - arr_nrt))
# when the source file does not contain overviews, the max of the difference array
# is > 0


Sean Gillies
 

HI Loïc,

I don't think we currently have any option other than to remove the overviews. In theory GDAL's OVERVIEW_LEVEL open option could help us, but the implementation isn't there. For example, if you used gdalwarp with `-ovr NONE` it will set an internal overview level to the value of -1. But that bypasses GDALOpenEx (used by rasterio). GDALOpenEx won't let us pass `OVERVIEW_LEVEL=-1. It fails with a "Cannot open overview level -1" error.

To illustrate

>>> rasterio.open("tests/data/green.tif").shape
(64, 64)
>>> rasterio.open("tests/data/green.tif", OVERVIEW_LEVEL=0).shape
(32, 32)
>>> rasterio.open("tests/data/green.tif", OVERVIEW_LEVEL="NONE").shape
(32, 32)
>>> rasterio.open("tests/data/green.tif", OVERVIEW_LEVEL=-1).shape
Traceback (most recent call last):
  File "rasterio/_base.pyx", line 261, in rasterio._base.DatasetBase.__init__
    self._hds = open_dataset(filename, flags, driver, kwargs, None)
  File "rasterio/_shim.pyx", line 78, in rasterio._shim.open_dataset
    return exc_wrap_pointer(hds)
  File "rasterio/_err.pyx", line 213, in rasterio._err.exc_wrap_pointer
    raise exc
rasterio._err.CPLE_OpenFailedError: Cannot open overview level -1 of tests/data/green.tif

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/sean/projects/rasterio/rasterio/env.py", line 433, in wrapper
    return f(*args, **kwds)
  File "/home/sean/projects/rasterio/rasterio/__init__.py", line 221, in open
    s = DatasetReader(path, driver=driver, sharing=sharing, **kwargs)
  File "rasterio/_base.pyx", line 263, in rasterio._base.DatasetBase.__init__
    raise RasterioIOError(str(err))
rasterio.errors.RasterioIOError: Cannot open overview level -1 of tests/data/green.tif


On Fri, Nov 13, 2020 at 4:33 AM Loïc Dutrieux <loic.dutrieux@...> wrote:
Hi Denis,

Thanks for pointing to that issue. I may have misunderstood it but I don't think it helps answering my question.

Cheers,
Loïc
________________________________________
From: main@rasterio.groups.io <main@rasterio.groups.io> on behalf of Denis Rykov <rykovd@...>
Sent: 10 November 2020 21:36:13
To: main@rasterio.groups.io
Subject: Re: [rasterio] Is it possible to ignore existing overview when performing decimated read?

You can find an example how to do that here: https://github.com/mapbox/rasterio/issues/1929<https://urldefense.com/v3/__https://github.com/mapbox/rasterio/issues/1929__;!!DOxrgLBm!XaInc3_J0VCLXQo6gYXo0rBWHs4b0p4ZkMuimU2g9HtJy0eRb55BN-J7JDPt-MB3Dhg5K3E$>.

On Tue, Nov 10, 2020, 8:12 PM Loïc Dutrieux <loic.dutrieux@...<mailto:loic.dutrieux@...>> wrote:
Hi everyone,

I'm trying to perform a decimated read of a dataset that already contains overviews. Regardless of which value I pass to the resampling= argument of the read method, it seems that the already existing overview is used. Is there any way to ignore it? See the reproducible example below.

Cheers,
Loïc

---

import tempfile
import os

import numpy as np
import rasterio
from rasterio.enums import Resampling


filename = os.path.join(tempfile.gettempdir(), 'overview_test.tif')

# Create random int array
shape = (100, 100)
arr = np.random.randint(0, 9999, size=shape, dtype=np.uint16)

meta = {'height': 100,
        'width': 100,
        'driver': 'GTiff',
        'dtype': np.uint16,
        'count': 1}

# Write random array to file and compute first overview
with rasterio.open(filename, 'w', **meta) as dst:
    dst.write(arr, 1)
    dst.build_overviews([2], Resampling.nearest)

# Read with downsample
with rasterio.open(filename) as src:
    arr_avg = src.read(1, out_shape=(1,50,50), resampling=Resampling.average,
                                  out_dtype=np.float)
    arr_nrt = src.read(1, out_shape=(1,50,50), resampling=Resampling.nearest)

print(np.max(arr_avg - arr_nrt))
# when the source file does not contain overviews, the max of the difference array
# is > 0


--
Sean Gillies


Even Rouault
 

On vendredi 13 novembre 2020 09:05:41 CET Sean Gillies via groups.io wrote:
HI Loïc,

I don't think we currently have any option other than to remove the
overviews. In theory GDAL's OVERVIEW_LEVEL open option could help us, but
the implementation isn't there. For example, if you used gdalwarp with
`-ovr NONE` it will set an internal overview level to the value of -1. But
that bypasses GDALOpenEx (used by rasterio). GDALOpenEx won't let us pass
`OVERVIEW_LEVEL=-1. It fails with a "Cannot open overview level -1" error.
Was left as an exercice for a contributor:
https://github.com/OSGeo/gdal/pull/3181

Even

--
Spatialys - Geospatial professional services
http://www.spatialys.com


Sean Gillies
 

Hi Even,

On Fri, Nov 13, 2020 at 10:38 AM Even Rouault <even.rouault@...> wrote:
On vendredi 13 novembre 2020 09:05:41 CET Sean Gillies via groups.io wrote:
> HI Loïc,
>
> I don't think we currently have any option other than to remove the
> overviews. In theory GDAL's OVERVIEW_LEVEL open option could help us, but
> the implementation isn't there. For example, if you used gdalwarp with
> `-ovr NONE` it will set an internal overview level to the value of -1. But
> that bypasses GDALOpenEx (used by rasterio). GDALOpenEx won't let us pass
> `OVERVIEW_LEVEL=-1. It fails with a "Cannot open overview level -1" error.

Was left as an exercice for a contributor:
https://github.com/OSGeo/gdal/pull/3181

Even

Thanks!

--
Sean Gillies


Loïc Dutrieux
 

Many thanks Sean and Even,

So with rasterio compiled against gdal 3.3 I'll be able to force ignore overviews?
If that deserves a new rasterio unit test I'm happy to send a pull request in the coming days.

Cheers,
Loïc
________________________________________
From: main@rasterio.groups.io <main@rasterio.groups.io> on behalf of Sean Gillies via groups.io <sean=mapbox.com@groups.io>
Sent: 13 November 2020 20:57
To: main@rasterio.groups.io
Subject: Re: [rasterio] Is it possible to ignore existing overview when performing decimated read?

Hi Even,

On Fri, Nov 13, 2020 at 10:38 AM Even Rouault <even.rouault@...<mailto:even.rouault@...>> wrote:
On vendredi 13 novembre 2020 09:05:41 CET Sean Gillies via groups.io<https://urldefense.com/v3/__http://groups.io__;!!DOxrgLBm!U8N393pmVlI170WWlMCwvmrIV7A8JSl9WHWOD7E3re7ET8O3tNpqP7DhM2SV30dTeRRzVSk$> wrote:
HI Loïc,

I don't think we currently have any option other than to remove the
overviews. In theory GDAL's OVERVIEW_LEVEL open option could help us, but
the implementation isn't there. For example, if you used gdalwarp with
`-ovr NONE` it will set an internal overview level to the value of -1. But
that bypasses GDALOpenEx (used by rasterio). GDALOpenEx won't let us pass
`OVERVIEW_LEVEL=-1. It fails with a "Cannot open overview level -1" error.
Was left as an exercice for a contributor:
https://github.com/OSGeo/gdal/pull/3181<https://urldefense.com/v3/__https://github.com/OSGeo/gdal/pull/3181__;!!DOxrgLBm!U8N393pmVlI170WWlMCwvmrIV7A8JSl9WHWOD7E3re7ET8O3tNpqP7DhM2SV30dTK8LUNCg$>

Even

Thanks!

--
Sean Gillies


Sean Gillies
 

Hi Loïc,

A test that requires 3.3 would be welcome. 

The next time we have a rasterio release I will patch GDAL the GDAL library in the wheels we publish on PyPI.


On Mon, Nov 16, 2020, 3:26 AM Loïc Dutrieux <loic.dutrieux@...> wrote:
Many thanks Sean and Even,

So with rasterio compiled against gdal 3.3 I'll be able to force ignore overviews?
If that deserves a new rasterio unit test I'm happy to send a pull request in the coming days.

Cheers,
Loïc
________________________________________
From: main@rasterio.groups.io <main@rasterio.groups.io> on behalf of Sean Gillies via groups.io <sean=mapbox.com@groups.io>
Sent: 13 November 2020 20:57
To: main@rasterio.groups.io
Subject: Re: [rasterio] Is it possible to ignore existing overview when performing decimated read?

Hi Even,

On Fri, Nov 13, 2020 at 10:38 AM Even Rouault <even.rouault@...<mailto:even.rouault@...>> wrote:
On vendredi 13 novembre 2020 09:05:41 CET Sean Gillies via groups.io<https://urldefense.com/v3/__http://groups.io__;!!DOxrgLBm!U8N393pmVlI170WWlMCwvmrIV7A8JSl9WHWOD7E3re7ET8O3tNpqP7DhM2SV30dTeRRzVSk$> wrote:
> HI Loïc,
>
> I don't think we currently have any option other than to remove the
> overviews. In theory GDAL's OVERVIEW_LEVEL open option could help us, but
> the implementation isn't there. For example, if you used gdalwarp with
> `-ovr NONE` it will set an internal overview level to the value of -1. But
> that bypasses GDALOpenEx (used by rasterio). GDALOpenEx won't let us pass
> `OVERVIEW_LEVEL=-1. It fails with a "Cannot open overview level -1" error.

Was left as an exercice for a contributor:
https://github.com/OSGeo/gdal/pull/3181<https://urldefense.com/v3/__https://github.com/OSGeo/gdal/pull/3181__;!!DOxrgLBm!U8N393pmVlI170WWlMCwvmrIV7A8JSl9WHWOD7E3re7ET8O3tNpqP7DhM2SV30dTK8LUNCg$>

Even

Thanks