Date   
Re: NameError: name 'Window' is not defined

quinsen.joel@...
 

Thanks for both your help! I see the import statement in the docs now, cheers!

-Quin

Re: NameError: name 'Window' is not defined

Sean Gillies
 

Thanks for the help, rickD!

In https://github.com/mapbox/rasterio/commit/eebf54b74fa0f718cb8baaa44aa20616e405b4e4 I added the missing Window class import. Read the Docs should be updated in a couple minutes. I think this should help.


On Mon, May 6, 2019 at 1:38 PM <debboutr@...> wrote:

you just need the class in your namespace, try:

`from rasterio.windows import Window`

I would agree it's not easy from the docs to know where to go and find this.

cheers

 

~rickD



--
Sean Gillies

Re: NameError: name 'Window' is not defined

debboutr@...
 

you just need the class in your namespace, try:

`from rasterio.windows import Window`

I would agree it's not easy from the docs to know where to go and find this.

cheers

 

~rickD

Re: NameError: name 'Window' is not defined

Sean Gillies
 

Hi Quin,

The Window class is defined in the rasterio.windows module and you must explicitly import it into your program's code like this:

    from rasterio.windows import Window

Example code in our docs might obscure this need, if you see any docs that don't show import of the Window class, let me know and I'll fix them.

On Mon, May 6, 2019 at 1:23 PM <quinsen.joel@...> wrote:
Hello. I'm new to this forum, so let me know if there are any problems with my question here.

I am trying to use rasterio's windowed reading ability and I am hoping to use the following structure as shown on rasterios documentation page (https://rasterio.readthedocs.io/en/latest/topics/windowed-rw.html):
 
 
```
with rasterio.open('file.tif') as src:
   window = src.read(1, window=Window(0, 0, new_height, new_width)) 
```
 
 
However, when I try to do this I get:
 
```NameError: name 'Window' is not defined```
 
Shouldn't Window() be imported with rasterio? Is this a problem with the version of rasterio I have downloaded or am I missing something really easy here?
 
```
print (rasterio.__version__)
1.0.7
```
 
Thank you for your guidance,
Quin



--
Sean Gillies

NameError: name 'Window' is not defined

quinsen.joel@...
 

Hello. I'm new to this forum, so let me know if there are any problems with my question here.

I am trying to use rasterio's windowed reading ability and I am hoping to use the following structure as shown on rasterios documentation page (https://rasterio.readthedocs.io/en/latest/topics/windowed-rw.html):
 
 
```
with rasterio.open('file.tif') as src:
   window = src.read(1, window=Window(0, 0, new_height, new_width)) 
```
 
 
However, when I try to do this I get:
 
```NameError: name 'Window' is not defined```
 
Shouldn't Window() be imported with rasterio? Is this a problem with the version of rasterio I have downloaded or am I missing something really easy here?
 
```
print (rasterio.__version__)
1.0.7
```
 
Thank you for your guidance,
Quin

Re: WarpedVRT and resampling ?

vincent.sarago@...
 

Hi Sean, sorry for not being really clear. 

> Can you explain what processes produced those images? What program produced the blurry one and what program produced the non-blurry one? Which resampling algorithms were used? 

Those two image were produced by `rio-tiler`. One on linux (blurry) the other on Mac OS (both with wheels, but also tried with gdal sources). The resampling algorithm that produced blurry lines are non-nearest: cubic, bilinear, cubic spline.  

>  if you use the WarpedVRT, you should define its extents so that you don't need to use any resampling at all when you read from it. Also, I don't fully understand the edge effects when you resample in read().

in https://github.com/cogeotiff/rio-tiler/blob/master/rio_tiler/utils.py#L296-L321, what we do first is to calculated the bounds of the VRT (mercator tiles) and then do (decimated) reads. 

The interesting part is the bilinear resampling introduce the blurry lines only when using it on the `WarpedVRT` params and not on the `read`. 

here is a gist showing how to reproduce it: https://gist.github.com/vincentsarago/0b229741c3261a9a447c7ab5dbd9eb05

Re: WarpedVRT and resampling ?

Sean Gillies
 

Hi Vincent,

Can you explain what processes produced those images? What program produced the blurry one and what program produced the non-blurry one? Which resampling algorithms were used?

Yes, you can introduce resampling in two places. I think it is not a good idea to do so: if you use the WarpedVRT, you should define its extents so that you don't need to use any resampling at all when you read from it. Also, I don't fully understand the edge effects when you resample in read().


On Fri, Apr 19, 2019 at 9:37 PM <vincent.sarago@...> wrote:
This was previously discussed on multiple Issues and PR: 
https://github.com/mapbox/rasterio/issues/1206
https://github.com/mapbox/rasterio/pull/1238
https://github.com/mapbox/rasterio/pull/1239

While working with https://github.com/cogeotiff/rio-tiler on different data format (Int16, Float ...) I've encountered some strange behavior when using non-nearest resampling algorithm. 

## Contexte 

When working with `WarpedVRT` you can choose the resampling algorithm twice: 
```
with WarpedVRT(src, resampling=enums.Resampling.nearest) as vrt:
    arr = vrt.read(out_shape=(1, 512, 512, resampling=enums.Resampling.nearest))
```
Sean try to explain this in https://github.com/mapbox/rasterio/pull/1238#issuecomment-353123776

rio-tiler code: https://github.com/cogeotiff/rio-tiler/blob/master/rio_tiler/utils.py#L296-L321

## Behavior 

On the images attached we can see that there are `blurry` lines on the bottom. Those lines appear on the bottom of the tile returned by `rio-tiler` (bottom of the WarpedVRT returned value). 

Observation: 
- Only visible when working on Linux (https://github.com/RemotePixel/amazonlinux-gdal/tree/gdal2.4.0, or with rasterio wheels) 
- Only with Int16 data with negative NoData 
- Not visible when using gdalwarp directly
- Visible only when not using `resampling=enums.Resampling.nearest` in WarpedVRT params

I'm not 100% to understand what is going on and if I should or should not use `enums.Resampling.nearest`  as default resampling in WarpedVRT params so I'm looking for guidance here. 

Thanks,

Vincent 



--
Sean Gillies

WarpedVRT and resampling ?

vincent.sarago@...
 

This was previously discussed on multiple Issues and PR: 
https://github.com/mapbox/rasterio/issues/1206
https://github.com/mapbox/rasterio/pull/1238
https://github.com/mapbox/rasterio/pull/1239

While working with https://github.com/cogeotiff/rio-tiler on different data format (Int16, Float ...) I've encountered some strange behavior when using non-nearest resampling algorithm. 

## Contexte 

When working with `WarpedVRT` you can choose the resampling algorithm twice: 
```
with WarpedVRT(src, resampling=enums.Resampling.nearest) as vrt:
    arr = vrt.read(out_shape=(1, 512, 512, resampling=enums.Resampling.nearest))
```
Sean try to explain this in https://github.com/mapbox/rasterio/pull/1238#issuecomment-353123776

rio-tiler code: https://github.com/cogeotiff/rio-tiler/blob/master/rio_tiler/utils.py#L296-L321

## Behavior 

On the images attached we can see that there are `blurry` lines on the bottom. Those lines appear on the bottom of the tile returned by `rio-tiler` (bottom of the WarpedVRT returned value). 

Observation: 
- Only visible when working on Linux (https://github.com/RemotePixel/amazonlinux-gdal/tree/gdal2.4.0, or with rasterio wheels) 
- Only with Int16 data with negative NoData 
- Not visible when using gdalwarp directly
- Visible only when not using `resampling=enums.Resampling.nearest` in WarpedVRT params

I'm not 100% to understand what is going on and if I should or should not use `enums.Resampling.nearest`  as default resampling in WarpedVRT params so I'm looking for guidance here. 

Thanks,

Vincent 

Re: tempfile.NamedTemporaryFile behaving as /vsimem and eating all the machine memory

Sean Gillies
 

Vincent.

At https://github.com/mapbox/rasterio/blob/master/rasterio/__init__.py#L191, a big GeoTIFF is created in RAM. Then at https://github.com/mapbox/rasterio/blob/master/rasterio/__init__.py#L199 that GeoTIFF is read into memory *again* so that it can be written to the Python file object. There will be two copies in memory. It's terribly inefficient, but I don't want to spend the time to optimize this case when I should be documenting the limitations instead.

On Tue, Apr 16, 2019 at 12:47 PM <vincent.sarago@...> wrote:
Thanks Sean this is really helpful and love the `temp.name` solution. 

About the second point, do you have any idea why `/vsimem` driver need so much memory when exiting/closing ? Should I raise this to the gdal list? 



--
Sean Gillies

Re: tempfile.NamedTemporaryFile behaving as /vsimem and eating all the machine memory

vincent.sarago@...
 

Thanks Sean this is really helpful and love the `temp.name` solution. 

About the second point, do you have any idea why `/vsimem` driver need so much memory when exiting/closing ? Should I raise this to the gdal list? 

Re: tempfile.NamedTemporaryFile behaving as /vsimem and eating all the machine memory

Sean Gillies
 

Hi Vincent,

This is expected (if not well-documented) behavior. tempfile.NamedTemporaryFile() returns an open Python file object, not a filename. GDAL can't use a Python file object, so in that case rasterio.open reads all the bytes from the file object, copies them to the vsimem filesystem, and opens that vsimem file.

I think what you want do do is pass the name of the temp file object to GDAL. Like this:

with tempfile.NamedTemporaryFile() as temp:
    with rasterio.open(temp.name) as dataset:
        print(dataset)

No copy in the vsimem filesystem will be made.

On Tue, Apr 16, 2019 at 6:55 AM <vincent.sarago@...> wrote:
While working on https://github.com/cogeotiff/rio-cogeo/pull/75 we noticed strange behaviors with `vsimem` driver (this could be a GDAL but TBH). 

1. When using `
tempfile.NamedTemporaryFile()` Rasterio uses `vsimem` driver

with tempfile.NamedTemporaryFile() as tmpfile:

   print(tmpfile)

   with rasterio.open(tmpfile, "w", **meta) as tmp_dst:

       print(tmp_dst)

<tempfile._TemporaryFileWrapper object at 0x1151bb2e8>

<open DatasetWriter name='/vsimem/d26d4650-3010-4d39-8b0c-3d947b94f1d5.' mode='w+'>

Here I was expecting Rasterio/GDAL to behave as `tempfile` was a regular file.

2. When closing
 a `vsimem`  (`MemoryFile` or `tempfile`) we observe a huge memory surge when working with big images.

code: 
https://github.com/cogeotiff/rio-cogeo/pull/75#issuecomment-482745580



Tested on Mac OS and linux with python 3.7 (gdal 2.4 and 2.3) 

Thanks 



--
Sean Gillies

Re: Is it possible to create a rasterio object without exporting a file?

Leah Wasser
 

Thank you, Alan!
that example is very helpful. i'll dig into this a bit deeper. xarray seems like a powerful tool for raster processing.
i've wanted to plan with it a bit more regardless.
Leah

Re: Is it possible to create a rasterio object without exporting a file?

Alan Snow
 

Hi Leah,

If you would like to try the rioxarray as an xarray extension, it is now available on pypi.

An example of what you would like to do is here:
https://corteva.github.io/rioxarray/html/examples/clip_geom.html

Best,
Alan

tempfile.NamedTemporaryFile behaving as /vsimem and eating all the machine memory

vincent.sarago@...
 

While working on https://github.com/cogeotiff/rio-cogeo/pull/75 we noticed strange behaviors with `vsimem` driver (this could be a GDAL but TBH). 

1. When using `
tempfile.NamedTemporaryFile()` Rasterio uses `vsimem` driver

with tempfile.NamedTemporaryFile() as tmpfile:

   print(tmpfile)

   with rasterio.open(tmpfile, "w", **meta) as tmp_dst:

       print(tmp_dst)

<tempfile._TemporaryFileWrapper object at 0x1151bb2e8>

<open DatasetWriter name='/vsimem/d26d4650-3010-4d39-8b0c-3d947b94f1d5.' mode='w+'>

Here I was expecting Rasterio/GDAL to behave as `tempfile` was a regular file.

2. When closing
 a `vsimem`  (`MemoryFile` or `tempfile`) we observe a huge memory surge when working with big images.

code: 
https://github.com/cogeotiff/rio-cogeo/pull/75#issuecomment-482745580



Tested on Mac OS and linux with python 3.7 (gdal 2.4 and 2.3) 

Thanks 

Re: Is it possible to create a rasterio object without exporting a file?

Alan Snow
 

Hi Leah,

So, this project uses xarray which has many similar operations to numpy, but with slight differences.
In the case of the example, the `drop=True` is an option they provide that is described here:
http://xarray.pydata.org/en/stable/generated/xarray.DataArray.where.html

For numpy, it is not as simple if I remember correctly for a 2D array. An example of doing this is here:
https://github.com/CI-WATER/gsshapy/blob/00fd4af0fd65f1614d75a52fe950a04fb0867f4c/gsshapy/grid/grid_to_gssha.py#L616-L631

Hopefully this is helpful and gets you in the right direction.

Best,
Alan

Re: Is it possible to create a rasterio object without exporting a file?

Leah Wasser
 

Hey Alan,
this is super helpful.

digging through your code. how does this line work:

cropped_ds = self._obj.where(clip_mask_xray, drop=True).astype(self._obj.dtype)

i'm familiar with numpy.where however the drop=True is not an argument i'm familiar with.
thank you so much!
Leah

Re: Is it possible to create a rasterio object without exporting a file?

Alan Snow
 

Hi Leah,

Here is an example of what you want to do:
https://github.com/corteva/geocube/blob/582ea0c2f3f0ba91326de098b75412e5033bee21/geocube/xarray_extensions/rioxarray.py#L617

It uses xarray, but the concept is the same.

Hopefully this helps,
Alan

Is it possible to create a rasterio object without exporting a file?

Leah Wasser
 

Hi All,
I am running into a consistent workflow issue. i'd like to crop a numpy array. I have metadata dict and a numpy array but then want to crop it. is there a way to crop a numpy array to a new extent (using rio.mask()) using just the meta file rather than needing to export the array and reimport it to instantiate a rasterio object and create a new numpy array and associated meta?

Perhaps i'm missing something very basic!!
thank you
Leah

Re: Is it possible to open a binary file with rasterio.open?

Sean Gillies
 

Hi,

If you pass a Python file object opened in "r" mode to rasterio.open, the contents will be read and stored in a MemoryFile. See https://github.com/mapbox/rasterio/blob/master/rasterio/__init__.py#L176. Rasterio's open function doesn't accept bytes as an argument. Note that if the JP2 file is large, on the order of your computer's RAM, your program will be at risk of running out of memory.


On Sun, Apr 7, 2019 at 7:51 AM <htaidirt@...> wrote:
Hi all,

I've been using rasterio for few days, and now experiencing a situation where I can't find any help.

I read in the doc that when using `rasterio.open(fp,...)` it is possible to pass a binary content as `fp` (doc in the source file -> https://github.com/mapbox/rasterio/blob/c2df12979a5e07f96f108b0be8329e79fe950532/rasterio/__init__.py#L74

So instead of passing a file path or URL, I wanted to pass the content of the jp2 file to rasterio. Here is the full code for reproductability:


import boto3
import rasterio
s3 = boto3.resource('s3')

bucket = 'sentinel-s2-l1c'
key = 'tiles/31/U/DQ/2019/3/2/0/B01.jp2'

content = s3.Object(bucket, key).get(RequestPayer='requester')['Body'].read()
rio = rasterio.open(content, driver="JP2OpenJPEG").read(1)

But I'm getting an invalid path or file: b'\x00\x00\x00\x0cjP... error.

Any help on how to do so?

Thanks



--
Sean Gillies

Is it possible to open a binary file with rasterio.open?

htaidirt@...
 

Hi all,

I've been using rasterio for few days, and now experiencing a situation where I can't find any help.

I read in the doc that when using `rasterio.open(fp,...)` it is possible to pass a binary content as `fp` (doc in the source file -> https://github.com/mapbox/rasterio/blob/c2df12979a5e07f96f108b0be8329e79fe950532/rasterio/__init__.py#L74

So instead of passing a file path or URL, I wanted to pass the content of the jp2 file to rasterio. Here is the full code for reproductability:


import boto3
import rasterio
s3 = boto3.resource('s3')

bucket = 'sentinel-s2-l1c'
key = 'tiles/31/U/DQ/2019/3/2/0/B01.jp2'

content = s3.Object(bucket, key).get(RequestPayer='requester')['Body'].read()
rio = rasterio.open(content, driver="JP2OpenJPEG").read(1)

But I'm getting an invalid path or file: b'\x00\x00\x00\x0cjP... error.

Any help on how to do so?

Thanks