1. Topics
  2. Is it possible to open a binary file with rasterio.open?

Is it possible to open a binary file with rasterio.open?

htaidirt@...
 

Hi all,

I've been using rasterio for few days, and now experiencing a situation where I can't find any help.

I read in the doc that when using `rasterio.open(fp,...)` it is possible to pass a binary content as `fp` (doc in the source file -> https://github.com/mapbox/rasterio/blob/c2df12979a5e07f96f108b0be8329e79fe950532/rasterio/__init__.py#L74

So instead of passing a file path or URL, I wanted to pass the content of the jp2 file to rasterio. Here is the full code for reproductability:


import boto3
import rasterio
s3 = boto3.resource('s3')

bucket = 'sentinel-s2-l1c'
key = 'tiles/31/U/DQ/2019/3/2/0/B01.jp2'

content = s3.Object(bucket, key).get(RequestPayer='requester')['Body'].read()
rio = rasterio.open(content, driver="JP2OpenJPEG").read(1)

But I'm getting an invalid path or file: b'\x00\x00\x00\x0cjP... error.

Any help on how to do so?

Thanks
Sean Gillies
 

Hi,

If you pass a Python file object opened in "r" mode to rasterio.open, the contents will be read and stored in a MemoryFile. See https://github.com/mapbox/rasterio/blob/master/rasterio/__init__.py#L176. Rasterio's open function doesn't accept bytes as an argument. Note that if the JP2 file is large, on the order of your computer's RAM, your program will be at risk of running out of memory.


On Sun, Apr 7, 2019 at 7:51 AM <htaidirt@...> wrote:
Hi all,

I've been using rasterio for few days, and now experiencing a situation where I can't find any help.

I read in the doc that when using `rasterio.open(fp,...)` it is possible to pass a binary content as `fp` (doc in the source file -> https://github.com/mapbox/rasterio/blob/c2df12979a5e07f96f108b0be8329e79fe950532/rasterio/__init__.py#L74

So instead of passing a file path or URL, I wanted to pass the content of the jp2 file to rasterio. Here is the full code for reproductability:


import boto3
import rasterio
s3 = boto3.resource('s3')

bucket = 'sentinel-s2-l1c'
key = 'tiles/31/U/DQ/2019/3/2/0/B01.jp2'

content = s3.Object(bucket, key).get(RequestPayer='requester')['Body'].read()
rio = rasterio.open(content, driver="JP2OpenJPEG").read(1)

But I'm getting an invalid path or file: b'\x00\x00\x00\x0cjP... error.

Any help on how to do so?

Thanks



--
Sean Gillies
Ryan Avery
 

Hey all,

I'm trying something similar, trying to read a bytes object from a POST request, which originally comes from a tif file. Here's my code:

with open(image_bytes, 'rb') as f, MemoryFile(f) as memfile:
with memfile.open() as src:
arr = reshape_as_image(src.read())


but I get this error
```

TypeError: expected str, bytes or os.PathLike object, not _io.BytesIO

```

```
{

    "TaskId": 1828,

    "Status": "failed: <class 'TypeError'>\nTraceback (most recent call last):\n  File \"/app/keras_iNat_api/runserver.py\", line 60, in detect\n    arr_for_detection, image_for_drawing = keras_detector.open_image(image_bytes)\n  File \"./keras_detector.py\", line 32, in open_image\n    with open(image_bytes, 'rb') as f, MemoryFile(f) as memfile:\nTypeError: expected str, bytes or os.PathLike object, not _io.BytesIO\n",

    "Timestamp": "2019-07-22 18:33:06",

    "Endpoint": "uri"

 

}
```

I can' seem to find how to convert a _io.BytesIO object into a bytes object, any suggestions?

Cheers,
vincent.sarago@...
 

Hi Ryan, 
This is just an idea but can you try with

```
with open(image_bytes, 'rb') as f, MemoryFile(f.read()) as memfile:
    with memfile.open() as src:
        arr = reshape_as_image(src.read())
```
Ryan Avery
 

Thanks for the suggestion Vincent, it looks like I still get the error 

TypeError: expected str, bytes or os.PathLike object, not _io.BytesIO\

Because this happens in the python open(image_bytes, 'rb') call not the  MemoryFile call. 

When I tried following this post to convert the BytesIO object to a bytes object like so but I don't think it actually converts it to a bytes object

with open(image_bytes.read(), 'rb') as f, MemoryFile(f) as memfile:

I get this error: ValueError: embedded null byte within the python open() call

Any other suggestions are super appreciated!
Sean Gillies
 

Ryan,

It looks like you're trying to pass an opened file object into Python's open() function. That won't work. It only takes strings and paths.

On Mon, Jul 22, 2019 at 2:18 PM Ryan Avery <ravery@...> wrote:
Thanks for the suggestion Vincent, it looks like I still get the error 

TypeError: expected str, bytes or os.PathLike object, not _io.BytesIO\

Because this happens in the python open(image_bytes, 'rb') call not the  MemoryFile call. 

When I tried following this post to convert the BytesIO object to a bytes object like so but I don't think it actually converts it to a bytes object

with open(image_bytes.read(), 'rb') as f, MemoryFile(f) as memfile:

I get this error: ValueError: embedded null byte within the python open() call

Any other suggestions are super appreciated!




--
Sean Gillies
1 - 6 of 6