Date   

Representing many raster files as one big file

rasterio@...
 

Hi all,

I'm new to rasterio and it's so far working very nicely for reading some GeoTIFF files (specifically, the EU DEM elevation data set). I have a question I hope someone can answer: it's about whether it's possible to take the 27 contiguous individual TIFF files that form this data set and represent them as if they were one big file.

Currently I have to load each file in a loop to perform my analysis, but ideally I would like to be able to give all of the file names and their coordinates to some function that then lets me fetch data from anywhere in the whole dataset using a single `dataset.read` with my window settings. This would avoid me having to deal with the edges where the window I'm looking at would overlap with other tiles. This would ideally work without requiring the whole dataset to be in memory, since it's huge (10s of GB).

Is this possible already in rasterio?

Cheers,


Sean


problems installing GDAL and Rasterio in virtual environment in Windows

Eyal Saiet
 

Hello,
I was successful in installing GDAL and Rasterio in my Windows10 machine using pip. But for some reason, I am failing to do so in the virtual environment, getting the limited error message in red. I did try to install Numpy with no difficulty. Both GDAL and Rasterio wheel files were downloaded and installed successfully using pip on the system. But I am failing to instal the same files in the virtual environment.
Thanks for any help,
image.png

--


Eyal Saiet

The mind is not a vessel to be filled, but a fire to be kindled. Plutarch


Re: Problem with AAIGrid overwrite

Sean Gillies
 

I suspect the error is coming from the gdal module function, not from rasterio. If it was coming from rasterio, you'd see a log message instead.


On Mon, Dec 2, 2019 at 3:32 AM Amine Aboufirass <amine.aboufirass@...> wrote:
The strange thing is that it actually updates the file but still throws the error. 

On Mon, Dec 2, 2019 at 11:17 AM Amine Aboufirass via Groups.Io <amine.aboufirass=gmail.com@groups.io> wrote:
Dear Sean,

I tried your example under an administrator command prompt and I got no problems. So I then tried running my own example under the same prompt and got the error again. 

I have attached a zip file which contains my example. Would you care to take a look? I really can't pinpoint why this behavior occurs.

Thanks in advance for your help,

Regards,

Amine
 
...

--
Sean Gillies


Re: Problem with AAIGrid overwrite

Amine Aboufirass <amine.aboufirass@...>
 

The strange thing is that it actually updates the file but still throws the error. 

On Mon, Dec 2, 2019 at 11:17 AM Amine Aboufirass via Groups.Io <amine.aboufirass=gmail.com@groups.io> wrote:
Dear Sean,

I tried your example under an administrator command prompt and I got no problems. So I then tried running my own example under the same prompt and got the error again. 

I have attached a zip file which contains my example. Would you care to take a look? I really can't pinpoint why this behavior occurs.

Thanks in advance for your help,

Regards,

Amine

On Fri, Nov 29, 2019 at 8:14 PM Sean Gillies <sean.gillies@...> wrote:
Amine,

I'm unable to reproduce this with rasterio 1.1.0. The following script can be executed over and over again and the shade.asc file is overwritten with no error and no messages.

import rasterio

with rasterio.open("/Users/seang/code/rasterio/tests/data/shade.tif") as tif:
    profile = tif.profile
    profile["driver"] = "AAIGrid"
    with rasterio.open("/tmp/shade.asc", "w", **profile) as asc:
        asc.write(tif.read())

It might actually be a permissions issue on your computer.


On Fri, Nov 29, 2019 at 7:38 AM Amine Aboufirass <amine.aboufirass@...> wrote:
Dear All,

For one project I am reading and writing these test raster datasets as ASCII grids. I have found that overwriting these grids is impossible and I get the following error:

ERROR 1: Deleting test_data/out.asc failed:
Permission denied

This is quite annoying because for my testing I need to overwrite and delete files repeatedly. I have to keep deleting them again before I run my application each time and this is quite time consuming.

I have the impression this is related to the GDAL creation options but I am not sure. Could someone please enlighten me? Thanks.

Regards,

Amine



--
Sean Gillies


Re: Problem with AAIGrid overwrite

Amine Aboufirass <amine.aboufirass@...>
 

Dear Sean,

I tried your example under an administrator command prompt and I got no problems. So I then tried running my own example under the same prompt and got the error again. 

I have attached a zip file which contains my example. Would you care to take a look? I really can't pinpoint why this behavior occurs.

Thanks in advance for your help,

Regards,

Amine

On Fri, Nov 29, 2019 at 8:14 PM Sean Gillies <sean.gillies@...> wrote:
Amine,

I'm unable to reproduce this with rasterio 1.1.0. The following script can be executed over and over again and the shade.asc file is overwritten with no error and no messages.

import rasterio

with rasterio.open("/Users/seang/code/rasterio/tests/data/shade.tif") as tif:
    profile = tif.profile
    profile["driver"] = "AAIGrid"
    with rasterio.open("/tmp/shade.asc", "w", **profile) as asc:
        asc.write(tif.read())

It might actually be a permissions issue on your computer.


On Fri, Nov 29, 2019 at 7:38 AM Amine Aboufirass <amine.aboufirass@...> wrote:
Dear All,

For one project I am reading and writing these test raster datasets as ASCII grids. I have found that overwriting these grids is impossible and I get the following error:

ERROR 1: Deleting test_data/out.asc failed:
Permission denied

This is quite annoying because for my testing I need to overwrite and delete files repeatedly. I have to keep deleting them again before I run my application each time and this is quite time consuming.

I have the impression this is related to the GDAL creation options but I am not sure. Could someone please enlighten me? Thanks.

Regards,

Amine



--
Sean Gillies


Re: Geodatabase, FileGeodatabase, and Esri Geodatabase

Alan Snow
 

Hello,

It appears the GeoPackage format can support vector and raster data in GDAL similar to the FIleGDB format.

https://gdal.org/drivers/raster/gpkg.html
https://gdal.org/drivers/vector/gpkg.html

Not sure how to do both in the same file as I haven't tried to do it myself. But, sounds worth a shot. Hope that helps.

Best,
Alan


Re: Problem with AAIGrid overwrite

Sean Gillies
 

Amine,

I'm unable to reproduce this with rasterio 1.1.0. The following script can be executed over and over again and the shade.asc file is overwritten with no error and no messages.

import rasterio

with rasterio.open("/Users/seang/code/rasterio/tests/data/shade.tif") as tif:
    profile = tif.profile
    profile["driver"] = "AAIGrid"
    with rasterio.open("/tmp/shade.asc", "w", **profile) as asc:
        asc.write(tif.read())

It might actually be a permissions issue on your computer.


On Fri, Nov 29, 2019 at 7:38 AM Amine Aboufirass <amine.aboufirass@...> wrote:
Dear All,

For one project I am reading and writing these test raster datasets as ASCII grids. I have found that overwriting these grids is impossible and I get the following error:

ERROR 1: Deleting test_data/out.asc failed:
Permission denied

This is quite annoying because for my testing I need to overwrite and delete files repeatedly. I have to keep deleting them again before I run my application each time and this is quite time consuming.

I have the impression this is related to the GDAL creation options but I am not sure. Could someone please enlighten me? Thanks.

Regards,

Amine



--
Sean Gillies


Re: metadata and profile: What's the difference?

Amine Aboufirass <amine.aboufirass@...>
 

Thanks for your reply. I guess the profile is more comprehensive then. Is that a good argument for dropping metadata then? 

On Fri, Nov 29, 2019 at 4:11 PM Sean Gillies <sean.gillies@...> wrote:
Indeed they are largely redundant. The meta property came first. Then we introduced a profile property to better support the pattern of using one dataset's profile as keyword arguments for creating new similar datasets. Note that the profile may contain items that don't appear in the meta property. 

On Fri, Nov 29, 2019, 9:57 AM Amine Aboufirass <amine.aboufirass@...> wrote:
Also, what is the difference between a dataset's metadata and profile? I have the impression one of these values is redundant. Please see example below:

>>> dataset.profile
{'driver': 'AAIGrid', 'dtype': 'int32', 'nodata': -9999.0, 'width': 8, 'height': 8, 'count': 1, 'crs': None, 'transform': Affine(1.0, 0.0, 0.0,
       0.0, -1.0, 8.0), 'tiled': False}
>>> dataset.meta
{'driver': 'AAIGrid', 'dtype': 'int32', 'nodata': -9999.0, 'width': 8, 'height': 8, 'count': 1, 'crs': None, 'transform': Affine(1.0, 0.0, 0.0,
       0.0, -1.0, 8.0)}


Re: metadata and profile: What's the difference?

Sean Gillies
 

Indeed they are largely redundant. The meta property came first. Then we introduced a profile property to better support the pattern of using one dataset's profile as keyword arguments for creating new similar datasets. Note that the profile may contain items that don't appear in the meta property. 


On Fri, Nov 29, 2019, 9:57 AM Amine Aboufirass <amine.aboufirass@...> wrote:
Also, what is the difference between a dataset's metadata and profile? I have the impression one of these values is redundant. Please see example below:

>>> dataset.profile
{'driver': 'AAIGrid', 'dtype': 'int32', 'nodata': -9999.0, 'width': 8, 'height': 8, 'count': 1, 'crs': None, 'transform': Affine(1.0, 0.0, 0.0,
       0.0, -1.0, 8.0), 'tiled': False}
>>> dataset.meta
{'driver': 'AAIGrid', 'dtype': 'int32', 'nodata': -9999.0, 'width': 8, 'height': 8, 'count': 1, 'crs': None, 'transform': Affine(1.0, 0.0, 0.0,
       0.0, -1.0, 8.0)}


metadata and profile: What's the difference?

Amine Aboufirass <amine.aboufirass@...>
 

Also, what is the difference between a dataset's metadata and profile? I have the impression one of these values is redundant. Please see example below:

>>> dataset.profile
{'driver': 'AAIGrid', 'dtype': 'int32', 'nodata': -9999.0, 'width': 8, 'height': 8, 'count': 1, 'crs': None, 'transform': Affine(1.0, 0.0, 0.0,
       0.0, -1.0, 8.0), 'tiled': False}
>>> dataset.meta
{'driver': 'AAIGrid', 'dtype': 'int32', 'nodata': -9999.0, 'width': 8, 'height': 8, 'count': 1, 'crs': None, 'transform': Affine(1.0, 0.0, 0.0,
       0.0, -1.0, 8.0)}


Re: Geodatabase, FileGeodatabase, and Esri Geodatabase

Sean Gillies
 

Hi Armstrong, 

Rasterio relies on a library named GDAL for data access and GDAL doesn't support Esri raster databases. There are people in the open source community who store raster data in a PostGIS database and manipulate it using SQL. I don't have any experience with that. My own applications use GeoTIFF and numpy arrays for storage and all the manipulation is done using the numpy and rasterio APIs, in python.

Open Data Cube is one project that aims to provide a more comprehensive raster data management system. It may be worth your time to look at it. I'm sure subscribers on this list can recommend other systems.

Yours, 

On Thu, Nov 28, 2019, 10:14 PM Armstrong Manuvakola Ezequias Ngolo <arngolo@...> wrote:
Hi everyone,

After relying on GIS softwares for a long time with a bit of python help with rasterio, I decided to move on using rasterio (and geopandas) for AI applications.

I would like to have access to some common tools that I have been using in GIS softwares!!!. 

One important thing is the ability to store and manipulate the datasets using only one file such as the Esri FileGeodatabase (FileGDB). 

My question is: Are there any FileGDBs that we can use in Python to store and manipulate spatial data?

I found that Fiona can create, read and write FileGDB but only for storing shapefile kind of datasets.

thank you in advance, 

Armstrong Ngolo


Problem with AAIGrid overwrite

Amine Aboufirass <amine.aboufirass@...>
 

Dear All,

For one project I am reading and writing these test raster datasets as ASCII grids. I have found that overwriting these grids is impossible and I get the following error:

ERROR 1: Deleting test_data/out.asc failed:
Permission denied

This is quite annoying because for my testing I need to overwrite and delete files repeatedly. I have to keep deleting them again before I run my application each time and this is quite time consuming.

I have the impression this is related to the GDAL creation options but I am not sure. Could someone please enlighten me? Thanks.

Regards,

Amine


Geodatabase, FileGeodatabase, and Esri Geodatabase

Armstrong Manuvakola Ezequias Ngolo
 

Hi everyone,

After relying on GIS softwares for a long time with a bit of python help with rasterio, I decided to move on using rasterio (and geopandas) for AI applications.

I would like to have access to some common tools that I have been using in GIS softwares!!!. 

One important thing is the ability to store and manipulate the datasets using only one file such as the Esri FileGeodatabase (FileGDB). 

My question is: Are there any FileGDBs that we can use in Python to store and manipulate spatial data?

I found that Fiona can create, read and write FileGDB but only for storing shapefile kind of datasets.

thank you in advance, 

Armstrong Ngolo


Re: reading raster with masked=True leads to error

Eyal Saiet
 

This is quite embarrassing, I cannot believe I did not catch that myself.

The link is very useful thanks!


On Fri, Nov 15, 2019 at 12:32 AM Guillaume Lostis <g.lostis@...> wrote:

Hi,

I think the reason you get an error is that you have misspelled the argument (you wrote maksed instead of masked).

FYI, the list of arguments of the read() function can be found in the documentation, here: https://rasterio.readthedocs.io/en/stable/api/rasterio.io.html#rasterio.io.DatasetReader.read

Best,

Guillaume Lostis



--


Eyal Saiet


The mind is not a vessel to be filled, but a fire to be kindled. Plutarch


Re: reading raster with masked=True leads to error

Guillaume Lostis <g.lostis@...>
 

Hi,

I think the reason you get an error is that you have misspelled the argument (you wrote maksed instead of masked).

FYI, the list of arguments of the read() function can be found in the documentation, here: https://rasterio.readthedocs.io/en/stable/api/rasterio.io.html#rasterio.io.DatasetReader.read

Best,

Guillaume Lostis


reading raster with masked=True leads to error

Eyal Saiet
 

Hello,
I wanted to try using read(1,masked=True) on an imported DEM, but I get the following error below:
image.png

Since I could not verify what arguments the read() method accepts, I am asking here.

thanks

--


Eyal Saiet

The mind is not a vessel to be filled, but a fire to be kindled. Plutarch


Rasterio 1.1.1

Sean Gillies
 

Hi all,

Rasterio 1.1.1 source distribution and wheels for macosx and manylinux1 are on PyPI now. Here are the changes from the change log:

Bug fixes:

- Calling a dataset's sample method with coordinates outside the extent of a
  dataset with a nodata value of None has raised a TypeError (#1822). Now, it
  gives the values we would get from a boundless read of the dataset's values.
- Use new set_proj_search_path() function to set the PROJ data search path. For
  GDAL versions before 3.0 this sets the PROJ_LIB environment variable. For
  GDAL version 3.0 this calls OSRSetPROJSearchPaths(), which overrides
  PROJ_LIB (#1823).
- Check for header.dxf file instead of pcs.csv when looking for installed GDAL
  data. The latter is gone with GDAL 3.0 but the former remains (#1823).
- RGB rasters are now properly displayed by rasterio.plot.show (#1650).

Notes:

- The wheels on PyPI include GDAL 2.4.3 with a patch that addresses the
  multithreading issue reported in #1828.

Some of you have been reporting trouble with concurrent reads of GeoTIFFs on S3 and I'm optimistic that we have a fix for at least one class of these problems. Please give the 1.1.1 wheels a try (or patch your own installations of GDAL) and let us know how it goes.

--
Sean Gillies


Re: reproject from a tiff to another tiff gives "destination band 1 appears to be read only"

Sean Gillies
 

Hi,

It's a good thing that you opened the destination (population) file in "r" mode. Otherwise, it could have been overwritten.

You must create an output array for the reprojected data and pass it as the second argument, along with dst_crs and dst_transform keyword arguments.

with rasterio.open("population") as population, rasterio.open("lights") as lights:

    reprojected = numpy.zeros(population.shape, dtype=lights.dtypes[0])

    reprojected, transform = rasterio.warp.reproject(rasterio.band(lights, 1), reprojected, dst_crs=population.crs, dst_transform=population.transform)

The reprojected array will be aligned with and have the same shape as lights.read(1).

On Mon, Nov 11, 2019 at 6:28 AM <simonm3@...> wrote:
Please can someone explain how I align a tiff file of lights with another of population to create two numpy arrays the same size and aligned. I tried:

    population = rio.open(population)
    lights = rio.open(lights)
    res = rio.warp.reproject(rio.band(lights,1), rio.band(population,1))

This gives "CPLE_IllegalArgError: Destination band 1 appears to be read-only."

I was expecting it to read the meta data (crs, height, width and affine transform) from source and destination then return something with the data of source but the shape of destination.



--
Sean Gillies


reproject from a tiff to another tiff gives "destination band 1 appears to be read only"

simonm3@...
 

Please can someone explain how I align a tiff file of lights with another of population to create two numpy arrays the same size and aligned. I tried:

    population = rio.open(population)
    lights = rio.open(lights)
    res = rio.warp.reproject(rio.band(lights,1), rio.band(population,1))

This gives "CPLE_IllegalArgError: Destination band 1 appears to be read-only."

I was expecting it to read the meta data (crs, height, width and affine transform) from source and destination then return something with the data of source but the shape of destination.


Re: DataReader.crs is empty with new environment?

Denise Draper
 

Yup, that is it: the environment variables are not set.  I may have set them manually before.
I'll have to investigate how conda is supposed to handle this (since the variables should only be set when you are in the right environment), but in the meantime it seems to work to set them manually.

-- 
  Denise Draper
  draperd@...


581 - 600 of 945