|
Representing many raster files as one big file
rasterio@...
Hi all,
I'm new to rasterio and it's so far working very nicely for reading some GeoTIFF files (specifically, the EU DEM elevation data set). I have a question I hope someone can answer: it's about whether it's possible to take the 27 contiguous individual TIFF files that form this data set and represent them as if they were one big file. Currently I have to load each file in a loop to perform my analysis, but ideally I would like to be able to give all of the file names and their coordinates to some function that then lets me fetch data from anywhere in the whole dataset using a single `dataset.read` with my window settings. This would avoid me having to deal with the edges where the window I'm looking at would overlap with other tiles. This would ideally work without requiring the whole dataset to be in memory, since it's huge (10s of GB). Is this possible already in rasterio? Cheers, Sean
|
|
|
|
problems installing GDAL and Rasterio in virtual environment in Windows
Eyal Saiet
Hello, I was successful in installing GDAL and Rasterio in my Windows10 machine using pip. But for some reason, I am failing to do so in the virtual environment, getting the limited error message in red. I did try to install Numpy with no difficulty. Both GDAL and Rasterio wheel files were downloaded and installed successfully using pip on the system. But I am failing to instal the same files in the virtual environment. Thanks for any help,  -- Eyal Saiet
|
|
|
|
Re: Problem with AAIGrid overwrite
Sean Gillies
I suspect the error is coming from the gdal module function, not from rasterio. If it was coming from rasterio, you'd see a log message instead.
On Mon, Dec 2, 2019 at 3:32 AM Amine Aboufirass <amine.aboufirass@...> wrote:
--
Sean Gillies
|
|
|
|
Re: Problem with AAIGrid overwrite
Amine Aboufirass <amine.aboufirass@...>
The strange thing is that it actually updates the file but still throws the error.
On Mon, Dec 2, 2019 at 11:17 AM Amine Aboufirass via Groups.Io <amine.aboufirass=gmail.com@groups.io> wrote:
|
|
|
|
Re: Problem with AAIGrid overwrite
Amine Aboufirass <amine.aboufirass@...>
Dear Sean, I tried your example under an administrator command prompt and I got no problems. So I then tried running my own example under the same prompt and got the error again. I have attached a zip file which contains my example. Would you care to take a look? I really can't pinpoint why this behavior occurs. Thanks in advance for your help, Regards, Amine
On Fri, Nov 29, 2019 at 8:14 PM Sean Gillies <sean.gillies@...> wrote:
|
|
|
|
Re: Geodatabase, FileGeodatabase, and Esri Geodatabase
Alan Snow
Hello,
It appears the GeoPackage format can support vector and raster data in GDAL similar to the FIleGDB format. https://gdal.org/drivers/raster/gpkg.html https://gdal.org/drivers/vector/gpkg.html Not sure how to do both in the same file as I haven't tried to do it myself. But, sounds worth a shot. Hope that helps. Best, Alan
|
|
|
|
Re: Problem with AAIGrid overwrite
Sean Gillies
Amine, I'm unable to reproduce this with rasterio 1.1.0. The following script can be executed over and over again and the shade.asc file is overwritten with no error and no messages. import rasterio with rasterio.open("/Users/seang/code/rasterio/tests/data/shade.tif") as tif: profile = tif.profile profile["driver"] = "AAIGrid" with rasterio.open("/tmp/shade.asc", "w", **profile) as asc: asc.write(tif.read()) It might actually be a permissions issue on your computer.
On Fri, Nov 29, 2019 at 7:38 AM Amine Aboufirass <amine.aboufirass@...> wrote:
-- Sean Gillies
|
|
|
|
Re: metadata and profile: What's the difference?
Amine Aboufirass <amine.aboufirass@...>
Thanks for your reply. I guess the profile is more comprehensive then. Is that a good argument for dropping metadata then?
On Fri, Nov 29, 2019 at 4:11 PM Sean Gillies <sean.gillies@...> wrote:
|
|
|
|
Re: metadata and profile: What's the difference?
Sean Gillies
Indeed they are largely redundant. The meta property came first. Then we introduced a profile property to better support the pattern of using one dataset's profile as keyword arguments for creating new similar datasets. Note that the profile may contain items that don't appear in the meta property.
On Fri, Nov 29, 2019, 9:57 AM Amine Aboufirass <amine.aboufirass@...> wrote:
|
|
|
|
metadata and profile: What's the difference?
Amine Aboufirass <amine.aboufirass@...>
Also, what is the difference between a dataset's metadata and profile? I have the impression one of these values is redundant. Please see example below: >>> dataset.profile {'driver': 'AAIGrid', 'dtype': 'int32', 'nodata': -9999.0, 'width': 8, 'height': 8, 'count': 1, 'crs': None, 'transform': Affine(1.0, 0.0, 0.0, 0.0, -1.0, 8.0), 'tiled': False} >>> dataset.meta {'driver': 'AAIGrid', 'dtype': 'int32', 'nodata': -9999.0, 'width': 8, 'height': 8, 'count': 1, 'crs': None, 'transform': Affine(1.0, 0.0, 0.0, 0.0, -1.0, 8.0)}
|
|
|
|
Re: Geodatabase, FileGeodatabase, and Esri Geodatabase
Sean Gillies
Hi Armstrong, Rasterio relies on a library named GDAL for data access and GDAL doesn't support Esri raster databases. There are people in the open source community who store raster data in a PostGIS database and manipulate it using SQL. I don't have any experience with that. My own applications use GeoTIFF and numpy arrays for storage and all the manipulation is done using the numpy and rasterio APIs, in python. Open Data Cube is one project that aims to provide a more comprehensive raster data management system. It may be worth your time to look at it. I'm sure subscribers on this list can recommend other systems. Yours,
On Thu, Nov 28, 2019, 10:14 PM Armstrong Manuvakola Ezequias Ngolo <arngolo@...> wrote: Hi everyone,
|
|
|
|
Problem with AAIGrid overwrite
Amine Aboufirass <amine.aboufirass@...>
Dear All, For one project I am reading and writing these test raster datasets as ASCII grids. I have found that overwriting these grids is impossible and I get the following error: ERROR 1: Deleting test_data/out.asc failed: Permission denied This is quite annoying because for my testing I need to overwrite and delete files repeatedly. I have to keep deleting them again before I run my application each time and this is quite time consuming. I have the impression this is related to the GDAL creation options but I am not sure. Could someone please enlighten me? Thanks. Regards, Amine
|
|
|
|
Geodatabase, FileGeodatabase, and Esri Geodatabase
Armstrong Manuvakola Ezequias Ngolo
Hi everyone,
After relying on GIS softwares for a long time with a bit of python help with rasterio, I decided to move on using rasterio (and geopandas) for AI applications. I would like to have access to some common tools that I have been using in GIS softwares!!!. One important thing is the ability to store and manipulate the datasets using only one file such as the Esri FileGeodatabase (FileGDB). My question is: Are there any FileGDBs that we can use in Python to store and manipulate spatial data? I found that Fiona can create, read and write FileGDB but only for storing shapefile kind of datasets. thank you in advance, Armstrong Ngolo
|
|
|
|
Re: reading raster with masked=True leads to error
Eyal Saiet
This is quite embarrassing, I cannot believe I did not catch that myself. The link is very useful thanks!
On Fri, Nov 15, 2019 at 12:32 AM Guillaume Lostis <g.lostis@...> wrote:
-- Eyal Saiet The mind is not a vessel to be filled, but a fire to be kindled. Plutarch
|
|
|
|
Re: reading raster with masked=True leads to error
Guillaume Lostis <g.lostis@...>
Hi, I think the reason you get an error is that you have misspelled the argument (you wrote FYI, the list of arguments of the Best, Guillaume Lostis
|
|
|
|
reading raster with masked=True leads to error
Eyal Saiet
Hello, I wanted to try using read(1,masked=True) on an imported DEM, but I get the following error below:  Since I could not verify what arguments the read() method accepts, I am asking here. thanks -- Eyal Saiet
|
|
|
|
Rasterio 1.1.1
Sean Gillies
Hi all, Rasterio 1.1.1 source distribution and wheels for macosx and manylinux1 are on PyPI now. Here are the changes from the change log: Bug fixes: - Calling a dataset's sample method with coordinates outside the extent of a dataset with a nodata value of None has raised a TypeError (#1822). Now, it gives the values we would get from a boundless read of the dataset's values. - Use new set_proj_search_path() function to set the PROJ data search path. For GDAL versions before 3.0 this sets the PROJ_LIB environment variable. For GDAL version 3.0 this calls OSRSetPROJSearchPaths(), which overrides PROJ_LIB (#1823). - Check for header.dxf file instead of pcs.csv when looking for installed GDAL data. The latter is gone with GDAL 3.0 but the former remains (#1823). - RGB rasters are now properly displayed by rasterio.plot.show (#1650). Notes: - The wheels on PyPI include GDAL 2.4.3 with a patch that addresses the multithreading issue reported in #1828. Some of you have been reporting trouble with concurrent reads of GeoTIFFs on S3 and I'm optimistic that we have a fix for at least one class of these problems. Please give the 1.1.1 wheels a try (or patch your own installations of GDAL) and let us know how it goes. Sean Gillies
|
|
|
|
Re: reproject from a tiff to another tiff gives "destination band 1 appears to be read only"
Sean Gillies
Hi, It's a good thing that you opened the destination (population) file in "r" mode. Otherwise, it could have been overwritten. You must create an output array for the reprojected data and pass it as the second argument, along with dst_crs and dst_transform keyword arguments. with rasterio.open("population") as population, rasterio.open("lights") as lights: reprojected = numpy.zeros(population.shape, dtype=lights.dtypes[0]) reprojected, transform = rasterio.warp.reproject(rasterio.band(lights, 1), reprojected, dst_crs=population.crs, dst_transform=population.transform) The reprojected array will be aligned with and have the same shape as lights.read(1).
On Mon, Nov 11, 2019 at 6:28 AM <simonm3@...> wrote: Please can someone explain how I align a tiff file of lights with another of population to create two numpy arrays the same size and aligned. I tried: -- Sean Gillies
|
|
|
|
reproject from a tiff to another tiff gives "destination band 1 appears to be read only"
simonm3@...
Please can someone explain how I align a tiff file of lights with another of population to create two numpy arrays the same size and aligned. I tried:
population = rio.open(population) lights = rio.open(lights) res = rio.warp.reproject(rio.band(lights,1), rio.band(population,1)) This gives "CPLE_IllegalArgError: Destination band 1 appears to be read-only." I was expecting it to read the meta data (crs, height, width and affine transform) from source and destination then return something with the data of source but the shape of destination.
|
|
|
|
Re: DataReader.crs is empty with new environment?
Denise Draper
Yup, that is it: the environment variables are not set. I may have set them manually before. I'll have to investigate how conda is supposed to handle this (since the variables should only be set when you are in the right environment), but in the meantime it seems to work to set them manually. -- Denise Draper draperd@...
|
|
|