Date   

Re: metadata and profile: What's the difference?

Amine Aboufirass <amine.aboufirass@...>
 

Thanks for your reply. I guess the profile is more comprehensive then. Is that a good argument for dropping metadata then? 

On Fri, Nov 29, 2019 at 4:11 PM Sean Gillies <sean.gillies@...> wrote:
Indeed they are largely redundant. The meta property came first. Then we introduced a profile property to better support the pattern of using one dataset's profile as keyword arguments for creating new similar datasets. Note that the profile may contain items that don't appear in the meta property. 

On Fri, Nov 29, 2019, 9:57 AM Amine Aboufirass <amine.aboufirass@...> wrote:
Also, what is the difference between a dataset's metadata and profile? I have the impression one of these values is redundant. Please see example below:

>>> dataset.profile
{'driver': 'AAIGrid', 'dtype': 'int32', 'nodata': -9999.0, 'width': 8, 'height': 8, 'count': 1, 'crs': None, 'transform': Affine(1.0, 0.0, 0.0,
       0.0, -1.0, 8.0), 'tiled': False}
>>> dataset.meta
{'driver': 'AAIGrid', 'dtype': 'int32', 'nodata': -9999.0, 'width': 8, 'height': 8, 'count': 1, 'crs': None, 'transform': Affine(1.0, 0.0, 0.0,
       0.0, -1.0, 8.0)}


Re: metadata and profile: What's the difference?

Sean Gillies
 

Indeed they are largely redundant. The meta property came first. Then we introduced a profile property to better support the pattern of using one dataset's profile as keyword arguments for creating new similar datasets. Note that the profile may contain items that don't appear in the meta property. 


On Fri, Nov 29, 2019, 9:57 AM Amine Aboufirass <amine.aboufirass@...> wrote:
Also, what is the difference between a dataset's metadata and profile? I have the impression one of these values is redundant. Please see example below:

>>> dataset.profile
{'driver': 'AAIGrid', 'dtype': 'int32', 'nodata': -9999.0, 'width': 8, 'height': 8, 'count': 1, 'crs': None, 'transform': Affine(1.0, 0.0, 0.0,
       0.0, -1.0, 8.0), 'tiled': False}
>>> dataset.meta
{'driver': 'AAIGrid', 'dtype': 'int32', 'nodata': -9999.0, 'width': 8, 'height': 8, 'count': 1, 'crs': None, 'transform': Affine(1.0, 0.0, 0.0,
       0.0, -1.0, 8.0)}


metadata and profile: What's the difference?

Amine Aboufirass <amine.aboufirass@...>
 

Also, what is the difference between a dataset's metadata and profile? I have the impression one of these values is redundant. Please see example below:

>>> dataset.profile
{'driver': 'AAIGrid', 'dtype': 'int32', 'nodata': -9999.0, 'width': 8, 'height': 8, 'count': 1, 'crs': None, 'transform': Affine(1.0, 0.0, 0.0,
       0.0, -1.0, 8.0), 'tiled': False}
>>> dataset.meta
{'driver': 'AAIGrid', 'dtype': 'int32', 'nodata': -9999.0, 'width': 8, 'height': 8, 'count': 1, 'crs': None, 'transform': Affine(1.0, 0.0, 0.0,
       0.0, -1.0, 8.0)}


Re: Geodatabase, FileGeodatabase, and Esri Geodatabase

Sean Gillies
 

Hi Armstrong, 

Rasterio relies on a library named GDAL for data access and GDAL doesn't support Esri raster databases. There are people in the open source community who store raster data in a PostGIS database and manipulate it using SQL. I don't have any experience with that. My own applications use GeoTIFF and numpy arrays for storage and all the manipulation is done using the numpy and rasterio APIs, in python.

Open Data Cube is one project that aims to provide a more comprehensive raster data management system. It may be worth your time to look at it. I'm sure subscribers on this list can recommend other systems.

Yours, 

On Thu, Nov 28, 2019, 10:14 PM Armstrong Manuvakola Ezequias Ngolo <arngolo@...> wrote:
Hi everyone,

After relying on GIS softwares for a long time with a bit of python help with rasterio, I decided to move on using rasterio (and geopandas) for AI applications.

I would like to have access to some common tools that I have been using in GIS softwares!!!. 

One important thing is the ability to store and manipulate the datasets using only one file such as the Esri FileGeodatabase (FileGDB). 

My question is: Are there any FileGDBs that we can use in Python to store and manipulate spatial data?

I found that Fiona can create, read and write FileGDB but only for storing shapefile kind of datasets.

thank you in advance, 

Armstrong Ngolo


Problem with AAIGrid overwrite

Amine Aboufirass <amine.aboufirass@...>
 

Dear All,

For one project I am reading and writing these test raster datasets as ASCII grids. I have found that overwriting these grids is impossible and I get the following error:

ERROR 1: Deleting test_data/out.asc failed:
Permission denied

This is quite annoying because for my testing I need to overwrite and delete files repeatedly. I have to keep deleting them again before I run my application each time and this is quite time consuming.

I have the impression this is related to the GDAL creation options but I am not sure. Could someone please enlighten me? Thanks.

Regards,

Amine


Geodatabase, FileGeodatabase, and Esri Geodatabase

Armstrong Manuvakola Ezequias Ngolo
 

Hi everyone,

After relying on GIS softwares for a long time with a bit of python help with rasterio, I decided to move on using rasterio (and geopandas) for AI applications.

I would like to have access to some common tools that I have been using in GIS softwares!!!. 

One important thing is the ability to store and manipulate the datasets using only one file such as the Esri FileGeodatabase (FileGDB). 

My question is: Are there any FileGDBs that we can use in Python to store and manipulate spatial data?

I found that Fiona can create, read and write FileGDB but only for storing shapefile kind of datasets.

thank you in advance, 

Armstrong Ngolo


Re: reading raster with masked=True leads to error

Eyal Saiet
 

This is quite embarrassing, I cannot believe I did not catch that myself.

The link is very useful thanks!


On Fri, Nov 15, 2019 at 12:32 AM Guillaume Lostis <g.lostis@...> wrote:

Hi,

I think the reason you get an error is that you have misspelled the argument (you wrote maksed instead of masked).

FYI, the list of arguments of the read() function can be found in the documentation, here: https://rasterio.readthedocs.io/en/stable/api/rasterio.io.html#rasterio.io.DatasetReader.read

Best,

Guillaume Lostis



--


Eyal Saiet


The mind is not a vessel to be filled, but a fire to be kindled. Plutarch


Re: reading raster with masked=True leads to error

Guillaume Lostis
 

Hi,

I think the reason you get an error is that you have misspelled the argument (you wrote maksed instead of masked).

FYI, the list of arguments of the read() function can be found in the documentation, here: https://rasterio.readthedocs.io/en/stable/api/rasterio.io.html#rasterio.io.DatasetReader.read

Best,

Guillaume Lostis


reading raster with masked=True leads to error

Eyal Saiet
 

Hello,
I wanted to try using read(1,masked=True) on an imported DEM, but I get the following error below:
image.png

Since I could not verify what arguments the read() method accepts, I am asking here.

thanks

--


Eyal Saiet

The mind is not a vessel to be filled, but a fire to be kindled. Plutarch


Rasterio 1.1.1

Sean Gillies
 

Hi all,

Rasterio 1.1.1 source distribution and wheels for macosx and manylinux1 are on PyPI now. Here are the changes from the change log:

Bug fixes:

- Calling a dataset's sample method with coordinates outside the extent of a
  dataset with a nodata value of None has raised a TypeError (#1822). Now, it
  gives the values we would get from a boundless read of the dataset's values.
- Use new set_proj_search_path() function to set the PROJ data search path. For
  GDAL versions before 3.0 this sets the PROJ_LIB environment variable. For
  GDAL version 3.0 this calls OSRSetPROJSearchPaths(), which overrides
  PROJ_LIB (#1823).
- Check for header.dxf file instead of pcs.csv when looking for installed GDAL
  data. The latter is gone with GDAL 3.0 but the former remains (#1823).
- RGB rasters are now properly displayed by rasterio.plot.show (#1650).

Notes:

- The wheels on PyPI include GDAL 2.4.3 with a patch that addresses the
  multithreading issue reported in #1828.

Some of you have been reporting trouble with concurrent reads of GeoTIFFs on S3 and I'm optimistic that we have a fix for at least one class of these problems. Please give the 1.1.1 wheels a try (or patch your own installations of GDAL) and let us know how it goes.

--
Sean Gillies


Re: reproject from a tiff to another tiff gives "destination band 1 appears to be read only"

Sean Gillies
 

Hi,

It's a good thing that you opened the destination (population) file in "r" mode. Otherwise, it could have been overwritten.

You must create an output array for the reprojected data and pass it as the second argument, along with dst_crs and dst_transform keyword arguments.

with rasterio.open("population") as population, rasterio.open("lights") as lights:

    reprojected = numpy.zeros(population.shape, dtype=lights.dtypes[0])

    reprojected, transform = rasterio.warp.reproject(rasterio.band(lights, 1), reprojected, dst_crs=population.crs, dst_transform=population.transform)

The reprojected array will be aligned with and have the same shape as lights.read(1).

On Mon, Nov 11, 2019 at 6:28 AM <simonm3@...> wrote:
Please can someone explain how I align a tiff file of lights with another of population to create two numpy arrays the same size and aligned. I tried:

    population = rio.open(population)
    lights = rio.open(lights)
    res = rio.warp.reproject(rio.band(lights,1), rio.band(population,1))

This gives "CPLE_IllegalArgError: Destination band 1 appears to be read-only."

I was expecting it to read the meta data (crs, height, width and affine transform) from source and destination then return something with the data of source but the shape of destination.



--
Sean Gillies


reproject from a tiff to another tiff gives "destination band 1 appears to be read only"

simonm3@...
 

Please can someone explain how I align a tiff file of lights with another of population to create two numpy arrays the same size and aligned. I tried:

    population = rio.open(population)
    lights = rio.open(lights)
    res = rio.warp.reproject(rio.band(lights,1), rio.band(population,1))

This gives "CPLE_IllegalArgError: Destination band 1 appears to be read-only."

I was expecting it to read the meta data (crs, height, width and affine transform) from source and destination then return something with the data of source but the shape of destination.


Re: DataReader.crs is empty with new environment?

Denise Draper
 

Yup, that is it: the environment variables are not set.  I may have set them manually before.
I'll have to investigate how conda is supposed to handle this (since the variables should only be set when you are in the right environment), but in the meantime it seems to work to set them manually.

-- 
  Denise Draper
  draperd@...



Re: DataReader.crs is empty with new environment?

Yann-Sebastien Tremblay-Johnston
 

Is your GDAL_DATA  and/or PROJ_LIB environment variables are set in your new windows 10 environment? Typically these are set for you when you create a new conda environment and install GDAL. Is it possible that they are set but are somehow not readable? Have you tried opening another image and reading the crs? does it happen only with this one specific image?


DataReader.crs is empty with new environment?

Denise Draper
 

Hi ---

I just got a new (Windows 10) laptop, and recreated my software environment from my old laptop.  Code that had previously worked didn't.  In particular, on opening a file, I find the crs field is empty:

import rasterio
foo = rasterio.open('path_to_file.tif')
foo.crs  # returns nothing

I assume it is some sort of version difference at fault.   However, I have tried upgrading on an old machine and downgrading on the new one, and neither helped.   So far I've only looked at the rasterio version and the gdal version.  What else should I be looking at?    I use miniconda to create the software environment if that matters.

Here's a summary of version combinations that I tried

Old Windows 10 machine
rasterio version 1.0.22   gdal version 2040100 ==> crs is CRS.from_epsg(32629)

Cloud Linux VM
rasterio version 1.0.28  gdal version 3000100 ==> crs is CRS.from_epsg(32629)
rasterio version 1.1.00  gdal version 3000100 ==> crs is CRS.from_epsg(32629)

New Windows 10 machine
rasterio version 1.1.00  gdal version 3000100 ==> crs is NULL
rasterio version 1.0.28  gdal version 3000100 ==> crs is NULL

Also: I've independently verified (with gdalinfo) that the data file does have a proper CRS, which is the same from all machines.


Re: setting BIGTIFF='IF_SAFER' not solving BigTiff problems

Luke
 

How did you install rasterio?

If you installed via Anaconda / conda using the default channel, your gdal may lack bigtiff support.

https://github.com/ContinuumIO/anaconda-issues/issues/9887


setting BIGTIFF='IF_SAFER' not solving BigTiff problems

Amine Aboufirass <amine.aboufirass@...>
 

Dear All,

I would like to point to this issue in which I brought up the BIGTIFF limitation with regards to merging rasters using rasterio:


It turns out that the suggested solution of adding BIGTIFF="IF_SAFER" to the rasterio.open arguments does not remove the problem. I still get an error:

rasterio._err.CPLE_NotSupportedError: A 140401 pixels x 36001 lines x 1 bands Int16 image would be larger than 4GB but this is the largest size a TIFF can be, and BigTIFF is unavailable.  Creation failed.

Is there a way to circumvent or workaround this? Why is BIGTIFF="IF_SAFER" not working?

Thanks,

Amine


Re: Clarification on rio bounds --sequence/--collection

Sean Gillies
 

Hi,

I agree, it is confusing. And there is a bug here. The --sequence/--collection switch was intended to allow printing a sequence of JSON texts (GeoJSON feature or bbox) or a single GeoJSON feature collection. But the latter feature is disabled by the reuse of --collection to determined the type of JSON output. 


On Thu, Oct 17, 2019 at 10:44 PM Yann-Sebastien Tremblay-Johnston <yanns.tremblay@...> wrote:
Hi,

In the usage description of `rio bounds --help` 

```
...

 --sequence / --collection  Write a LF-delimited sequence of texts containing
                             individual objects (the default) or write a
                             single JSON text containing a feature collection
                             object.
...

 --collection               Output as GeoJSON feature collection(s).
```

confuses me. The first instance of `--collection` seems to imply that if several input files are passed like `rio bounds --collection *.tif`, then the output would be a single GeoJSON document with a single FeatureCollection containing multiple Features corresponding to each tif file expanded in the shell. The second instance of `--collection` is perhaps more honest in setting expectation that a FeatureCollection object will be created for each tif file. I'm wondering if the first I described was ever meant to be intended behavior? or has there been a regression? 

Thank you so much!



--
Sean Gillies


Re: masking using another raster

Sean Gillies
 

Hi Matt,

Yes, I think that's the only builtin command that will do it. Enhancing rio-stack to more clearly provide this feature might be feasible. Note that few formats support addition of a mask and that adding a mask generally requires creation of a new file.


On Fri, Oct 11, 2019 at 3:12 PM Gregory, Matthew <matt.gregory@...> wrote:
Hi all,

Simple question, but I want to make sure I'm doing this the canonical way. 

Assume two single-band rasters with identical projections/transforms and one of the rasters has NODATA pixels defining an irregular shape and 1 otherwise.  Is the best way to mask the other raster to use rio calc? e.g.:

  rio calc "(* (read 1) (read 2))" -o out.tif input.tif mask.tif

This works, but didn't know if a different utility is better suited.

thanks, matt





--
Sean Gillies


Re: reading hdf4 files with rasterio

Sean Gillies
 

Rasterio doesn't improve the situation with subdatasets very much. You would still need to discover them via a rasterio dataset's subdatasets property. Something like

with rasterio.open("file.hdf4") as dataset:
    for name in dataset.subdatasets:
        with rasterio.open(name) as subdataset:
            print(subdataset.profile)


On Mon, Oct 21, 2019 at 2:39 AM Amine Aboufirass <amine.aboufirass@...> wrote:
Dear All,

Is there a way to read and write hdf4 datasets using rasterio? I am working with DAAC data, for example https://e4ftl01.cr.usgs.gov//DP107/MOLA/MYD13Q1.006/2013.01.09/MYD13Q1.A2013009.h09v07.006.2015254175244.hdf

I found some examples of reading the data using gdal but extracting all relevant subdatasets and writing to a new GeoTiff using gdal would be a headache, I think.

Please let me know your thoughts. Thanks in advance for your help.

Regards,

Amine



--
Sean Gillies

341 - 360 of 698