Topics

Writing ESRI sidecar files


James David Smith
 

Hello,

I'm trying to use rasterio to manipulate a file and then write it out,
along with accompanying ESRI-type files. So the output files will be:

raster.tif,
raster.tif.aux.xml
raster.tif.ovr

I think before writing the file I can do something like this to create
the ovr file.

rasterio.Env(TIFF_USE_OVR=True)

Is there a similar command to create the aux.xml file please?

Thanks, James


Sean Gillies
 

Hi James,

I have tried this myself, but I think you'll want to add both GDAL_PAM_ENABLED=YES and ESRI_XML_PAM=YES to the Env.



On Thu, Jun 27, 2019 at 11:11 AM James David Smith <james.david.smith@...> wrote:
Hello,

I'm trying to use rasterio to manipulate a file and then write it out,
along with accompanying ESRI-type files. So the output files will be:

raster.tif,
raster.tif.aux.xml
raster.tif.ovr

I think before writing the file I can do something like this to create
the ovr file.

rasterio.Env(TIFF_USE_OVR=True)

Is there a similar command to create the aux.xml file please?

Thanks, James





--
Sean Gillies


James David Smith
 

Thanks for the tip Sean, but that doesn't work for me.

with rasterio.Env(TIFF_USE_OVR=True, GDAL_PAM_ENABLED=YES, ESRI_XML_PAM=YES):
----- name 'YES' is not defined

If I change 'YES' to True then the code runs .... but I'm not sure
it's recognizing the commands as the aux.xml file isn't created (but
the .ovr is).

Anyone got a good idea?


On Thu, 27 Jun 2019 at 18:53, Sean Gillies via Groups.Io
<sean=mapbox.com@groups.io> wrote:

Hi James,

I have tried this myself, but I think you'll want to add both GDAL_PAM_ENABLED=YES and ESRI_XML_PAM=YES to the Env.

https://trac.osgeo.org/gdal/wiki/ConfigOptions#GDAL_PAM_ENABLED
https://gdal.org/drivers/raster/gtiff.html#configuration-options


On Thu, Jun 27, 2019 at 11:11 AM James David Smith <james.david.smith@gmail.com> wrote:

Hello,

I'm trying to use rasterio to manipulate a file and then write it out,
along with accompanying ESRI-type files. So the output files will be:

raster.tif,
raster.tif.aux.xml
raster.tif.ovr

I think before writing the file I can do something like this to create
the ovr file.

rasterio.Env(TIFF_USE_OVR=True)

Is there a similar command to create the aux.xml file please?

Thanks, James



--
Sean Gillies


Sean Gillies
 

Sorry, James, I messed up and used the GDAL vernacular. Replace YES with True and I think you'll be good to go.


On Thu, Jun 27, 2019 at 3:09 PM James David Smith <james.david.smith@...> wrote:
Thanks for the tip Sean, but that doesn't work for me.

with rasterio.Env(TIFF_USE_OVR=True, GDAL_PAM_ENABLED=YES, ESRI_XML_PAM=YES):
----- name 'YES' is not defined

If I change 'YES' to True then the code runs .... but I'm not sure
it's recognizing the commands as the aux.xml file isn't created (but
the .ovr is).

Anyone got a good idea?


On Thu, 27 Jun 2019 at 18:53, Sean Gillies via Groups.Io
<sean=mapbox.com@groups.io> wrote:
>
> Hi James,
>
> I have tried this myself, but I think you'll want to add both GDAL_PAM_ENABLED=YES and ESRI_XML_PAM=YES to the Env.
>
> https://trac.osgeo.org/gdal/wiki/ConfigOptions#GDAL_PAM_ENABLED
> https://gdal.org/drivers/raster/gtiff.html#configuration-options
>
>
> On Thu, Jun 27, 2019 at 11:11 AM James David Smith <james.david.smith@...> wrote:
>>
>> Hello,
>>
>> I'm trying to use rasterio to manipulate a file and then write it out,
>> along with accompanying ESRI-type files. So the output files will be:
>>
>> raster.tif,
>> raster.tif.aux.xml
>> raster.tif.ovr
>>
>> I think before writing the file I can do something like this to create
>> the ovr file.
>>
>> rasterio.Env(TIFF_USE_OVR=True)
>>
>> Is there a similar command to create the aux.xml file please?
>>
>> Thanks, James
>>
>>
>>
>
>
> --
> Sean Gillies
>





--
Sean Gillies


James David Smith
 

This didn't work unfortunately Sean. No errors .... but nothing else
was wrote out as I was hoping. If you get a few minutes would you mind
testing to ensure I'm not going crazy? Thanks, James

On Thu, 27 Jun 2019 at 23:05, Sean Gillies via Groups.Io
<sean=mapbox.com@groups.io> wrote:

Sorry, James, I messed up and used the GDAL vernacular. Replace YES with True and I think you'll be good to go.

On Thu, Jun 27, 2019 at 3:09 PM James David Smith <james.david.smith@gmail.com> wrote:

Thanks for the tip Sean, but that doesn't work for me.

with rasterio.Env(TIFF_USE_OVR=True, GDAL_PAM_ENABLED=YES, ESRI_XML_PAM=YES):
----- name 'YES' is not defined

If I change 'YES' to True then the code runs .... but I'm not sure
it's recognizing the commands as the aux.xml file isn't created (but
the .ovr is).

Anyone got a good idea?


On Thu, 27 Jun 2019 at 18:53, Sean Gillies via Groups.Io
<sean=mapbox.com@groups.io> wrote:

Hi James,

I have tried this myself, but I think you'll want to add both GDAL_PAM_ENABLED=YES and ESRI_XML_PAM=YES to the Env.

https://trac.osgeo.org/gdal/wiki/ConfigOptions#GDAL_PAM_ENABLED
https://gdal.org/drivers/raster/gtiff.html#configuration-options


On Thu, Jun 27, 2019 at 11:11 AM James David Smith <james.david.smith@gmail.com> wrote:

Hello,

I'm trying to use rasterio to manipulate a file and then write it out,
along with accompanying ESRI-type files. So the output files will be:

raster.tif,
raster.tif.aux.xml
raster.tif.ovr

I think before writing the file I can do something like this to create
the ovr file.

rasterio.Env(TIFF_USE_OVR=True)

Is there a similar command to create the aux.xml file please?

Thanks, James



--
Sean Gillies


--
Sean Gillies


Sean Gillies
 

If you are creating a GeoTIFF it is possible that no XML file will be written. As Even Rouault explains in https://lists.osgeo.org/pipermail/gdal-dev/2010-May/024522.html, the PAM file is only created if needed.

In the script below, I'm using JPEG as an output format and do see creation of a PAM file.

from rasterio import Env
from rasterio.shutil import copy

with Env(GDAL_PAM_ENABLED=True, ESRI_XML_PAM=True):
    copy("/Users/seang/code/rasterio/tests/data/RGB.byte.tif", "/tmp/foo.jpg", driver="JPEG")

$ ls -l foo.jpg*
-rw-r--r--  1 seang  wheel  103112 Jun 28 08:19 foo.jpg
-rw-r--r--  1 seang  wheel    2252 Jun 28 08:19 foo.jpg.aux.xml

I hope this gets you on the right track!


On Fri, Jun 28, 2019 at 7:36 AM James David Smith <james.david.smith@...> wrote:
This didn't work unfortunately Sean. No errors .... but nothing else
was wrote out as I was hoping. If you get a few minutes would you mind
testing to ensure I'm not going crazy? Thanks, James

On Thu, 27 Jun 2019 at 23:05, Sean Gillies via Groups.Io
<sean=mapbox.com@groups.io> wrote:
>
> Sorry, James, I messed up and used the GDAL vernacular. Replace YES with True and I think you'll be good to go.
>
> On Thu, Jun 27, 2019 at 3:09 PM James David Smith <james.david.smith@...> wrote:
>>
>> Thanks for the tip Sean, but that doesn't work for me.
>>
>> with rasterio.Env(TIFF_USE_OVR=True, GDAL_PAM_ENABLED=YES, ESRI_XML_PAM=YES):
>> ----- name 'YES' is not defined
>>
>> If I change 'YES' to True then the code runs .... but I'm not sure
>> it's recognizing the commands as the aux.xml file isn't created (but
>> the .ovr is).
>>
>> Anyone got a good idea?
>>
>>
>> On Thu, 27 Jun 2019 at 18:53, Sean Gillies via Groups.Io
>> <sean=mapbox.com@groups.io> wrote:
>> >
>> > Hi James,
>> >
>> > I have tried this myself, but I think you'll want to add both GDAL_PAM_ENABLED=YES and ESRI_XML_PAM=YES to the Env.
>> >
>> > https://trac.osgeo.org/gdal/wiki/ConfigOptions#GDAL_PAM_ENABLED
>> > https://gdal.org/drivers/raster/gtiff.html#configuration-options
>> >
>> >
>> > On Thu, Jun 27, 2019 at 11:11 AM James David Smith <james.david.smith@...> wrote:
>> >>
>> >> Hello,
>> >>
>> >> I'm trying to use rasterio to manipulate a file and then write it out,
>> >> along with accompanying ESRI-type files. So the output files will be:
>> >>
>> >> raster.tif,
>> >> raster.tif.aux.xml
>> >> raster.tif.ovr
>> >>
>> >> I think before writing the file I can do something like this to create
>> >> the ovr file.
>> >>
>> >> rasterio.Env(TIFF_USE_OVR=True)
>> >>
>> >> Is there a similar command to create the aux.xml file please?
>> >>
>> >> Thanks, James
>> >>
>> >>
>> >>
>> >
>> >
>> > --
>> > Sean Gillies
>> >
>>
>>
>>
>
>
> --
> Sean Gillies
>





--
Sean Gillies


Even Rouault
 

On vendredi 28 juin 2019 08:26:35 CEST Sean Gillies wrote:
If you are creating a GeoTIFF it is possible that no XML file will be
written. As Even Rouault explains in
https://lists.osgeo.org/pipermail/gdal-dev/2010-May/024522.html, the PAM
file is only created if needed.
Indeed. You can play with the GTIFF specific PROFILE creation option
(see https://gdal.org/drivers/raster/gtiff.html#creation-issues)

PROFILE=BASELINE: .aux.xml only created if presence of georeferencing and/or
user metadata
PROFILE=GEOTIFF: .aux.xml only created if there's user metadata
PROFILE=GDALGeoTIFF: .aux.xml only created if user metadata doesn't fit in
TIFF tag

Even

--
Spatialys - Geospatial professional services
http://www.spatialys.com


James David Smith
 

Thanks again Sean, and Evan too. It seems I just need to get the right
combination of options.

The options that you have shown Even ....how are they implemented?
They don't seem to go in the Env like the other options?

My code at the moment as follows. It create the .tif and .ovr file .
If anyone can improve it, and help me get the aux.xml file too I'd be
very grateful. I need the ovr and aux.xml files to be created
independently as it's what the client needs.
-------
import rasterio

with rasterio.open("meri50year.tif") as file:
data = file.read(1)
metadata = file.profile
metadata['nodata'] = 255
data[data == 1 ] = 4
data[data == 15 ] = 255

outfile = 'meri50_processed.tif'

with rasterio.Env(GDAL_PAM_ENABLED=True, ESRI_XML_PAM=True, TIFF_USE_OVR=True):
with rasterio.open(outfile, 'w', **metadata) as new_temp_file:
new_temp_file.write(data, 1)
overviews = [2,4,8,16]
new_temp_file.build_overviews(overviews, Resampling.nearest)
--------

On Fri, 28 Jun 2019 at 16:09, Even Rouault <even.rouault@spatialys.com> wrote:

On vendredi 28 juin 2019 08:26:35 CEST Sean Gillies wrote:
If you are creating a GeoTIFF it is possible that no XML file will be
written. As Even Rouault explains in
https://lists.osgeo.org/pipermail/gdal-dev/2010-May/024522.html, the PAM
file is only created if needed.
Indeed. You can play with the GTIFF specific PROFILE creation option
(see https://gdal.org/drivers/raster/gtiff.html#creation-issues)

PROFILE=BASELINE: .aux.xml only created if presence of georeferencing and/or
user metadata
PROFILE=GEOTIFF: .aux.xml only created if there's user metadata
PROFILE=GDALGeoTIFF: .aux.xml only created if user metadata doesn't fit in
TIFF tag

Even

--
Spatialys - Geospatial professional services
http://www.spatialys.com



Sean Gillies
 

Hi,

On Fri, Jun 28, 2019 at 10:00 AM James David Smith <james.david.smith@...> wrote:
Thanks again Sean, and Evan too. It seems I just need to get the right
combination of options.

The options that you have shown Even ....how are they implemented?
They don't seem to go in the Env like the other options?

Correct, those dataset creation options need to be passed into rasterio.open().

One way to do that is to add to your metadata dict below

  metadata["profile"] = "GeoTIFF"

Any extra keyword argument passed to rasterio.open() is collected into a kwargs dict (see https://rasterio.readthedocs.io/en/latest/api/rasterio.html#rasterio.open) and items of that dict are turned into GDAL dataset creation options (by uppercasing the keyword argument names and converting their values appropriately).


My code at the moment as follows. It create the .tif and .ovr file .
If anyone can improve it, and help me get the aux.xml file too I'd be
very grateful. I need the ovr and aux.xml files to be created
independently as it's what the client needs.
-------
import rasterio

with rasterio.open("meri50year.tif") as file:
    data               = file.read(1)
    metadata           = file.profile
    metadata['nodata'] = 255
    data[data == 1 ]   = 4
    data[data == 15 ]  = 255

outfile = 'meri50_processed.tif'

with rasterio.Env(GDAL_PAM_ENABLED=True, ESRI_XML_PAM=True, TIFF_USE_OVR=True):
    with rasterio.open(outfile, 'w', **metadata) as new_temp_file:
        new_temp_file.write(data, 1)
        overviews = [2,4,8,16]
        new_temp_file.build_overviews(overviews, Resampling.nearest)
--------


On Fri, 28 Jun 2019 at 16:09, Even Rouault <even.rouault@...> wrote:
>
> On vendredi 28 juin 2019 08:26:35 CEST Sean Gillies wrote:
> > If you are creating a GeoTIFF it is possible that no XML file will be
> > written. As Even Rouault explains in
> > https://lists.osgeo.org/pipermail/gdal-dev/2010-May/024522.html, the PAM
> > file is only created if needed.
>
> Indeed. You can play with the GTIFF specific PROFILE creation option
> (see https://gdal.org/drivers/raster/gtiff.html#creation-issues)
>
> PROFILE=BASELINE: .aux.xml only created if presence of georeferencing and/or
> user metadata
> PROFILE=GEOTIFF: .aux.xml only created if there's user metadata
> PROFILE=GDALGeoTIFF: .aux.xml only created if user metadata doesn't fit in
> TIFF tag
>
> Even
>
> --
> Spatialys - Geospatial professional services
> http://www.spatialys.com
>
>
>





--
Sean Gillies