Date
1 - 8 of 8
Reading NetCDF file as inMemoryFile
|
vincent.sarago@...
I'm seeking some advice here,
I'm trying to reduce the memory/disk usage of one of my script where I try to translate a netcdf file to COG. Ideally I'd love not to save any file to disk. The problem I'm facing is when opening the file from disk everything works fine and the netcdf is recognise as `netCDF` but when load the same file in memory it is then recognized as a HDF5... ``` import rasterio
from rasterio.io import MemoryFile
src_path = "OR_ABI-L1b-RadF-M6C04_G16_s20193221600287_e20193221609595_c20193221610025.nc"
with rasterio.open(src_path) as src_dst:
print(src_dst.name)
print(src_dst.meta)
OR_ABI-L1b-RadF-M6C04_G16_s20193221600287_e20193221609595_c20193221610025.nc
{'driver': 'netCDF', 'dtype': 'float_', 'nodata': None, 'width': 512, 'height': 512, 'count': 0, 'crs': None, 'transform': Affine(1.0, 0.0, 0.0,
0.0, 1.0, 0.0)}
with open(src_path, "rb") as f:
with rasterio.open(f) as src_dst:
with src_dst.open() as mem:
print(mem.name)
print(mem.meta)
/vsimem/8ee5dd37-9f49-47c9-bce7-a6732c72d4b5.nc
{'driver': 'HDF5', 'dtype': 'float_', 'nodata': None, 'width': 512, 'height': 512, 'count': 0, 'crs': None, 'transform': Affine(1.0, 0.0, 0.0,
0.0, 1.0, 0.0)}
```
The problem with HDF5 is that it seems to loose any geographical information and thus is not usable for the process I'm doing. thanks for your help |
|
|
|
Even Rouault
On vendredi 20 décembre 2019 05:03:36 CET vincent.sarago@... wrote:
I'm seeking some advice here,RTF(anstatic)M :-) https://gdal.org/drivers/raster/netcdf.html#vsi-virtual-file-system-api-support "Since GDAL 2.4, and with Linux kernel >=4.3 and libnetcdf >=4.5, read operations on /vsi file systems are supported." When building GDAL, you must see "NetCDF has netcdf_mem.h: yes" in the summary output of ./configure If you don't meet those requirements, /vsimem/ access on netCDF4/HDF5 files will fallback to the HDF5 driver, which has proper support for pluggable I/O since libhdf5 allows it The netCDF library has no pluggable I/O layer, hence the GDAL support uses the netCDF in-memory file API combined with the Linux userfaultfd mechanism to populated the in-memory mapping with data. That said, for a file hosted in /vsimem/ (to be opposed to /vsi network file systems), we could probably improve that to avoid any Linux specificities. Even -- Spatialys - Geospatial professional services http://www.spatialys.com |
|
|
|
Thanks for your answer Even, and be assured that I always read the manual before asking question :-)
``` $ more /proc/version
Linux version 4.9.184-linuxkit (root@a8c33e955a82) (gcc version 8.3.0 (Alpine 8.3.0) ) #1 SMP Tue Jul 2 22:58:16 UTC 2019 ... # From gdal /configure NetCDF support: yes
NetCDF has netcdf_mem.h: yes
```... # NetCDF install ENV NETCDF_VERSION=4.6.3
# NetCDF RUN mkdir /tmp/netcdf \
&& curl -sfL https://github.com/Unidata/netcdf-c/archive/v$NETCDF_VERSION.tar.gz | tar zxf - -C /tmp/netcdf --strip-components=1
RUN cd /tmp/netcdf \
&& CPPFLAGS="-I${PREFIX}/include" LDFLAGS="-L${PREFIX}/lib" \
./configure \
--with-default-chunk-size=67108864 \
--with-chunk-cache-size=67108864 \
--prefix=$PREFIX \
--disable-static \
--enable-netcdf4 \
--enable-dap \
--with-pic \
&& make -j $(nproc) --silent && make install && make clean \
&& rm -rf /tmp/netcdf
My configuration meets the current spec from the docs, and still fallback to HDF5 |
|
|
|
Sean Gillies
Vincent, Can you try naming the driver when you open the in-memory dataset? Something like with MemoryFile(...) as memfile: with memfile.open(driver="netCDF") as dataset: .... On Fri, Dec 20, 2019 at 8:23 AM <vincent.sarago@...> wrote:
-- Sean Gillies |
|
|
|
Even Rouault
On vendredi 20 décembre 2019 07:22:11 CET vincent.sarago@... wrote:
$ more /proc/versionCan you check for the following too ? userfaultfd support: yes to check that you actually built against sufficiently recent kernel headers. -- Spatialys - Geospatial professional services http://www.spatialys.com |
|
|
|
vincent.sarago@...
Then I get "not recognized as a supported file format."
``` >>> f = open("/local/OR_ABI-L1b-RadF-M6C04_G16_s20193221600287_e20193221609595_c20193221610025.nc", "rb")
>>> with MemoryFile(f) as mem:
... with mem.open(driver="netCDF") as mem_dst:
... print(mem_dst.meta)
...
Traceback (most recent call last):
File "rasterio/_base.pyx", line 216, in rasterio._base.DatasetBase.__init__
File "rasterio/_shim.pyx", line 78, in rasterio._shim.open_dataset
File "rasterio/_err.pyx", line 205, in rasterio._err.exc_wrap_pointer
rasterio._err.CPLE_OpenFailedError: '/vsimem/808d1a21-9154-43a9-a7a8-459294b33fe4.' not recognized as a supported file format. ``` same when using ``` with open("/local/OR_ABI-L1b-RadF-M6C04_G16_s20193221600287_e20193221609595_c20193221610025.nc", "rb") as f: with rasterio.open(f, driver="netCDF") as src_dst: Traceback (most recent call last):
File "rasterio/_base.pyx", line 216, in rasterio._base.DatasetBase.__init__
File "rasterio/_shim.pyx", line 78, in rasterio._shim.open_dataset
File "rasterio/_err.pyx", line 205, in rasterio._err.exc_wrap_pointer
rasterio._err.CPLE_OpenFailedError: '/vsimem/b370432d-9a53-4c40-9315-bc9ad408589d.' not recognized as a supported file format.
``` |
|
|
|
vincent.sarago@...
Yes it is set to yes,
here the full log https://gist.github.com/vincentsarago/36473e6322336e84cf928ef445db64cc |
|
|
|
Even Rouault
On vendredi 20 décembre 2019 09:45:45 CET vincent.sarago@... wrote:
Yes it is set to yes,You were responding to "userfaultfd support: yes" ? Hum, then I'm not sure. Are you running in a container ? Maybe there are some restrictions by default. Dunno. But you should see GDAL error messages if userfaulfd system calls fail at runtime. There are quite a lot of them in https://github.com/OSGeo/gdal/blob/master/gdal/port/cpl_userfaultfd.cpp And fallback to the magic solution of open source projects (the reason why we all use open source, right ;-) ?): take your favorite debugger and break at https://github.com/OSGeo/gdal/blob/master/gdal/frmts/netcdf/netcdfdataset.cpp#L7274 and follow what happens then... It should normally go to the call to nc_open_mem() -- Spatialys - Geospatial professional services http://www.spatialys.com |
|
|