Topics

[rasterio-dev] Writing azure blobs does not create blobs

Sean Gillies
 

Hi Niklas,

First off, in my original reply I missed the chance to suggest that we move this discussion to the main (not dev) discussion group, which has more users. The dev group is for discussing and planning new features for the library. I'm going to cross post over to main@rasterio.groups.io and would appreciate it if you would follow up there.

Nothing in the logs jumps out at me, so I'm stumped. I don't have any experience using Azure blob storage and have never used GDAL or Rasterio to write to cloud objects of any kind. In my own applications, I write local files and then upload them to AWS S3 using the AWS Python SDK (boto3 &c). There may be some limitations or required GDAL configuration that I am not aware of.

Are you able to write blobs to the same Azure bucket using gdal_translate instead of a Rasterio program?

And of course, we really need to know what versions of Rasterio and GDAL are employed here and where they come from: PyPI or Conda-forge or where else?

On Thu, Jan 3, 2019 at 5:25 AM Niklas Heim <heim.niklas@...> wrote:
Hi Sean,

Thank you for your reply!
After setting the CURL_CA_BUNDLE environment variable I could run the above script without GDAL_HTTP_UNSAFESSL.
The logs with CPL_CURL_VERBOSE enabled are attached below. It looks like the new blob is not created?

DEBUG:rasterio.env:Entering env context: <rasterio.env.Env object at 0x7f36b3c0c630>
DEBUG:rasterio.env:Starting outermost env
DEBUG:rasterio.env:No GDAL environment exists
DEBUG:rasterio.env:New GDAL environment <rasterio._env.GDALEnv object at 0x7f3699eab198> created
DEBUG:rasterio._env:GDAL_DATA not found in environment, set to '/home/nihe/.envs/azure/lib/python3.6/site-packages/rasterio/gdal_data'.
DEBUG:rasterio._env:Started GDALEnv <rasterio._env.GDALEnv object at 0x7f3699eab198>.
DEBUG:rasterio.env:Got a copy of environment <rasterio._env.GDALEnv object at 0x7f3699eab198> options
DEBUG:rasterio.env:Entered env context: <rasterio.env.Env object at 0x7f36b3c0c630>
DEBUG:rasterio.env:Got a copy of environment <rasterio._env.GDALEnv object at 0x7f3699eab198> options
DEBUG:rasterio.env:Entering env context: <rasterio.env.Env object at 0x7f36b3cc8b70>
DEBUG:rasterio.env:Got a copy of environment <rasterio._env.GDALEnv object at 0x7f3699eab198> options
DEBUG:rasterio.env:Got a copy of environment <rasterio._env.GDALEnv object at 0x7f3699eab198> options
DEBUG:rasterio.env:Entered env context: <rasterio.env.Env object at 0x7f36b3cc8b70>
* Couldn't find host AZURE-STORAGE-NAME.blob.core.windows.net in the .netrc file; using defaults
*   Trying 52.239.213.4...
* TCP_NODELAY set
* Connected to AZURE-STORAGE-NAME.blob.core.windows.net (52.239.213.4) port 443 (#0)
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: none
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=*.blob.core.windows.net
*  start date: Nov  9 05:42:30 2017 GMT
*  expire date: Nov  9 05:42:30 2019 GMT
*  subjectAltName: host "AZURE-STORAGE-NAME.blob.core.windows.net" matched cert's "*.blob.core.windows.net"
*  issuer: C=US; ST=Washington; L=Redmond; O=Microsoft Corporation; OU=Microsoft IT; CN=Microsoft IT TLS CA 2
*  SSL certificate verify ok.
> GET /products?comp=list&delimiter=%2F&prefix=data%2F&restype=container HTTP/1.1
Host: AZURE-STORAGE-NAME.blob.core.windows.net
Accept: */*
x-ms-date: Thu, 03 Jan 2019 11:55:58 GMT
x-ms-version: 2015-02-21
Authorization: SharedKey AZURE-STORAGE-NAME:AZURE-KEY

< HTTP/1.1 200 OK
< Transfer-Encoding: chunked
< Content-Type: application/xml
< Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0
< x-ms-request-id: REQUEST-ID
< x-ms-version: 2015-02-21
< Date: Thu, 03 Jan 2019 11:55:58 GMT
< 
* Connection #0 to host AZURE-STORAGE-NAME.blob.core.windows.net left intact
* Couldn't find host AZURE-STORAGE-NAME.blob.core.windows.net in the .netrc file; using defaults
* Found bundle for host AZURE-STORAGE-NAME.blob.core.windows.net: 0x1b25870 [can pipeline]
* Re-using existing connection! (#0) with host AZURE-STORAGE-NAME.blob.core.windows.net
* Connected to AZURE-STORAGE-NAME.blob.core.windows.net (52.239.213.4) port 443 (#0)
> GET /products/data/9440fcbd-1e7f-48af-92ab-70c4c11c1fa7.tif HTTP/1.1
Host: AZURE-STORAGE-NAME.blob.core.windows.net
Accept: */*
Range: bytes=0-10161
x-ms-date: Thu, 03 Jan 2019 11:55:59 GMT
x-ms-version: 2015-02-21
Authorization: SharedKey AZURE-STORAGE-NAME:AZURE-KEY

< HTTP/1.1 206 Partial Content
< Content-Length: 10162
< Content-Type: application/octet-stream
< Content-Range: bytes 0-10161/10162
< Last-Modified: Thu, 20 Dec 2018 10:25:26 GMT
< Accept-Ranges: bytes
< ETag: "0x8D666657591C03E"
< Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0
< x-ms-request-id: REQUEST-ID
< x-ms-version: 2015-02-21
< x-ms-lease-status: unlocked
< x-ms-lease-state: available
< x-ms-blob-type: BlockBlob
< Date: Thu, 03 Jan 2019 11:55:58 GMT
< 
* Connection #0 to host AZURE-STORAGE-NAME.blob.core.windows.net left intact
/home/nihe/.envs/azure/lib/python3.6/site-packages/rasterio/__init__.py:216: NotGeoreferencedWarning: Dataset has no geotransform set. The identity matrix may be returned.
  s = DatasetReader(path, driver=driver, **kwargs)
DEBUG:rasterio._base:No projection detected.
DEBUG:rasterio._base:Nodata success: 0, Nodata value: -10000000000.000000
DEBUG:rasterio._base:No projection detected.
DEBUG:rasterio._base:Dataset <open DatasetReader name='/vsiaz/products/data/9440fcbd-1e7f-48af-92ab-70c4c11c1fa7.tif' mode='r'> is started.
DEBUG:rasterio.env:Exiting env context: <rasterio.env.Env object at 0x7f36b3cc8b70>
DEBUG:rasterio.env:Cleared existing <rasterio._env.GDALEnv object at 0x7f3699eab198> options
DEBUG:rasterio._env:Stopped GDALEnv <rasterio._env.GDALEnv object at 0x7f3699eab198>.
DEBUG:rasterio.env:No GDAL environment exists
DEBUG:rasterio.env:New GDAL environment <rasterio._env.GDALEnv object at 0x7f3699eab198> created
DEBUG:rasterio._env:GDAL_DATA not found in environment, set to '/home/nihe/.envs/azure/lib/python3.6/site-packages/rasterio/gdal_data'.
DEBUG:rasterio._env:Started GDALEnv <rasterio._env.GDALEnv object at 0x7f3699eab198>.
DEBUG:rasterio.env:Exited env context: <rasterio.env.Env object at 0x7f36b3cc8b70>
DEBUG:rasterio._base:Entering Dataset <open DatasetReader name='/vsiaz/products/data/9440fcbd-1e7f-48af-92ab-70c4c11c1fa7.tif' mode='r'> context.
DEBUG:rasterio._io:Output nodata value read from file: None
DEBUG:rasterio._io:Output nodata values: [None]
DEBUG:rasterio._io:all_valid: True
DEBUG:rasterio._io:mask_flags: ([<MaskFlags.all_valid: 1>],)
DEBUG:rasterio._io:Jump straight to _read()
DEBUG:rasterio._io:Window: None
DEBUG:rasterio._io:IO window xoff=0.0 yoff=0.0 width=100.0 height=100.0
* Couldn't find host AZURE-STORAGE-NAME.blob.core.windows.net in the .netrc file; using defaults
* Found bundle for host AZURE-STORAGE-NAME.blob.core.windows.net: 0x1b25870 [can pipeline]
* Re-using existing connection! (#0) with host AZURE-STORAGE-NAME.blob.core.windows.net
* Connected to AZURE-STORAGE-NAME.blob.core.windows.net (52.239.213.4) port 443 (#0)
> GET /products/data/9440fcbd-1e7f-48af-92ab-70c4c11c1fa7.tif HTTP/1.1
Host: AZURE-STORAGE-NAME.blob.core.windows.net
Accept: */*
Range: bytes=162-10161
x-ms-date: Thu, 03 Jan 2019 11:55:59 GMT
x-ms-version: 2015-02-21
Authorization: SharedKey AZURE-STORAGE-NAME:AZURE-KEY

< HTTP/1.1 206 Partial Content
< Content-Length: 10000
< Content-Type: application/octet-stream
< Content-Range: bytes 162-10161/10162
< Last-Modified: Thu, 20 Dec 2018 10:25:26 GMT
< Accept-Ranges: bytes
< ETag: "0x8D666657591C03E"
< Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0
< x-ms-request-id: REQUEST-ID
< x-ms-version: 2015-02-21
< x-ms-lease-status: unlocked
< x-ms-lease-state: available
< x-ms-blob-type: BlockBlob
< Date: Thu, 03 Jan 2019 11:55:58 GMT
< 
* Connection #0 to host AZURE-STORAGE-NAME.blob.core.windows.net left intact
DEBUG:rasterio._base:Dataset <open DatasetReader name='/vsiaz/products/data/9440fcbd-1e7f-48af-92ab-70c4c11c1fa7.tif' mode='r'> has been stopped.
DEBUG:rasterio._base:Dataset <closed DatasetReader name='/vsiaz/products/data/9440fcbd-1e7f-48af-92ab-70c4c11c1fa7.tif' mode='r'> has been closed.
DEBUG:rasterio._base:Exited Dataset <closed DatasetReader name='/vsiaz/products/data/9440fcbd-1e7f-48af-92ab-70c4c11c1fa7.tif' mode='r'> context.
DEBUG:rasterio.env:Got a copy of environment <rasterio._env.GDALEnv object at 0x7f3699eab198> options
DEBUG:rasterio.env:Entering env context: <rasterio.env.Env object at 0x7f3699b73080>
DEBUG:rasterio.env:Got a copy of environment <rasterio._env.GDALEnv object at 0x7f3699eab198> options
DEBUG:rasterio.env:Got a copy of environment <rasterio._env.GDALEnv object at 0x7f3699eab198> options
DEBUG:rasterio.env:Entered env context: <rasterio.env.Env object at 0x7f3699b73080>
DEBUG:rasterio._io:Path: UnparsedPath(path='/vsiaz/ready/data/9440fcbd-1e7f-48af-92ab-70c4c11c1fa7.tif'), mode: w, driver: GTiff
DEBUG:rasterio._io:Option: ('TILED', b'FALSE')
DEBUG:rasterio._io:Option: ('INTERLEAVE', b'BAND')
* Couldn't find host AZURE-STORAGE-NAME.blob.core.windows.net in the .netrc file; using defaults
* Found bundle for host AZURE-STORAGE-NAME.blob.core.windows.net: 0x1b25870 [can pipeline]
* Re-using existing connection! (#0) with host AZURE-STORAGE-NAME.blob.core.windows.net
* Connected to AZURE-STORAGE-NAME.blob.core.windows.net (52.239.213.4) port 443 (#0)
> GET /ready?comp=list&delimiter=%2F&prefix=data%2F&restype=container HTTP/1.1
Host: AZURE-STORAGE-NAME.blob.core.windows.net
Accept: */*
x-ms-date: Thu, 03 Jan 2019 11:55:59 GMT
x-ms-version: 2015-02-21
Authorization: SharedKey AZURE-STORAGE-NAME:AZURE-KEY

< HTTP/1.1 200 OK
< Transfer-Encoding: chunked
< Content-Type: application/xml
< Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0
< x-ms-request-id: REQUEST-ID
< x-ms-version: 2015-02-21
< Date: Thu, 03 Jan 2019 11:55:58 GMT
< 
* Connection #0 to host AZURE-STORAGE-NAME.blob.core.windows.net left intact
* Couldn't find host AZURE-STORAGE-NAME.blob.core.windows.net in the .netrc file; using defaults
* Found bundle for host AZURE-STORAGE-NAME.blob.core.windows.net: 0x1b25870 [can pipeline]
* Re-using existing connection! (#0) with host AZURE-STORAGE-NAME.blob.core.windows.net
* Connected to AZURE-STORAGE-NAME.blob.core.windows.net (52.239.213.4) port 443 (#0)
> HEAD /ready/data/9440fcbd-1e7f-48af-92ab-70c4c11c1fa7.tif HTTP/1.1
Host: AZURE-STORAGE-NAME.blob.core.windows.net
Accept: */*
x-ms-date: Thu, 03 Jan 2019 11:55:59 GMT
x-ms-version: 2015-02-21
Authorization: SharedKey AZURE-STORAGE-NAME:AZURE-KEY

< HTTP/1.1 404 The specified blob does not exist.
< Transfer-Encoding: chunked
< Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0
< x-ms-request-id: REQUEST-ID
< x-ms-version: 2015-02-21
< Date: Thu, 03 Jan 2019 11:55:58 GMT
< 
* Connection #0 to host AZURE-STORAGE-NAME.blob.core.windows.net left intact
* Couldn't find host AZURE-STORAGE-NAME.blob.core.windows.net in the .netrc file; using defaults
* Found bundle for host AZURE-STORAGE-NAME.blob.core.windows.net: 0x1b25870 [can pipeline]
* Re-using existing connection! (#0) with host AZURE-STORAGE-NAME.blob.core.windows.net
* Connected to AZURE-STORAGE-NAME.blob.core.windows.net (52.239.213.4) port 443 (#0)
> GET /ready?comp=list&delimiter=%2F&maxresults=1&prefix=data%2F9440fcbd-1e7f-48af-92ab-70c4c11c1fa7.tif%2F&restype=container HTTP/1.1
Host: AZURE-STORAGE-NAME.blob.core.windows.net
Accept: */*
x-ms-date: Thu, 03 Jan 2019 11:55:59 GMT
x-ms-version: 2015-02-21
Authorization: SharedKey AZURE-STORAGE-NAME:AZURE-KEY

< HTTP/1.1 200 OK
< Transfer-Encoding: chunked
< Content-Type: application/xml
< Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0
< x-ms-request-id: REQUEST-ID
< x-ms-version: 2015-02-21
< Date: Thu, 03 Jan 2019 11:55:58 GMT
< 
* Connection #0 to host AZURE-STORAGE-NAME.blob.core.windows.net left intact
* Couldn't find host AZURE-STORAGE-NAME.blob.core.windows.net in the .netrc file; using defaults
* Found bundle for host AZURE-STORAGE-NAME.blob.core.windows.net: 0x1b25870 [can pipeline]
* Re-using existing connection! (#0) with host AZURE-STORAGE-NAME.blob.core.windows.net
* Connected to AZURE-STORAGE-NAME.blob.core.windows.net (52.239.213.4) port 443 (#0)
> HEAD /ready/data/9440fcbd-1e7f-48af-92ab-70c4c11c1fa7.tif.xml HTTP/1.1
Host: AZURE-STORAGE-NAME.blob.core.windows.net
Accept: */*
x-ms-date: Thu, 03 Jan 2019 11:55:59 GMT
x-ms-version: 2015-02-21
Authorization: SharedKey AZURE-STORAGE-NAME:AZURE-KEY

< HTTP/1.1 404 The specified blob does not exist.
< Transfer-Encoding: chunked
< Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0
< x-ms-request-id: REQUEST-ID
< x-ms-version: 2015-02-21
< Date: Thu, 03 Jan 2019 11:55:58 GMT
< 
* Connection #0 to host AZURE-STORAGE-NAME.blob.core.windows.net left intact
* Couldn't find host AZURE-STORAGE-NAME.blob.core.windows.net in the .netrc file; using defaults
* Found bundle for host AZURE-STORAGE-NAME.blob.core.windows.net: 0x1b25870 [can pipeline]
* Re-using existing connection! (#0) with host AZURE-STORAGE-NAME.blob.core.windows.net
* Connected to AZURE-STORAGE-NAME.blob.core.windows.net (52.239.213.4) port 443 (#0)
> GET /ready?comp=list&delimiter=%2F&maxresults=1&prefix=data%2F9440fcbd-1e7f-48af-92ab-70c4c11c1fa7.tif.xml%2F&restype=container HTTP/1.1
Host: AZURE-STORAGE-NAME.blob.core.windows.net
Accept: */*
x-ms-date: Thu, 03 Jan 2019 11:55:59 GMT
x-ms-version: 2015-02-21
Authorization: SharedKey AZURE-STORAGE-NAME:AZURE-KEY

< HTTP/1.1 200 OK
< Transfer-Encoding: chunked
< Content-Type: application/xml
< Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0
< x-ms-request-id: REQUEST-ID
< x-ms-version: 2015-02-21
< Date: Thu, 03 Jan 2019 11:55:58 GMT
< 
* Connection #0 to host AZURE-STORAGE-NAME.blob.core.windows.net left intact
DEBUG:rasterio._io:Skipped delete for overwrite.  Dataset does not exist: b'/vsiaz/ready/data/9440fcbd-1e7f-48af-92ab-70c4c11c1fa7.tif'
/home/nihe/.envs/azure/lib/python3.6/site-packages/rasterio/__init__.py:225: NotGeoreferencedWarning: The given matrix is equal to Affine.identity or its flipped counterpart. GDAL may ignore this matrix and save no geotransform without raising an error. This behavior is somewhat driver-specific.
  **kwargs)
DEBUG:rasterio._base:No projection detected.
DEBUG:rasterio._base:Nodata success: 0, Nodata value: -10000000000.000000
DEBUG:rasterio._base:No projection detected.
DEBUG:rasterio.env:Exiting env context: <rasterio.env.Env object at 0x7f3699b73080>
DEBUG:rasterio.env:Cleared existing <rasterio._env.GDALEnv object at 0x7f3699eab198> options
DEBUG:rasterio._env:Stopped GDALEnv <rasterio._env.GDALEnv object at 0x7f3699eab198>.
DEBUG:rasterio.env:No GDAL environment exists
DEBUG:rasterio.env:New GDAL environment <rasterio._env.GDALEnv object at 0x7f3699eab198> created
DEBUG:rasterio._env:GDAL_DATA not found in environment, set to '/home/nihe/.envs/azure/lib/python3.6/site-packages/rasterio/gdal_data'.
DEBUG:rasterio._env:Started GDALEnv <rasterio._env.GDALEnv object at 0x7f3699eab198>.
DEBUG:rasterio.env:Exited env context: <rasterio.env.Env object at 0x7f3699b73080>
DEBUG:rasterio._base:Entering Dataset <open DatasetWriter name='/vsiaz/ready/data/9440fcbd-1e7f-48af-92ab-70c4c11c1fa7.tif' mode='w'> context.
DEBUG:rasterio._io:Dataset <closed DatasetWriter name='/vsiaz/ready/data/9440fcbd-1e7f-48af-92ab-70c4c11c1fa7.tif' mode='w'> has been stopped.
DEBUG:rasterio._base:Dataset <closed DatasetWriter name='/vsiaz/ready/data/9440fcbd-1e7f-48af-92ab-70c4c11c1fa7.tif' mode='w'> has been closed.
DEBUG:rasterio._base:Exited Dataset <closed DatasetWriter name='/vsiaz/ready/data/9440fcbd-1e7f-48af-92ab-70c4c11c1fa7.tif' mode='w'> context.
DEBUG:rasterio.env:Got a copy of environment <rasterio._env.GDALEnv object at 0x7f3699eab198> options
DEBUG:rasterio.env:Entering env context: <rasterio.env.Env object at 0x7f3699b73080>
DEBUG:rasterio.env:Got a copy of environment <rasterio._env.GDALEnv object at 0x7f3699eab198> options
DEBUG:rasterio.env:Got a copy of environment <rasterio._env.GDALEnv object at 0x7f3699eab198> options
DEBUG:rasterio.env:Entered env context: <rasterio.env.Env object at 0x7f3699b73080>
DEBUG:rasterio._io:Path: UnparsedPath(path='/vsiaz/ready/data/9440fcbd-1e7f-48af-92ab-70c4c11c1fa7.tif'), mode: w, driver: GTiff
DEBUG:rasterio._io:Option: ('TILED', b'FALSE')
DEBUG:rasterio._io:Option: ('INTERLEAVE', b'BAND')
DEBUG:rasterio._io:Skipped delete for overwrite.  Dataset does not exist: b'/vsiaz/ready/data/9440fcbd-1e7f-48af-92ab-70c4c11c1fa7.tif'
DEBUG:rasterio._base:No projection detected.
DEBUG:rasterio._base:Nodata success: 0, Nodata value: -10000000000.000000
DEBUG:rasterio._base:No projection detected.
DEBUG:rasterio.env:Exiting env context: <rasterio.env.Env object at 0x7f3699b73080>
DEBUG:rasterio.env:Cleared existing <rasterio._env.GDALEnv object at 0x7f3699eab198> options
DEBUG:rasterio._env:Stopped GDALEnv <rasterio._env.GDALEnv object at 0x7f3699eab198>.
DEBUG:rasterio.env:No GDAL environment exists
DEBUG:rasterio.env:New GDAL environment <rasterio._env.GDALEnv object at 0x7f3699eab198> created
DEBUG:rasterio._env:GDAL_DATA not found in environment, set to '/home/nihe/.envs/azure/lib/python3.6/site-packages/rasterio/gdal_data'.
DEBUG:rasterio._env:Started GDALEnv <rasterio._env.GDALEnv object at 0x7f3699eab198>.
DEBUG:rasterio.env:Exited env context: <rasterio.env.Env object at 0x7f3699b73080>
DEBUG:rasterio._base:Entering Dataset <open DatasetWriter name='/vsiaz/ready/data/9440fcbd-1e7f-48af-92ab-70c4c11c1fa7.tif' mode='w'> context.
DEBUG:rasterio._io:Dataset <closed DatasetWriter name='/vsiaz/ready/data/9440fcbd-1e7f-48af-92ab-70c4c11c1fa7.tif' mode='w'> has been stopped.
DEBUG:rasterio._base:Dataset <closed DatasetWriter name='/vsiaz/ready/data/9440fcbd-1e7f-48af-92ab-70c4c11c1fa7.tif' mode='w'> has been closed.
DEBUG:rasterio._base:Exited Dataset <closed DatasetWriter name='/vsiaz/ready/data/9440fcbd-1e7f-48af-92ab-70c4c11c1fa7.tif' mode='w'> context.
DEBUG:rasterio.env:Exiting env context: <rasterio.env.Env object at 0x7f36b3c0c630>
DEBUG:rasterio.env:Cleared existing <rasterio._env.GDALEnv object at 0x7f3699eab198> options
DEBUG:rasterio._env:Stopped GDALEnv <rasterio._env.GDALEnv object at 0x7f3699eab198>.
DEBUG:rasterio.env:Exiting outermost env
DEBUG:rasterio.env:Exited env context: <rasterio.env.Env object at 0x7f36b3c0c630>



--
Sean Gillies

Sean Gillies
 

In https://github.com/OSGeo/gdal/issues/1189#issuecomment-452716305, Even reminds us that GDAL does not support writing TIFFs to S3 and I'm pretty sure that this applies to Azure as well. See the note about sequential writing at https://www.gdal.org/gdal_virtual_file_systems.html#gdal_virtual_file_systems_vsiaz.

On Fri, Jan 4, 2019 at 1:40 PM Sean Gillies <sean.gillies@...> wrote:
Hi Niklas,

First off, in my original reply I missed the chance to suggest that we move this discussion to the main (not dev) discussion group, which has more users. The dev group is for discussing and planning new features for the library. I'm going to cross post over to main@rasterio.groups.io and would appreciate it if you would follow up there.

Nothing in the logs jumps out at me, so I'm stumped. I don't have any experience using Azure blob storage and have never used GDAL or Rasterio to write to cloud objects of any kind. In my own applications, I write local files and then upload them to AWS S3 using the AWS Python SDK (boto3 &c). There may be some limitations or required GDAL configuration that I am not aware of.

Are you able to write blobs to the same Azure bucket using gdal_translate instead of a Rasterio program?

And of course, we really need to know what versions of Rasterio and GDAL are employed here and where they come from: PyPI or Conda-forge or where else?

On Thu, Jan 3, 2019 at 5:25 AM Niklas Heim <heim.niklas@...> wrote:
Hi Sean,

Thank you for your reply!
After setting the CURL_CA_BUNDLE environment variable I could run the above script without GDAL_HTTP_UNSAFESSL.
The logs with CPL_CURL_VERBOSE enabled are attached below. It looks like the new blob is not created?
...

Niklas Heim
 

Hi Sean,

Thank you for you help and excuse the late reply! I will find another way of accomplishing this.

Best regards,
Niklas

Am Mi., 9. Jan. 2019 um 22:12 Uhr schrieb Sean Gillies via Groups.Io <sean=mapbox.com@groups.io>:

In https://github.com/OSGeo/gdal/issues/1189#issuecomment-452716305, Even reminds us that GDAL does not support writing TIFFs to S3 and I'm pretty sure that this applies to Azure as well. See the note about sequential writing at https://www.gdal.org/gdal_virtual_file_systems.html#gdal_virtual_file_systems_vsiaz.

On Fri, Jan 4, 2019 at 1:40 PM Sean Gillies <sean.gillies@...> wrote:
Hi Niklas,

First off, in my original reply I missed the chance to suggest that we move this discussion to the main (not dev) discussion group, which has more users. The dev group is for discussing and planning new features for the library. I'm going to cross post over to main@rasterio.groups.io and would appreciate it if you would follow up there.

Nothing in the logs jumps out at me, so I'm stumped. I don't have any experience using Azure blob storage and have never used GDAL or Rasterio to write to cloud objects of any kind. In my own applications, I write local files and then upload them to AWS S3 using the AWS Python SDK (boto3 &c). There may be some limitations or required GDAL configuration that I am not aware of.

Are you able to write blobs to the same Azure bucket using gdal_translate instead of a Rasterio program?

And of course, we really need to know what versions of Rasterio and GDAL are employed here and where they come from: PyPI or Conda-forge or where else?

On Thu, Jan 3, 2019 at 5:25 AM Niklas Heim <heim.niklas@...> wrote:
Hi Sean,

Thank you for your reply!
After setting the CURL_CA_BUNDLE environment variable I could run the above script without GDAL_HTTP_UNSAFESSL.
The logs with CPL_CURL_VERBOSE enabled are attached below. It looks like the new blob is not created?
...