Hello,
I'm trying to run a python flask app, that opens a file hosted on an S3 bucket and analyzes the data. When I deploy the flask app on my local host, it runs like expected. Once I deploy the app to the cloud, it suddenly cannot find the file. I feel like this might be a bug, but i'm too afraid to post in the github. I'll attach the relevant snippet of code, and the error I'm receiving in the GCP Logs. I apologize for terrible formatting as I'm unaware of how to do it properly in this forum. Here's the code:
session = boto3.Session(aws_access_key_id=access_key, aws_secret_access_key=secret_access_key)
def start_process(map, lon, lat): with rasterio.Env(AWSSession(session)): return getCoordinatePixel(map,lon,lat) url = 's3://metcon-test-bucket/high_plus.tif'
app = Flask(__name__) @app.route('/api') def api_id(): lat = int(request.args['lat']) lon = int(request.args['lon']) return jsonify(start_process(url, lon, lat))
if __name__ == '__main__': app.run()
and here's the error: 
|
|
Apologies for the late reply. There may be a bug, yes, but there are more likely sources of trouble. Can you explain more about how you are installing the rasterio package on GCP and how you are configuring your AWS credentials?
toggle quoted message
Show quoted text
Hello,
I'm trying to run a python flask app, that opens a file hosted on an S3 bucket and analyzes the data. When I deploy the flask app on my local host, it runs like expected. Once I deploy the app to the cloud, it suddenly cannot find the file. I feel like this might be a bug, but i'm too afraid to post in the github. I'll attach the relevant snippet of code, and the error I'm receiving in the GCP Logs. I apologize for terrible formatting as I'm unaware of how to do it properly in this forum. Here's the code:
session = boto3.Session(aws_access_key_id=access_key, aws_secret_access_key=secret_access_key)
def start_process(map, lon, lat): with rasterio.Env(AWSSession(session)): return getCoordinatePixel(map,lon,lat) url = 's3://metcon-test-bucket/high_plus.tif'
app = Flask(__name__) @app.route('/api') def api_id(): lat = int(request.args['lat']) lon = int(request.args['lon']) return jsonify(start_process(url, lon, lat))
if __name__ == '__main__': app.run()
and here's the error:

|
|
Hi Sean,
The only way that I’m installing the package is through my requirements.txt which I include in my deploy folder. I have
rasterio==1.1.5
in the requirements.txt. For the credentials I’ve tried a couple different ways. I’ve tried setting them using
os.environ['GS_SECRET_ACCESS_KEY']
= secret_access_key
os.environ['GS_ACCESS_KEY_ID']
= access_key
I’ve also tried
session
= boto3.Session(aws_access_key_id=access_key,
aws_secret_access_key=secret_access_key)
and then using that session object when I open the raster file. Was there any other info you’d like?
Thanks a ton for the help,
Judson
From: <main@rasterio.groups.io> on behalf of "Sean Gillies via groups.io" <sean.gillies@...>
Reply-To: "main@rasterio.groups.io" <main@rasterio.groups.io>
Date: Saturday, August 8, 2020 at 10:43 PM
To: "main@rasterio.groups.io" <main@rasterio.groups.io>
Subject: Re: [rasterio] rasterio opens file from AWS S3 bucket on local machine, but can't find file when deployed to Google App Engine
Apologies for the late reply. There may be a bug, yes, but there are more likely sources of trouble. Can you explain more about how you are installing the rasterio package on GCP and how you are configuring your AWS credentials?
toggle quoted message
Show quoted text
On Thu, Jul 23, 2020 at 7:06 AM < judson.buescher@...> wrote:
Hello,
I'm trying to run a python flask app, that opens a file hosted on an S3 bucket and analyzes the data. When I deploy the flask app on my local host, it runs like expected. Once I deploy the app to the cloud, it suddenly cannot find the file. I feel like this
might be a bug, but i'm too afraid to post in the github. I'll attach the relevant snippet of code, and the error I'm receiving in the GCP Logs. I apologize for terrible formatting as I'm unaware of how to do it properly in this forum.
Here's the code:
session = boto3.Session(aws_access_key_id=access_key, aws_secret_access_key=secret_access_key)
def start_process(map, lon, lat):
with rasterio.Env(AWSSession(session)):
return getCoordinatePixel(map,lon,lat) url = 's3://metcon-test-bucket/high_plus.tif'
app = Flask(__name__) @app.route('/api')
def api_id():
lat = int(request.args['lat'])
lon = int(request.args['lon'])
return jsonify(start_process(url, lon, lat))
if __name__ == '__main__':
app.run()
and here's the error:

--
|
|
Hi Judson,
Hi Sean,
The only way that I’m installing the package is through my requirements.txt which I include in my deploy folder. I have
rasterio==1.1.5
in the requirements.txt. For the credentials I’ve tried a couple different ways. I’ve tried setting them using
os.environ['GS_SECRET_ACCESS_KEY']
= secret_access_key
os.environ['GS_ACCESS_KEY_ID']
= access_key
I’ve also tried
session
= boto3.Session(aws_access_key_id=access_key,
aws_secret_access_key=secret_access_key)
and then using that session object when I open the raster file. Was there any other info you’d like?
Thanks a ton for the help,
Judson
If the source data for your App Engine app is on AWS, you'll need to make sure that you're providing AWS keys to access it. You'll want
AWS_SECRET_ACCESS_KEY = secret_access_key etc.
in your environment, not GS_SECRET_ACCESS_KEY (etc.). I've never used boto3 on App Engine and wouldn't recommend that approach of configuring AWS for GDAL and rasterio.
--
|
|
Judson Buescher <judson.buescher@...>
Sean,
Sorry, I’ll try to be a bit more explicit. I’m getting the following error after setting the keys in my app.yaml file. I’m entirely sure what it means but wonder if it has something to do with trying to run it on GAE.
Traceback (most recent call last): File "rasterio/_base.pyx", line 216, in rasterio._base.DatasetBase.__init__ File "rasterio/_shim.pyx", line 67, in rasterio._shim.open_dataset
File "rasterio/_err.pyx", line 205, in rasterio._err.exc_wrap_pointer rasterio._err.CPLE_HttpResponseError: CURL error: error setting certificate verify locations: CAfile: /etc/pki/tls/certs/ca-bundle.crt CApath: none
From: <main@rasterio.groups.io> on behalf of "Sean Gillies via groups.io" <sean@...>
Reply-To: "main@rasterio.groups.io" <main@rasterio.groups.io>
Date: Friday, August 14, 2020 at 7:03 PM
To: "main@rasterio.groups.io" <main@rasterio.groups.io>
Subject: Re: [rasterio] rasterio opens file from AWS S3 bucket on local machine, but can't find file when deployed to Google App Engine
toggle quoted message
Show quoted text
On Wed, Aug 12, 2020 at 9:42 AM Judson Buescher < judson.buescher@...> wrote:
Hi Sean,
The only way that I’m installing the package is through my requirements.txt which I include in my deploy folder. I have
rasterio==1.1.5
in the requirements.txt. For the credentials I’ve tried a couple different ways. I’ve tried setting them using
os.environ['GS_SECRET_ACCESS_KEY']
= secret_access_key
os.environ['GS_ACCESS_KEY_ID']
= access_key
I’ve also tried
session = boto3.Session(aws_access_key_id=access_key,
aws_secret_access_key=secret_access_key)
and then using that session object when I open the raster file. Was there any other info you’d like?
Thanks a ton for the help,
Judson
If the source data for your App Engine app is on AWS, you'll need to make sure that you're providing AWS keys to access it. You'll want
AWS_SECRET_ACCESS_KEY = secret_access_key
in your environment, not GS_SECRET_ACCESS_KEY (etc.). I've never used boto3 on App Engine and wouldn't recommend that approach of configuring AWS for GDAL and rasterio.
--
|
|
Judson Buescher <judson.buescher@...>
Hi Sean,
I’ve tried using os.environ and also env_variables in my app.yaml. Is there a better way that you know of to set the aws access keys?
From: <main@rasterio.groups.io> on behalf of "Sean Gillies via groups.io" <sean@...>
Reply-To: "main@rasterio.groups.io" <main@rasterio.groups.io>
Date: Friday, August 14, 2020 at 7:03 PM
To: "main@rasterio.groups.io" <main@rasterio.groups.io>
Subject: Re: [rasterio] rasterio opens file from AWS S3 bucket on local machine, but can't find file when deployed to Google App Engine
toggle quoted message
Show quoted text
On Wed, Aug 12, 2020 at 9:42 AM Judson Buescher < judson.buescher@...> wrote:
Hi Sean,
The only way that I’m installing the package is through my requirements.txt which I include in my deploy folder. I have
rasterio==1.1.5
in the requirements.txt. For the credentials I’ve tried a couple different ways. I’ve tried setting them using
os.environ['GS_SECRET_ACCESS_KEY']
= secret_access_key
os.environ['GS_ACCESS_KEY_ID']
= access_key
I’ve also tried
session = boto3.Session(aws_access_key_id=access_key,
aws_secret_access_key=secret_access_key)
and then using that session object when I open the raster file. Was there any other info you’d like?
Thanks a ton for the help,
Judson
If the source data for your App Engine app is on AWS, you'll need to make sure that you're providing AWS keys to access it. You'll want
AWS_SECRET_ACCESS_KEY = secret_access_key
in your environment, not GS_SECRET_ACCESS_KEY (etc.). I've never used boto3 on App Engine and wouldn't recommend that approach of configuring AWS for GDAL and rasterio.
--
|
|
Hi Judson,
Sean,
Sorry, I’ll try to be a bit more explicit. I’m getting the following error after setting the keys in my app.yaml file. I’m entirely sure what it means but wonder if it has something to do with trying to run it on GAE.
Traceback (most recent call last): File "rasterio/_base.pyx", line 216, in rasterio._base.DatasetBase.__init__ File "rasterio/_shim.pyx", line 67, in rasterio._shim.open_dataset
File "rasterio/_err.pyx", line 205, in rasterio._err.exc_wrap_pointer rasterio._err.CPLE_HttpResponseError: CURL error: error setting certificate verify locations: CAfile: /etc/pki/tls/certs/ca-bundle.crt CApath: none
--
|
|