Topics

rasterio opens file from AWS S3 bucket on local machine, but can't find file when deployed to Google App Engine

Judson Buescher
 

Hello,

 

I'm trying to run a python flask app, that opens a file hosted on an S3 bucket and analyzes the data. When I deploy the flask app on my local host, it runs like expected. Once I deploy the app to the cloud, it suddenly cannot find the file. I feel like this might be a bug, but i'm too afraid to post in the github. I'll attach the relevant snippet of code, and the error I'm receiving in the GCP Logs. I apologize for terrible formatting as I'm unaware of how to do it properly in this forum.
Here's the code: 

session = boto3.Session(aws_access_key_id=access_key, aws_secret_access_key=secret_access_key)    

def start_process(map, lon, lat):    
    with rasterio.Env(AWSSession(session)):        
        return getCoordinatePixel(map,lon,lat)  url = 's3://metcon-test-bucket/high_plus.tif'  

app = Flask(__name__) @app.route('/api')
def api_id():    
    lat = int(request.args['lat'])    
    lon = int(request.args['lon'])          
    return jsonify(start_process(url, lon, lat))  

if __name__ == '__main__':    
    app.run()

 and here's the error:


Sean Gillies
 

Apologies for the late reply. There may be a bug, yes, but there are more likely sources of trouble. Can you explain more about how you are installing the rasterio package on GCP and how you are configuring your AWS credentials?

On Thu, Jul 23, 2020 at 7:06 AM <judson.buescher@...> wrote:

Hello,

 

I'm trying to run a python flask app, that opens a file hosted on an S3 bucket and analyzes the data. When I deploy the flask app on my local host, it runs like expected. Once I deploy the app to the cloud, it suddenly cannot find the file. I feel like this might be a bug, but i'm too afraid to post in the github. I'll attach the relevant snippet of code, and the error I'm receiving in the GCP Logs. I apologize for terrible formatting as I'm unaware of how to do it properly in this forum.
Here's the code: 

session = boto3.Session(aws_access_key_id=access_key, aws_secret_access_key=secret_access_key)    

def start_process(map, lon, lat):    
    with rasterio.Env(AWSSession(session)):        
        return getCoordinatePixel(map,lon,lat)  url = 's3://metcon-test-bucket/high_plus.tif'  

app = Flask(__name__) @app.route('/api')
def api_id():    
    lat = int(request.args['lat'])    
    lon = int(request.args['lon'])          
    return jsonify(start_process(url, lon, lat))  

if __name__ == '__main__':    
    app.run()

 and here's the error:




--
Sean Gillies

Judson Buescher
 

Hi Sean,

The only way that I’m installing the package is through my requirements.txt which I include in my deploy folder. I have

 

rasterio==1.1.5

 

in the requirements.txt. For the credentials I’ve tried a couple different ways. I’ve tried setting them using

 

os.environ['GS_SECRET_ACCESS_KEY'] = secret_access_key

os.environ['GS_ACCESS_KEY_ID'] = access_key

 

I’ve also tried

 

session = boto3.Session(aws_access_key_id=access_key,

                        aws_secret_access_key=secret_access_key)

 

and then using that session object when I open the raster file. Was there any other info you’d like?

Thanks a ton for the help,
Judson

 

From: <main@rasterio.groups.io> on behalf of "Sean Gillies via groups.io" <sean.gillies@...>
Reply-To: "main@rasterio.groups.io" <main@rasterio.groups.io>
Date: Saturday, August 8, 2020 at 10:43 PM
To: "main@rasterio.groups.io" <main@rasterio.groups.io>
Subject: Re: [rasterio] rasterio opens file from AWS S3 bucket on local machine, but can't find file when deployed to Google App Engine

 

Apologies for the late reply. There may be a bug, yes, but there are more likely sources of trouble. Can you explain more about how you are installing the rasterio package on GCP and how you are configuring your AWS credentials?

 

On Thu, Jul 23, 2020 at 7:06 AM <judson.buescher@...> wrote:

Hello,

 

I'm trying to run a python flask app, that opens a file hosted on an S3 bucket and analyzes the data. When I deploy the flask app on my local host, it runs like expected. Once I deploy the app to the cloud, it suddenly cannot find the file. I feel like this might be a bug, but i'm too afraid to post in the github. I'll attach the relevant snippet of code, and the error I'm receiving in the GCP Logs. I apologize for terrible formatting as I'm unaware of how to do it properly in this forum.
Here's the code: 

session = boto3.Session(aws_access_key_id=access_key, aws_secret_access_key=secret_access_key)    

def start_process(map, lon, lat):    
    with rasterio.Env(AWSSession(session)):        
        return getCoordinatePixel(map,lon,lat)  url = 's3://metcon-test-bucket/high_plus.tif'  

app = Flask(__name__) @app.route('/api')
def api_id():    
    lat = int(request.args['lat'])    
    lon = int(request.args['lon'])          
    return jsonify(start_process(url, lon, lat))  

if __name__ == '__main__':    
    app.run()

 and here's the error:



--

Sean Gillies