If multiple credentials exits, many users have profiles in their ~/.boto as such-
[profile Data-Robot]
aws_access_key_id = a key
aws_secret_access_key = a secret
[Credentials]
aws_access_key_id = default key
aws_secret_access_key = default secret
you can pass
In [70]: con = boto.connect_s3(profile_name="Data-Robot")
``` to load from a profile. However, you can not do this in pandas.
Comment From: cpcloud
What's wrong with connecting with boto
and then get the data as a string and pass it in a StringIO
object to read_csv
?
Comment From: cpcloud
alternatively you could write a read_s3
function to do this. you could put up a pull request and we'll happily discuss it here.
Comment From: hunterowens
Cool. I'll try and get on that tonight.
On Mon, Jul 7, 2014 at 11:18 AM, Phillip Cloud notifications@github.com wrote:
you could write a read_s3 function to do this. you could put up a pull request and we'll happily discuss it here.
— Reply to this email directly or view it on GitHub https://github.com/pydata/pandas/issues/7682#issuecomment-48192444.
Comment From: cpcloud
cool, just FYI, not guaranteeing this will get in, but i think worth putting up a first implementation to see if this is something that users would like to have.
Comment From: hunterowens
Hey, finally got around to looking at this. Anyways, boto config is as follows
The Credentials section is used to specify the AWS credentials used for all boto requests. The order of precedence for authentication credentials is:
Credentials passed into the Connection class constructor.
Credentials specified by environment variables
Credentials specified as named profiles in the shared credential file.
Credentials specified by default in the shared credential file.
Credentials specified as named profiles in the config file.
Credentials specified by default in the config file.
Anyways, behavior could be that if a couple variable names are set (AWS_ACCESS_KEY
and AWS_SECRET_KEY
) it would use those keys instead of the default environment variables. Thoughts?
Comment From: jreback
we use s3fs
for access, so out-of-scope (this may actually work though)