CIRRUS S3 Storage¶

To help enable cloud native application development there is a S3 compatible storage system on site that can be used from CIRRUS. This is an object storage system and does not provide a POSIX compliant file system.

The system provides HA object storage that is replicated between ML and NWSC. If a site goes down then objects will continue to be available. S3 systems are most often accessed programmatically and at this time we do not offer any GUI browsability of buckets and objects.

Endpoint¶

https://s3.k8s.ucar.edu:5443

Access¶

Accounts are automatically created in the system upon your first connection using UCAR credentials. A special access token is created based on your username and password.

Create a file

{
  "RGW_TOKEN": {
    "version": 1,
    "type": "ldap",
    "id": "your_username",
    "key": "your_clear_text_password_here"
  }
}

Base64 encode that file
```
cat <file>|base64
```

Now set your environment variables or use your ~/.aws/credentials file

# export AWS_ACCESS_KEY_ID=<base64-encoded token>
# export AWS_SECRET_ACCESS_KEY="asdf"  #it's best to set this to something as some tools complain if it is completely empty

or in ~/.aws/credentials

[cirrus] #or your chosen profile name
aws_access_key_id=<base64 encoded token>
aws_secret_access_key="asdf"

Now use aws, s5cmd, or your tool of choice to run a command to make a bucket. (See the section below for access methods)
This method of connecting works well but requires your UCAR password to be kept in a plaintext method. Due to this we do not recommend using this authentication token beyond the initial connection to set up your account. Accounts also get normal s3 access keys created for them that are specific to CIRRUS. To access your CIRRUS S3 credentials there is an application in Open OnDemand named "CIRRUS S3 Keys" launch that application and it will report what your aws_access_key_id and aws_secret_access_key are. This site is updated every hour so it may take a small amount of time from automatic account creation to being able to access your credentials.

If you encounter issues open a jira ticket or email cirrus-admin@ucar.edu.

Access Methods¶

S3 was created by Amazon AWS and is a way to access objects over http calls. Thus it can be access via curl but that is tedious.

Bash¶

To access S3 via a bash/sh/csh/etc CLI it is best to use a widely available CLI utility such as the aws CLI provided by Amazon or an open source project such as s5cmd.

AWS CLI

AWS CLI

The official documentation is the best way to familiarize yourself with this tool. You will use the s3 subcomamnd to work with objects. AWS CLI S3 Command Reference. The --endpoint-url option will be used with a value of endpoint from above.

The aws command needs to be initially configured with credentials see the aws configure subcommand documented here.

Profiles and credentials can be set up manually via the ~/.aws/config and ~/.aws/credentials files.

# Example of ~/.aws/config file using cirrus as the default
[default]
endpoint_url = https://s3.k8s.ucar.edu:5443

# Example of ~/.aws/credentials file 
# this is where the credentials from the cirrus team are used

[default]
aws_access_key_id=ASDF1234
aws_secret_access_key=JKL987

When using either aws or s5cmd you need to use an additional flags --endpoint-url=https://s3.k8s.ucar.edu:5443.

s5cmd

s5cmd

s5cmd is a community project for interacting with s3 resources. It uses the same configuration files above. Generally it is much faster than aws cli or s3cmd project. This is our recommended method for accessing S3 objects.

Python¶

There are many libraries that can access s3 resources but the aws official boto3 is often the best way to go about it.

boto3

boto3 - Official docs
Examples

Installation in python environment

pip install boto3