Boreas object storage system¶

System overview¶

Boreas is an S3-compatible object storage service backed by IBM Storage Scale (GPFS). It provides scalable, high-throughput access to long-term data and integrates with NCAR’s HPC environment. Boreas is operated by CISL and is not available for university projects.

Architecture¶

Boreas uses IBM Cluster Export Services (CES) to provide a Storage Scale (GPFS) filesystem through an S3-compatible interface. NooBaa provides the S3 service layer and handles API request processing.

Core Components¶

Cluster Export Services (CES) - CES provides the S3 protocol interface and connects directly to the underlying GPFS storage system.

NooBaa S3 Services - NooBaa services run on the CES nodes and process all S3 API requests.

Clients and Applications - Users and applications access Boreas using standard S3 APIs. Client requests are distributed across CES nodes using DNS round-robin.

Performance and capacity¶

• Single-client throughput: ~1–4 GB/s, depending on workload.

• Total usable capacity: ~6.9 PB.

• Maximum objects per bucket: Up to 100 million.

• Request distribution: DNS round-robin across CES nodes.

• Hardware platform: Uses the same as Campaign Storage.

Network Connectivity¶

• Protocol (CES) nodes are connected to other HPC resources via 100 Gb/s networking over Bifrost, enabling high-throughput data transfers between compute systems and storage.

Data Sharing and Access Controls¶

Boreas supports multiple access models based on security and collaboration requirements. All access is restricted to the UCAR VPN unless explicitly approved.

Anonymous Object Access (Signed URLs)

• Allow users to grant temporary access to individual objects without requiring authentication.
• URLs may include an expiration time.
• No S3 account or Access/Secret key credentials are required.
• Intended for short-term or ad hoc sharing.

Identity-Based Bucket Policies

• Identity-based policies allow controlled sharing of buckets with authenticated users.
• Requires S3 accounts and Access/Secret key credentials.
• Users may grant read and/or write access to specific S3 users or to all authenticated users.
• Recommended for collaborative project workflows.

Resource-Based Bucket Policies (Anonymous Bucket Access)

• Resource-based policies allow anonymous access to entire buckets when approved.
• Users may request that anonymous access be enabled for a bucket.
• No S3 account is required to access an anonymous bucket.
• Intended for controlled public distribution or broad internal sharing of datasets.

Choosing an Access Model¶

Access Methods¶

• Programmatic access using standard S3 libraries and tools.
• List Buckets via Python SDK.

  #!/usr/bin/python3
  import boto3
  import os
  from botocore.config import Config
  import urllib3
  urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
  
  config = Config(retries={'max_attempts': 40})
  
  session = boto3.session.Session()
  
  s3_client = session.client('s3', verify=False,
                             endpoint_url="https://boreas.ucar.edu",
                             aws_access_key_id='xxx',
                             aws_secret_access_key='xxx',
                             config=config)
  
  response = s3_client.list_buckets()
  for bucket in response['Buckets']:
      print(bucket['Name'])
  

• Generate a Presigned URL — Create a temporary URL (24,000s) to share myfile.txt from my-bucket without requiring credentials.

  aws s3 presign --expires 24000 s3://my-bucket/myfile.txt --endpoint https://boreas.hpc.ucar.edu:6443.
  

• Programmatic S3 Access via boto3: Initialization and Core Operations

  import boto3
  from botocore.config import Config
  
  config = Config(retries={'max_attempts': 10})
  
  s3 = boto3.client(
      "s3",
      endpoint_url="https://boreas.hpc.ucar.edu:6443",
      aws_access_key_id="YOUR_ACCESS_KEY",
      aws_secret_access_key="YOUR_SECRET_KEY",
      verify=False  # set to True if certificates are properly configured
  )
  
  # List buckets
  response = s3.list_buckets()
  for bucket in response["Buckets"]:
      print(bucket["Name"])
  
  # Upload a file
  s3.upload_file("local_file.nc", "my-bucket", "local_file.nc")
  
  # Download a file
  s3.download_file("my-bucket", "local_file.nc", "local_file.nc")
  

• Apply a Bucket Access Policy — Grants user tomb read and list access to the benchmarking bucket.

  aws --endpoint-url https://boreas.ucar.edu:6443 s3api put-bucket-policy \
  --bucket benchmarking \
  --policy '{
    "Version": "2021-10-17",
    "Statement": [{
     "Sid": "read and list bucket",
     "Effect": "Allow",
     "Principal": { "AWS": "tomb" },
     "Action": [ "s3:GetObject", "s3:ListBucket" ],
     "Resource": "*"
    }]
   }'
  

• Make a Bucket Publicly Readable — Allows any authenticated user on the Boreas endpoint to read and list objects in proj1-bucket.

  aws --endpoint-url https://boreas.ucar.edu:6443 --profile ncar-boreas \
  s3api put-bucket-policy \
  --bucket proj1-bucket --policy '{
    "Version": "2021-10-17",
    "Statement": [
     {
      "Effect": "Allow",
      "Principal": "*",
      "Action": [
         "s3:GetObject"
      ],
      "Resource": "arn:aws:s3:::proj1-bucket/*"
     },
     {
      "Effect": "Allow",
      "Principal": "*",
      "Action": [
         "s3:ListBucket"
      ],
      "Resource": "arn:aws:s3:::proj1-bucket"
     }
    ]
   }'
  

• Web-based access and transfers via the “NCAR Boreas S3” Globus collection.

Logging into Globus¶

Boreas is available through the Globus collection “NCAR Boreas S3.” Users authenticate using their S3 credentials and NCAR identity. For users logging into Boreas using Globus, you need to follow the prompts to enter the Secret and Access Keys and authenticate with your CIT username and password. Below are the screenshots and steps to log into Boreas on Globus.

Refer to the NSF NCAR Globus documentation for data transfer instructions.

Policies¶

• The system is not backed up.
• Support will be provided during business hours on business days.
• CISL will create only one admin account per lab. The admin will be able to create accounts for other users. Because the secret key-based logins do not expire, the admin will also delete accounts as appropriate – for example, when a user leaves NSF NCAR.

Requesting account¶

Contact CISL to request an account. You will be asked to:

• Specify how much disk space you need.
• Give a brief description (one sentence) of your intended use case.
• Acknowledge that you will be the admin and will manage buckets and users.