ballerinax/aws.s3 Ballerina library

Overview

Amazon S3 (Simple Storage Service) is a highly scalable, durable, and secure object storage service provided by Amazon Web Services (AWS). It is designed to store and retrieve any amount of data from anywhere on the web, making it ideal for a wide range of use cases, including data backup, archiving, content distribution, and big data analytics.

The ballerinax/aws.s3 connector offers APIs to connect and interact with Amazon S3, specifically based on the 2006-03-01 version of the Amazon S3 REST API. It supports creating, listing, and deleting buckets, uploading, retrieving, and deleting objects, managing object metadata and tagging, multipart uploads, and bucket and object access control lists (ACLs).

Setup guide

To use the Ballerina AWS S3 connector, you need an AWS account with the necessary IAM user credentials. For detailed steps on obtaining these credentials, refer to the Obtaining IAM user credentials guide.

Quickstart

To use the aws.s3 connector in your Ballerina application, update your .bal file as follows.

Step 1: Import the module

Import the aws.s3 module and the aws module.

import ballerinax/aws;
import ballerinax/aws.s3;

Step 2: Instantiate a new connector

1. Create a Config.toml file and configure the credentials obtained above:

accessKeyId = "<ACCESS_KEY_ID>"
secretAccessKey = "<SECRET_ACCESS_KEY>"

2. Instantiate an s3:Client with the obtained credentials and initialize the connector with it.

configurable string accessKeyId = ?;
configurable string secretAccessKey = ?;

final s3:Client s3Client = check new ({
   region: aws:US_EAST_1,
   auth: {
      accessKeyId,
      secretAccessKey
   }
});

Alternative authentication methods

Profile-based authentication

You can use AWS profile-based authentication as an alternative to static credentials.

final s3:Client s3Client = check new ({
   region: aws:US_EAST_1,
   auth: {
      profileName: "myAwsProfile",
      credentialsFilePath: "/path/to/custom/credentials"
   }
});

Note: Ensure your AWS credentials file follows the standard format.

[default]
aws_access_key_id = YOUR_ACCESS_KEY_ID
aws_secret_access_key = YOUR_SECRET_ACCESS_KEY

[myAwsProfile]
aws_access_key_id = ANOTHER_ACCESS_KEY_ID
aws_secret_access_key = ANOTHER_SECRET_ACCESS_KEY

Default credential provider chain

The standard default credential provider chain, trying each of the following in order and taking the first source that yields credentials:

1. Environment variables (AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY, and AWS_WEB_IDENTITY_TOKEN_FILE if set)
2. The shared config/credentials file's active profile (AWS_PROFILE, or default if unset) — which may itself resolve via SSO, an external process, or a chained AssumeRole call, depending on that profile's configuration
3. Container credentials (ECS/EKS)
4. EC2 instance profile (IMDS)

import ballerinax/aws.auth;

final s3:Client s3Client = check new ({
   region: aws:US_EAST_1,
   auth: auth:DEFAULT_CREDENTIALS
});

Step 3: Invoke the connector operations

Now, utilize the available connector operations. A sample use case is shown below.

public function main() returns error? {
   check s3Client->createBucket("add-unique-bucket-name");
}

Step 4: Run the Ballerina application

Use the following command to compile and run the Ballerina program.

bal run

Examples

The ballerinax/aws.s3 connector provides practical examples illustrating usage in various scenarios. Explore these examples, covering the following use cases.

1. S3 Report Archiver: Implements an ETL-style workflow that processes CSV reports and archives them to Amazon S3. Reads report data, transforms it, and uploads the results to a designated S3 bucket for long-term storage.

2. FTP to S3 Sync: Syncs files from an FTP server to Amazon S3. Downloads files from the FTP source, uploads them to an S3 bucket, and generates a summary report of skipped or failed transfers.