S3 Batch Operations Overview
S3 Batch Operations can perform a single operation on lists of Amazon S3 objects while providing a fully managed, auditable, and serverless experience. It is found under /s3/jobs on the AWS console. I have used Amazon Batch Operations mainly for copying objects in bulk between accounts.
S3 Batch Operations Benefits
- Serverless: Provides a fully managed operations
- Automated: Automates repetitive or bulk operations, saving time and money
- Scalable: Can manage up to billions of objects containing exabytes of data
- Observable: Monitors running time and percentage of objects completed
- Auditable: Tracks progress, sends notifications, and stores a detailed completion report
S3 Batch Operations Features
Amazon S3 Batch can manage objects in the batch using various operations
- Copying objects between buckets
- Replicating objects
- Encrypting objects
- Replacing object tags
- Deleting object tags
- Modifying access controls
- Setting object lock retention
- Setting object lock legal hold
- Restoring archived objects from S3 Glacier
- Invoking an AWS Lambda function
The objects in the batch are defined in a manifest file. The manifest can be populated in various ways:
- by specifying a filter which can include the bucket, prefix, suffix, creation date, or storage class
- by providing inventory reports or CSV files to drive the batch operations
S3 Batch Operations Pricing
Charges are as follows:
- $0.25 per job
- $1.00 per million object operations performed
- plus any standard costs for accessing S3 objects
- Plus any transfers costs for inter-region transfers
- Plus any KMS encryption costs
- Plus any Lambda costs when invoking Lamdba functions
Pricing source: https://aws.amazon.com/blogs/storage/data-management-at-scale-using-amazon-s3-batch-operations/
You can read more about S3 Batch Operations at https://docs.aws.amazon.com/AmazonS3/latest/userguide/batch-ops.html
