As part of this effort and patchset related to the RGW auth subsystem, Radek was kind enough to include my last patches supporting the AWS4 authentication for S3 Post Object API as part of this big patchset.
This entry comments on this AWS4 feature upgrade and how it works with Ceph RGW S3.
Browser-Based Uploads Using POST (AWS Signature Version 4)
Those HTML forms consist of a form declaration and form fields. The form declaration contains high-level information about the request and the form fields contain detailed request information.
The process for sending browser-based POST requests is as follows:
- Create a security policy specifying conditions restricting what you want to allow in the request.
- Create a signature that is based on the policy. For authenticated requests, the form must include a valid signature and the policy.
- Create a HTML form that your users can access in order to upload objects to your Amazon S3 bucket directly.
Using the feature with Ceph RGW S3 and AWS4
To test the feature you can use a browser, the boto3 client or the AWS command line interface. Try the following commands:
1. Create a new bucket
$ aws s3 mb s3://test-1-2-1-bucket --region eu-central-1 \ > --endpoint-url http://s3.eu-central-1.amazonaws.com:8000 make_bucket: test-1-2-1-bucket
2. Generate some test html code with the minimal and required data form fields to auth under aws4, proper policy encoding, etc. Feel free to use this script in Python to get a simple and tested skeleton.
$ ./rgw-s3-aws4-form.py test-rgw-s3-aws4-form.html created.
3. Load test-rgw-s3-aws4-form.html in some browser and upload a test file. You should receive a 204 message.
4. Verify the object is in place and the content is good.
$ md5sum test-1-2-1-key aaf3b5e3b7505131a6baf9fb6ec1f9dc test-1-2-1-key $ aws s3 cp s3://test-1-2-1-bucket/test-1-2-1-key --region eu-central-1 \ > --endpoint-url http://s3.eu-central-1.amazonaws.com:8000 - | md5sum aaf3b5e3b7505131a6baf9fb6ec1f9dc -
Note: The example uses s3.eu-central-1.amazonaws.com as an example box name in the local network. You should use the names of your RGWs here.
- Ceph RGW AWS4 presigned URLs working with the Minio Cloud client
- AWS4 chunked upload goes upstream in Ceph RGW S3
- Ansible AWS S3 core module now supports Ceph RGW S3
- The Ceph RGW storage driver goes upstream in Libcloud
- Scalable placement of replicated data in Ceph
- Requester Pays Bucket goes upstream in Ceph
- AWS Signature Version 4 goes upstream in Ceph
- Ceph, a free unified distributed storage system
- On S3, endpoints, regions, signatures and Boto 3