Some days ago Matt committed the great Radek's effort to have a more coherent and structured scaffolding in the Ceph RGW auth subsystem supporting the differences among the available auth algorithms.

As part of this effort and patchset related to the RGW auth subsystem, Radek was kind enough to include my last patches supporting the AWS4 authentication for S3 Post Object API as part of this big patchset.

This entry comments on this AWS4 feature upgrade and how it works with Ceph RGW S3.

Browser-Based Uploads Using POST (AWS Signature Version 4)

The Amazon S3 feature documentation is available here. It describes how users upload content to Amazon S3 by using their browsers via authenticated HTTP POST requests and HTML forms.

Those HTML forms consist of a form declaration and form fields. The form declaration contains high-level information about the request and the form fields contain detailed request information.

The technical details to craft a S3 HTML form are available here. The HTML form also requires a proper POST policy (have a look here to create a POST policy!).

The process for sending browser-based POST requests is as follows:

  1. Create a security policy specifying conditions restricting what you want to allow in the request.
  2. Create a signature that is based on the policy. For authenticated requests, the form must include a valid signature and the policy.
  3. Create a HTML form that your users can access in order to upload objects to your Amazon S3 bucket directly.

Using the feature with Ceph RGW S3 and AWS4

Ceph RGW S3 supports HTTP POST requests under AWS2. With the new patch in place Ceph RGW S3 also authenticates HTTP POST requests under AWS4.

To test the feature you can use a browser, the boto3 client or the AWS command line interface. Try the following commands:

1. Create a new bucket

$ aws s3 mb s3://test-1-2-1-bucket --region eu-central-1 \
> --endpoint-url
make_bucket: test-1-2-1-bucket

2. Generate some test html code with the minimal and required data form fields to auth under aws4, proper policy encoding, etc. Feel free to use this script in Python to get a simple and tested skeleton.

$ ./
test-rgw-s3-aws4-form.html created.

3. Load test-rgw-s3-aws4-form.html in some browser and upload a test file. You should receive a 204 message.

4. Verify the object is in place and the content is good.

$ md5sum test-1-2-1-key
aaf3b5e3b7505131a6baf9fb6ec1f9dc test-1-2-1-key

$ aws s3 cp s3://test-1-2-1-bucket/test-1-2-1-key --region eu-central-1 \
> --endpoint-url - | md5sum
aaf3b5e3b7505131a6baf9fb6ec1f9dc -


Note: The example uses as an example box name in the local network. You should use the names of your RGWs here.


My work in Ceph has been made possible by Igalia and the invaluable help of the Ceph development team!


comments powered by Disqus

Recent Entries