Some fellows are using the Minio Client (mc) as their primary client-side tool to work with S3 cloud storage and filesystems. As you may know, mc works with the AWS v4 signature API and it provides a modern alternative under the Apache 2.0 License to UNIX commands (ls, cat, cp, diff, etc).
To resolve this issue you need to set to 'false' a new configuration parameter in the RGW S3 configuration file:
rgw s3 auth aws4 force boto2 compat = false
With this configuration in place, RGW S3 will be able to handle mc and other client-side tools experimenting the same issue properly. This configuration option is already available upstream.
By the way, if you are interested to know the origin of this issue you can have a look in this old boto2 bug.
While computing the signature a buggy boto2 version will craft the host using the port number twice while a proper implementation (mc, etc) uses it once only. The result will be two different outputs to compute the same URL.
Amazon S3 will accept as valid both signatures.
In the case of RGW S3, with the new configuration option set to 'false', RGW S3 will compute a second signature in the case of presigned URLs if the first signature computation does not match. The AWS4 presigned URL will be valid if any of the two signatures match.
My work in Ceph is sponsored by Outscale and has been made possible by Igalia and the invaluable help of the Ceph development team. Thanks Pritha, Matt Benjamin and Yehuda for all your support to go upstream!
- Multipart Upload (Copy part) goes upstream in Ceph
- AWS4 chunked upload goes upstream in Ceph RGW S3
- Ansible AWS S3 core module now supports Ceph RGW S3
- The Ceph RGW storage driver goes upstream in Libcloud
- The Outscale OSU driver goes upstream in Libcloud
- AWS Signature Version 4 goes upstream in Ceph
- Ceph, a free unified distributed storage system
- On S3, endpoints, regions, signatures and Boto 3