Lately I am attending some industry events and talking about technical aspects related to Ceph RGW, Amazon S3, web APIs, etc.
In my last talk in LibreCon I missed having some step-by-step technical notes, with focus on these issues, to share with the audience and thus help to deploy a minimum sandbox environment in a matter of minutes. Ideally, these notes should also be useful to install common basic tools in a deterministic way.
This entry documents technically how to deploy Ceph RGW/S3 together with an S3 Open Source client (S3cmd) and a command-line packet analyzer (tcpdump) to follow the content of these talks from a practical environment.
Deploying the Ceph RGW/S3 demo container with Docker
There are several ways to set up a demo with Ceph RGW/S3 but the Ceph demo container fits really well here.
We will work on Ubuntu 16.04
devel@trooper:~$ lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 16.04.4 LTS Release: 16.04 Codename: xenial
If Docker is not in place you will need to install the 'docker.io' package.
devel@trooper:~$ sudo apt-get install docker.io
Using default tag: latest latest: Pulling from ceph/daemon 22dc81ace0ea: Pull complete 1a8b3c87dba3: Pull complete 91390a1c435a: Pull complete 07844b14977e: Pull complete b78396653dae: Pull complete 619ae7252650: Pull complete 8ddbda056ca6: Pull complete f8273215bd70: Pull complete f9729ced846b: Pull complete 34df4751f08d: Pull complete dd64a267d72f: Pull complete 8d1b92c3f0fb: Pull complete 6350dca2289f: Pull complete Digest: sha256:c66f0dec1bd41211faafe8110bae179d986d7c0b1fc63d83102917139c799204 Status: Downloaded newer image for ceph/daemon:latest
Get your IP and network addresses. In this example they are 192.168.124.249 and 192.168.124.0/24
devel@trooper:~$ ip route default via 192.168.124.1 dev ens3 proto static metric 100 169.254.0.0/16 dev docker0 scope link metric 1000 linkdown 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 192.168.124.0/24 dev ens3 proto kernel scope link src 192.168.124.249 metric 100
Deploy the demo image.
devel@trooper:~$ sudo docker run -d --name sandbox \
-e MON_IP=192.168.124.249 \
-e CEPH_PUBLIC_NETWORK=192.168.124.0/24 \
--net=host -v /var/lib/ceph:/var/lib/ceph \
-v /etc/ceph:/etc/ceph \
-e CEPH_DEMO_UID=sandboxId \
-e CEPH_DEMO_ACCESS_KEY=sandboxAccessKey \
-e CEPH_DEMO_SECRET_KEY=sandboxSecretKey \
-e CEPH_DEMO_BUCKET=sandboxBucket ceph/daemon \
demo
1f819cb49cdade4ca8d55f4a1749824366908de5a48b5012981f554e907d1a22
Run the status command to check all things were ok.
devel@trooper:~$ sudo docker exec sandbox ceph status
cluster:
id: c7138739-1d14-49fd-a83d-d4c98d47aa42
health: HEALTH_OK
services:
mon: 1 daemons, quorum trooper
mgr: trooper(active)
mds: cephfs-1/1/1 up {0=0=up:active}
osd: 1 osds: 1 up, 1 in
rbd-mirror: 1 daemon active
rgw: 1 daemon active
data:
pools: 7 pools, 56 pgs
objects: 216 objects, 4875 bytes
usage: 1056 MB used, 9183 MB / 10240 MB avail
pgs: 56 active+clean
Basic commands to work with the demo
To avoid typing 'sudo docker exec sandbox' with every command, you may find more comfortable running an interactive shell.
devel@trooper:~$ sudo docker exec -i -t sandbox /bin/bash root@trooper:/# ceph version ceph version 12.2.4 (52085d5249a80c5f5121a76d6288429f35e4e77b) luminous (stable)
You may need to stop/start the container in some point.
devel@trooper:~$ sudo docker stop sandbox sandbox devel@trooper:~$ sudo docker start sandbox sandbox
The Open Source client S3cmd
S3cmd ships with the demo container so you don't need to install it.
root@trooper:/# s3cmd --version s3cmd version 1.6.1
As an S3 API basic use case we can store/retrieve S3 objects.
root@trooper:/# s3cmd ls 2018-03-23 00:00 s3://sandboxBucket root@trooper:/# s3cmd ls s3://sandboxBucket root@trooper:/# echo test > /tmp/test.txt root@trooper:/# s3cmd put /tmp/test.txt s3://sandboxBucket upload: '/tmp/test.txt' -> 's3://sandboxBucket/test.txt' [1 of 1] 5 of 5 100% in 0s 105.49 B/s done root@trooper:/# s3cmd ls s3://sandboxBucket 2018-03-23 00:00 5 s3://sandboxBucket/test.txt root@trooper:/# s3cmd get s3://sandboxBucket/test.txt /tmp/test-2.txt download: 's3://sandboxBucket/test.txt' -> '/tmp/test-2.txt' [1 of 1] 5 of 5 100% in 0s 573.26 B/s done root@trooper:/# sha256sum /tmp/test*txt f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2 /tmp/test-2.txt f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2 /tmp/test.txt
Capturing and decoding raw packets
To have a look in the S3 API you will need to install a packet analyzer. You can install the 'tcpdump' package with 'apt-get' as usual. Remember running an update before the install step in the container.
To record the traffic.
root@trooper:/# tcpdump -i lo -s0 port 8080 -w s3cmd-ls-s3-sandboxBucket.bin
Run some testing command.
root@trooper:/# s3cmd ls s3://sandboxBucket 2018-03-23 00:00 5 s3://sandboxBucket/test.txt
Replay the traffic and have a look in the S3 API under the hood.
root@trooper:/# tcpdump -r s3cmd-ls-s3-sandboxBucket.bin
...
15:41:52.299463 IP localhost.42362 > trooper.8080: Flags [P.], seq 1:222, ack 1,
win 342, options [nop,nop,TS val 1088684439 ecr 2276747101],
length 221: HTTP: GET /sandboxBucket/?delimiter=/ HTTP/1.1
...
15:41:52.301064 IP trooper.8080 > localhost.42362: Flags [P.], seq 1:176, ack 222,
win 350, options [nop,nop,TS val 2276747103 ecr 1088684439],
length 175: HTTP: HTTP/1.1 200 OK
...
15:41:52.301097 IP trooper.8080 > localhost.42362: Flags [P.], seq 176:713, ack 222,
win 350, options [nop,nop,TS val 2276747103 ecr 1088684440],
length 537: HTTP
...
To dump more details add the '-v' switch.
root@trooper:/# tcpdump -r s3cmd-ls-s3-sandboxBucket.bin -v ... GET /sandboxBucket/?delimiter=/ HTTP/1.1 Host: trooper:8080 Accept-Encoding: identity Content-Length: 0 Authorization: AWS sandboxAccessKey:yimS5hvzq771AFM8uDppnoPGPEg= x-amz-date: Fri, 23 Mar 2018 00:00:00 +0000 ... HTTP/1.1 200 OK x-amz-request-id: tx00000000000000000001b-005ab52040-1010-default Content-Type: application/xml Content-Length: 537 Date: Fri, 23 Mar 2018 00:00:00 GMT ...
To dump even more readable information add the '-A' switch.
root@trooper:/# tcpdump -r s3cmd-ls-s3-sandboxBucket.bin -A ... GET /sandboxBucket/?delimiter=/ HTTP/1.1 Host: trooper:8080 Accept-Encoding: identity Content-Length: 0 Authorization: AWS sandboxAccessKey:yimS5hvzq771AFM8uDppnoPGPEg= x-amz-date: Fri, 23 Mar 2018 00:00:00 +0000 ... HTTP/1.1 200 OK x-amz-request-id: tx00000000000000000001b-005ab52040-1010-default Content-Type: application/xml Content-Length: 537 Date: Fri, 23 Mar 2018 00:00:00 GMT ... <?xml version="1.0" encoding="UTF-8"?> <ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"> <Name>sandboxBucket</Name> <Prefix></Prefix> <Marker></Marker> <MaxKeys>1000</MaxKeys> <Delimiter>/</Delimiter> <IsTruncated>false</IsTruncated> <Contents> <Key>test.txt</Key> <LastModified>2018-03-23T00:00:00.373Z</LastModified> <ETag>"d8e8fca2dc0f896fd7cb4cb0031ba249"</ETag> <Size>5</Size> <StorageClass>STANDARD</StorageClass> <Owner> <ID>sandboxId</ID> <DisplayName>Ceph demo user</DisplayName> </Owner> </Contents> </ListBucketResult> ...
Enjoy!
Subscribe to this weblog's