Ceph RGW/S3 demo container technical notes

| | Comments ()

Lately I am attending some industry events and talking about technical aspects related to Ceph RGW, Amazon S3, web APIs, etc.

In my last talk in LibreCon I missed having some step-by-step technical notes, with focus on these issues, to share with the audience and thus help to deploy a minimum sandbox environment in a matter of minutes. Ideally, these notes should also be useful to install common basic tools in a deterministic way.

This entry documents technically how to deploy Ceph RGW/S3 together with an S3 Open Source client (S3cmd) and a command-line packet analyzer (tcpdump) to follow the content of these talks from a practical environment.

Deploying the Ceph RGW/S3 demo container with Docker

There are several ways to set up a demo with Ceph RGW/S3 but the Ceph demo container fits really well here.

We will work on Ubuntu 16.04

devel@trooper:~$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 16.04.4 LTS
Release:	16.04
Codename:	xenial

If Docker is not in place you will need to install the 'docker.io' package.

devel@trooper:~$ sudo apt-get install docker.io
Using default tag: latest
latest: Pulling from ceph/daemon
22dc81ace0ea: Pull complete
1a8b3c87dba3: Pull complete
91390a1c435a: Pull complete
07844b14977e: Pull complete
b78396653dae: Pull complete
619ae7252650: Pull complete
8ddbda056ca6: Pull complete
f8273215bd70: Pull complete
f9729ced846b: Pull complete
34df4751f08d: Pull complete
dd64a267d72f: Pull complete
8d1b92c3f0fb: Pull complete
6350dca2289f: Pull complete
Digest: sha256:c66f0dec1bd41211faafe8110bae179d986d7c0b1fc63d83102917139c799204
Status: Downloaded newer image for ceph/daemon:latest

Get your IP and network addresses. In this example they are 192.168.124.249 and 192.168.124.0/24

devel@trooper:~$ ip route
default via 192.168.124.1 dev ens3  proto static  metric 100
169.254.0.0/16 dev docker0  scope link  metric 1000 linkdown
172.17.0.0/16 dev docker0  proto kernel  scope link  src 172.17.0.1 linkdown
192.168.124.0/24 dev ens3  proto kernel  scope link  src 192.168.124.249  metric 100

Deploy the demo image.

devel@trooper:~$ sudo docker run -d --name sandbox \
                                 -e MON_IP=192.168.124.249 \
				 -e CEPH_PUBLIC_NETWORK=192.168.124.0/24 \
				 --net=host -v /var/lib/ceph:/var/lib/ceph \
				 -v /etc/ceph:/etc/ceph \
				 -e CEPH_DEMO_UID=sandboxId \
				 -e CEPH_DEMO_ACCESS_KEY=sandboxAccessKey \
				 -e CEPH_DEMO_SECRET_KEY=sandboxSecretKey \
				 -e CEPH_DEMO_BUCKET=sandboxBucket ceph/daemon \
				 demo
1f819cb49cdade4ca8d55f4a1749824366908de5a48b5012981f554e907d1a22

Run the status command to check all things were ok.

devel@trooper:~$ sudo docker exec sandbox ceph status
  cluster:
    id:     c7138739-1d14-49fd-a83d-d4c98d47aa42
    health: HEALTH_OK

  services:
    mon:        1 daemons, quorum trooper
    mgr:        trooper(active)
    mds:        cephfs-1/1/1 up  {0=0=up:active}
    osd:        1 osds: 1 up, 1 in
    rbd-mirror: 1 daemon active
    rgw:        1 daemon active

  data:
    pools:   7 pools, 56 pgs
    objects: 216 objects, 4875 bytes
    usage:   1056 MB used, 9183 MB / 10240 MB avail
    pgs:     56 active+clean

Basic commands to work with the demo

To avoid typing 'sudo docker exec sandbox' with every command, you may find more comfortable running an interactive shell.

devel@trooper:~$ sudo docker exec -i -t sandbox /bin/bash
root@trooper:/# ceph version
ceph version 12.2.4 (52085d5249a80c5f5121a76d6288429f35e4e77b) luminous (stable)

You may need to stop/start the container in some point.

devel@trooper:~$ sudo docker stop sandbox
sandbox
devel@trooper:~$ sudo docker start sandbox
sandbox

The Open Source client S3cmd

S3cmd ships with the demo container so you don't need to install it.

root@trooper:/# s3cmd --version
s3cmd version 1.6.1

As an S3 API basic use case we can store/retrieve S3 objects.

root@trooper:/# s3cmd ls
2018-03-23 00:00  s3://sandboxBucket
root@trooper:/# s3cmd ls s3://sandboxBucket
root@trooper:/# echo test > /tmp/test.txt
root@trooper:/# s3cmd put /tmp/test.txt s3://sandboxBucket
upload: '/tmp/test.txt' -> 's3://sandboxBucket/test.txt'  [1 of 1]
 5 of 5   100% in    0s   105.49 B/s  done
root@trooper:/# s3cmd ls s3://sandboxBucket
2018-03-23 00:00         5   s3://sandboxBucket/test.txt
root@trooper:/# s3cmd get s3://sandboxBucket/test.txt /tmp/test-2.txt
download: 's3://sandboxBucket/test.txt' -> '/tmp/test-2.txt'  [1 of 1]
 5 of 5   100% in    0s   573.26 B/s  done
root@trooper:/# sha256sum /tmp/test*txt
f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2  /tmp/test-2.txt
f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2  /tmp/test.txt

Capturing and decoding raw packets

To have a look in the S3 API you will need to install a packet analyzer. You can install the 'tcpdump' package with 'apt-get' as usual. Remember running an update before the install step in the container.

To record the traffic.

root@trooper:/# tcpdump -i lo -s0 port 8080 -w s3cmd-ls-s3-sandboxBucket.bin

Run some testing command.

root@trooper:/# s3cmd ls s3://sandboxBucket
2018-03-23 00:00         5   s3://sandboxBucket/test.txt

Replay the traffic and have a look in the S3 API under the hood.

root@trooper:/# tcpdump -r s3cmd-ls-s3-sandboxBucket.bin
...
15:41:52.299463 IP localhost.42362 > trooper.8080: Flags [P.], seq 1:222, ack 1,
                win 342, options [nop,nop,TS val 1088684439 ecr 2276747101],
		length 221: HTTP: GET /sandboxBucket/?delimiter=/ HTTP/1.1
...
15:41:52.301064 IP trooper.8080 > localhost.42362: Flags [P.], seq 1:176, ack 222,
                win 350, options [nop,nop,TS val 2276747103 ecr 1088684439],
		length 175: HTTP: HTTP/1.1 200 OK
...
15:41:52.301097 IP trooper.8080 > localhost.42362: Flags [P.], seq 176:713, ack 222,
                win 350, options [nop,nop,TS val 2276747103 ecr 1088684440],
		length 537: HTTP
...

To dump more details add the '-v' switch.

root@trooper:/# tcpdump -r s3cmd-ls-s3-sandboxBucket.bin -v
...
GET /sandboxBucket/?delimiter=/ HTTP/1.1
Host: trooper:8080
Accept-Encoding: identity
Content-Length: 0
Authorization: AWS sandboxAccessKey:yimS5hvzq771AFM8uDppnoPGPEg=
x-amz-date: Fri, 23 Mar 2018 00:00:00 +0000
...
HTTP/1.1 200 OK
x-amz-request-id: tx00000000000000000001b-005ab52040-1010-default
Content-Type: application/xml
Content-Length: 537
Date: Fri, 23 Mar 2018 00:00:00 GMT
...

To dump even more readable information add the '-A' switch.

root@trooper:/# tcpdump -r s3cmd-ls-s3-sandboxBucket.bin -A
...
GET /sandboxBucket/?delimiter=/ HTTP/1.1
Host: trooper:8080
Accept-Encoding: identity
Content-Length: 0
Authorization: AWS sandboxAccessKey:yimS5hvzq771AFM8uDppnoPGPEg=
x-amz-date: Fri, 23 Mar 2018 00:00:00 +0000
...
HTTP/1.1 200 OK
x-amz-request-id: tx00000000000000000001b-005ab52040-1010-default
Content-Type: application/xml
Content-Length: 537
Date: Fri, 23 Mar 2018 00:00:00 GMT
...
<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Name>sandboxBucket</Name>
<Prefix></Prefix>
<Marker></Marker>
<MaxKeys>1000</MaxKeys>
<Delimiter>/</Delimiter>
<IsTruncated>false</IsTruncated>
<Contents>
<Key>test.txt</Key>
<LastModified>2018-03-23T00:00:00.373Z</LastModified>
<ETag>&quot;d8e8fca2dc0f896fd7cb4cb0031ba249&quot;</ETag>
<Size>5</Size>
<StorageClass>STANDARD</StorageClass>
<Owner>
<ID>sandboxId</ID>
<DisplayName>Ceph demo user</DisplayName>
</Owner>
</Contents>
</ListBucketResult>
...

Enjoy!

Comments

comments powered by Disqus

Recent Entries