Ceph RGW/S3 demo container technical notes

| Comments ()

Lately I am attending some industry events and talking about technical aspects related to Ceph RGW, Amazon S3, web APIs, etc.

In my last talk in LibreCon I missed having some step-by-step technical notes, with focus on these issues, to share with the audience and thus help to deploy a minimum sandbox environment in a matter of minutes. Ideally, these notes should also be useful to install common basic tools in a deterministic way.

This entry documents technically how to deploy Ceph RGW/S3 together with an S3 Open Source client (S3cmd) and a command-line packet analyzer (tcpdump) to follow the content of these talks from a practical environment.

Continue reading ...

Open Source UAV and survival analysis with R

| Comments ()

The previous week I played a bit with some reliability models and estimators to explore how a drone-based company could run survival analysis over its fleet in order to make decissions on the hardware plane while keeping its operational risks and costs under control.

This entry contains some references, pointers and experiments on applying survival analysis over simulated UAV samples with R

Continue reading ...

Open Source UAV and mobile cellular networks

| Comments ()

The safe operation of an UAV requires a communication link to handle telemetry data, control commands and other information between the vehicle and the ground control station (GCS).

A simple and affordable way to overcome the range limitation is running the UAVs missions over the mobile radio infrastructure.

This blog post contains technical notes, references and pointers on Open Source UAVs and the feasiblity of these communication links from a vehicle's command and control perspective.

This is the third entry on Open Source UAVs. The previous entries are available here and here.

Continue reading ...

CVE-2005-3252 - Snort 2.4.0-2 remote code execution

| Comments ()

This blog post contains my technical notes on the old Snort Back Orifice (BO) pre-preprocessor buffer overflow vulnerability (CVE-2005-3252).

The bug is a stack buffer overflow in the BO pre-processor module included with Snort versions 2.4.0, 2.4.1 and 2.4.2. This vulnerability may be used to completly compromise a Snort sensor.

This content is part of my interest to design and test more secure networks monitored by Intrusion Detections Systems (IDS) and Intrusion Prevention Systems (IPS) such as Snort and Suricata.

The entry contains the technical details to get a static Snort vulnerable build together with a new payload disabling the detection capabilities and keeping the NIDS alive after a successful exploitation.

Continue reading ...

Attending LibreCon 2017

| Comments ()

This week I will be attending LibreCon 2017, one of the largest international events on open source technologies. It will be held on 19 and 20 October in Santiago de Compostela (Spain).

This year’s theme is the application of open source technologies in the industrial and primary sector, as well as the new opportunities that these technologies offer in areas like Cloud Computing, Big Data, Internet of Things (IoT) and the Sharing Economy.

I will be delivering one talk, under the sponsorship of my company Igalia, on Ceph Object Storage and its S3 API. I will introduce the Ceph architecture and the basics to understand how make cloud storage products and services based on Ceph/RGW. I will also comment on the most useful and supported S3 API and tooling working with Ceph.

See you there!

Open Source UAV API, DroneKit-Python and Geopy

| Comments ()

This blog entry is a continuation of my notes related to Open Source UAV autopilots based on Ardupilot and Pixhawk.

In this post I will explore the model, the API and the snippets of code needed to command, control and monitor a vehicle running Ardupilot with DroneKit-Python.

The entry also contains some technical comments on the communication layer interface (MAVLink), flight use cases implementation and geocoding with Geopy.

Continue reading ...

Open Source UAV Autopilot with Ardupilot and Pixhawk

| Comments ()

Recently I attended a local event related to Drones and Unmanned Aerial Vehicles (UAV) here in Galicia.

The event was a nice place to meet some interesting projects and people in the community although, on the low-level technical side, I missed some local company/individual using Open Source flight stacks.

This blog entry contains my notes to build and test an Open Source UAV autopilot based on Ardupilot and Pixhawk, together with the simulation setup needed to explore autonomous missions, flight logs, control commands and open development.

Continue reading ...

My configuration and scripts to build and run the RISC-V Linux kernel (rev 1.9) on RISC-V QEMU.

Continue reading ...

Some days ago Matt committed the great Radek's effort to have a more coherent and structured scaffolding in the Ceph RGW auth subsystem supporting the differences among the available auth algorithms.

As part of this effort and patchset related to the RGW auth subsystem, Radek was kind enough to include my last patches supporting the AWS4 authentication for S3 Post Object API as part of this big patchset.

This entry comments on this AWS4 feature upgrade and how it works with Ceph RGW S3.

Continue reading ...

CVE-2017-7269 - Binary patch diffing

| Comments ()

This Tuesday, Microsoft released additional updates for older platforms to protect against potential nation-state activity. Among the updates, the official patch fixing the CVE-2017-7269 vulnerability is available.

This official patch is a good way to confirm the root cause of the bug commented on my previous notes, allocating the target buffer size with the number of characters instead of the number of bytes in the functions HrCheckIfHeader and HrGetLockIdForPath.

This blog entry contains my notes running patch diffing on CVE-2017-7269.

Continue reading ...

This blog post contains my technical notes on the Microsoft Internet Information Services (IIS) 6.0 WebDAV 'ScStoragePathFromUrl' buffer overflow vulnerability (CVE-2017-7269).

While the root cause of this bug is a simple stack overflow, the public proof of concept (PoC) leverages the bug in a creative way to take the control avoiding the different protections and security checks in the operating system.

The result is an interesting sequence of instructions where the original and attack code interleave to run remote arbitrary code.

Continue reading ...

Some fellows are using the Minio Client (mc) as their primary client-side tool to work with S3 cloud storage and filesystems. As you may know, mc works with the AWS v4 signature API and it provides a modern alternative under the Apache 2.0 License to UNIX commands (ls, cat, cp, diff, etc).

In the case you are using mc in the client side and Ceph RGW S3 in the server side, you could be experimenting some issues with AWS4 presigned URLs and the error code '403 Forbidden'.

Continue reading ...

The last Upload Part (Copy) patches went upstream in Ceph some days ago. This new feature is available in the master branch now, and it will ship with the first development checkpoint for Kraken.

In S3, this feature is used to copy/move data using an existing object as data source in the storage backend instead of downloading/uploading the object to achieve the same effect via the request body.

This extension, part of the Multipart Upload API, reduces the required bandwidth between the RGW cluster and the final user when copying/moving existing objects in specific use cases.

In this post I will introduce the feature to know this concept maps to Ceph and how it works under the hood.

Continue reading ...

Attending ApacheCon and Apache Big Data Europe 2016

| Comments ()

This year I will be attending my first ApacheCon and Apache Big Data here in Europe. Two major events related to open source technologies, techniques and best practices shaping the data ecosystem and cloud computing.

I will be in Seville (Spain) all week, November 14-18, under the sponsorship of my company Igalia. It will be great meeting with some fellows in the Apache Libcloud community, one of the open source projects where I contribute code upstream to support Ceph and custom solutions for cloud providers.

If you are interested in topics such as cloud, distributed systems, data, massive storage, scalability, security, devops, ML... or you would like to chat about some Apache project, feel free to approach me and talk about anything. You can also contact me via mail, linkedin, twitter, etc. See you there!

AWS4 chunked upload goes upstream in Ceph RGW S3

| Comments ()

With AWS Signature Version 4 (AWS4) you have the option of uploading the payload in fixed or variable-size chunks.

This chunked upload option, also known as Transfer payload in multiple chunks or STREAMING-AWS4-HMAC-SHA256-PAYLOAD feature in the Amazon S3 ecosystem, avoids reading the payload twice (or buffer it in memory) to compute the signature in the client side.

AWS4 chunked upload support is now upstream code in Ceph. It will also ship with the next Jewel release.

Continue reading ...

Two of the promised promises in Software-Defined Storage (SDS) are higher automation and flexibility in order to reduce the costs around the storage administration tasks.

This entry will go over the required concepts to understand the virtual data and control paths in SDS solutions and how metadata are used to convey the data requirements into the automation software with the proper granularity and flexibility.

The entry will also include one of the most simple use cases ('x-amz-website-redirect-location') you can find to illustrate how all these concepts fit in Ceph.

Continue reading ...

The Ansible AWS S3 core module now supports Ceph RGW S3. The patch was upstream today and it will be included in Ansible 2.2

This post will introduce the new RGW S3 support in Ansible together with the required bits to run Ansible playbooks handling S3 use cases in Ceph Jewel.

Continue reading ...

The Ceph RGW storage driver was upstream in Apache Libcloud today. It is available in the Libcloud trunk repository and it will ship with the next release Apache Libcloud 1.0.0.

This post will introduce the new RGW driver together with the proper configuration parameters to run some examples uploading/downloading objects in Ceph Jewel.

Continue reading ...

Scalable placement of replicated data in Ceph

| Comments ()

One of the most interesting and powerful features in Ceph is the way how it computes the placement and storage of a growing amount of data at hyperscale.

This computation avoids the need to look up data locations in a central directory in order to allow nodes to be added or removed, moving as few objects as possible while still maintaining balance across new cluster configurations.

Continue reading ...

Apache Libcloud 1.0.0-rc2 (preview) was released today and it contains the new Outscale storage driver I contributed upstream several days ago.

This release together with the digital signatures are available in the download section. You can read the change log here.

Along this entry I will introduce the Apache Libcloud project, the Outscale driver and how a new provider can be used to connect with the Outscale object storage service.

Continue reading ...

Requester Pays Bucket goes upstream in Ceph

| Comments ()

The last Requester Pays Buckets patches went upstream in Ceph some days ago. This new feature is available in the master branch now, and it will be part of the next Ceph Jewel release.

In S3, this feature is used to configure buckets in such a way that the user who request the contents will pay transfer fee.

Along this post I will introduce the feature in order to know how this concept maps to Ceph and how it works under the hood.

Continue reading ...

AWS Signature Version 4 goes upstream in Ceph

| Comments ()

The first stable AWS4 implementation in Ceph went upstream some days ago. It is now available in the master branch and it will ship with the next Ceph release Jewel as planned.

I will use this blog post to talk about this new feature shipping in Ceph Jewel and the current effort by Outscale and Igalia to raise the level of compatibility between the Ceph RGW S3 and Amazon S3 interfaces.

In detail, I will describe the signing process in AWS4, how it works in Ceph RGW, the current coverage and the next steps in the pipeline around this authentication algorithm.

Continue reading ...

Ceph, a free unified distributed storage system

| Comments ()

Over the last few months I have been working in Ceph, a free unified distributed storage system, in order to implement some missing features in RADOS gateway, help some customers with Ceph clusters in production and fixing bugs.

This effort is part of my daily work here in Igalia working in upstream projects. As you could know, Igalia works in the Cloud arena providing services on development, deployment and orchestration around interesting open projects.

Together with Ceph (storage) we are also working upstream in Qemu (compute) and Snabb (networking). All these projects are in the core to create private and public clouds with Open Source.

My goal with this first post is introducing Ceph in a simple and easy way to understand this marvelous piece of software. I will cover the design and main innovations in Ceph together with its architecture, major use cases and relationship with OpenStack (a well-known free and open-source software platform for cloud computing).

Continue reading ...

On S3, endpoints, regions, signatures and Boto 3

| Comments ()

Boto3, the AWS SDK for Python, is the reference implementation to consume the Amazon cloud services.

The SDK was designed to support the AWS lifecycle so any possible solution using this SDK will require valid Amazon endpoints or regions to get the things working.

With this post I will have a look in the current Boto3 implementation to know how endpoints and regions are supported in S3, and how it would be possible to use Boto3 with compatible S3 REST interfaces if needed.

I will use the two available request signature processes, v2 and v4, to confirm all things work as expected.

I will also comment on setting up new and compatible regions with Boto3 to consume compatible S3 API, and how the current region constraints can be enabled or disabled.

Continue reading ...

Windows 10 Kernel debugging on QEMU

| Comments ()

My quick notes and configurations to debug the Windows 10 Kernel on QEMU...

Continue reading ...

Recent Entries