Some fellows are using the Minio Client (mc) as their primary client-side tool to work with S3 cloud storage and filesystems. As you may know, mc works with the AWS v4 signature API and it provides a modern alternative under the Apache 2.0 License to UNIX commands (ls, cat, cp, diff, etc).
In S3, this feature is used to copy/move data using an existing object as data source in the storage backend instead of downloading/uploading the object to achieve the same effect via the request body.
This extension, part of the Multipart Upload API, reduces the required bandwidth between the RGW cluster and the final user when copying/moving existing objects in specific use cases.
In this post I will introduce the feature to know this concept maps to Ceph and how it works under the hood.
This year I will be attending my first ApacheCon and Apache Big Data here in Europe. Two major events related to open source technologies, techniques and best practices shaping the data ecosystem and cloud computing.
I will be in Seville (Spain) all week, November 14-18, under the sponsorship of my company Igalia. It will be great meeting with some fellows in the Apache Libcloud community, one of the open source projects where I contribute code upstream to support Ceph and custom solutions for cloud providers.
If you are interested in topics such as cloud, distributed systems, data, massive storage, scalability, security, devops, ML... or you would like to chat about some Apache project, feel free to approach me and talk about anything. You can also contact me via mail, linkedin, twitter, etc. See you there!
With AWS Signature Version 4 (AWS4) you have the option of uploading the payload in fixed or variable-size chunks.
This chunked upload option, also known as Transfer payload in multiple chunks or STREAMING-AWS4-HMAC-SHA256-PAYLOAD feature in the Amazon S3 ecosystem, avoids reading the payload twice (or buffer it in memory) to compute the signature in the client side.
Two of the promised promises in Software-Defined Storage (SDS) are higher automation and flexibility in order to reduce the costs around the storage administration tasks.
This entry will go over the required concepts to understand the virtual data and control paths in SDS solutions and how metadata are used to convey the data requirements into the automation software with the proper granularity and flexibility.
This post will introduce the new RGW driver together with the proper configuration parameters to run some examples uploading/downloading objects in Ceph Jewel.
One of the most interesting and powerful features in Ceph is the way how it computes the placement and storage of a growing amount of data at hyperscale.
This computation avoids the need to look up data locations in a central directory in order to allow nodes to be added or removed, moving as few objects as possible while still maintaining balance across new cluster configurations.
Apache Libcloud 1.0.0-rc2 (preview) was released today and it contains the new Outscale storage driver I contributed upstream several days ago.
Along this entry I will introduce the Apache Libcloud project, the Outscale driver and how a new provider can be used to connect with the Outscale object storage service.
In S3, this feature is used to configure buckets in such a way that the user who request the contents will pay transfer fee.
Along this post I will introduce the feature in order to know how this concept maps to Ceph and how it works under the hood.
I will use this blog post to talk about this new feature shipping in Ceph Jewel and the current effort by Outscale and Igalia to raise the level of compatibility between the Ceph RGW S3 and Amazon S3 interfaces.
In detail, I will describe the signing process in AWS4, how it works in Ceph RGW, the current coverage and the next steps in the pipeline around this authentication algorithm.
Over the last few months I have been working in Ceph, a free unified distributed storage system, in order to implement some missing features in RADOS gateway, help some customers with Ceph clusters in production and fixing bugs.
This effort is part of my daily work here in Igalia working in upstream projects. As you could know, Igalia works in the Cloud arena providing services on development, deployment and orchestration around interesting open projects.
My goal with this first post is introducing Ceph in a simple and easy way to understand this marvelous piece of software. I will cover the design and main innovations in Ceph together with its architecture, major use cases and relationship with OpenStack (a well-known free and open-source software platform for cloud computing).
The SDK was designed to support the AWS lifecycle so any possible solution using this SDK will require valid Amazon endpoints or regions to get the things working.
With this post I will have a look in the current Boto3 implementation to know how endpoints and regions are supported in S3, and how it would be possible to use Boto3 with compatible S3 REST interfaces if needed.
I will also comment on setting up new and compatible regions with Boto3 to consume compatible S3 API, and how the current region constraints can be enabled or disabled.
My quick notes and configurations to debug the Windows 10 Kernel on QEMU...
Several weeks ago I received a phone call coming from Juan. Andy was looking for some mate ready to jump in a new opportunity related to high performance networking, hypervisors, packet filtering and LuaJIT. Hey! this mix sounded great so I joined Andy and we went ahead.
Six weeks later, and with Diego joining the project too, one first implementation (Pflua) of the libpcap packet filtering language (pflang), together with the proper testing code and benchmarking (Pflua-bench) went live.
Along those weeks, I hacked in bindings/FFI implementation, performance/benchmarking, testing stuff and kernel-space to user-space code adaptation (Linux BPF JIT wrapped as a dynamic library!). With this post I will share a quick overview of the project and the proper links to explore it in detail.
As you may know, the mission of the National Institute for Communication Technologies (INTECO), located in León (Spain), is to strengthen cybersecurity, trust, and the protection of privacy with respect to services offered within the information society, providing value to the public, businesses, the Spanish Government, the Spanish academic and research network, the information technology sector and strategic sectors in general. It is a huge responsibility though.
Those last weeks I was really busy here in Igalia. We were hacking in Chromium/Blink broadly, attending to BlinkOn 2, held our Assembly, enjoyed one of our summits and so on. On the top of all these things we started to collaborate with the Carnegie Mellon Software Engineering Institute (SEI) around browsers security too. Great news!
On the other hand, Robert commented on the SEI's blog about the importance of having in mind secure coding practices to prevent vulnerabilities while coding, and how the CERT Secure Coding Initiative at the SEI is supporting this approach with completed standards for C and Java. By the way, coding standards for C++, Perl and other languages are under development too.
CERT coding standards are valuable resources for the programmer taking care of Information Security. As Robert highlights, secure coding standards itemize coding errors that are the root causes of current software vulnerabilities, prioritizing them by severity, likelihood of exploitation, and remediation costs. Each rule in the standard includes examples of insecure code, as well as secure alternative implementations.
If you are interested about our collaboration with SEI and the research project to evaluate the costs of producing a CERT-conforming implementation of the Chromium browser you should not skip his post. It introduces the rest of lines and colleagues collaborating in this project too.
In the '90s, a lot of active exploration and research got done around self-replicating code with the aim of pushing software infection limits and crafting some proof of concept virus in PC platforms. In that period malware was primitive and simple, but seminal in papers and applied techniques.
Those days, while reversing malware and reading dead listings for some of the most recent sophisticated and aggresive virus and worms, I needed to implement some support code to verify as the infections were happening and how hosts could be uninfected. I used OCaml to implement some antivirus routines and I spent some time observing the impact and consecuences of using functional programming in this arena.
Along this post I am going to write about virus signature matching, bit string abstractions and the blending of both worlds using OCaml. In order to understand the beauty of pattern matching in this domain, this post will comment on antivirus architectures and signature scanners too. A real Win32 virus will be shredded to explore how functional programming could be used to detect/disinfect malware.
As a result of our work in the Kernel and Virtualization team here in Igalia, Samuel and I were invited to take part at the first conference on control system's technologies used by High Energy Physics facilities. This event was hosted in the National Center of Scientific Research NCSR DEMOKRITOS, the biggest and most acclaimed research center in Greece.
After our talk titled Driving and virtualizing control systems: the Open Source approach used in WhiteRabbit, we joined the round table to discuss about the future of controls for accelerators and detectors. It was great sensing how the open hardware makes its way in this community.
One of the most extended definitions about Continuous integration (CI) is the practice, in software engineering, of merging all developer working copies with a shared mainline several times a day. This approach reduces long periods between build and test runs while simplifying automatic tasks.
Recently, I wrote some lines in Go language to watch like a concurrent Personal Continuous Integration (PCI) code, exporting a REST API over HTTP, could look. Landscape in computing has evolved from desktop computer and client-server architectures to more diverses computing devices and architectures (clusters, cloud, embedded devices ...) Nowadays, running some kind of build bot in your multi-core smartphone or personal device makes sense in some scenarios.
With this post I am releasing my last snippets of code exploring VAX architecture. Those snippets of code contain the required code developed from scratch to bootstrap a simple kernel supporting an interactive shell. Among the goals for programming this simple kernel were checking the minimal bootstrapping code, MMU programming, interrupt handling, I/O (console support) and multitasking on VAX.
The seventh White Rabbit workshop took place in CDTI, the Spanish Centre for Industrial Technological Development in Madrid on 27 and 28 November 2012. If you don't know about this project you might be interested in my last technical entry about White Rabbit project.
In this workshop Igalia reported on FMC TDC experience and some of the new technical ways used to fuel the project while developing and testing low level software using virtualization techniques.
Working in open projects is always a great experience and this time is not an excepction. Along this year we were collaborating and working with CERN, the European Organization for Nuclear Research, in several projects.
One the these projects goes under the name White Rabbit. But, what is White Rabbit? and how we are partnering with some of major European accelerators and research institutions to support it?
They were two days talking about Erlang and OTP framework, an open-source general-purpose programming language and runtime environment developed by Ericsson to build distributed and reliable soft real-time concurrent systems.
In the previous edition, I was among the speakers where I talked about self-replicating computer code, infection techniques and how security software was handling all this stuff.
This year, Ross Anderson was among the speakers. Good news having one world-class security expert talking about cryptology and security. If you don’t know Ross maybe you would like to check his personal web page on Cambridge. Ross is professor of Security Engineering at the Cambridge’s Computer Laboratory where he runs serious and pragmatic research on topics resolving global security issues.
Just blogging a quick post after caming back from Root3d CON in Madrid. This year I have to congratulate speakers again. They shared another year interesting ideas and good technical hacks. I would say this CON speaks loud and clear about the global security scene and the industry around it too. Congrats guys!
Related to technical work I would like to highlight some hot topics covered in talks such as banking attacks, loading malware in Domain Name Servers (DNS), subverting domotic facilities, cracking industrial embedded devices or bouncing along IP videos and on-line weather stations across the globe.
This past weekend I ended my lessons on our Master Software Libre.
If you follow this blog you will know I usually write down the topics I teach along these lessons. It is always good thing getting feedback and getting in touch with persons reading these lines.
By the way, this year our Master runs its fifth edition. I am proud to watch how it is working and how old and new students, teachers, collaborators, community advisors and all our friends build this knowledge community daily.
These lessons were the first ones happening before my usual lessons on Networking, Security Networking and Linux Kernel.
On Physical Security time we worked on well-know physical system security methodologies, together with two new relevant topics: environmental design and design and evaluation of physical protection systems.
It was a lesson covering broad and detailed topics; ranging from designing defensible spaces, where you are able to use different elements and aspects to get natural social control and crime prevention, till a full description of technology and sensor availability to protect different facilities. Security standards or some notes to understand social behaviour (The Bronx study case) were worked out too.
On Cryptography, we walked along its history and development in order to understand cryptographic models and current crytographic systems, free/open software tooling, integration and usual use cases. At the end, everybody got their crypto stuff in place, ready to take part in keysigning parties and next social community events.
Ah! I almost forgot. This year, students will elaborate on the right design to build a safe and secure physical protection system for one embassy.
First patch adds the new PCI PM in order to let the PCI core code handling the PCI-specific details of power translations. It was tested in kernel version 2.6.38, including standby and hibernation support. I would like to thank to Wu Zhangjin. He was kind enough to run this testing.
Second patch implements dynamic framebuffer mode setting support. Previous code works with mode setting in a hard code way. It was tested with SM712 supporting 1024x600x16 as default hardware resolution.