During my work life, I spend a lot of time working with MapReduce-style workflows, particularly with Hadoop infrastructure. A lot of this work is spent with larger amounts of data in order to implement the batch layer of the Lambda architecture. Due to this, the largest concern is that the behaviour is consistent across both the batch layer and the realtime layer - naturally you wouldn't want sporadic behaviour across the two. The easiest way to do this is to share code across the layers, to avoid having to keep implementations in sync. We have recently been working with Rust, which has been a little difficult to integrate with Hadoop MapReduce flows due to the fact it's mainly written in Java. It's because of this that I began to work on a small library named Efflux. It's designed as a very small interface to the MapReduce pattern, and implemented in Rust to allow us to share code across the batch layer easily.
Interesting new or updated crates.
My favorite way to explore Lambdas is to build Alexa skills because of the immediate feedback: you write a little code, and a home device talks to you. It’s a peek into the long promised of world of easy service composition.
Unfortunately, Rust didn’t have complete Alexa skill request/response handling (there is a crate from 2 years ago that handled only the basics), so I wrote one called alexa_sdk. (It’s basically a struct plus serde wrapper around the Alexa JSON spec, with some helpers.
A recent project has led me to have a go at writing an XML parser. I thought I’d document my experiences using pest to implement a lexer using the EBNF-esque formal grammar.
In short, when writing a daemon or a service, we have the „muscle“ of the application ‒ whatever we write the daemon for. And we have a whole lot of infrastructure around that: logging, command line parsing, configuration. And while there are Rust libraries for all that, one needs nontrivial amount of boilerplate code to bridge all this together. Spirit aims to be this bridge.
Less than 2 weeks ago, I was working on improving the integration of Rust with GNOME libraries at the fourth Hackfest, which happened this time in Thessaloniki.
In this post, I present a wait-free thread-local storage using the Rust language.
This is the second article about my experience at supporting pest in my glsl crate – without, for now, removing the nom parser.
Hello everyone! I would like to share an update on an open source project which I have been developing for a little while now. It is a Rust project called Wither which attempts to “provide a simple, sane & predictable interface into MongoDB, based on data models”. This post is about the 0.6 release of this crate, and I would like to dive into some of the aspects of developing this release which I really enjoyed or which I found interesting.
This is the first article out of a (I guess?!) series of upcoming articles about… parsing. More specifically, I’ve been writing the glsl crate for a while now and the current, in-use parser is backed with nom. nom is a parser combinator crate written originally by @geal and there has been a lot of fuzz around nom vs. pest lately.
Soooooooooooo. Because glsl is written with nom in its third iteration and because nom is now at version 4, I decided it was time to update the parser code of glsl. I heard about the comparison between pest and nom and decided to write an implementation with pest.
This is the first article of a series about how writing a pest looks like is fun compared to writing the nom parser. I’ll post several articles as I advance and see interesting matter to discuss and share.
Gutenberg changes name to Zola and gets a big release to celebrate.
Mundane is a cryptography library written in Rust and backed by BoringSSL. It aims to be difficult to misuse, ergonomic, and performant (in that order). It was originally created to serve the cryptography needs of Fuchsia, but we’ve decided to split it off as a general-purpose crate.
I realized I kept copy and pasting (or re-writing) functions to take user input or read simple files into Rust variables. So I’ve been working on a Rust library that attempts to make these tasks easier.
In contrast to many other ECS, iteration in Pyro is fully linear. Different combinations of components always live in the same storage. The advantage is that iteration is always fully linear and no cache is wasted. The storage behind the scene is a SoA storage.
Today I realized a new crate called pin-cell. This crate contains a type called PinCell, which is a kind of cell that is similar to RefCell, but only can allow pinned mutable references into its interior. Right now, the crate is nightly only and no-std compatible.
How is the API of PinCell different from RefCell? When you call borrow_mut on a RefCell, you get a type back that implements DerefMut, allowing you to mutate the interior value.
Rust SGX SDK, maintained by Baidu X-Lab, is a convenient framework to develop secure trusted computing applications for Intel SGX enclaves. Based on it, developers can easily build trusted SGX enclaves with memory safety guarantees. This adds an extra strong (and strongest ever) security layer over the SGX isolation, further keeping attackers away from the secrets in enclave even if they compromised the privileged software environment (operating system, hypervisor, etc.). Rust SGX SDK thus means a lot to privacy protection and trusted computing on public cloud platforms and blockchains.
For many of my use cases for running Rust applications, storing configuration in file format is less attractive as I’m typically running Rust inside docker containers and container orchestrators typically encourage the use of standard interfaces like the env for configuration. So I pondered 🤔, “What if I could treat my program’s env parameterization with the same level of typing I treat my functions and enclosing types with while getting everything one get’s from using serde for free?” That would be the bee’s knees 🐝 . Enter: envy.
Recently, I wrote ndarray-csv, a Rust crate for converting between CSV files and 2D arrays. There are already crates for CSV and arrays, so how exciting could this possibly be? Actually, there was a lot more to it than I had thought: although it started out as a two-hour project, I ended up rewriting the entire thing three times!
pest is a general purpose parser written in Rust with a focus on accessibility, correctness, and performance. It uses parsing expression grammars (or PEG) as input, which are similar in spirit to regular expressions, but which offer the enhanced expressivity needed to parse complex languages.
Merlin is a small Rust library that performs the Fiat-Shamir transformation in software, maintaining a STROBE-based transcript of the proof protocol and allowing the prover to commit messages to the transcript and compute challenges bound to all previous messages. It also provides a transcript-based RNG for use by the prover, generalizing “deterministic” and “synthetic” nonces to arbitrarily complex zero-knowledge protocols.
orion is another attempt at cryptography implemented in pure Rust. Its main focus is usability. This is in part achieved by providing a thorough documentation of the library. High-level abstractions are also provided, which are an attempt at guiding the users towards safe usage of the lower-level functionality of the library.
After almost 6 months, a new release of the GStreamer Rust bindings and the GStreamer plugin writing infrastructure for Rust is out. As usual this was coinciding with the release of all the gtk-rs crates to make use of all the new features they contain.
Recently I've been working again in the rust port of libgepub, libgepub is C code, but in the rust-migration branch almost all the real functionality is done with rust and the GepubDoc class is a GObject wrapper around that code. For this reason I was thinking about to use gnome-class to implement GepubDoc. Gnome-class is a rust lib to write GObject code in rust that's compatible with the C binary API so then you can call this new GObject code written with gnome-class from C. So, libgepub is the excuse to start to implement GIR in gnome-class.
Sonnerie is a time-series database. Map a timestamp to a floating-point value. Store multiple of these series in a single database. Insert tens of millions of samples in minutes, on rotational media.
Spirit is a crate that cuts down on boilerplate when creating unix daemons, with support for live configuration reloading.
Access Control Lists (ACLs) are an integral part of the Microsoft Windows security model. In addition to controlling access to secured resources, they are also used in sandboxing, event auditing, and specifying mandatory integrity levels. They are also exceedingly painful to programmatically manipulate, especially in Rust. Today, help has arrived — we released windows-acl, a Rust crate that simplifies the manipulation of access control lists on Windows.
I wrote this bit to give a some background about why Rand looks the way it does, and why it is time to make some changes. The new release, Rand 0.5, is getting almost ready fro release. Also I hope it is a bit entertaining to read about the history in combination with early Rust.
In this blog post I’d like to present toykio, a simple futures executor intended for learning about how executors with an event loop work. Toykio only provides a very minimal feature set: An event loop and TCP streams and listeners. However, it turns out that due to the fact that futures are composable, this is enough to build complex clients and servers.
A compatibility layer between 0.3 an 0.1 was developed. It is now possible to convert an 0.3 future into an 0.1 future and vice versa. Similar conversions for streams and sinks are also supported. Additionally, it is now possible to run 0.3 futures and async functions on Tokio’s executor. We have a dedicated blog post coming up that explains this in more detail.
The short version is, Tower Web 0.2 was just released and regular Rust attributes are now used instead of magic comments. The doc comment is replaced with #[get("/")]. This is thanks to Rust macro wizard David Tolnay. I also thought that it would be best to immediately push out 0.2 and then we can all pretend 0.1 didn’t happen.
We’ve just published a rust port of our PrettySize.NET library, now available via cargo and github. Like its .NET predecessor, PrettySize-rs aims to provide a comprehensive API for dealing with file sizes, covering both manipulation and human-readable formatting.
intl_pluralrules is a Rust crate, built to handle pluralization. Pluralization is the foundation for all localization and many internationalization APIs. With the addition of intl_pluralrules, any locale-aware date-, time- or unit-formatting (“1 second” vs “2 seconds”) and many other pluralization-dependent APIs can be added to Rust.
I’ve been working with diesel and serde. I use diesel for my postgres datastore, and serde for serializing/deserializing data to the web. Recently I came across a situation where I needed to define my type in diesel as well as implement deserialize in serde. The example below is a fairly simple so it makes for a good example to share so others can learn (and so I can remember how all this works next time I need it).
I’m excited to announce the brand new website/user-guide for the nphysics2d and nphysics3d crates: pure-rust 2D and 3D real-time physics engines with rigid bodies and joints! Online wasm-based demos are also provided (see for example the Multibody joints 34 demo).
With a brand new tutorial and a ton of new features, including prefabs, controller support, MP3 audio, localisation and an even better ergonomics!
Gutenberg 0.4.0 is out with custom taxonomies, image processing, improved shortcodes and more.
In this post I'd like to introduce a serdebug helper which is a drop-in replacement for #[derive(Debug)] with some of the advanced features that serde can provide.
This past fall, three former GnuPG developers began working on a new OpenPGP implementation in Rust called Sequoia. As it’s starting to shape up and become useful, I feel now is a good time to announce the project to the larger Rust community, and hopefully get some feedback before our first release.
Over the past several months, I’ve been working a web framework in Rust. I wanted to make use of the new hyper 0.12 changes, so the framework is just as fast, is asynchronous, and benefits from all the improvements found powering Linkerd. More importantly, I wanted there to be a reason for making a new framework; it couldn’t just be yet another framework with the only difference being I’ve written it. Instead, the way this framework is used is quite different than many that exist. In doing so, it expresses a strong opinion, which might not match your previous experiences, but I believe it manages to do something really special.
I’m super excited to reveal warp, a joint project with @carllerche.
Several new changes while working towards 0.3.
Using C libraries in a portable way involves a bit of work: finding the library on the system or building it if it's not available, checking if it is compatible, finding C headers and converting them to Rust modules, and giving Cargo correct linking instructions. Often every step of this is tricky, because operating systems, package managers and libraries have their unique quirks that need special handling.
Fortunately, all this work can be done once in a build script, and published as a
-sys Rust crate. This way other Rust programmers will be able to use the C library without having to re-invent the build script themselves.
wayland-rs is a set of crates providing generic APIs to manipulate the Wayland protocol, successor of X11 for linux windowing.
Here I am finally, after having hinted at the possibility and finally taken the time to write and merge quite an epic pull request, I can finally say it: wayland-rs is now a pure rust implementation of the protocol, rather than a crate of bindings to the wayland system C libraries.
Welcome to the inaugural post of the new futures-rs blog!
After several months of work, we’re happy to announce an alpha release of the new edition of future-rs, version 0.3. The immediate goal of this work is to support async/await notation (with borrowing) in Rust itself, which has entailed significant changes to the futures crate.
Hi. I’m Bradlee. I’ve mostly been a lurker in Rust for a while, making a couple small contributions here and there. So launching dtparse feels like nice step towards becoming a functioning member of society. But not too much, because then you know people start asking you to pay bills, and ain’t nobody got time for that.
But I built dtparse, and you can read about my thoughts on the process. Or don’t. I won’t tell you what to do with your life (but you should totally keep reading).
Thirty times faster than Handlebars, half the features! I am introducing a new templating engine for Rust that is designed to be robust and very fast! Why Zapper? Runtime templating is amazing, since you can reload templates on the fly or even allow users to provide their own templates, yet runtime templating engines are rarely fast. Templates that are statically compiled into your application can be super fast, but are completely inflexible. Recompiling and restarting your application just to change a template is especially boring. Zapper combines the flexibility of runtime templating with great performance!
Better HTTP Upgrades with hyper It’s been possible to handle HTTP Upgrades (like Websockets) in hyper if you made use of the low-level APIs in the server and client, but it wasn’t especially nice to...
In one of my recent blog posts, I talked about Event Sourcing with Aggregates in Rust. In that post, I was just beginning to explore how the Rust language and its strongly typed native data structures would allow me to express event sourcing concepts and primitives. I have now created an initial version of an Event Sourcing crate that you can explore on crates.io
I've been writing some code in Rust recently, and I thought it would be cool if I could take some of this Rust code and provide it as a native extension that I can call from Python. It turns out there are some amazing tools like PyO3 that make it easy to write fully featured Python extensions in Rust, with considerably less effort than writing a CPython extension manually.
To test out PyO3 I wrote a small Python extension in Rust, and I thought I would share some of the tips and tricks I encountered in getting this going. This post aims to serve as a quick tutorial showing how to write extensions in Rust, talking about why you might want to use something more powerful than just exposing a C library called using CFFI, and how PyO3 lets you write Python aware extensions in Rust.
I’ve been working on a project called Thruster recently, and needed a way that a developer could reasonably use templates. Thruster is a middleware based web server written in Rust (get it, th-rust-er? I’m working on my tight 5 for amateur night at The Apollo,) and as such, I needed a way to load HTML templates and insert variables into them in a performant way. Rather than poking around the numerous existing libraries and choosing one made by someone I don’t know — stranger danger! — I decided to make it myself. This article is about that journey, the unbelievably thrilling adventures of writing my first proc_macro_derive in Rust.
I gave a talk about lyon at RustFest Paris. This post is the introduction of the talk, wherein I introduce vector graphics and try to get the audience somewhat excited about it. Things will get technical in the follow-up posts.
Compressing your files is a good way to save space on your hard drive. At Dropbox’s scale, it’s not just a good idea; it is essential. Even a 1% improvement in compression efficiency can make a huge difference. That’s why we conduct research into lossless compression algorithms that are highly tuned for certain classes of files and storage, like Lepton for jpeg images, and Pied-Piper-esque lossless video encoding. For other file types, Dropbox currently uses the zlib compression format, which saves almost 8% of disk storage.
We introduce DivANS, our latest open-source contribution to compression, in this blog post.
This is a post about an interesting testing technique which feels like it should be well known. However, I haven’t seen it mentioned anywhere. I don’t even have a good name for it, I’ve semi-discovered it in the wild. If you know how this thing is called, please leave a comment!
Tarpaulin (or cargo-tarpaulin) is a code coverage tool for Rust. Last year was pretty busy with the launch of the project and the rush of issues as people started to use it so this is just a chance to look at what’s new with version 0.6.0 and what’s planned for the rest of this year.
In this blog post, I would like to present a research project I have been working on: Trying to use QML from Rust, and in general, using a C++ library from Rust. The project is a Rust crate which allows to create QMetaObject at compile time from pure Rust code. It is available here: https://github.com/woboq/qmetaobject-rs
Crates.rs, an alternative opinionated front-end to crates.io:
All readmes are displayed whenever possible, and if there’s no or poor readme, doc comments are shown too.
It combines information from multiple sources, e.g. byline is a compact amalgamation of Cargo.toml, crates.io, and GitHub contributors, so you get an idea who wrote the code even if authors forgot to keep Cargo.toml up to date.
Crate popularity is displayed as top-N position in its most relevant category, e.g. “#5 in Cryptography”, which is more meaningful than absolute download numbers.
Optional dependencies display which feature or platform they’re for.
Version history is summarized to help see at a glance whether a crate gets regular updates and how often it has breaking changes.
Recognizes sys crates even if they’re not called -sys and shows when build.rs is used.
Category pages fit more crates on screen despite looking less cluttered and having easier to read descriptions.
All categories and their representative crates are right there on the homepage.
Syntax highlighting everywhere, including code blocks.
Everything works without JS and gets indexed by search engines, so I’m hoping the site will help find crates.
Today sees the release of hyper v0.12.0, a fast and correct HTTP library for the Rust language.
This release adds support for several new features, while taking the opportunity to fix some annoyances, and improve the extreme speeds!
While the http crate generally has a great API I have been unsatisfied how it handles URLs. To create a HTTP request a full URL is needed with a scheme (http/https), authority (example.org) and a path (/search?q=rust) but http does enforce this and allows you to only state the path. This means both clients and servers are either unable to determine protocol and and authority information or have to do this manually.
More than a year ago a friend of mine wanted to learn a bit more about Rust by trying out a project. He had a nice project in mind which suits Rust quite well I think. For fun I joined his effort and created an implementation at the same time as he did, discussing and comparing along the way. In this post I’ll tell you about the project specifics, but the point of the post is more an encouragement. If you’ve read about Rust before but haven’t tried it yet, find a small project like the one below, and learn Rust in a fun and hands-on way yourself. It’s a great programming language, I highly recommend it.
It took nearly 6 months of development and the library went through nearly 5 entire rewrites. Compare that to previous major releases, which took a month at most to do. But it was worth it! This new release cleans up a lot of old bugs and unintuitive behaviours, simplifies some common patterns, is faster, uses less memory, gives better errors, but the way parsers are written stay the same. It’s like an entirely new engine under the same body work!
Currently, testing UIs is difficult, but with gtk-test you can test basically everything UI-related way more simply.
As happy Rust users ourselves, it makes us even happier to be able to say that we now have a Sentry Rust SDK. This means you and your Fungiculture can now report panics, failures, and other types of incidents to Sentry.
Not only is the SDK new and fancy, it is also one of the first ones that follows our new API guidelines for Sentry SDKs, which makes it even newer and fancier than you might normally expect.
It took a bit longer than I had initially hoped (as it always does), but a new Tokio version has been released. This release includes, among other features, a new set of APIs that allow performing filesystem operations from an asynchronous context.
Last week, I was working on improving the integration of Rust with GNOME libraries at the third Hackfest, which happened this time in Madrid.
A suite of cryptographic libraries and protocol implementations, written in the systems programming language Rust, for creating blazingly-fast, production-quality cryptographic applications.
This library implements several of the more commonly useful immutable data structures for Rust. They rely on structural sharing to keep most operations fast without needing to mutate the underlying data store, leading to more predictable code without necessarily sacrificing performance.
MesaLink is a memory-safe and OpenSSL-compatible TLS library. Since 2014, the industry has seen a huge loss due to memory vulnerabilities in TLS stacks, such as the infamous "Heartbleed" bug. MesaLink is created with the goal of eliminating memory vulnerabilities in TLS stacks. MesaLink is written in Rust, a programming language that guarantees memory safety. This significantly reduces the attack surfaces, which facilitates auditing and restricting the remaining attack surfaces. MesaLink is cross-platform and provides OpenSSL-compatible APIs. It works seamlessly in desktop, mobile, and IoT devices. With the growth of the ecosystem, MesaLink would also be adopted in the server environment in the future.
The newest release of hyper includes some lower-level connection APIs for both the server and client. Notably, this allows using hyper send and receive HTTP upgrade requests. The most popular of these is Websockets.
I am learning Clojure for the past one year and I thought making open source contributions is a great way to interact with the community. I made a post previously on using Clojars metadata to analyse JDK 9 and Clojure 1.9 issues that helped me file issues to ensure compatibility. I used the same method here to find the modules that were broken on a nightly version of a rustc due to a recent stabilisation.
To close out a great week, there is a new release of Tokio. This release includes a brand new timer implementation.
I’m happy to announce a new release of Tokio. This release includes the first iteration of the Tokio Runtime.
On behalf of the futures-rs team, I’m very happy to announce that the master branch is now at 0.2: we have a release candidate! Barring any surprises, we expect to publish to crates.io in the next week or two.
You can peruse the 0.2 API via the hosted crate docs, or dive right in to the master branch. Note that Tokio is not currently compatible with Futures 0.2; see below for more detail.
I’m planning to release a 1.0.0 version of failure on March 15. Once this happens, I don’t plan to release any further breaking changes to the failure crate (though maybe someday in the distant future).
Breaking changes in 1.0 failure is in a somewhat unique position as being a significant part of the public API of other libraries that depend on it. Whether they use the Error struct or derive Fail for a custom error type, this becomes a part of the API they expose to other users.
It’s been a while since I last suggested Mutation Testing in Rust, almost two years ago. Since then I got sidetracked a lot, and later lost interest. Just one more cool project I couldn’t afford to take on. But as things go, my interest in mutation testing was rekindled, and I decided to give it a shot and do more than blogging about it.
iui, the Improved User Interface crate, has just gotten its 0.2 release. Improved User Interface is a set of safe, idiomatic Rust bindings to platform native GUI libraries (Win32API, Cocoa, and GTK+) via libui and ui-sys. Highlights of this release include: Correct, semantic use of mutability, GTK+ theme application per window, and menu bars and file open/create modal support.
Bincode is a serializer implementation for Serde. If you stick a #[derive(Deserialize, Serialize)] on your struct, Bincode can efficiently serialize and deserialize those structs to and from bytes. Bincode is unique in that it’s a format that was built specifically for the Rust serialization ecosystem. Tight coupling with Serde allows Bincode to be very fast and serialize to very small payloads.
After a couple of weeks of work, I'm now happy to release the first version of tql, the easy-to-use ORM for Rust. While the focus was to make tql work on the stable version of the compiler, I also added some new features. The most notable new feature is the support for SQLite: now tql supports SQLite as well as PostgreSQL. The support for SQLite is almost as complete as the one for PostgreSQL: the only missing function is not implemented because the backend (SQLite) does not support it.
For some time now I develop a Rust library for asynchronous programming with coroutines, called Corona (note there’s a version 0.4.0-pre.1, but Crates prefer the „stable“ 0.3.1). I believe it is starting to be useful, so I wrote this description to show what it is good for and how it fits into the big picture of Rust. There’ll be some more changes, though, at least because Tokio just released a new version (and Futures plan to do so soon), so Corona will have to adapt.
Combine is a parser combinator library for the Rust programming language. I first announced version 3 of Combine back in August and back then I definitely expected to have a stable version by now. However other projects (cough gluon cough) got in the way and Combine fell to the wayside. It didn’t help that I didn’t have a killer feature for 3.0 either, user-defined error types make it possible to define parsers usable in #[no_std] crates which is great when you need it but it is still a fairly niche use-case.
I'm happy to announce that today, the changes proposed in the reform RFC have been released to crates.io as tokio 0.1. The primary changes are: Add a default global event loop, eliminating the need for setting up and managing your own event loop in the vast majority of cases, and decouple all task execution functionality from Tokio.
Criterion.rs is a statistics-driven benchmarking library for Rust. It provides precise measurements of changes in the performance of benchmarked code, and gives strong statistical confidence that apparent performance changes are real and not simply noise. Clear output, a simple API and reasonable defaults make it easy to use even for developers without a background in statistics. Unlike the benchmarking harness provided by Rust, Criterion.rs can be used with stable versions of the compiler.
faster began as a yak shave, created to aid base💯 in its quest to become the fastest meme on Github. Writing an explicit AVX2-accelerated version of base💯's encoder and decoder, then realizing I'd have to do the same thing again to see the speedups on my Ivy Bridge desktop, pushed me to make this library. Months later, it has blossomed into its own project, and has eclipsed base💯 in both popularity and promise.