We are excited to announce a beta release of async-std with the intent to publish version 1.0 by September 26th, 2019. async-std is a library that looks and feels like the Rust standard library, except everything in it is made to work with async/await exactly as you would expect it to. The library comes with a book and polished API documentation, and will soon provide a stable interface to base your async libraries and applications on. While we don't promise API stability before our 1.0 release, we also don't expect to make any breaking changes.
Interesting new or updated crates.
glam is a simple and fast Rust linear algebra library for games and graphics. mathbench is a set of unit tests and benchmarks comparing the performance of glam with the popular Rust linear algebra libraries cgmath and nalgebra. The following is a table of benchmarks produced by mathbench comparing glam performance to cgmath and nalgebra on f32 data.
We’re pleased to announce the release of the first Tokio alpha with async & await support. This includes updating all of the Tokio crates to use std::future instead of futures 0.1. It also includes adding async fn versions of the APIs.
One of my more recent projects that I have been putting a lot of effort into is a Rust HTTP client called cHTTP, which I introduced on this blog over 18 months ago. Here I want to share an update on the direction of the project, and also give some detail on what months of late evenings and weekends produced in version 0.5 just published today.
If you are familiar with Python ecosystem, probably you had heard about psutil package — a cross-platform library for retrieving information about system processes and system utilization (CPU, memory, disks, network and so on). It is very popular and actively used package, which has analogs in other languages: gopsutil for Golang, oshi for Java, you name it. Rust, of course, is not an exception here: we do have psutil, sysinfo, sys-info and systemstat crates.
Now, despite the tremendous work that had been done already by the authors of these crates, I’m excited to announce what I’ve been working on for the past three months: “heim” project — library for system information fetching.
Thanks to the diligent work of Laurent Mazare on his tch-rs crate, the Rust community can now enjoy an easy access to the powerful Torch neural net framework. Being personally an avid user of both Rust and Torch, stumbling on this repo has been nothing but a belated birthday present. In this post, I would like to dive into two examples to present its most fundamental functionalities.
Earlier this month we released Abscissa: our security-oriented Rust application framework. After releasing v0.1, we’ve spent the past few weeks further polishing it up in tandem with this blog post, and just released a follow-up v0.2.
🌵 CactusRef is a single-threaded, cycle-aware, reference counting smart pointer [docs] [code]. CactusRef is nearly a drop-in replacement for std::rc1 from the Rust standard library.
I recently migrated a small/medium-sized crate from Futures 0.1 to 0.3. It was fairly easy, but there were some tricky bits and some things that were not well documented, so I think it is worth me writing up my experience.
I am a big fan of simplifying the existent code in futures-rs using async/await syntax. My goal was to rewrite the combinators in such a way that even a newbie can understand what was going on. However I met several issues with Stream combinators because it was a little bit hard to construct an impl Stream without defining a struct with a ::poll_next method. So I used Stream::unfold that can create streams from a closure.
How to write a well behaved C-API library in Rust.
A blog about learning computer science concepts with practical projects
I’m one of the maintainers of Winit, the main pure-Rust window creation library. Even if you haven’t used it directly, you’ve probably heard of projects that depend on it - Servo and Alacritty being the best-known applications that depend on our codebase. If you’ve done any graphics programming in Rust using Glutin (or dependent projects including gfx-rs, Glium, and Amethyst) we’ve been the ones making the windows actually show up on your desktop.
This announcement details the major changes since Winit 0.19. Also, we are looking for new contributors! If you are interested in working on the foundations of Rust’s GUI story, now is a great time to join the project.
Today we'd like to introduce async-log, a general-purpose crate that extends the standard log crate with asynchronous metadata. This is a first step in introducing full-fledged asynchronous tracing capabilities to Rust.
A simple tool to display static or dynamic Markdown snippets in the terminal, with skin isolation. Based on crossterm so works on most terminals (even on windows).
swym is a transactional memory library that prioritizes performance. It’s not lock-free, but it does have progress guarantees. This post will explore some of the recent work on swym’s progress promises, as well as some comparisons with non-blocking algorithms. I’m not an expert on schedulers or OS’s, so please correct me if anything is wrong. It might benefit swym!
Today I’ll be continuing my series of posts on the rust implementation of the Mercurial version control system I’ve been working on. In this post I’ll be focusing on what I learned this week about the rust module system as well as a few helpful crates I discovered to aid in command-line argument parsing and error handling.
nom, the Rust parser combinators library, is now available at version 5. This is the most mature version of nom. This is the one that feels “done”. This is the parser library that I wanted when I started nom 5 years ago. It’s here at last. nom 5 is a complete rewrite of the internal architecture, to use functions instead of macros, while keeping backward compatibility with existing macros based parsers, and making the error type completely generic.
Orkhon is Rust framework for Machine Learning to run/use inference/prediction code written in Python, frozen models and process unseen data. It is mainly focused on serving models and processing unseen data in a performant manner. Instead of using Python directly and having scalability problems for servers this framework tries to solve them with built-in async API.
Welcome everyone to this whole new gtk-rs release! Time to check what was added/updated in this new version.
The base of Rust users and contributors is growing steadily. The amount of libraries (aka crates) at http://crates.io is growing quickly; the overall “noise” is increasing. Some libraries might not be maintained any longer 🙁
This annotated catalogue shall help the Rust-users to find specific, popular, mature Rust crates. This list is WIP (Work In Progress), reflecting my personal shortlist. The ordering in the table top-down doesn’t express any preference.
Actix web 1.0.0 is released - a small, pragmatic, and extremely fast web framework.
Recently Gorka Irazoqui Apecechea and me proudly published a VRF crate as an open source project under the MIT license. The library is a fast, flexible and general-purpose Verifiable Random Function (VRF) library written in Rust, which follows the IETF standard draft written by Sharon Goldberg, Moni Naor, Dimitris Papadopoulos, Leonid Reyzin, and Jan Včelák.
This is partly an announcement of a new crate folks might find useful, partly a call for participation and help and partly a journal like story how the crate came to being. Read on (or not) or skip to the parts that seem interesting to you.
j4rs stands for “Java for Rust” and allows effortless calls to Java code, from Rust. Some time ago, on a need to call Java code from Rust, I started the j4rs project. The main idea was to implement a crate that would give the ability to its users to make calls to Java easily, so that they can benefit from the huge Java ecosystem.
We’re proud to announce mockiato! For the last few months, we tackled the issue of creating a usable mocking library. Our primary goals were: Ease of use: The mocks are written in idiomatic Rust and don’t rely on custom macro syntax. Maintainability: The entire code base strives to follow the rules of Clean Code and Clean Architecture as specified by Robert C. Martin. Strict expectation enforcement: Mockiato catches unexpected behavior as soon as it happens instead of returning default values.
During my work at Adgear, I’ve been working for a while on an async IO library for Rust. This post will present this new library. This library takes a very different approach than most other async IO libraries in Rust: it is actually inspired by the Pony programming language. So, it does not use futures, it does not use async/await: it just provides simple trait
My crate offers a powerset enum - an enum that can be parametrized to any subset of it's variants. I also included a macro for doing this parametrization by providing the types of the variants - so for example Error![std::io::Error, serde_json::Error] would generate a subset of the enum that can only have IO and JSON parsing errors.
Custom completion behavior is configured using a special bash built-in called complete. complete can be used to designate either a bash function or any other command as the completion script for a particular command. When the user requests completions for a command, complete will run specified code, passing in as args information about what the user has already typed, and expecting as output the completion suggestions. Typically these completion scripts are written in bash, but we’ll look at how it is possible to write them in Rust.
With these crates in a basic but usable state, I went to write papers, Rust code to gather data from the above sources, and inject them into Wikidata. I wrote a Rust trait to represent a generic source, and then wrote adapter structs for each of the sources. Finally, I added some wrapper code to take a list of adapters, query them about a paper, and update Wikidata accordingly.
Last week, I went to the fifth Rust+GNOME hackfest which was in Berlin again. My goal for this hackfest was to fix this issue I opened nearly three years ago. The problem is that sometimes you want to create a widget or an object and set some properties at construction time. This might be needed when you want to set construct-only properties. For instance, you might want to create a webkit2gtk::WebView with a WebContext and a UserContentManager at the same time. That’s why a constructor was manually written for this use case.
In this tutorial, we’ll see how to generate a GNOME library using the gir crate. A few things to note first: It only works on GObject-based libraries. You need .gir files.
A little explanation about those requirements: the gir crate needs .gir files to generate the library API. You can generally find them alongside the library header files (as you can see here for example, look for “.gir”).
The .gir files “describes” the library API (objects, arguments, even ownership!). This is where the gir crate comes in: it reads those .gir files and generates the Rust crates from them. You can learn more about the GIR format here.
Today we're announcing paw, a first step by the CLI Working Group to make command line applications more first class in Rust.
We introduce a procedural macro paw::main that allows passing arguments to fn main, and a new trait ParseArgs that must be implemented by the arguments to main. This allows passing not only the classic std::env::Args to main but also, for example, structopt instances.
A beginners guide to using StructOpt for parsing command line arguments.
In general, using phrase falls into 3 steps: Counting n-grams, Exporting scored models, Significant term/phrase extraction/transform N-gram counting is done continuously, providing batches of documents as they come in. Model export reads all n-gram counts so far and calculates mutual information-based collocations - you can then deploy the models by shipping the binary and data/scores_* files to a server. Labeling (identifying all significant terms and phrases in text) or transforming (eager replace of longest found phrases in text) can be done either via the CLI or the web server. Providing labels for documents is not necessary for learning phrases, but does help, and allows for significant term labeling also.
At Pixability my team and I recently created a pretty neat AWS Batch driven system. AWS Batch can can handle almost any task seamlessly and it’s pretty easy to manage with the console. This is great already, but I was curious to see what it would look like to monitor Batch with Rust. Watchrs was inspired by this question and currently provides basic functionality to do so. In this post we will be briefly going over how the main components of watchrs were built and how to use them all together.
Hello people. It’s been weeks I have started to work on luminance-1.0.0. For a brief recap, luminance is a graphics crate that I originally created in Haskell, when I ripped it off from a demoscene engine called quaazar in order to make and maintain tiner packages. The Rust port was my first Rust project and it became quickly the default language I would develop graphics applications in.
This provides a high-level binding to emacs-module, Emacs's support for dynamic modules.
After the previous two posts this post will go step by step through how the mail crate can be used to create mails based on a handlebars template and send them to a Mail Submission Agent (MSA).
The mail crate is a modular Rust library for creating, modifying and then encoding mails. It also has bindings to our new crate new-tokio-smtp to allow sending mails asynchronously, as well as bindings to handlebars for creating mails from templates. It currently does not support parsing mails, but is designed in a way that decoding capabilities could be easily added in the future (contact me if that is something you'd be interested in working on!). At 1aim, we are already using mail in production.
Rust’s futures ecosystem is currently split in two: On the one hand we have the vibrant ecosystem built around email@example.com with its many libraries working on stable Rust and on the other hand there’s the unstable std::future ecosystem with support for the ergonomic and powerful async/await language feature. To bridge the gap between these two worlds we have introduced a compatibility layer as part of the firstname.lastname@example.org extension to std::future. This blog post aims to give an overview over how to use it.
today, I’m going to talk about the splines crate. And more specifically, the splines-1.0.0-rc.1 release candidate I uploaded today on crates.io. Maybe you’re wondering what a spline is, in the first place. A spline is a mathematic curve that is defined by several polynomials. You can picture them mentally by several small and simple curves combined to each others, giving the curve an interesting shape and properties. Now why we want splines is easy to understand: imagine a curve, something smooth and a bit complex (maybe even with loops). Now, imagine you want to make an object move along that curve. How do you represent that curve and how to you “make something advance along it?”
I’ve just crossed a 20K LOC in one of my bigger Rust projects, and thought about pausing for a moment and sharing some great Rust libraries that I’ve used.
Below are 6 Rust macros that are worth taking a look at to improve your project. Rust macros are a great feature that can reduce code boilerplate and be a time saver for programmers. They also offer flexibility for developers to use metaprogramming to add new features to the language and package them in a way that is easy to integrate into code. They are one of the more powerful features of the language and this led me to search github and cargo to see what was out there. Below are some interest macros that are not as well known.
The vision of the Async Ecosystem WG is to refine the async Rust experience until it matches the quality and ease of working with today's std. There are a lot of components in that vision, including async/await syntax and borrow checker integration. Today, though, we'd like to introduce another component: Runtime, a crate that makes working with async code feel closer to working with std, and a stepping stone toward ecosystem standardization.
There’s a dearth of blog posts online that cover the details of implementing a custom protocol in tokio, at least that I’ve found. I’m going to cover some of the steps I went through in implementing an async version i3wm’s IPC.
AV1 is a modern video codec brought to you by an alliance of many different bigger and smaller players in the multimedia field. rav1e: The safest and fastest AV1 encoder, built by many volunteers and Mozilla/Xiph developers. crav1e: A companion crate, written by yours truly, that provides a C-API, so the encoder can be used by C libraries and programs. This article will just give a quick overview of the API available right now and it is mainly to help people start using it and hopefully report issues and problem.
Last time we defined a minimum viable implementation for mutagen statement removal: Remove only function call statements whose results are not returned from the surrounding block and whose AST do not contain any Assign expressions.
Zola gets the beginning of multi-lingual support and perf improvements.
This time in our “Mutating Rust” series, we want to tackle the most complex mutation so far: Statement removal. Now why do I think this is complex? It’s just removing the statement (or, as we bake our mutations into the code, activating at runtime, putting it behind an if), right?
MeiliES is an Event Sourcing database that uses the RESP (REdis Serialization Protocol) to communicate. We use the Redis protocol to simplify clients implementation. The portability problematics are resolved by using a full Rust implementation (we are using Sled as internal storage).
This is the second post in a series detailing the issues, progress, and design decisions used in clap v3. This post details removing the "stringly typed" nature of clap.
Sonic is a fast, lightweight and schema-less search backend. It ingests search texts and identifier tuples that can then be queried against in a microsecond's time.
barrel makes writing migrations for different databases as easy as possible. It provides you with a common API over SQL, with certain features only provided for database specific implementations. This way you can focus on your Rust code, and stop worrying about SQL.
Welcome to the fourth edition of This month in rustsim. This monthly newsletter will provide you with a summary of important update that occurred within the rustsim community. This includes in particular updates about the nphysics, ncollide, nalgebra, and alga crate.
Over the past few months, I’ve implemented hmmm, a Rust library for Hidden Markov Models (HMMs). HMMs are a well-established statistical machine learning technique for modeling sequences of data. They have been applied to problems like speech recognition and bioinformatics. They are called “hidden” because each discrete time step is associated with a hidden state. Below, I’ll briefly discuss some challenges that I ran into while implementing this library.
After the librsvg team finished the rustification of librsvg's main library, I wanted to start porting the high-level test suite to Rust. This is mainly to be able to run tests in parallel, which cargo test does automatically in order to reduce test times. However, this meant that librsvg needed a Rust API that would exercise the same code paths as the C entry points. At the same time, I wanted the Rust API to make it impossible to misuse the library.
Persy is a single file storage engine, all the data, referencing structures and logs are kept in a single file. Persy support read-committed transactions, using copy on write to guarantee high concurrency and isolation, the data consistency is guaranteed by a transaction log that recover the operations in case of crash. Persy provide segments to organize records of different kinds and allow the scan on a single segment. Persy provide index implementation that can be used to associate any simple value to another value or a record reference. In Persy a record is a simple Vec
, the content of the record is ignored by Persy, is just stored and retrieved on request.
pulldown_cmark, a fast pull parser for the CommonMark markdown standard written in Rust, has just seen its 0.3 release. It marks a milestone for renewed CommonMark test compliance and performance. In this blog post, we'll have a quick look at the goals of the rewrite, how they were achieved and what's next for the crate.
I’m super excited about WebAssembly! It’s fast, (can be) small, and extremely portable. In fact, I wouldn’t be surprised if in the near future most web developers write code that eventually gets compiled to Wasm.
But currently there’s not a huge selection of resources showing how to get started with WebAssembly, and I couldn’t find any tutorials that worked with create-react-app. Most focus on writing and compiling a module, but rush over the details of actually using Wasm code. What follows is a basic setup for a React app using WebAssembly that should serve as a good foundation for more complex applications.
Serde serializable and deserializable trait objects. This crate provides a macro for painless serialization of &dyn Trait trait objects and serialization + deserialization of Box
A new version packed with new features and improvements: Blink leds without blocking main thread and adjust brightness. ⚙️ Work with servo motors and adjust motor speed. 〜 Work with software PWM.
We’ve been working steadily to get Wasmer to execute WebAssembly modules on the server-side as fast as possible. TL;DR — We got 100x improvement on startup time on Wasmer 0.2.0.
I have a problem. I don't know what to call it, but there is this itch I have when I find something that can obviously be made more efficient. The way I get about scratching that itch these days is by writing some Rust. I just got such an itch when looking at static site generators for this blog, Hugo (written in Go) is pretty much state-of-the art. In Rust land we have Zola, which is feature-rich, mature (for v0.5), and definitely fast enough for most users, and yet it is not as fast as it could be. Being written in Rust is, by itself, not a guarantee of top performance. The implementation matters. After some looking around I've narrowed down a problem I want to tackle (for now) to one area: template engines.
Hey rustaceans! With a little help from your feedback just a few hours ago, I'm proud to present to you all Starling, a Binary Indexed Merkle tree! I've been working on this data structure for the last few months, and I think it is ready now for the community to have a look. Let me explain what it is and why it might be useful for your project.
We encrypt these secrets in so-called ansible vaults. Kuberwave also has the need to access these secrets. Because our staff is already comfortable with using these vaults, we’ve decided to also employ them for our Kubernetes setup. For this I created ansible-vault-rs, a library that can decrypt ansible vaults. Note that it can not create or edit vaults, because I have no need (yet) for this functionality.
This post is to announce a new Rust library for low-level text layout, called “skribo” (the Esperanto word for “writing”). This has been a major gap in the Rust ecosystem, and I hope the new crate can improve text handling across the board.
The error handling features within Rust are some of my favorite things about the language. This system works great when you are in a function which returns a Result and you want to exit at the first error you come to. However, it can be challenging if your goal is to try a few failure-prone things and return each of the errors, rather than just the first error. This is the problem multi_try attempts to solve.
My new pet project for reading batteries information in Rust.
It’s time for a new release! Main adds/changes this time are: We added the generation of the Atk crate. We now generate functions taking callback as parameters. We improved the channels handling in GLib. The whole new GString type! The minimum Rust version supported is now the 1.31. The minimum version of all libraries has been changed to GNOME 3.14. The maximum version of all libraries has been upgraded to GNOME 3.30. Added subclassing support in GLib. Even more bindings generated. Let’s see those in details.
Integrate Ruby with your Rust application. Or integrate Rust with your Ruby application. This project allows you to do either with relative ease.
161 votes and 6 comments so far on Reddit
This is a follow-up post to Lock-freedom without garbage collection from 2015, which introduced Crossbeam, a Rust library that implements efficient lock-free data structures without relying on a tracing garbage collector.
In 2018 my activity on the project has varied depending on the time and energy I have had left after work and other activities. As it turns out, working on getting WebRender shipped in Firefox is at the same time amazing and very demanding, and what's left of my brain after a good day of work isn't always up to some of the ambitions I have planned for lyon. Fortunately I am not the only one who contributed to the project, and while progress was slow on the most ambitious plans, I did spend some time on smaller features and polish.
I'll get to these big plans towards the end of this post. In the mean time let's look at some of the highlights of what changed in lyon in 2018.
I’m pleased to announce the release of metered-rs, a crate that helps live measurements of code, inspired by Coda Hale’s Java metrics, with the philosophy that measuring program performance at runtime is valuable, and independent from benchmarking.
I’ve released the ieee802154 crate, a partial implementation of the IEEE 802.15.4 standard, earlier this week. IEEE 802.15.4 is a standard for low-rate wireless personal area networks. It is used as the basis for higher-level protocols like 6LoWPAN, Zigbee, or Thread. The ieee802154 crate is only a partial implementation of this standard, but I hope it can be used as a basis for future work by extending it as required.
This is the first part in a three part blog post about (e-)mails and how to create, encode and send them using the mail crate (a library).
The recent release of Flutter 1.0 was quite exciting for me. I’m not much of anGUI person nor do I make a lot of mobile apps but after looking through some examples, I started to like their take on UI frameworks. In particular, the three aspects mentioned above seemed to be handled very well!
Back to Rust. I wondered how to adopt a similar API under the strict eyes 👀 of the borrow- and typechecker — which resulted in the experimental UI framework paw (in progress..)
Let’s talk about Layout. Layout is fundamental to any UI application. The layout engine is what takes a set of rules and figures out where to place elements on the screen. This sounds simple enough but as UIs become increasingly complex we rely more on the layout engine to be able to create these UIs as easily as possible. Not only should it be easy to build these UIs but the engine performing this work is also required to do so at minimum cost as it runs possibly on every frame (for example when performing layout animations).
I'm announcing a new library for procedural macro authors: proc-macro-rules (and on crates.io). It allows you to do macro_rules-like pattern matching inside a procedural macro. The goal is to smooth the transition from declarative to procedural macros (this works pretty well when used with the quote crate).
Thanks to Mozilla and Igalia I have the opportunity to work on Servo, adding it HTML5 multimedia features. First, with the help of Fernando Jiménez, we finished what my colleague Philippe Normand and Sebastian Dröge (one of my programming heroes) started: a media player in Rust designed to be integrated in Servo. This media player lives in its own crate: servo/media along with the WebAudio engine. A crate, in Rust jargon, is like a library. This crate is (very ad-hocly) designed to be multimedia framework agnostic, but the only backend right now is for GStreamer. Later we integrated it into Servo adding an initial support for audio and video tags.
These days, we hardly think about URLs. Popular content management tools default to so-called “pretty” slugs, and even here, on secretfader.com, I remove stop-words to ensure the tidiest, most SEO-friendly URLs possible.
It would be easy to forget the struggles that led to best practices of today’s web. However, to my generation of internet hackers, URLs riddled with ampersands and question marks were entirely normal. In those days, assuming URLs would be comprised of hyphenated alphanumeric characters was obviously unsafe; instead, we learned the official standard for parsing and constructing URLs.
generic-array is a method of achieving fixed-length fixed-size stack-allocated generic arrays without needing const generics in stable Rust.
A couple of weeks ago we had the fourth GNOME+Rust hackfest, this time in Thessaloniki, Greece. We held the hackfest at the CoHo coworking space, a small, cozy office between the University and the sea. Every such hackfest I am overwhelmed by the kind hackers who work on [gnome-class], the code generator for GObject implementations in Rust.
This is yet another library for writing parsers in Rust. What makes this one different is that I've combined some existing academic work in a way that I think is novel. The result is an unusually flexible parsing library while still offering competitive performance and memory usage.
Announcement of version 0.2 of smithay, now providing the fundamentals of a wayland compositor.
err-derive A failure-like derive macro for the std Error. The source code is mostly copied from failure-derive.
Recently there has been some discussion about interprocedural borrowing conflicts in rust. This is something I’ve been fighting with a lot, especially while working on my SAT solver varisat. Around the time Niko Matsakis published his blog post about this, I realized that the existing workarounds I’ve been using in varisat have become a maintenance nightmare. Making simple changes to the code required lots of changes in the boilerplate needed to thread various references to the places where they’re needed.
While I didn’t think that a new language feature to solve this would be something I’d be willing to wait for, I decided to sit down and figure out how such a language feature would have to look like. I knew that I wanted something that allows for partial borrows across function calls. I also prefer this to work with annotations instead of global inference. While trying to come up with a coherent design that fits neatly into the existing type and trait system, I realized that most of what I wanted can be realized in stable rust today.
I think it’s part of the human condition to ignore perfectly good advice when it comes our way. A bit over a month ago, I was dispensing sage wisdom for the ages: I had a really great idea: build a custom allocator that allows you to track your own allocations. I gave it a shot, but learned very quickly: never write your own allocator. I proceeded to ignore it, because we never really learn from our mistakes. There’s another part of the human condition that derives joy from seeing things explode. And that’s the part I’m going to focus on.
A project to make Postgres extensions in Rust easy, you might learn how to use macro_rules, attribute macros, allocators and some FFI in this post.
A year ago, Tokio was a very different library. It includes the (now deprecated) tokio-core which provided a future executor, I/O selector, and basic TCP/UDP types in a single library. It also included tokio-proto, but we won't talk about that. Over the past year, Tokio has grown to become Rust's asynchronous I/O platform. It has been adopted by a number of large companies to build apps.
encoding_rs is a high-decode-performance, low-legacy-encode-footprint and high-correctness implementation of the WHATWG Encoding Standard written in Rust. In Firefox 56, encoding_rs replaced uconv as the character encoding library used in Firefox. This wasn’t an addition of a component but an actual replacement: uconv was removed when encoding_rs landed. This writeup covers the motivation and design of encoding_rs, as well as some benchmark results.
During my work life, I spend a lot of time working with MapReduce-style workflows, particularly with Hadoop infrastructure. A lot of this work is spent with larger amounts of data in order to implement the batch layer of the Lambda architecture. Due to this, the largest concern is that the behaviour is consistent across both the batch layer and the realtime layer - naturally you wouldn't want sporadic behaviour across the two. The easiest way to do this is to share code across the layers, to avoid having to keep implementations in sync. We have recently been working with Rust, which has been a little difficult to integrate with Hadoop MapReduce flows due to the fact it's mainly written in Java. It's because of this that I began to work on a small library named Efflux. It's designed as a very small interface to the MapReduce pattern, and implemented in Rust to allow us to share code across the batch layer easily.
My favorite way to explore Lambdas is to build Alexa skills because of the immediate feedback: you write a little code, and a home device talks to you. It’s a peek into the long promised of world of easy service composition.
Unfortunately, Rust didn’t have complete Alexa skill request/response handling (there is a crate from 2 years ago that handled only the basics), so I wrote one called alexa_sdk. (It’s basically a struct plus serde wrapper around the Alexa JSON spec, with some helpers.
A recent project has led me to have a go at writing an XML parser. I thought I’d document my experiences using pest to implement a lexer using the EBNF-esque formal grammar.
In short, when writing a daemon or a service, we have the „muscle“ of the application ‒ whatever we write the daemon for. And we have a whole lot of infrastructure around that: logging, command line parsing, configuration. And while there are Rust libraries for all that, one needs nontrivial amount of boilerplate code to bridge all this together. Spirit aims to be this bridge.
Less than 2 weeks ago, I was working on improving the integration of Rust with GNOME libraries at the fourth Hackfest, which happened this time in Thessaloniki.
In this post, I present a wait-free thread-local storage using the Rust language.
This is the second article about my experience at supporting pest in my glsl crate – without, for now, removing the nom parser.
Hello everyone! I would like to share an update on an open source project which I have been developing for a little while now. It is a Rust project called Wither which attempts to “provide a simple, sane & predictable interface into MongoDB, based on data models”. This post is about the 0.6 release of this crate, and I would like to dive into some of the aspects of developing this release which I really enjoyed or which I found interesting.
This is the first article out of a (I guess?!) series of upcoming articles about… parsing. More specifically, I’ve been writing the glsl crate for a while now and the current, in-use parser is backed with nom. nom is a parser combinator crate written originally by @geal and there has been a lot of fuzz around nom vs. pest lately.
Soooooooooooo. Because glsl is written with nom in its third iteration and because nom is now at version 4, I decided it was time to update the parser code of glsl. I heard about the comparison between pest and nom and decided to write an implementation with pest.
This is the first article of a series about how writing a pest looks like is fun compared to writing the nom parser. I’ll post several articles as I advance and see interesting matter to discuss and share.
Gutenberg changes name to Zola and gets a big release to celebrate.
Mundane is a cryptography library written in Rust and backed by BoringSSL. It aims to be difficult to misuse, ergonomic, and performant (in that order). It was originally created to serve the cryptography needs of Fuchsia, but we’ve decided to split it off as a general-purpose crate.
I realized I kept copy and pasting (or re-writing) functions to take user input or read simple files into Rust variables. So I’ve been working on a Rust library that attempts to make these tasks easier.
In contrast to many other ECS, iteration in Pyro is fully linear. Different combinations of components always live in the same storage. The advantage is that iteration is always fully linear and no cache is wasted. The storage behind the scene is a SoA storage.
Today I realized a new crate called pin-cell. This crate contains a type called PinCell, which is a kind of cell that is similar to RefCell, but only can allow pinned mutable references into its interior. Right now, the crate is nightly only and no-std compatible.
How is the API of PinCell different from RefCell? When you call borrow_mut on a RefCell, you get a type back that implements DerefMut, allowing you to mutate the interior value.
Rust SGX SDK, maintained by Baidu X-Lab, is a convenient framework to develop secure trusted computing applications for Intel SGX enclaves. Based on it, developers can easily build trusted SGX enclaves with memory safety guarantees. This adds an extra strong (and strongest ever) security layer over the SGX isolation, further keeping attackers away from the secrets in enclave even if they compromised the privileged software environment (operating system, hypervisor, etc.). Rust SGX SDK thus means a lot to privacy protection and trusted computing on public cloud platforms and blockchains.
For many of my use cases for running Rust applications, storing configuration in file format is less attractive as I’m typically running Rust inside docker containers and container orchestrators typically encourage the use of standard interfaces like the env for configuration. So I pondered 🤔, “What if I could treat my program’s env parameterization with the same level of typing I treat my functions and enclosing types with while getting everything one get’s from using serde for free?” That would be the bee’s knees 🐝 . Enter: envy.
Recently, I wrote ndarray-csv, a Rust crate for converting between CSV files and 2D arrays. There are already crates for CSV and arrays, so how exciting could this possibly be? Actually, there was a lot more to it than I had thought: although it started out as a two-hour project, I ended up rewriting the entire thing three times!
pest is a general purpose parser written in Rust with a focus on accessibility, correctness, and performance. It uses parsing expression grammars (or PEG) as input, which are similar in spirit to regular expressions, but which offer the enhanced expressivity needed to parse complex languages.
Merlin is a small Rust library that performs the Fiat-Shamir transformation in software, maintaining a STROBE-based transcript of the proof protocol and allowing the prover to commit messages to the transcript and compute challenges bound to all previous messages. It also provides a transcript-based RNG for use by the prover, generalizing “deterministic” and “synthetic” nonces to arbitrarily complex zero-knowledge protocols.
orion is another attempt at cryptography implemented in pure Rust. Its main focus is usability. This is in part achieved by providing a thorough documentation of the library. High-level abstractions are also provided, which are an attempt at guiding the users towards safe usage of the lower-level functionality of the library.
After almost 6 months, a new release of the GStreamer Rust bindings and the GStreamer plugin writing infrastructure for Rust is out. As usual this was coinciding with the release of all the gtk-rs crates to make use of all the new features they contain.
Recently I've been working again in the rust port of libgepub, libgepub is C code, but in the rust-migration branch almost all the real functionality is done with rust and the GepubDoc class is a GObject wrapper around that code. For this reason I was thinking about to use gnome-class to implement GepubDoc. Gnome-class is a rust lib to write GObject code in rust that's compatible with the C binary API so then you can call this new GObject code written with gnome-class from C. So, libgepub is the excuse to start to implement GIR in gnome-class.
Sonnerie is a time-series database. Map a timestamp to a floating-point value. Store multiple of these series in a single database. Insert tens of millions of samples in minutes, on rotational media.
Spirit is a crate that cuts down on boilerplate when creating unix daemons, with support for live configuration reloading.
Access Control Lists (ACLs) are an integral part of the Microsoft Windows security model. In addition to controlling access to secured resources, they are also used in sandboxing, event auditing, and specifying mandatory integrity levels. They are also exceedingly painful to programmatically manipulate, especially in Rust. Today, help has arrived — we released windows-acl, a Rust crate that simplifies the manipulation of access control lists on Windows.
I wrote this bit to give a some background about why Rand looks the way it does, and why it is time to make some changes. The new release, Rand 0.5, is getting almost ready fro release. Also I hope it is a bit entertaining to read about the history in combination with early Rust.
In this blog post I’d like to present toykio, a simple futures executor intended for learning about how executors with an event loop work. Toykio only provides a very minimal feature set: An event loop and TCP streams and listeners. However, it turns out that due to the fact that futures are composable, this is enough to build complex clients and servers.
A compatibility layer between 0.3 an 0.1 was developed. It is now possible to convert an 0.3 future into an 0.1 future and vice versa. Similar conversions for streams and sinks are also supported. Additionally, it is now possible to run 0.3 futures and async functions on Tokio’s executor. We have a dedicated blog post coming up that explains this in more detail.
The short version is, Tower Web 0.2 was just released and regular Rust attributes are now used instead of magic comments. The doc comment is replaced with #[get("/")]. This is thanks to Rust macro wizard David Tolnay. I also thought that it would be best to immediately push out 0.2 and then we can all pretend 0.1 didn’t happen.
We’ve just published a rust port of our PrettySize.NET library, now available via cargo and github. Like its .NET predecessor, PrettySize-rs aims to provide a comprehensive API for dealing with file sizes, covering both manipulation and human-readable formatting.
intl_pluralrules is a Rust crate, built to handle pluralization. Pluralization is the foundation for all localization and many internationalization APIs. With the addition of intl_pluralrules, any locale-aware date-, time- or unit-formatting (“1 second” vs “2 seconds”) and many other pluralization-dependent APIs can be added to Rust.
I’ve been working with diesel and serde. I use diesel for my postgres datastore, and serde for serializing/deserializing data to the web. Recently I came across a situation where I needed to define my type in diesel as well as implement deserialize in serde. The example below is a fairly simple so it makes for a good example to share so others can learn (and so I can remember how all this works next time I need it).
I’m excited to announce the brand new website/user-guide for the nphysics2d and nphysics3d crates: pure-rust 2D and 3D real-time physics engines with rigid bodies and joints! Online wasm-based demos are also provided (see for example the Multibody joints 34 demo).
With a brand new tutorial and a ton of new features, including prefabs, controller support, MP3 audio, localisation and an even better ergonomics!
Gutenberg 0.4.0 is out with custom taxonomies, image processing, improved shortcodes and more.
In this post I'd like to introduce a serdebug helper which is a drop-in replacement for #[derive(Debug)] with some of the advanced features that serde can provide.
This past fall, three former GnuPG developers began working on a new OpenPGP implementation in Rust called Sequoia. As it’s starting to shape up and become useful, I feel now is a good time to announce the project to the larger Rust community, and hopefully get some feedback before our first release.
Over the past several months, I’ve been working a web framework in Rust. I wanted to make use of the new hyper 0.12 changes, so the framework is just as fast, is asynchronous, and benefits from all the improvements found powering Linkerd. More importantly, I wanted there to be a reason for making a new framework; it couldn’t just be yet another framework with the only difference being I’ve written it. Instead, the way this framework is used is quite different than many that exist. In doing so, it expresses a strong opinion, which might not match your previous experiences, but I believe it manages to do something really special.
I’m super excited to reveal warp, a joint project with @carllerche.
Several new changes while working towards 0.3.
Using C libraries in a portable way involves a bit of work: finding the library on the system or building it if it's not available, checking if it is compatible, finding C headers and converting them to Rust modules, and giving Cargo correct linking instructions. Often every step of this is tricky, because operating systems, package managers and libraries have their unique quirks that need special handling.
Fortunately, all this work can be done once in a build script, and published as a
-sys Rust crate. This way other Rust programmers will be able to use the C library without having to re-invent the build script themselves.
wayland-rs is a set of crates providing generic APIs to manipulate the Wayland protocol, successor of X11 for linux windowing.
Here I am finally, after having hinted at the possibility and finally taken the time to write and merge quite an epic pull request, I can finally say it: wayland-rs is now a pure rust implementation of the protocol, rather than a crate of bindings to the wayland system C libraries.
Welcome to the inaugural post of the new futures-rs blog!
After several months of work, we’re happy to announce an alpha release of the new edition of future-rs, version 0.3. The immediate goal of this work is to support async/await notation (with borrowing) in Rust itself, which has entailed significant changes to the futures crate.
Hi. I’m Bradlee. I’ve mostly been a lurker in Rust for a while, making a couple small contributions here and there. So launching dtparse feels like nice step towards becoming a functioning member of society. But not too much, because then you know people start asking you to pay bills, and ain’t nobody got time for that.
But I built dtparse, and you can read about my thoughts on the process. Or don’t. I won’t tell you what to do with your life (but you should totally keep reading).
Thirty times faster than Handlebars, half the features! I am introducing a new templating engine for Rust that is designed to be robust and very fast! Why Zapper? Runtime templating is amazing, since you can reload templates on the fly or even allow users to provide their own templates, yet runtime templating engines are rarely fast. Templates that are statically compiled into your application can be super fast, but are completely inflexible. Recompiling and restarting your application just to change a template is especially boring. Zapper combines the flexibility of runtime templating with great performance!
Better HTTP Upgrades with hyper It’s been possible to handle HTTP Upgrades (like Websockets) in hyper if you made use of the low-level APIs in the server and client, but it wasn’t especially nice to...
In one of my recent blog posts, I talked about Event Sourcing with Aggregates in Rust. In that post, I was just beginning to explore how the Rust language and its strongly typed native data structures would allow me to express event sourcing concepts and primitives. I have now created an initial version of an Event Sourcing crate that you can explore on crates.io
I've been writing some code in Rust recently, and I thought it would be cool if I could take some of this Rust code and provide it as a native extension that I can call from Python. It turns out there are some amazing tools like PyO3 that make it easy to write fully featured Python extensions in Rust, with considerably less effort than writing a CPython extension manually.
To test out PyO3 I wrote a small Python extension in Rust, and I thought I would share some of the tips and tricks I encountered in getting this going. This post aims to serve as a quick tutorial showing how to write extensions in Rust, talking about why you might want to use something more powerful than just exposing a C library called using CFFI, and how PyO3 lets you write Python aware extensions in Rust.
I’ve been working on a project called Thruster recently, and needed a way that a developer could reasonably use templates. Thruster is a middleware based web server written in Rust (get it, th-rust-er? I’m working on my tight 5 for amateur night at The Apollo,) and as such, I needed a way to load HTML templates and insert variables into them in a performant way. Rather than poking around the numerous existing libraries and choosing one made by someone I don’t know — stranger danger! — I decided to make it myself. This article is about that journey, the unbelievably thrilling adventures of writing my first proc_macro_derive in Rust.
I gave a talk about lyon at RustFest Paris. This post is the introduction of the talk, wherein I introduce vector graphics and try to get the audience somewhat excited about it. Things will get technical in the follow-up posts.
Compressing your files is a good way to save space on your hard drive. At Dropbox’s scale, it’s not just a good idea; it is essential. Even a 1% improvement in compression efficiency can make a huge difference. That’s why we conduct research into lossless compression algorithms that are highly tuned for certain classes of files and storage, like Lepton for jpeg images, and Pied-Piper-esque lossless video encoding. For other file types, Dropbox currently uses the zlib compression format, which saves almost 8% of disk storage.
We introduce DivANS, our latest open-source contribution to compression, in this blog post.
This is a post about an interesting testing technique which feels like it should be well known. However, I haven’t seen it mentioned anywhere. I don’t even have a good name for it, I’ve semi-discovered it in the wild. If you know how this thing is called, please leave a comment!
Tarpaulin (or cargo-tarpaulin) is a code coverage tool for Rust. Last year was pretty busy with the launch of the project and the rush of issues as people started to use it so this is just a chance to look at what’s new with version 0.6.0 and what’s planned for the rest of this year.
In this blog post, I would like to present a research project I have been working on: Trying to use QML from Rust, and in general, using a C++ library from Rust. The project is a Rust crate which allows to create QMetaObject at compile time from pure Rust code. It is available here: https://github.com/woboq/qmetaobject-rs
Crates.rs, an alternative opinionated front-end to crates.io:
All readmes are displayed whenever possible, and if there’s no or poor readme, doc comments are shown too.
It combines information from multiple sources, e.g. byline is a compact amalgamation of Cargo.toml, crates.io, and GitHub contributors, so you get an idea who wrote the code even if authors forgot to keep Cargo.toml up to date.
Crate popularity is displayed as top-N position in its most relevant category, e.g. “#5 in Cryptography”, which is more meaningful than absolute download numbers.
Optional dependencies display which feature or platform they’re for.
Version history is summarized to help see at a glance whether a crate gets regular updates and how often it has breaking changes.
Recognizes sys crates even if they’re not called -sys and shows when build.rs is used.
Category pages fit more crates on screen despite looking less cluttered and having easier to read descriptions.
All categories and their representative crates are right there on the homepage.
Syntax highlighting everywhere, including code blocks.
Everything works without JS and gets indexed by search engines, so I’m hoping the site will help find crates.
Today sees the release of hyper v0.12.0, a fast and correct HTTP library for the Rust language.
This release adds support for several new features, while taking the opportunity to fix some annoyances, and improve the extreme speeds!
While the http crate generally has a great API I have been unsatisfied how it handles URLs. To create a HTTP request a full URL is needed with a scheme (http/https), authority (example.org) and a path (/search?q=rust) but http does enforce this and allows you to only state the path. This means both clients and servers are either unable to determine protocol and and authority information or have to do this manually.
More than a year ago a friend of mine wanted to learn a bit more about Rust by trying out a project. He had a nice project in mind which suits Rust quite well I think. For fun I joined his effort and created an implementation at the same time as he did, discussing and comparing along the way. In this post I’ll tell you about the project specifics, but the point of the post is more an encouragement. If you’ve read about Rust before but haven’t tried it yet, find a small project like the one below, and learn Rust in a fun and hands-on way yourself. It’s a great programming language, I highly recommend it.
It took nearly 6 months of development and the library went through nearly 5 entire rewrites. Compare that to previous major releases, which took a month at most to do. But it was worth it! This new release cleans up a lot of old bugs and unintuitive behaviours, simplifies some common patterns, is faster, uses less memory, gives better errors, but the way parsers are written stay the same. It’s like an entirely new engine under the same body work!
Currently, testing UIs is difficult, but with gtk-test you can test basically everything UI-related way more simply.
As happy Rust users ourselves, it makes us even happier to be able to say that we now have a Sentry Rust SDK. This means you and your Fungiculture can now report panics, failures, and other types of incidents to Sentry.
Not only is the SDK new and fancy, it is also one of the first ones that follows our new API guidelines for Sentry SDKs, which makes it even newer and fancier than you might normally expect.
It took a bit longer than I had initially hoped (as it always does), but a new Tokio version has been released. This release includes, among other features, a new set of APIs that allow performing filesystem operations from an asynchronous context.
Last week, I was working on improving the integration of Rust with GNOME libraries at the third Hackfest, which happened this time in Madrid.
A suite of cryptographic libraries and protocol implementations, written in the systems programming language Rust, for creating blazingly-fast, production-quality cryptographic applications.
This library implements several of the more commonly useful immutable data structures for Rust. They rely on structural sharing to keep most operations fast without needing to mutate the underlying data store, leading to more predictable code without necessarily sacrificing performance.
MesaLink is a memory-safe and OpenSSL-compatible TLS library. Since 2014, the industry has seen a huge loss due to memory vulnerabilities in TLS stacks, such as the infamous "Heartbleed" bug. MesaLink is created with the goal of eliminating memory vulnerabilities in TLS stacks. MesaLink is written in Rust, a programming language that guarantees memory safety. This significantly reduces the attack surfaces, which facilitates auditing and restricting the remaining attack surfaces. MesaLink is cross-platform and provides OpenSSL-compatible APIs. It works seamlessly in desktop, mobile, and IoT devices. With the growth of the ecosystem, MesaLink would also be adopted in the server environment in the future.
The newest release of hyper includes some lower-level connection APIs for both the server and client. Notably, this allows using hyper send and receive HTTP upgrade requests. The most popular of these is Websockets.
I am learning Clojure for the past one year and I thought making open source contributions is a great way to interact with the community. I made a post previously on using Clojars metadata to analyse JDK 9 and Clojure 1.9 issues that helped me file issues to ensure compatibility. I used the same method here to find the modules that were broken on a nightly version of a rustc due to a recent stabilisation.
To close out a great week, there is a new release of Tokio. This release includes a brand new timer implementation.
I’m happy to announce a new release of Tokio. This release includes the first iteration of the Tokio Runtime.
On behalf of the futures-rs team, I’m very happy to announce that the master branch is now at 0.2: we have a release candidate! Barring any surprises, we expect to publish to crates.io in the next week or two.
You can peruse the 0.2 API via the hosted crate docs, or dive right in to the master branch. Note that Tokio is not currently compatible with Futures 0.2; see below for more detail.
I’m planning to release a 1.0.0 version of failure on March 15. Once this happens, I don’t plan to release any further breaking changes to the failure crate (though maybe someday in the distant future).
Breaking changes in 1.0 failure is in a somewhat unique position as being a significant part of the public API of other libraries that depend on it. Whether they use the Error struct or derive Fail for a custom error type, this becomes a part of the API they expose to other users.
It’s been a while since I last suggested Mutation Testing in Rust, almost two years ago. Since then I got sidetracked a lot, and later lost interest. Just one more cool project I couldn’t afford to take on. But as things go, my interest in mutation testing was rekindled, and I decided to give it a shot and do more than blogging about it.
iui, the Improved User Interface crate, has just gotten its 0.2 release. Improved User Interface is a set of safe, idiomatic Rust bindings to platform native GUI libraries (Win32API, Cocoa, and GTK+) via libui and ui-sys. Highlights of this release include: Correct, semantic use of mutability, GTK+ theme application per window, and menu bars and file open/create modal support.
Bincode is a serializer implementation for Serde. If you stick a #[derive(Deserialize, Serialize)] on your struct, Bincode can efficiently serialize and deserialize those structs to and from bytes. Bincode is unique in that it’s a format that was built specifically for the Rust serialization ecosystem. Tight coupling with Serde allows Bincode to be very fast and serialize to very small payloads.
After a couple of weeks of work, I'm now happy to release the first version of tql, the easy-to-use ORM for Rust. While the focus was to make tql work on the stable version of the compiler, I also added some new features. The most notable new feature is the support for SQLite: now tql supports SQLite as well as PostgreSQL. The support for SQLite is almost as complete as the one for PostgreSQL: the only missing function is not implemented because the backend (SQLite) does not support it.
For some time now I develop a Rust library for asynchronous programming with coroutines, called Corona (note there’s a version 0.4.0-pre.1, but Crates prefer the „stable“ 0.3.1). I believe it is starting to be useful, so I wrote this description to show what it is good for and how it fits into the big picture of Rust. There’ll be some more changes, though, at least because Tokio just released a new version (and Futures plan to do so soon), so Corona will have to adapt.
Combine is a parser combinator library for the Rust programming language. I first announced version 3 of Combine back in August and back then I definitely expected to have a stable version by now. However other projects (cough gluon cough) got in the way and Combine fell to the wayside. It didn’t help that I didn’t have a killer feature for 3.0 either, user-defined error types make it possible to define parsers usable in #[no_std] crates which is great when you need it but it is still a fairly niche use-case.
I'm happy to announce that today, the changes proposed in the reform RFC have been released to crates.io as tokio 0.1. The primary changes are: Add a default global event loop, eliminating the need for setting up and managing your own event loop in the vast majority of cases, and decouple all task execution functionality from Tokio.
Criterion.rs is a statistics-driven benchmarking library for Rust. It provides precise measurements of changes in the performance of benchmarked code, and gives strong statistical confidence that apparent performance changes are real and not simply noise. Clear output, a simple API and reasonable defaults make it easy to use even for developers without a background in statistics. Unlike the benchmarking harness provided by Rust, Criterion.rs can be used with stable versions of the compiler.
faster began as a yak shave, created to aid base💯 in its quest to become the fastest meme on Github. Writing an explicit AVX2-accelerated version of base💯's encoder and decoder, then realizing I'd have to do the same thing again to see the speedups on my Ivy Bridge desktop, pushed me to make this library. Months later, it has blossomed into its own project, and has eclipsed base💯 in both popularity and promise.