R2 began from a desire to learn the internals of the HFSC scheduler by writing it from scratch. It started with our decision to use HFSC at a startup I was working at. We were having troubles with a home grown scheduler and decided to stop investing time and effort on that – rightly so since our engineering team’s size was in single digits. And the question was which open source scheduler to use. Someone else in the team had used BSD’s HFSC before, so the decision was quick. I took on the task of porting it and making it work the way we wanted. I didn’t want to just port it blindly without understanding the internals, because I knew I would have to fix bugs in it :). There really is just ONE piece of documentation on hfsc, which is the original paper with densely packed theorems and math.
Years flew by, and my anxieties went away, only because we stopped using HFSC and built something custom as the startup grew. But that paper remained in my bag and I kept reading it in every flight, it still stays in my bag wherever I travel. And at some point I thought I understood enough of the paper that it was time to write the code myself, but in a new language. Go was the popular kid in town and I thought I will write it in Go. At the same time my brother was experimenting with this new kid in town Rust and writing blogs about his experiments, so I started looking at Rust also. And for whatever reason, I just liked Rust more than Go and dropped Go and switched to Rust. And at some point when I got enough understanding of Rust and I was starting to write the scheduler, I thought why just write a scheduler, why not write an entire packet forwarding system. By then I had worked on plenty of networking code bases big and small, good and bad, success and failures, proprietary and open. And I had seen good designs and bad designs from others, made good designs and bad designs myself, and I thought why not put all the learnings together and build something bigger.
Web and Network ServicesWeb applications, web assembly, network daemons, etc.
In this tutorial I assume that you have already obtained a certificate from LetsEncrypt. If not, please start with that process first. Once you have obtained the certificate, we will need two files that are included int certbots output: fullchain.pem and privkey.pem. Place these into your certificates folder inside the Rust directory.
When working with Rust + Webassembly, you might want to use some crates in your project. Not all crates work out of the box with Webassembly yet, especially those that rely on System Libraries, File I/O, Networking, etc. With proposals such as WASI or WebAssembly Interface Types, these might work eventually but it isn’t the case yet.
The Rust Wasm book suggests: A good rule of thumb is that if a crate supports embedded and #![no_std] usage, it probably also supports WebAssembly.
I've long wanted to be able to index and search my own content -- to set up ad hoc "search engines" for different purposes as they come up, with sophisticated search strategies and customized, queryable metadata.
And when I heard about Tantivy, it immediately struck me as a potential vehicle for the search aspirations I'd harbored, as it was reportedly 1) fast, 2) powerful, and 3) written in Rust (all critical components in the Jonathan Strong "potential hobbyist project" rubric).
When it came time to set up the search engine for this site, I knew what I wanted: a Tantivy-based search engine, even if it was a tad bit overkill.
I recently was in the situation that I had to implement a connection to the JIRA API, which means the privilege of having to deal with their OAuth 1.0a implementation.
Unfortunately, JIRA at this point in time uses RSA-SHA1 in their OAuth 1.0a scheme, which I wasn’t able to find a crate for I was happy with (the OAuth client part, not the crypto). So I decided to write it myself and this post is a description of some parts that came out of that endeavour.
Calling all Rustaceans, Rustafarians, Ferrosities, and rustlers - we’ve got an exciting update! Active full time work on a Rust-IPFS implementation has commenced, building on the great work by Parity in rust-libp2p. Equilbrium is spearheading the new community and implementation with support from Protocol Labs, and is looking for additional Rust devs itching to help build a new language implementation of the InterPlanetary File System combining the performance and resource utilization benefits of Rust with a keen eye on conformance to the IPFS spec.
The story of the implementation of the BroadcastChannel Web API in Servo, a large multi-process and multi-threaded Rust codebase.
One of the interesting prospects of WebAssembly is that it can be used for safe and portable sandboxing on systems that aren’t web browsers. This is somewhat undersold by at least some parts of the wasm ecosystem, in my opinion, but has some real value. Being able to distribute executables and libraries in a portable bytecode that gets locally compiled seems to me like a strict improvement on distributing platform-specific binaries, even if the programs are open source and you can just compile them from source if you really want to. Given that we’re increasingly living less in an x86_64 monoculture, with diverse good non-x86 hardware widely available, this idea seems more and more useful. So, as someone who mostly wants to write portable programs that run on desktop-style hardware, let’s try to actually use webassembly for this and see how it goes.
This post is Part one of a series on how to build a highly fault tolerant system using bastion-rs, and expose an async web api with Tide. Part one is about setting up a simple Tide server, and covers initial steps such as adding dependencies, some basic routing and the request/reply model.
In this tutorial, we are going to build pagination for our API results. To solve this problem we need to hook into Diesel’s query builder, so I will also cover how we can do that. This tutorial builds on my tutorial on building a REST API, so I highly recommend reading that first or cloning the code from GitHub.
In this article I'm going to write about my journey of building Commentable.rs - a privacy oriented Disqus-like comment hosting solution written in Rust and running on AWS Lambda.
In the Foundation’s engineering roadmap for 2020, we overviewed our plans for Zebra, our Rust implementation of Zcash. Announced last summer at Zcon1, Zebra aims to support the core strength of Zcash – its best-in-class cryptography – by placing it on a solid foundation, providing a modern, modular implementation that can be broken into components and used in many different contexts. In that post, we briefly described the new network stack we designed and implemented for Zebra. As a fork of Bitcoin, Zcash inherited Bitcoin’s network protocol; in this post, we’ll do a deep dive on Zebra’s network stack.
After reading the Cloudflare blog post on how to receive 1M packets per second, I wondered: How fast can we go with Rust and Tokio?
So. Serializing IPv4 packets. Easy? Well, not exactly.
IPv4 was annoying to parse, because we had 3-bit integers, and 13-bit integers, and who knows what else. Serializing it is going to be exactly the same.
Right now, we don't have a way to serialize that.
Let's take the version and ihl fields, both of which are supposed to take 4 bits, together making a byte.
In this tutorial, we are going to create a REST API in Rust with Actix web 2.0 and Diesel. We will be using Postgres as our database.
The point of having Rust-based WebAssembly in Node.js is to offload some compute-heavy parts from Node.js to Rust, which runs significantly faster for tasks that require algorithmic or memory optimization. The Rust compiler is smart enough to optimize its base functions, which makes it run faster. However, the combination of a single-threaded Node.js and Rust-based WebAssembly, which also runs without threading, won’t do much to save you time or resources.
The idea is to use a Node.js module worker_threads to spin up a new Rust WebAssembly computation without having a sync operation waiting in the main thread.
So we've managed to look at real network traffic and parse it completely. We've also taken some ICMP packets, parsed them, and then serialized them right back and we got the exact same result.
So I know what you're thinking - let's just move our way down the stack again - stuff that ICMP packet in an IP packet, then in an Ethernet frame, and then serialize the whole thing.
We need to think about what we're doing. We (a host) are attempting to ping another host. By sending ICMP packets that are inside IPv4 packets that are inside Ethernet frames. That means we need four principal pieces of information.
In this post, we will learn how to use Rust Warp. We will start from the current official example at GitHub. Then, we will learn how to modularize it.
WebSocket (WS) support for Tide has been a long-anticipated feature. More recently requests for Server Sent Events (SSE) support have started to pop up as well. In this post we'll look at the motivation, requirements, and design for WS and SSE support in Tide.
Today I want to show how to build a simple microservice. We will use Actix, Tokio-Postgress, and other libraries. We will use Postgress as our source of truth and we will run it in docker (for development sake). We also will use Barrel + some customs migration structure I created. The code will be all async and non-blocking IO. I hope you have fun, let’s get started.
Back in 2014 I was fetching frontpages of the top million websites to scan them for a particular vulnerability. Not only have I found 99,9% websites to be vulnerable to a trivial attack, I’ve also found that curl command was randomly crashing with a segmentation fault, indicating a likely vulnerability in libcurl — the HTTP client library that the whole world seems to depend on.
By that time I was already disillusioned in the security of software written in C and the willingness of maintainers to fix it, so I never followed up on the bug. However, this year I decided to repeat the test with software written in a language that’s less broken by design: Rust.
Here’s how 7 different HTTP clients fared.
Warp is a Rust web server framework focusing on composability and strongly-typed APIs. Today sees the release of v0.2! The most exciting part of this release is the upgrade to std::future, so you can now use async/await for cleaner flow control. Due to how warp encourages composition of filters, this is most noticeable at the “ends” of a filter chain, where an application is doing its “business logic”, converting input into actions and replies. And that’s where most of the app code is! Services In the original release of warp, I wrote: We’d like for warp to be able to make use of all the great tower middleware that already exists. As part of this release, that is now possible! Any Filter which returns a Reply can now be converted into a Service using warp::service(filter).
This is the second part of a blog post series. You can read the first part on our website.
This article is part 9 of the series Making our own ping. Now that we’ve found the best way to find the “default network interface”… what can we do with that interface?
Well, listen for network traffic of course!
Okay, I lied.
I’m deciding - right this instant - that using wmic is cheating too. Oh, it was fair game when we were learning about Windows, but we’re past that now.
We know there’s IPv4 routing tables, and we know network interfaces have indices (yes, they do change when you disable/enable one, so ill-timed configuration changes may make our program blow up).
We also know how to call Win32 APIs directly, and we know that WMI really is more geared toward system administrators than it is toward developers.
There’s no way a query language is the canonical way to retrieve that kind of information. Also, we’ve gone slightly over our crate budget.
Let’s fix that right now.
I’m playing with the usual format to do a medium dive into an active frontier: web technology in Rust. The language and its ecosystem have seen a lot of change over the last few years and I would have advocated against it in serious projects for its evanescent APIs alone. But there’s good reason to be optimistic about the state of Rust coming into 2020 – critical features and APIs have stabilized, substrate libraries are ready and available, tooling is polished to the point of outshining everything else. The keystones are finally set, ready to bear load.
tokio heroku async
In an effort to understand the new Rust async/await syntax, I made a super-simple app that simply responds to all HTTP requests with Hello! and deployed on heroku.
If you want to skip right to the punchline, the source code and README instructions can be found on https://github.com/ultrasaurus/hello-heroku-rust
I've been studying WebAssembly recently, which has included porting some of my m4vga graphics demos. I started with the Rust and WebAssembly Tutorial, which has you use fancy tools like wasm-pack, wasm-bindgen, webpack, and npm to produce a Rust-powered webpage.
And that's great! But I want to know how things actually work, and those tools put a lot of code between me and the machine.
The resulting WebAssembly module will be less than 300 bytes. That's about the same size as the previous paragraph.
Let’s set aside our sup project for a while.
Don’t get me wrong - it’s a perfectly fine project, and, were we simply rewriting “ping” for Windows in Rust, we could (almost) stop there.
We’re currently using the operating system’s facility to speak ICMP, which is great for a bunch of reasons: we can be sure that whatever flaws there are in the implentation, all “native” Windows programs suffer from it as well.
As a bonus, the binary size is reasonable, (looks like 175KiB for me at time of this writing - built with cargo build --release, of course).
But we didn’t come here to make sensible decisions for production-grade software. We came here to bind Win32 APIs and dig our way down the network stack, and we’re all out of APIs to bind.
I have recently started digging deeper into Amazon Web Services (AWS), and particularly into serverless lambda functions. After a few successful experiments with Python, I started wondering how hard it would be to get this stack running with Rust. Turns out, it wasn’t too hard after I had overcome a few of the initial hurdles which mostly consisted of figuring out how the different parts work together under the hood. This article is intended to serve as a guide on how to write serverless AWS lambda functions in Rust and how to deploy them onto Amazon Web Services.
Wasm Pack is a flexible framework in the Rust ecosystem to compile code to WebAssembly. It allows web developers to outsource their most demanding data processing tasks to some safe and high-performance Rust code. But it also provides Rust developers with an easy access to Web technologies for creating GUIs for their libraries. Elm is a fairly new language that has revolutionised the way highly interactive web pages are built. This tutorial presents a way to use Elm in conjunction with Rust to produce powerful web applications.
reqwest is a higher-level HTTP client for Rust. Let me introduce you the v0.10 release that adds async/await support!
Recently I've been working on a rust web service written using Rocket and Diesel. A big issue I came across was handling muitpart forms (AKA content-type: multipart/form-data). Rocket doesn't support them currently (although it is planned!) but you can still get it working right now.
This is the third post in a series of posts exploring web services related technologies.
With the web service implementations in place, I evaluated them using an off-the-shelf benchmarking tool. This post documents the client-side observations from this evaluation.
I originally wrote Sorta Secret a year ago in Rust using actix-web and deployed it, like most services we write at FP Complete, to our Kubernetes cluster. When Rust 1.39 was released with async/await support, and then Hyper 0.13 was released using that support, I decided I wanted to try rewriting against Hyper. But that's a story for another time.
After that, more out of curiosity than anything else, I decided to rewrite it as a serverless application using Cloudflare Workers, a serverless platform that supports Rust and WASM.
This post will describe my experiences doing this, what I thought worked well (and not so well), and why you may consider doing something like this yourself.
A review of preparing Trust-DNS for async/await in Rust
What started as a brief sojourn to learn the new std::future::Future in Rust 1.36, slowly became a journey to fully adopt the new async/await syntax in Rust. The plan had been to merely update to the new Future API, trying to keep the minimum Rust version as low as possible. This was ideally to keep the libraries compatible with more Rust users, but it became aparent that this wasn’t really feasible. For a number of reasons, primarily, all of the underlying libraries Trust-DNS relies upon were moving in this direction, which made the task a fools errand. Additionally, adopting async/await simplified much of the code. This post is the announcement of the 0.18 release, representing a few months of work.
We’ve just spent a lot of time abstracting over LoadLibrary, but we still have all the gory details of the Win32 ICMP API straight in our main.rs file! That won’t do.
This time will be much quicker, since we already learned about carefully designing an API, hiding the low-level bits and so on.
I’ve just released the version 0.3.0 of Meuse, an alternative registry for the Rust programming language.
Using a Rust image manipulation library via WebAssembly, service workers can manipulate images on the fly. You can see a simple example of the code here and a fully-fledged example here.
After a few months of alpha development, the final release of hyper v0.13.0 is now ready!
In this post, I will start to talk about the exciting technical stuff around Paddlers. If you haven’t read my other posts (#0, #1, #2), Paddlers is a game in which your goal is to make ducks happy. As of recently, I also have an online live-demo where you can look around and see the current progress. (You can log in using a test user: Username: Tester, password: 1) Just don’t expect too much, it is not a playable game, yet.
Paddland, the world in which the game plays, is a colorful place to relax and have fun. And today, I show you how building this is also fun, using Rust, WebAssembly, and some good old creative freedom to glue everything together.
A couple years ago, we released the beginning of the http crate. It’s purpose was to allow a common API for the ecosystem to interact with HTTP types, without those types referring to a specific implementation. We’ve seen great things sprout up since then!
This is a very simple authentication server but I hope this post gave you the building blocks needed to expand it for your own needs. I strongly recommend taking a look at the warp documentation and if you need help, don't hesitate to ask me. Also, any feedback is welcome!
Today we're happy to announce the release of Tide 0.4.0 which has an exciting new design. This post will (briefly) cover some of the exciting developments around Tide, and where we're heading.
Setting up a GraphQL server with Rust, Juniper, Diesel, and Actix; learning about Rust's web frameworks and powerful macros along the way.
Multi-value is a proposed extension to core WebAssembly that enables functions to return many values, among other things. It is also a pre-requisite for Wasm interface types.
I’ve been adding multi-value support all over the place recently: I added multi-value support to all the various crates in the Rust and WebAssembly toolchain, so that Rust projects can compile down to Wasm code that uses multi-value features. I added multi-value support to Wasmtime, the WebAssembly runtime built on top of the Cranelift code generator, so that it can run Wasm code that uses multi-value features.
Now, as my multi-value efforts are wrapping up, it seems like a good time to reflect on the experience and write up everything that’s been required to get all this support in all these places.
I was unhappy about Deliveroo's search interface and built my own using some web scraping, indexing with Tantivy search library and a bit of React on frontend.
Today we announce the formation of the Bytecode Alliance, a new industry partnership coming together to forge WebAssembly’s outside-the-browser future by collaborating on implementing standards and proposing new ones. Our founding members are Mozilla, Fastly, Intel, and Red Hat, and we’re looking forward to welcoming many more.
Wasmtime is a standalone wasm-only optimizing runtime for WebAssembly and WASI, using Cranelift. It runs WebAssembly code outside of the Web, and can be used both as a command-line utility or as a library embedded in a larger application.
This experiment continues the work done in our pretend suite of microservices exposed via API Gateway to form an API with a code name of Slipspace in a mock company called STG. Slipspace drives are how the ships in the Halo universe travel so quickly to different sectors of the galaxy through something called Slipstream Space, so thought it was cool for a name requiring awesome warp API speeds.
Several months ago we announced that we were providing a new public time service. Part of what we were providing was the first major deployment of the new Network Time Security (NTS) protocol, with a newly written implementation of NTS in Rust. In the process, we received helpful advice from the NTP community, especially from the NTPSec and Chrony projects. We’ve also participated in several interoperability events. Now we are returning something to the community: Our implementation, cfnts, is now open source and we welcome your pull requests and issues.
There is a wonderful Magic: The Gathering scene here in Durham, NC. Unfortunately, Magic often deserves the trope that it gets of being an expensive hobby. In order to enjoy the game on a budget, myself and a group of locals created a new format: Paper Dreadful. The format is simple: create a 60 card deck with a maximum of 4 of any card (other than basic lands) and the total cost must be $20 or less on that day. The same rule applies with the sideboard, except it must by $5 or less.
The problem here is that Magic cards change in price all the time. What happens if I build a deck and then a card skyrockets in price?
In this post, we will learn how to use Rust Tonic gRPC crate. We will learn how to implement CRUD with Postgresql database. The purpose of it is to help you to have the working Rust Tonic code and start your own porject immediately with it.
I’ve written an event sourcing bank simulation in Clojure (a lisp build for Java virtual machines or JVMs) called open-bank-mark, which you are welcome to read about in my previous blog post explaining the story behind this open source example. As a next step, specifically for this article I’ve added SSL and combined some topics together, using the subject name strategy option of Confluent Schema Registry, making it more production like, adding security, and making it possible to put multiple kinds of commands on one topic. We will examine how the application works, and what was needed to change one of the components from Clojure to Rust. We’ll also take a look at some performance tests to see if Rust might be a viable alternative for Java applications using Apache Kafka®.
The thing about contributing to Servo is that you keep learning new things about the Web platform. Personally, I had never used messaging on the web when developing web applications, and I find it a fascinating idea.
Web-messaging enables developers to provide cross-site API’s without having to go through a server, all the while leveraging the client-side security model of the Web. And since it happens on the client, it could be more transparent to the end-user, and probably easier to block if necessary.
Implementing message-ports also raises interesting architectural questions. In an earlier Web(like, in 2017), an API like message-ports could have been implemented with some sort of cross-thread communication. In 2019 however, it’s going to have to go across process. Why? Something known as “Spectre”.
At Meili, we needed a tool that allows us to monitor our pods, we already have vigil which health checks our front page and backend, but the number of these services is limited. We do not pop new front or backend servers dynamically (for now). When we create new search engines for the user we instanciate a kubernetes pod, we need to monitor the health of this service. Adding each of those URLs by hand in the vigil config file is not a solution.
So we decided that we needed a simple tool, a tool that can accept HTTP requests to register/unregister URLs to health check. We use the new async/await Rust syntax along with tide for the http server, no big deal here.
I wrote a basic search module that you can add to your static website. It's very lightweight (around 50kB-100kB gzipped), should work with Hugo, Zola, and Jekyll. Only searching entire words is supported. Try the search box on the left side for a demo. The code is on Github.
So… You want to write an HTTP server. Well, you’re in luck, The Internet Engineering Task Force (IETF) is here to help us. Whenever they’re not fighting with fiber cables (at least that’s what I imagine they do), they are writing useful specifications for us! Well, first you prepare yourself a big pot of coffee. Once you’re appropriately caffeinated, you read the 57897 word RFC 2616 specification, written by none other, than the IETF. Exciting.
I am pleased to finally announce a crate that I have been working very hard on for the past few months. tonic has finally hit the initial 0.1.0-alpha.1 release! Stable releases will follow in the coming months. Tonic is a gRPC-over-HTTP/2 implementation focused on high performance, interoperability, and flexibility. This library was created to have first class support of async/await and to act as a core building block for production systems written in Rust.
I recently had to create a simple webapplication for a project i was working on. Luckily i was free to choose the technology for that task so i decided that it was great oportunity to try out how applicable is Rust for web development. There is already a bunch of mature web servers available such as Rocket or Actix. However as far as i can tell Yew is the only actual framework that allows you to create the frontend application.
Over the summer, I prototyped a bunch of web apps whose ideas had been floating in my mind for a long time. I spent quite a bit of time learning about REST APIs and, as part of these exercises, implemented skeletons of REST servers in both Go and Rust. The app prototypes have gone nowhere but I thought of sharing the skeletons I built if only to serve as templates for myself in future work. Thus, in this post, I will cover these skeleton demos and I will compare them.
JS/TS is killing me. Now that I use Rust everyday, I have to try Yew, a front framework for Rust to create WASM apps, I spotted it long time ago but it wasn’t mature enough for my use case… guess what — now is the time ! The project is still alive and well! Rust fullstack dev it is.
This is a quick & dirty guide to building a web app using an all-rust stack: Diesel as an ORM, Rocket as a web framework, and Seed for the [WebAssembly] frontend. Read this book to learn how to quickly throw together a prototype application, all with your favorite programming language.
It is already possible to run WebAssembly inside C, C++, PHP, Python, Ruby, Go, C#, R. Now it is time to run WebAssembly inside Postgres.
In the previous post How to use a modal in Rust, we learnt how to write a simple image modal with Rust frontend. We could find that we can build components visible only in specific conditions. In this post, we will learn how to use routers in Rust frontend with Yew. You will find that it is easy with an example.
As you might have heard, async/await is coming to Rust soon. This is a big deal. Rust has already has popular crates (tokio, actix) that provide asynchronous concurrency, but the async syntax coming to stable in 1.39 is much, much more approachable. My experience has been that you can produce and reason about application flow much more easily, which has made me significantly more productive when dealing with highly concurrent systems. To kick the tires of this new syntax I dug into the nightly branch, and built a high-performance TCP client called clobber. In this post I'll talk about why I think async/await in Rust is a big deal, and walk you some of the code in clobber.
Build a basic chat app with the Rust Programming Language.
I start to get sessions working by POSTs OAuth requests to an API endpoint and create accounts using Diesel to talk to the database.
Playing around with Rust+WebAssembly and wrote a little crate to perform Levenberg-Marquardt fits. Brief comparison of SciPy & Rust fitting.
I implement Amazon's DynamoDB for my Rust web app session management storage with the Rusoto crate and some deeply nested code.
Welcome to the second installation in my series on taking the Practical Networked Applications in Rust course, kindly provided by the PingCAP company, where you develop a networked and multithreaded/asynchronous key-value store in the amazing Rust language. In the previous, and initial, post I implemented the course module of making the fundamental key-value store functionality, based around the Bitcask algorithm, which would only allow for local usage on your own computer. In the second module of my course work, I add networking functionality, dividing the application into a client/server architecture so that clients can connect to servers across the network.
It is time to get acquainted with Metal IO, low-level cross-platform abstraction over epoll/kqueue written in Rust. In this article I will show and explain how to write simple single-threaded asynchronous TCP server, then teach it to mock HTTP protocol, and then benchmark it with ab/wrk. The results are about to be impressive.
The PingCAP company, makers of the TiDB NewSQL database and the TiKV key-value store, have kindly made publicly available, as well as open-sourced, a set of training courses that they call the "PingCAP Talent Plan". These courses train programmers in writing distributed systems in the Go and Rust languages. They are originally intended by PingCAP to train students, new employees and new contributors to TiDB and TiKV and focus as such on subjects relevant to those projects, but are still appropriate to anyone with an interest in learning to make distributed systems in Go and/or Rust.
In the previous post How to modulize your Rust Frontend, we learnt how to use impl, functions and Yew components. They help you to find errors and organize your Rust frontend project. In this post, we will include server side code with ws-rs. It will help us to build complete Rust Full Stack chat app similar to what we made at How to start Rust Chat App.
In the previous post How to use markdown with Rust Frontend, we learnt how to render markdown in Rust frontend and include CSS files from your previous frontend project. You can use a text, image, video, markdown or any HTML elements you want for your Rust frontend. In this post, we will learn how to modulize your Rust Yew frontend app. It will be easy after you learn how to use impl, function, and components for that.
After taking a break from working on Arrow and DataFusion for a couple of months to focus on some deliverables at work, I have now started a new PoC project to have a second attempt at building a distributed platform with Rust, and this time around I have the advantage of already having some foundational pieces in place, namely Arrow and DataFusion. I have also been gaining experience with Kubernetes recently and I could clearly see how this would simplify the creation of a distributed platform. The pieces really are starting to fall into place.
The new project is called Ballista and is a fast moving PoC taking a top down approach to building a distributed platform.
I'm happy to announce, that we've just finished one of the biggest European music festival Atlas Weekend which took place in Kyiv, Ukraine. As a year ago, backend for technical purposes of festival is written with Rust: Tickets exchange, entrance control, powerful user management with distributed configurable permissions, backstage pass, control of security workers, their work time, cars entrance, warehouse system for rfid bracelets with full history and control of each rfid. This year we registered more than 6000 workers, each of them has full info tied to bracelet: name, phone, photo, available permissions, time to access festival.
In this blog series, I will experiment with Rust as a safer and simpler C/C++ replacement. The idea is to combine a couple of C dependencies in Rust, to do some work using the dependencies in Rust and to expose a final API from a Rust library via C ABI. Then I will consume the same exported Rust methods from a number of modern languages/platforms such as C#, Python, Java, Nodejs, R, Go and even back from C/C++.
In the previous post full-stack Rust with Yew, we learnt how to prepare minimal files to build a full stack Rust web app. You can do whatever Rust allows with it.
In the previous post How to use Rust Yew, we learnt how to prepare minimal files to build webassembly files with Yew for Rust frontend. We will advance it with some Rust server side code and write a bash file to automate the process.
Creating a simple console application in Rust for fun and profit, including some nice tricks to push async programming to the limits.
Dev's offline page is fun. Can we do that with Rust and WebAssembly? The answer is yes. Let us do it.
What are we gonna do? Create a WebAssembly application that takes a string in markdown format and converts that into HTML.
In this article we are going to create a WebAssembly Hello World program with Rust. You might be thinking that a WebAssembly Hello World tutorial could be found almost everywhere, including one on Mozilla website and rustwasm.github.io.
In this post, we will prepare development environment for Rust Yew. Then, we will write minimal code with it and learn how to deploy it in your website also.
A hands-on introduction into WebAssembly ( Wasm ). Containing simple examples and tutorials on how to implement concepts and various tasks using Wasm.
It’s been a year since I wrote a little proxy with Rust, it is one of my first projects with this language and I learned a lot writing it. To sum up a bit, the main goal of this proxy is to be really simple to use and easy to extend with middlewares. It targets HTTP APIs and can be used in front of a lot of services. This proxy has been used in multiple projects, with microservices architecture and so on…
As with most hobby projects, my htsp-rs implementation became a side-side-side project, and it will probably stay that way. Another property of those hobby projects is that I now-and-then pick them up again, and htsp-rs has come to that point now. The reason for picking it up again is different from the reason of its existence though, and that’s a good thing: it means I have multiple reasons for the crate’s existence. Initially, I wanted it as a back-end for a livestreaming app on SailfishOS. Today, I want it as back-end for mpd’s TVHeadend support.
I've been experimenting with techniques for collaborative editing in Rust recently and I'd like to share my first functional prototype with you. The demo uses Rust and WebAssembly, and I implemented it using a conflict-free replicated data type (CRDT) based on LOGOOT, which I modified to work with variable-sized strings rather than individual characters or lines.
In a previous post, we created an Actix 0.7 Web App, which was not fully non-blocking. In this post, we'll become fully non-blocking and upgrade the app to Actix 1.0
The vision is to build a livecoding / design hybrid program, where procedural design and code are fused in one environment. If you have missed 'learnable programming' please check this out: http://worrydream.com/LearnableProgramming/ Makepad aims to fulfill (some) of these ideas using a completely from-scratch renderstack built on the GPU and Rust. It will be like an IDE meets a vector designtool, and had offspring. Direct manipulation of the vectors modifies the code, the code modifies the vectors.
However before we can make this awesome application, we need to build a UI stack. The aim of this toolkit is to be our stepping stone into building a livecoding IDE and designtools that don't suck or fall to pieces along the way.
We are going to create a web-server in rust that only deals with user registration and authentication. I will be explaining the steps in each file as we go. The complete project code is here repo. Please take all this with a pinch of salt as I’m a still a noob to rust 😉.
Actix web 1.0.0 is released - a small, pragmatic, and extremely fast web framework.
I find Gopher really cool. I think it’s a really nice way to organize information into trees and hierarchies, and as we all know programmers can’t resist trees. So recently I took an interest in Gopher and started writing my own server.
But recently it’s been gaining traction; so we should provide a decent landscape for new gophers, full of oxidised servers. Since I started using Gopher more often, it’s beneficial for me if there’s more content out there. So I’m writing this blog post to walk you through how to write your own server. We’ll be doing this in Rust.
In this series, we are creating a basic static HTTP 1.0 server with Rust. If you haven’t seen Part 1 yet, go do that first. At the end of Part 2, our server will do the following: Read and serve files from a predefined directory on the host server, Generate appropriate HTTP responses to incoming requests, Log information about the response to standard output.
It all started with a simple issue, a “quick fix”, or so I thought.
General purpose computation on the blockchain using Web Assembly System Interface (WASI).
This is the story of our journey running Wasmer on the Linux kernel.
In this series, we will create a basic static HTTP 1.0 server with Rust. At the end of Part 1 of this tutorial, our server will do the following: Listen for and handle TCP connections on a specific port, Accept HTTP 1.0 GET requests, Parse and validate requests for further use, Log incoming requests. We will avoid using libraries that make this trivial (i.e. the http crate) and focus on the fundamentals of how a server works.
Stretch is a cross-platform Flexbox engine written in Rust. At Visly we are building a design tool for front-end engineers and we needed to ensure components looked the same across web, iOS, and Android without making use of WebViews. This meant replicating the web layout system on mobile.
In this post, I’ll cover the test setup we use in Stretch, how and why we need to generate unit dynamically. I’ll also cover an example of contributing another test to Stretch, and finally I’ll walk through how we also make use of this system for benchmarking.
This article is more of a how-to on getting up and running with a production-grade web project which incorporates Rust (or any other language for that matter) and WebAssembly into your web pages.
Lightbeam is a new streaming compiler for WebAssembly, designed to produce the best possible assembly while still being fast enough to produce assembly faster than the WebAssembly is received over the wire.
When writing a web service, I often lean towards using tools that are as minimal as possible. One pretty obvious reason for this is the avoidance of dependencies you either don't want or don't need in your project. Whilst I'm not someone who goes out of their way to avoid dependencies, this is a pain point particularly in Rust because of the cost of building them repeatedly rather than shipping extra interpreted files around (especially if you use "pure" build environments).
If you are coming from NodeJS, Futures in Rust don't make much sense. In NodeJS, everything happens asynchronously. Therefore for you to be able to say "Hey, I really need to wait for the answer for this GET HTTP call", you are putting.then() on a Promise, so you can make sure you just execute the code inside the .then() when the HTTP call is finished.
In Rust, everything is blocking and synchronous by default, so you might ask yourself: "Why bothering with the complexity, that's exactly what I wanted in the first place!"
Rust is one of the newest kid on the block in the modern programming languages. We tried to take advantage of its great features by using it in an Image Recognition blueprint project. This project can be traded with any existing security system deployed at any organization. Hawk uses AWS services integrated with Rust.
In the last three posts of this series we covered all of the things we would need to use Wasmer as the base for a plugin system. In part one we went over the basics of passing simple data in and out of a web assembly module, in part two we dug deeper into how you might do the same with more complicated data. In the last part we eased the experience of plugin developers by encapsulating all of our work into a library that exports a procedural macro. In this post we are going to explore what it would take to extend an existing plugin system to allow for wasm plugins.
In this article, we’ll implement and deploy a Gotham full-stack web framework using the Tera template system, Webpack for a complete front-end asset management, a minimal VueJS and CoffeeScript web app and deploy to Heroku. Gotham is a Rust framework which is focused on safety, speed, concurrency and async everything. Webpack is a NodeJS website asset preprocessor and bundler which can let you use any of your favorite front end technologies. Combining these technologies allow for a small footprint on the server, which means saving money on infrastructure, very fast performance in page load for higher visitor retention and the full flexibility of client side code available to you for web design without limitations.
Build a WebAssembly application with a hash router using Smithy.
Using sonr-extras to build a very basic chat using the provided connection handling objects.
Plume is a federated blogging engine, based on ActivityPub. It uses the Rocket framework, and Diesel to interact with the database.
In the last two posts of this series we covered all of the things we would need to use Wasmer as the base for a plugin system. In part one we went over the basics of passing simple data in and out of a web assembly module, in part two we dug deeper into how you might do the same with more complicated data. In this part we are going to explore how we might ease the experience for people developing plugins for our application.
In this post we are going to cover how we could pass more complicated data from the wasm module back to the runner.
A few months ago, the Wasmer team announced a Web Assembly (aka wasm) interpreter that could be embedded into rust programs. This is particularly exciting for anyone looking to add plugins to their project and since Rust provides a way to directly compile programs to wasm, it seems like a perfect option. In this series of blog posts we are going to investigate what building a plugin system using wasmer and rust would take.
The most common pattern for creating APIs is REST. We will discover how we can build an API in Rust which conforms with the REST pattern.
Last article I wrote about how to use tensorflow with rust. This time we're going to take what we've built on, and serve it as an HTTP API call. As Actix Web is nearing its inevitable 1.0 release, I thought it would be a great time to build something with it.
In this tutorial, we'll create web app using the Rust actix-web framework and implement magic link authentication powered by ApproveAPI's Rust library.
As a first little project in Rust I thought I'd do something familiar, so I created a small web application, which we'll check out in this post.
Linkerd 2.0 introduced a substantial rewrite of the widely adopted service mesh, using a split between Go and Rust. In this article, we discuss the lessons learned in the "cauldron of production adoption", and how those lessons became the basis of Linkerd 2.x’s philosophy, design, and implementation.
Ocypod is a language-agnostic, Redis-backed job queue server with an easy to use HTTP interface. Its focus is on handling and monitoring long running jobs.
Lucet is Fastly’s native WebAssembly compiler and runtime. Using the Lucet runtime and Rust’s wasm32-unknown-wasi target, we can create a WASM program that runs on the server.
I’m extremely excited to announce the 0.0.2 release of Smithy, a web development framework for Rust! While it is a very pre-alpha version, it should be functional enough for others to start playing around with. Please, get your feet wet and provide feedback.
Today, we're open sourcing and announcing wrangler, a CLI tool for building, previewing, and publishing Rust and WebAssembly Cloudflare Workers. If that sounds like some word salad to you, that's a reasonable reaction.
Today, we are thrilled to announce the open sourcing of Lucet, Fastly’s native WebAssembly compiler and runtime. WebAssembly is a technology created to enable web browsers to safely execute programs at near-native speeds. It has been shipping in the four major browsers since early 2017.
I'm distinguishing Seed through clear examples and documentation, and using wasm-bindgen/web-sys internally. I started this project after being unable to get existing frameworks working due to lack of documented examples, and inconsistency between documentation and published versions. My intent is for anyone who's proficient in a frontend framework to get a standalone app working in the browser within a few minutes, using just the quickstart guide.
Seed's different approach to view syntax also distinguishes it: rather than use an HTML-like markup similar to JSX, it uses Rust builtin types, thinly-wrapped by macros that allow flexible composition. This decision will not appeal to everyone, but I think it integrates more naturally with the language.
tarssh is an SSH tarpit — a server that trickles an endlessly repeating introductory banner to clients for as long as it remains connected, in order to expend the resources of attackers. It's based on the same concept as Chris Wellons' Endlessh, a similar service written in C.
If you learn something new, always have an updated version of your project in production. It keeps you motivated. We cover the most common options of how to do this in and with Rust.
Dodrio is a virtual DOM library written in Rust and WebAssembly. It takes advantage of both Wasm’s linear memory and Rust’s low-level control by designing virtual DOM rendering around bump allocation. Preliminary benchmark results suggest it has best-in-class performance.
ZEIT is proud to announce official support for Rust on Now through `@now/rust`. Our mission at ZEIT is to make cloud computing accessible for all. Rust has seen rapid growth in its adoption, and we're proud to be able to support the community.
WASP is ‘a LISP programming language for extremely performant and concise web assembly modules.’ That means that you can use WASP to generate WASM modules.
Using Rust and AWS Lambda to thumbnail images, cheaply, relablity, and quickly.
In this article about the state of the Rust and Wasm ecosystem, I'll try to explain why Rust is the language that can unlock the true potential of WebAssembly.
In my previous articles I talked about how WebAssembly allows you to bring the library ecosystem of C/C++ to the web. One app that makes extensive use of C/C++ libraries is squoosh, our web app that allows you compress images with a variety of codecs that have been compiled from C++ to WebAssembly.
In my experience, most performance problems on the web are caused by forced layout and excessive paint but every now and then an app needs to do a computationally expensive task that takes a lot of time. WebAssembly can help here.
Almost by accident, it turned out that I implemented a pretty simple, but non trivial task in both C and Rust and blogged about them. Now that I’m done with both of them, I thought it would be interesting to talk about the differences in the experiences. The Rust version clocks at exactly 400 lines of code and uses 12 external crates. The C version has 911 lines of C code and another 140 lines in headers and depends on libuv and openssl.
At HAL24K, we benefit a lot from open source software. That is why, to contribute back, we’ve started an internal program to open source some of the internal tools and libraries we’ve used to build our platform and machine learning solutions, starting with OOProxy. OOProxy is a reverse OpenID and OAuth2 proxy that we use to protect our HTTP-based machine learning APIs.
Wasmer is a WebAssembly runtime designed to run both standalone and embedded. The crate wasmer-runtime exposes an easy to use and safe api for compiling, creating imports, and calling WebAssembly from your own library. This tutorial goes over how to make a simple wasm application and run it using the wasmer-runtime!
During last year’s Birthday Week we announced early support for QUIC, the next generation encrypted-by-default network transport protocol designed to secure and accelerate web traffic on the Internet.
We are not quite ready to make this feature available to every Cloudflare customer yet, but while you wait we thought you might enjoy a slice of quiche, our own open-source implementation of the QUIC protocol written in Rust.
In this tutorial we’ll discuss the ideas and concepts behind rendering water and then talk through some demo code.
Rust 2018 has shipped, and we’re closing in on the end of the year. While we didn’t manage to ship async/await as part of the edition itself, the community has made quite a lot of progress toward that goal. This post summarizes the state of play, and announces the publication of several crates intended to facilitate use of async/await on the nightly ecosystem.
Take a developer deep dive into Terrarium, our multi-language, browser-based editor and deployment platform at the edge. Learn how to compile Rust programs to WebAssembly right on your local machine, interact with the Terrarium system, and explore some applications we’ve built with it.
WebAssembly (abbreviated Wasm) is a binary instruction format for a stack-based virtual machine. This essentially means that is is fast, because the program is compiled to a much more compact format, making it faster to parse. Wasm can be written by hand if you're looking for a challenge, but is primarily meant to be written in another language, and then compiled to Wasm. You may know a little about Assembly language and how it works - here's a quick refresher in-case you're rusty.
I am elated to announce that the next major release of Rocket is now available! Rocket 0.4 is a step forward in every direction: it is packed with features and improvements that increase developer productivity, improve application security and robustness, provide new opportunities for extensibility, and deliver a renewed degree of toolchain stability.
This blog post is about a project called Romio that I’ve been working on over the past two or three weeks. Romio is a port of a small part of the Tokio project to the newer futures APIs.
I started the project to get some experience porting code from the old futures API to the new API. However, we realized that this code could also be useful to other people who want to experiment with networking code using the new async/await syntax, so with the help of others we polished it up during the RustFest Rome “impl days” and now its being released for people to experiment with.
Enjoy this quick introduction to creating a web server with Actix and monitoring it with Sentry.
This past weekend I made a game for Ludum Dare 43. Tools used: Aseprite, quicksilver. Inspired by Zachtronics. It is written in Rust and compiled to WebAssembly.
Rust support on AWS Lambda was recently released, which seems like as good an opportunity as any to share some code and the solutions to challenges I encountered along the way ☺. I’ve decided to create a little diceware service, and the lambda-runtime crate provides a great API to make this a breeze.
Taking advantage of AWS Lambda Runtime support for Rust
AWS Lambda, which makes it easy for developers to run code for virtually any type of application or backend service with zero administration, has just announced the Runtime APIs. The Runtime APIs define an HTTP-based specification of the Lambda programming model which can be implemented in any programming language. To accompany the API launch, we have open sourced a runtime for the Rust language.
Throw down your main! Rustlang Serverless HTTP applications won’t need them where they’re going
We recently sent out a survey regarding the state of the current Rust web ecosystem and we got over a 1000 responses! We really appreciate the feedback from the community. This will help us continue to improve upon the state of the Rust web ecosystem. Today, we would like to go over the responses and understand the results.
Since the last post on Tide, there have been a number of excellent contributions from a bunch of new contributors! In this post, I want to talk about the work that @tirr-c has done to substantially improve the middleware story.
Adjusting an existing Rust project to build a native binary and library for WASM, load in JS. Details on working through build issues.
In this article, we describe how we ported the Exonum framework to actix-web using generic programming.
I really like Elm. It is a delightful language with an amazing ecosystem. It has an interesting architecture called TEA, The Elm Architecture. Another language I like is Rust. On paper, Rust is completely different from Elm, but in using them both, I have seen some resemblance. Having used both Elm and Rust I had something I wanted to try. Would it be possible to create The Elm Architecture in Rust?
Abusing Go runtime in AWS to run Rust binaries
Say you want to store a huge number of very small filesthat you will only access over HTTP.For example:You are using rustdoc to render the documentation of a library.Without much work you’ll end up with about 100k HTML filesthat are about 10kB each.As it turns out,this number of small files is very annoying for any kind of file system performance.Best case: making copies/backups is slow.Worst case: You’re using an anti virus software and it takes ages.
After the positive response to the routing and extraction proposal in Tide, I’m pleased to say that an initial implementation is available on GitHub! As a testament to the strong foundation that Rust’s ecosystem provides, the basic framework implementation took only about 1,000 lines of code.
This is going to be the first post ( and hopefully not last ) in a series of posts about writing (and thinking about) serverless applications in Rust. Stay tuned…
First of all, hello there! This is the first announcement after the call for maintainers back in summer, and includes the efforts of several new faces: @whitfin, @nyarly, @colinbankier. Together, as well as input from the original authors @bradleybeddoes and @smangelsdorf, we are the new maintainers of the Gotham project. Today we’re excited to announce the release of Gotham 0.3, which follows the evolution of the broader Rust ecosystem.
Picking up from part one, we now have as server that takes an email address from a request and spits out a JSON response with an invitation object. In part one I said that we will send an email to the user, after some thought and feedback, we will be skipping this part now (look out for part 3). For now we will use the http response from the server to verify the email so to speak.
This is part four in a series of Rust on Azure Functions. The other parts are about performance comparisons (part 1 and part 2), and explaining the PMX algorithm (part 3). If you want to learn more about Genetic Algorithms be sure to read part 3 first).
When WebAssembly was first shipped it was an MVP which, while minimal, has spawned a huge number of exciting projects which work today across all major browsers. Rust has capitalized on the wasm MVP’s success as well with tools like wasm-bindgen and wasm-pack by making the MVP feel less minimal. WebAssembly is yet more ambitious, though! Since inception it’s always been intended to extend the WebAssembly specification with new features and functionality.
In this post, I look at how WebAssembly can be used to create serverless functions and demonstrate an AWS Lambda function written entirely in Rust.
This post continues the series on Tide, sketching a possible design for routing and extraction that combines some of the best ideas from frameworks like Rocket, Actix, and Gotham.
The API is written in Rust, a language new to the Pi-hole project. Rust is a safe and fast language which matches well with our goals for the API. It is statically typed and prevents whole categories of errors while being productive and extendable.
We are going to create a web-server in rust that only deals with user registration and authentication. I will be explaining the steps in each file as we go. The complete project code is here repo. Please take all this with a pinch of salt as I’m a still a noob to Rust.
WebAssembly is a binary instruction format for a stack-based virtual machine. Wasm is designed as a portable target for compilation of high-level languages like C/C++/Rust, enabling deployment on the web for client and server applications.
Rust has its goals set on to be a primary WASM language and it would be awesome to use it both in backend and frontend web. Ruukh is one of such efforts to realise that dream. Ruukh, a frontend web framework, is inspired by both VueJS and ReactJS.
This blog post is an update to the preceeding article “A web application completely written in Rust” and summarizes the projects’ progress over the last months.
Today, Tower Web 0.3 has been released and it comes with two major new features: Experimental support for async/await. Support for using templates to render responses.
I am proud to announce that I’m working on a book for the Pragmatic Programmers. The title hasn’t been set in stone yet, but the book will be about programming WebAssembly with Rust.
We are pleased to announce the first release of the web-sys crate! It provides raw bindings to all the Web’s APIs: everything from DOM manipulation to WebGL to Web Audio to timers to fetch and more!
Today we are releasing RedBPF and ingraind, our eBPF toolkit that integrates with StatsD and S3, to gather feedback, and see where others in the Rust community might take this framework. If you are looking to up your company’s monitoring game, gather more data about your Raspberry Pi cluster at home, or just have a strong academic interest in Rust and low-level bit shepherding, you might want to read on.
Smithy, a web development framework written in Rust that compiles to WebAssembly. In the mean time, there has been substantial progress, and Smithy is on the verge of being ready for alpha use! In this post, I want to describe the improvements that have been made, and what’s on the Smithy roadmap!
The 2.0 release of Linkerd brings two very significant changes. First, we’ve completely rewritten Linkerd to be orders of magnitude faster and smaller than Linkerd 1.x. Linkerd 2.0’s data plane is comprised of ultralight Rust proxies which consume around 10mb of RSS and have a p99 latency of <1ms. Linkerd’s minimalist control plane (written in Go) is similarly designed for speed and low resource footprint.
The Network Services Working Group aims to improve the story for web development this year in several respects: by bolstering foundations like async/await, by improving the ecosystem of web-related crates, and by pulling these pieces together into a framework and book called Tide. The name “Tide” refers to “a rising tide lifts all boats”; the intent is to improve sharing, compatibility, and improvements across all web development and frameworks in Rust.
tower-web version 0.2.2 has been released. It comes with a number of new features, which I will talk about in this post. Primarily, the middleware story is starting to come together. I will be expanding some on how middleware fits into Tower and web in general.
Let’s take a look at how in Rust you can have an algorithm generic over T, where T is further bound by a trait, which itself is generic over one of several parameters.
I set out out my goal 9 for Rustnish: Write benchmark code that compares runtime performance of Rustnish against Varnish. Use cargo bench to execute the benchmarks.
The basic idea of a performance test here is to send many HTTP requests to the web service (the reverse proxy in this case) and measure how fast the responses arrive back. Comparing the results from Rustnish and Varnish should give us an idea if our performance expectations are holding up.
Markdown rendering is very important to the performance of Semaphor - every message you send and read is a Markdown document - so we're always looking for ways to improve the performance of rendering Markdown. A couple months ago Jonathan Moore and I wondered how easy it would be to integrate WebAssembly into a React component, replacing the render() function, and we thought that moving Markdown parsing into Rust would be a great way to test this idea out.
In Servo, task-sources are implemented via a channel, whose sender is cloned for each specific task-source, and where tasks are messages sent on the channel and containing a closure representing the actual task.
In "Oxidizing sourmash: Python and FFI" I described my road to learn Rust, but something that I omitted was that around the same time the WebAssembly support in Rust started to look better and better and was a huge influence in my decision to learn Rust. Reimplementing the sourmash C++ extension in Rust and use the same codebase in the browser sounded very attractive, and now that it was working I started looking into how to use the WebAssembly target in Rust.
Today, let’s go through an entire fetch in Servo, starting with an example in JS.
Rust's web frameworks ecosystem is in constant change, but recently a new framework called warp came out implementing a new, original way to solve the old problem of transforming a request into a response, and I wanted to give it a try.
And, as I use GraphQL massively at work, I also wanted to check how well Juniper implements it. To add some spice, I used MongoDB as a storage engine instead of the ubiquitous and well-supported SQL databases.
The short version is, Tower Web 0.2 was just released and regular Rust attributes are now used instead of magic comments. The doc comment is replaced with #[get("/")]. This is thanks to Rust macro wizard David Tolnay. I also thought that it would be best to immediately push out 0.2 and then we can all pretend 0.1 didn’t happen.
This is a demonstration of a pure-rust library to generate syntax diagrams for macro_rules!().Diagrams are generated fully automatically from rust-source as Scalable Vector Graphics, using customizable CSS for layout.
I previously announced Tower and mentioned that a web framework was in the works. It took longer than I had hoped (as it sometimes does with software), but today, I am opening up Tower Web.
Tower Web is an asynchronous HTTP web framework that focuses on removing boilerplate. It is built on top of Tokio, Hyper, and of course Tower. It works today on stable Rust.
Over the past several months, I’ve been working a web framework in Rust. I wanted to make use of the new hyper 0.12 changes, so the framework is just as fast, is asynchronous, and benefits from all the improvements found powering Linkerd. More importantly, I wanted there to be a reason for making a new framework; it couldn’t just be yet another framework with the only difference being I’ve written it. Instead, the way this framework is used is quite different than many that exist. In doing so, it expresses a strong opinion, which might not match your previous experiences, but I believe it manages to do something really special.
I’m super excited to reveal warp, a joint project with @carllerche.
Over the past few months, I’ve been writing Smithy, a very work-in-progress front-end WebAssembly framework written in Rust.
My goal for Smithy is to enable you to use idiomatic Rust to write front-end code. This has costs: for example worrying about lifetimes and using Rc<RefCell<State>> to share state. But this also has the potential to give you the safety guarantees that the Rust compiler provides when writing browser code!
For this post, I’m going to make three comparisons: to Flash, to Java Applets, and occasionally to PNaCL. Secondly, this post is going to focus on the web use-case for WebAssembly, even though the previous post was about non-web uses. We’ll make that comparison in the next post. Finally, this post is kind of like eating tapas, there’s a bunch of little sections.
A three day tour of Yew and WASM with Rust
A sensible, modern pastebin written in Rust
We've just finished music festival Atlas Weekend which took place in Kyiv, Ukraine. This year there were for about 450,000 visitors during 6 days. I'm glad to announce that Rust and actix-web are used as a backend for main technical purposes of festival.
In the first article of this series, we created a RestAPI for our application using Postgres, Rust, and Rocket. For the second half, we’ll be using React to create the front end.
My latest software architectural experiment is to write a complete real-world web application in Rust with as less as boilerplate as possible. Within this post I want to share my findings with you to answer the question on how much web Rust actually is.
A tutorial on how to compile Rust to WebAssembly and integrate it into a React application
This is the first part in a two-part introductory series to PRRR Stack (Postgres, Rust, Rocket, React) application.
The last blog post laid out the Rust and WebAssembly domain working group’s overall vision for Rust and WebAssembly. In this blog post, we will dive into the details of wasm-bindgen, the future we envision for it, and how you can help us build that future.
In our previous post, we saw how combining channels with an event-loop could be a useful technique to ‘drive’ the concurrent logic of your system, and it was hinted at that shared mutable state might be more complicated. A good example of such ‘shared mutable state’ in Servo is the HTTP cache.
In our last post about actix we introduced you to the Rust programming language and the actix actor framework. This week we will build a basic TCP client with actix.
Better HTTP Upgrades with hyper It’s been possible to handle HTTP Upgrades (like Websockets) in hyper if you made use of the low-level APIs in the server and client, but it wasn’t especially nice to...
Rust and WebAssembly can combine in many delightful ways. To consolidate our efforts and have the largest, most-positive impact we can, the Rust and WebAssembly domain working group is focusing on one vision:
This blog post will expand on these aspirations and describe where we stand in relation to them right now. In a series of follow up posts, we will talk about the next steps for each major component of the Rust and WebAssembly ecosystem.
This game was coded in Rust and is playable in web browsers by means of WebAssembly, WebGL and Howler.js. The software I developed is partly open source in the form of Gate, which is the Rust library that powers this game and can power other similar games. Special thanks to the tools I used to create assets: Gimp, FL Studio and BFXR.
The idea was simple, choose a package and recursively traverse npm to find all of the packages that depend upon it, after all the npm website offers a ‘Dependents’ tab on a package’s page. I had not anticipated the issues that made this an interesting technical problem.
I had this goal of making a web app that was very fast, stable and easy to deploy. This is what I was hoping to accomplish: strongly typed server-side and client-side languages (Rust & Elm respectively), push-button deployments to the cloud with free hosting initially, and sub-second response times for API calls and page loads. With this setup, I think I'm well on my way to accomplishing each of these.
Lately I’ve been working on a side project that involves aggregating the indexes of media libraries and allowing one library to “lend” a file to another. For example, I might use this system if I want my home tower to “lend” a file to my laptop before I get on a plane. I’ll be committing my work-in-progress on this project to github shortly (just want to get an MVP working first and do some refactoring), but I thought I would take a step back for a moment and share my thoughts on one aspect of this project: gRPC. In particular: gRPC streaming.
The 3rd of a multi-part series where we build a small REST client for the Marvel Comics Web API using Rust. This time we focus on how to use the hyper and futures crates to make HTTP requests to a Web API, then use serde to parse JSON data from the response.
Welcome to the third post in a series where I share my experience learning Rust. I’m building out a conspiracy theories API to help me get more familiar with Rust and have a little fun. Since I am new to Rust, I welcome all feedback, especially from developers who have been using Rust for quite some time. Leave a comment below or send your feedback via Twitter. Now it’s time to draw the blackout curtains, put on your foil-lined hat on and start serving up conspiracies from a REST API.
The “actor model” is the main primitive that powers the Erlang programming language and its descendant, Elixir. It describes a programming model that simplifies the development of concurrent and multi-threaded applications or even applications that run distributed on multiple machines.
The complexity of actors is relatively low, and that is because the complexity is usually hidden in the actor frameworks that are used to run these types of primitives in the end. One example of such an actor framework is actix, which we will have a closer look at now.
I was inspired by Nick’s post to migrate my code from Rocket to Actix. I have also been nagged by recent struggles with the nightly compiler and its compatibility with all the other crates that I use and other the platforms that I use; my code runs on 32⁄64 bit arm and 32⁄64 bit x86. When I started using Rocket around a year ago, the nightly compiler was just as stable as the stable compiler, but things have changed as rust is a growing eco system.
For me, once the differences below were addressed, porting the code was relatively straightforward. The vast majority of my code remained the same.
While the http crate generally has a great API I have been unsatisfied how it handles URLs. To create a HTTP request a full URL is needed with a scheme (http/https), authority (example.org) and a path (/search?q=rust) but http does enforce this and allows you to only state the path. This means both clients and servers are either unable to determine protocol and and authority information or have to do this manually.
Today sees the release of hyper v0.12.0, a fast and correct HTTP library for the Rust language.
This release adds support for several new features, while taking the opportunity to fix some annoyances, and improve the extreme speeds!
Creating a simple web service for automating Travis builds across repositories using actix-web and Github webhooks.
I write code for the web every day, and as such, I naturally wanted to find a web server that I could write some beautiful code in Rust with. There are a few options out there, Actix, Rocket, but none of them are quite as simple and succinct as KoaJS for node. As such, I wanted to write something as simple as Koa and as performant as a Rust framework, and I did. With that, I give you Thruster, an elegant, performant, web server written in Rust.
A little while ago I came across this Snowflake generator. It’s a project by Raph Levein that takes a hash string and uses it to procedurally generate a unique snowflake. He explains that the original motivation was as a cryptographically secure visual hash, so that people would reliably be able to tell by visual inspection whether two hashes were identical. I thought that was a pretty cool idea.
I previously wrote an article back in November 2017: Replacing Elasticsearch with Rust and SQLite. In it, I needed to create a few HTTP endpoints that ingested JSON, perform a database lookup, and return JSON. Very simple. No query / path parameters, authentication, authorization, H2, or TLS. I didn’t understand how stability was such an important feature. I was familiar with needing new versions of the nightly compiler to stay current with clippy and rustfmt, but it was a blindspot when it came to dependencies.
Here I go with my first post that fully focuses on Rust. After spending a few months doing a bit here and there I decided to just dive right in as I was going through the Rust book at too slow a pace to keep myself interested. So, in this post I decided to write about setting up a simple REST API which is something that I have done in Java plenty of times but with Rust it is a different story.
The 2nd of a multi-part series where we build a small REST client for the Marvel Comics Web API using Rust. This time we focus on how to use the hyper and futures crates to make HTTP requests to a Web API, then use serde to parse JSON data from the response.
I had the need for a very small API for this website. So small, in fact, that only one endpoint was required. I've been doing a lot of development in Rust lately, so naturally Rust seemed like a good candidate to build this API in. I also wanted to try out a newer Rust web framework called Actix web. It claims to be a "small, pragmatic, and extremely fast rust web framework", which sounded perfect for what I needed.
A project used for a meetup talk about getting started with rust & wasm
This is a small idea that I have been wanting to put together for quite some time now and finally have managed to get the time and most importantly experience in Rust to finally try something a little more than just small projects. One area that I think Rust is really making a decent headway in in the web domain, which I am assuming is likely due to its origin from Firefox. So I wanted to see if I could put together a really basic CRUD micro-service doing the ever so original TODO functionality.
Using Rust on Microsoft Azure Functions with web assembly
Hopefully this post helps jumpstart those who want to use LLVM from Rust. I also include how one could use this to generate WebAssembly, but the first part of this post is target platform agnostic.
C, C++, and Rust all have some capability for dynamic dispatch: function pointers, virtual methods, and trait objects. On native targets like x86, all these forms compile down into a jump to a dynamic address. What do these forms compile down into when targeting WebAssembly?
The Rust language is one of the earliest adopters of WebAssembly, and it has more than one way to compile to it:
wasm-pack is a tool for assembling and packaging Rust crates that target WebAssembly. These packages can be published to the npm Registry and used alongside other packages. This means you can use them side-by-side with JS and other packages, and in many kind of applications, be it a Node.js server side app, a client-side application bundled by Webpack, or any other sort of application that uses npm dependencies. You can find wasm-pack on crates.io and GitHub.
I wrote the NES emulator with Rust and WebAssembly to learn Rust. It’s not perfect and have some audio bugs, but it’s good enough to play Super Mario bros.
Rust aims to be the foundation of fast and robust software. A thorn in that side is the choice of timeout-centric APIs for I/O and multithreading. I posit this was a bad choice and it needs to be remedied as soon as possible before more APIs proliferate this (possibly accidental) design decision.
Yew is a new web framework written in Rust that takes concepts from React JS and Elm. It takes the two web libraries I love and combines them with the new language I love — Rust. I wrote this article because I haven’t found a lot of documentation online on how to get the Yew examples running, and I’m sure there will be people that will run into the same problems I did. I wrote this in hopes that it helps someone out there.
Tower is a library for writing robust network services with Rust. It is being built in service of the Conduit proxy, which is using the Tokio ecosystem to build the world’s smallest, fastest, most secure network proxy. Tower will also provide a batteries included experience for implementing HTTP and gRPC services.
Serving content from a Rust web server running on a Raspberry Pi from your home to the world, with a Cloudflare Argo Tunnels.
For one of our customers at Centricular we were working on a quite interesting project. Their use-case was basically to receive an as-high-as-possible number of audio RTP streams over UDP, transcode them, and then send them out via UDP again. Due to how GStreamer usually works, they were running into some performance issues.
This blog post will describe the first set of improvements that were implemented for this use-case, together with a minimal benchmark and the results. My colleague Mathieu will follow up with one or two other blog posts with the other improvements and a more full-featured benchmark.
The short version is that CPU usage decreased by about 65-75%, i.e. allowing 3-4x more streams with the same CPU usage. Also parallelization works better and usage of different CPU cores is more controllable, allowing for better scalability. And a fixed, but configurable number of threads is used, which is independent of the number of streams.
Hello and welcome to the first issue of This Week in Rust and WebAssembly! Rust is a systems language pursuing the trifecta: safety, concurrency, and speed. WebAssembly is designed as a portable target for compilation of high-level languages like C, C++, and Rust, enabling deployment on the web for client and server applications. This is a weekly summary of its progress and community.
Please read the Part 1 of the series here :
The newest release of hyper includes some lower-level connection APIs for both the server and client. Notably, this allows using hyper send and receive HTTP upgrade requests. The most popular of these is Websockets.
I started writing mob, an multi-echo server using mio, in 2015. I coded mob into a mostly working state and then left it mostly alone, only updating it to work with the latest stable mio. Recently, I started looking at the code again and had the urge to improve it. In a previous post, I talked about managing the state of connections in mob. In this post, I will walk through what I did to remove the need to track connection state. I wanted to remove the state because the implementation required an O(n) operation every tick of the mio event loop. It also added a fair amount of complexity to the code.
Blazing fast, low requirements, computationally intensive operations on Node.js using Rust
Central to the problem are the edges. People will reliably do a good job of building and testing the happy paths, but as humans we’re terrible at considering the edge conditions, and it’s those edges and corners that cause trouble over the years that a program is in service.
Since my initial Node/Rust REST comparison, I’ve wanted to follow up with a comprehensive guide for getting simple CRUD operations up and running in Rust.
A brief post to help others multicast in Rust
With the combined goal of gaining a deep understanding of DNS, of doing something interesting with Rust, and of scratching some of my own itches, I originally set out to implement my own DNS server. This document is not a truthful chronicle of that journey, but rather an idealized version of it, without all the detours I ended up taking. We'll gradually implement a full DNS server, starting from first principles.
WebAssembly has been interesting me lately, specifically the prospect of doing arithmetically-intensive operations with it, not unlike the fantastic physics engine Emscripten port Ammo.js. Compiling something like that is out of the scope of this little post (but should be getting easier!), and I’ve left some links at the bottom so you can clear more about WebAssembly if you’re interested.
IPFS is a peer-to-peer protocol that allows you to access and publish content in a decentralized fashion. It uses hashes to refer to files. Short of someone posting hashes on a website, discoverability of content is pretty low. In this article, we’re going to write a very simple crawler for IPFS.
It’s challenging to have a traditional search engine in IPFS because content rarely links to each other. But there is another way than just blindly following links like a traditional crawler.
I’m writing a reverse proxy in Rust using Hyper and I want to measure performance a bit to know if I’m doing something terribly wrong. By doing that I discovered a Denial of Service vulnerability in Hyper when IO errors are not properly handled. Note that a workaround has been released in the meantime in Hyper 0.11.20, more background info can be found in this Hyper issue.
I’ve been writing a small toy project with Tokio in my spare time’s spare time. I’ll write more about it at a later date. What I’ve found writing it, though, is that there is a specific pattern to writing servers, both UDP and TCP (and presumably others) in Tokio, and it’s not super obvious at first glance. So here it is.
Vyacheslav Egorov, who goes by mraleph on the Web, wrote a response to my article “Oxidizing Source Maps with Rust and WebAssembly” titled “Maybe you don’t need Rust and WASM to speed up your JS”.
It’s been a few months since I shifted my focus full time to Rust, and in that time, we’ve gotten a lot of work done! I wanted to update you here what all that is, since it’s spread around on multiple repositories.
I recently spent some effort trying to make reproto run in a browser. Here I want to outline the problems I encountered and how I worked around them. I will also provide a number of suggestions for how things might be improved for future porters.
Fanta is a web framework that aims for developers to be productive and consistent across projects and teams. Its goals are to be: Opinionated, Fast, and Intuitive. Based heavily off of the work here: https://github.com/tokio-rs/tokio-minihttp
My nice brother Johannes Ridderstedt sent me some old files a few weeks ago (in late 2017), stuff that he had preserved from an age-old computer of ours. One of these was the file named gameland.zip (not published yet, but I might put it up here some day.) I managed to get this running, and liked what I saw (you'll find the YouTube link to it further down on this page.) Around this time I was reading a bit about WebAssembly which I think will redefine and help reshape the web as we see it today. I was also looking at the Hello, Rust web page, and the "FizzleFade effect using a Feistel network" page in particular.
Everyone loves event sourcing, right up until the moment they have to implement it. At that moment all the wonderful whiteboard drawings…
Introducing wee_alloc. wee_alloc is a work-in-progress memory allocator designed for WebAssembly. It has a tiny code size footprint, compiling down to only a kilobyte of .wasm code.
概要 以前、JSで書いた(ファミコンのエミュレータを書いた - undefined)ファミコンのエミュレータをRustで書き直してみた。 また、技術的な内容はQiitaの方にも書いているので興味のある方は参照してみてください。(まだ Hello, World!までしか書けてませんが。) qiita.com もともとファミコンのエミュレータって新しい言語を習得するのにちょうどいい題材だったりするのでは、って話しからスタートしてて、よくわからないのでJSで書いてみて、ようやくRustで一通りは実装できた感じ。まだバグや未実装（音声周りやマッパー）も多いんですが、ひとまずはお腹いっぱいな感じ。 成果…
DataFusion is an open-source Big Data platform implemented in the Rust programming language with a similar programming style to Apache Spark.
Let me begin this article on Writing a Microservice in Rust by talking about C++. I’ve been a reasonably active member of the C++ community for quite a while now, attending and contributing talks to conferences, following the development and evangelism of the language’s more modern features and of course writing lots of it. C++ is a language that gives its users very fine-grained control over all aspects of the program they are writing, at the cost of a steep learning curve and a large body of knowledge required to write effective C++. However, C++ is also a very old language, conceived by Bjarne Stroustrup in 1985, and thus caries a lot of technical debt even into modern standards.
In this post, I'll walk through a few of the highlights of getting Turtle, a Rust library for creating animated drawings, to run in the browser with WebAssembly.