Bastion-rs blog

Celebrate the 1st Bastioniversary with the 0.4 release! :tada: :confetti_ball: :rocket:

Tue, 21 Jul 2020 00:00:00 +0000

The Excitement of a 1st birthday!

Exactly 1 year passed since the initial release of Bastion. In this one year we’ve done so many amazing things together as Bastion team.

We are proud to announce our 0.4 release on our 1st birthday! :tada: :confetti_ball: :rocket:

What is Bastion?

Bastion is a distributed runtime which is inspired from the design principles of Erlang and adapts these into Rust ecosystem.

What is new?

This release is coming with exciting features:

Agnostik executor runtime
- With Agnostik, Bastion can run on any runtime. Head over to the agnostik project for more information.
Nuclei, an agnostic proactive IO system
- With nuclei and agnostik, IO and executors are separated.
- You can mix and match any executor with agnostik and use the same IO system.
- Nuclei is based on Proactive IO.
- Nuclei supports io_uring and works well with completion based evented IO. Windows support is currently in the works.
- Completely async, can be independently used from Bastion.
  - It will power Bastion’s IO system and ecosystem.

Autoscaling feature for actors

Right now with the scaling feature enabled, you can create actor groups in Bastion that will adapt to incoming workload according to given resizer parameters.

Example construction is like:

  children
  // Start with 3 actors
  .with_redundancy(3)
  // Do heartbeat every 5 seconds
  .with_heartbeat_tick(Duration::from_secs(5))
  .with_resizer(
      OptimalSizeExploringResizer::default()
          // A minimal acceptable size of group
          .with_lower_bound(1)
          // Max 10 actors in runtime
          .with_upper_bound(UpperBound::Limit(10))
          // Scale up when a half of actors have more than 3 messages
          .with_upscale_strategy(UpscaleStrategy::MailboxSizeThreshold(3))
          // Increase the size of group on 10%, if necessary to scale up
          .with_upscale_rate(0.1)
          // Decrease the size of group on 20%, if there are too many idle actors
          .with_downscale_rate(0.2)
  )

Cluster/distributed actors
- After some extensive testing of the Artillery protocol, we are super thrilled to show SWIM based epidemic cluster for Bastion. (SWIM stands for Scalable Weakly-consistent Infection-style Membership, we know it’s a mouthful, read this blog post if you would like an introduction). Here is an example code for cluster execution.
- Here’s a live example of a ping pong cluster.
Named child in children groups
- Right now you can get the names of the child in children groups. This is very handy when logging for example.
- With this enabled everything is user readable content addressable in Bastion runtime.
Heartbeat for children
- The bastion runtime continuously watches children with heartbeat to check their status and do sampling over their internals.
- This feature doesn’t require any feature flags.
- It is shown in the scaling example seen above:
```
  children
  // Do heartbeat each 5 seconds
  .with_heartbeat_tick(Duration::from_secs(5))
```
Dispatchers allow you to route specific messages to specific groups and collect messages from them.
- This enables creating various fan-outs and fan-ins, and streaming data processing implementations.
- Here’s how to use a round robin dispatcher.
Dead letters mailbox so undelivered messages can be processed or monitored later on.
Lockfree runtime statistics sampling for run queue offloading.
Actor registry support

Restart strategy for actors:

We have written a restart strategy for actors, so they can be restarted with various backoff strategies like linear, exponential etc.

Moreover, if panic or error occurs in your actors, you can define a max restart policy to give up after n attempts, or maybe you’re never going to give up ? Maybe you don’t want to restart at all:

  // At the beginning we're creating a new instance of RestartStrategy
  // we then provide a policy and a back-off strategy.
  let restart_strategy = RestartStrategy::default()
      // Set the limits for supervisor, so that it could stop
      // after 3 attempts. If the actor can't be started, the supervisor
      // will remove the failed actor from tracking.
      .with_restart_policy(RestartPolicy::Tries(3))
      // Set the desired restart strategy. By default supervisor will
      // try to restore the failed actor as soon as possible. However,
      // in our case we want to restart with a small delay between the
      // tries. Let's say that we want a regular time interval between the
      // attempts which is equal to 1 second.
      .with_actor_restart_strategy(ActorRestartStrategy::LinearBackOff {
          timeout: Duration::from_secs(1),
      });

Adapted NUMA allocator to latest Alloc API to make use of it easily with Bastion.

What’s on our roadmap?

Bastion
- Management subtrees of actors in runtime
- Local and shared state API for actors
- Link and Monitor implementations
- Refactoring the msg! macro
Alcazar
- Implement an early prototype of web framework based on top of the existing Bastion codebase. The goal is to expose a simple web api to send / receive messages to a Bastion.

Wanna join us?

We are working a lot towards letting Bastion become a prominent runtime, and help rustaceans build scalable and reliable distributed applications.

We are looking for maintainers who will:

Simplify the code and guide the development
Develop distributed operations
Triage and help Prioritizing bugs/feature requests
Develop new features or help with optimizations and refactoring
Make documentation and infrastructure shine
Join the fun and hang out with us!

If you are interested in these topics above and want to deep dive into Rust. This is a good start. This project and other crates that will be developed around with this project will reside under Bastion RS. Looking forward to hearing from future contributors!

We need your support!

We would love to invest into better testing on different platforms and architectures, verification, and constant improvement in our infrastructure and nourish the Rust ecosystem with a new approach for applications.

All contributors are dedicating their free time and trying to get the most of it at a continuous pace. But good mood and a cheerful team only got us so far. We unfortunately don’t have enough funding at the Bastion project and all projects under it at the moment.

Since the beginning of the project we have collected 35$ total. This is why kindly ask you to consider supporting us on https://opencollective.com/bastion.

We are also considering other approaches like providing consulting and integration advice, and advanced support features that will allow us to get feedback and spend more time making Bastion the best framework for distributed systems out there.

If you are currently using bastion in your company, or if you’re considering using it in production, please reach out to us!

If you (use Bastion and project ecosystem/considering using Bastion/want to help to project to get going), please consider donating over our OpenCollective. All these features mentioned above prepped in a long time, and we want to do more together with you!

If we start getting enough funding, we will begin writing posts about where the money goes and what we built with it!

If you have any questions or would like to have a chat with us, don’t hesitate to reach out to the Bastion team on Github, Twitter, and Discord!

Bastion floating on Tide - Part 2

Sun, 14 Jun 2020 00:00:00 +0000

Part 2: Building a Bastion

Introduction

This post is Part two of a series on how to build a highly fault tolerant system using bastion-rs, and expose an async web api with Tide.

Part one is about setting up a simple Tide server, and covers initial steps such as adding dependencies, some basic routing and the request/reply model.
In Part two we will first do a bit of refactoring, and then transform our function into a bastion child.

The complete example file is available in this github gist and you can reach out to me anytime if you have a question.

Before we dive into the bastion

I can’t remember where I read this sentence. (If anyone knows please let me know so I can give due credit) But it goes something like this:

Software architecture is like sewing, you want to intertwine threads two by two at most, otherwise you ll end up with a ball of knots.

This is a principle I tend to follow as much as possible, which allows me to read the code I wrote a long time ago. So before we dive into setting up a bastion, let’s refactor the code we wrote before, so we can only expose one function, that will serve prime numbers.

mod prime_number {
        pub fn get(num_digits: usize) -> u128 {
            get_prime(num_digits)
        }
        // the previous code
}

I put it in a separate module here, but It would probably be in an other file if it wasn’t intended to live in a github gist. As you can see, we just wrapped all of the logic from our previous code in a module, and exposed only one prime_number::get(). We don’t expose the other functions anymore (number_or_panic, is_prime etc.), except our Tide handler.

The tide request handler will now call prime_number::get():

async fn prime_number(req: tide::Request<()>) -> String {
    use std::time::Instant;
    let d: usize = req.param("digits").unwrap_or(1);
    // Start a stopwatch
    let start = Instant::now();
    // Get a prime number
    let prime_number = prime_number::get(d);
    // Stop the stopwatch
    let elapsed = Instant::now().duration_since(start).as_secs();
    format!(
        "{} is a prime number with {} digits.\nIt was computed in {} seconds.\n",
        prime_number, d, elapsed
    )
}

I think we could do better with the name of this function, but let’s keep it that way for now.

We now have a single entry point to receive prime_numbers, and our example is still working fine. Time to build a bastion!

Let’s wrap our business logic in a bastion child

Now there’s a lot of features we can use in bastion, so we’ll keep it as simple as possible, and only use what we need. There’s now a couple of examples in the bastion examples folder as well as a couple of cool showcases if you would like do dig deeper.

If you have any question or just wanna hang out with us, please join our discord and let us know what you’re working on! :D

Step one, our response structure

We are now going to wrap our prime number generator in a bastion child, that will receive requests for Prime numbers, and reply with the generated prime number, and a couple of informations as well.

We will first define a Response struct that will allow prime_number::get() to give us the prime number we asked for, as well as how long it took it to find it. An impl Display for Response will allow us to simplify the tide request handler even further:

mod prime_number {
    use std::time::{Duration, Instant};

    #[derive(Debug)]
    pub struct Response {
        prime_number: u128,
        num_digits: usize,
        compute_time: Duration,
    }

    impl std::fmt::Display for Response {
        fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
            write!(
                f,
                "{} is a prime number that has {} digits.\nIt was found in {}s and {}ms",
                self.prime_number,
                self.num_digits,
                self.compute_time.as_secs(),
                self.compute_time.as_millis() % 1000
            )
        }
    }

    // our prime number getter now returns a Response
    pub fn get(num_digits: usize) -> Response {
        let start = Instant::now();
        // Get a prime number
        let prime_number = get_prime(num_digits);
        // Stop the stopwatch
        let elapsed = Instant::now().duration_since(start);

        Response {
            prime_number,
            num_digits,
            compute_time: elapsed,
        }
    }
}

We #[derive(Debug)] here because currently, in Bastion, structures that serve to communicate need to implement the Debug trait. This may however change in the future.

Our request handler now looks like this:

async fn prime_number(req: tide::Request<()>) -> String {
    let d: usize = req.param("digits").unwrap_or(1);
    let prime_number_response = prime_number::get(d);
    format!(
        "{}\n",
        prime_number_response
    )
}

Step two, create a message processor

A bastion child is an autonomous being, that will wait for messages and handle them if it can. We will now build an async function that loops for ever, and processes ouf prime_numbers requests.

// `serve_prime_numbers` is the bastion child's behavior.
async fn serve_prime_numbers(ctx: BastionContext) -> Result<(), ()> {
    // let's put the context in an arc, so we can pass it to other threads
    let arc_ctx = std::sync::Arc::new(ctx);
    // a child will keep processing messages until it crashes
    // (or until it gets told to shutdown)
    loop {
        // msg! is our message receiver helper.
        // we will only use one variant here
        // =!> means messages that can be replied to
        // usize means it will only match against messages that are a usize
        msg! { arc_ctx.clone().recv().await?,
            nb_digits: usize =!> {
                // clone the context to send it to a thread
                let ctx2 = arc_ctx.clone();
                // answer! takes a context and will automagically figure out whom to reply to
                blocking!(answer!(ctx2, prime_number::prime_number(nb_digits)).expect("couldn't reply :("));
            };
            // this is a catch all for any other message we might receive
            unknown:_ => {
                println!("uh oh, I received a message I didn't understand\n {:?}", unknown);
            };
        }
    }
}

Now there’s a lot of information here, so let’s walk trough it together:

BastionContext

A child processor is an async function that takes a BastionContext as a parameter, and returns a Result<(),()>. In our case it will hopefully never return, because we are running it in a loop{}.

Refer to the BastionContext documentation if you would like to dig deeper.

msg!

The msg! macro is where most of the magic is happening. We are not using all of the features available in this match statement on steroids, but the fibonacci example dives deeper on the topic.

Refer to the msg! documentation if you would like to dig deeper.

It used to be one of my biggest struggles as I started using Bastion, so we are trying to find ways to express this behavior without macros. If you have ideas you would like to share, please let me know in the corresponding issue.

answer!

The answer! macro does a bit of magic. It uses the BastionContext to deduce who to reply to, and sends the reply.

Refer to the answer! documentation if you would like to dig deeper.

blocking!

Let’s talk about my favorite macros here, the “runners”. There are two preferred ways to run an async function on bastion:

blocking! puts a task in our blocking threadpool, which focuses on CPU bound tasks.

spawn! does the same, but tasks aren’t put in a separate threadpool, so you want to use this for I/O bound tasks.

run! allows you to wait until a task is complete.

If you’re not familiar with CPU bound and I/O bound, this blog post is an excellent resource to get started and to figure out what kind of tasks are CPU or I/O bound. Bonus points for one of my favourite illustrations on multi-threaded code.

Now that we have our runner, let’s spawn a child!

Step three, create a child

Let’s now head back to our main function, and build our bastion:

fn main() {

    // We need a bastion in order to run everything
    Bastion::init();
    Bastion::start();

    // Spawn 1 child that will serve prime numbers
    let prime_number_children =
        Bastion::children(|children| children.with_redundancy(1).with_exec(serve_prime_numbers))
            .expect("couldn't create children");

    let nb_digits = 6;

    // Ask our child for a prime number
    let reply = prime_number_children.elems().first().expect("no child?")
        .ask_anonymously(nb_digits)
        .expect("couldn't ask for a prime number");
    
    Bastion::stop();
    Bastion::block_until_stopped();
}

A bastion needs to be initialized and started before it can do anything. stop() and block_until_stopped() allow the bastion to gracefully shut down.

The bastion struct documentation is very exhaustive and will allow you to dig deeper.

Bastion::children allows us to set up redundancy (1 for now), and the message processing function (serve_prime_numbers in our case)

We can then pick our first (and only) child and ask_anonymously for a prime number.

Time for our last explanation before we can tie bastion and tide (this one sounds funny doesn’t it?).

Ask/Tell anonymously or not ?

Messages can either expect a reply or not.

ask() and ask_anonymously() will allow a child to use the answer! macro we saw earlier.

tell() and tell_anonymously() won’t.

The anonymously variants are used from “outside the bastion”. They add a mechanism that will allow to identify message senders even if they aren’t other children, which could come in handy when say… you want to answer! to a tide request handler :)

The ChildRef documentation shows examples on how to use each function, if you would like to dig deeper.

Let’s give it a run

At that point, we should be able to run it, so let’s try it out:

$ cargo run 
    Finished dev [unoptimized + debuginfo] target(s) in 0.11s
     Running `target/debug/floating_on_tide_2`
$ # nothing happened

Wait what? How comes nothing happened? Hmmm it turns out something happened, but we didn’t catch it! We just sent a request, and we didn’t await a reply. let’s do it now.

Writing our request/reply function

Our request/reply function will:

make a request
wait for a reply
return the reply

In order to do this, we need to have a handle to our bastion children, so we can ask_anonymously for prime numbers. Let’s write this structure:

// Our PrimeClient will keep a children_handle
// to be able to send messages to the bastion
struct PrimeClient {
    children_handle: ChildrenRef,
}

impl PrimeClient {
    pub fn new(children_handle: ChildrenRef) -> Self {
        Self { children_handle }
    }

    // request_prime will allow us to perform a request,
    // and reply when we receive an answer
    pub async fn request_prime(&self, nb_digits: usize) -> Result<prime_number::Response, ()> {
        // Ask our child for a prime number
        let reply = self
            .children_handle
            .elems()
            .first()
            .expect("no child?")
            .ask_anonymously(nb_digits)
            .expect("couldn't ask for a prime number");

        // wait for the reply
        msg! { reply.await?,
            response: prime_number::Response => {
                // This is a prime number Response, let's return it
                Ok(response)
            };
            unknown:_ => {
                // Something wrong happened
                println!("uh oh, I received a message I didn't understand\n {:?}", unknown);
                Err(())
            };
        }
    }
}

Ok we now have a structure that will allow us to perform a request, and hopefully get a reply, let’s edit our main function so it uses our new structure.

fn main() {
    // We need a bastion in order to run everything
    Bastion::init();
    Bastion::start();

    // Spawn 1 child that will serve prime numbers
    let prime_number_children =
        Bastion::children(|children| children.with_redundancy(1).with_exec(serve_prime_numbers))
            .expect("couldn't create children");

    // Create a prime client...
    let prime_client = PrimeClient::new(prime_number_children);
    let nb_digits = 6;

    use bastion::run;
    let response = run!(async { prime_client.request_prime(nb_digits).await })
        .expect("couldn't perform request");

    println!("{}", response);

    Bastion::stop();
    Bastion::block_until_stopped();
}

Let’s see how it goes…

$ cargo run
    Finished dev [unoptimized + debuginfo] target(s) in 1.59s
     Running `target/debug/floating_on_tide_2`
483991 is a prime number that has 6 digits.
It was found in 0s and 0ms

Yay \o/ it works, it’s now time to plug it to our tide server!

Tide and Bastion

As an aside, I really want to let you know how amazing tide’s documentation is. It was a pleasure to briefly skim the doc, and find exactly what it was looking for, with super nice examples, and no bloat around it. Kudos to the team, you’re doing an amazing job!

Tide has two ways to create an app, new() and with_state(). We are now going to change our new() to a with_state() call so we can pass our prime_client to the tide app:

// Create a prime client...
let prime_client = PrimeClient::new(children_handle);

task::block_on(async move {
    // ...That will populate our tide state
    let mut app = tide::with_state(prime_client);
    app.at("/prime/:digits").get(prime_number);
    app.listen("127.0.0.1:8080").await
})?;

In order to receive this state, we will update our prime_number async fn so it can use it:

// tide::Request is now parameterized over PrimeClient, which is our state
async fn prime_number(req: tide::Request<PrimeClient>) -> Result<String, tide::Error> {
    let d: usize = req.param("digits").unwrap_or(1);

    // Use the PrimeClient to ask for a prime number
    // The state is available under req.state()
    let response = req.state().request_prime(d).await.map_err(|_| {
        tide::Error::from_str(
            tide::StatusCode::InternalServerError,
            "I'm sorry, I couldn't get a prime number",
        )
    })?;

    Ok(format!("{}\n", response))
}

Let’s give it a run!

$ cargo run               
   Compiling floating_on_tide_2 v0.1.0 (/Users/ignition/Projects/oss/bastion/floating_on_tide_2)
    Finished dev [unoptimized + debuginfo] target(s) in 3.37s
     Running `target/debug/floating_on_tide_2`

We can now spawn an other shell and start performing requests:

$ curl http://127.0.0.1:8080/prime/15
623361288611851 is a prime number that has 15 digits.
It was found in 0s and 350ms

$ curl http://127.0.0.1:8080/prime/15
947499788349449 is a prime number that has 15 digits.
It was found in 0s and 410ms

$ curl http://127.0.0.1:8080/prime/15
510096850144577 is a prime number that has 15 digits.
It was found in 0s and 327ms

$ curl http://127.0.0.1:8080/prime/15
826290052329481 is a prime number that has 15 digits.
It was found in 0s and 427ms

Congratulations! We have successfully plugged Bastion to our tide server!

Conclusion

What do you mean by conclusion? Where’s the Supervisor? What is a dispatch mechanism? How can we add fault tolerance to our system? Our system still isn’t panic! proof yet is it?

You’re correct, we needed to do all of this work so we can learn about all of these amazing things in part 3 of this series, which will probably be the last one!

Again, I hope you enjoyed reading this post as much as I enjoyed writing it, and I can’t wait to share part three with you!

The complete example file is available in this Github gist and you can reach out to me anytime if you have a question.

If you have any questions or would like to have a chat with us, don’t hesitate to reach out to the Bastion team on Github, Twitter, and Discord!

Bastion floating on Tide - Part 1

Sun, 23 Feb 2020 00:00:00 +0000

Part 1: Building a Tide server

Introduction

This post is Part one of a series on how to build a highly fault tolerant system using bastion-rs, and expose an async web api with Tide.

Part one is about setting up a simple Tide server, and covers initial steps such as adding dependencies, some basic routing and the request/reply model.
In Part two we will do a bit of refactoring, and then transform our function into a bastion child.

The complete example file is available in this Github gist and you can reach out to me anytime if you have a question.

Creating a simple Tide project

If you haven’t yet, I highly recommend you to install cargo-edit, a tool that will allow you to add dependencies to a project without the need to edit Cargo.toml manually (among other really cool features). If you don’t want to install it, you need to skip the cargo add statements in the example, and edit Cargo.toml to add the relevant crates as we move along.

$ cargo install cargo-edit

Let’s start with a Tide example:

$ cargo new bastion_with_tide # Create a new project
$ cd bastion_with_tide # Go to the project folder
$ cargo add async-std # Add async-std
$ cargo add tide # Add tide

Now that we created a project and added the required dependencies, let’s edit the src/main.rs file:

fn main() -> Result<(), std::io::Error> {
    use async_std::task;
    task::block_on(async {
        let mut app = tide::new();
        app.at("/").get(|_| async move { "Hello, world!" });
        app.listen("127.0.0.1:8080").await?;
        Ok(())
    })
}

And run the application:

cargo run
[…]
   Compiling bastion_with_tide v0.1.0 (/Users/ignition/Projects/oss/bastion_with_tide)
    Finished dev [unoptimized + debuginfo] target(s) in 35.37s
     Running `target/debug/bastion_with_tide`

Now let’s open a browser and go to http://127.0.0.1:8080

Ok we now have a Tide web server up and running, but we’re not doing too much for now. Let’s try to build an application that will do some heavy computing behind the scenes.

Let’s use some CPU cycles!

~~For the purpose of this example, we will create a fibonacci adder.~~

Florian skade Gilcher suggested an amazing example, that some of you might have read about already. For the purpose of this example, we will create an amazing service for delivering prime numbers to the internet!

If this exercise doesn’t ring a bell, don’t worry, I didn’t know about it either, and I had an amazing journey discovering “Programming Erlang”, written by Joe Armstrong.

If you were interested simple examples on how to use Tide, head over to the Tide examples. If you would like to know how a Fibonacci example would look like, I have submitted a pull request to add it to the Tide examples.

Our prime number service will, given a number of digits, return a prime number with the expected number of digits.

In pseudocode it would look like this:

GET: /prime/:digits => return prime_number(digits)

First of all, we want to extract some data from the route we create. We can leverage the power of Tide and get named parameters by specifying a route and a handler function:

app.at("/prime/:digits").get(prime_number);

The Tide router will extract d as a parameter, which will be queryable in our prime_number function:

async fn prime_number(req: tide::Request<()>) -> String {
    let d: usize = req.param("digits").unwrap_or(1);
    format!("you asked me to compute a prime number that has {} digits.\n", d)
}

Go for a cargo run and give it a try with different numbers:

➜ curl "127.0.0.1:8080/prime/10"
you asked me to compute a prime number that has 10 digits.

Let’s now do some heavy computation!

Well maybe not… We’re going to create a couple of functions that will allow us to compute a prime number with the correct width. Long story short we will iterate over numbers with the correct number of digits starting from a random one, and use a test to find out if the random number is a prime number. It will look like this:

fn get_prime(num_digits: usize) -> u128 {
    let min_bound = get_min_bound(num_digits);
    // with num_digits = 4, max_bound == 10000
    let max_bound = get_max_bound(num_digits);
    // maybe_prime is a number in range [1000, 10000)
    // the closing parentheses means it won't reach the number.
    // the maximum allowed value for maybe_prime is 9999.
    use rand::Rng;
    let mut maybe_prime = rand::thread_rng().gen_range(min_bound, max_bound);
    loop {
        if is_prime(maybe_prime) {
            return maybe_prime;
        }
        // for any integer n > 3,
        // there always exists at least one prime number p
        // with n < p < 2n - 2
        maybe_prime += 1;
        // We don't want to return a number
        // that doesn't have the right number of digits
        if maybe_prime == max_bound {
            maybe_prime = min_bound;
        }
    }
}

If you want to follow along and be able to do cargo runs, here’s what you want to do:

Add a dependency to the rand crate, so we can generate random numbers:

$ cargo add rand

In src/main.rs, we want to add a missing use statement:

use std::iter;

Here’s the contents of the missing functions:

// in order to determine if n is prime
// we will use a primality test.
// https://en.wikipedia.org/wiki/Primality_test#Pseudocode
fn is_prime(n: u128) -> bool {
    if n <= 3 {
        n > 1
    } else if n % 2 == 0 || n % 3 == 0 {
        false
    } else {
        for i in (5..=(n as f64).sqrt() as u128).step_by(6) {
            if n % i == 0 || n % (i + 2) == 0 {
                return false;
            }
        }
        true
    }
}

// given a sequence of digits, return the corresponding number
// eg: assert_eq!(1234, digits_to_number(vec![1,2,3,4]))
fn digits_to_number(iter: impl Iterator<Item = usize>) -> u128 {
    iter.fold(0, |acc, b| acc * 10 + b as u128)
}

fn get_min_bound(num_digits: usize) -> u128 {
    let lower_bound_iter =
        iter::once(1usize).chain(iter::repeat(0usize).take(num_digits - 1 as usize));
    digits_to_number(lower_bound_iter)
}

fn get_max_bound(num_digits: usize) -> u128 {
    let lower_bound_iter = iter::once(1usize).chain(iter::repeat(0usize).take(num_digits));
    digits_to_number(lower_bound_iter)
}

I won’t go too much into the implementation details here, but the whole example is available in this Github gist and you can reach out to me anytime if you have a question.

If you would like to know more about the theory behind prime numbers generation, I will instead urge you to dig more into Joe Armstrong’s book, and have a look at Bertrand’s postulate, and Fermat primality test. If you (like me) are still amazed by what you just learnt about prime numbers, have a look at a special subset of prime numbers, called Mersenne prime numbers.

Ok enough with this rabbit hole, let’s go for a cargo run.

We need to edit our get_prime function get a prime number, and return it among a couple of information on how long it took us to compute it:

async fn prime_number(req: tide::Request<()>) -> String {
    use std::time::Instant;
    let d: usize = req.param("digits").unwrap_or(1);
    // Start a stopwatch
    let start = Instant::now();
    // Get a prime number
    let prime_number = get_prime(d);
    // Stop the stopwatch
    let elapsed = Instant::now().duration_since(start).as_secs();
    format!(
        "{} is a prime number with {} digits.\nIt was computed in {} seconds.\n",
        prime_number, d, elapsed
    )
}

Go ahead and give it a run, and notice how the computation time grows depending on the number of digits (and luck, mostly).

For example computing this sequence on my machine for d = 15 took me less than a second. It took me 9 seconds for d = 18, and 237 seconds for d = 20 !

➜ curl "http://127.0.0.1:8080/prime/15"
729994968290557 is a prime number with 15 digits.
It was computed in 0 seconds.
➜ curl "http://127.0.0.1:8080/prime/16"
7717274638523017 is a prime number with 16 digits.
It was computed in 1 seconds.
➜ curl "http://127.0.0.1:8080/prime/17"
76559666666708933 is a prime number with 17 digits.
It was computed in 3 seconds.
➜ curl "http://127.0.0.1:8080/prime/18"
654693728093206181 is a prime number with 18 digits.
It was computed in 9 seconds.
➜ curl "http://127.0.0.1:8080/prime/19"
1832975536808651843 is a prime number with 19 digits.
It was computed in 17 seconds.
➜ curl "http://127.0.0.1:8080/prime/20"
34003387272791828929 is a prime number with 20 digits.
It was computed in 237 seconds.

Depending on your computer, the duration might vary a little bit. Find a sweet spot you’re comfortable with, we will keep using it for the rest of the tutorial.

Add a little twist

The more complex a system becomes, the more likely it is to fail. In order to introduce Bastion and a couple of its basic features, we will introduce a failure mode that will happen randomly in our example, and see how Bastion can help us reason about failures and complex systems. But first, a disclaimer.

Disclaimer

THIS EXAMPLE COMES AS IS… Yeah no that’s not what I meant by a disclaimer.

async-rs and the async-std library allow you to achieve a lot, and by a lot I really mean A LOT. The twist I’m about to introduce (leading to a panic) is not idiomatic at all, and you’re much better off leveraging Result, Option and other awesome Rust structures that were built to do this. The purpose of this example is not to make you think Bastion can solve this problem and async-rs cannot. async-rs can totally handle failures (and generating prime numbers for that matter) in a graceful way.

I would also like to let you know that the async-rs team has been doing an amazing job, and that they are an amazing team, always friendly, super helpful, and they helped me a lot while I was preparing the examples and this blog post.

When it comes to choosing a runtime, I encourage you to first define your goals and what matters the most, and then pick the best tool for the job. Sometimes an actor based framework is not necessary because your system will never reach a high level of complexity. Sometimes you should rather start with a very simple web server, and let the tools evolve as your system grows.

As John Gall states it:

A complex system that works is invariably found
to have evolved from a simple system that worked.
A complex system designed from scratch never works
and cannot be patched up to make it work.
You have to start over with a working simple system.

As a more general thought, when it comes to libraries and frameworks focusing on topics in the same area, I really believe having different approaches and being helpful to each other will improve the quality of each framework, the ecosystem, and the community as a whole.

With that in mind, let’s introduce a bit of chaos to our server!

Chaos !

We will create a number_or_panic function that will well… either return the prime number or panic :)

fn number_or_panic(number_to_return: u128) -> u128 {
    // Let's roll a dice
    if rand::random::<u8>() % 6 == 1 {
        panic!(format!(
            "I was about to return {} but I chose to panic instead!",
            number_to_return
        ))
    };
    number_to_return
}

We can now edit our get_prime function to sometimes panic, and watch the server fail:

loop {
    if is_prime(maybe_prime) {
        return number_or_panic(maybe_prime);
    }
    // [...]
}

Let’s take our example for a cargo run and watch it fail after a couple of calls:

$ cargo run
    Finished dev [unoptimized + debuginfo] target(s) in 0.08s
     Running `target/debug/bastion_with_tide`
thread 'async-std/executor' panicked at 'I was about to return 2940675629 but I chose to panic instead!', src/
main.rs:41:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace.
[1]    26676 abort      cargo run

And here’s what happened on the client’s side:

$ curl "http://127.0.0.1:8080/prime/10"
2864551423 is a prime number with 10 digits.
It was computed in 0 seconds.
$ curl "http://127.0.0.1:8080/prime/10"
3718438259 is a prime number with 10 digits.
It was computed in 0 seconds.
$ curl "http://127.0.0.1:8080/prime/10"
8959500691 is a prime number with 10 digits.
It was computed in 0 seconds.
$ curl "http://127.0.0.1:8080/prime/10"
curl: (52) Empty reply from server
$ curl "http://127.0.0.1:8080/prime/10"
curl: (7) Failed to connect to 127.0.0.1 port 8080: Connection refused

A couple of prime numbers were generated, then the server sent an empty reply. What’s even more unfortunate is that once an empty reply was returned, further connections are simply refused. In hindsight it makes sense. Because the panic caused the server to crash. It can’t serve requests anymore.

Conclusion

As said in the disclaimer, there are several ways to recover from this, and in part two, we will explore one way to escape this prime numbers russian roulette. We will add fault tolerance, and dive a bit into the actor model by building a Bastion on Tide!

I hope you enjoyed reading this blog post as much as I enjoyed writing it, and I can’t wait to share part two with you!

The complete example file is available in this Github gist and you can reach out to me anytime if you have a question.

I you have any questions or would like to have a chat with us, don’t hesitate to reach out to the Bastion team on Github, Twitter, and Discord!

Writing AWS Lambda Functions with Bastion

Sat, 01 Feb 2020 00:00:00 +0000

In this post, we will look how AWS Lambda Functions can be implemented using Bastion. AWS Lambda functions are well known serverless platform used for tasks varying from infrastructure management, serving APIs, data processing to orchestration tasks.

Let’s start! We are going to use the Bastion AWS Lambda Example in our showcase repo.

First let’s clone the Bastion showcase repository:

$ git clone git@github.com:bastion-rs/showcase.git

Get into the containing directory of our example lambda declaration:

$ cd showcase/bastion-aws-lambda

We are using serverless utility to leverage compilation and configuration of our lambda. In this directory you will see how our lambda is configured by serverless.yml. The serverless invoke local command we will run needs to build a Docker container, so keep in mind you need to run Docker for this example to work locally.

In order to install serverless and its dependencies for Rust environment we will do:

$ npm i

Now we have installed both dependencies for compiling and configuring our lambda. Our lambda is taking a list of sites as input and concurrently making GET requests. Immediately returns their body strings whenever they are available. Main code for lambda is in page_fetcher/src/main.rs. Let’s take a look at the Lambda code and walk through it:

/// This is the JSON payload we expect to be passed to us by the client accessing our lambda.
#[derive(Deserialize, Debug)]
struct InputPayload {
    sites: Vec<String>
}

This is our input payload that contains sites that will be requested when lambda is triggered.

/// This is the JSON payload we will return back to the client if the request was successful.
#[derive(Serialize, Debug)]
struct OutputPayload {
    status: String
}

When we process sites and receive body of the sites we are going to return the lambda status. Think that these are the cluster internal applications which we concurrently trigger their endpoints. Instead of sites you can use your cluster internal naming.

Message Handler (Dispatcher)

fn dispatcher(
    payload: InputPayload,
    _c: Context,
) -> Result<OutputPayload, HandlerError> {
    let (p, mut c) = unbounded::<bool>();

    Bastion::children(|children: Children| {
        children.with_exec(move |_ctx: BastionContext| {
            let sites = payload.sites.clone();
            let workers = worker_pool(payload.sites.len());
            let p = p.clone();

            async move {
                info!("Dispatching started");

                for (worker, site) in workers.elems().iter().zip(sites) {
                    info!("Site sent for processing!");
                    let answer = worker.ask_anonymously(site).unwrap();
                    // Or use the returned body
                    let _ = answer.await.unwrap();
                }

                let _ = p.unbounded_send(true);

                Ok(())
            }
        })
    }).unwrap();

    // Wait for completion signal OR data itself
    while let Err(_) = c.try_next() {}

    Ok(OutputPayload { status: "OK".into() })
}

Let’s break this dispatcher down. Lambda runtime unfortunately doesn’t allow us to write direct runtimeless lambda application so we need to write a dispatcher that will be registered to underlying runtime. So this is our dispatcher that will use the underlying runtime to dispatch requests to Bastion for processing.

We created an unbounded channel for receiving completion that we made requests to all given sites and received a response from each other:

let (p, mut c) = unbounded::<bool>();

We are creating a children group to setup workers and fan out requests to workers. So we will make our workers run individually:

let workers = worker_pool(payload.sites.len());

The line above sets up workers using worker_pool method with given amount of sites. We will explain setup method later.

async move {
    info!("Dispatching started");

    for (worker, site) in workers.elems().iter().zip(sites) {
        info!("Site sent for processing!");
        let answer = worker.ask_anonymously(site).unwrap();
        // Or use the returned body
        let _ = answer.await.unwrap();
    }

    let _ = p.unbounded_send(true);

    Ok(())
}

We link each worker and the site. Using ask, we are sending site to the workers. Here, we need to know the difference between ask and tell. tell method works like a fire and forget messaging system. Message passed with tell won’t expect response from the message receiving actor. In Bastion we have ask and ask_anonymously. Latter is usable across BastionContext’s. What we call context is basically a hierarchy branch manager which allows granular microconcurrency between children, child and supervisors. ask is usable within the same context which allows us to directly identify children inside the same hierarchy. We are using ask_anonymously in here because workers and dispatcher are separate branches.

Our workers are sending back the body. Even though we are not using it we wait for the response from all the workers. If we don’t wait inside the for loop, order will be different across responses. In Bastion order of messages belong to user’s code.

After these explanations for the dispatcher. Let’s take a look at the workers:

Worker Actors

fn worker_pool(pool_size: usize) -> ChildrenRef {
    Bastion::children(|children: Children| {
        children
            .with_redundancy(pool_size)
            .with_exec(move |ctx: BastionContext| {
                async move {
                    info!("Worker started!");

                    // Start receiving work
                    loop {
                        msg! { ctx.recv().await?,
                            site: String =!> {
                                info!("Received site: {}!", site.clone());
                                let body = surf::get(site.clone()).recv_string().await.unwrap();
                                warn!("Site: {} Body: {}", site, body);
                                let _ = answer!(ctx, body);
                            };
                            _: _ => ();
                        }
                    }
                }
            })
    })
    .expect("Couldn't start a new children group.")
}

We are passing the given pool_size as redundancy:

children
    .with_redundancy(pool_size)

Then we are giving the actor body:

.with_exec(move |ctx: BastionContext| {
    async move {
        info!("Worker started!");

        // Start receiving work
        loop {
            msg! { ctx.recv().await?,
                site: String =!> {
                    info!("Received site: {}!", site.clone());
                    let body = surf::get(site.clone()).recv_string().await.unwrap();
                    warn!("Site: {} Body: {}", site, body);
                    let _ = answer!(ctx, body);
                };
                _: _ => ();
            }
        }
    }
})

In this body you might have realized that we receive messages with msg! macro. And it has an interesting syntax. While designing Bastion we’ve thought about tell, ask and broadcast scenarios between actors. =!> corresponds to asked messages. So as you know we’ve asked our workers to fetch a site. We are receiving this match from ask arm of our mailbox.

Oh yes, we are using surf to issue our call. and returning the response with our answer! macro.

Application’s Main

These were the methods that were doing the processing. It’s time to take a look at the application protected against crashes. Our main entry:

#[fort::root]
async fn main(_: BastionContext) -> Result<(), ()> {
    let _ = simple_logger::init_with_level(log::Level::Info);
    lambda!(dispatcher);

    Ok(())
}

Fort is a proc macro attribute that supplies fault tolerance to main method and wraps your main method into Bastion. It gives you root context of the Bastion system as argument for further hierarchy and let you run your application in panic handler context. Rest is basically initializing our logging system, and lambda handler registration.

That’s all we have inside the code. For more information take a look at to our Documentation.

Testing locally

For testing locally what we need to do is:

$ ./node_modules/.bin/serverless invoke local -f page_fetcher -d \
    '{
      "sites": [
        "https://www.bastion-rs.com",
        "https://blog.bastion-rs.com",
        "http://google.com",
        "https://docs.rs/",
        "https://crates.io/",
        "https://twitter.com/",
        "https://news.ycombinator.com/",
        "http://play.rust-lang.org/",
        "http://catb.org/jargon/html/hates.html"
      ]
    }'

Then wait for container build finish, artefact extracted, and run inside the container. Then hopefully you will see the lambda logs.

Conclusion

In this post, we introduced Bastion to AWS Lambda runtime. For your own applications feel free to explore more advanced methods, use message passing with different approaches mentioned above, design your own hierarchy and concurrency nodes. Don’t forget to share your suggestions and ideas with us. Join our Discord server, open an issue/feature request/support ticket in our GitHub.

New age of Bastion

Thu, 30 Jan 2020 00:00:00 +0000

Disclaimer: This post is not only an announcement of new features of Bastion, which comes with 0.3.4; it also declares our future perspective for the project.

So we had a long way since our project had started in June 2019. Right now, we are evolving from a runtime that enables fault-recovery for your server-side applications to an agnostic runtime, which has its executor but also works with other executors. Since the beginning we can interoperate with async-std without the hassle.

We have started with an idea of runtime, which can recover from failures in the application domain, no matter what happens. Async/await landed, and we got stronger and well known in the community. Then Bastion deployed to AWS lambdas, did, and still doing its job in data processing pipelines. Benchmarks of sparse queries outperformed plenty of sparse queries and aggregations over NYC taxi data in various inmem benchmarks.

Where futures-executor performs:

test count_aggregation      ... bench: 108,916,527 ns/iter (+/- 11,909,528)

Bastion performed:

test count_aggregation      ... bench: 87,096,485 ns/iter (+/- 3,874,812)

Everything started with an idea of replicating what Erlang does in its runtime. Having the idea of using the runtime for data processing came right after it. Bastion’s purpose is to provide a generic runtime for fault-tolerant middlewares, distributed systems, and data processing pipelines.

I didn’t aim for faster runtime, but I aimed to exploit the whole resources whenever it’s possible. Bastion also doesn’t aim to replace every other existing runtimes. It just unfolds a different approach to design systems. This system design is very graspable for the people who come from a functional background like Scala, Erlang, or Haskell. When we released Bastion 0.1 in June 2019 with Tokio runtime, I realized that it’s not the way that this project can continue. With premature design assumptions of the runtime, I struggled with developing the project with Tokio runtime. Having to implement a monolithic runtime with several lack of reduction based concurrency principles made me write an executor for Bastion. So we end up having a multiple run queue based, redundancy statistics sampled, async executor which is a primer of the runtime. Throughout time, people wanted to use blocking tasks also. So we came with an idea of using an adaptive blocking pool, which can automatically scale the resources based on the throughput and prevents IO hogs.

What is next with Bastion?

We are currently working on being runtime agnostic. Working in everywhere but still supplying our own executor to the people who want to have fault-tolerant systems in their projects.
We are exploring HCS(Hot code swap) case. And currently, we have design discussions going on. We would be really happy to gather your feedback and ideas on HCS!
WASM is a rising star in the ecosystem. We would like to support it, first we want to make everything #[no_std], then we want to move towards to make Bastion projects compile for WASM.
We are working on Bastion Streams! A streaming framework implementation which is backpressure aware and works with a push interface like API. Many people are asking about how this will be implemented, and I have pretty concrete ideas about this. Especially from the perspective of Rust! So now I am quite bit busy enabling parallel streams in Rust. Not with this, or this, or via this. Come join us if you find this interesting!
Distributed properties are still needs to be solved. We are slow at that side but wrote initial epidemic protocol implementation We aren’t moving too fast on this, but an initial epidemic protocol implementation can be seen and used in our Artillery repository.

New version is in here! Tell me more!

We have added features to 0.3.4, you can see it in our CHANGELOG. But I will showcase a bunch of excellent features of the new Bastion version!

At Bastion itself, at the surface API level, here are our changes:

0- We added restart strategies, linear and exponential backoff between restarts. And maximum restarts for the failed children. Check out our example in the repo. Thanks to @Relrin and for the clean cut of their PR!

// Here we are specifying the used restart strategy for our supervisor.
// By default the bastion's supervisors are always trying to restart
// failed actors with unlimited amount of tries.
//
// At the beginning we're creating a new instance of RestartStrategy
// and then provides a policy and a back-off strategy.
let restart_strategy = RestartStrategy::default()
    // Set the limits for supervisor, so that it could stop
    // after 3 attempts. If the actor can't be started, the supervisor
    // will remove the failed actor from tracking.
    .with_restart_policy(RestartPolicy::Tries(3))
    // Set the desired restart strategy. By default supervisor will
    // try to restore the failed actor as soon as possible. However,
    // in our case we want to restart with a small delay between the
    // tries. Let's say that we want a regular time interval between the
    // attempts which is equal to 1 second.
    .with_actor_restart_strategy(ActorRestartStrategy::LinearBackOff {
      timeout: Duration::from_secs(1),
    });

// Define the supervisor
supervisor
	 // That uses our restart strategy defined earlier
	 .with_restart_strategy(restart_strategy)
	 // And tracks the child group, defined in the following function
	 .children(|children| failed_actors_group(children))

1- Now we have a signature method to address all the actors in the system. Check our context addressing system here

Bastion::children(|children| {
    children.with_exec(|ctx: BastionContext| {
        async move {
            ctx.tell(&ctx.signature(), "Hello to myself");

            Ok(())
        }
    })
}).expect("Couldn't create the children group.");

2- We’ve added meta-architecture instantiation, so you can have a boilerplate free hierarchy for your actors and/or workers.

let children = children! {
	// the default redundancy is 1
	redundancy: 100,
	action: |msg| {
	   // do something with the message here
	},
};

// simpler supervisor declaration.
let sp = supervisor! {};

let children = children! {
  	supervisor: sp,
  	redundancy: 10,
  	action: |msg| {
    	// do something with the message here
    },
};

3- You can now mark your blocking code and execute it inside Bastion’s blocking adaptive thread pool.

let task = blocking! {
	thread::sleep(time::Duration::from_millis(3000));
};

4- If you want to run a process and wait for it’s completion, you can call run!. run is equivalent to what other Rust frameworks call block_on. We decided to call it run because that’s the name implementations in other languages use. We think sticking to cross languages conventions will avoid confusion among users, regardless of their favorite language.

With the blocking task written above, we can use:

run!(task);

And that’s it! This is equivalent of zio running effects in Rust environment. Now we have it!

The Executor, our huge steampunk engine

This is the core, the piece everything is running on top of.

It is way more diverged and different from any runtime out there. The approach we took is very different from the other frameworks in the ecosystem.

What are the differences ? Let’s have a look!

0- When under contention we skip work-stealing. We directly force queue consumers to handle messages coming from the run queue and continue working without work-stealing for a small fraction of time. Think of it like a congestion unblocking mechanism. It works pretty well so far! Works pretty well.

1- Bastion executor relies mostly on interconnect messages and tries to do single clock advancement without OS migration of threads. We are pinning our threads which means we need to be aware of how many vCPU Bastion is running on, and we maintain thread pinning on the fly. A lot of efforts were put into our pinning API in this release.

2- We have a blocking pool based on the paper A Novel Predictive and Self–Adaptive Dynamic Thread Pool Management and you can see this implementation documentation here. That’s our blocking congestion prevention mechanism and it enables high throughput for the blocking operation under various cases ranging from long running, spike oriented, bursting, decaying task timing behaviors. Moreover, this thread pool is also configurable with an environment variable for constrained environments.

3- We improved the performance by making less syscalls when yielding. We are now passively looking for an opportunity to run tasks, and yield only once, as soon as possible.

4- Added convenience methods for blocking pool with spawn_blocking in addition to existing spawn, run. If you don’t want any supervision and restart mechanisms, and are just looking for a quick dive into bastion, add bastion-executor to your Cargo.toml file and you can start spawning sync and async tasks in no time !

Bastion uses LightProc, and it’s the crucial implementation that makes fault tolerance easier for the applications.

In LightProc we have added:

An interface for storing state! This will be used to persist messages or data in memory and can be backed by anything that you can define.

use lightproc::proc_stack::ProcStack;

pub struct GlobalState {
   pub amount: usize
}

ProcStack::default()
    .with_pid(1)
    .with_state(GlobalState { amount: 1 });

From Rust HashMap’s to your NoSQL backend etc. Wire up anything to your processes. Soon we will expose convenience APIs for mailbox preservation across restarts. @onsails is working on it here.

A Big Thanks

Yes, a big thanks, to everyone who enabled this project to run, supported, wrote code and made it production grade in plenty of aspects. Thanks for everything and thanks for paving the road with us! We are getting bigger and I really like how passionate people are about what are we trying to do here! I can’t thank enough everyone who shared their ideas on our Discord channel, on issues and over the PRs! Thank you all, let’s continue building together! 2020 is a year that we (as Rust community) will unite over plenty of subjects, and topics. I like to close my thanks with these words (though this was a famous US Marshall Plan propaganda, this sentence pretty much sums up all of us as people working on Rust): Whatever the weather we must move together!

Conclusion

In the upcoming blog posts, we will talk about how the Bastion runtime is designed, which design decisions were made (and why) and our newest initiatives and next goals. Meanwhile we are expanding our ecosystem with various projects and would love to welcome maintainers who want to work on distributed systems and data processing. We have pretty high aims for our projects, and would love to help and mentor you. Come join us! Join our Discord, have a look at our issues and projects our org’s GitHub!