Release the Kraken! An open-source pub/sub server for the real-time web

🐙Today, we are releasing Kraken, the distributed pub/sub server we wrote to handle the performance and scalability demands of real-time web apps like Asana.

What problem does Kraken solve?

One of the key promises of real-time web applications is that users will see changes made by other users as they happen, without hitting reload or refresh. Many of the up-and-coming reactive web frameworks initially accomplish this by periodically re-executing queries against the database. When we were building the early prototypes of Asana, our reactive framework (Luna) was no different.

Unfortunately, most production-ready databases, from Mongo to Mysql, are unable to keep up with the query volume that even a moderate userbase creates when you frequently poll the database for changes. We recognized this problem early, and worked around it by designing an algorithm that incrementally updates queries in real time without putting any additional load on the database server. Key to this solution is the requirement that every query be notified about changes to objects that could potentially match, or stop matching the result set of each query.

That’s where Kraken comes into play. Kraken is responsible for routing data-invalidation messages between the processes that are running these queries so that they stay up-to-date.

Why is Kraken the right solution?

Before building Kraken, we searched for an existing open-source pub/sub solution that would satisfy our needs. At the time, we discovered that most solutions in this space were designed to solve a much wider set of problems than we had, and yet none were particularly well-suited to solve the specific requirements of real-time apps like Asana. Our team had experience writing routing-based infrastructure and ultimately decided to build a custom service that did exactly what we needed – and nothing more.

The decision to build Kraken paid off. For the last three years, Kraken has been fearlessly routing messages between our servers to keep your team in sync. During this time, it has yet to crash even once. We’re excited to finally release Kraken to the community!

You can get Kraken yourself at Github