![]() ![]() Neil also has experience in end-user compute and enterprise-scale systems architecture.Everyone is welcome to contribute to Dendrite! We aim to make it as easy as possible to get started. He is interested in security, privacy, decentralised systems and computer networking. Neil Alexander is a software developer working on Stay tuned to see what interesting updates lay ahead next year for Dendrite! About the Author As part of our evolution we made the shift from Kafka to NATS, and have not looked back. In summary, our team is always on the lookout for the right tools and architectures for the performance and flexibility our users require. ![]() We’ve even had it successfully running in application-embedded scenarios on iOS and Android as a part of our peer-to-peer demos. We’ve been able to successfully manage pools of workers from NATS streams and considerably reduce the amount of complexity in our codebase in a number of places which no longer need their own persistent storage. We take advantage of being able to manually acknowledge items to ensure they were processed successfully before cleaning up. We make heavy use of the interest-based retention policies to ensure that work is cleaned up when it is complete. Reduced codebase complexity – simpler code stack due to elimination of complicated persistent storage requirementsįor us, switching to JetStream was transformational.Reduced management overhead – Easier for developers to manage and maintain than Kafka.Simpler data retention – Using JetStream, easier to manage and configure retention than Kafka.Written in Go – Easy to embed, no need for separate server instances like in Kafka.Consumes less system resources – Less infrastructure spend than Kafka.We contributed some patches to make it possible to communicate with the NATS Server in-process without using external sockets this made it possible to both easily target mobile devices, and venture into the web browser under WebAssembly too. Being written in Go, we were able to embed NATS into our own monolith-style binaries so we didn’t even need a separate server instance in those smaller deployments. The NATS Server consumed far less system resources. For a start, it was considerably easier to get running. We evaluated a number of different alternatives and eventually landed on NATS JetStream, which in effect gave us everything we used Kafka for but had a number of additional benefits. We weren’t satisfied with the performance of either mode of operation and found that data retention was especially difficult to manage. To successfully run under a single process without a standalone server handling the events and tasks, we were also forced to mock up our own lightweight Kafka-ish queue implementation, which carried with it additional maintenance burden. While Kafka allowed us to get earlier versions of Dendrite working quickly, we eventually found it to be relatively heavy when used in a full polylith microservice deployment and didn’t scale down as well as we had hoped for smaller deployments. We wanted to be able to target embedded and small platforms, as well as the datacentre. Given our need for deployment flexibility, it’s equally important for all of the microservices to run under a single process as it is to be able to split them out and run them under separate processes - or even on different machines. In addition to being able to scale up and down as needed, we also needed durability in the message queue to ensure that if parts of the service were restarted, no tasks or events were lost. When we first started the Dendrite project, we were using Kafka as a means to distribute events and asynchronous tasks between microservice components. As a result, flexibility, performance, and scalability have been three critical areas of focus in our development. It was originally intended to help us to overcome some load issues that were present in earlier monolithic implementations at the time. Is a federated protocol for real-time communications built on top of distributed data structures and Dendrite is an implementation of a Matrix homeserver built following a microservice architecture. ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |