Transforming Microservices async events using Protobuf

In Augury you can be a Backend Developer in a product squad and still lead long complex technological projects. Each developer is part of a squad in which they invest 80% of their time. The remaining 20% is invested in Backend initiatives which usually occur on Backend Days every other week.

The squad that I am in, is responsible for handling the downstream events coming from our Algo’s Pipelines. The Algo team has several pipelines, each pipeline is written in python and creates events. In my squads’ domain we have several services which act as consumers written in Go. These services handle the events and distribute them to different domains in our ecosystem.

Why improve our schemas definition?

Currently these events are json based, meaning that each producer in python creates the related json, sends it to the relevant message broker and the Go services consumes these events. The problem here is that we have 7 different developers working on the same interface in 2 different languages and on top of that, the schema is not strictly typed. This causes us issues from time to time, for example a value type mismatch or an upper lower case misspell. Due to the complexity of the domain and the event itself, it is pretty much impossible to create integration tests that will cover all possible flows.

So, after realizing that we can’t improve our integration quality with testing, we decided to improve our event’s schema. We chose to do this by using protobuf instead of json as the format of the event. Because protobuf is language neutral and is strictly typed, it seemed like a good fit to solve my issue. But… protobuf is already used in some areas of our system, mostly in the data and the iot section.

In case you are not familiar with protobuf, it’s a way to define schemas and send them as binary data. To send and read this binary data, we need to serialize and deserialize the values. This is done via a language specific compiled class, which exposes the schema fields, this acts a “serializer”. To create this “serializer” you need to use the compiler “protoc”, the input will be the “.proto” schema file and the output will be the language specific file. The file differs a bit between languages, but has the same serialization under the hood. This enables the “source of truth” of these events to originate from the same place.

See more at: https://developers.google.com/protocol-buffers

Creating standardization cross disciplines in R&D

This got me thinking, if we are using protobuf in several disciplines already, and are planning on adding it to the Backend discipline, we should probably all be aligned. Having a standard way for defining events in our system has a lot of benefits. Starting with development process perks like ease of use, onboarding new employees and having a structured way to define the interfaces between the disciplines to technical benefits like versioning.

Like every initiative in Augury, I started with some research and understanding of our needs. What I’ve learned is that we need a solution that will be easy to use and extendable, that will not require each developer to choose the compilation version and support multiple languages. With all these thoughts and all the proposed solutions, I wrote a Design review document that was shared and discussed with everyone, especially with the discipline leads.

I came up with two main concepts:

1. Have each “producing service” define its own events in a public package in the service repository, and each consuming service will compile and save them for use.

2. Have all events defined as schemas in a central repository and have another central repository for each language of compiled events.

The main pros for each approach (respectively):

1. Autonomy at its best.
2. Having a central repo will make the dependencies clear and less complicated to change as we remodel our services.

Even though it seems that option one is the best solution on paper, it will be problematic to have a simple solution that will support multiple languages. We would need to enhance the build step of all services (4 different languages and tens of current services). On top of that we are scaling and one of our biggest threads in the company is the remodelling of our business domains. This includes Bounded Contexts, Subdomain and Aggregates to fit our growth and business needs. This means that we have a lot of changes happening right now and more to come.

Read about DDD definitions at: https://airbrake.io/blog/software-design/domain-driven-design

Although, looking at option two, we would only need to develop automation for the central repo. To mitigate the fact that we are “losing” ownership, we can add a structure definition to the central repository and define ownership with code owners.

Defining a POC and implement

As you can understand by now, we chose option 2 for its simplicity. Augury is growing, reorganizing and remodelling, so there are a lot of unknowns and we should optimize for easy transitions. I defined a straightforward way to decide how to structure that central repo and I developed a process so the compilation of the protobufs and versioning is completely automatic, meaning executed by our ci tool.

Due to the fact that in my domain I need to implement the usage of protobuf and I wanted a quick win, but also to test my flow as fast as possible, I chose a Go service from our iot domain which already uses protobufs and migrated it to the new solution. Now one of our services uses a central schema project instead of having its events defined in it.

Next Steps

1. Extending the automation solution to support python, java and node.
2. Currently our infrastructure for NSQ, which is our main queueing solution in the Backend, doesn’t support protobuf, so enhancing our infra is also in the near future.

Win Win

I started by saying that each Backend developer puts 20% of their time into the discipline, but it doesn’t mean that this 20% can’t impact the whole R&D department. It all started by a need in my day 2 day squad and transformed into this cross discipline initiative. I love being in a product squad, I feel like I’m making a real difference in the company. But also it’s important for me to have impactful technological initiatives under my belt and this 20% enables it.