Sorry, there is no results for this query
Site Reliability Engineers (SREs) design and manage efficient processes and operations, and they keep a company’s infrastructure in healthy working order.
Here at Algolia, our team has grown from 4 to 10 in less than two years, a growth rate similar to the company as a whole. The team has grown not only in number but in how we work together and create sound operational processes.
This blog is about how a group of hard-working individuals, with unique skills and working methods, managed to create a successful SRE team.
My own journey into the SRE field reflects this maturation process. Before I joined Algolia, I was traveling around the world as an Integration Engineer for a telco company. On more than one occasion, I found the opportunity to build small tools that helped me be more productive. This increased my confidence to go further professionally. That’s when I started digging into what an SRE is. Here’s what Google VP Ben Trayner says about SREs:
An SRE is what happens when a software engineer is tasked with what used to be called operations.
Although I wasn’t technically a software engineer, my career followed this same pattern: writing code to manage operational tasks. I was therefore excited by the SRE role and ready for a change.
Every member of the SRE team gets involved in all three of these activities:
Projects: We work on different kinds of projects. Some of them have a direct impact on the business and others help improve the global infrastructure.
Operations: The majority of our infrastructure is bare metal, which means that we require a decent amount of automation and work on installing/destroying/repairing servers. We need to debug live applications and support other teams searching for technical advice. On top of that, we have to contact providers to manage any issue with the providers.
On call: We need to provide support for the whole infrastructure. This means that each one of us has to be on call 1 week every 4 weeks, 24/7.
Every Monday we have a one-hour meeting where we review the previous week and plan the current one. We talk about:
Operations at Algolia has changed over time. For instance, when I joined, operations were performed on a daily basis, which made it difficult to gain a deep context about what tasks had been completed. Our golden rule regarding priority is
Operations are now performed on a weekly basis. You can see the rotation plan below. On top of this, we have two levels of on call. In the end, on call should not really be different from the normal operations we do. Overall, operations is more than maintaining and improving our infrastructure.
In my first days at Algolia, I noticed right away that my colleagues were communicating mainly through Slack – even if they were just a few meters from each other. This felt a bit cold, especially given my natural inclination to get up and talk to people. Additionally, the team was scattered across the office. It didn’t feel like a cohesive unit. For these reasons, communication was difficult.
Interestingly enough, I wasn’t the only one experiencing this. Newcomers arrived and noticed the same issues.
Critical mass probably pushed us in the right direction: there were just too many tasks to continue functioning as we did before. Our first step was to set expectations for every member, so that everyone would know in advance what they can get from each other.
New people bring in unexpected benefits and qualities to the team that the older members wouldn’t have expected. Additionally, if they are open to it, older members can actually learn from the incoming ones and have these new people/qualities change the team for the better. The right mix of old and new is what makes a team great.
One initiative we took was to create a coffee break culture, two days a week. During these breaks, we speak about different things, work or not work related. We get to know each other and communicate better.
Initially each member of the squad worked individually on their given projects, choosing for themselves the subject they wanted to work on. This kind of autonomy and personal initiative wasn’t all that bad, but for a newcomer it was overwhelming; it forced me to switch gears often: to learn and do my daily support and come up with project ideas and complete them, all on my own.
Because we were growing, we quickly realized that we needed to change this, we needed to start working together. The first step was to do some pairing, working on teams of two people. This gave us an opportunity to interact and get to know other members on the team.
On this project, Paul and I worked on finding out how the current deployment worked, how we can recover in a disaster scenario, and how much time the recovery would take.
The project took 2 weeks. Once we were done, we had the choice to change partners or continue working together. At the end of these two weeks we had:
The next two weeks I continued to work with Paul on getting more insights on the new Load Balancer. If you have not read his blog post on one year of load balancing, I suggest you do it.
The main problem we found was that during operations or on call, any request regarding the load balancer had to be forward to Paul. This knowledge sharing has two benefits: first, it provides insights to more than one person; secondly, it offloads tasks from the person having all the knowledge.
Recently, a new member joined Algolia and started working on the load balancer full-time. Dedicated support on the load balancer not only provides the new member with more operational skills, but also gives the team more people to consult.
In this project, three members of the Foundation squad worked together to bring up a new backup solution. The interesting part here is that we decided to start using Scrum methodology. This was a huge success as it allowed us to:
Our efforts have worked so well, and our team has become so stable, that I’ve taken a fresh look at the original “problems” I encountered when I started: the overuse of Slack and a scattered team not sitting together. Today, this is not a problem: we use Slack constantly, and yet the conversation feels natural and direct. And I can sit on a different floor or desk every day and easily collaborate with my team – because I know how the people on my team work. This is what makes for a great SRE experience: effective communication, efficient processes.
Site Reliability Engineer