Site Reliability Engineers (SREs) design and manage efficient processes and operations, and they keep a company’s infrastructure in healthy working order. 

Here at Algolia, our team has grown from 4 to 10 in less than two years, a growth rate similar to the company as a whole. The team has grown not only in number but in how we work together and create sound operational processes. 

This blog is about how a group of hard-working individuals, with unique skills and working methods, managed to create a successful SRE team.

My own journey into the SRE field reflects this maturation process. Before I joined Algolia, I was traveling around the world as an Integration Engineer for a telco company. On more than one occasion, I found the opportunity to build small tools that helped me be more productive. This increased my confidence to go further professionally. That’s when I started digging into what an SRE is. Here’s what Google VP Ben Trayner says about SREs:

An SRE is what happens when a software engineer is tasked with what used to be called operations.

Although I wasn’t technically a software engineer, my career followed this same pattern: writing code to manage operational tasks. I was therefore excited by the SRE role and ready for a change.

What SREs do at Algolia

Every member of the SRE team gets involved in all three of these activities:

• Projects
• Operations
• On call

Projects: We work on different kinds of projects. Some of them have a direct impact on the business and others help improve the global infrastructure. 

Operations: The majority of our infrastructure is bare metal, which means that we require a decent amount of automation and work on installing/destroying/repairing servers. We need to debug live applications and support other teams searching for technical advice. On top of that, we have to contact providers to manage any issue with the providers.

On call: We need to provide support for the whole infrastructure. This means that each one of us has to be on call 1 week every 4 weeks, 24/7.

Meetings

Every Monday we have a one-hour meeting where we review the previous week and plan the current one. We talk about:

• On-call issues
• Operations
• Project statuses 
• Personal objectives and initiatives

Operations

Operations at Algolia has changed over time. For instance, when I joined, operations were performed on a daily basis, which made it difficult to gain a deep context about what tasks had been completed.  Our golden rule regarding priority is 

• Responding to customers first
• Responding internally to questions on Slack
• Solving incidents
• Provisioning infrastructure 

Operations are now performed on a weekly basis. You can see the rotation plan below. On top of this, we have two levels of on call. In the end, on call should not really be different from the normal operations we do. Overall, operations is more than maintaining and improving our infrastructure.

How we work as a team

Communications   

In my first days at Algolia, I noticed right away that my colleagues were communicating mainly through Slack – even if they were just a few meters from each other. This felt a bit cold, especially given my natural inclination to get up and talk to people. Additionally, the team was scattered across the office. It didn’t feel like a cohesive unit. For these reasons, communication was difficult. 

Interestingly enough, I wasn’t the only one experiencing this. Newcomers arrived and noticed the same issues. 

Newcomers

Critical mass probably pushed us in the right direction: there were just too many tasks to continue functioning as we did before. Our first step was to set expectations for every member, so that everyone would know in advance what they can get from each other.

New people bring in unexpected benefits and qualities to the team that the older members wouldn’t have expected. Additionally, if they are open to it, older members can actually learn from the incoming ones and have these new people/qualities change the team for the better. The right mix of old and new is what makes a team great. 

One initiative we took was to create a coffee break culture, two days a week. During these breaks, we speak about different things, work or not work related. We get to know each other and communicate better.

Pairing creates a team

Initially each member of the squad worked individually on their given projects, choosing for themselves the subject they wanted to work on. This kind of autonomy and personal initiative wasn’t all that bad, but for a newcomer it was overwhelming; it forced me to switch gears often: to learn and do my daily support and come up with project ideas and complete them, all on my own. 

Because we were growing, we quickly realized that we needed to change this, we needed to start working together. The first step was to do some pairing, working on teams of two people. This gave us an opportunity to interact and get to know other members on the team. 

Three Projects

1 – Reverse engineering a Vault/Consul server

On this project, Paul and I worked on finding out how the current deployment worked, how we can recover in a disaster scenario, and how much time the recovery would take.   

The project took 2 weeks. Once we were done, we had the choice to change partners or continue working together. At the end of these two weeks we had: 

• Deep understanding of the project
• Automated deployment
• Replicate data

2 – Load Balancer knowledge sharing

The next two weeks I continued to work with Paul on getting more insights on the new Load Balancer. If you have not read his blog post on one year of load balancing, I suggest you do it.

The main problem we found was that during operations or on call, any request regarding the load balancer had to be forward to Paul. This knowledge sharing has two benefits: first, it provides insights to more than one person; secondly, it offloads tasks from the person having all the knowledge.

Recently, a new member joined Algolia and started working on the load balancer full-time. Dedicated support on the load balancer not only provides the new member with more operational skills, but also gives the team more people to consult.

3 – Backup solution

In this project, three members of the Foundation squad worked together to bring up a new backup solution. The interesting part here is that we decided to start using Scrum methodology. This was a huge success as it allowed us to:

• Define small tasks inside more complex ones, with the benefit that all three of us would work on it at some point.
• Establish a time estimation, which helped us foresee how further in the project we were.
• Auto-assign tasks. This was the best part for me, being able to start working on something else once I was done with what I had to do.  
• Have greater visibility on the project due to the fact that we all work on almost all parts.

The journey continues ..

Our efforts have worked so well, and our team has become so stable, that I’ve taken a fresh look at the original “problems” I encountered when I started: the overuse of Slack and a scattered team not sitting together. Today, this is not a problem: we use Slack constantly, and yet the conversation feels natural and direct. And I can sit on a different floor or desk every day and easily collaborate with my team – because I know how the people on my team work. This is what makes for a great SRE experience: effective communication, efficient processes.