Making interruptions worth it
A Practical Guide to Code Yellows
Imagine it's your first week at a new job, and the CEO assigns you a critical project: ensuring that Client A, a key enterprise client, does not churn. During a meeting with the recently joined Head of Sales at Client A's office, you discover a series of unmet promises and mismatched expectations about critical service changes due soon.
As you get back with the product team, you discover that one of the most junior teams is working on marginally related changes, and the tech team is not very engaged when the sales team asks for help because they have cried wolf too many times before. It's clear you will fail, and the client will churn. It's time for drastic action: a Code Yellow.
Such interruptions are commonplace in many tech organizations despite well-established quarterly prioritization cycles and the use of OKRs (Objectives and Key Results). These tools are intended to keep teams on track but often fall short when unexpected challenges arise, as illustrated by the crisis with Client A.
In this article, we'll explore Code Yellows and how the process can transform chaotic interruptions into structured opportunities for growth and learning, ensuring that the most pressing problems are not just addressed but solved comprehensively.
Interruptions are a part of the real world. They come from failures in critical (frequently considered legacy) systems that stretched to their limits; other times, sudden changes to the market or critical clients can make strong pushbacks or requests not aligned with the short-term roadmap. Some incidents happen so frequently that it's usual to meet teams that have mastered the drill of emergency palliative support ("Let's just run that script that deletes that file and restarts the service once again and move on"). Teams and businesses can get trapped in a doom loop, struggling to overcome the root causes for good.
Interruptions can also be self-imposed when the leadership team realizes that the current course of action does not show evidence of progress or move the business toward the declared goals, deciding to change the plan to increase the chances of success. Leadership interruptions are polemic and can be especially disturbing for tech teams, which tend to have a high cost of interruption. This political component can lead teams to engage half-heartedly on the redirection in a self-fulfilling weak results prophecy.
Given these interruptions' frequent and varied nature, traditional methods often fail to address them effectively. While at Google, I witnessed a highly effective process called Code Yellow, specifically designed to tackle such critical interruptions head-on. Its name was drawn from hospital emergency color codes and adapted for the corporate tech world. I experimented with it in the startup world and believe it applies well to their reality.
In response to such critical situations, a Code Yellow assembles a strong team for a few weeks with the mindset of performing intense and excellent work to solve a critical problem. This addresses the immediate issue and sets a precedent within the organization for handling crises efficiently, serving as a live case study in teamwork and decision-making.
Declaring a Code Yellow
Understanding what a Code Yellow entails is crucial for recognizing when such a measure is necessary and fitting. Deciding to initiate a Code Yellow (CY for short) is a strategic choice, not made lightly. The tool will only remain effective if used consistently well over time. Declare too many CYs; teams will see them as pompous names for interruptions and work half-heartedly. Declare too complex or ambiguous CYs; they will take too long, teams get tired, and cynicism about their effectiveness creeps up.
Here are some of the legitimate reasons for which I have declared Code Yellows before:
A critical Enterprise client was about to churn while resources and plans were completely misaligned with what mattered to avoid the loss. For the attentive reader, the anecdote from the beginning of the article is a true story. As a result of the CY, we developed a trusted advisor relationship with the client, founded a new Sales Engineering department, and translated several of the requested changes into significant improvements for the client base.
Launching a new service that should solve critical production issues was bound to be delayed on its third quarterly OKR rollover. As a result of the CY, we brought together a senior advising team to make quality and robustness evaluations of the new technology more pragmatic, negotiated more resources to speed up quality evaluations by operators, and resolved newly discovered issues quickly to roll out the new technology to clients.
The company's quality of service was deteriorating in what seemed to be a death by a thousand cuts, and there was no clarity on how to reverse the trend swiftly. As a result of the CY, we made adjustments to the quality metric to contemplate a new class of failures, established new routines with the operations team, and prioritized issues to recover quality standards for most clients, identifying a new class of issues that were of shared responsibility with a few clients.
These diverse situations, though challenging, were turned around by implementing focused interventions, demonstrating the potential effectiveness of the Code Yellow approach.
Drawing from these experiences, consider declaring a Code Yellow only when:
The cost of not declaring it is high (clients are churning, high-performers are leaving, or essential business goals are threatened).
The leadership team will support and sponsor the work throughout the organization (e.g., internal communications, performance evaluation cycles, and protecting the team's focus).
The company hasn't gotten addicted to frequent Code Yellows.
The scope is feasible within a few weeks of work.
An example of when it was inappropriate to use a Code Yellow was one instance at a previous company when an aspect of the B2C strategy was not progressing well, and senior leadership decided to declare a Code Yellow with the goal of "launching something.” In addition to the very ambiguous exit criteria, given that this was a new value that demanded significant exploration, there were several organizational-level reasons for which progress was questionable, including strong misalignment at the leadership level of what the goals were, shared ownership and redundant bets. While this situation was important to address, a more fitting approach was in the senior management sphere. In this situation, the Code Yellow intensified the confusion and dragged on for a long time without a clear target.
Code Yellow leader
A Code Yellow needs a leader, ideally one or two levels more senior than it would take to solve the problem. The leader should align sponsorship with the broader senior leadership and own the CY end-to-end. The CY leader's first responsibilities are to define the scope and the initial team and declare the CY across the organization.
Due to the intense coordination and communication demands, the leader is typically someone from the context's senior product management leadership. Still, sometimes, it can be the engineering management leader when the scope requires less cross-organizational coordination.
Code Yellow's scope
The leader should assemble a senior team to discuss the CY scope and communicate it as a set of exit conditions. The team will remain in CY mode until all exit conditions are met.
The exit conditions must be feasible, clearly measurable/observable, and focused on the root causes that triggered the CY. Only include critical goals and nothing too ambiguous. The aim is for the exit conditions to be met in at most five weeks, ideally less than 3. It's hard to keep the intense dynamics for longer. Even though it's essential to be mindful of the CY's duration, avoid promising a deadline. Work with the team to make fast and robust progress and even set milestone dates internally from a project management perspective, but keep hard dates within the CY.
A set of exit conditions could look like this:
* Migrate 70% of all transactions from service A to B.
* Set up alarms and on-call rotation to speed up recovery. Run a complete smoke test with at most five minutes of downtime.
* Cover root causes from the last three incidents with system-level integration tests.
It's prudent to underpromise on the exit conditions to increase the chances of guaranteeing structural progress while avoiding getting the team stuck at a high bar of expectations during a first iteration. The focus should be on the structural progress and solving the root cause rather than elevating the standards of the current solutions.
Code Yellow's initial team
It's part of the practice that the leader can invite anyone in the company to help with the CY, so it's ideal to keep the initial team small and senior to move fast over robust foundations.
Since the CY leader oversees communication and represents the senior leadership sponsorship, technical experts and solid individual contributors can fill the remaining team. Evaluating and recruiting people who haven't been in too many CYs or could see it as an exciting career opportunity is a nice touch.
Code Yellow's declaration
Communication around a CY should be excellent, starting from its declaration. Everyone in the organization should know there is a new CY, the reasons for declaring it, and who its leader is. Hence, it's a good idea to communicate in multiple formats and channels: Slack, email, and company-wide videoconference are the most common.
A CY is a corporate ritual; just like all other corporate world rituals, it can develop cynicism. The quality and discipline of the communication around it are essential traits that help validate that the organization means it, is vested in it and wants to see it through. Getting the senior leadership involved is also good practice, and the declaration of the CY is an excellent opportunity for VPs and the C-level to weigh in on their sponsorship. If no one from the C-/VP level is the CY leader, have one communicate the declaration.
The declaration should include the CY's name, the motivation for declaring it, the exit conditions, and the leader's name. It should be objective and transparent, reminding people the leader might summon anyone to help. It's a nice touch to wish the CY team good luck and thank everyone in advance for their support.
Executing a Code Yellow
The weeks in a CY should feel like the most productive weeks of the year. The team should work intensively but within a reasonable setup and not cross the lines of work-life balance with extra hours. The work in a CY should be like the one we all wish to perform all year but understand it's virtually impossible to sustain for such a long time. It draws from peak motivation.
CY team members should work only on the CY tasks. The definitive prioritization helps with focus and pace. It also contributes to communicating the costs of running CYs and avoiding addiction: several other tasks are interrupted in favor of the CY.
Three coordination components are essential for a good CY execution: team syncs, stakeholder syncs, and Code Yellow updates.
Team syncs
The team should meet briefly most days each week to discuss progress and challenges and explicitly align each member's next steps.
The CY leader should decide when the regular syncs happen. The frequency can vary over time depending on the work and the need for coordination. Sometimes, teams find it helpful to meet at the beginning of the day and a second touchpoint in mid-afternoon.
At each sync, most team members should report their progress and blockers. The leader can summon other people from the organization to help remove such blockers if necessary. The leader should also feel comfortable dismissing team members who are done with their contributions. This practice helps keep the team agile and laser-focused.
The team should produce notes from every sync containing explicit action items for each member involved. This list of action items can guide the discussions for the following sync.
Stakeholder syncs
Since the CY is communicated widely and typically addresses a prominent problem, many organization members might want to be involved. It's good practice for the CY leader to hold a separate meeting, which happens less often, to inform whoever wants to join and help address tangent concerns, which may include communication with external clients, legal considerations, questions, and general suggestions.
Holding stakeholder syncs is an excellent way to keep the sales and support teams involved without drawing from the focused execution of the CY team.
Code Yellow updates
The leader is responsible for keeping the organization well informed of the progress, especially regarding the exit conditions. It's a good practice to set a particular time of the day to share updates and keep the content objective and focused on each exit condition.
The clocklike regularity and discipline of communication can be a great motivator for keeping the team moving quickly. The update should state this even if the leader has nothing new to share.
It is helpful to post regular communication in a public Slack channel and to an email group that anyone can subscribe to. The leader should auto-subscribe the senior leadership and primary stakeholders to receive the updates.
Wrapping it up
A successful CY means more than the team meeting the exit criteria. It's a testament to the company's resilience and capability to course-correct or adapt to unexpected challenges.
It's essential that the leader be mindful of what CYs represent and care for a firm closure as if they are relaying the baton to the leader of the following CY that might come ahead. Impacts should be made clear, or risk a lack of sponsorship from senior leadership the next time the company would benefit from a CY. Individuals should be widely recognized or risk half-hearted engagement from the following team members.
As the CY ends, it's essential that the leader dedicate energy to a comprehensive analysis of the impacts and visual presentations. This will validate all the team's efforts and reinforce the value of such initiatives for the entire organization.
The communication about the CY's conclusion should echo its declaration: use the same concepts and channels and involve senior leadership, who are crucial in highlighting the CY's success to a broader audience and celebrating the achievements.
The careful impact analysis and presentations become valuable assets for team members to demonstrate their contributions in the following performance evaluations. This content is a strategic resource since the CY will probably be the main undertaking of the performance cycle for the team members who deserve recognition.
Lastly, gather the team and primary stakeholders to celebrate the results and consolidate the CY as an opportunity for team bonding and trust-building.
As we bring a Code Yellow to a close, the focus shifts from immediate resolution to long-term impact. Successfully concluding a CY unfolds the opportunity to integrate the lessons learned and the momentum gained into ongoing organizational practices.
Successful Code Yellows set a precedent for how challenges are addressed, turning what could be a disruptive interruption into a demonstration of proactive problem-solving and excellence. Code Yellows can turn pressing challenges into opportunities for the business and team members.
Making interruptions worth it was originally published in Desirable Difficulty on Medium, where people are continuing the conversation by highlighting and responding to this story.