Organizations with high resilience, digital maturity and high observability through dashboards and other tools should embrace chaos engineering, as they can take immediate action on issues that occur through experiments. Organizations that lack this observability⁴ (link resides outside ibm.com) can take too long to resolve the experiments they create through chaos engineering.

Chaos engineering is also a must-have for organizations that are using cloud, particularly public cloud and cloud-native apps. The public cloud introduces potential outage issues that require coordination with the cloud provider, which creates a different approach than dealing with on-premises issues. 

Enterprises that use the cloud still often approach IT incidents without considering how the cloud and software-as-a-service (SaaS) impact those incidents differently according to Constellation Research⁵ (link resides outside ibm.com). 

In addition, the rise of using microservices, which increases how many hosts or containers are running in a system, creates unique challenges (link resides outside ibm.com) that can be unearthed and solved through chaos experiments. It shifts complexities from code design into system operations, which does not eliminate the complexities but allows for greater automation.

Chaos engineering can also help organizations to enhance the velocity of their continuous integration and continuous delivery (CI/CD) pipelines. Incorporating chaos engineering into CI/CD as Netflix did⁶ (link resides outside ibm.com) enables organizations to automate continuous experiments while controlling their potential impact. 

Finally, the fact that organizations increasingly connect with partners through APIs means that an issue in their systems can have a knock-on impact on other organizations. Deploying chaos engineering helps organizations understand the weak points in their architecture and correct them, ultimately creating the ability to anticipate future failures.

Successful chaos engineering helps organizations to minimize technical failures with any significant customer impact and it also supports the construction of stronger and more resilient complex system architectures. Once an organization decides to pursue chaos engineering, the next step is determining whether to execute it in the pre-production or production environment.

DevOps teams have several options for running chaos engineering experiments to test various system processes.

• Latency injection: DevOps teams intentionally create scenarios that emulate a slow or failing network connection. This includes the introduction of network delays or slower response times.
• Fault injection: This involves purposefully introducing errors into the system to determine how it affects other dependent systems and whether it interrupts services. Examples of fault injections include inducing disk failures, terminating processes, shutting down a host or introducing power or temperature increases. Fault injections can help organizations identify any single points of failure, which can cause the entire system to fail if something happens to them.
• Load generation: This relates to intentionally stressing the system by sending significant traffic levels well beyond normal operations. This helps the site reliability engineers (SREs) to understand any bottlenecks in the system, which in turn allows them to build more scalable systems. 
• Canary testing: This involves releasing a new product or feature to a small group of users. That way, any glitches or bugs will only affect a percentage of visitors, leaving the rest of the audience to access the existing website experience.

Creating the ideal chaos engineering process requires several principles to ensure that an organization can have a distributed system at scale. 

• Understand the system: This involves a comprehensive knowledge of the holistic system, its emergent properties and functions and its topology, architecture, dependencies, steady-state behavior, output response and characteristics such as availability, latency and throughput.
  
• Embrace failure: It seems paradoxical initially for software engineers to allow an incident to happen when they are wired to prevent such occurrences. However, disruptions will always occur in IT services and it's better to experience them in a controlled environment to identify the solution preemptively instead of after hours when an organization's team is off the clock or has not encountered that specific problem before.
• Establish steady-state behavior: First, the engineering team must define how the system should behave when running correctly, so it can compare how experiments impact that steady state.
• Identify real-world incidents: Chaos engineering experiments should hew as closely as possible to what might happen on a normal day instead of creating unlikely situations. Network and infrastructure failures, bad code, power issues and traffic overload are potential occurrences.
• Create a game day: Chaos engineering can study the environment on a game day, where multiple tests occur during a specific day to maximize their resources to identify and resolve as many issues as possible. 
• Use automation: Organizations of all sizes can use chaos engineering by automating experiments, which would be too labor intensive if companies manually conducted them. This reduces some of the burden on IT teams during the chaos engineering process. Experiment design, failure injection and infrastructure provisioning are all aspects of experimentation that organizations can automate.
• Be mindful of the blast radius: A chaos engineer must take great pains to minimize the blast radius so that the actual harm to customers is as minor as possible. Some ways to minimize the blast radius are:
  • Target a subset of services: Chaos engineering, especially in a production setting, should not fundamentally disrupt an organization's service. Targeting a specific subset of services can minimize the impact of an incident if it occurs, ensuring that it does not take down the entire system.
  • Run the experiment for a finite time: The experiment should have a beginning and an end. The point of the experiment is to create an incident and resolve it versus letting it run unchecked for a long time.
  • Run the experiment away from peak traffic: Organizations should try to avoid running experiments during peak times unless they are specifically trying to gauge how high capacity affects the system during an incident. 
  • Run the experiment in the development environment: The easiest way to ensure that no customers experience service interruption is to run in the pre-production environment. However, that means that the conditions will differ from the production environment, potentially giving a false picture of what is occurring. To minimize this, ensure that your pre-production and production environments mirror each other as much as possible. 
  • Experiment on every component: Chaos experimentation never ends as an organization's system is continually changing. Another goal should be to test "everything"-examine all components, layers, services-and their dependencies throughout the process.

Organizations that use chaos engineering must decide whether to use chaos testing in their production or pre-production environments. There are several reasons why chaos engineering is most beneficial in production environments.

Live environments provide the most accurate environment for understanding how an incident impacts the customer experience. Another reason is that the pre-production environment might not have the exact settings as the live environment, therefore introducing some variability to the experiments.

For instance, an incident in a pre-production environment might not create a realistic response because it lacks the same traffic levels as the live environment. It also might not have the same security configurations as that environment. 

Some organizations fear intentionally causing issues with their live site, so they run their experiments on their pre-production or developmental site. This ensures that any issues that occur do not impact the live customer experience. To mitigate this, some organizations begin in pre-production environments to get a handle on the process before moving to the live production environment.

Organizations choose which environment to use based on their risk tolerance. Ultimately, chaos engineering aims to test actual large-scale issues, which is why production environments give the most accurate picture of what's happening and what requires fixing.