As enterprises expand their technology portfolios, system complexity increases and their ability to gain insight into those systems decreases. This reduced operational system awareness leads to delayed service restoration, unstable systems, and a less resilient IT operational environment. Observability solutions bridge visibility gaps to provide real-time visibility into environments, including cloud-native and microservices-based architecture, and they enable organizations to take a more proactive approach to operations.1
IBM Instana Observability is a real-time observability platform that monitors and analyzes applications, services, infrastructure, and web and mobile applications with distributed tracing and high-fidelity 1-second metrics. Instana’s ease of deployment and use, and the visibility and data it provides, enable better, faster decision-making for IT operations and development teams. This reduces the time and effort it takes to remediate incidents and ultimately supports teams’ efforts to improve application stability and end-user experiences.
IBM commissioned Forrester Consulting to conduct a Total Economic Impact™ (TEI) study and examine the potential return on investment (ROI) enterprises may realize by deploying Instana Observability.2 The purpose of this study is to provide readers with a framework to evaluate the potential financial impact of Instana Observability on their organizations.
To better understand the benefits, costs, and risks associated with this investment, Forrester interviewed seven representatives at four organizations with experience using Instana Observability. For the purposes of this study, Forrester aggregated the interviewees’ experiences and combined the results into a single composite organization with 10,000 employees and $2 billion in annual revenue.
Interviewees said that prior to using Instana Observability, their organizations’ observability tools were disjointed and didn’t cover their full environment. In some cases, teams didn’t have access to any observability tools due to financial, licensing, or technological constraints. As a result, monitoring was difficult, and identifying and solving issues was time- and resource-intensive.
After the investment in Instana Observability, the interviewees had a single view of their environment, providing better visibility and end-to-end monitoring. Key results from the investment include fewer and less-severe incidents, reduced developer time spent troubleshooting, operational productivity improvements, and cost savings.
Key Findings
Quantified benefits. Three-year, risk-adjusted present value (PV) quantified benefits for the composite organization include:
- Reduced revenue-impacting incidents by up to 60% and mean time to repair by 70%. Instana provides better data capture, translating to better visibility, allowing developers to improve application stability and address problems faster, reducing incident occurrence and shortening the mean time to detect and the mean time to restore for remaining incidents. For customer-facing applications, this means decreased lost revenue caused by incidents. This benefit is worth almost $481,000 to the composite organization over three years.
- Reduced business productivity-impacting incidents by up to 60% and mean time to repair by 70%. Fewer incidents and faster restoration times for internal applications reduce business productivity loss for business employees. This benefit is worth nearly $4.4 million to the composite organization over three years.
- Decreased developer troubleshooting time by up to 90%. Previously, incidents required dozens of individuals to spend hours trying to identify where a problem was before they could start fixing it. Instana helps teams narrow in on the issue faster and with significantly fewer people, saving developers’ time due to traces, end-to-end visibility, and increased granularity of alerting. This benefit is worth more than $442,000 to the composite organization over three years.
- Operational productivity improvements of up to 40%. Site reliability engineer and DevOps teams benefit from a single tool that provides end-to-end monitoring and presents data clearly and consistently. They can more easily create more-effective alerts, and monitoring is more manageable and effective from a single source. They are also able to push more ownership of issues to developer teams. This benefit is worth close to $1.9 million to the composite organization over three years.
- Enabled cost savings of $213,000 per year. The composite organization can retire legacy monitoring tools and identify underutilized resources with Instana, allowing it to save additional money by retiring them. This benefit is worth more than $528,000 to the composite organization over three years.
Unquantified benefits. Benefits that provide value for the composite organization but are not quantified for this study include:
- Ability to be more proactive and focus on other strategic work. With less time and effort spent monitoring and fixing applications, developers and operations teams can focus on proactive improvements to applications and infrastructure.
- Better customer experience. Instana helps developers get ahead of problems, and in some cases, preempt outages through customer outreach.
- Increased employee satisfaction. Support for troubleshooting and the ability to focus on proactive, strategic initiatives provide more engaging work for developers.
Costs. Three-year, risk-adjusted PV costs for the composite organization include:
- License costs of $1.2 million over three years. The composite organization pays licensing costs for its Instana deployment based on the number of hosts.
- Implementation costs of $892,000 over three years. These include the organization’s internal time and effort to: launch the Instana agent during initial deployment as well as additional deployment to new applications each year; build automations; develop policies, processes, and documentation; create and deliver training; and set up user roles and permissions.
- Ongoing management costs of $340,000 over three years. Ongoing management costs include internal time and effort to manage updates; maintain users, policies, and permissions; and provide ongoing training and support to users in the organization.
The representative interviews and financial analysis found that a composite organization experiences benefits of $7.68 million over three years versus costs of $2.41 million, adding up to a net present value (NPV) of $5.27 million and an ROI of 219%.
Key Statistics
-
Return on investment (ROI):
219%
-
Benefits PV:
$7.68M
-
Net present value (NPV):
$5.27M
Benefits (Three-Year)
Reduced revenue-impacting incidents Reduced business productivity-impacting incidents Developer troubleshooting time savings Operational productivity improvements Cost savingsTEI Framework And Methodology
From the information provided in the interviews, Forrester constructed a Total Economic Impact™ framework for those organizations considering an investment Instana Observability.
The objective of the framework is to identify the cost, benefit, flexibility, and risk factors that affect the investment decision. Forrester took a multistep approach to evaluate the impact that Instana Observability can have on an organization.
-
Due Diligence
Interviewed IBM stakeholders and Forrester analysts to gather data relative to Instana Observability.
-
Interviews
Interviewed seven representatives at four organizations using Instana Observability to obtain data about costs, benefits, and risks.
-
Composite Organization
Designed a composite organization based on characteristics of the interviewees’ organizations.
-
Financial Model Framework
Constructed a financial model representative of the interviews using the TEI methodology and risk-adjusted the financial model based on issues and concerns of the interviewees.
-
Case Study
Employed four fundamental elements of TEI in modeling the investment impact: benefits, costs, flexibility, and risks. Given the increasing sophistication of ROI analyses related to IT investments, Forrester’s TEI methodology provides a complete picture of the total economic impact of purchase decisions. Please see Appendix A for additional information on the TEI methodology.
Disclosures
Readers should be aware of the following:
This study is commissioned by IBM and delivered by Forrester Consulting. It is not meant to be used as a competitive analysis.
Forrester makes no assumptions as to the potential ROI that other organizations will receive. Forrester strongly advises that readers use their own estimates within the framework provided in the study to determine the appropriateness of an investment in Instana Observability.
IBM reviewed and provided feedback to Forrester, but Forrester maintains editorial control over the study and its findings and does not accept changes to the study that contradict Forrester’s findings or obscure the meaning of the study.
IBM provided the customer names for the interviews but did not participate in the interviews.
Consulting Team:
Elizabeth Preston
Jonny Cook
Drivers leading to the Instana Observability investment
Interviews
| Role | Industry | Geography | Annual Revenue | Applications | Percentage Of Applications Monitored By Instana |
|---|---|---|---|---|---|
| Product head for observability | Banking | South Africa | $1.5 billion | 285 | 65% |
| Director of observability platform, Platform engineer | Apparel | Global | $23.5 billion | 285 | 35% |
| Senior performance tech lead | Software | Global | $250 million | 30 | 90% |
| Senior IT architect, IT architect, IT architect | Government | Spain | $3 billion | 1,000 | 50% |
Key Challenges
Prior to deploying Instana Observability, interviewees shared that their organizations either didn’t have observability tools in place or used a variety of tools for different purposes.
The interviewees noted how their organizations struggled with common challenges, including:
- Difficulty managing disjointed tools and systems. Interviewees shared that their organizations often deployed various tools to support monitoring needs, but those tools were not deployed consistently across applications or environments, and they didn’t connect to one another. In some cases, the existing tools were too complicated or expensive to deploy broadly, hampering adoption. As a result, significant effort was required to manage disparate tools and processes without corresponding value. The director of observability platform for an apparel company shared, “Every area, every application, every product had its own observability.” The product head for observability at a banking organization said that they used four different monitoring tools. However, they said, “[They were] not capable of monitoring to the extent we required.”
- Lack of visibility into their environment. Whether due to disjointed tools or a complete lack of tools, interviewees shared that their organizations lacked good visibility into their environments. They particularly struggled to monitor their Kubernetes environments. The director of observability platform for an apparel company shared that despite having multiple monitoring tools in place, “The capacity of observability was really almost nothing.” The senior performance tech lead for a software organization explained that because their other solution was difficult to deploy broadly, they had to pick and choose which applications to monitor.
- Time- and labor-intensive processes to identify and resolve issues. Without good visibility into their environments and applications, interviewees’ organizations struggled to identify the cause of incidents and resolve them in a timely manner. The senior performance tech lead for a software organization said: “It [took] a tribe because you didn’t know where the performance problems were because you didn’t have the visibility. It would irk me to no end to get on a call, and there were 20 people who have been on the phone for 6 hours.”
Solution Requirements
The interviewees’ organizations searched for a solution that could:
- Be deployed easily.
- Provide end-to-end visibility across environments including Kubernetes and applications including microservices, as well as the ability to see how services interact with one another.
- Allow them to identify and respond to issues quickly and effectively.
- Enable broad adoption across the organization.
- Remain cost-effective.
Composite Organization
Based on the interviews, Forrester constructed a TEI framework, a composite company, and an ROI analysis that illustrates the areas financially affected. The composite organization is representative of the seven interviewees at four organizations, and it is used to present the aggregate financial analysis in the next section. The composite organization has the following characteristics:
Description of composite. The composite organization is a global company with $2 billion in revenue and 10,000 employees. It has 300 applications, which are a mix of on-premises and cloud-based, monolithic and microservice-based. Some applications are customer-facing, while others are internal.
Deployment characteristics. Prior to deploying Instana Observability, the composite organization uses multiple monitoring tools that do not provide a comprehensive view of its environment and applications. The composite organization initially deploys Instana as a SaaS solution to 50% of its applications. It continues to expand deployment over time, with the goal of eventually covering 80% to 90% of its applications.
Quantified benefit data as applied to the composite
Total Benefits
| Ref. | Benefit | Year 1 | Year 2 | Year 3 | Total | Present Value |
|---|---|---|---|---|---|---|
| Atr | Reduced revenue-impacting incidents | $190,185 | $193,606 | $197,027 | $580,819 | $480,930 |
| Btr | Reduced business productivity-impacting incidents | $1,728,982 | $1,760,083 | $1,791,185 | $5,280,250 | $4,372,160 |
| Ctr | Developer troubleshooting time savings | $167,962 | $178,459 | $188,957 | $535,378 | $442,145 |
| Dtr | Operational productivity improvements | $645,469 | $753,047 | $860,625 | $2,259,141 | $1,855,743 |
| Etr | Cost savings | $212,500 | $212,500 | $212,500 | $637,500 | $528,456 |
| Total benefits (risk-adjusted) | $2,945,097 | $3,097,696 | $3,250,295 | $9,293,087 | $7,679,434 |
Reduced Revenue-Impacting Incidents
Evidence and data. Interviewees shared that without good visibility into their environments, their organizations were reactive when it came to incident response and remediation. Prior to adopting Instana, identifying the cause of an issue was time- and resource-intensive, and data needed to solve the problem wasn’t always readily available. When customer-facing applications experienced incidents, interviewees said their organizations saw revenue impacts and financial penalties. After deploying Instana, organizations were able to detect incidents sooner, troubleshoot and resolve them faster, and ultimately be more proactive in improving application stability.
- The director of observability platform for an apparel organization explained that when a customer-facing application went down, it could cost the organization millions of dollars in revenue, and any reduction in the mean time to detect (MTTD) and resolve (MTTR) would provide a massive financial benefit. After deploying Instana Observability, the company implemented better alerts, and its MTTD decreased by up to 40%.
- The product head for observability for a banking organization shared that about 10% of their applications were consumer-facing, and outages impacting those applications affected revenue. The organization reduced its critically unstable applications for customers by almost half after deploying Instana and saw the number of multi-hour monthly outages decrease a corresponding amount. For the remaining incidents, the MTTR decreased by 50%.
- The senior performance tech lead for a software organization said that most of the firm’s applications were consumer-facing, and there were financial penalties when it experienced incidents that caused it to miss service-level agreements (SLAs). Before Instana, the company experienced incidents causing missed SLAs multiple times per week, but with Instana, these decreased by more than 90%. Instana also enabled the company to identify the cause of the incident in minutes instead of hours.
- An IT architect for a government organization estimated that incident response time decreased by 75% to 80% because Instana made it easy to locate the source of the problem.
Modeling and assumptions. Based on the interviews, Forrester assumes the following about the composite organization:
- The organization experiences 12 incidents per year that impact customers and lead to a loss of revenue.
- Instana Observability enables a 50% reduction in incidents in Year 1, increasing to 60% by Year 3 as the deployment of Instana increases and teams take proactive steps to improve application stability.
- The MTTD incidents is 0.1 hours, and the MTTR is 8 hours before the organization deploys Instana Observability.
- Instana Observability enables the organization to reduce its MTTD for remaining incidents by 30% due to more granular data and the ability to create more sensitive alerting.
- Instana Observability enables the organization to reduce its MTTR for remaining incidents by 70% due to better tracing and visibility.
- The composite organization’s operating margin is 11%.
Risks. An organization’s ability to realize benefits related to reduced revenue-impacting incidents will vary based on a variety of factors, including:
- Current application types, end users, and overall stability.
- How closely tied the organization’s revenue is to its applications.
- Existing monitoring tools and alerts.
- Developer and operations team practices.
- The breadth of Instana deployment, time and effort spent creating alerts, and how teams choose to use Instana to support decision-making.
Results. To account for these risks, Forrester adjusted this benefit downward by 15%, yielding a three-year, risk-adjusted total PV of $481,000.
Reduced Revenue-Impacting Incidents
| Ref. | Metric | Source | Year 1 | Year 2 | Year 3 | |
|---|---|---|---|---|---|---|
| A1 | Annual revenue-impacting incidents before Instana | Composite | 12 | 12 | 12 | |
| A2 | Reduction in incidents due to Instana | Interviews | 50% | 55% | 60% | |
| A3 | Annual revenue-impacting incidents after Instana | A1-A1*A2 | 6.0 | 5.4 | 4.8 | |
| A4 | Average MTTD before Instana (hours) | Composite | 0.10 | 0.10 | 0.10 | |
| A5 | Reduction in MTTD with Instana | Interviews | 30% | 30% | 30% | |
| A6 | Average MTTR before Instana (hours) | Composite | 8 | 8 | 8 | |
| A7 | Reduction in MTTR with Instana | Interviews | 70% | 70% | 70% | |
| A8 | Average revenue cost of an incident | Composite | $200,000 | $200,000 | $200,000 | |
| A9 | Average hourly revenue cost of an incident | A8/(A4+A6) | $24,691 | $24,691 | $24,691 | |
| A10 | Operating margin | Composite | 11% | 11% | 11% | |
| At | Reduced revenue-impacting incidents | (A1*A2*A8+A3*A4*A5*A9+A3*A6*A7* A9)*A10 | $223,747 | $227,772 | $231,797 | |
| Risk adjustment | ↓15% | |||||
| Atr | Reduced revenue-impacting incidents (risk-adjusted) | $190,185 | $193,606 | $197,027 | ||
| Three-year total: $580,819 | Three-year present value: $480,930 | |||||
Reduced Business Productivity-Impacting Incidents
Evidence and data. While some incidents impacted customer-facing applications, many more affected internal applications, impacting employee productivity. Interviewees shared that after deploying Instana Observability, internal applications saw similar benefits to customer-facing applications with regard to fewer incidents, faster MTTD and MTTR, and improved application stability.
- The product head for observability for a banking organization shared that 90% of their applications are internal rather than consumer-facing applications. The organization experienced a 50% decrease in unstable applications and a corresponding reduction in outages. It was also able to decrease its resolution time after deploying Instana. The product head explained that with Instana, “[W]e get more accurate alerting … [and] we have a better understanding of underlying architecture within [our environment].”
- An IT architect for a government organization explained that while constituents used some of their applications, a large number were used by civil servants within the government. They said, “The stability of applications has improved a lot, and [so has] the performance because this was a measure that our teams didn’t have before Instana.”
Modeling and assumptions. Based on the interviews, Forrester assumes the following about the composite organization:
- The organization experiences 48 incidents per year that impact employee productivity.
- Instana Observability enables a 50% reduction in incidents in Year 1, increasing to 60% by Year 3 as the deployment of Instana increases and teams take proactive steps to improve application stability.
- The MTTD for incidents is 0.1 hours, and the MTTR is 8 hours before the organization deploys Instana Observability.
- Instana Observability enables the organization to reduce its MTTD for remaining incidents by 30% due to more granular data and the ability to create more sensitive alerting.
- Instana Observability enables the organization to reduce its MTTR for remaining incidents by 70% due to better tracing and visibility.
- Forrester assumes that not all recovered time will be used for productive or value-added activities, so a 50% productivity recapture rate is applied.
Risks. An organization’s ability to realize benefits related to reduced business productivity-impacting incidents will vary based on a variety of factors, including:
- Current application types, end users, and overall stability.
- How closely tied application availability is to employee productivity.
- Existing monitoring tools and alerts.
- Developer and operations team practices.
- The breadth of Instana deployment, time and effort spent creating alerts, and how teams choose to use Instana to support decision-making.
Results. To account for these risks, Forrester adjusted this benefit downward by 15%, yielding a three-year, risk-adjusted total PV (discounted at 10%) of $4.4 million.
Reduced Business Productivity-Impacting Incidents
| Ref. | Metric | Source | Year 1 | Year 2 | Year 3 | |
|---|---|---|---|---|---|---|
| B1 | Annual productivity-impacting incidents before Instana | Composite | 48 | 48 | 48 | |
| B2 | Reduction in incidents due to Instana | Interviews | 50% | 55% | 60% | |
| B3 | Annual productivity-impacting incidents after Instana | Composite | 24.0 | 21.6 | 19.2 | |
| B4 | Average MTTD before Instana (hours) | Composite | 0.10 | 0.10 | 0.10 | |
| B5 | Reduction in MTTD with Instana | Interviews | 30% | 30% | 30% | |
| B6 | Average MTTR before Instana (hours) | Composite | 8 | 8 | 8 | |
| B7 | Reduction in MTTR with Instana | Interviews | 70% | 70% | 70% | |
| B8 | Average productivity cost of an incident | Composite | $100,000 | $100,000 | $100,000 | |
| B9 | Average hourly productivity cost of an incident | B8/(B4+B6) | $12,346 | $12,346 | $12,346 | |
| B10 | Productivity recapture | TEI standard | 50% | 50% | 50% | |
| Bt | Reduced business productivity-impacting incidents | (B1*B2*B8+B3*B4* B5*B9+B3*B6*B7* B9)*B10 | $2,034,096 | $2,070,686 | $2,107,277 | |
| Risk adjustment | ↓15% | |||||
| Btr | Reduced business productivity-impacting incidents (risk-adjusted) | $1,728,982 | $1,760,083 | $1,791,185 | ||
| Three-year total: $5,280,250 | Three-year present value: $4,372,160 | |||||
Developer Troubleshooting Time Savings
Evidence and data. Prior to deploying Instana, identifying the cause of an incident and which team needed to resolve it could take many hours and dozens of individuals. Instana provided the ability to pinpoint the problem quickly so the right people could get to work, and the previous all-hands-on-deck scenarios became the exception rather than the rule.
- The senior performance tech lead for a software organization explained: “It used to be a support nightmare — just troubleshooting it took a long time — and with IBM Instana in place, we can quickly jump in and find the root problem and give us the ability to quickly troubleshoot it.” They shared that before deploying Instana, they could have 20 people on a call for every incident, spanning developers, DevOps, and infrastructure teams, and that each call could last 6 or more hours while they tried to determine the cause of an incident. After deploying Instana, they typically had two or three people on a call, and they could identify the problem in minutes.
- A senior IT architect for a government organization said: “Before Instana, many of us, especially developers, were blind because it [was] really complicated to see or to know all the architecture that we have. … So when one of these applications had an issue, it was really complicated to know [where it was]. … [Now], you can share all of this information with our developers to solve these issues.” They noted that multiple times a year, they experienced incidents requiring dozens of individuals to spend 8 or more hours trying to identify the cause of an incident and resolve it. With Instana, they needed 80% fewer people and could resolve the incident much faster.
- The product head for observability for a banking organization shared: “Now we have a lot more focus. Now we know — the application team is you and the downstream is you. We get two teams in to start speaking. So getting the correct people to speak to one another is happening a lot faster.”
Modeling and assumptions. Based on the interviews, Forrester assumes the following about the composite organization:
- Before deploying Instana Observability, the composite organization experiences 60 incidents per year that require developer troubleshooting.
- Each incident requires 20 developers to spend 6 hours of troubleshooting.
- Instana Observability enables the organization to reduce troubleshooting time by 80% in Year 1, increasing to 90% by Year 3 due to increased environment stability, expanded environment coverage, and increased automation.
- Instana Observability enables the organization to decrease the number of developers required for troubleshooting by 80% due to improved visibility.
- Forrester assumes that not all recovered time will be used for productive or value-added activities, so a 50% productivity recapture rate is applied.
- The average fully burdened hourly salary for a developer is $81.
Risks. An organization’s ability to realize benefits related to developer troubleshooting time savings will vary based on a variety of factors, including:
- Current application types and overall stability.
- Existing monitoring tools and alerts.
- Developer and operations team troubleshooting practices.
- The breadth of Instana deployment, time and effort spent creating alerts, and how teams choose to use Instana to support decision-making.
Results. To account for these risks, Forrester adjusted this benefit downward by 10%, yielding a three-year, risk-adjusted total PV (discounted at 10%) of $442,000.
Developer Troubleshooting Time Savings
| Ref. | Metric | Source | Year 1 | Year 2 | Year 3 | |
|---|---|---|---|---|---|---|
| C1 | Annual incidents before Instana | Composite | 60 | 60 | 60 | |
| C2 | Time spent troubleshooting before Instana (hours) | Composite | 6 | 6 | 6 | |
| C3 | Developers involved in troubleshooting before Instana | Composite | 20 | 20 | 20 | |
| C4 | Reduction in time spent troubleshooting with Instana | Interviews | 80% | 85% | 90% | |
| C5 | Reduction in developers required from troubleshooting with Instana | Interviews | 80% | 80% | 80% | |
| C6 | Productivity recapture | TEI standard | 50% | 50% | 50% | |
| C7 | Average fully burdened hourly salary | Composite | $81 | $81 | $81 | |
| Ct | Developer troubleshooting time savings | C1*C2*C3*C4*C5* C6*C7 | $186,624 | $198,288 | $209,952 | |
| Risk adjustment | ↓10% | |||||
| Ctr | Developer troubleshooting time savings (risk-adjusted) | $167,962 | $178,459 | $188,957 | ||
| Three-year total: $535,378 | Three-year present value: $442,145 | |||||
Operational Productivity Improvements
Evidence and data. Interviewees shared that deploying Instana improved application stability and performance, improved monitoring and alerting capabilities, and allowed developers to take more ownership of application performance. As a result, site reliability engineering (SRE) and DevOps teams saw productivity gains and were able to focus on more strategic and proactive initiatives at their organizations.
- The director of observability platform for an apparel organization said: “[Our SREs] have end-to-end monitoring, so they can see the full trace from the beginning to the end. … They also have the metrics for different kinds of infrastructure in a single point, so they can create more complex alerts that can generate incidents and be more proactive. … [Having] a single application … makes them more efficient.” They estimated that their SREs saw a 30% productivity improvement from deploying Instana Observability.
- The same director also shared: “One important point for the support team is that [Instana’s] way of presenting the data … it’s quite clear and quite consistent. They can go from the whole application to the different endpoints, seeing the same information with more granularity. So I think that this is something that can help a lot, seeing in a consistent way the percentage of errors, the percentage of failed calls, this kind of thing in a standard way, [and] you don’t have to create anything — it’s just supported by default.”
- A senior IT architect for a government organization said, “We are more productive, and we spend less time to detect problems.” They estimated their support team was 30% to 40% more productive after deploying Instana Observability. As a result, they could focus on more proactive work for their organization.
Modeling and assumptions. Based on the interviews, Forrester assumes the following about the composite organization:
- The composite organization has 30 SRE and DevOps employees.
- Those employees experience a 30% productivity improvement in Year 1, increasing to 40% by Year 3 as the organization’s deployment of Instana Observability increases.
- Forrester assumes that not all recovered time will be used for productive or value-added activities, so a 50% productivity recapture rate is applied.
- The average fully burdened annual salary for SREs and DevOps employees is $168,750.
Risks. An organization’s ability to realize benefits related to operational productivity improvements will vary based on a variety of factors, including:
- Current application types and overall stability.
- Existing monitoring tools and alerts.
- The breadth of Instana deployment, time and effort spent creating alerts, and how teams choose to use Instana to support decision-making.
Results. To account for these risks, Forrester adjusted this benefit downward by 15%, yielding a three-year, risk-adjusted total PV (discounted at 10%) of $1.9 million.
Operational Productivity Improvements
| Ref. | Metric | Source | Year 1 | Year 2 | Year 3 | |
|---|---|---|---|---|---|---|
| D1 | SRE and DevOps employees | Composite | 30 | 30 | 30 | |
| D2 | Productivity improvement due to Instana | Interviews | 30% | 35% | 40% | |
| D3 | Productivity recapture | TEI standard | 50% | 50% | 50% | |
| D4 | Average blended fully burdened annual salary for SRE and DevOps employees | Composite | $168,750 | $168,750 | $168,750 | |
| Dt | Operational productivity improvements | D1*D2*D3*D4 | $759,375 | $885,938 | $1,012,500 | |
| Risk adjustment | ↓15% | |||||
| Dtr | Operational productivity improvements (risk-adjusted) | $645,469 | $753,047 | $860,625 | ||
| Three-year total: $2,259,141 | Three-year present value: $1,855,743 | |||||
Cost Savings
Evidence and data. Interviewees whose organizations were previously using other monitoring tools were able to reduce usage or retire those solutions entirely as they deployed Instana Observability and adoption increased. In some cases, the improved visibility provided by Instana allowed organizations to identify potentially underutilized or overallocated resources, monitor the impact of reducing them, and confidently retire them.
- The product head for observability for a banking organization shared that they scaled back their other monitoring tools after deploying Instana, leading to annual cost savings.
- The director of observability platform for an apparel organization said they retired their previous solution after deploying Instana.
- The senior performance tech lead for a software organization described how Instana helped them identify underutilized resources, including Kubernetes pods, in their environment. They were able to retire and consolidate resources and monitor the performance to ensure the changes didn’t cause any issues. They said, “With IBM Instana, being able to see the complete picture, it gave us the confidence to be able to make those changes and see the impact and be able to track it.”
Modeling and assumptions. Based on the interviews, Forrester assumes the following about the composite organization:
- The organization retires $100,000 per year in legacy monitoring tools replaced by Instana Observability.
- Instana Observability helps the organization identify underutilized resources costing $150,000, which can be retired.
Risks. An organization’s realization of benefits related to cost savings will depend upon a variety of factors, including:
- The previously deployed monitoring tools that Instana Observability can replace.
- Other systems and technologies deployed at the organization, their utilization rate, and their ability to retire any underutilized resources.
- The adoption rate of Instana Observability within the organization.
Results. To account for these risks, Forrester adjusted this benefit downward by 15%, yielding a three-year, risk-adjusted total PV of $528,000.
Cost Savings
| Ref. | Metric | Source | Year 1 | Year 2 | Year 3 | |
|---|---|---|---|---|---|---|
| E1 | Retired legacy solutions | Composite | $100,000 | $100,000 | $100,000 | |
| E2 | Retired underutilized technologies identified by Instana | Composite | $150,000 | $150,000 | $150,000 | |
| Et | Cost savings | E1+E2 | $250,000 | $250,000 | $250,000 | |
| Risk adjustment | ↓15% | |||||
| Etr | Cost savings (risk-adjusted) | $212,500 | $212,500 | $212,500 | ||
| Three-year total: $637,500 | Three-year present value: $528,456 | |||||
Unquantified Benefits
Interviewees mentioned the following additional benefits that their organizations experienced but were not able to quantify:
-
Ability to be more proactive and focus on strategic work. Instana provides better, more granular data and improved visibility, enabling teams to improve application stability and remediate issues faster. With more time and better tools, teams could look for new ways to improve products and services for customers rather than simply fixing things as they broke.
- A senior IT architect for a government organization said, “Now we can improve some of the more important services … like authentication, authorization, and other infrastructure and architecture.”
- A senior performance tech lead at a software organization explained: “What’s changed developers’ lives is the ability to see performance issues. … There are things you can always make better, and that’s one of the things we’ve been able to do with Instana — the whole proactive part is to go fishing for problems and find [problem] areas we hadn’t thought about.”
-
Better customer experience. Interviewees said their organizations were able to proactively reach out to customers to alert them to potential issues and, in some cases, prevent issues from occurring.
- The product head for observability for a banking organization said: “It is definitely a better client experience. We can preempt the outages, phone them, and say, ‘Hey guys, we see you’ve got a problem with your device; you are on the incorrect version. We require you to update, please.’ Instead of waiting for them to come to us, we go to them.”
- A senior performance tech lead at a software organization noted that customer feedback was shifting from performance-related issues to other types of concerns, which they saw as an indication of improved customer experience with the applications.
- Increased employee satisfaction. The ability to focus on more proactive, strategic work made developers’ jobs more engaging for them, according to interviewees. A senior performance tech lead for a software organization said: “The whole ability to go fishing and find interesting problems to work on and get visibility to see different ways applications behave — it’s just made it more interesting. You learn more insights about behaviors of applications you didn’t realize were there.” They have found that since deploying Instana, their developer turnover has decreased substantially, which they partially attribute to how much easier Instana has made the support side of their work.
Flexibility
The value of flexibility is unique to each customer. There are multiple scenarios in which a customer might implement Instana Observability and later realize additional uses and business opportunities, including:
- Support for product innovation. The product head for observability at a banking organization shared that they’ve used data from monitoring the end-user experience on mobile and web apps to create new applications and products for their customers, allowing them to reach new customers and drive revenue growth.
Flexibility would also be quantified when evaluated as part of a specific project (described in more detail in Appendix A).
Quantified cost data as applied to the composite
Total Costs
| Ref. | Cost | Initial | Year 1 | Year 2 | Year 3 | Total | Present Value |
|---|---|---|---|---|---|---|---|
| Ftr | License costs | $0 | $393,750 | $472,500 | $567,000 | $1,433,250 | $1,174,446 |
| Gtr | Implementation | $558,900 | $111,780 | $134,136 | $160,218 | $965,034 | $891,749 |
| Htr | Ongoing management | $0 | $125,654 | $125,654 | $162,779 | $414,087 | $340,376 |
| Total costs (risk-adjusted) | $558,900 | $631,184 | $732,290 | $889,997 | $2,812,371 | $2,406,571 |
License Costs
Evidence and data. Interviewees’ organizations paid Instana licensing costs based on the number of hosts or servers managed.
- All interviewees had a SaaS deployment of IBM Instana Observability.
- The product head for observability for a banking organization shared that they pay per server for infrastructure and application monitoring licenses.
- The director of observability platform for an apparel organization explained that their Instana Observability license costs for their Kubernetes deployment were effectively paid per node.
- A senior performance tech lead for a software organization shared that the licensing model made it easy and affordable to onboard more users compared to their previous solution.
Modeling and assumptions. Forrester modeled this cost based on the following:
- The composite organization has a SaaS deployment.
- The composite organization pays for 500 hosts in Year 1, increasing to 720 hosts by Year 3 as its adoption rate of Instana Observability increases.
- License costs for the composite organization are $750 per host.
- Pricing may vary. Contact IBM for additional details.
Risks. An organization’s experience of costs associated with licensing for IBM Instana Observability will depend upon a variety of factors, including:
- The type of deployment.
- The number of hosts.
- Add-ons, discounts, and local pricing.
Results. To account for these risks, Forrester adjusted this cost upward by 5%, yielding a three-year, risk-adjusted total PV (discounted at 10%) of $1.2 million.
License Costs
| Ref. | Metric | Source | Initial | Year 1 | Year 2 | Year 3 | |
|---|---|---|---|---|---|---|---|
| F1 | License cost per host | Composite | $0 | $750 | $750 | $750 | |
| F2 | Hosts | Composite | 0 | 500 | 600 | 720 | |
| Ft | License costs | F1*F2 | $0 | $375,000 | $450,000 | $540,000 | |
| Risk adjustment | ↑5% | ||||||
| Ftr | License costs (risk-adjusted) | $0 | $393,750 | $472,500 | $567,000 | ||
| Three-year total: $1,433,250 | Three-year present value: $1,174,446 | ||||||
Implementation
Evidence and data. Interviewees shared that deploying and setting up Instana at their organizations was a multistep process. For most applications, they could simply deploy the agent in their environments. For a small percentage of applications, additional rearchitecting of applications or other work was needed for the applications to work with Instana. Once Instana was deployed, teams generally created automations to support ongoing deployments, configurations, or updates. They also needed to create policies, processes, and documentation around the use of Instana, set up user roles and permissions, and train developers on the solution.
- The product head of observability at a banking organization shared that their initial implementation happened over six months, including creating automations. Their organization required new security checkpoints and management processes to be created as part of the implementation. They spent 40 to 45 hours per application on setup, deployment, service-level objective (SLO) development, and corresponding alert creation as well as organizational training. Training included creating a “welcome pack” for developers with training videos, IBM resources and documentation, and an overview of Instana.
- A senior IT architect for a government agency said that for most of their applications, initial deployment only took a few hours. For the remainder of their applications, they needed to reconfigure them and install new libraries. Additional time was spent on user setup and management, training, and technical troubleshooting. Training included watching tutorials and reading documentation provided by IBM. The total time spent was about a week per application.
- The head of observability platform for an apparel organization said four people spent a few weeks creating a process to automate Instana deployment for Kubernetes clusters. Non-Kubernetes deployments followed a different, manual process that required ongoing work.
Modeling and assumptions. Based on the interviews, Forrester assumes the following about the composite organization:
- The composite organization has a SaaS deployment.
- It initially deploys Instana to 150 applications across a mix of cloud and on-premises environments and different application types. During Year 1, it deploys Instana to an additional 30 applications. In Year 2, it deploys Instana to an additional 36 applications. In Year 3, it deploys Instana to another 43 applications.
- Deployment, configuration, and training take an average of 40 hours per application.
- The average fully burdened hourly salary for the operations employees supporting the deployment is $81.
Risks. An organization’s realization of implementation costs will depend upon a variety of factors, including:
- The number and type of applications to which Instana Observability is deployed.
- The number and types of users.
- Organizational needs and requirements around security and access.
Results. To account for these risks, Forrester adjusted this cost upward by 15%, yielding a three-year, risk-adjusted total PV (discounted at 10%) of $892,000.
Implementation
| Ref. | Metric | Source | Initial | Year 1 | Year 2 | Year 3 | |
|---|---|---|---|---|---|---|---|
| G1 | Applications | Composite | 150 | 30 | 36 | 43 | |
| G2 | Deployment, configuration, and training per application (hours) | Interviews | 40 | 40 | 40 | 40 | |
| G3 | Average fully burdened hourly salary for employees supporting deployment | TEI standard | $81 | $81 | $81 | $81 | |
| Gt | Implementation | G1*G2*G3 | $486,000 | $97,200 | $116,640 | $139,320 | |
| Risk adjustment | ↑15% | ||||||
| Gtr | Implementation (risk-adjusted) | $558,900 | $111,780 | $134,136 | $160,218 | ||
| Three-year total: $965,034 | Three-year present value: $891,749 | ||||||
Ongoing Management
Evidence and data. Ongoing maintenance for Instana Observability includes managing updates;;maintaining users, policies, and permissions, and continuing training and support for users.
- The product head for observability for a banking organization said that their team of five dedicates about 20% of their total time to ongoing management of Instana Observability, including managing access, roles, and licenses; creating automations; managing updates; and training and supporting the organization in its use and increasing adoption of Instana.
- A senior IT architect for a government organization explained that two people dedicate 10% of their time to managing Instana Observability for their organization, including maintaining users and groups.
Modeling and assumptions. Based on the interviews, Forrester assumes the following about the composite organization:
- The composite organization has a SaaS deployment of Instana Observability.
- Three operations employees dedicate 20% of their time to updates; ongoing training and support; and maintaining users, policies, and permissions related to Instana Observability.
- As new developers join the organization due to natural churn, they receive training on Instana Observability. This training averages 160 hours per year for the organization.
Risks. An organization’s realization of implementation costs will depend upon a variety of factors, including:
- The type and scale of deployment of Instana Observability.
- Organizational policies and requirements around deployment, access, and management.
- Organizational use cases and training requirements.
- Developer and other user churn and ongoing support needs.
Results. To account for these risks, Forrester adjusted this cost upward by 10%, yielding a three-year, risk-adjusted total PV (discounted at 10%) of $340,000.
Ongoing Management
| Ref. | Metric | Source | Initial | Year 1 | Year 2 | Year 3 | |
|---|---|---|---|---|---|---|---|
| H1 | Resources dedicated to management, governance, and training | Composite | 3 | 3 | 4 | ||
| H2 | Percentage of time dedicated | Interviews | 20% | 20% | 20% | ||
| H3 | Training hours for developer churn | Composite | 160 | 160 | 160 | ||
| H4 | Average fully burdened annual salary for resources dedicated to management, governance, and training | TEI standard | $168,750 | $168,750 | $168,750 | ||
| Ht | Ongoing management | H1*H2*H4+H3*H4/ 2,080 | $0 | $114,231 | $114,231 | $147,981 | |
| Risk adjustment | ↑10% | ||||||
| Htr | Ongoing management (risk-adjusted) | $0 | $125,654 | $125,654 | $162,779 | ||
| Three-year total: $414,087 | Three-year present value: $340,376 | ||||||
Consolidated Three-Year Risk-Adjusted Metrics
Cash Flow Chart (Risk-Adjusted)
Total costs Total benefits Cumulative net benefits-
The financial results calculated in the Benefits and Costs sections can be used to determine the ROI and NPV for the composite organization’s investment. Forrester assumes a yearly discount rate of 10% for this analysis.
These risk-adjusted ROI and NPV values are determined by applying risk-adjustment factors to the unadjusted results in each Benefit and Cost section.
Cash Flow Analysis (Risk-Adjusted Estimates)
| Initial | Year 1 | Year 2 | Year 3 | Total | Present Value | |
|---|---|---|---|---|---|---|
| Total costs | ($558,900) | ($631,184) | ($732,290) | ($889,997) | ($2,812,371) | ($2,406,571) |
| Total benefits | $0 | $2,945,097 | $3,097,696 | $3,250,295 | $9,293,087 | $7,679,434 |
| Net benefits | ($558,900) | $2,313,913 | $2,365,406 | $2,360,298 | $6,480,716 | $5,272,863 |
| ROI | 219% |
Appendix A: Total Economic Impact
Total Economic Impact is a methodology developed by Forrester Research that enhances a company’s technology decision-making processes and assists vendors in communicating the value proposition of their products and services to clients. The TEI methodology helps companies demonstrate, justify, and realize the tangible value of IT initiatives to both senior management and other key business stakeholders.
Total Economic Impact Approach
-
Benefits represent the value delivered to the business by the product. The TEI methodology places equal weight on the measure of benefits and the measure of costs, allowing for a full examination of the effect of the technology on the entire organization.
-
Costs consider all expenses necessary to deliver the proposed value, or benefits, of the product. The cost category within TEI captures incremental costs over the existing environment for ongoing costs associated with the solution.
-
Flexibility represents the strategic value that can be obtained for some future additional investment building on top of the initial investment already made. Having the ability to capture that benefit has a PV that can be estimated.
-
Risks measure the uncertainty of benefit and cost estimates given: 1) the likelihood that estimates will meet original projections and 2) the likelihood that estimates will be tracked over time. TEI risk factors are based on “triangular distribution.”
The initial investment column contains costs incurred at “time 0” or at the beginning of Year 1 that are not discounted. All other cash flows are discounted using the discount rate at the end of the year. PV calculations are calculated for each total cost and benefit estimate. NPV calculations in the summary tables are the sum of the initial investment and the discounted cash flows in each year. Sums and present value calculations of the Total Benefits, Total Costs, and Cash Flow tables may not exactly add up, as some rounding may occur.
Appendix C: Endnotes
1 Source: “The Forrester Observability Reference Architecture: Putting It Into Practice,” Forrester Research, Inc., October 21, 2022.
2 Total Economic Impact is a methodology developed by Forrester Research that enhances a company’s technology decision-making processes and assists vendors in communicating the value proposition of their products and services to clients. The TEI methodology helps companies demonstrate, justify, and realize the tangible value of IT initiatives to both senior management and other key business stakeholders.
PDF
