airoi.org

AIROI - Artificial Intelligence Return on Invest
The AI strategy for decision-makers and managers

Business excellence for decision-makers & managers by and with Sanjay Sauldie

AIROI - Artificial Intelligence Return on Invest: The AI strategy for decision-makers and managers

Start » Testing AI tools: How decision-makers choose the winners

21 May 2026

Testing AI tools: How decision-makers choose the winners

Automation Big data and smart data Digital transformation Artificial intelligence AIROI Step 3: Big Data and Smart Data

Why do seven out of ten companies fail to implement intelligent software solutions despite having conducted extensive evaluations beforehand?

This question concerns executives in almost all sectors of the economy, because the selection of the right technological tools decisively determines the future success of the company. Testing AI tools It’s no longer just about technical specifications or price comparisons; it’s about a holistic evaluation process that must take into account strategic, operational, and cultural dimensions alike. The complexity of these decision-making processes overwhelms many organizations because they approach modern challenges with outdated evaluation methods and overlook essential success factors. In this article, you’ll learn which systematic approaches have proven effective in practice and how you, as decision-makers, can identify the winners among the diverse solutions.

Understand the fundamentals of systematic technology assessment

Before decision-makers even begin the evaluation process, they must first define their own requirements precisely and understand the specific challenges of their organization. Many companies begin their search for suitable solutions with vague ideas and unclear objectives, which inevitably leads to misjudgments. The definition of success criteria forms the foundation of any serious technology evaluation, because only those who know what they want to achieve can assess whether a tool supports these goals. A medium-sized company from the manufacturing sector, for example, initially sought a general automation solution before realizing that the real challenge lay in quality control. A service company with several hundred employees found during the evaluation process that it was not the technology itself but the lack of data base that posed the real obstacle. Another example comes from a retail company that wanted to optimize its customer service and realized that internal processes had to be standardized first before technological solutions could be effectively implemented.

Why traditional selection methods fail to test AI tools

Conventional procurement processes based on tenders, checklists, and price comparisons often fall short when evaluating intelligent systems. These established methods were developed for standardized products whose functionality is clearly defined and whose performance is predictable. Intelligent software solutions, on the other hand, learn, adapt, and further develop their capabilities in use, which is why static evaluation criteria cannot capture their actual performance. A financial services provider evaluated several providers using an extensive catalog of criteria and chose the supposedly best solution only to find that it performed significantly worse under real conditions than in the test environment. One logistics company experienced the opposite when, after its solution was initially ranked second best, it exceeded all expectations after being able to adapt it better to the company’s specific data structures. A third example concerns an energy provider, which realized that the cheaper solution was the most expensive in the long run because hidden costs for adjustments and training had not been taken into account.

Testing practical methods for effective AI tools

Developing a meaningful pilot project represents the crucial step to test the suitability of a technological solution under realistic conditions. The pilot project should be designed in such a way that it is representative of typical use cases, but also takes into account boundary cases and exceptional situations. The duration of such testing phases is underestimated by many companies, because they do not take into account that intelligent systems require time to learn from the available data and optimize their performance. For example, a healthcare provider conducted a six-month pilot phase during which the solution continuously delivered better results and only after four months did it show its full potential. An insurance company implemented parallel pilot projects with three different providers to enable direct comparisons under identical conditions. An industrial company developed a staged evaluation process in which basic technical functions were tested first, before more complex application scenarios were added.

Best practice with a AIROI customer

An internationally operating company in the machine and plant engineering sector faced the challenge of optimizing its maintenance processes and significantly reducing the downtime of its production facilities. The company had already evaluated several solutions, but was dissatisfied with the superficial demonstrations and marketing presentations of the providers, as they did not provide any reliable statements about the actual performance under real production conditions. As part of a transruptive coaching process, the project team developed together with the management team a structured evaluation framework that included both quantitative and qualitative criteria. The inclusion of employees from various departments, who were able to bring their practical experiences and concerns to the table, proved particularly valuable. The company defined specific performance metrics against which the solutions were evaluated, and conducted realistic test scenarios that simulated typical incidents and emergency situations. After a three-month evaluation process, the company was able to make a well-informed decision that was supported by all stakeholders. After full implementation, the chosen solution achieved a reduction in unplanned downtime of more than forty percent, which amortized the investment within the first operating year.

The role of data quality and data integration

No system, no matter how powerful, can deliver convincing results if the underlying data is incomplete, inaccurate, or poorly structured. Decision-makers often underestimate the effort required to prepare and integrate existing data sets before a new solution can be put into productive use. The evaluation of data quality should therefore be an integral part of any evaluation process, because it significantly determines the later project success. A telecommunications company had to postpone its original timeline by several months because cleaning up historical customer data was significantly more complex than expected. A retailer chose a solution that did not use the most modern algorithms, but was better compatible with the existing data formats and required less customization. A pharmaceutical company invested six months in improving its data infrastructure before actually selecting the technology, which proved to be a crucial success factor for the subsequent implementation [1].

Testing decision criteria beyond technical specifications for AI tools

Long-term collaboration with a technology provider requires more than just a powerful product, as factors such as corporate culture, support quality, and strategic alignment also strongly influence project success. Decision-makers should therefore also assess the stability and future viability of potential partners to minimize the risk that a provider will shut down its business or fundamentally change its product strategy. The ability of a provider to respond to changing requirements and further develop its solution represents a significant value factor that is often overlooked in traditional evaluation models. A media company deliberately chose a smaller provider because it promised closer partnerships and faster response times when adapting to changing requirements. One construction company, however, chose an established market leader because the long-term availability of support and updates in the risk-averse industry was considered more valuable than innovative features. One tourism company explicitly considered the references and experiences of other customers from related industries to learn from their experiences [2].

Involving employees as key success factors

The acceptance by later users is crucial in determining whether a technological solution can achieve its potential or fail due to resistance. Many organizations make the mistake of making technology decisions solely at the senior management level, without involving the perspectives and concerns of those who will be working with the solution on a daily basis. The early involvement of employees from different levels and departments not only improves the quality of the decision, but also promotes later acceptance and shortens the implementation phase. A management company organized workshops where administrative staff could articulate their requirements and concerns, which led to a significantly better solution than initially planned. A transportation company trained a group of employees as internal multipliers who supported their colleagues during implementation and served as the first point of contact for questions. A real estate company conducted anonymous surveys to obtain honest feedback on the tested solutions, without employees having to fear any disadvantages [3].

Evaluate costs and benefits realistically

The economic evaluation of technological investments requires a comprehensive view of direct and indirect costs as well as quantifiable and qualitative benefits factors. Many companies focus too much on the obvious acquisition costs and overlook hidden costs for implementation, training, maintenance, and continuous optimization. A realistic economic assessment must also take into account opportunity costs, that is, the benefits that are missed when a decision is delayed or a suboptimal solution is chosen. A chemical company developed a detailed total cost of ownership model that depicted all cost categories over a five-year period and put the decision on a solid economic footing. A textile company also considered soft factors such as employee satisfaction and employer attractiveness in its benefit assessment, which had a positive impact on personnel costs in the medium term. A food manufacturer calculated the value of improved compliance processes that reduced the risk of fines and reputational damage.

Best practice with a AIROI customer

A medium-sized company in the professional services sector approached us because, despite several attempts, it had not found a satisfactory solution for automating its document processes. Previous evaluation efforts had failed because the company had not defined clear success criteria and the various departments formulated different, sometimes conflicting, requirements. As part of the transruptive coaching support, we first conducted a comprehensive stakeholder analysis to capture all relevant perspectives and make objective conflicts transparent. Subsequently, the project team developed a prioritized list of requirements that served as the basis for the structured evaluation. Particularly important was the distinction between essential core functions and desirable additional features, because this differentiation significantly simplified the subsequent decision-making process. The company conducted six-week pilot projects with the three most promising providers, whose results were compared with the previously defined metrics. The employees also evaluated the user friendliness and the quality of the provider’s support, which was included in the overall evaluation. At the end of the process, the company was able to make a decision that was both economically sound and accepted by all parties involved, and the implemented solution achieved a time savings of an average of fifteen percent per employee within the first year.

Risk management and scenario planning

Every technology decision is associated with uncertainties, which is why structured risk management should be an integral part of the selection process. Decision-makers should run through various scenarios and assess how the respective solution will perform under changed conditions, whether it be in terms of business growth, market changes, or technological developments. The flexibility of a solution to adapt to changing requirements constitutes an essential risk buffer that can be more valuable in the long term than short-term cost advantages. An automotive supplier explicitly evaluated the scalability of the solutions under review because it was foreseeable that the data volume would increase sharply in the coming years. A consulting firm defined exit strategies for the case where the chosen solution would not deliver the expected results and negotiated corresponding contractual clauses. An energy company conducted a risk analysis that also considered regulatory changes and their potential impact on the usability of the evaluated solutions [4].

My AIROI Analysis

The experiences from numerous projects clearly show that success in selecting intelligent software solutions depends less on the technical superiority of a particular product than on the quality of the decision-making process itself. Organizations that take a structured approach, clearly define their requirements and involve all relevant stakeholders make better decisions and implement them more successfully than companies that rely on superficial comparisons or intuitive assessments. The guidance provided by experienced partners such as transruptions coaching can help identify blind spots, introduce proven methods, and avoid typical mistakes that many organizations make during their first technology projects. What seems particularly important to me is the realization that there is no one best solution; rather, it is the fit between an organization’s specific requirements and the strengths of a technology that determines the outcome. Decision-makers should not be swayed by marketing promises or industry trends, but rather know their own priorities and pursue them consistently. The investment in a thorough evaluation process pays off in the long run, because misjudgments in this area not only cause financial losses but can also undermine employee motivation and the competitiveness of the company. Clients who have adopted the AIROI-approach often report that not only do they make better technology decisions, but they have also been able to sustainably improve their internal processes and their collaboration culture.