Don’t Use a Ruler to Measure Wind Speed: Establishing a Standard for Competitive Solutions Testing
None
<p>Competitive testing is a business-critical function for financial institutions seeking the ideal solutions provider to help optimize their risk management strategies. Don’t get seduced by inflated test results or flowery marketing claims, however.</p><p>Selecting the right risk solutions could be one of the most important tasks your business ever undertakes – and one of the costliest if it goes wrong. For the best chance of success, competitive testing grounded in scientific analysis is vital.</p><h3><strong>What is Competitive Testing?</strong></h3><p>Competitive testing is a careful evaluation of prospective solutions’ performance uninfluenced by dubious promises and conducted thoughtfully, in a way that enables fair comparisons with current system outputs and real-world outcomes.</p><p>It’s easy to overlook the true motivations of the solutions providers pitching for your business. Some are great at winning business – and will make a big song and dance about it – but the true value of competitive testing lies in establishing a meaningful and lasting partnership, as many of our returning clients will attest.</p><p>Here I’ve outlined several key insights based on extensive experience with similar processes in the hope that it helps others to build an optimal approach and perhaps goes some way to establishing an industry standard practice for this important process.</p><h3><strong>Key Insight #1: Avoid Only Looking at High-Level Metrics Without Context</strong></h3><p>In competitive testing, risk managers should use specific key performance indicators (KPIs) closely aligned with their business needs, not generic metrics like Kolmogorov-Smirnov Statistics (KS). High-level metrics like KS often require contextual interpretation to avoid suboptimal decisions. A higher KS score might indicate better differentiation between good and bad populations, but its significance depends on the operating range of the business, which in turn depends on the nature of the business.</p><p>For example, subprime lenders operate at different parameters to near-prime lenders who operate at different parameters to prime lenders. Fraud-related KPIs tend to be focused in the riskiest tail and depend on the specific fraud typology fraudsters use. Ultimately, testing requires tailoring measurements to operational contexts, being sure to inform decisions with relevant, actionable insights.</p><h3><strong>Key Insight #2: Don’t Review Scores in Isolation to Existing Strategy</strong></h3><p>Assess results beyond face value instead of simply prioritizing the highest-ranking predictive scores. Consider the broader context, including the <em>net benefit</em> and <em>incremental lift</em> a solution may provide over existing strategies.</p><p>For example:</p><ul> <li><strong>Lift Over Legacy: </strong>Always measure a prospective solution’s benefit in comparison to the existing decision strategy (referred to as “lift”). A solution that integrates complementary, non-correlated data with current systems may provide greater incremental value than one with the highest KPIs.</li> <li><strong>Avoid Redundant Data</strong>: Adding too much of the same type of information results in diminishing returns. Incorporate a mix of highly varied data sources like credit bureau data, alternative data, device information, email data, behavioral data and/or biometric data to create a holistic and multi-dimensional risk assessment. Multiple uncorrelated signals enrich predictive power, reduce risks and improve fraud detection.</li> </ul><h3><strong>Key Insight #3: Don’t Use a Ruler to Measure Wind Speed</strong></h3><p>It’s crucial for risk managers to align performance metrics with the specific problem they aim to address when testing and implementing scoring algorithms. Using a score calibrated for third-party fraud to tackle first-party fraud will yield suboptimal results, as the frauds differ significantly in their typologies.</p><p>Equally, specific fraud categories demand distinct performance definitions. A mismatch between a score’s calibration and a lender’s business metrics can lead to ineffective decisions. In that sense, the best-performing scoring models are not necessarily the ones with the most “accurate” definitions but those that are calibrated to align with your organization’s specific fraud problems and operational metrics. Misaligned definitions negatively impact outcomes for all parties, underscoring the importance of tailoring fraud analytics to meet individual performance needs.</p><h3><strong>Key Insight #4: Avoid the Risk of Overfitting</strong></h3><p>When testing competitor products, it’s crucial to avoid the pitfall of overfitting, where providers may intentionally or unintentionally manipulate algorithms to achieve high performance metrics on test samples, at the cost of long-term efficacy. Overfitted models often degrade quickly when applied to broader populations, leading to suboptimal results.</p><p>The most accurate and sustainable scoring models use a three-sample test:</p><ol> <li><strong>Development Sample:</strong> Share performance data with the solutions provider to optimize scoring algorithms.</li> <li><strong>Validation Sample:</strong> Provide an out-of-time sample from a different period to test the score’s robustness.</li> <li><strong>Independent Sample:</strong> Request scoring on a final sample without performance data to confirm validation across independent data sets.</li> </ol><p>This structured process minimizes risks and overfitting, and ensures high-performing, predictive scores tailored to the institution’s applicant base. Sharing performance data also enables prospective solution providers to fully leverage their expertise to deliver solutions aligned with the client’s unique needs.</p><h3><strong>Key Insight #5: Beware of Truncation Bias</strong></h3><p>Truncation bias refers to the development sample not accurately reflecting the broader “through the door” population. Most samples are based on legacy products and strategies, resulting in a skewed outcome during head-to-head testing because the sample was shaped by the legacy solution. This bias often places the legacy score at a disadvantage because it has already filtered outcomes, giving the challenger score an artificial advantage. The challenger score need only find a few additional red flags in the booked population to look like the stronger score. This is an unbalanced conclusion insofar as the legacy score doesn’t have the opportunity to add insight to a challenger score.</p><p>To mitigate truncation bias, adopt champion-challenger testing methods on the full population of applicants rather than solely on a booked sample. This approach ensures a more accurate assessment of both legacy and challenger solutions by reflecting their true potential impact on decision making.</p><p><a href="https://securityboulevard.com/wp-content/uploads/2025/11/Screen-Shot-2025-11-24-at-12.29.55-PM.png"><img fetchpriority="high" decoding="async" class="wp-image-2077127 alignleft" src="https://securityboulevard.com/wp-content/uploads/2025/11/Screen-Shot-2025-11-24-at-12.29.55-PM-284x300.png" alt="" width="391" height="413" srcset="https://securityboulevard.com/wp-content/uploads/2025/11/Screen-Shot-2025-11-24-at-12.29.55-PM-284x300.png 284w, https://securityboulevard.com/wp-content/uploads/2025/11/Screen-Shot-2025-11-24-at-12.29.55-PM-969x1024.png 969w, https://securityboulevard.com/wp-content/uploads/2025/11/Screen-Shot-2025-11-24-at-12.29.55-PM-768x812.png 768w, https://securityboulevard.com/wp-content/uploads/2025/11/Screen-Shot-2025-11-24-at-12.29.55-PM.png 1122w" sizes="(max-width: 391px) 100vw, 391px"></a></p><h3></h3><h3></h3><h3></h3><h3></h3><h3></h3><h3></h3><h3></h3><h3></h3><h3></h3><h3><strong>Key Insight #6: Never “Set-and-Forget” Scores</strong></h3><p>There is a false perception that scores don’t need much maintenance. However, changing economic conditions, business strategies, target markets and risk tolerances are among the drivers that cause current operational practices around a credit risk or fraud solution to lose effectiveness over time. In the case of fraud solutions, fraudsters change their tactics regularly to evade defenses. Practitioners should track and recalibrate their credit risk scores at least annually and fraud scores even more often.</p><h3><strong>Key Insight #7: Be Wary of Marketing Hype </strong></h3><p>Marketing hype or exaggeration is unfortunately common in our space. I see it every day and I urge risk managers to exercise caution, particularly when it involves buzzwords like Artificial Intelligence (AI) and Machine Learning (ML). Many providers pitching for business exaggerate the role of these in their solutions, leading to misunderstandings about the actual sophistication and effectiveness of the technology.</p><p>I advise asking critical questions to distinguish genuine AI solutions from inflated claims. At LexisNexis Risk Solutions we emphasize transparency. We only reference AI or ML in our offerings where these technologies are demonstrably in use. Don’t get caught out: choosing solutions that make overstated AI claims inevitably results in unnecessary operational and compliance burdens that will further negatively impact efficiency. In short, validate every marketing claim until you are satisfied it really does what it promises.</p><h3><strong>Conclusion</strong></h3><p>There are so many moving parts that make competitive testing a complicated balancing act. By following a structured approach and applying due diligence in all the right places, while avoiding the pitfalls of overstated promises, any business can discover the optimal mix of solutions to best serve their customers and business objectives. Above all, they can find the solutions provider that represents the perfect technical and cultural fit that will result in a long and productive partnership.</p><p> </p><div class="spu-placeholder" style="display:none"></div><div class="addtoany_share_save_container addtoany_content addtoany_content_bottom"><div class="a2a_kit a2a_kit_size_20 addtoany_list" data-a2a-url="https://securityboulevard.com/2025/11/dont-use-a-ruler-to-measure-wind-speed-establishing-a-standard-for-competitive-solutions-testing/" data-a2a-title="Don’t Use a Ruler to Measure Wind Speed: Establishing a Standard for Competitive Solutions Testing"><a class="a2a_button_twitter" href="https://www.addtoany.com/add_to/twitter?linkurl=https%3A%2F%2Fsecurityboulevard.com%2F2025%2F11%2Fdont-use-a-ruler-to-measure-wind-speed-establishing-a-standard-for-competitive-solutions-testing%2F&linkname=Don%E2%80%99t%20Use%20a%20Ruler%20to%20Measure%20Wind%20Speed%3A%20Establishing%20a%20Standard%20for%20Competitive%20Solutions%20Testing" title="Twitter" rel="nofollow noopener" target="_blank"></a><a class="a2a_button_linkedin" href="https://www.addtoany.com/add_to/linkedin?linkurl=https%3A%2F%2Fsecurityboulevard.com%2F2025%2F11%2Fdont-use-a-ruler-to-measure-wind-speed-establishing-a-standard-for-competitive-solutions-testing%2F&linkname=Don%E2%80%99t%20Use%20a%20Ruler%20to%20Measure%20Wind%20Speed%3A%20Establishing%20a%20Standard%20for%20Competitive%20Solutions%20Testing" title="LinkedIn" rel="nofollow noopener" target="_blank"></a><a class="a2a_button_facebook" href="https://www.addtoany.com/add_to/facebook?linkurl=https%3A%2F%2Fsecurityboulevard.com%2F2025%2F11%2Fdont-use-a-ruler-to-measure-wind-speed-establishing-a-standard-for-competitive-solutions-testing%2F&linkname=Don%E2%80%99t%20Use%20a%20Ruler%20to%20Measure%20Wind%20Speed%3A%20Establishing%20a%20Standard%20for%20Competitive%20Solutions%20Testing" title="Facebook" rel="nofollow noopener" target="_blank"></a><a class="a2a_button_reddit" href="https://www.addtoany.com/add_to/reddit?linkurl=https%3A%2F%2Fsecurityboulevard.com%2F2025%2F11%2Fdont-use-a-ruler-to-measure-wind-speed-establishing-a-standard-for-competitive-solutions-testing%2F&linkname=Don%E2%80%99t%20Use%20a%20Ruler%20to%20Measure%20Wind%20Speed%3A%20Establishing%20a%20Standard%20for%20Competitive%20Solutions%20Testing" title="Reddit" rel="nofollow noopener" target="_blank"></a><a class="a2a_button_email" href="https://www.addtoany.com/add_to/email?linkurl=https%3A%2F%2Fsecurityboulevard.com%2F2025%2F11%2Fdont-use-a-ruler-to-measure-wind-speed-establishing-a-standard-for-competitive-solutions-testing%2F&linkname=Don%E2%80%99t%20Use%20a%20Ruler%20to%20Measure%20Wind%20Speed%3A%20Establishing%20a%20Standard%20for%20Competitive%20Solutions%20Testing" title="Email" rel="nofollow noopener" target="_blank"></a><a class="a2a_dd addtoany_share_save addtoany_share" href="https://www.addtoany.com/share"></a></div></div>