Understanding Validation Samples Within Model Development

Updated: August 4, 2026 by Guest Contributor 3 min read June 18, 2018

An introduction to the different types of validation samples

Model validation is an essential step in evaluating and verifying a model’s performance during development before finalizing the design and proceeding with implementation. More specifically, during a predictive model’s development, the objective of a model validation is to measure the model’s accuracy in predicting the expected outcome. For a credit risk model, this may be predicting the likelihood of good or bad payment behavior, depending on the predefined outcome. Two general types of data samples can be used to complete a model validation. The first is known as the in-time, or holdout, validation sample and the second is known as the out-of-time validation sample.

So, what’s the difference between an in-time and an out-of-time validation sample? An in-time validation sample sets aside part of the total sample made available for the model development. Random partitioning of the total sample is completed upfront, generally separating the data into a portion used for development and the remaining portion used for validation. For instance, the data may be randomly split, with 70 percent used for development and the other 30 percent used for validation. Other common data subset schemes include an 80/20, a 60/40 or even a 50/50 partitioning of the data, depending on the quantity of records available within each segment of your performance definition. Before selecting a data subset scheme to be used for model development, you should evaluate the number of records available in your target performance group, such as number of bad accounts. If you have too few records in your target performance group, a 50/50 split can leave you with insufficient performance data for use during model development. A separate blog post will present a few common options for creating alternative validation samples through a technique known as resampling.

Once the data has been partitioned, the model is created using the development sample. The model is then applied to the holdout validation sample to determine the model’s predictive accuracy on data that wasn’t used to develop the model. The model’s predictive strength and accuracy can be measured in various ways by comparing the known and predefined performance outcome to the model’s predicted performance outcome.

The out-of-time validation sample contains data from an entirely different time period or customer campaign than what was used for model development. Validating model performance on a different time period is beneficial to further evaluate the model’s robustness. Selecting a data sample from a more recent time period having a fully mature set of performance data allows the modeler to evaluate model performance on a data set that may more closely align with the current environment in which the model will be used. In this case, a more recent time period can be used to establish expectations and set baseline parameters for model performance, such as population stability indices and performance monitoring.

Learn more about how Experian Decision Analytics can help you with your custom model development needs.

Related Posts

How GenAI Fraud Bots Fly Under the Radar (and How to Ground Them Before It’s Too Late)

Today’s AI-powered fraud bots are exceptionally good at mimicking human behavior. Learn why a modern defense strategy is key.

November 18, 2026 by Laura Burrows
What Is Incident Response? Key Steps, Planning and Best Practices

Learn what incident response is, the key steps in the process and how to build and test an incident response plan for a data breach.

November 17, 2026 by Laura Burrows
Rethinking Charged-Off Portfolio Prioritization

Experian’s new Recovery Scores model turns portfolio intelligence into more focused recovery strategies, helping organizations improve returns, direct effort and accelerate speed to cash. The challenge isn’t simply working more accounts. Whether a debt buyer is evaluating an acquisition, an agency is deciding where to assign collector time or a lender is managing its own post-charge-off accounts, the question is similar: How can limited capacity be directed toward accounts with the greatest potential for recovery? Prioritizing accounts mainly by age, balance or when they were last worked may not show which ones are most likely to generate a recovery. With the new Recovery Scores model, organizations can rank charged-off accounts by expected collectability, providing a consistent signal to inform placement strategy and treatment decisions across the recovery lifecycle. Make collector capacity a strategic resource Working every account with the same level of attention isn’t always practical. The more useful question is: Where should teams focus first to protect speed to cash? Our model performance analysis found: 80% of recovered dollars come from the top 40% of accounts worked. The value isn’t simply a better ranking. It’s how organizations can use it for collections optimization. Score bands can inform different strategies for placement, outsourcing, auction and legal channel treatment rather than applying the same approach across an entire portfolio. For debt buyers: The scores can inform acquisition pricing and how a newly purchased portfolio is divided across internal collections, legal channels and resale. Expected collectability can shape debt portfolio analysis, including how a portfolio is valued and segmented before collection work begins. For agencies: Score bands can help teams focus collector resources on accounts with stronger recovery potential. For first-party lenders: Managing recovery internally, the scores provide another signal to help structure decisions about which accounts to work and in what order. Look beyond collectability alone Prioritization rarely comes down to a single variable. Consider two accounts that both rank highly for expected collectability, but one has a substantially larger balance. Their recovery likelihood may be similar, but the potential dollars recovered are not. The Recovery Scores model uses debt collections analytics by combining Experian’s bureau-based score with current balance information. This helps organizations consider both the likelihood of recovery and the potential dollars at stake, while sensitive balance information remains within the organization’s working environment. Monitor accounts as they change The first decision about an account is rarely the last. Teams continually make choices about treatment, placement and resource allocation as accounts move through the recovery process. Recovery Scores is also available as a trigger. When underlying information changes, teams can receive an update and reconsider an account’s placement without waiting for the next portfolio-wide re-score. An account that shows lower recovery potential today may have a different recovery outlook later. Turn recovery potential into a portfolio strategy When portfolios contain more accounts than teams can work at once, the question isn’t simply, “How many accounts can we work?” It is, “Where can our resources have the greatest impact?” The latest Recovery Scores model helps organizations sharpen placement and treatment decisions. It gives debt buyers a way to price and segment portfolios with greater confidence and gives lenders another tool to direct limited collector time across their own post-charge-off portfolios. Learn more about Experian Recovery Scores

November 10, 2026 by Zohreen Ismail

Subscribe to our Newsletter

Enter your name and email for the latest updates.

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

Subscribe to our Newsletter

Don't miss out on the latest industry trends and insights!
Subscribe