Understanding Validation Samples Within Model Development

by Guest Contributor 3 min read June 18, 2018

An introduction to the different types of validation samples

Model validation is an essential step in evaluating and verifying a model’s performance during development before finalizing the design and proceeding with implementation. More specifically, during a predictive model’s development, the objective of a model validation is to measure the model’s accuracy in predicting the expected outcome. For a credit risk model, this may be predicting the likelihood of good or bad payment behavior, depending on the predefined outcome. Two general types of data samples can be used to complete a model validation. The first is known as the in-time, or holdout, validation sample and the second is known as the out-of-time validation sample.

So, what’s the difference between an in-time and an out-of-time validation sample? An in-time validation sample sets aside part of the total sample made available for the model development. Random partitioning of the total sample is completed upfront, generally separating the data into a portion used for development and the remaining portion used for validation. For instance, the data may be randomly split, with 70 percent used for development and the other 30 percent used for validation. Other common data subset schemes include an 80/20, a 60/40 or even a 50/50 partitioning of the data, depending on the quantity of records available within each segment of your performance definition. Before selecting a data subset scheme to be used for model development, you should evaluate the number of records available in your target performance group, such as number of bad accounts. If you have too few records in your target performance group, a 50/50 split can leave you with insufficient performance data for use during model development. A separate blog post will present a few common options for creating alternative validation samples through a technique known as resampling.

Once the data has been partitioned, the model is created using the development sample. The model is then applied to the holdout validation sample to determine the model’s predictive accuracy on data that wasn’t used to develop the model. The model’s predictive strength and accuracy can be measured in various ways by comparing the known and predefined performance outcome to the model’s predicted performance outcome.

The out-of-time validation sample contains data from an entirely different time period or customer campaign than what was used for model development. Validating model performance on a different time period is beneficial to further evaluate the model’s robustness. Selecting a data sample from a more recent time period having a fully mature set of performance data allows the modeler to evaluate model performance on a data set that may more closely align with the current environment in which the model will be used. In this case, a more recent time period can be used to establish expectations and set baseline parameters for model performance, such as population stability indices and performance monitoring.

Learn more about how Experian Decision Analytics can help you with your custom model development needs.

Related Posts

Rewriting the Road Ahead with Longer Loan Terms and Increased Refinancing Options

The automotive market is entering a new phase defined not just by what consumers are buying, but by how they’re choosing to finance it. According to Experian Automotive’s State of the Automotive Finance Market Report: Q1 2026, nearly one-third (35.55%) of all new vehicle loans now stretch more than six years, up from 30.83% in Q1 2025. Similarly on the used side, 31.54% of loans extended more than six years, an increase from 28.60% last year. The shift highlights why affordability is reshaping how consumers are financing their vehicles, particularly in larger and higher-priced vehicles. Refinancing gains traction as interest rates stabilize In addition to longer-term loans, consumers are becoming increasingly deliberate with their financing decisions and managing monthly payments as refinancing activity has gained momentum. For instance, consumers who refinanced this quarter lowered their interest rate by 2.2% and saved an average of $81 on their monthly payment. Credit unions, in particular, continued to play a major role in helping consumers secure more affordable payment options. In Q1 2025, credit unions accounted for the lion’s share of automotive refinancing at 63.43%, from 62.31% a year ago. By comparison, banks went from 23.51% to 22.59% year-over-year. Furthermore, those who refinanced with a credit union saved an average of $101 this quarter, whereas those who refinanced with banks saved $60. Expanding credit access through flexible financing Another notable trend this quarter was the incessant growth in subprime financing as credit accessibility across the market continues to increase. In the first quarter of this year, subprime borrowers made up 15.75% of total vehicle financing, from 14.40% last year. For new vehicles in particular, the subprime market went from 5.61% to 6.88% year-over-year, while subprime in used vehicle financing grew to 20.60% this quarter, from 19.36% a year ago. Increased activity in the subprime segment highlights continued confidence in the automotive market and underscores the importance of expanded financing options. As consumers seek greater flexibility with financing decisions that fit their lifestyle, lenders and dealers have the opportunity to approach them with more personalized solutions. These trends are helping keep both new and used vehicle markets moving forward, while creating new opportunities for consumers to manage payments and purchase confidently. To learn more about automotive finance trends, view the full State of the Automotive Finance Market Report: Q1 2026 presentation on demand.

Published: June 2, 2026 by Melinda Zabritski
Staying Competitive After Trigger Leads Evolve: A Roadmap For Lenders

Trigger leads have long been the preferred solution for identifying high-intent mortgage borrowers. But with the implementation of the Homebuyers Privacy Protection Act (HPPA), which introduces new limitations and consumer protections around trigger leads, that playbook will need to shift. Now, lenders are quickly facing a pivotal shift in how they discover, engage, and convert prospective borrowers into customers. The industry now stands at a crossroads. Lenders who adapt early—leaning into predictive tools, consent-based engagement, and smarter prescreening—will redefine borrower acquisition in a more privacy-centric era.  HPPA: A structural change to mortgage marketing  The HPPA amends the Fair Credit Reporting Act by significantly restricting the use of mortgage inquiries for prescreen purposes. As of March 5, 2026, credit bureaus may only provide or utilize mortgage inquiries to:  End users with explicit borrower consent  The originator of the consumer’s current mortgage  The servicer of the consumer’s current mortgage  An insured depository institution or credit union where the consumer has an existing account  While these exemptions may provide continuity for banks and credit unions, many mortgage brokers and nonbank lenders will need to overhaul their prescreen practices—or risk being cut off entirely from a previously high-performing acquisition channel.  Why this isn’t just a compliance shift—It’s a strategic recalibration  Mortgage triggers in prescreen allow lenders to react instantly to consumer intent. Lenders rely on a prompt and convincing narrative to entice applicants to switch lenders. Mortgage inquiry triggers are effective and were, therefore, a prospecting strategy for many lenders. Recent legislative changes significantly restrict the availability of these inquiry triggers, and impacted lenders are focusing on a more intentional prospecting strategy to compete.   Without these mortgage triggers in prescreen, lenders need to ask:  Who are we trying to reach?  What early signals can we act on?  How do we earn permission and attention before a mortgage inquiry ever happens?  Transforming the funnel: From reaction to anticipation  The shift in mortgage inquiry-based prescreen isn’t the end of high-intent lead targeting. It’s the beginning of a more strategic and intentional approach—one that leverages earlier indicators of mortgage readiness and focuses on building relationships, not just closing transactions.  Here’s where the momentum is evolving, creating a new and smarter funnel:  Prescreen marketing: Using credit and behavioral attributes to help identify consumers who meet specific lending criteria before they signal active intent.  Predictive modeling: Leveraging propensity scores or custom models to prioritize outreach based on conversion likelihood.  Consent-based engagement: Implementing compliant mechanisms to capture and manage borrower opt-ins at scale.  The power of predictive modeling  According to recent industry interviews, propensity modeling is emerging as one of the most effective replacements for trigger-based prescreen. These models analyze hundreds of credit attributes—such as utilization, account mix, account age, and depth—to help identify consumers statistically more likely to seek a mortgage.  For lenders just beginning to use predictive modeling, off-the-shelf models can be a quick way to identify potential borrowers. For example, when layering propensity scores on top of credit eligibility, which can improve borrower targeting, many lenders see an increase in open mortgage loan rates.  Meanwhile, custom-built models, which analyze a lender’s own campaign performance over time, offer the highest level of precise targeting. These models isolate the attributes most predictive of conversions within a specific product mix—optimizing not just volume, but fit.  Speed without traditional triggers? It’s possible  One of the biggest concerns among lenders is maintaining the speed historically enabled by trigger leads. But that concern may be overblown.  Self-service prescreen platforms now allow marketers to generate qualified lead lists in as little as 24 hours, enabling rapid response during rate drops, competitive shifts, or seasonal demand spikes.   For those new to prescreening, batch campaigns still offer value, especially with analyst support.   Don’t overlook retention  In an era of intense acquisition competition, retention becomes a key differentiator.  Lenders who monitor property status, cash flow, and consumer credit behavior can proactively identify when an existing borrower is likely to list, refinance, or exit. Armed with that intelligence, lenders can re-engage with the borrower at the right moment—sometimes before a competitor is considered or contacted.  This level of behavioral intelligence may soon separate proactive lenders from reactive ones.  Actions instead of reactions  The evolution of trigger-based prescreen doesn’t just require new tools; it demands new thinking. Lenders should begin by auditing their current pipelines and determining:  What percentage of our acquisition is dependent on triggers?  What share of our book falls under the HPPA exemptions?  How will we scale compliant opt-in collection?  Are our current prescreen or modeling capabilities future-ready?  Those who answer these questions today—and act on them—won’t just be in compliance with the new laws, they’ll lead in a transformed market. Lenders should also be asking:   Do we have the infrastructure to collect and act on borrower consent?  Are our acquisition teams equipped to run prescreen campaigns — both batch and self-service?  What predictive models are we using (or could we use) to prioritize leads?  Are we proactively monitoring our portfolio to catch retention risks early?  How are we preparing our sales teams for longer, more consultative buying journeys?  Conclusion  The HPPA signals a shift away from relying on passive, inquiry-based prescreen acquisition and the beginning of smarter, more strategic engagement with potential borrowers. Lenders who embrace this transition early will find themselves not just compliant, but competitive—with deeper borrower insights, better conversion rates, and stronger long-term customer relationships.  The market is moving. The only question is: will you lead the change or chase it?  Citation  Experian. (2025, November). Interview: How the Homebuyers Privacy Protection Act is reshaping mortgage marketing—and what lenders should do now [transcript]. Experian Mortgage Insights. Insights based on lender feedback, campaign performance data, and analysis of prescreen marketing strategies and predictive modeling outcomes were gathered from Experian client engagements and internal mortgage analytics between May and October 2025. Homebuyers Privacy Protection Act timeline and legal context referenced from legislation signed September 5, 2025, with implementation beginning March 5, 2026.   

Published: April 22, 2026 by Ivan Ahmed

Subscribe to our thought leadership

Enter your name and email for the latest updates.

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

Subscribe to our thought leadership

Don't miss out on the latest industry trends and insights!
Subscribe