Proxies are variables in an underwriting model whose predictive power is attributable mostly to their correlation with protected-class membership, such as race, gender, or age, rather than a valid credit characteristic. For example, ZIP codes can be a close proxy for race due to residential segregation while language preference may strongly correlate with national origin, making it another potential close proxy. The elimination of proxies from underwriting models isn’t just a matter of fairness, it’s a regulatory requirement. The Equal Credit Opportunity Act (ECOA) states that the use of protected class characteristics, and variables that act as close proxies for them, is generally prohibited. But a greater understanding of what exactly makes a variable a proxy is required before an organization can find and remove it from a model.
One might assume that any variable correlated with a demographic attribute is a proxy, but that’s too broad a definition. Even traditional credit scores, a cornerstone of today’s lending industry, are correlated with demographic attributes. In fact, it’s that correlation between a variable’s predictive power and protected-class membership that matters. Let’s look at an example to illustrate this:
If we included an applicant’s name in the model, we might observe that individuals named James and Mary pay back their loans less often than those named Michael and Jessica. The model then learns to associate these names with a lack of creditworthiness. But it turns out that James and Mary were the most common baby names in the 1950s, while Michael and Jessica were the most common in the 1990s. While there is no reason to believe that a person’s name has any effect on their creditworthiness, we know from experience that age can be a good predictor of credit performance. Older individuals may be more likely to see declines in their income as they move into retirement, which could make it difficult to pay bills or keep current on loans, resulting in poor credit. Therefore, the Jameses and Marys declined by our model may be in this older age demographic given the prevalence of the names 70 years ago. Individuals in their 30s, however, are likely to have steady jobs and a stable credit history after paying off school loans and taking on a home mortgage. Of course, this isn’t always true. Thirtysomethings may have a shorter credit history and less wealth, while older individuals may be more established financially and have a long history of responsible credit use.
In learning to associate a person’s name with creditworthiness (or the lack thereof), the model was using that information as a proxy for age, which would not be allowed under ECOA rules. It identified this trend because of an overall association between age and creditworthiness, stemming from the fact that older individuals may struggle financially in retirement after losing the income from a decades-long career. While this trend generally holds when you consider all individuals, it does not hold when you restrict yourself to specific age groups.
The above example leads to the main observation underlying Upstart’s patent-pending proxy detection methodology: that proxies don’t hold up as an accurate measure of a person’s creditworthiness when conditioned, or restricted, to protected-class membership. For example, we observed that, on the whole population, a person’s name seemed to have a correlation with credit risk. However, when we considered specific age groups, the association completely disappeared. Upstart performs this test for all model variables to ensure they are not operating as proxies. Here’s how it works:
We use a scoring system to rank each variable according to its likelihood of being a stand-in for a protected group. First, we measure the predictive power of a variable by computing its association with a loan default or prepayment. If the variable successfully predicts either outcome, it may be beneficial to the model. But then we restrict the variable to applicants from a specific demographic group and test it again. A variable that loses most of its predictive power for default or prepayment when conditioned on a demographic attribute has the risk of being a proxy, according to our definition. A comparison of the two results gives us what we call a “normalized proxy score,” allowing us to determine the severity of the variable’s “proxiness.” In essence, a high normalized proxy score means the variable is still a good predictor even within a specific demographic group. A low score means it’s not a good predictor, indicating that the variable might be acting as a proxy for the demographic attribute rather than directly predicting the outcome.
There are multiple benefits to Upstart’s proxy analysis: First, our definition of a proxy lends itself well to intuitive interpretation and computational speed due to its simplicity. Second, because the proxy score does not depend on the model, we can keep the detection process separate from the building of the model. This is important because we won’t miss proxies included in a model that the model may not currently be using, but may learn to use in the future. Third, the normalization of the proxy score allows us to put different variables on the same scale so that we can compare them to empirically derived thresholds that hold across all variables. Finally, and most importantly, having an efficient and effective way of screening out proxy variables enables us to more confidently include new information in our models, ultimately leading to more accurate assessments of risk and improving access to credit across all demographic groups.
¹ Upstart does not collect demographic information from its applicants. The demographic attributes are estimated via location and surname using the BISG methodology.