Susan Athey is interested in helping machine-learning applications look beyond correlations and root causes.
Machine learning based on data has made its way into many fields of Science, industry, and public policy. Today, it’s easy to classify complex things, like text, speech, and photos, or predict the website traffic for tomorrow. Asking a computer to investigate how a minimum wage increase might affect employment or design an algorithm that assigns optimal treatments to each patient in a medical facility is a different game.
Professor of Economics at Stanford Graduate School of Business Susan Athey says that most machine learning applications are highly functional versions of simple tasks. Computers are particularly good at sifting through large amounts of data to find patterns and connections and make accurate predictions. In a stable environment, how or why an algorithm works? It is not essential. It’s enough to see how the program performs on test data. This means you do not need to be an expert to use prediction algorithms.
Despite their profusion of data and computing power, machine-learning algorithms aren’t very good at distinguishing correlation from causation – determining whether a statistically-linked pattern is coincidental or caused by some cause-and-effect force. Athey explains that “some problems are not solvable by adding more data or using more complex algorithms.”
Athey says that if machine-learning methods are used to address public policy problems, we must develop new ways to combine them with causal inference methods. This would dramatically expand the potential for big-data applications and transform our abilities to design, evaluate, and improve public policy work.
What Predictive models Miss
Athey believes it is essential for government agencies to be aware of the limitations of machine-learning techniques. She outlined several scenarios in a recent paper that was published in Science. These scenarios highlight the difference between prediction and causal inference issues and how machine-learning programs would struggle to draw valuable conclusions about cause and effect.
Machine-learning techniques can help predict churn. The real challenge is to determine the most effective allocation of resources. This requires identifying the customers who will benefit the most from an intervention, such as sending targeted emails or offering discounts. It’s harder to determine. The firm may need to run a random experiment to find out where the most significant benefits are. Athey cites a study that showed that, in a firm that conducted a more thorough analysis, there was only a 50% overlap between customers at high risk of leaving and those for whom the intervention worked best.
Predictive models have already been used in another case to identify patients who, while eligible for hip replacement surgery, should not receive the operation because they are likely to die soon from other causes. These methods do not solve the more difficult problem of prioritizing the patients most benefit from the surgery.
Athey says, “If you are just crunching big data without thinking about all the things that can go wrong when you confuse correlation with causality, then you may think that adding a larger machine to your problem will solve it.” “But sometimes, the answer is not in the data.”
She says this is especially true in many real-world situations where public policy is shaped.
Randomized controlled experiments are the gold standard for determining correlations and causality. They allow for fairly straightforward conclusions about cause and effect. These experiments are often used to test new drugs. A randomly chosen group of patients with a specific illness receives the drug, while a second group is given a placebo. The drug was likely the reason if a large portion of the first group improves.
In many situations, such experiments would not be feasible. It would be impossible, both politically and practically, to run a large-scale controlled experiment that examined what happens when minimum wages are raised or lowered in different locations. Policy analysts must instead rely on “observational” data, or data that is not generated randomly. To draw valuable conclusions from uncontrolled, unreliable observational data is beyond the reach and capability of most predictive methods.
Athey hopes that her research will help push the boundaries of machine learning. She says that combining pure prediction and causal inference will allow us to solve the most challenging problems, which involve determining all the possible outcomes of different policies.
Athey questions, “How can we continue developing and building new technologies that fully exploit big data?” Athey notes that many public policy problems have questions about causal inference as their core. “That is the hard stuff. You have to be cautious to understand what something will do fully. “But that’s the majority of the world.”
People’s Computational Power
Athey believes that while these advances are still a few years away, the momentum behind big data and machine learning in academic research and practical applications is energizing. She says that the gap between research and practice is closing. It’s fantastic when our research is adopted within months.
She is especially pleased to see the broad adoption of predictive methods, which were the sole domain of a small group of highly skilled data scientists not long ago. Athey says, “It is amazing because it empowers people who would not have used computers for anything but word processing in the past.” Now, not only the geeky engineers are interested in the latest research. People at the highest levels of an organization also care about it. They understand the importance of using data to optimize investments and decisions. They are building open-source software and big-data models to make cutting-edge predictions using cutting-edge technologies. It has been wholly democratized, and I think this is a great success story.”