Publications
Publications
- 2023
- HBS Working Paper Series
Nailing Prediction: Experimental Evidence on the Value of Tools in Predictive Model Development
By: Daniel Yue, Paul Hamilton and Iavor Bojinov
Abstract
Predictive model development is understudied despite its centrality in modern artificial
intelligence and machine learning business applications. Although prior discussions
highlight advances in methods (along the dimensions of data, computing power, and
algorithms) as the primary driver of model quality, the tools that implement those
methods have been neglected. In a field experiment leveraging a predictive data science
contest, we study the impact of tools by restricting access to software libraries
for machine learning models. By only allowing access to these libraries in our control
group, we find that teams with unrestricted access perform 30% better in log-loss error
— a statistically and economically significant amount, equivalent to a 10-fold increase
in the training data set size. We further find that teams with high general data-science
skills are less affected by the intervention. In contrast, teams with high tool-specific
skills significantly benefit from access to modeling libraries. Our findings are consistent
with a mechanism we call ‘Tools-as-Skill,’ where tools automate and abstract some
general data science skills but, in doing so, create the need for new tool-specific skills.
Keywords
Citation
Yue, Daniel, Paul Hamilton, and Iavor Bojinov. "Nailing Prediction: Experimental Evidence on the Value of Tools in Predictive Model Development." Harvard Business School Working Paper, No. 23-029, December 2022. (Revised April 2023.)