To newcomers to social science research with backgrounds in mathematics and

engineering like myself, one question about the choice of methodology in quantitative
social science constantly catches our attention: why do social scientists treat linear
regression as qualitatively different from other methods like Lasso regression or ridge
regression, with the former being considered as the default option whereas the latter
being labeled as “machine learning” methods, while mathematically speaking they are
from the same family of algorithms and share similar ways of interpretation. What Lasso
and ridge regressions do is literally limiting the size of coefficients of linear regression to
facilitate interpretation of coefficients, differing only by the notion of size of coefficients
they choose. In this sense, penalized regression is just like the continuous version of
subset selection. It is therefore sometimes not entirely clear why using Lasso or ridge
regressions seem to require much more justification than using linear regression.

I think this bias toward existing methods is a general phenomenon in scientific research
that is of interest in itself. From a science-of-science perspective, it is interesting to
study how scientists of a particular field draw the boundary of “standard” methods of
data analysis, and what it takes for a new method to be considered as a valid approach.
The former question calls for understanding how scientists from different fields
categorize data analysis methods. For example, computer scientists and applied
mathematicians might think of methods from their mathematical expressions, therefore
consider various regression methods, which often function by minimizing similar
objective functions, as belonging to the same class. Natural scientists and social
scientists, on the other hand, might think of methods more from the literal meaning of
their names and the high-level descriptions provided by their developers. To them,
linear regression and Lasso regression sounds like two distinct methods, the former
being linear and the latter apparently non-linear. Moreover, the explanation that Lasso
regression “shrinks coefficients to zero” does not sound intuitive to most people. The
boundary of standard methods could therefore differ by not just discipline, but also by
the academic background of individual researchers.

For the second question, we need to understand the criteria scientists use to gauge the
effectiveness of a new method, which could be different, often a much higher bar, than
what they ask of standard methods. For example, even though it is well-known that
linear regression models are susceptible to problems such as overfitting, p-hacking, and
model cherry-picking, researchers still use it as the default way of analysis. In other
words, it is likely that some form of status quo bias is working against novel machine
learning methods in some fields of science. It would be interesting to see how this
affects adoption of new methods and what strategies advocates of new methods can
use.