verbal explanations of complex machine learning models can potentially mislead users. Take linear models again as an example. Looking at which features get high weight, absent broader context, might yield dubious interpretations. What if one weight is high, but it’s for a feature which takes the same value 95% of the time? In this case, this weight may not be informative for explaining how some decisions are made.For truly complex models, it may be unreasonable to account for its full dynamics with terse verbiage. And perhaps the difficulty of producing simple explanations shouldn’t be surprising. Sometimes, our very purpose in building complex models is so that they can express complex hypotheses that we might not have been able to specify concisely. We should question the proposition that it’s reasonable to expect a short summary to meaningfully explain a complex model.
Source: The Myth of Model Interpretability
