Scientists from University College London (UCL) and the Academy of Mathematics and Systems Science, Chinese Academy of Sciences (CAS, AMSS), have reported progress in understanding problems associated with Bayesian model selection.   

The research suggests that Bayesian methods tend to produce very high posterior probabilities for estimated evolutionary trees even if the trees are clearly wrong, and offers a possible explanation for this phenomenon.  

Model comparison is widely used in various branches of sciences in which scientific hypotheses are formulated as statistical models and tested using observed data. However, model comparison is a thorny issue in both classical statistics and Bayesian statistics.   

In classical statistics, two nested models are compared, and the framework does not work when the models being compared are not nested. In contrast, Bayesian statistics compares different models by calculating their posterior probabilities, which show our confidence or belief in the model.   

Not only do these two methodologies spring from drastically different philosophies, they may also produce opposite conclusions in the analysis of the same data. Bayesian model selection is known to converge upon the true model if the true model is included among the models under consideration.   

That is, when scientists collect more and more data, the posterior probability for the right model will approach 100%. Scientists will thus be more and more certain which is the true model.  

However, if all the models under consideration are wrong, the behavior of the Bayesian method is unknown.   

Scientists have characterized Bayesian model selection problems, and categorized them into three types, each of which shows a different behavior.   

In the most scientifically interesting case, i.e., when the models being compared are distinct and nearly equally wrong, Bayesian model selection shows problematic polarized behavior: It tends to support one model with full force in some datasets, while supporting another model in other datasets.  

The result may be summarized using the following analogy: Suppose the world is gray, but we ask a sage whether it is black or white. He takes a deep look at the world and says it is black, with total confidence. But the next time we ask the same question, he says it is white, again with total confidence.  

This study was motivated by problems in molecular phylogenetics, which is the science of working out the relationships among species using genetic data, represented by evolutionary trees.   

These different trees are opposing statistical models in the Bayesian analysis of the data. Evolutionary biologists have long observed that the method tends to produce very high posterior probabilities for hypothetical evolutionary trees (very often 100%) even if the trees are clearly wrong.  

Our results provide a possible explanation for this unpleasant behavior. The implications of the results for the use of Bayesian model selection in testing opposing scientific hypotheses in general are yet to be explored. This study was supported by grants from the Biotechnological and Biological Sciences Research Council and the Natural Science Foundation of China. 

Classification of Bayesian model-selection problems involving two equally right or equally wrong models (Image by ZHU Tianqi)