Overrepresentation Bias Leads to Performance Overestimation in Blood−Brain Barrier Permeability Prediction Models: Characterization and Mitigation
My eye was drawn directly to the title of this paper and I was not disappointed. Before reading, it had already brought up memories from 20 years ago when I worked in large pharma as a medicinal chemist. I was told the new thing to watch out for was phospholipidosis, but usefully there was a model to predict this. I checked out the model and it was poor because the thousand or so compounds used to build it were from one project and one chemical series, so for my compounds the prediction was meaningless. This is how I was drawn to the title of this paper – just what effect do you get when a dataset is dominated by large quantities of similar compounds? I was not disappointed, this is a great paper.
The cheminformatics field has had some 6 to 8 years of Machine Learning (ML) model building, with Deep Learning (DL) being the more recent addition to the likes of tree methods like Random Forest and Support Vector. Some of the great and the good, in this time frame, have already highlighted the problem of comparing one model to another. Especially when many have claimed their model is superior to others.
First, the authors took a dataset and set out to build models in an area we care about in drug discovery; Blood Brain Barrier (BBB) penetration. We have found that many of the new cancer drugs are not brain penetrant so ineffective at treating secondary tumours, for example many breast cancer patients experience metastasis to the brain and poor prognosis.
Second, they set about to build models with the whole dataset (Random) using multiple compound descriptors and multiple ML methods with additions like gradient boosting – thus allowing a fair comparison of techniques.
Third, they describe new methodology to remove compounds that are highly similar to other compounds and data points that are highly similar. This produced compound datasets with a more even distribution of compound structures.
Fourth, they chose two methods to assess the quality of the models and fifth, they chose methods to determine predictions for in domain and out of domain compounds. As you can imagine the table of results are quite comprehensive and this really shows the quality of the work. All the methods are well described and should be reproducible. One short coming – they did not state the name of the GitHub repo – it is here https://github.com/bdslab-upv/bbbp-overrepresentation-bias
The full dataset, when used to build ML models, produces excellent models as judged by the both assessment methods. Both methods use the classic split method of dividing the dataset into a build dataset and a test dataset. Thus, how does the model predict the test dataset. Here is the flaw in the method, if the full dataset is dominated by several chemical series and not highly diverse, within the test set there will be compounds very similar to those on which the model is built. Thus, any prediction is just selecting the result of the most similar compound – what is now being called ‘memorising’ stuff. The models built on the datasets without the bias of similar compounds and data did not perform as well. This is comprehensively shown across the broad spectrum of descriptors and methods used. All models also did not fare well with out of domain predictions – this is not a surprise. The authors offer this methodology as a fairer way of comparing model builds. Interestingly they showed the DL model did not fare better that the tree method methods once the correction had been applied. This tends to indicate that many claims of superiority with DL may not be true.
It is a reality in drug discovery projects that different issues occur with different series of compounds with the addition of different goals. Not everyone is trying to get BBB penetration thus some compounds are never tested – we simply do not know if they are good or bad. The pragmatic flow of project data gathering, leading to just one compound that matters, means that datasets are incomplete from the perspective of building great models. Add in synthetic availability and it is no wonder we have datasets of similar compounds with similar results. The work in this paper also clearly shows, and they go to pains to point this out, that the techniques are relevant for measuring the performance (or quality) of the model build when comparing to others. In other words, if you want the best model of BBB penetration then build the model with as much data as possible, just be careful how you compare it to others.
If you would like to read more, click here
