Initial metrics on model performance
Volv, Sanofi and Optimum Patient Care (OPC): collaborating for people living with disease
Volv, supported by Sanofi, and leveraging the data and expertise of Optimum Patient Care in the UK, is creating a unique collaboration that does not stop here.
The first phase of this project was to collaborate to build new types of models for two rare diseases: Fabry and Pompe. To do this, we focussed on primary health care records, i.e. the records that general practitioners use.
Both diseases are difficult to diagnose for primary care clinicians, and as a result, remain underdiagnosed. For Pompe disease in the UK, it is estimated that 50% of people with the disease are not being diagnosed, leading to a longer delay until they eventually do get diagnosed. This data is managed by Optimum Patient Care UK, which provides de-identified data, of around 18 million patient records, for research purposes. Data security and protection are paramount. This means that the data remains anonymous and secure during the disease model development process.
The data complies with:
The first phase of the inTriguemethodology involved an iterative process of finding a way to determine what makes patients with Fabry and Pompe disease stand out from all other patients. We used a combination of data science (or AI) approaches to get to a list of patients that plausibly have a disease.
Within this phase, crucially and differentiatingly, we also needed to validate whether the approach has worked by checking the inTrigue results with an expert clinician. We did this with a consultant in a specialist Fabry and Pompe department in a UK teaching hospital. The results of this evaluation can be seen in the results section.
Once the clinician's validation was complete, we then take those inputs and optimise the algorithm, which will again boost the performance. Once this is done, we are ready to move to Phase 2.
In this second phase, the algorithm is applied to the data, and clinicians are asked if they want to participate in the model deployment programme. The clinicians need to give their consent to be part of this quality improvement (QI) programme. Several QI programmes are already in place and if they agree, they can then check to see if any of the patients in their practice are at risk of these diseases. This is done through the remote installation of reports in the GP system. We can then monitor to see if there is an improvement in terms of quality of clinical care.
More results on this aspect of the deployment of the models will be published at a later stage, but the optimisation steps post clinician validation shows significant improvement on these results presented here.
After this programme, consideration is being given to deploying the models more widely by embedding them into GP systems nationwide.
Initial metrics on model performance
Model performance: Fabry disease in UK
Use model learned via Algorithm SLSL to find undiagnosed FD patients in OPCRD EHR database GP-EHR-DB-UK (18M patients).
Request that FD specialist practicing in UK review EHRs of top 50 candidate patients (candidates have predicted probabilities exceeding FD threshold FD).
Results are very promising showing that out of 50 patients the top 25 have a precision of 88%, and when the total 50 patients are considered the precision remains high at 76% using the precision@k metric.
Model performance: Pompe disease in UK
Use model learned via Algorithm SLSL to find undiagnosed PD patients in OPCRD EHR database GP-EHR-DB-UK (18M patients).
Request that PD specialist practicing in UK review EHRs of top 30 candidate patients (candidates have predicted probabilities exceeding PD threshold xPD).
The results for Pompe are also very promising showing that out of 30 patients the top 20 have a precision of 80%, and when the total 30 patients are considered the precision remains high at 73% using the precision@k metric.
In the refinement steps following the clinical review phase, we see a significant performance boost. However, these results tend to have an upward bias, so we do not report them here. Instead, these results will be tested out as we deploy the models in this next phase of the programme.
