Moscow's AI medical services rated
AI-based solutions are connected to the system that connects all radiology departments of Moscow polyclinics and hospitals - the unified radiology information service of the Unified Medical Information and Analytical System.
As part of the experiment, a funnel of intelligent algorithms was organised - the rules of selection, testing, validating and commissioning AI-based services in order to apply the highest quality solutions in the clinical practice.
Currently, 19 services for the diagnosis of oncological diseases (lung and breast cancer), Covid-19, osteoporosis and coronary heart disease are connected to the main circuit of the system, which is used by all radiologists of urban health facilities.
The results of AI work in the field of medicine in the world are commonly evaluated according to three parameters: sensitivity, specificity and diagnostic accuracy of the service.
Based on lessons learned during the experiment, Moscow specialists supplemented the existing parameters.
Rating
The rating was initiated by the Moscow Centre for Diagnostics and Telemedicine, which organised the experiment.
The rating includes assessment not only of the service, but also of its adaptability to the unified city information system and the ability to work with large research flows.
In the future, such a rating scale may be useful for the global scientific community to scale the obtained results and implement evolving AI algorithms.
The new methodology for evaluating AI includes comparing service results with radiologists' protocols, the doctor's agreement with the localisation of pathological findings, the proportion of examinations without defects and the speed of data processing.
Some of the parameters are evaluated automatically and others are manually verified by a group of doctors - experts from the Diagnostic and Telemedicine Centre.
For each criterion, the service receives a certain number of points, which are added up to a final score.
Services for automatic analysis of radiology research have been tested in the city's healthcare system since spring 2020.
The publication will help to stimulate competition between AI developers to develop promising technologies and objectively evaluate existing solutions before implementation in the healthcare system of other regions
