Neurological diseases affect over 50% of the world population, which contribute to an increasingly needs of imaging examination for screening and diagnosis. The shortage of neuroradiologists led to the excessive workload, linked with increasing of error rates and decreasing of report quality. Inexperienced radiologists struggle to make accurate diagnoses for some neurological diseases, especially brain tumors, which may lead to additional delays and a potentially meaningless examination for patients.
Artificial intelligence (AI) has shown potential to be "a tireless resident", which can rival human performance in medical imaging interpretation and assist radiologists in multiple links of clinical diagnostic work. However, clinicians recognize that "Black-box" models lacking clinical utility and interpretability face significant barriers to real-world deployment. While the reasoning-enhanced AI model, trained by a hundred thousand data, can not only achieve accurate and efficient diagnosis but also but also address the lack of explainability, transparency, and trust.
Owing to the heterogeneity of multi-modal real clinical data and complexity of neurological diseases, whether the developed AI-assisted system can truly serve as an "AI resident" facing challenges and doubts. Clinical trial is the best approach to verify the performance of diagnosis and human-machine collaboration of AI model with a wide range of users and prospective data before being approved for clinical use. Key aspects of the study design have been established in conjunction with a multi-disciplinary scientific advisory board, consisting with experts in AI, radiology, neurology, and pathology, to ensure meaningful validation of reasoning-enhanced AI model towards clinical translation.
This clinic trial contains two sub-studies:
1. Clinical silence trial study for AI tools only:
There are three approaches may be used in this stage to compare the working performance of our AI model with radiologists and the other models.
First, a curated mini-dataset of totally 1,000 scans of brain tumor cases on MRI and cerebrovascular disease cases on brain CT will be publicly released. Teams can use this dataset to train or test on their AI models, and submit their trained algorithms or prediction results for evaluation. The performance of the uploaded models will be tested on the same testing cohort of us, and the uploaded prediction results will be compared with the generated content of us.
Second, a fine-tuned offline AI-assisted systems will be embedded in the picture archiving and communication system (PACS) in the radiology department after getting permission of medical affairs department and ethics committee in some medical centers. However, radiologists in these medical centers will not get AI-generated diagnoses and reports during this stage, while the imaging reports of brain MRI and CT generated by AI and radiologists will be recorded at the same time.
Third, prospective data of brain MRI and CT with corresponding imaging reports and gold standard diagnostic results will be collected from all over the world as the test data inputting to our AI model.
Finally, the consistency and accuracy of the report between AI models and radiologists will be compared, with respect to the histopathology and discharge diagnosis (\< 3 month) as reference.
2. AI-radiologist collaboration study:
In this stage, we aim to perform generalization verification on our reasoning-enhanced neuroimaging AI in the real clinical scenarios.
Over 50 international radiologists will be recruited to perform a diagnosis study using a subset of 400 scans from the testing cohort of us. The workflow including such steps in order: imaging reading, writing initial report, making diagnosis, reading AI reasoning, revising initial report, and evaluating content generated by AI. When writing reports, radiologists will be will be randomly assigned to two groups and asked to complete half of cases with AI-assisted tools and the other half of cases without, to ensure every case will be completing by two radiologists of the same level with and without AI assistance, respectively.
The first goal of this study is to estimate the performance of the average radiologist with or without AI as well as their working efficiency and confidence at neurological diseases diagnosis on CT and MRI.
The secondary goal is to evaluate the clinical viability and superiority of reasoning-enhanced AI on human-machine collaborative task. The accuracy and completion will be compared between revised reports with initial version written by radiologists, while radiologists' inclination and diagnostic confidence will be recorded as subjective evaluation indexes.
