Artificial Intelligence Integration in Breast Cancer Screening: The ANIHTA Study

Breast cancer screening can identify malignancies before they become clinically apparent and support earlier diagnosis and treatment. In organized programs, this benefit must be balanced against false-positive findings, unnecessary recalls and diagnostic procedures, interval cancers, overdiagnosis, radiologist workload, and workforce availability. These issues are especially relevant in double-reading programs, where two radiologists interpret the same mammogram and discrepant readings may require arbitration. Artificial intelligence (AI) systems for mammography can estimate the likelihood of malignancy and highlight suspicious regions. Their potential value extends beyond stand-alone diagnostic accuracy: depending on how AI output is incorporated into the pathway, it may affect additional human review, management of discordant readings, recall to second-level assessment, and radiologist workload. The study therefore evaluates AI as part of a workflow rather than only as an isolated classifier. The ANIHTA project ("Introducing artificial intelligence in personalised breast screening programs: a Health Technology Assessment") was designed to evaluate the integration of AI into an organized breast cancer screening program and to generate evidence on clinical performance, organizational implications, economic sustainability, and implementation. The present ClinicalTrials.gov registration concerns the observational clinical evaluation based on routinely collected screening data and a retrospective target trial emulation. The originally proposed prospective randomized component is not part of the current study design. The central clinical question is whether routinely available AI information can be used to construct a more selective mammography review pathway in which an additional radiologist is involved only for examinations meeting predefined AI-based criteria. Rather than prospectively modifying the care received by screening participants, the study will reconstruct this alternative pathway retrospectively. This approach makes it possible to compare the observed screening strategy with a prespecified AI-guided strategy while preserving the clinical care that actually occurred. The study will be conducted within the organized breast cancer screening program of Local Health Authority Roma 2 (Azienda Sanitaria Locale Roma 2), Italy. The program serves a large population and routinely invites women in the target screening age range to undergo mammography. The screening infrastructure includes multiple territorial mammography facilities and second-level diagnostic centers, with centralized digital image storage and access through the local picture archiving and communication system. In accordance with the organized screening workflow, first-level mammograms are interpreted by breast radiologists using double reading. The reading process and subsequent management are documented in the screening information system. Depending on the radiological assessments, a woman may complete the screening episode without additional testing or may be referred to second-level diagnostic assessment. Follow-up information is available through the screening program and linked clinical sources. Lunit INSIGHT MMG has been integrated into the digital mammography environment of the screening program. The software uses deep-learning methods to analyze mammographic images, generate a malignancy-related AI score, and identify suspicious areas. In routine practice, the AI output is available as decision support for radiologists; it does not replace the radiologist's interpretation. The study will use AI scores and radiological readings that are available from the screening pathway, together with retrospectively generated or harmonized AI information where required by the study dataset. The study is intended to evaluate AI within this real-world organized screening context. Consequently, the unit of clinical interest is the screening examination and its associated screening episode, while data splitting and validation procedures will be performed at the woman level to avoid information leakage across analytic datasets when a woman contributes more than one examination over time. This is a retrospective observational cohort study with diagnostic-accuracy, workflow-evaluation, and target trial emulation (TTE) components. The cohort will be assembled from approximately 200,000 eligible mammography screening examinations collected over about six years within the Local Health Authority Roma 2 program, with follow-up sufficient to characterize screen-detected and interval cancers. The first component will characterize the AI score and standard diagnostic-accuracy measures in the observed data, including clinically relevant subgroups. The principal comparative component will use a TTE framework representing the hypothetical prospective comparison of two alternative workflow strategies. The observational dataset will be structured to emulate this comparison using information available at the relevant screening decision points. The two strategies are defined before evaluation in the hold-out validation dataset. Strategy A represents the observed routine screening workflow. Strategy B represents an emulated AI-guided workflow in which predefined AI score thresholds can trigger review by an independent third radiologist in specific reading scenarios. Strategy B will be reconstructed retrospectively and will not alter the screening decision originally communicated to the woman. The emulation will explicitly distinguish between observed information and counterfactual information. Outcomes under Strategy A are derived from the actual screening pathway. Outcomes under Strategy B are reconstructed according to prespecified rules using the original first- and second-reader assessments, the AI score, and, when triggered, an offline third-reader assessment. Because Strategy B is not implemented prospectively, its downstream consequences are modelled or reconstructed counterfactual outcomes and will be interpreted as such. Strategy A corresponds to the screening workflow actually observed in the study cohort. Mammograms are assessed by the two radiologists participating in the routine double-reading pathway, with the AI system available as decision support. The clinical screening result, recall decision, subsequent diagnostic assessment, biopsy referral when applicable, and final outcome are those recorded in routine care. No additional third reader is systematically activated on the basis of the prespecified AIpt or AInt thresholds under Strategy A. Therefore, Strategy A represents the reference pathway against which the alternative AI-guided workflow will be evaluated. For the target trial emulation, the relevant observed data will be aligned to a common index time corresponding to the screening examination and the associated reading process. Baseline and screening characteristics available before or at that time will be used to describe the cohort and assess comparability of the analytic populations. Strategy B is an alternative workflow reconstructed retrospectively. It preserves the two-reader structure of organized screening but uses predefined AI thresholds to identify selected examinations for an additional independent radiological review. Two AI thresholds will be used. The AI positivity threshold (AIpt) is intended to identify examinations with a sufficiently high AI malignancy score to warrant additional review in a scenario in which the routine radiological readings would otherwise support a negative screening result. The AI negativity threshold (AInt) is intended to identify examinations with a sufficiently low AI malignancy score in the setting of disagreement between the two routine radiologists, where an additional review may help determine whether recall is necessary. The exact values of AIpt and AInt will be estimated only in the derivation dataset and will then be locked before application to the validation dataset. The emulated workflow will combine the first-reader result, second-reader result, and AI score. For concordant negative readings, the AI score will be compared with AIpt. Examinations meeting the escalation rule will be routed, within the emulation only, to offline third-reader assessment, whose result will be used to reconstruct whether the examination would have remained negative or resulted in recall under Strategy B. For discordant first- and second-reader results, the AI score will be evaluated against the predefined AInt rule. Examinations meeting this criterion will undergo offline third-reader review to reconstruct the Strategy B decision and evaluate selective AI-guided arbitration. For examinations with concordant positive readings by the first and second radiologists, the emulated strategy will preserve referral to second-level assessment without requiring AI-triggered third-reader arbitration. Thus, the alternative strategy is not designed to replace clearly concordant radiological decisions but to use AI information at specific points where additional review could plausibly change workflow efficiency or diagnostic performance. All rules used to assign the emulated workflow will be prespecified before evaluation of the primary outcome in the validation dataset. The analysis will report both the number and proportion of examinations for which Strategy B differs from Strategy A and the number and proportion of examinations that would require third-reader review. The third-reader component is a research procedure performed offline and does not constitute an additional clinical reading affecting patient care. The independent third radiologist will review only examinations selected according to the prespecified Strategy B rules. The third reader will not modify the original screening report, will not contact participants, and will not trigger clinical recall, biopsy, treatment, or any other patient-management action. The offline review will be conducted in a silent mode with respect to routine clinical care. The third reader will assess the mammographic examination independently for the research reconstruction, with the AI score available as specified in the study procedure. The original patient management remains unchanged regardless of the third-reader assessment. The third-reader result will be incorporated into the emulation variables used to reconstruct the alternative pathway. These variables will identify whether an examination met the AIpt or AInt criterion, whether third-reader review was triggered, the third-reader assessment, the emulated recall decision, and the resulting emulated workflow classification. Maintaining separate observed and emulated variables will allow the study to distinguish actual clinical outcomes from counterfactual workflow decisions. A central design feature is the separation of threshold development from evaluation. The analytic cohort will therefore be divided into a derivation dataset and an independent hold-out validation dataset. The split will be performed at the woman level rather than the examination level. This prevents mammograms from the same woman from appearing in both datasets and reduces the risk of information leakage caused by correlated repeated screening examinations. The split will preserve the distribution of key characteristics, including cancer status, screening period, screening site, screening round, and mammographic density, and will be documented before validation outcome evaluation. The derivation dataset will be used to describe AI score distributions, estimate diagnostic performance, and determine the AIpt and AInt values according to the prespecified clinical objectives. AIpt will be selected with an emphasis on identifying examinations with a high probability of malignancy in the relevant escalation scenario, whereas AInt will be selected with an emphasis on identifying examinations with a low probability of malignancy in the relevant discordant-reading scenario. Sensitivity, specificity, positive predictive value, negative predictive value, and the consequences for recall and arbitration workload will be considered when characterizing candidate thresholds. Because threshold selection is part of workflow development, derivation-set estimates will not be treated as primary comparative evidence. Once AIpt and AInt are chosen, both thresholds and the complete Strategy B rules will be frozen, with no further outcome-driven optimization in the validation dataset. The validation dataset will be used to compare Strategy A with Strategy B using the locked rules. This separation is intended to provide a more credible estimate of how the proposed workflow would perform when applied to data not used to define its decision thresholds. Clinical and screening information will be obtained primarily from the regional screening information infrastructure used by the Local Health Authority Roma 2, including the SIPSO screening database (Sistema Informativo Prevenzione Screening Oncologici), together with mammography image systems and linked clinical information required to ascertain diagnostic outcomes. The study dataset may include the screening date, screening site, screening round, age at screening, mammographic density when available, first- and second-reader interpretations, AI score, screening result, recall status, second-level diagnostic assessment, biopsy referral, histopathological confirmation, breast cancer diagnosis, and follow-up information. Tumor-related information may include whether the cancer is invasive or ductal carcinoma in situ and, where available, tumor size, histologic grade, lymph-node involvement, and molecular or pathological characteristics. Cancer outcomes will be ascertained from the screening program, clinical records, pathology information, and relevant cancer-registry or follow-up sources. Cross-referencing will improve completeness and resolve discrepancies, with particular attention to interval cancers after a negative screening episode. The Lunit INSIGHT MMG examination-level score will be analyzed continuously for diagnostic-accuracy analyses and according to prespecified thresholds for workflow emulation. Radiologist interpretations and AI outputs will remain distinct variables to evaluate agreement, discordance, and incremental AI information. The research dataset will be pseudonymized for statistical analysis. Linkage keys and directly identifying information will be managed according to institutional procedures and applicable data-protection requirements. Analytic datasets will contain only the variables required for the scientific objectives of the study. The primary comparative endpoint will be the difference in cancer detection rate (CDR) between Strategy B and Strategy A in the hold-out validation dataset. CDR will be expressed using the prespecified screening denominator, including presentation per 1,000 screening examinations where appropriate. The primary effect measure will be the absolute difference between the emulated AI-guided strategy and the observed strategy, accompanied by a 95% confidence interval. The study will also evaluate several secondary outcomes that characterize the clinical and organizational consequences of the alternative workflow. These include recall rate, biopsy referral, positive predictive value of recall, false-positive findings, interval cancers, the proportion of examinations requiring AI-triggered third-reader arbitration, and measures of radiologist workload. These outcomes are intended to show whether any change in cancer detection is accompanied by an increase or reduction in unnecessary assessment and how much additional human reading would be required to implement the alternative strategy. Interval cancers will be identified using the follow-up definition specified in the study protocol and available screening data. They will be incorporated into diagnostic-accuracy analyses to avoid evaluating performance only among cancers detected during the immediate screening episode. Additional descriptive outcomes will include distributions of AI scores according to final cancer status, patterns of agreement between radiologists and AI, the frequency of the different first-reader/second-reader combinations, and the proportion of examinations affected by each AI threshold. These analyses will help explain the mechanisms through which Strategy B differs from the observed pathway. The diagnostic performance of the AI score will be assessed independently of the workflow comparison. Receiver operating characteristic (ROC) curves will be generated to describe discrimination across the full range of AI scores. The area under the ROC curve (AUC) and corresponding 95% confidence intervals will be estimated. Where comparisons between correlated diagnostic assessments are required, appropriate methods for paired ROC data will be used. Sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) will be calculated at clinically relevant score thresholds, including the final AIpt and AInt values. Because predictive values depend on disease prevalence, results will be interpreted within the screening population from which the cohort was drawn. The analysis will distinguish screen-detected cancers from cancers occurring after a negative screening result when follow-up permits this classification. Interval cancers will contribute to the evaluation of false-negative screening outcomes according to the protocol definition. This is important because an AI strategy that appears accurate only at the initial screening episode could still be clinically inadequate if relevant cancers emerge during the subsequent screening interval. The relationship between AI score and cancer characteristics will be explored when detailed tumor information is available. Analyses may examine whether AI scores differ for invasive cancers and ductal carcinoma in situ, and according to tumor size, grade, nodal status, or other pathological features. These analyses are exploratory and will be interpreted as complementary to the primary workflow comparison. The TTE will operationalize the eligibility conditions, strategies, time zero, follow-up, outcomes, and analysis plan of the hypothetical target trial using the retrospective cohort, making the comparative question explicit. Eligibility for the emulated comparison will be determined using information available at the index screening episode. Strategy assignment is conceptual rather than randomized: each eligible examination will be evaluated under the observed Strategy A and, where the required information is available, reconstructed under Strategy B using the locked rules. Because the study does not actually randomize women, the assumptions underlying the emulation and the distinction between observed and counterfactual decisions will be made explicit in interpretation. The primary validation analysis will calculate CDR under both strategies and estimate the between-strategy difference with a 95% confidence interval. Secondary comparative analyses will estimate corresponding differences in recall, biopsy referral, false positives, PPV of recall, interval-cancer classification where applicable, arbitration rate, and workload-related measures. Paired methods will be used where appropriate because both strategies are evaluated on the same examinations. Confidence intervals will account for pairing and, when necessary, repeated examinations from the same woman. The TTE will not assume that every downstream event under Strategy B is directly observed. When Strategy B changes the hypothetical screening decision, subsequent consequences may depend on assumptions or reconstructed information. The study will therefore distinguish endpoints that can be directly reconstructed from available data from those that require stronger counterfactual assumptions. Results will be reported with this distinction so that the precision of statistical estimates is not confused with certainty about the emulated causal pathway. Prespecified or clinically motivated subgroup analyses will examine heterogeneity according to mammographic density, screening round, age group, screening period, and screening site, subject to data availability and adequate sample size. Mammographic density is of particular interest because dense breast tissue can reduce mammographic sensitivity and may influence both radiologist interpretation and AI performance. Diagnostic measures and workflow consequences will therefore be summarized across density categories when density information is sufficiently complete. Screening round may distinguish prevalent from subsequent screening examinations and may affect cancer prevalence, prior-image availability, recall behavior, and expected diagnostic performance. Analyses by screening round will be used to assess whether the proposed thresholds behave consistently across different screening contexts. Temporal analyses may be performed to evaluate whether changes in technology, reader experience, screening practice, or the implementation of AI within the local program are associated with differences in score distributions or diagnostic performance. Site-level analyses may similarly be used to assess whether results are robust across facilities within the screening network. Subgroup results will primarily assess consistency and potential heterogeneity. Confidence intervals and sample sizes will be reported to avoid overinterpretation of small subgroups. Before analysis, completeness, plausibility, and logical consistency will be assessed for key variables including radiologist readings, AI scores, screening results, follow-up, cancer diagnoses, and subgroup variables. When information required to establish the primary screening outcome can be recovered from another validated source, cross-referencing will be used to resolve missing or discordant records. Examinations lacking essential information required to define the observed or emulated pathway may be excluded from the relevant analysis if the missing information cannot be reliably reconstructed. The primary analysis will use the prespecified complete and evaluable dataset required to define both strategies and the primary outcome. The pattern and extent of missing data will be described. Sensitivity analyses will assess the robustness of the findings to alternative assumptions about missing information. Where statistically and clinically appropriate, multiple imputation may be considered for covariates or secondary variables; variables that define the emulated intervention itself will not be imputed in a manner that creates unobserved treatment rules without justification. Potential bias may arise from incomplete follow-up, calendar-time changes, reader variation, and missing images or AI scores. The derivation/validation split, standardized definitions, threshold locking, outcome linkage, and sensitivity analyses are intended to reduce these risks. Because the emulated strategy relies on offline third-reader assessments for selected examinations, inter-reader variation is another potential source of uncertainty. The third-reader procedure will therefore be standardized and documented. Analyses may quantify the frequency with which third-reader decisions alter the emulated pathway and explore the robustness of findings to alternative classifications when appropriate. A central purpose of the study is to evaluate AI as part of a screening workflow rather than as a stand-alone classifier. Organizational outcomes will therefore be analyzed alongside diagnostic outcomes. The number and proportion of examinations triggering third-reader review under Strategy B will be used as a direct measure of arbitration workload. This will be compared with the volume of readings and recalls occurring in the observed pathway. The analysis will examine whether selective use of a third reader could concentrate additional radiological effort on examinations in which AI and the original readings indicate a higher potential value of further review. Workload indicators may include the number of first-, second-, and third-reader assessments required under each strategy, the number of examinations routed to second-level assessment, and the number of additional reviews required per cancer detected or per change in screening disposition. These measures will be interpreted together with CDR and false-positive outcomes because workload reduction alone would not represent an improvement if achieved at the expense of clinically important cancer detection. The organizational analysis will also consider the practical implications of introducing threshold-based routing into an existing screening information system, including the need for workflow rules, availability of radiologists for arbitration, compatibility with image and data systems, monitoring indicators, and training requirements. These elements will contribute to the broader assessment of whether the proposed strategy could be implemented sustainably in routine organized screening. The clinical study is embedded within a broader Health Technology Assessment (HTA) of AI-supported breast cancer screening. The purpose of the HTA is to integrate evidence on clinical effectiveness with organizational, economic, ethical, legal, and implementation considerations relevant to the Italian National Health Service. Clinical effectiveness and safety inputs will be derived primarily from the retrospective cohort, diagnostic-accuracy analyses, and target trial emulation described above. The HTA will use these results to characterize the expected clinical consequences of alternative AI integration strategies rather than relying only on the stand-alone accuracy of the algorithm. The economic component will examine AI-related resources, radiologist time, diagnostic assessment, and downstream resource use using study data and relevant costing sources. Cost-effectiveness and budget-impact perspectives may be developed where suitable inputs are available, with uncertainty explicitly represented. Organizational analyses will address workload, arbitration, workflow efficiency, data infrastructure, training, and implementation requirements. Ethical and legal considerations will include human oversight, transparency, data protection, and the use of algorithmic information for routing decisions. The HTA is intended to prevent implementation decisions from being based on a single diagnostic metric by considering clinical outcomes together with recalls, workload, resources, costs, and implementation complexity. A separate ANIHTA component will investigate preferences, acceptability, trust, and perceived utility of AI-supported screening among healthcare professionals. It is distinct from the retrospective cohort of screened women and does not modify their clinical management. Preferences will be investigated through a Discrete Choice Experiment (DCE) in approximately 75 healthcare professionals, in which respondents choose between hypothetical implementation alternatives defined by selected attributes. Repeated-choice models, including mixed logit methods where specified, will estimate trade-offs and preference heterogeneity. DCE findings will not define the primary clinical endpoint or modify the locked AI thresholds; they will provide complementary implementation evidence for the HTA. The observational clinical component uses retrospective screening data and offline research readings. No participant will be assigned prospectively to an experimental screening pathway as part of this study, and the emulated Strategy B will not alter the clinical result of any historical screening examination. The AI-triggered third-reader review is performed solely for research. A third-reader opinion generated for the emulation will not result in participant recontact, recall, biopsy, treatment, or retrospective alteration of the medical record. The study therefore evaluates what the alternative workflow might have produced without exposing participants to an experimental routing decision. Data will be handled under applicable institutional and data-protection requirements. Analytic data will be pseudonymized, access to identifiable linkage information will be restricted, and processing will follow approved European and national requirements. The study also recognizes that algorithmic performance can vary across populations and screening contexts. Reporting will therefore emphasize the characteristics of the study population, the local workflow in which the AI system is evaluated, and the distinction between internal validation of the emulated strategy and broader generalizability to other screening programs. The study is designed to answer a workflow question rather than only an algorithm-performance question: whether predefined AI information can identify situations in which additional radiologist review may be most useful while controlling recalls, false positives, biopsy referrals, and workload. The derivation/validation separation and locked-threshold evaluation are intended to reduce optimism from selecting and testing thresholds in the same data. The TTE framework makes the Strategy A versus Strategy B comparison explicit and distinguishes directly observed quantities from counterfactual reconstructions. Findings will be interpreted within the limitations of a retrospective observational emulation. Historical data can show how the prespecified rules would have changed screening decisions, but cannot fully reproduce behavioral, organizational, and downstream effects of prospective implementation. By combining diagnostic performance, workflow consequences, workload, and HTA-relevant evidence, ANIHTA aims to inform further evaluation and possible implementation of a selective AI-guided third-reader pathway.