Benchmarking Large Language Models Against Tumour Boards for Oncology Treatment Recommendations

BEACON is a prospective, multicentre, blinded benchmark using automated, criteria-based scoring. It is built on three design decisions that distinguish it from the existing literature: (i) synthetic, standardised cases remove the record-completeness variability that confounds retrospective comparisons and allow the identical input to be given to every board and every model; (ii) two independent tumour boards per localisation let human-to-human agreement be measured rather than assumed, and that agreement is the level against which model performance is read; and (iii) an independent panel, blinded to the source, adjudicates every discordance and every case-domain where the two boards diverge, so that clinically equivalent options are recognised as such and dangerous ones are named. Reference standard. For each case-domain, where the two boards select the same options their agreement is the reference; where they differ, the case-domain is classified EQUIPOISE, analysed separately and adjudicated blind. Five decision domains. Every recommendation is decomposed into D1 Intent, D2 Surgery, D3 Radiotherapy, D4 Systemic therapy (class and line), and D5 Work-up and biomarkers before any comparison. Deliberate silence on a domain is ticked on the form as an explicit category and is scored as a recommendation, never as missing data. Model selection. Models are chosen by a rule fixed before the first call: for each organisation, the most capable text model its API exposes on the freeze date, whether stable or in preview. Data collection. Tumour boards record their recommendations during the meeting itself, on a dedicated platform (https://etude-beacon.fr). Each board receives a single access link, with no account to create. Outside a session the platform shows case numbers only; a case is revealed when the board presents it, and that moment is time-stamped. The recommendation form must be saved before the session closes, failing which the case cannot be opened again. Case text cannot be selected or copied and carries a watermark with the board name and the time of display. At closure the coordinator attests that the rules of the study were followed during the session. Session opening times, case reveal times, intervals to submission and attestations are exported with the data.