NICE recognizes that the use of artificial intelligence (AI) methods, from relatively well-established machine learning approaches to newer and more complex generative AI, offers benefits such as processing and analyzing large datasets to reveal patterns and relationships that may not otherwise be readily apparent, and creating novel outputs based upon what is observed in the data.
NICE expects that evidence it considers will be informed by AI methods and published a position statement in August 2024. Anyone considering the use of AI in a submission should take careful note of NICE’s expectations. If history is any guide, these are likely to be picked up, at least as a starting point, by markets other than the UK that use HTA to determine the cost-effectiveness of treatments.
NICE recognizes that although using AI in HTA has benefits, it also has risks. These include algorithmic bias, reduced human oversight, and reduced transparency to non-experts. NICE considers that AI should therefore only be used when there is a clear rationale for doing so. Submissions should not lead with AI-generated data where there are alternatives. In these circumstances, AI output might be included as supplementary data. Because of their complexity and consequent lack of transparency, submissions should include validation of AI methods used and should aim to augment rather than replace involvement of people.
NICE encourages submitting organizations to discuss proposed use of AI methods in advance. The use of AI methods to estimate comparative treatment effects are potentially very influential and carry higher risks. Because of this, submission using these methods should include sensitivity analysis, be checked against other suitable methods, and the results should be triangulated with available clinical evidence. The guidance further states that use of machine-learning methods should be accompanied by pre-specified outcome-blind simulations, conducted independently, to demonstrate their statistical properties in similar settings.
Submissions should make the use of AI methods explicit and because of their risk and complexity, explain the methods used fully, to include reporting identified risks and the steps taken to mitigate these. This information should be presented to make it as accessible as possible to the non-expert. NICE’s position statement references a number of tools and approaches to enhance the explainability and transparency of AI methods:
- LIME (Local Interpretable Model-agnostic Explanations)
- SHAP (SHapley Additive exPlanations)
- Decision Trees
- Rule-Based Models
- Feature Importance
- Partial Dependence Plots (PDPs)
- Heatmaps
- Saliency Maps
- What-If Tool
- AI Explainability 360
Also referenced are several guidelines and checklists, including:
- The PALISADE Checklist, developed by ISPOR’s ML Task Force, which provides guidance for the application of ML in HEOR with a focus on transparency in methods development and findings
- TRIPOD+AI Checklist, an updated guidance to harmonize the reporting of studies involving prediction modelling.
- Algorithmic Transparency Reporting Standard (ATRS), developed by the UK’s Central Digital and Data Office and the Responsible Technology Unit.
- The Cochrane Dissemination Checklist and Guidance which aims to ensure that dissemination products are useful, accessible, and understandable.
The statement also suggests that submissions using AI methods should take account of NICE’s 2022 real-world evidence framework (ECD9).
Submitting organizations are responsible for ensuring that the methods they use are consistent with the UK Government framework for regulating AI and all applicable ethical, technical, scientific, and regulatory standards as well as relevant IP legislation.
NICE also points to the issue of cybersecurity, including manipulation of data and injecting malicious content into prompts and expects submitting organizations to provide evidence of the steps taken to ensure robust security.
The position statement suggests that while AI may augment a submission, it is unlikely to be a short-cut to a successful result. Using AI methods may get you ahead of the opposition but until they are more familiar to HTA bodies, their use will require particularly careful justification and explanation if any conclusions based on AI findings are going to be accepted by NICE.
As always, HTA submissions to NICE must be evidence based and properly supported. It is helpful that NICE has clearly stated its expectations on how AI can support a submission.
Contact us for support on the preparation of a NICE submission, with or without AI