Artificial Intelligence/ Machine Learning Medical Device Regulatory Handbook by Cooley's life sciences and healthcare regulatory team Cooley AI/ML-Based Medical Device Regulatory Handbook 1 Even before the Biden administration issued its landmark executive order on October 30, 2023, establishing new standards for artificial intelligence (AI) safety and security,1 the US Food and Drug Administration (FDA) had been exploring how to regulate medical devices that incorporate AI and machine learning (ML) and evaluating the role of AI/ML in medical product development.2 FDA has sought to strike a balance between facilitating patient access to innovative medical device technologies and providing oversight in a manner that adequately protects public health. This handbook outlines the key issues to consider as device software manufacturers attempt to navigate this complex regulatory area. 1 The executive order "establishes new standards for AI safety and security, protects Americans' privacy, advances equity and civil rights, stands up for consumers and workers, promotes innovation and competition, advances American leadership around the world, and more." 2 See, e.g., "Proposed Regulatory Framework for Modifications to Artificial Intelligence/Machine Learning (AI/ML)-Based Software as a Medical Device (SaMD) - Discussion Paper and Request for Feedback." See also "Artificial Intelligence/Machine Learning (AI/ML)-Based Software as a Medical Device (SaMD) Action Plan" and "Using Artificial Intelligence & Machine Learning in the Development of Drug & Biological Products - Discussion Paper and Request for Feedback." Artificial Intelligence/Machine Learning Medical Device Regulatory Handbook This handbook is for general information purposes only. It is not intended to be, and should not be taken as, legal advice. Cooley AI/ML-Based Medical Device Regulatory Handbook 2 I. Overview of the regulatory landscape of AI/ML-based software as a medical device (SaMD) A. What is AI/ML? FDA defines AI as "the science and engineering of making intelligent machines, especially intelligent computer programs," that "can use different techniques, including models based on statistical analysis of data, expert systems that primarily rely on if-then statements, and machine learning."3 As an AI technique and subset of AI, ML "can be used to design and train software algorithms to learn from and act on data."4 ML can be used to create a "locked algorithm" (meaning the software function does not change) or "adaptive algorithm" (meaning the software function can change over time based on new data).5 Other government agencies have defined AI differently. The Centers for Medicare & Medicaid Services (CMS) recently defined AI as "a machine-based system that can, for a given set of human-defined objectives, make predictions, recommendations, or decisions influencing real or virtual environments," based on the use of such definition in the National Artificial Intelligence Initiative Act of 2020.6 Although different terms are used to describe AI/ML-based software programs, generally a technology can be classified as AI/ML-based if sponsors use expressions such as "deep learning," "black box," "machine learning," "deep neural networks," and/or "artificial learning" to describe their technology.7 3 FDA, Artificial Intelligence and Machine Learning in Software as a Medical Device. 4 Id. 5 Id. 6 See CMS, "Frequently Asked Questions related to Coverage Criteria and Utilization Management Requirements in CMS Final Rule (CMS-4201-F)," February 6, 2024. 7 See Benjamens, Stan, et al., "The state of artificial intelligence-based FDA-approved medical devices and algorithms: an online database," Nature Partner Journal/Digital Medicine (2020) 3:118. Cooley AI/ML-Based Medical Device Regulatory Handbook 3 B. Software in a medical device versus software as a medical device AI/ML technology can be applied to software in a medical device (SiMD) - namely, software that is part of a hardware medical device, such as software used to "drive or control" the motors and the pumping of medication in an infusion pump. AI/ML technology also can be applied to software as a medical device (SaMD) - namely, "software intended to be used for one or more medical purposes that perform these purposes without being part of a hardware medical device," such as software that allows a smartphone to view images obtained from a magnetic resonance imaging (MRI) medical device for diagnostic purposes. C. FDA pathways: 'Teaching an old dog new tricks' The statutory standard for premarket review of medical devices was not designed with AI/ML products in mind. Rather, applying a risk-based framework, FDA must determine what regulatory controls are necessary to provide a "reasonable assurance of safety and effectiveness" of the device, including any modifications to medical devices that could "significantly affect" the safety or effectiveness of the device.8 Because FDA's existing regulatory paradigm for medical devices was not designed for adaptive AI/ ML technologies, many of the AI/ML-driven software products authorized by FDA have used the de novo pathway. De novo is the pathway for the manufacturer to receive marketing authorization for novel low-to-moderate risk SaMDs.9 This is in contrast to either the more common 510(k) pathway, whereby a device can receive "clearance" by FDA after submission of a premarket notification, often simply referred to as a "510(k)," so named for the provision of the Federal Food, Drug, and Cosmetic Act (FDCA) that codifies this process, and the more laborious premarket approval (PMA) pathway that requires a full showing of safety and effectiveness before FDA can approve the application.10 8 21 CFR § 807.81(a)(3). 9 21 USC § 360c(f)(2). 10 21 USC §§ 360(k), 360c(a)(1)-(2). Cooley AI/ML-Based Medical Device Regulatory Handbook 4 D. Predetermined change control plans: The new frontier? Because the "reasonable assurance of safety and effectiveness" standard is not absolute, FDA applies the standard to AI/ML-based SaMD using a risk-based approach. FDA recognizes that "[o]ne of the greatest benefits of AI/ML in software resides in its ability to learn from real-world use and experience, and its capability to improve its performance"11 over time, but the regulatory paradigm was not designed to allow for such realtime change. Under the law, each time an algorithm makes a change that could significantly affect the safety or effectiveness of the device, technically a new 510(k) submission would be required.12 In reviewing "adaptive algorithms," which generally present higher risks than "locked algorithms," and understanding the limitations of the current regulatory framework, FDA created a mechanism called a "predetermined change control plan" (PCCP) to account for the evolving nature of adaptive algorithms. The PCCP approach permits future modifications to SaMD within certain parameters - without the need for premarket submissions - while still ensuring that the regulatory standard is met. Congress recently provided FDA with the authority to approve PCCPs as part of premarket review of AI/ML-based devices in section 3308 of the Food and Drug Omnibus Reform Act of 2022 (FDORA).13 While implementing this new authority, FDA issued draft guidance on PCCPs in April 2023 that "proposes a science-based approach to ensuring that AI/ML-enabled devices can be safely, effectively, and rapidly modified, updated, and improved in response to new data."14 Use of PCCPs, as proposed in the draft guidance, would accelerate innovation and enable more personalized medicine by "put[ting] safe and effective advancements in the hands of health care providers and users faster."15 A PCCP can decrease the regulatory burden on sponsors developing AI/ML-enabled 11 FDA has stated that "AI and ML technologies have the potential to transform health care by deriving new and important insights from the vast amount of data generated during the delivery of health care every day," and that "[m]edical device manufacturers are using these technologies to innovate their products to better assist health care providers and improve patient care." 12 21 CFR § 807.81(a)(3). 13 FDORA was enacted as part of the Consolidated Appropriations Act, 2023, Pub. L. No. 117-328 (2022). 14 FDA press release, "CDRH Issues Draft Guidance on Predetermined Change Control Plans for Artificial Intelligence/Machine Learning-Enabled Medical Devices," March 30, 2023. 15 Id. Cooley AI/ML-Based Medical Device Regulatory Handbook 5 device software functionality because the sponsors can avoid the need to prepare submissions to FDA for changes covered by the PCCP. Furthermore, the PCCP approach is a necessity for adaptive ML products involving frequent postmarket learnings and automatic updates. Under the PCCP framework, such products can continuously retrain themselves based on new data and feedback and make updates "on the go." This iterative process allows the products to rapidly evolve and become more useful, consistent with the regulatory parameters. In contrast, models that require FDA review for each update may face delays and hinder the ability to provide the most up-to-date and impactful functionality. Such constant retraining may result in numerous daily updates - which is impractical, if not impossible - without the PCCP approach to regulation. Based on FDA's guidance document concerning PCCPs, sponsors providing a premarket submission to FDA would need to include: 1. A detailed description of the specific, planned device modifications. 2. A description of the methodology that would be used to develop, validate and implement those modifications, including a description of how necessary information about these modifications will be clearly communicated to users. 3. An assessment of the benefits and risks of the planned modifications.16 II. What have you done for me lately?: Recent FDA AI/ML authorizations Using the risk-based approach discussed above, the FDA Center for Devices and Radiological Health (CDRH) has granted marketing authorization17 for more than 950 AI/ML-based medical devices, and the number continues to grow almost daily.18 Table 1 provides a flavor of the range of AI/ML-based SaMDs that have received FDA's marketing authorization. 16 Id. 17 The term "marketing authorization" broadly includes approval of a premarket approval application, granting of a de novo authorization request and clearance of a 510(k). 18 FDA, "Artificial Intelligence and Machine Learning (AI/ML)-Enabled Medical Devices." Cooley AI/ML-Based Medical Device Regulatory Handbook 6 Table 1 - Examples of AI/ML-based authorizations Device name Company Brief description Premarket pathway Medical specialty Date AVIEW CAC Coreline Soft Providing quantitative analysis of calcified plaques in the coronary arteries using non-contrast/ non-gated chest CT scans 510(k) predicate: AVIEW (K214036) Radiology March 2024 Irregular Rhythm Notification Feature (IRNF) Apple Analyzing pulse rate data to identify episodes of irregular heart rhythms 510(k) predicate: IRNF 2.0 (K212516) Cardiovascular July 2023 EyeArt v2.2.0 Eyenu Detecting more than mild diabetic retinopathy and vision-threatening diabetic retinopathy 510(k) predicate: EyeArt (K200667) Ophthalmology June 2023 Caption Interpretation Automated Fraction Software Caption Health Processing previously acquired transthoracic cardiac ultrasound images to provide automated estimation of left ventricular ejection fraction De novo (with PCCP) Radiology February 2023 Belun Sleep System BLS-10 Belun Technology Aiding in evaluating moderate-to-severe sleep-related breathing disorders of adult patients suspected of sleep apnea 510(k) predicate: NightOwl (K220028) Anesthesiology February 2023 VISIONAIR PacificMD Biotech Measuring the nasal respiratory airway 510(k) predicate: Eccovision System (K170071) Ear, nose and throat October 2022 SomnoMetry Neumetry Medical Assessing sleep quality and aiding diagnosis of sleep- and respiratory-related sleep disorders in adults 510(k) predicate: EnsoSleep (K162627) Neurology September 2022 Minuteful (kidney test) Healthy.io Measuring albumin and creatine in urine 510(k) predicate: URiSCAN Optima (K141874) Clinical chemistry July 2022 DreaMed DreaMed Diabetes Managing Type 1 diabetes De novo Endocrinology June 2018 Guardian Connect System Medtronic Predicting blood glucose changes PMA Endocrinology March 2018 In 2023, FDA compiled the following useful statistics concerning AI/ ML-enabled devices on the FDA-created list: * The year-over-year increase of AI/ML-enabled devices slowed in 2021 (15%) and 2022 (14%) after an increase of 39% in 2020 (compared to 2019). Based on projected volume in 2023, the increase of AI/ ML-enabled devices (compared to 2022) is expected to reach more than 30%. * 87% of devices on this list authorized in calendar year 2022 are in radiology (122), followed by 7% in cardiovascular (10), and 1% each Cooley AI/ML-Based Medical Device Regulatory Handbook 7 in neurology (2), hematology (1), gastroenterology/urology (1), ophthalmology (2), clinical chemistry (1), and ear, nose and throat (1). * Through the end of July 2023, 79% of devices authorized in 2023 are in radiology (85), 9% in cardiovascular (10), 5% in neurology (5), 4% in gastroenterology/urology (4), 2% in anesthesiology (2), and 1% each in ear, nose and throat (1) and ophthalmology (1). * In addition to having the largest number of submissions, radiology has experienced the steadiest increase of AI/ML-enabled device submissions of any specialty. * In general, ML models have ranged in complexity from shallow models (less than two hidden layers) to more complex models (deep learning models). * Models have generally trended toward more hybrid approaches, combining different algorithmic methods to achieve the result of a safe and effective device (for example, using one model to generate features and using another model for classification). III. How do I get my AI/ML device through FDA? A. Is the AI/ML product a device? The first step is to determine whether FDA has jurisdiction over the product - namely, is the product a device? A "device" is any product intended to diagnose, cure, mitigate, prevent or treat a disease or condition, or intended to affect the structure or any function of the body, and that does not achieve its primary intended purposes through chemical action within or on the body.19 Excluded from this definition are certain non-device software functions under section 520(o) of the FDCA20 (see chart below). 19 21 USC § 321(h). 20 21 USC § 360j(o). Cooley AI/ML-Based Medical Device Regulatory Handbook 8 Low-risk general wellness devices Admin support software Electronic health records Medical device data systems Clinical decision support software and mobile medical apps FDA-regulated devices Software/ Saas SaMD B. Use datasets appropriate for intended use that reduce or minimize bias After it has been determined that the product at issue is a "device" and subject to FDA's jurisdiction, FDA evaluates AI/ML-based software during premarket review for potential bias, as such bias may directly impact a device's safety and effectiveness. Although there may be different reasons for bias to exist in AI/ML-enabled SaMDs, one primary reason is the use of datasets that are incomplete or unrepresentative of the overall population to develop, train and/or validate the AI/ML-enabled SaMD, as such use could lead to the SaMD being ineffective in a more diverse, realworld context.21 For example, a research paper found that when men represented most (94%) of the patients in the datasets used in training an algorithm to predict which patients were most likely to experience decline in kidney 21 Pew, "FDA Review Can Limit Bias Risks in Medical Devices Using Artificial Intelligence," October 7, 2021. Pew explains: "Unlike many traditional medical devices, such as hip implants that are prescribed or used on a per-patient basis, software products tend to be embedded within an institution's broader information technology infrastructure and run automatically in the background for all patients -- beyond the control of individual providers. Depending on their purpose, such products may affect the care of any patient treated at a particular clinic or hospital, not just those with a particular condition. This increases the potential scope of impact if the product proves biased in some way." Cooley AI/ML-Based Medical Device Regulatory Handbook 9 function, the algorithm proved less effective when tested on women.22 As such, FDA subject matter experts assess the benefit-risk profile of a device for its proposed intended use and evaluate devices for potential bias during premarket review. When reviewing AI/ML-based SaMDs, FDA expects sponsors of AI/ ML-based SaMDs to present data showing that their algorithms are tested on a representative population - including data broken down by demographic groups, such as sex, age, race and ethnicity, as appropriate - to reduce, if not eliminate, potential bias. C. Develop 'transparent' labeling Transparency to users is a significant factor that FDA considers in its oversight. Because labeling is the primary way sponsors communicate their products' functionality to users, FDA expects such labeling "to clearly describe the data that were used to train the algorithm, the relevance of its inputs, the logic it employs (when possible), the role intended to be served by its output, and the evidence of the device's performance."23 This type of transparency also is important in FDA's evaluation of whether an AI/ ML-enabled product is a SaMD as opposed to a non-device clinical decision support (CDS) software.24 D. Use real-world data and evidence - if you have it Real-world data (RWD) is defined as "data relating to patient health status and/or the delivery of health care routinely collected from a variety of sources."25 Moreover, real-world evidence (RWE) is defined as "the clinical evidence regarding the usage, and potential benefits or risks, of a medical product derived from analysis of RWD."26 These could include, for example, "data derived from electronic health records (EHRs), medical claims data, data from product and disease registries, and data gathered 22 Id. 23 FDA 2021 AI/ML Action Plan. 24 FDA draft guidance, "Use of Real-World Evidence to Support Regulatory Decision-Making for Medical Devices," December 19, 2023, at 1. 25 Id. 26 Id at 2. Cooley AI/ML-Based Medical Device Regulatory Handbook 10 from other sources (such as digital health technologies) that can inform on health status."27 On December 19, 2023, FDA issued a draft guidance document to clarify how it evaluates RWD to determine the quality of such data for purposes of generating RWE for use in FDA regulatory decision-making for medical devices.28 Because sponsors are in the best position to understand how their products are used, identify opportunities for improvements and respond proactively to safety or usability concerns by gathering performance data on the real-world use of the SaMD,29 the agency allows sponsors to leverage RWD as part of a device's benefit-risk profile.30 For example:31 * RWE served as the primary source of clinical evidence in submissions for new devices and expanded indications for currently marketed devices. * Prospective, randomized trials were nested within RWD sources. * Control arms and objective performance goals were generated for evaluating performance of the next generation of devices. * Registry infrastructure addressed important premarket and postmarket needs. * Diverse RWD sources were, at times, combined to generate RWE. * RWD was used in submissions to fulfill study requirements under postmarket surveillance orders. * RWE was obtained from use of an emergency use authorization (EUA) device. In the SaMD context, certain RWD sources have been used to generate RWE to support FDA premarket review and marketing authorization, as shown in Table 2. 27 Id. 28 Id. 29 FDA 2021 AI/ML Action Plan. 30 Id. 31 Id at 33-35. For additional examples of RWE used in regulatory decision-making, see "Examples of Real-World Evidence (RWE) Used in Medical Device Regulatory Decisions." Cooley AI/ML-Based Medical Device Regulatory Handbook 11 Table 2 - Examples of RWE used in regulatory decision-making concerning SaMDs32 File Sponsor Device RWD source(s) RWE used in premarket review K172959 PeraHealth PeraServer and PeraTrend System Data from medical records (electronic health records, electronic medical records and/or chart reviews) were used for validation of SaMD product Three publications were submitted for this 510(k), in which the subject SaMD product was tested on data from retrospective medical records of adult and pediatric patients. DEN170073 Viz.AI ContaCT Radiological reports and real-world literature This is a radiological computer-aided triage and notification software. A secondary RWE analysis compared the standard-of-care notification time extracted from radiologist reports against a comparable metric from stand-alone testing. DEN170052 Natural Cycles Natural Cycles Outside-the-US data from a web- and mobile-based application for conception that contains patient-generated or patient-entered data This is a web- and mobile-based SaMD application for conception. For this submission, the sponsor performed a retrospective analysis of data from approximately 15,000 users of the mobile application. This was the primary source of clinical evidence supporting the de novo classification request. E. Don't forget about international policies and standards For clients looking to develop products to be marketed globally, and who have been stymied by the need to comply with different regulatory regimes, there is good news: FDA has made efforts to encourage harmonization of its regulatory approach with international policies and standards. Specifically, in 2021, FDA, Health Canada and the UK's Medicines and Healthcare Products Regulatory Agency (MHRA) jointly identified 10 guiding principles that can inform good machine learning practice (GMLP) development. In 2023, the same group jointly published the "Predetermined Change Control Plans for Machine Learning-Enabled Medical Devices: Guiding Principles." These PCCP guiding principles mirror those described in the FDA's PCCP guidance. 32 These examples were extracted from FDA's "Examples of Real-World Evidence (RWE) Used in Medical Device Regulatory Decisions." Cooley AI/ML-Based Medical Device Regulatory Handbook 12 IV. Looking around regulatory corners A. FDA is not the only US regulatory agency looking at AI/ML As AI technology evolves rapidly,33 FDA has been exploring how to regulate AI/ML-based medical devices and evaluating the role of AI/ML in medical product development.34 While FDA, as part of the Department of Health and Human Services (HHS), has been developing a regulatory framework for AI/ML-driven software modifications to provide appropriate safety and effectiveness guidelines, the National Institutes of Health (NIH) has collaborated and invested in AI-based projects to discover health solutions across research and medical settings, including analysis of biomedical imaging to diagnose diseases such as COVID-19. To effectively advance the AI ambition outlined in this strategy, HHS will establish an AI Council to support AI governance, strategy execution and development of strategic AI priorities. Transparency (truthfulness) about AI/ML-enabled products also is the focus of federal agencies other than FDA - including the Securities and Exchange Commission (SEC) and Federal Trade Commission (FTC). For example, on December 5, 2023, SEC Chair Gary Gensler cautioned companies against making false or exaggerated AI-related claims - or "AI washing" - advising that companies must make "full, fair and truthful" disclosures about AI products. The FTC also has warned companies to "[k]eep [their] AI claims in check," advising that such claims must be transparent and truthful.35 This is consistent with the FTC's mission to ensure truthful and nonmisleading advertisement. Health regulatory agencies have not yet made a concerted attempt to regulate the use of AI, though many are considering it. The Office of the National Coordinator for Health Information Technology (ONC) published a final rule implementing the provisions of the 21st Century Cures Act in February 2024. The final rule establishes transparency requirements for 33 The executive order "establishes new standards for AI safety and security, protects Americans' privacy, advances equity and civil rights, stands up for consumers and workers, promotes innovation and competition, advances American leadership around the world, and more." 34 See, e.g., "Proposed Regulatory Framework for Modifications to Artificial Intelligence/Machine Learning (AI/ML)-Based Software as a Medical Device (SaMD) - Discussion Paper and Request for Feedback." See also "Artificial Intelligence/Machine Learning (AI/ML)-Based Software as a Medical Device (SaMD) Action Plan" and "Using Artificial Intelligence & Machine Learning in the Development of Drug & Biological Products - Discussion Paper and Request for Feedback." 35 See also FTC, "Aiming for truth, fairness, and equity in your company's use of AI," April 19, 2021. Cooley AI/ML-Based Medical Device Regulatory Handbook 13 use of AI and other predictive algorithms that are part of certified health information technology. The rule also requires the implementation of risk management practices covering a broad array of topics - including validity, reliability, robustness, fairness, intelligibility, safety, security and privacy. CMS has addressed Medicare Advantage Plans' ability to use AI to make coverage decisions; however, CMS has not otherwise provided meaningful guidance in this space.36 B. Use of AI in clinical research AI is being used more often in the drug development space, with FDA reporting more than 100 submissions utilizing AI/ML components in drug and biologic applications in 2021.37 In 2023, FDA published a discussion paper on the use of AI/ML in the drug development process. While not endorsing or providing guidance, the paper was meant to promote learning and discussion in this area. The paper highlighted several areas where AI/ML is being used, such as in the phase of drug discovery, nonclinical research and clinical research. Particularly in clinical research, where medical device manufacturers also may be conducting clinical trials, it is critical to assess the risks of using AI/ML. Areas where we are now seeing sponsors utilize AI/ML for clinical trials include: * Recruitment * Selection and stratification of trial participants * Using AI applications to match patients to appropriate clinical trials * Tools to help improve participants' adherence to the trial and retention * Site selection * Clinical trial data collection, management and analysis * Particularly with digital health technologies (DHT), AI/ML can be utilized, either as embedded in algorithms within the DHT or utilized after the data has been collected from the DHT, and can be used to predict the status of a chronic disease and its response to treatment or to identify novel characteristics of an underlying condition.38 Because these DHTs may continuously monitor the 36 CMS, "Frequently Asked Questions Related to Coverage Criteria and Utilization Management Requirements in CMS Final Rule (CMS-4201-F)," February 6, 2024. 37 FDA, "Artificial Intelligence and Machine Learning (AI/ML) for Drug Development." 38 Id. Cooley AI/ML-Based Medical Device Regulatory Handbook 14 individual using the technology, the AI/ML can be utilized to analyze these large sets of data. * Clinical endpoint assessment * Postmarket safety surveillance Large amounts of clinical data, potentially including sensitive identifiable protected health information (PHI),39 are generated during a clinical trial. Researchers must seek and obtain regulatory approval to collect and utilize such data. When conducting clinical trials, institutional review boards (IRBs) may prohibit the use of AI in the clinical trial, particularly when accessing PHI that may be embedded in the AI tool.40 An IRB may request that a sponsor provide details in the protocol and informed consent about how AI is being utilized in the clinical trial. In addition to protecting patient health information, sponsors should consider how clinical trial data integrity may be compromised if a breach of the AI tool occurs. C. Don't forget about HIPAA If a sponsor's AI is accessing patient health records, an IRB may require that the Health Insurance Portability and Accountability Act of 1996 (HIPAA) consent form also include information about how the sponsor is utilizing AI, how patient confidentiality will be protected and what rights participants have to remove their information from the AI tool.41 Some IRBs may permit the use of AI where HIPAA identifiers have been removed or "de-identified"42 from the data prior to feeding the information into the AI model. In addition to HIPAA's application to clinical trials, a major consideration for healthcare entities relates to data sharing. HHS co-hosted a series of roundtable discussions, including a "Roundtable on Sharing and 39 45 CFR § 160.103 (defining protected health information as "individually identifiable health information" and exempting a small number of categories of individually identifiable health information, such as individually identifiable health information found in employment records held by a covered entity in its role as an employer). 40 See, e.g., University of Tennessee Knoxville, Guidance on the Use of AI Tools in Human Subject Research. 41 45 CFR § 46.117(a). 42 45 CFR § 164.514(b)(2) (de-identification standard under HIPAA). Note that under the HHS regulations at 45 CFR § 46.117(a), IRB review and approval of HIPAA authorizations is only required if the authorization language is integrated in the informed consent document for human subjects research. Cooley AI/ML-Based Medical Device Regulatory Handbook 15 Utilizing Health Data for AI Applications" in 2019.43 The roundtable report noted that AI requires high-quality, accurate data from large, multifaceted datasets to develop algorithms - the more data inputs, the better the predictive and diagnostic results.44 Researchers are obtaining this data from various sources - including EHRs and data collected from wearable devices and sensors.45 This data may be considered PHI, and thus, the exchange of such sensitive data would be subject to laws and regulations on the protection of individual protected health information, including HIPAA.46 A few key compliance issues that HIPAA-regulated entities should consider before incorporating AI into their operations include the type of AI model utilized (e.g., supervised versus unsupervised algorithms47) and how the AI application collects and shares patient data. Entities should seek to limit who has access to the AI model to prevent or mitigate the potential for an accidental breach or unauthorized disclosure. HIPAAcovered entities should, at a minimum, provide training on risks specifically associated with the use of AI and implement the training into the organization's annual HIPAA training. Further, HIPAA-regulated entities should ensure that all patient health information used to train the AI model is de-identified to fit into the HIPAA safe harbor.48 HIPAA does not restrict the use or disclosure of de-identified health information, as it is no longer considered PHI.49 Entities subject to HIPAA will want to ensure business associate agreements (BAAs) are in place with any AI vendors they utilize. The Cooley life sciences and healthcare regulatory team continues to track global developments and is available to counsel on any of these complex regulatory questions. 43 HHS, "Sharing and Utilizing Health Data for AI Applications - Roundtable Report." 44 Id. 45 Id. 46 HHS, "Covered Entities and Business Associates." HIPAA requires "covered entities" (which include healthcare providers that conduct certain standard administrative and financial transactions in electronic form, healthcare clearinghouses or health plans) and their "business associates" (thirdparty organizations that provide certain services for or on behalf of a covered entity and receive PHI from a covered entity in connection with those services) to comply with standards relating to the privacy and security of patient health information. 47 HHS, "Sharing and Utilizing Health Data for AI Applications - Roundtable Report." 48 HHS, "Guidance Regarding Methods for De-identification of Protected Health Information in Accordance with the Health Insurance Portability and Accountability Act (HIPAA) Privacy Rule," October 25, 2022. 49 Id. Cooley AI/ML-Based Medical Device Regulatory Handbook 16 Sonia Nath Partner [email protected] Sonia is chair of Cooley's global life sciences and healthcare regulatory practice group. She has deep experience in matters involving the Food and Drug Administration (FDA). Her practice encompasses investigations, litigation and regulatory counseling, as well as advising on transactional matters. Having served for almost 12 years at the FDA's Office of the Chief Counsel as a litigation and enforcement attorney, Sonia represents clients before the FDA, including in FDA enforcement actions and regulatory meetings. Son Nguyen Special Counsel [email protected] Son is an experienced US Food and Drug Administration (FDA) lawyer, having spent almost 14 years at the agency as associate chief counsel for enforcement, associate chief counsel for devices and combination products, and senior counsel. During his tenure at the FDA, Son advised the Office of the Commissioner, the Office of Combination Products (OCP), the medical-product centers (the Center for Drug Evaluation and Research, Center for Devices and Radiological Health, and Center for Biologics Evaluation and Research), and agency leaders on statutory and regulatory issues concerning medical devices and combination products. Alexis Finkelberg Bortniker Partner [email protected] Alexis has extensive experience in digital health, behavioral health, population health management, risk-based reimbursement systems and payer/provider alignment. She regularly counsels investors, entrepreneurs and startup businesses in the healthcare arena, and she advises on a variety of developing legal areas - including structural considerations, fraud and abuse, antitrust risk and payment models. About the authors