IBM Seeks Niche for Watson for Oncology

Nathan Levitan, MD, MBA

After getting off to a rocky start, the artificial intelligence (AI) system that IBM designed for oncology has demonstrated clinical utility in recent studies, generating excitement among AI advocates that the technology may have a productive role as a decision support tool.

Twenty-two studies about Watson for Oncology (WFO), as the system is known, were presented at the 2019 American Society of Clinical Oncology (ASCO 2019) Annual Meeting by IBM Watson Health and its partners. IBM hopes that the new information, coupled with an effort to improve WFO’s integration with electronic health records (EHRs), will encourage broader adoption of the system among US oncologists.

Although WFO was once hailed as having the potential to identify “even new approaches” to cancer treatment, critics have said the technology has yet to demonstrate that it can revolutionize the practice of oncology.¹

Proponents of the technology, however, describe it as an AI tool that can do far more than function as a refined search engine for medical information. To them, it is a “cognitive” computing tool that can “learn,” “reason with purpose,” and “interact with humans naturally.”²

WFO offers a tablet-sized library of clinical data on 13 cancer types, including information on third-line or beyond treatment for 10 metastatic cancers, which is developed in collaboration with experts from Memorial Sloan Kettering Cancer Center (MSK) in New York, New York. By inputting patient information, physicians can obtain suggestions for treatment.³

“WFO does not diagnose. It does not replace the physician—it is information at the point of care to help the physician make the best decision,” Nathan Levitan, MD, MBA, chief medical officer at IBM Watson Health oncology and genomics, said in an interview with OncologyLive^®.

AI, Meet Healthcare

The technology has met with varying degrees of success. In 2013, IBM partnered with physicians from The University of Texas MD Anderson Cancer Center in Houston to develop Oncology Expert Advisor (OEA), a web-based clinical decision support tool that used Watson technology to issue care advice, match patients with clinical trials, and facilitate patient evaluation.^4,5 IBM would power OEA and an MD Anderson oncologist would operate the tool; the plan appeared workable until the cancer center abruptly canceled the collaboration in 2016 after pouring $62 million into OEA.

Although MD Anderson gave OEA a try in leukemia and later lung cancer,⁶ the institution halted the pilot program after a University of Texas audit found that the Watson tool could not sync with MD Anderson’s Epic electronic health record (EHR) system, thereby requiring oncologists to reference outdated drug protocol and clinical trial data from the EHR system that MD Anderson used prior to Epic, ClinicStation.⁴ OEA’s use of outdated information was particularly problematic because the “evidence-based” recommendations generated may not have been in accord with the latest clinical evidence.⁷

The audit report included details of the closure agreement with IBM, which supported MD Anderson’s decision to suspend development of OEA. IBM acknowledged in the report that OEA “was not ready for human investigational or clinical use” and instituted a moratorium on its use for the treatment of patients, except in further tests of OEA.⁴

As MD Anderson physicians struggled with OEA, MSK oncologists programmed the nascent system that would go on to become WFO. MSK fed WFO a variety of medical data, including clinical studies, guidelines, patient records, and doctors’ notations.⁷

The library of information was intended to inform WFO’s treatment recommendations and enable WFO to interpret the inputted data and uncover patterns among patients with recent cancer diagnoses whose records were stored in WFO. The supercomputer would also keep pace with newly published medical literature.

Critics of the technology contended that WFO was limited in its ability to provide physicians with information that was not already at their disposal. Furthermore, it was hampered by difficulties in interfacing with EHR systems.^1,6,7

In contrast to the broad applications of AI in other fields, the AI that is directly applied to clinical care is still frontier science, according to John Frownfelter, MD, FACP, chief medical information officer for Jvion, which specializes in patient harm prevention and clinical cost savings. Frownfelter is optimistic that AI will one day play a crucial role in oncology despite its current fledgling status. “AI will be needed to deliver insights that we won’t otherwise have,” Frownfelter said in an interview with OncologyLive^®.

The degree of AI’s usefulness in the health setting depends largely on whether the data can improve oncologists’ treatment decisions. “It’s [easy] to underestimate the complexity of developing AI that’s clinically relevant,” Frownfelter said. “Making it clinically relevant means that it also needs to be actionable. So if I get insights that aren’t really news to me—that I’m not learning anything from— then [the information is] irrelevant.”

Proving Watson's Worth in Oncology

Watson the supercomputer initially made a splash in 2011 when it was introduced as a contestant on Jeopardy! and triumphed over 2 game champions. To carve out a niche for the technology in oncology, IBM has turned to clinical trials for proof of relevance.

Levitan cited results of a concordance study conducted at Manipal Hospitals in India as evidence of WFO’s potential to generate clinically meaningful information. Presented at ASCO 2019, the study measured changes in treatment decisions by a multidisciplinary tumor board that evaluated 1000 patients with breast, lung, and colorectal cancers between 2016 and 2018.

Findings showed that board members changed their treatment decisions in 13.6%⁸ of cases based on information supplied by WFO (Table^2,8,9,10). Investigators said that 55% of the changed decisions occurred after WFO offered recent evidence for newer treatment and 30% followed WFO’s identification of more personalized alternatives. Information from genotypic and phenotypic data and evolving clinical experiences prompted another 15% of the changes.

Table. Findings From Recent Studies Involving Watson for Oncology^2,8,9,10 (Click to Enlarge)

For Shridar Ganesan, MD, PhD, chief of molecular oncology and associate director of translational research at Rutgers Cancer Institute of New Jersey in New Brunswick, the actionability of WFO’s treatment recommendations in the study and the caliber of its “decision impact” require clarification. “It’s not clear those changes were actually for the better,” Ganesan told OncologyLive^®, calling 13.6% a relatively small amount. “[The study results] show that you can integrate WFO into workflow, but whether it has a positive impact on outcome or not has yet to be seen,” he said.

At ASCO 2019, IBM Watson Health and its partners presented another study whose results showed improvement in patient care confidence following WFO’s inclusion in a 7-step patient engagement and consultation process. Although patient perception of WFO was positive, the clinical benefit of WFO was not measured. The study was conducted in the oncology department of the Beijing Chaoyang Integrative Medicine Emergency Medical Center in China.

Investigators said WFO helped patients better comprehend their disease and treatment options.⁹ Patients reported higher satisfaction with treatment options, greater confidence in their physicians, and increased willingness to follow the suggested regimen when WFO was implemented as a supplemental aid versus when it was not used at all.

Ganesan noted that these favorable findings might not be wholly attributable to WFO. “You imagine that the 7-step process alone may have an impact on the patients’ confidence, not the Watson part of it,” Ganesan said. “That [process] itself may be very different from what the standard is.” If an intensive patient engagement and consultation plan is not a part of the oncology department’s patient engagement procedure, then the increase in patient comprehension and overall confidence that investigators witnessed during the study could simply be a byproduct of its more involved method of patient interaction, according to Ganesan.

Assessing Acceptance of WFO

Some US patients appear to be open to the use of WFO as a clinical decision support tool. In a study reported in 2019 by MSK, investigators divided 46 patients with breast, lung, or colorectal cancer into 9 focus groups to analyze their attitudes and perspectives on WFO. The majority (63%) of participants agreed that WFO would help their physician identify the best chemotherapy, and 35% were unsure.¹⁰ Similarly, 63% of patients said they would be glad that their physician had consulted WFO and 35% remained uncertain. Investigators said that a prominent subtheme expressed in the focus groups was the sentiment that WFO would be valuable if its use were appropriate, which patients defined as being “a supplement to physician decision making.”

“[Patients] really felt that providing this additional information to their physician didn’t diminish the autonomy or intelligence of the physician but simply provided a tool to optimize the recommendation,” Levitan said.

Patients still had their hesitations: Nearly every focus group included in the 2019 report expressed concerns about the quality of the curated and manually fed data. Patients questioned the origins of the data and the potential for human error at the point of input.

Finding a US Following

The unsteady start of IBM’s AI undertaking has not discouraged the developer from embarking on a fresh path to commercial and clinical success. IBM is working to address the deficiencies of WFO’s EHR compatibility and hopes that enhanced EHR functionality will make WFO a more attractive asset to US hospitals and oncologists. “This year, we are in the process of really optimizing our capabilities for integration with multiple EHRs,” Levitan said. “We think [EHR] integration is very important to making the workflow most efficient for US physicians.” Notably, the system is already able to sync with EHRs, but many oncologists instead opt to manually input patients’ medical records into WFO because of the expense of EHR connection. Physicians at hospitals that use WFO must consequently manually register patient information, which can be time-consuming and generally inconvenient. Only when the data are entered can WFO formulate care recommendations. Improved EHR integration may ease patient concerns regarding data entry errors, as well as streamline and simplify how oncologists interact with WFO. Still, the specifics of IBM’s plans to enhance WFO’s EHR integration are unclear.

Support in the United States for WFO has been weak compared with support in other countries, where top-level oncology resources and expertise may be harder to come by, said Andre Goy, MD, chairman and director of the John Theurer Cancer Center (JTCC) at Hackensack University Medical Center in New Jersey.

Findings from a recent study involving investigators from JTCC’s Division of Breast Oncology; Cota Inc in New York, New York; and IBM Watson Health in Cambridge, Massachusetts, illustrate some of the challenges.

The study results showed that 3 blinded oncologists specializing in breast cancer chose treatment strategies that were in accord with WFO “acceptable” strategies 87.9% of the time and chose a preferred option in 78.5% of cases.²

Conversely, oncologists who did not regularly work with patients with breast cancer chose WFO “recommended” options 75.5% of the time. When using WFO, these oncologists chose WFO “recommended” or “for consideration” options 95.3% of the time.

WFO in this study was supported by the Cota real-world evidence (RWE) platform. Authors of the study concluded that based on these results, WFO/Cota RWE can help novice oncologists improve their clinical decisions.

“If you use WFO with our oncologists who are really experts in the field [and who] really know the disease because they are highly subspecialized, WFO doesn’t really bring much,” Goy said in an interview. “If you use this with a physician who is doing general oncology, where the landscape is getting so complex and is changing so rapidly, WFO could improve the decision by 25% and improve the outcome of the patient.”

The study findings, Goy added, have helped convince him that WFO can be of significant value for novice or generalist oncologists. “There’s no question that outside a very subspecialized area, having an artificial intelligence resource will help at the point of care.”

References

1. Ross C, Swetlitz I. IBM pitched its Watson supercomputer as a revolution in cancer care. it’s nowhere close. STAT website. statnews.com/2017/09/05/watson-ibm-cancer/. Published September 5, 2017. Accessed June 14, 2019.
2. McNamara DM, Goldberg SL, Latts L, et al. Differential impact of cognitive computing augmented by real world evidence on novice and expert oncologists [published online September 11, 2019]. Cancer Med. doi: 10.1002/cam4.2548.
3. IBM Watson Health. Solution brief: Elevate cancer care with the Watson oncology suite. IBM website. ibm.com/downloads/ cas/AONROW12.
4. The University of Texas System Audit Office. Special Review of Procurement Procedures Related to the M.D. Anderson Cancer Center Oncology Expert Advisor Project. The University of Texas System website. utsystem.edu/sites/default/files/ documents/UT%20System%20Administration%20Special%20 Review%20of%20Procurement%20Procedures%20Related%20 to%20UTMDACC%20Oncology%20Expert%20Advisor%20Project/ ut-system-administration-special-review-procurement-procedures- related-utmdacc-oncology-expert-advis.pdf. Published November 2016. Accessed July 15, 2019.
5. Takahashi K, Kantarjian HM, Garcia-Manero G, et al. MD Anderson’s Oncology Expert Advisor powered by IBM Watson: a web-based cognitive clinical decision support tool. J Clin Oncol. 2014;32(suppl 15; abstr 6506). ascopubs.org/doi/ abs/10.1200/jco.2014.32.15_suppl.6506.
6. Herper M. MD Anderson benches IBM Watson in setback for artificial intelligence in medicine. Forbes website. forbes.com/ sites/matthewherper/2017/02/19/md-anderson-benches- ibm-watson-in-setback-for-artificial-intelligence-in-medicine/# 6e3547163774. Published February 19, 2017. Accessed July 10, 2019.
7. Strickland E. How IBM Watson overpromised and underdelivered on AI health care. IEE website. spectrum.ieee.org/ biomedical/diagnostics/how-ibm-watson-overpromised-andunderdelivered- on-ai-health-care. Published April 2, 2019. Accessed June 17, 2019.
8. Somashekhar SP, Sepúlveda MJ, Puglielli S, et al. Watson for Oncology and breast cancer treatment recommendations: agreement with an expert multidisciplinary tumor board. Ann Oncol. 2018;29(2):418-423. doi: 10.1093/annonc/ mdx781.
9. Wang Z, Yu Z, Zhang Z, et al. Artificial intelligence-based clinical decision-support system improves cancer treatment and patient satisfaction. Abstract presented at: 2019 ASCO Annual Meeting; May 31, 2019-June 4, 2019; Chicago, Illinois. e18303. http://abstracts.asco.org/239/Abst- View_239_266219.html.
10. Hamilton JG, Garzon MG, Westerman JS, et al. “A Tool, Not a Crutch”: Patient Perspectives About IBM Watson for Oncology Trained by Memorial Sloan Kettering. Journal of Oncology Practice. 2019; 15(4): e277-e288. Jan 28, 2019. June 14, 2019.

“[The change rate] is among the most important [findings presented] because it really is about not only educating the physician but also about the actual decision impact,” Levitan said.