eContour: Evidence-Based Interactive Contouring Resources at the Point of Care

About the lead author:

Erin F. Gillespie, MD

University of California San Diego

Department of Radiation Medicine and Applied Sciences

La Jolla, CA 92093

Expert’s Perspective

Brandi R. Page, MD

Assistant Professor

Radiation Oncology and Molecular Sciences

Johns Hopkins University School of Medicine

Why is this article contemporary?

The utilization of internet-based educational forums has been developing over the last few years for the sharing of information and standardization of practice patterns.

As technical details of treatment planning become more involved and the complexity of treatment planning increases with our ability to deliver higher doses in fewer treatment fractions—and as newer treatment modalities are being developed—accurate target and organ delineation, and sometimes even the ability to obtain expert opinions, have become even more critical. Numerous reports state inter-observer variability in practice exists not only between experts in our field in each disease site, but especially among trainees and in centers that have lower volumes of certain types of cases. Currently there are limited resources for residents and practitioners to refer to for specific help on contouring both target volumes and organs at risk in the modern era. The utilization of online contouring atlases has started to address this need, but more comprehensive educational materials are an unmet need for this very critical problem.

The e-contouring program documented in this article has a unique niche in this regard, allowing for a novel, web-based, user-friendly, accessible, interactive approach to facilitate interaction between colleagues for decision support. As our field is a dynamically and rapidly changing its practices, we are in great need of these types of innovative educational interventions. Here, the authors present a novel approach to ongoing education and quality improvement. While it is clear that the eContour website is not meant to replace experienced clinician judgment in any case, and that more cases are needed, it represents an excellent innovation that hopefully just represents the beginning of the development of more comprehensive online educational resources for learning residents and continuing-to-learn practitioners.

Abstract

Background and Purpose

Target delineation in radiation oncology is critical to delivering safe and effective radiation therapy. Evidence from clinical trials suggests that poor radiation treatment plan quality decreases the rates of overall survival and increases rates of toxicity. The introduction of new technologies in radiation oncology, including intensity-modulated radiotherapy (IMRT), image-guided stereotactic radiosurgery (SRS), and stereotactic body radiation therapy (SBRT) have improved our ability to deliver higher doses of radiation to the tumor and areas at risk while avoiding normal tissues. These highly conformal radiation fields leave little margin for error, highlighting the critical importance of accurate contouring.

Results and Conclusions

There is substantial inter-observer variability among radiation oncologists in target delineation, which is most notable in head-and-neck and lung cancers, although it has also been shown in prostate cancer due to variable definitions of the prostate apex. This variation can be reduced when contouring atlases and consensus guidelines are referenced. Unfortunately, barriers exist that limit clinicians from seeking answers to their clinical questions (such as details of high-risk clinical target volume coverage), which results in decreased adherence to the standard of care. Effective strategies to implement clinical guidelines emphasize intervention at the time of the patient encounter, which is also a recommended approach to motivating adult learning. eContour (eContour.org) is a novel, web-based, interactive contouring resource that aims to facilitate access to decision support through high-yield radiation treatment recommendations that are free and available at the point of care.

Introduction

Target delineation in radiation oncology—commonly referred to as contouring—represents the crux of delivering safe and effective radiation therapy. Contours not completely encompassing the tumor or regions at risk of microscopic disease lead to increased rates of disease recurrence and decreased survival.^1-3 On the other hand, contours that are too large have the potential to lead to increased rates of toxicity.⁴ The introduction of new technologies in radiation oncology, including intensity-modulated radiotherapy (IMRT), image-guided stereotactic radiosurgery (SRS), and stereotactic body radiation therapy (SBRT), has improved our ability to deliver higher doses of radiation to the tumor and areas at risk while avoiding normal tissues.

Outcomes in Radiation Oncology

Poor Contouring Quality Affects Patient Outcomes

Impact of Low-Volume Centers and Geography on Patient Outcomes

These highly conformal radiation fields leave little margin for error, which highlights the critical importance of accurate contouring. Here, we review the potential impact of suboptimal contouring on patient outcomes, currently available educational resources to aid radiation oncology providers in contour delineation, effective strategies to improve access to high-yield information for clinical management, and finally propose a recently released novel, open-access contouring resource called eContour (eContour.org).There is increasing evidence that a radiation plan of poor quality leads to inferior patient outcomes. A re-analysis of the head-and-neck trial TROG 02.02 found a 20% decrease in overall survival at 2 years (70% vs 50%) for patients with radiation protocol violations compared to compliant treatment plans.¹ Likewise, the head-and-neck trial RTOG 0129 found that patients with radiation plan protocol violations had a 22% decrease in survival at 5 years (63% vs 41%) compared to patients receiving protocol compliant treatment plans.² A third head and neck trial, RTOG 0022, reported a 46% increase in locoregional failure at 2 years (6% vs 50%) in patients with major protocol deviations.⁵ In pancreatic cancer, RTOG 9704 found a decrease in overall survival (1.74 years vs 1.46 years) as well as a trend toward increasing grade 4/5 non-hematologic toxicity in patients with protocol violations.³ Finally, the HD4 trial of patients with early-stage Hodgkins lymphoma found recurrence- free survival at 7 years was decreased by 12% (84% vs. 72%) in patients with non-compliant treatment plans.⁶ Multiple aspects of the radiation planning process can lead to protocol violations4, although significant evidence suggests an integral role for contouring. In the TROG 02.02 head-and-neck trial, 25% of deviations were due to “incorrect target definition.”¹ In the RTOG 00-22 head and neck trial, 83% of major deviations related to “PTV66 coverage.”⁴ In the RTOG 9704 pancreas trial, 70% of deviations were due to “field placement or size.”³ This trial also found that 13% of plans identified as “unacceptable” were due to an over-contouring bowel. In the HD4 Hodgkins trial, 29% of protocol deviations were due to “volume too small.”⁶ Taken together, these findings suggest that radiation plan quality—of which contouring represents a central component—substantially impacts patient survival.Multiple studies demonstrate that rates of protocol deviations correlate inversely with the number of patients treated per center.⁸ For example, TROG 02.02 found that treatment centers enrolling <5 patients had a 30% protocol deviation rate in comparison to 5% in centers enrolling 20 or more patients. RTOG 0129 found a similar result with a cutoff of 13 or more patients, reducing the rate of protocol deviations from 18% to 6%. RTOG 0022 found a cutoff of 20 patients to be significant. Similar patterns have been observed in early-stage Hodgkins (HD4) and breast cancer (NSABP B04).⁷ A review of clinical trial Q&A by Weber et al raised the provocative and controversial question of whether low-volume treatment centers should be authorized to enroll patients into clinical trials due to the cost for institutional activation and apparent detrimental effect on radiation quality.⁸ Population-based studies further support the link between center experience and patient health outcomes.

Multiple studies using the National Cancer Data Base (NCDB) have shown decreased survival in patients treated at low-volume centers with chemoradiotherapy for larynx cancer⁹, definitive chemoradiotherapy for stage III non— small cell lung cancer¹⁰, and radiation for high-risk prostate cancer.¹¹ Recently, a Surveillance, Epidemiology, and End Results (SEER)-Medicare study looked specifically at individual provider experience in head-and-neck cancer and found a correlation between provider volume and outcomes.

This study found that the risk of death decreased by 21% for every additional 5 patients treated, with a similar decrease in the risk of aspiration pneumonia.¹² Of note, this survival difference was seen in patients treated with IMRT but not 3D-conformal radiotherapy—which emphasizes the need for high-quality radiation delivery, particularly in the current era of advanced radiation technology.

Figure. Screenshot of a representative case in eContour.org

Improving Quality in the Digital Age The Basis of Evidence-Based Medicine

Decision Support through Peer Review

The findings from the clinical trials and population-based studies reviewed above all point to a link between radiation quality and patient health outcomes. While one solution might to be to centralize radiation delivery in high-volume specialty centers, the reality and complexity of cancer care delivery makes this solution difficult. The average course of non-palliative radiation takes weeks or months to deliver, and relocating patients to a high-volume center presents substantial logistical challenges. Personal and socioeconomic strains on patients and caregivers represent clear barriers to either patient relocation or daily travel to tertiary care facilities. Additionally, within the US, distance to radiation presents a clear impediment to delivery of effective radiation. In early-stage breast cancer, rural areas have been found to have the lowest rates of breast-conserving surgery, which is hypothesized to be due to geographic distance from radiation oncology centers, and therefore, access to therapy.¹³ In stage II/III rectal cancer, a recent population-based study using NCDB data found that patients living a significant distance from a radiation oncology clinic were less likely to receive radiation (OR 0.75 for >50 miles and OR 0.46 for >250 miles).¹⁴ The preferred approach in radiation oncology therefore has been to improve quality of care in the community.Radiation therapy in particular among medical specialties should be conducive to delivering high-quality care remotely in the digital age. Radiation treatment planning is rarely performed emergently, allowing time for physicians to review the literature and ask clinical questions of colleagues. It is unclear, though, how often radiation oncologists ask and seek answers to their clinical questions. Data from general medicine characterizes a phenomenon whereby clinical questions arise frequently at the point of care, but often go unanswered. In a systematic review of 11 studies including direct clinical observation and clinician interviews, Del Fiol et al found that 0.57 clinical questions were asked per patient encounter, of which clinicians pursued answers to questions 50% of the time.¹⁵ This percentage has been remarkably stable over time, although the source of where physicians look for answers appears to have shifted. In a seminal study by Covell et al in 1985, 29% of questions were answered by discussing with a colleague¹⁶, compared to only 14% in a study from 2007.¹⁷ Meanwhile, the use of internet searches such as Google and UpToDate for research have increased to up to 30% of clinical inquiries.In radiation oncology, peer review commonly takes the form of routine prospective “chart rounds” to review radiation treatment plans. While data linking peer reviews to improved outcomes are lacking, several studies catalog significant rates of plan changes with chart rounds, with one study from British Columbia Cancer Agency reporting that 7% of radiation plans were changed.¹⁸ The American Society for Radiation Oncology (ASTRO)’s Target Safely Campaign provides recommendations regarding procedures for routine peer review.¹⁹ Peer review, however, is not without limitations. One recent study out of MD Anderson found that 17% of peer-reviewed patient cases still did not meet the standard of care.²⁰ Outside of radiation oncology, the general medical literature suggests that peer review results fall short of literature searches in terms of finding answers to clinical questions¹⁷, possibly because colleagues have an equally difficult time keeping up-to-date with the medical literature. A systematic review assessing the relationship between clinician experience and quality of care delivered reported the sobering statistic that over half (52%) of 62 published studies on the subject report that physician knowledge, adherence to standards of practice, and quality of care decline with clinician age and experience.²¹

In radiation oncology, two mechanisms have recently been implemented to facilitate access to expert review. In 2010, Chartrounds.com formed out of a need for decision support among physicians without easy access to peer review. Chartrounds.com is a conferencing website that allows community radiation oncologists to present patient cases (including diagnostic imaging and radiation treatment plans) to disease site specialists on a scheduled basis. Chartrounds.com has reported, in abstract, form outcomes that include broad usership, and user feedback showing the significant impact of the sessions on treatment decisions. ²² In 2012, theMedNet.org was developed to facilitate knowledge sharing through social media in which radiation oncologists ask clinical questions that initiate discussion among academic radiation oncologists. Conference abstracts report responses frequently incorporating new studies into recommendations to guide clinical practice.²³

The Right Information at the Right Time

When physicians do decide to seek answers to their clinical questions, they are successful an average of 80% of the time.15 With only 50% of questions being pursued, however, the barriers are significant, and most commonly reported to be lack of time. On average, clinicians spend a mean of 2-3 minutes seeking an answer to a specific question. Unanswered questions are viewed as an important opportunity to improve patient outcomes by filling gaps in medical knowledge in the context of clinical decisions.To help clinicians digest the large quantities of information, which may sometimes even be conflicting, practice guidelines have become increasingly available. However, implementation research demonstrates that practice guidelines rarely translate into real-world clinical practice, primarily due to suboptimal methods of development, delivery, and access.^24-26 In fact, journal publication represents one the least effective mechanisms to change practice, while patient- specific interventions at the time of the encounter are one of the most effective mechanisms.²⁶

Contouring Resources

Limitations with Current Resources

Enthusiasm in the digital age has built around the potential for electronic clinical knowledge support systems such as Up- ToDate to provide evidence-based decision support at the point of care. Recent evidence from two studies, though retrospective, suggest that use of UpToDate resulted in improved patient outcomes using quality metrics including hospital length of stay and 30-day risk-adjusted mortality in patients presenting with acute myocardial infarction and pneumonia.^27,28 This effect appears to be dose-dependent and most significant in small- to medium-sized non-teaching facilities. In addition to just-in-time answers providing opportunities for improved clinical outcomes for the current patient, educational research supports this as the most effective means of adult learning, and therefore, the ideal focus of continuing medical education.Contouring provides an ideal point in the radiation oncology care continuum to intervene with high-quality information. Given the impact of contouring on patient outcomes in radiation oncology, many resources have been developed, including consensus guidelines, contouring atlases, and books.^29-31 Numerous studies have demonstrated substantial interobserver variability among radiation providers^32-36, which can be reduced if providers are given access to contouring reference aids.³⁷ However, radiation oncologists routinely report barriers to accessing currently available resources.First, current contouring atlases are thorough, although they can be cumbersome to use (they are often PDF documents, with some over 100 pages long), requiring considerable study to apply to clinical practice. While the RTOG website includes 10-15 disease-specific visual atlases, the majority of consensus guidelines are available only in the form of journal publication. Another limitation of current contouring resources is that they often focus on normal anatomy, while patient clinical presentations include significant heterogeneity. For example, the consensus guideline for head and neck lymph nodes by multiple cooperative groups (Danish Head and Neck Cancer Group [DAHANCA], European Organisation for Research and Treatment of Cancer [EORTC], Hong Kong Nasopharyngeal Carcinoma Study Group [HKNPCSG], National Cancer Institute of Canada Clinical Trials Group [NCIC CTG], Nakayama Cancer Research Institute [NCRI], Radiation Therapy Oncology Group [RTOG], and Tasman Radiation Oncology Group [TROG])29 includes only to the node-negative neck, which does not translate well to the node-positive patient, highlighting the lack of generalizability of current resources.

Innovation with eContour

Other resources aimed at education and peer review in contouring include live contouring webinars^38,39, which offer a useful educational opportunity, though can be costly and do not serve as an on-demand reference for providers. One interactive website called EduCase (www.educase.com) allows the submission of contours for offline review by a paid expert, but has not attracted high user volume, potentially due to the complexity of the platform, delays in obtaining feedback, or the subscription fee. Lastly, a product called Anatom-e (anatom-e.com) aims to serve as an on-demand resource for providers, although it requires hardware installation, and has a membership fee.eContour (eContour.org) stands to challenge the existing paradigm in image-based contouring resources. We sought to design a tool that could overcome the limitations of current contouring resources and build on the success of other interactive online medical resources. With eContour, we have constructed a user-friendly, web-based tool that is free and facilitates point-of-care reference using a DICOM- based platform that integrates MRI and PET image overlays and enables active interaction with 3D images, mirroring current radiation-planning software, but in a quickly accessible website (see Figure). This web-based imaging platform style parallels eAnatomy (www.imaios. com), a website that has been shown to be the radiographic anatomy resource most frequently used by radiology residents.⁴⁰ eContour collects and synthesizes consensus guidelines, recently published protocols, and key publications guiding management decisions around target delineation into an easily digestible resource that physicians can review in a couple minutes. By providing a busy provider with valuable case-specific information critical to safe and effective treatment planning at the point of care, eContour strives to serve as an imaging-based companion to UpTo- Date for radiation oncology providers.

The public release of eContour took place on March 8, 2016. Within one month, the website has gained over 1000 registered users from 50 states and 65 countries, primarily through social media and professional organization newsletters within the United States. In addition to practicing radiation oncologists and resident physicians, there has been significant interest among medical dosimetrists, particularly due to the website’s incorporation of organs at risk and normal anatomy. To address deficiencies in radiographic anatomy identified in radiation oncology training⁴¹, each case includes critical normal anatomy to facilitate understanding of the rationale behind treating various at-risk areas. The current library includes a basic set of head-and-neck and lymphoma cases, although we are progressively expanding to multiple disease sites through the solicitation of case submissions from experts that will be peer-reviewed and published electronically to create a comprehensive searchable library of individual patient cases. User feedback will be critical to shaping future directions of eContour to ensure that it develops into a robust resource that meets the needs of radiation oncologists in a health care environment characterized by increasing the complexity of both medical decision- making and technical radiation treatment planning.

About the Authors: Department of Radiation Medicine and Applied Sciences (EG, NP, AM, JM), University of California San Diego, La Jolla, CA. Address correspondence to: Erin F. Gillespie, MD, University of California San Diego, Department of Radiation Medicine and Applied Sciences, 3960 Health Sciences Dr., MC0865, La Jolla, CA 92093-0865, Phone: 858-822-6080. Fax: 858-246-1505. E-mail: efgillespie@ucsd.edu

Disclosures: None.

Conflicts of Interest: None.

References

1. Peters, LJ, O’Sullivan B, Giralt J, et al. Critical impact of radiotherapy protocol compliance and quality in the treatment of advanced head and neck cancer: results from TROG 02.02. J Clin Oncol. 2010 Jun 20;28(18):2996-3001. doi: 10.1200/JCO.2009.27.4498.
2. Wuthrick EJ, Zhang Q, Machtay M, et al. Institutional clinical trial accrual volume and survival of patients with head and neck cancer. J Clin Oncol. 2015 Jan 10;33(2):156-64. doi: 10.1200/JCO.2014.56.5218.
3. Abrams, RA, Winter KA, Regine WF,et al., Failure to adhere to protocol specified radiation therapy guidelines was associated with decreased survival in RTOG 9704--a phase III trial of adjuvant chemotherapy and chemoradiotherapy for patients with resected adenocarcinoma of the pancreas. Int J Radiat Oncol Biol Phys. 2012 Feb 1;82(2):809-16. doi: 10.1016/j.ijrobp.2010.11.039.
4. Chao KS, Deasy JO, Markman J. A prospective study of salivary function sparing in patients with head-and-neck cancers receiving intensity-modulated or three-dimensional radiation therapy: initial results. Int J Radiat Oncol Biol Phys. 2001 Mar 15;49(4):907-16.
5. Eisbruch A1, Harris J, Garden AS, et al. Multi-institutional trial of accelerated hypofractionated intensity-modulated radiation therapy for early-stage oropharyngeal cancer (RTOG 00-22). Int J Radiat Oncol Biol Phys. 2010 Apr;76(5):1333-8. doi: 10.1016/j.ijrobp.2009.04.011.
6. Duhmke, E., et al., Low-dose radiation is sufficient for the noninvolved extended- field treatment in favorable early-stage Hodgkin’s disease: long-term results of a randomized trial of radiotherapy alone. J Clin Oncol, 2001. 19(11): p. 2905-14.
7. Fairchild, A., et al., Does quality of radiation therapy predict outcomes of multicenter cooperative group trials? A literature review. Int J Radiat Oncol Biol Phys, 2013. 87(2): p. 246-60.
8. Weber, D.C., et al., QA makes a clinical trial stronger: evidence-based medicine in radiation therapy. Radiother Oncol, 2012. 105(1): p. 4-8.
9. Chen, A.Y., et al., Improved survival is associated with treatment at high-volume teaching facilities for patients with advanced stage laryngeal cancer. Cancer, 2010. 116(20): p. 4744-52.
10. Wang, E.H., et al., Patients Selected for Definitive Concurrent Chemoradiation at High-volume Facilities Achieve Improved Survival in Stage III Non- Small-Cell Lung Cancer. J Thorac Oncol, 2015. 10(6): p. 937-43.
11. Chen, Y.W., et al., Association Between Treatment at a High-Volume Facility and Improved Survival for Radiation-Treated Men With High-Risk Prostate Cancer. Int J Radiat Oncol Biol Phys, 2016. 94(4): p. 683-90.
12. Boero, I.J., et al., Importance of Radiation Oncologist Experience Among Patients With Head-and-Neck Cancer Treated With Intensity-Modulated Radiation Therapy. J Clin Oncol, 2016. 34(7): p. 684-90.
13. Nattinger, A.B., et al., Geographic variation in the use of breast-conserving treatment for breast cancer. N Engl J Med, 1992. 326(17): p. 1102-7.
14. Lin, C.C., et al., Association Between Geographic Access to Cancer Care and Receipt of Radiation Therapy for Rectal Cancer. Int J Radiat Oncol Biol Phys, 2016. 94(4): p. 719-28.
15. Del Fiol, G., T.E. Workman, and P.N. Gorman, Clinical questions raised by clinicians at the point of care: a systematic review. JAMA Intern Med, 2014. 174(5): p. 710-8.
16. Covell, D.G., G.C. Uman, and P.R. Manning, Information needs in office practice: are they being met? Ann Intern Med, 1985. 103(4): p. 596-9.
17. Graber, M.A., et al., Answering clinical questions in the ED. Am J Emerg Med, 2008. 26(2): p. 144-7.
18. Lefresne, S., et al., Impact of quality assurance rounds in a Canadian radiation therapy department. Int J Radiat Oncol Biol Phys, 2013. 85(3): p. e117-21.
19. Marks, L.B., et al., Enhancing the role of case-oriented peer review to improve quality and safety in radiation oncology: Executive summary. Pract Radiat Oncol, 2013. 3(3): p. 149-156.
20. Thaker, N.G., et al., ReCAP: Assessing the Quality of a Radiation Oncology Case-based Peer-review Program in an Integrated Academic and Community Cancer Center Network. J Oncol Pract, 2016.
21. Choudhry, N.K., R.H. Fletcher, and S.B. Soumerai, Systematic review: the relationship between clinical experience and quality of health care. Ann Intern Med, 2005. 142(4): p. 260-73.
22. Hardenbergh, P.H. Driving Health Care Transformation and Better Outcomes. in American Society of Radiation Oncology (ASTRO) Annual Meeting. 2014. San Francisco, CA.
23. Housri, N. Can social media be used to convey reliable radiation oncology information? in American Society for Clinical Oncology (ASCO) Annual Meeting. 2014. Chicago, IL.
24. Grimshaw, J.M., et al., Disseminating and implementing guidelines: article 13 in Integrating and coordinating efforts in COPD guideline development. An official ATS/ERS workshop report. Proc Am Thorac Soc, 2012. 9(5): p. 298-303.
25. Grimshaw, J.M. and I.T. Russell, Achieving health gain through clinical guidelines II: Ensuring guidelines change medical practice. Qual Health Care, 1994. 3(1): p. 45-52.
26. Grimshaw, J.M. and I.T. Russell, Effect of clinical guidelines on medical practice: a systematic review of rigorous evaluations. Lancet, 1993. 342(8883): p. 1317-22.
27. Bonis, P.A., et al., Association of a clinical knowledge support system with improved patient safety, reduced complications and shorter length of stay among Medicare beneficiaries in acute care hospitals in the United States. Int J Med Inform, 2008. 77(11): p. 745-53.
28. Isaac, T., J. Zheng, and A. Jha, Use of UpToDate and outcomes in US hospitals. J Hosp Med, 2012. 7(2): p. 85-90.
29. Gregoire, V., et al., Delineation of the neck node levels for head and neck tumors: a 2013 update. DAHANCA, EORTC, HKNPCSG, NCIC CTG, NCRI, RTOG, TROG consensus guidelines. Radiother Oncol, 2014. 110(1): p. 172-81.
30. Lee, N.Y. and J.J. Lu, Target Volume Delineation and Field Setup: A Practical Guide for Conformal and Intensity-Modulared Radiation Therapy, ed. Springer. 2013.
31. NRG Oncology: Contouring Atlases. Available from: https://www.nrgoncology. org/Resources/Contouring-Atlases.
32. Fiorino, C., et al., Intra- and inter-observer variability in contouring prostate and seminal vesicles: implications for conformal treatment planning. Radiother Oncol, 1998. 47(3): p. 285-92.
33. Kosztyla, R., et al., Evaluation of dosimetric consequences of seroma contour variability in accelerated partial breast irradiation using a constructed representative seroma contour. Int J Radiat Oncol Biol Phys, 2012. 84(2): p. 527-32.
34. Yamazaki, H., et al., Quantitative assessment of inter-observer variability in target volume delineation on stereotactic radiotherapy treatment for pituitary adenoma and meningioma near optic tract. Radiat Oncol, 2011. 6: p. 10.
35. Hong, T.S., et al., Interobserver variability in target definition for hepatocellular carcinoma with and without portal vein thrombus: radiation therapy oncology group consensus guidelines. Int J Radiat Oncol Biol Phys, 2014. 89(4): p. 804-13.
36. Hong, T.S., W.A. Tome, and P.M. Harari, Heterogeneity in head and neck IMRT target design and clinical practice. Radiother Oncol, 2012. 103(1): p. 92-8.
37. Cui, Y., et al., Contouring variations and the role of atlas in non-small cell lung cancer radiation therapy: Analysis of a multi-institutional preclinical trial planning study. Pract Radiat Oncol, 2015. 5(2): p. e67-75.
38. ASTRO eContouring Webinar Series. Available from: https://www.astro.org/ Educational-Resources/Webinars/eContouring-Webinar-Series.aspx.
39. AAMD Foundation: Anatomy and Contouring Webinar Series. 2014; Available from: https://www.regonline.com/Register/Checkin.aspx?EventID=1471648.
40. Berkowitz, S.J., et al., Resident iPad use: has it really changed the game? J Am Coll Radiol, 2014. 11(2): p. 180-4.
41. Nabavizadeh, N., et al., Results of the 2013-2015 Association of Residents in Radiation Oncology Survey of Chief Residents in the United States. Int J Radiat Oncol Biol Phys, 2016. 94(2): p. 228-34.