Collective wisdom and decision making in surgical oncology

Colorectal cancer management (Europe) 

The International Conference on “Multidisciplinary Rectal Cancer Treatment: Looking for a European Consensus” (EURECA-CC2) provides an example of the Delphi process in action. The Conference was organised in Italy and endorsed by the European Society of Medical Oncology (ESMO), European Society of Surgical Oncology (ESSO), and the European Society of Therapeutic Radiation Oncology (ESTRO), and 34 specialists with a range of expertise were invited to form the Panel. The Delphi document was available to all Committee members as a web-based document customized for the consensus process. Eight chapters were identified: epidemiology, diagnostics, pathology, surgery, radiotherapy and chemotherapy, treatment toxicity and quality of life, follow-up, and research questions. For each chapter, and a series of statements were developed. Each member commented and voted three times.

The votes were anonymous and rounds one and two were asynchronous, with three weeks available for voting. Statements upon which an agreement was not reached after voting round 1 (24th September to 20th October) and voting round 2 (10th November to 1st December) were openly debated during the Consensus Conference in Perugia in Italy from 11th–13th December 2008. A hand-held televoting system collected the opinions of both the Committee members and the audience after each debate.

A final voting round took place by email after the conference between 20th January and 10th February 2009. There were 207 voted sentences. Of these, 86% achieved large consensus, 13% achieved moderate consensus, and only 3 resulted in minimum consensus. The results were summarised and published.8

Hepatic resection for metastatic colorectal cancer (Canada) 

The Candian Hepatico-Pancreatico-Biliary (HPB) Society nominated a Delphi panel of experts to provide quality of care guidelines for the management of patients undergoing hepatic resection for metastatic colorectal cancer (Ref). The CHPBS group of 16 experts included two medical oncologists, one anaesthetist, one hepatologist, one pathologist, one palliative care physicians and nine surgeons. A literature search produced a list of evidence-based statements as possible quality indicators. The voting took place by post.

In each Delphi round, each statement was scored on a 5-point Likert scale as follows: (1) do not include, (2) little reason to include, (3) could include, (4) should include, (5) must include. Replies were anonymous and asynchronous. The response was compiled as an Excel spreadsheet and those statements with a mean score of >4 were kept as quality indicators. Those with an average score of >3 but <4 were re-sent to participants to score again with any new statements developed from round 1. Statements scoring <3 were discarded. The process was repeated for the final round 3, with participants being asked to add a weighting score to the established quality indicators. A 100% response was received for all rounds. The panel identified 18 quality indicators for care of the patient undergoing hepatic resection from metastatic colorectal cancer.

These indicators included items of structure and organisation, including hospital volumes, the establishment of HPB MDTs, equipment and review systems; process and service targets for specialist referral, investigation and treatment; and outcome indicators.9

The research agenda of the American Society of Colon and Rectal Surgeons 

The Delphi method can be extended to the polling of the entire membership of a specialist Society, as illustrated by the following exercise conducted by the American Society of Colon and Rectal Surgeons (ASCRS) to reach a consensus on the research questions of highest importance in terms of clinical care.

A modified Delphi process was used. 203 responding members in Round 1 submitted 746 questions. These were reduced to 105 questions encompassing 21 topics in colorectal surgical practice. In Rounds 2 and 3, 399 and 360 respondents, respectively, further prioritised and refined the list of questions. The final 20 items included 14 questions related to colorectal cancer, and six on benign disease topics, which were scored and tabulated.

The authors reported that the research agenda produced by this study reflects the clinical issues of greatest importance to the ASCRS members, and would help direct future research.10

Oncosurge 

In 2005, an international panel of HPB surgeons set out to create a therapeutic decision model identifying appropriate procedure sequences for the management of colorectal metastases. The RAND Corporation/University of California, Los Angeles Appropriateness Method (RAM) was used to assess strategies of resection, local ablation and chemotherapy. After a comprehensive literature review, an expert panel rated the appropriateness of each treatment option for a total of 1,872 ratings decisions in 252 cases. A decision model was constructed, consensus measured and results validated using 48 virtual cases, and 34 real cases with known outcomes. The group described its methodology as follows:

“The RAM methodology is a modified Delphi process which invites experts to apply their judgments on the appropriateness of treatments for an average patient within each category, treated by the average provider in the average hospital. The term “average” is clearly defined. “Appropriateness” describes the relative weight of the benefits and risks of a medical procedure, such that a procedure is “appropriate” when the expected benefits outweigh the possible negative consequences by a sufficient margin. The definition of a “sufficient margin” recognises the subjectivity of the expectations and risk perceptions of the individual patient, and their clinician(s).11 Therefore, to be of clinical use, the decision model must rank and stratify benefit/risk and benefits ratios for each treatment strategy in each clinical presentation. A procedure is judged to be “uncertain” when benefits and risks are similar, or if there is disagreement within the expert panel”.

In this study, the appropriateness of different treatments was scored on a scale of 1 (highly inappropriate) to 9 (clearly appropriate) against each clinical presentation. For the purpose of analysis, ratings from 1-3 for any given treatment were considered inappropriate, 4 to 6 represented uncertainty, and 7 to 9 indicated appropriateness. The group rating was taken as the median judgment of the panelists. Uncertainty could indicate that the experts were uncertain, or that there was disagreement among the experts, which would indicate a need for further clinical research. Bimodal disagreement (1 to 3 v 7 to 9) led to a discussion preceding the second round of ratings.

An international panel of 16 oncologists, surgeons and radiologists with wide experience in the diagnosis and treatment of liver metastases were provided with a systematic review of the indications for surgical resection of colorectal liver metastases prepared by two of the authors of this study and funded by the British Cancer Research Campaign (now CRUK). Each model patient was assigned to one indication, but considered for each possible treatment.

A preliminary panel meeting established three treatment groups for consideration: liver resection, local ablation, and chemotherapy. For each treatment segment, all relevant possible patient characteristics were included in the list of indications.

A second meeting reviewed the results of the first round of ratings, and modifications were made to the list of treatment options and indications. Panelists then privately rated each of the three treatment groups separately. The strategies that emerged from the second round of ratings were incorporated into a decision matrix in the form of a computer program, the OncoSurge decision model. Validation of the model by the expert panel was accomplished using 48 representative virtual theoretical cases using the real decision characteristics identified by the appropriateness analysis. These 48 cases were created specifically to ensure that the expert panel had considered the full range of all potential clinical presentations, making the validation process span the full range of potential patients. The validation process using the 48 virtual theoretical cases was reinforced by the panelists using 34 real cases from their panelists' own clinical practices with known outcomes.

Overall consensus rates for a series of statements ranged from 93.4 to 99.1%. For example, absolute contraindications to liver resection were judged to be unresectable extrahepatic disease, more than 70% liver involvement, liver failure, and lack of surgically fitness. Recommendations were also made in respect of factors which did not influence treatment strategy; indications for immediate resection and after chemotherapy, and in respect of surgery in relation to other ablation techniques.

The model identified criteria for resectability for individual patients; made recommendations for optimal treatment strategies; and provided opportunities for medical education.

The authors concluded that the OncoSurge decision model for colorectal liver metastases combines the best available scientific evidence with the collective judgment of worldwide experts. It allows a clinician to optimise treatment in the context of local skills, resources for each patient. The methodology can be applied efficiently across health care systems worldwide, to decision making in multimodality therapy for other cancers, such as rectal cancer or adjuvant therapy for breast, lung, and pancreatic cancer.12

Development of quality indicators for patients undergoing colorectal cancer surgery 

McGory et al. reported using a modification of the RAND/UCLA Appropriateness Methodology to update National Cancer Institute-sponsored consensus panel guidelines for colorectal cancer surgery from 2000. They used structured interviews with experienced colorectal cancer surgeons and systematic reviews of the literature to identify candidate quality indicators addressing perioperative care for patients undergoing surgery for colorectal cancer. A panel of 14 colorectal surgeons, general surgeons, and surgical oncologists then evaluated and formally rated the indicators using the modified Delphi method.142 candidate indicators were identified in six broad domains: privileging (which addresses surgical credentials), preoperative evaluation, patient-provider discussions, medication use, intraoperative care, and postoperative management. The expert panel rated 92 indicators as valid, which address all domains of perioperative care and which could be used as quality performance measures and for quality-improvement programs.13

The Delphi approach to attain consensus in methodology of local regional therapy for peritoneal surface malignancy 

At the Fifth International Workshop on Peritoneal Surface Malignancy (PSM) in Milan, December 2006, the Delphi process was used to seek a consensus on technical aspects of cytoreductive surgery (CRS) and hyperthermic intraperitoneal chemotherapy (HIPEC). Topics discussed included pre-operative workup; eligibility to CRS+HIPEC; intra-operative staging system; technical aspects of surgery; residual disease classification systems; HIPEC: nomenclature and modalities; drugs, carrier solution and optimal temperature; morbidity grading systems. Conflicting points were voted upon in two rounds by an international panel in local-regional therapy.

Peritoneal surface malignancies (PSM) have been traditionally regarded as uniformly terminal conditions. The combination of cyto-reductive surgery and perioperative intraperitoneal chemotherapy has changed PSM management from palliation to possible cure. Due to the inherent differences in biological and clinical behaviour, the optimal adaptation of comprehensive treatment to each PSM is still a matter of debate. A session of “The Fifth International Workshop on Peritoneal Surface Malignancy” (Milan, Italy, December 4–6, 2006) was committed to reach a consensus pertaining to conceptual and technical aspects of the loco-regional treatment of each PSM. The consensus developing process was based on principles of the Delphi method. A total of 103 international experts from 17 countries were included in six Working Groups for each of the following PSM: peritoneal mesothelioma, abdominal sarcomatosis, carcinomatosis of gastric, colo-rectal, appendiceal, and ovarian origin. The respective working groups wrote evidence reports. The main conflicting points regarding preoperative evaluation, patient eligibility, combined treatment methodology, postoperative follow-up and future investigational perspectives were listed as multiple-choice questions. 160 Conflicting Points were identified. A consensus of more than 50% of voters favoring one option was reached in 143 of these.14, 15

Other oncological applications of the Delphi process 

The Delphi process has been used on an occasional basis to identify priorities in cancer nursing service provision and cancer educational priorities for GPs.16 Hall et al. reported the use of the Delphi method to prioritise funding decisions at the National Cancer Institute,17 while Brook et al. reported its use in assessing medical technologies.18

Other collective knowledge systems 

The Internet is driving new commercial systems to capture the power of collective intelligence embodied in the conscious and subconscious choices and preferences that individuals make whenever they use the Web. While we are a long way from capturing the collective decision making of clinicians and MDTs on a case-by-case basis and hence from learning how to optimise individual cancer outcomes, we can at least start to see how this might be possible in a wired and networked world.

The wisdom of crowds is a concept which describes the aggregation of information in groups, resulting in decisions that may be better than could have been made by any single member of the group. The most meaningful outputs are derived from groups and “crowds” where there is diversity of opinion; independence of opinion; decentralisation and specialisation; and a method of aggregating the collective judgements into a collective decision.19 Groups and crowds are also capable of irrational behaviour and of collective failures of intelligence and common sense, where the groups are too homogeneous, centralised, bureaucratised or divided. Crowdsourcing is the study of information inherent in the actions of large numbers of independently acting individuals, leading to measurable collective outputs (for example in marketing or politics).