Understanding disagreement

papers review: One line summary: The paper looked at doctors diagnosing sleep disorders from reading an EEG and categorizing it as one of three broad disorders – Alzheimers, Sleep apnea, Parkinsons – and also healthy individuals. It focused on how argumentation could change a doctor’s recommendation, and where and why disagreements persisted.

Title: Understanding Expert Disagreement in Medical Data

Analysis through Structured Adjudication

Authors: Mike Schaekermann, University of Waterloo, Canada Graeme Beaton, University of Waterloo, Canada Minahz Habib, University of Toronto, Canada Andrew Lim, University of Toronto, Canada Kate Larson, University of Waterloo, Canada Edith Law, University of Waterloo, Canada

My highlights:

2. Related work – 2.2 Systems for group Deliberation

This section outlines a set of researched deliberation systems (that I had no idea about, you might have heard of them). Namely the Delphi method https://en.wikipedia.org/wiki/Delphi_method 3 – which has panels of experts fill out questionnaires. They also bring up MicroTalk (described better here: http://edithlaw.ca/papers/ambiguity.pdf 2) That linked paper is also interesting on this topic. Basically, people make a decision and provide a justification for it – an algorithm selects certain arguments that they have made, and presents counter arguments. The person is asked to re-evaluate their decision. This has some overlap with committee – though not fully. We basically do this in person, with no record, and often only partially explained.

I am also thinking of Chantal Mouffe’s Agonism https://en.wikipedia.org/wiki/Agonism 1 – Still reading a lot and thinking a lot. Not sure where this will lead.

Structured Adjunction

Here the researchers present the structured adjunction they developed in order to bring the members of the expert panel into agreement. The basic form is similar to Delphi, the medical researchers were given a format to fill out (see rule based representation) to argue their classification of an EEG. They were then shown a counter argument from another expert and allowed to adjust their decision (like in MicroTalk). This expanded into a rational and a set of evidence.

7 Results 7.2 Why do disagreements persist?

This is a neat section – it covers what did and did not get resolved. For example – when experts had variability in experience level, disagreements were more quickly resolved with adjunction. What was not resolved quickly, was when a feature was detected by one medical researcher, and not another.

Most importantly, however, across all variable groups, it were the feature-level variables that showed the greatest effect sizes for case resolvability overall. This finding confirms our hypothesis H2d, the claim that the structure of a disagreement, with respect to feature-level rationales , holds the greatest explanatory power for why disagreements remain unresolved even after adjudication.

Emphasis is mine. This is really exciting for me. If the same holds true for committee – this might be something that is solvable through experimentation. We can verify or exclude a feature. But we would need a way to do this efficiently.

The method they used was also neat. They also tracked information about experience, but they marked it differently.