Keywords and Phrases
Artificial Intelligence; Human Machine Teams; Human Systems Integration; Recommendation System; Uncertainty
“The use of Artificial Intelligence (AI) decision support systems is increasing in high-stakes contexts, such as healthcare, defense, and finance. Uncertainty information may help users better leverage AI predictions, especially when combined with domain knowledge. I conducted two human-subject experiments to examine the effects of uncertainty information with AI recommendations. The experimental stimuli are from an existing image recognition deep learning model, one popular approach to AI. In Paper I, I evaluated the effect of the number of AI recommendations and provision of uncertainty information. For a series of images, participants identified the subject and rated their confidence level. Results suggest that AI recommendations, especially multiple, increased accuracy and confidence. However, uncertainty information, which was represented visually with bars, did not significantly improve participants' performance. In Paper II, I tested the effect of AI recommendations in a within-subject comparison and the effect of more salient uncertainty information in a between-subject comparison in the context of varying domain knowledge. The uncertainty information combined both numerical (percent) and visual (color-coded bar) formats to make the information easier to interpret and more noticeable. Consistent with Paper I, results suggest that AI recommendations improved participants’ accuracy and confidence. In addition, the more salient uncertainty information significantly increased accuracy, but not confidence. Based on a subjective measure of domain knowledge, participants had higher domain knowledge for animals. In general, AI recommendations and uncertainty information had less of an effect as domain knowledge increased. Results suggest that uncertainty information, can improve accuracy and potentially decrease over-confidence”--Abstract, page iv.
Canfield, Casey I.
Shank, Daniel Burton
Dagli, Cihan H., 1949-
Engineering Management and Systems Engineering
M.S. in Engineering Management
Missouri University of Science and Technology
Journal article titles appearing in thesis/dissertation
- Communicating Uncertain Information from Deep Learning Models in Human Machine Teams
- Role of Uncertainty Information and Domain Knowledge in Use of AI Recommendations
x, 68 pages
© 2021 Harishankar Vasudevanallur Subramanian, All rights reserved.
Thesis - Open Access
Subramanian, Harishankar Vasudevanallur, "Communicating uncertain information from deep learning models to users" (2021). Masters Theses. 8001.