The initial experiment will deploy a previously validated machine learning algorithm trained on existing clinical datasets within simulation scenarios in which a patient with acute gastrointestinal bleeding (at low, moderate, and high risk for poor outcome) is evaluated.~Prior to the simulation, a baseline educational module about artificial intelligence, machine learning, and clinical decision support will be provided to all participants. The investigators will establish psychological safety by detailing what is available in the room, the opportunity to call a consultant, and availability of laboratory and radiographic studies. Each clinical scenario will run for approximately 10 minutes based on real patient cases where vital signs change over time and laboratory values are made available at specific points in the assessment. The study follows a 3 (scenario risk: low, medium, vs. high) by 2 (AI framing: tool vs. team member) by 2 (transparency: low vs. high) factorial design. Each participant receives all three scenarios in increasing order of risk (within-subject factor), but is randomized to one AI framing and one transparency treatment condition (between-subjects factors).~For the AI team member framing treatment, during the pre-simulation briefing, participants will be provided a framework of artificial intelligence / machine learning algorithms as a team member, and not just a tool, with the introduction of the concept of a theory of mind for algorithms to understand how their reasoning may complement human reasoning. Furthermore, the computer workstation upon which the algorithm is run will be placed by patient bedside. In addition, the AI-CDSS will be presented in a ChatGPT/GPT-4 chatbot style interface that providers can naturally interact with at the bedside. In the tool framing condition, they will be provided a typical explanation of AI-CDSS as a tool. In this case, the computer with the algorithm will be placed in a corner away from the patient. As such, it will be presented as a calculator-type stool.~For the transparency treatment, the algorithmic output will have either low transparency (with only the risk assessment and recommendation) or high transparency (with an explanation of the model inputs and the ability to query inputs either through user directed clicks in a web app in the tool framing condition or directly with the ChatGPT derived interface that can process user directed queries in a user-friendly natural language chatbot in the team member framework, resulting in presentation of inputs that affect the model output to provide a mental model of the algorithm).
The purpose of this research study is to measure usability, attitudes, and provider trust when interacting with a machine learning algorithm and clinical decision support system in the setting of bleeding from the upper gastrointestinal tract (upper GIB). Specifically, the investigators are looking to assess the optimal implementation of such machine learning algorithms in simulation scenarios to best increase utilization and improve outcomes. Participants will be randomized to different implementation settings for the machine learning algorithm and exposed to simulation cases of upper GIB to determine how they interact with the machine learning algorithm.