title: Creating General User Models from Computer Use
summary: Human-computer interaction has long imagined technology that understands
  us-from our preferences and habits, to the timing and purpose of our everyday
  actions. Yet current user models remain fragmented, narrowly tailored to
  specific apps, and incapable of the flexible reasoning required to fulfill
  these visions. This paper presents an architecture for a general user model
  (GUM) that learns about you by observing any interaction you have with your
  computer. The GUM takes as input any unstructured observation of a user (e.g.,
  device screenshots) and constructs confidence-weighted propositions that
  capture that user knowledge and preferences. GUMs can infer that a user is
  preparing for a wedding they're attending from messages with a friend. Or
  recognize that a user is struggling with a collaborator's feedback on a draft
  by observing multiple stalled edits and a switch to reading related work. GUMs
  introduce an architecture that infers new propositions about a user from
  multimodal observations, retrieves related propositions for context, and
  continuously revises existing propositions. To illustrate the breadth of
  applications that GUMs enable, we demonstrate how they augment chat-based
  assistants with context, manage OS notifications to selectively surface
  important information, and enable interactive agents that adapt to preferences
  across apps. We also instantiate proactive assistants (GUMBOs) that discover
  and execute useful suggestions on a user's behalf using their GUM. In our
  evaluations, we find that GUMs make calibrated and accurate inferences about
  users, and that assistants built on GUMs proactively identify and perform
  actions that users wouldn't think to request explicitly. Altogether, GUMs
  introduce methods that leverage multimodal models to understand unstructured
  context, enabling long-standing visions of HCI and entirely new interactive
  systems that anticipate user needs.
paper_authors._id: 682acae37cb97ecc4a4cd8f0
paper_authors.name: Omar Shaikh
paper_authors._id: 682acae37cb97ecc4a4cd8f1
paper_authors.name: Shardul Sapkota
paper_authors._id: 682acae37cb97ecc4a4cd8f2
paper_authors.name: Shan Rizvi
paper_authors._id: 682acae37cb97ecc4a4cd8f3
paper_authors.name: Eric Horvitz
paper_authors._id: 682acae37cb97ecc4a4cd8f4
paper_authors.name: Joon Sung Park
paper_authors._id: 682acae37cb97ecc4a4cd8f5
paper_authors.name: Diyi Yang
paper_authors._id: 682acae37cb97ecc4a4cd8f6
paper_authors.name: Michael S. Bernstein
paper_title: Creating General User Models from Computer Use
paper_summary: Human-computer interaction has long imagined technology that understands
  us-from our preferences and habits, to the timing and purpose of our everyday
  actions. Yet current user models remain fragmented, narrowly tailored to
  specific apps, and incapable of the flexible reasoning required to fulfill
  these visions. This paper presents an architecture for a general user model
  (GUM) that learns about you by observing any interaction you have with your
  computer. The GUM takes as input any unstructured observation of a user (e.g.,
  device screenshots) and constructs confidence-weighted propositions that
  capture that user knowledge and preferences. GUMs can infer that a user is
  preparing for a wedding they're attending from messages with a friend. Or
  recognize that a user is struggling with a collaborator's feedback on a draft
  by observing multiple stalled edits and a switch to reading related work. GUMs
  introduce an architecture that infers new propositions about a user from
  multimodal observations, retrieves related propositions for context, and
  continuously revises existing propositions. To illustrate the breadth of
  applications that GUMs enable, we demonstrate how they augment chat-based
  assistants with context, manage OS notifications to selectively surface
  important information, and enable interactive agents that adapt to preferences
  across apps. We also instantiate proactive assistants (GUMBOs) that discover
  and execute useful suggestions on a user's behalf using their GUM. In our
  evaluations, we find that GUMs make calibrated and accurate inferences about
  users, and that assistants built on GUMs proactively identify and perform
  actions that users wouldn't think to request explicitly. Altogether, GUMs
  introduce methods that leverage multimodal models to understand unstructured
  context, enabling long-standing visions of HCI and entirely new interactive
  systems that anticipate user needs.
paper_ai_summary: GUMs, a multimodal user model architecture, learns user preferences and habits from unstructured observations to enhance chat-based assistants and interactive agents by inferring context and making proactive suggestions.
paper_ai_keywords: multimodal models unstructured context user model propositions inferring context proactive assistants interactive agents