  For video or web services, it is crucial to measure user-perceived quality of
experience (QoE) at scale under various video quality or page loading delays.
However, fast QoE measurements remain challenging as they must elicit
subjective assessment from human users. Previous work either (1) automates QoE
measurements by letting crowdsourcing raters watch and rate QoE test videos or
(2) dynamically prunes redundant QoE tests based on previously collected QoE
measurements. Unfortunately, it is hard to combine both ideas because
traditional crowdsourcing requires QoE test videos to be pre-determined before
a crowdsourcing campaign begins. Thus, if researchers want to dynamically prune
redundant test videos based on other test videos' QoE, they are forced to
launch multiple crowdsourcing campaigns, causing extra overheads to
re-calibrate or train raters every time.
  This paper presents VidPlat, the first open-source tool for fast and
automated QoE measurements, by allowing dynamic pruning of QoE test videos
within a single crowdsourcing task. VidPlat creates an indirect shim layer
between researchers and the crowdsourcing platforms. It allows researchers to
define a logic that dynamically determines which new test videos need more QoE
ratings based on the latest QoE measurements, and it then redirects
crowdsourcing raters to watch QoE test videos dynamically selected by this
logic. Other than having fewer crowdsourcing campaigns, VidPlat also reduces
the total number of QoE ratings by dynamically deciding when enough ratings are
gathered for each test video. It is an open-source platform that future
researchers can reuse and customize. We have used VidPlat in three projects
(web loading, on-demand video, and online gaming). We show that VidPlat can
reduce crowdsourcing cost by 31.8% - 46.0% and latency by 50.9% - 68.8%.

Human-Computer Interaction;Networking and Internet Architecture