Cross-Reality Re-Rendering: Manipulating between Digital and Physical Realities

The advent of personalized reality has arrived. Rapid development in AR/MR/VR enables users to augment or diminish their perception of the physical world. Robust tooling for digital interface modification enables users to change how their software operates. As digital realities become an increasingly-impactful aspect of human lives, we investigate the design of a system that enables users to manipulate the perception of both their physical realities and digital realities. Users can inspect their view history from either reality, and generate interventions that can be interoperably rendered cross-reality in real-time. Personalized interventions can be generated with mask, text, and model hooks. Collaboration between users scales the availability of interventions. We verify our implementation against our design requirements with cognitive walkthroughs, personas, and scalability tests.

Introduction
Cross-reality systems provide users with access to information and objects between multiple realities from either reality. In prior work in augmented and mixed reality, cross-reality systems are methods that transition the level of virtuality along the reality-virtuality continuum (Milgram and Kishino, 1994). Gruenefeld et al. (2022) demonstrated how an end-user can change the level of virtuality of their physical environment, from one where real objects become virtual objects, to increasing levels of virtuality of the surrounding physical environment, to a completely virtual environment. Prototypes such as VRoamer (Cheng et al., 2019) or ShareVR (Gugenheimer et al., 2017) require significant client-side hardware in enabling real-time cross-reality rendering along the continuum. Physical and virtual realities tend to be the most common target realities to sample objects, information, and scene artifacts. Head-mounted displays are the primary interface to access cross-reality rendering.
Motivated by how much time humans spend time in both the physical and digital reality, we investigate how to support the rendering of objects of one reality in the other. An underlying assumption is that the physical reality and virtual reality have semantic mappings, where the objects from one reality can be mapped to another reality. Unlike virtual reality, the digital reality is not viewed through egocentric vision but through heterogeneous graphical user interfaces (e.g. media ranging from text to video, programs ranging from webpages to apps, operating systems ranging from Android to Windows, devices ranging from mobile to desktop). The view is different, partly due to the modes of interaction in each reality. The physical reality requires physical movement (e.g. gaze, body) for a user to actively seek information, while the digital reality is designed such that information flows to the user with minimal interaction (at most requiring finger action). The difference in view and affordances contributes to a non-trivial semantic mapping of objects in the digital (a) User authentication: Secure gateway to view history, devices, and camera feed.
(b) Interface & interventions selection: Registered devices and available interventions.
(c) Interface access: Accessing a Linux desktop from another (Linux) desktop browser.
(d) Interface access: Camera feed on the secondary device (to be loaded in full-screen).
(e) Interface access: Accessing Android emulator from an Android device. reality to a physical reality. When considering alternative reality where this mapping is not provided, transitioning along the continuum between these realities become challenging. We are shifting away from transitioning between a physical and semantically-mapped reality. We investigate how end-users can assist cross-reality rendering by providing semantic information collaboratively. Other than a semi-supervised approach to constructing mappings between both realities, enabling end-users to author or reflect on their experiences have brought benefits in augmented (Qian et al., 2022) and digital (Cho et al., 2021) realities.
Contributions We are the first to contribute a cross-reality rendering system that allows users to manipulate their digital and physical realities in real-time with respect to each other. We are the first to implement an interface-agnostic modification framework, which modifies digital interfaces agnostic to operating system, program, or content. Human-inthe-loop learning enables users to construct interventions, and collaboration scales the pool of interventions. Along with implementation details, we validate requirements with walkthroughs, personas, and scalability tests.

Related Work
Distinguishing reality. Milgram and Kishino (1994) elicited the reality-virtuality continuum, enabling the interpolation of elements of reality and elements of virtuality. We refer a reality to the environment in which the user manifests, perceives, and interacts with objects. Objects are manifestations of information. The view refers to the user's point-ofview that allows them to perceive the reality at any given moment (e.g. first-person-view). An interface is a tool that enables the user to interact with their reality (e.g. device screen, head-mounted display). The virtual reality has been primarily used as a source of virtual objects, with the intention that these objects can be overlayed onto the physical reality at varying levels. The digital reality, contrarily, is not a specially-designed reality for objects to be overlayed onto a physical reality (e.g. the digital reality is not geo-spatially organized). Given the level of immersion a human has in both physical and digital realities, unlike prior work in AR/MR/VR, we aim to enable users to manipulate both their physical and digital reality, extensibly treating them as a single reality.
Manipulating physical reality. Rather than immersing in complete reality or virtuality, we blend between the two along the continuum. Two general approaches to the manipulation of real/virtual objects in the physical reality are augmented reality and diminished reality (Mann, 1994). Augmented reality adds virtual information onto a reality. NaviCam (Rekimoto and Nagao, 1995) was early work demonstrating the placement of messages on video screens. VRCeption (Gruenefeld et al., 2022) enables users to dynamically transition along the reality-virtually continuum. ScalAR (Qian et al., 2022) enables users to author virtual objects to be placed in their physical reality. Diminished reality, on the other hand, removes real objects from reality. It erases physical objects through inpainting (Herling and Broll, 2012), approximation (Yue et al., 2017), or multiple cameras (Lindlbauer and Wilson, 2018). Software from digital interfaces can be ported to be used alongside AR/MR/VR systems (e.g. Meta (2022)). While this enables the usage of the software itself, each reality is still compartmentalized. For example, what one does on their desktop browsing sessions would play no effect on their physical world viewing experience.
Manipulating digital reality. The main challenges faced by frameworks that modify digital interfaces are: interoperability between programs (apps, browsers) and OS, requiring escalation of privilege, significant development/maintenance effort of modifications (e.g. patches break with version changes). Code modifications make changes to source code, either installation code to modify software before installation, or run-time code to modify software during usage (e.g. browser extensions for desktop/mobile, Cydia Substrate (Freeman, 2020) for iOS, Xposed Framework (rovo89, 2020) for Android). External modifications require installing a program that affects other programs (e.g. usage tracking with HabitLab (Geza, 2019)). Overlay modifications render graphics on an overlay layer over an active interface instance (e.g. occluding inappropriate text/images with models (Datta et al., 2021;2022)).

Cross-Reality System Architecture
We detail the implementation of the system base, upon which functionality on rendering (Section 4) and playback (Section 5) is supported, and further functionality can be added. All images are captured with our working prototype.
We outline the architectural components as follows: 1 The user logs into the system to access their camera feed or a set of personal emulators, along with a set of interventions. The system admin has provisioned a set of emulated devices, hosted on virtual machines on a server. The user places a (secondary) hand-held device on their head-mounted display, where the camera feed is streamed to a server and a re-rendered feed is loaded. 2 The user selects their desired interventions and views re-rendered interfaces. 3 The user accesses their view history and annotates graphics or text for generating interventions, which then re-populate the list of interventions available to members in a network.
The user accesses a web application (compatible with desktop/mobile browsers). With their login credentials, the database loads the corresponding mapping of the user's virtual machines that are shown in the interface selection page (Figure 1(b)). The server carries information on accessing a set of emulated devices. Each emulator is rendered in virtual machines where input commands are redirected. While all digital realities are loaded server-side and streamed directly to the user device, the physical reality is captured from a (secondary) device placed on a head-mounted display (Figure 2), where images are sent from the device camera to the server, then processed and loaded in real-time on the camera feed page (Figure 1(d)). The moment-by-moment changes on a person's screen or camera feed can be captured as images at a configurable framerate. The captured images are processed and rendered for the user to observe their reality, and also displayed to the user in their view history. Further, the database loads the corresponding mapping of available interventions (generated by the user, or by the network of users) in the interventions selection page. The database also loads the view history in the view history page (Figure 8). This consists of images of timestamped, visited realities, including chronological images from the camera feed as well as that of the emulators.
Input commands for desktop/mobile are captured and directed to the emulator, including keystrokes (hardware keyboard, on-screen keyboard) and mouse/touch events (scrolling, swiping, pinching, etc). Screen and camera images are captured at 60 FPS into an images directory. Generated masks and fine-tuned models are stored under an interventions directory. Images and interventions are accessible to their corresponding user. Interventions are applied sequentially upon an image to return a perturbed image, which then updates the rendered image on the client web app.

Rendering functionality
We can decompose rendering functionality into two components: (i) mapping objects between realities, and (ii) manipulating the objects within a reality. In virtual reality, mapping between real and virtual objects tend to be performed based on relative location in a given view. As the digital reality is not a geo-spatial reality, we need an alternative mapping strategy. We pursue one where we map based on properties of the objects (e.g. physical Figure 2: Head-mounted display: The user's view (left), and the external look (right) (with the handheld device inside, its back-facing camera exposed for capturing images).
appearance/attributes) between realities. As such, we built hooks to support generalizable identification of object properties between realities. We ideate these reality-agnostic hooks based on observed patterns between realities. For example, objects in realities can be encoded in text or graphics, or can be processed by a learnt model if necessary. Hooks operate time-agnostically: they can be used to identify object properties during annotation in playback as well as facilitate object manipulation during real-time re-rendering. As images are the primary medium of rendering over a reality, the hooks deal with pixel-based inputs and outputs (e.g. though raw text can be extracted from a device emulator, we use OCR to enable generalizable text detection such as in physical realities). Overlay modifications are interface-agnostic, enabling the same intervention to scale to all interfaces where the modification condition exists. It does not require escalation of privilege or modification of source code, and is thus easy to use for users and developers.
The input receivers for the hooks begin from the input devices (head-mounted feed, device emulators), and into the annotations on the view history. The output receivers for the hooks begin from the object renders superimposed on the base reality, and onto the stream of the reality displayed. Depending on the objective of the rendering task, how the output of the hooks manifest in the re-rendered reality can vary. The capability of output renders is independent of the hooks, given that sufficient information or attributes is provided by the hooks. We focus on diminished reality, where we work on reducing visibility on objects. On the other end of the spectrum, augmented reality would be more involved in the addition of new objects or adding/modifying properties of existing objects.
The text hook enables modifying the text on an interface. Character-level optical character recognition (OCR) takes an image as input and returns a set of characters and their geometric coordinates. We first identify a set of regions containing text with EAST text detection (Zhou et al., 2017). We then use Tesseract (Google, 2007) to extract characters within each region.
With the availability of real-time textual data from each image instance, an intervention developer can store information processed by the image to be processed by subsequent models via the model hook. A sample application of this hook include interventions against text of specific conditions (e.g. placing censor boxes over hate speech, or generating new text personalized to the user). Another example is the identification and highlighting of specific text used in one interface (e.g. product ads on Facebook) and appearing in another (e.g. search results in Amazon, or appearing in real-life when in a store).
The mask hook matches the current image against a target template of multiple images.
Multi-scale multi-template matching resizes an image multiple times and samples different subimages to compare against each mask instance. The mask hook can be augmented; for example, using the matching algorithm with contourized images (shapes, colour-independent) or coloured images depending on whether the mask contains (dynamic) objects. The mask hook could be connected to rendering functions such as highlighting the interface element with warning labels, or image inpainting (fill in the removed element pixels with newly generated pixels from the background). Given the higher likelihood of non-variability of object instances in the digital world, the applications of this hook would be expectedly predominant in the digital reality. An example application is the user can capture a mask of the share buttons on YouTube, and as long as this design is used across all interfaces of YouTube from Android to iOS to browser, this one mask can be reused. Having detected the coordinates of the mask on a given image, the detected object could be occluded, highlighted, or inpainted, etc.
A model hook loads a model to take an input and generate an output. This enables the embedding of models (i.e. model weights and architectures) to inform further overlay rendering. We can connect models trained on specific tasks (e.g. person pose detection, emotion/sentiment analysis) to return output given the image (e.g. bounding box coordinates to filter), and this output can then be passed to a pre-defined rendering function (e.g. draw filtering box).
We enable end-users to tune or adapt their own personalized models using an annotation interface and model adaptation mechanisms. Our implementation specifically relies on fine-tuning, but we also review few-shot learning and prompt-tuning. For model fine-tuning, the developer re-trains a pre-trained model on a new dataset. This is in contrast to training a model from a random initialization. Fine-tuning techniques for pre-trained models, which already contain representations for feature reuse, have indicated strong performance on downstream tasks (Galanti et al., 2022;Abnar et al., 2022;Neyshabur et al., 2020). To retain representations of older tasks or batches of data, online/continual learning methods can assist in reducing catastrophic forgetting (Kirkpatrick et al., 2017;Datta and Shadbolt, 2022c). If there is a large number of input distributions and few samples per distribution, few-shot learning is an approach where the developer separately trains a meta-model that learns how to change model parameters with respect to only a few samples. Few-shot learning has demonstrated successful test-time adaptation in updating model parameters with respect to limited test-time samples (Raghu et al., 2020;Koch et al., 2015;Finn et al., 2017;Datta, 2022). Some overlapping techniques even exist between few-shot learning and fine-tuning, such as constructing subspaces and optimizing with respect to intrinsic dimensions (Aghajanyan et al., 2021;Datta and Shadbolt, 2022b;Simon et al., 2020;Datta and Shadbolt, 2022a). Prompt tuning is an alternative adaptation approach that does not require changes in the parameters of the downstream model. It is a technique that leverages the use of specific conditioning inputs (e.g. a phrase at the start of a sentence) to condition a foundation model to perform a specific downstream task (Lester et al., 2021). Steps towards scaling the quantity of prompts have been undertaken, from PromptSource (Bach et al., 2022), a prompt repository and tool used for creating and sharing prompts, to PromptGen , a method for dynamic prompt generation.

Playback functionality
Human-in-the-Loop (HITL) learning is the procedure of integrating human knowledge and experience in the augmentation of machine learning models. It is commonly used to generate new data from humans or annotate existing data by humans. Examples are reviewed in Wu et al. (2021).
A view history ( Figure 8) refers to the historical record (e.g. time series sequence of images) of a view of reality from the perceived viewpoint of the user. A user can inspect their view history across their camera feed and digital devices, and use image segment highlighting techniques to annotate interface patterns to detect and subsequently intervene against these patterns. The user can go through the sequence of images to reflect on their viewing patterns. When the user identifies a GUI element they do not wish to see across interfaces and apps, they highlight the region of the image, and annotate it as mask-<name-of-intervention>, and the mask hook will store a mask of intervention <name-of-intervention>, which will then populate a list of available interventions with this option, and the user can choose to activate it during a session. When a user identifies text (images) that they do not wish to see of similar variations, they can highlight the text (image) region, and annotate it as text-<name-of-intervention> (image-<name-of-intervention>). The text hook retrieves text, and fine-tunes a pre-trained text classification model on the group of text <name-of-intervention>. For images, the highlighted region will be cropped as input to fine-tune a pre-trained image classification model.

Cognitive Walkthrough
We perform a cognitive walkthrough ("show and tell rather than use and test") (John and Packer, 1995;Rieman et al., 1995) to simulate a user's cognitive process and explicit actions during usage. We as the authors presume the role of a user. For each step of the walkthrough, we first report the data pertaining to each task, then provide a descriptive evaluation. To evaluate the process of constructing an intervention, we track the completion of a set of required tasks (Table 1) based on criteria from Parasuraman et al. (2000)'s 4 types of automation applications, which aim to measure the role of automation in the intervention self-development process. This evaluates the ease and usability in generating interventions and viewing realities.
Step 1: User logs in (Fig. 1a) The user enters their username/password. These credentials are stored in a database mapped to specific virtual machines that contain the interfaces the user registered for access.
This is a standard step for any secured or personalized system, where a user is informed they are accessing data and information that is tailored for their own usage.
Step 2: User activates interface & interventions ( Fig. 1b) The user is shown a set of available interventions. They select their target interventions, and select an interface (digital or physical) to view. Users can view the set of interfaces that they can access and use to facilitate their viewing experiences. The interface is available 24/7, retains all their personal data and storage, is recording their view history data for review, and accessible via a web browser from any other device/platform. They are less constrained by the hardware limitations of their personal device. The populated interventions liken to a marketplace and ecosystem of personalized and shareable interventions. Users can populate interventions that they themselves can generate through the view history tool, or access interventions collaboratively trained and contributed by multiple members in their network. The interventions are also modular enough that users are not restricted to a specific combination of interventions, and can apply any combination of interventions sequentially onto the interface. As the capabilities of generating interventions become extended, so do their ability to personalize their experience, and generate a distribution of experiences to match a similarly wide distribution of users. The autonomy to deploy interventions, with more options from community contributions, before usage of an interface satisfies Task 4.
Step 3: User accesses the interface and browses ( Fig. 1c-e) The user interacts with the re-rendered interface through their mobile/desktop browser.
After the user has chosen their desired interventions, the user can improve their viewing experience through the manipulation of exposure to certain objects. The altered viewing experience satisfies both Tasks 1 and 4. Not only is raw image data being collected, but the view is being altered by deployed interventions in real-time. It is a cyclical loop where users redesign and self-improve their viewing experiences with user-driven tools.
Step 4: User inspects view history to generate interventions (Fig.  8) After a viewing period, the user may inspect their personal view history to create interventions. They enter the view history page to inspect recorded intervals of their viewing activity across all interfaces, and they can choose to annotate certain regions to generate interventions.
The user is given autonomy in manipulating aspects of their perceived reality. Enabling the user to inspect their view history across all used interfaces to self-reflect and analyze activity patterns satisfies Task 2. Though the view history provides the user raw historical data, it may require additional processing (e.g. automated analysis, charts) to avoid information overload. Rather than waiting for a feedback loop for altruistic developers (e.g. app modifications for digital reality, or dedicated AR software for physical reality) to craft broad-spectrum interventions that may not fit their personal needs, the user can enjoy a personalized loop of crafting and deploying interventions, almost instantly for certain interventions such as element masks. The user can enter metadata pertaining to each annotated object, and not only contribute to their own experience improvement, but also contribute to the improvement of others who may not have encountered the object yet. By developing interventions based on their analysis, not only for themselves but potentially for other users, they can achieve Task 3.
All four tasks, used to determine whether a complete feedback loop between input collection and interface rendering through HITL, can be successfully completed.

Personas
Given a set of personalized requirements per user, we evaluate the ability for the system to render interventions. We evaluate specifically in physical realities, as distributional-diversity of objects are higher in physical realities than digital realities. Mask hooks work well in highly-uniform realities, such as handling GUI elements in digital realities. Attributing to the non-uniformity in physical realities, we rely on adaptation with model hooks.
We evaluate using personas, which are descriptions of individual people who represent groups of users that would interact with our system. To construct each persona, we populate each hypothetical individual with information pertaining to their background (e.g. context on scenarios or requirements), scenarios (situations/scenarios prompting the persona to use our system), and finally requirements (discrete items that need to be satisfied, generally consistent needs throughout most scenarios). In-line with Matthews et al. (2012), to evaluate each persona, we (i) identify persona requirements (i.e. hypothesize scenarios where the persona need our system), and (ii) evaluate scenario responses (i.e. create real-world scenarios where the personas use the system and evaluate the alleviation of requirements).
Based on the personas constructed in Table 2, the visual appearance of the required object tends to be consistent regardless of digital or physical reality. To support each person's requirements, we first manually collected a set of images for each object (graves, price labels, bikes, dogs). With our view history tool, we sample 100 images per object by searching for each object on Google Images and annotating images with object labels and bounding boxes. Though curated datasets could be found (e.g. dogs in CIFAR10), we remain consistent in our sampling strategy for objects without datasets. We fine-tune a Faster R- CNN (Ren et al., 2015) model, pre-trained on MSCOCO (Lin et al., 2014), on each object. We replace the pre-trained head of the model with a new one containing the new class, and fine-tune until early-stopping at loss 0.1. This results in four models for each of the four objects. With the predicted bounding box coordinates, we apply a Gaussian blur to occlude the object. After activating these interventions, the authors put on the head-mounted display and enacted the hypothetical scenario in real-life. We visit physical locations (cemetery, supermarket, street with a cycle path, and dog park), and observe consistent occlusion of the target objects ( Figure 6). We find that most objects can be successfully occluded in real-time. Objects that fail to be occluded tend to be at rotated angles inconsistent with sampled images, be a large distance away from the user (i.e. a small set of pixels in the image pertain to the object), or are distributionally-distant from the source distribution of images (e.g. variations of dogs).

Scalability Testing
To evaluate the collaborative component, we measure the improvement to the user experience of a single user from the efforts of multiple users. We do not recruit real users, as it would constrain our performance evaluation to the number of users available, the evaluation period, intervention quality control, and diversity of recruited users. Instead, we evaluate through scalability testing, a type of load testing (Meerts and Graham, 2010) that measures a system's ability to scale with respect to the number of users.
One application we can use as a base for evaluating our system is the mitigation of different digital harms. Harms tend to be highly individual and vary in how they manifest within users of digital systems. The spectrum of harms range from heavily-biased content (e.g. disinformation, hate speech), self-harm (e.g. eating disorders, self-cutting, suicide), cyber crime (e.g. cyber-bullying, harassment, promotion/recruitment for extreme causes such as terrorism), to demographic-specific exploitation (e.g. child-unsafe content, social engineering attacks). We refer the reader to the extensive literature (HM, 2019;Pater and Mynatt, 2017;Honary et al., 2020;Pater et al., 2019;Shelby et al., 2022). In each subsection, we demonstrate interface-agnostic intervention of different harms specific to GUI elements (Section 5.3.1) and content (Section 5.3.2).
We simulate the usage of the system to evaluate the scalable generation of one-shot mask detection, and scalable fine-tuning of text models, in order to evaluate the strengths/weaknesses of the system's scalability. We measure the ease of intervention development with the number of variations of interventions generated (specifically element removal) (Table 3), rather than development time. We do not replicate the scalability analysis on real users: the fine-tuning mechanism is still the same, and the main variable (in common) is the sentences highlighted (and their assigned labels and metadata, as well as the quality of the annotations), though error is expectedly higher in the real-world as the data may be sampled differently and of lower annotation quality. The primary utility of collaboration to an individual user is the scaled reduction of effort in intervention development. We evaluate this in terms of variety of individualized interventions (variations of masks), and the time saved in constructing a single robust intervention (time needed to construct an accurate model intervention).

Persona Background Scenarios Requirements
Persona 1: graves They are afraid of death. They went through the trauma of losing a spouse and being forced to quickly bury them in the local cemetery. The sight of gravestones may cause panic attacks.
They may pass a graveyard in their local vicinity. Gravestones may also be located in impromptu locations (e.g. points of cycling accidents). Media may contain imagery of deaths, such as in news coverage or leisurely content.
Objects, be it in the digital or physical reality, that pertain to death should be occluded from view. An example of such an object is a gravestone.

Persona 2: prices
They lack self-control on spending. They grew up in poverty and their parents forbade any unnecessary purchases. They purchase anything 'cheap', even if they do not need it. They cannot inhibit their vice when shopping online or in real-life.
Priced goods may be shown on ecommerce platforms, advertisements on other webpages or apps, or brickand-mortar stores.
Objects, be it in the digital or physical reality, that display a price tag should be occluded.

Persona 3: bikes
They have anger issues towards cyclists. They witnessed a cyclist run their brother over. Now they enter a fit of rage whenever they see one. Not only does this hurt any passing cyclist's feelings who hear their harsh words without any context (and sometimes this escalates into a physical fight), but they also fail to concentrate throughout the rest of the day.
They live in a city with extensive cycling routes, resulting in a high prevalence of cyclists.
Objects, be it in the digital or physical reality, that pertain to cyclists should be occluded from view. An example of such an object is a bicycle. Physical safety of the user should be considered. For example, only occluding the bike but not traffic lights, or blurring the bike so that the user still retains depth-perception of an incoming object.

Persona 4: dogs
They are easily distracted. They grew up in an intimately-small family where there were twice as many dogs as there were humans. They now cannot control themselves when they see a dog of any shape or size in real-life. When they see a canine, they stop what they were doing, and start chasing after them.
Dogs can be present on the street, in the park, in the (pet-friendly) workplace, in indoor settings (e.g. cafes/restaurants), etc. Targeted advertising tends to show them dog products with their demo dogs. Videos online may also contain dogs. Though they cannot chase them into a screen, the distraction absorbs them and they enter a rabbit hole of browsing funny dog videos.
Objects, be it in the digital or physical reality, that pertain to pets (specifically dogs) should be occluded from view.   Baseline 1 user, 5 sents/day 5 users, 5 sents/day 1 user, 10 sents/day 5 users, 10 sents/day = 10 user, 5 sents/day 10 user, 10 sents/day Figure 7: Convergence of fine-tuned models on hate speech.

Scaling mask hooks
We investigate the ease to annotate graphically-consistent GUI elements (Table 3). We sample elements to occlude that can exist across a variety of interfaces. We evaluate the occlusion of the stories bar (predominantly only found on mobile devices, not desktop/browsers). Some intervention tools exist on Android (Studios, 2021; MaaarZ, 2019; Datta et al., 2021) and iOS (Studio, 2022), though the tools are app-(and version-) specific. We evaluate the occlusion of like/share metrics. There are mainly desktop browser intervention tools (Grosser, 2012;2018;hidelikes.com, 2022), and one Android intervention tool (Datta et al., 2021). We evaluate the occlusion of recommendations. There are intervention tools that remove varying extents of the interface on browsers (such as the entire newsfeed) (West, 2012;unhook.app, 2022). Existing implementations and interest in such interventions indicate some users have overlapping interests in tackling the removal or occlusion of such GUI elements, though the implementations may not exist across all interface platforms, and may not be robust to version changes. We evaluate each intervention on a range of target interfaces, specifically native apps (for Android and iOS) and browsers (Android mobile browser, and Linux desktop browser).
We use the view history tool to annotate and tag the minimum number of masks needed per element to block across a set of apps. There tends to be small variations in the design of the element between browsers and mobile, hence we tend to require at least 1 mask from each device type. Android and iOS apps tend to have similar enough GUI elements that a single mask can be reused between them. We tabulate in Table 3 the successful generation and real-time occlusion of all evaluated GUI elements. We append screenshots of the removal of recommended items from the Twitter and Instagram apps on Android (Figure 5a). We append screenshots of the de-metrification (occlusion of like/share metrics) of YouTube across desktop browsers (macOS) and mobile browsers (Android, iOS) (Figure 4).

Scaling model hooks
We investigate the accuracy gains from fine-tuning pre-trained text models as a function of user numbers and annotated sentence contributions (Figure 7). Specifically, we evaluate the text censoring of hate speech, where the primary form of mitigation is still community standard guidelines and platform moderation, with little user tooling available on Android (Bodyguard, 2019;Datta et al., 2021). The premise of this empirical evaluation is that we have a group of simulated users M who each contribute N inputs (sentences) of a specific target class (hate speech, specifically against women) per timestep. Baselined against a pre-trained model fine-tuned with all sentences against women, we wish to observe how the test accuracy of a model fine-tuned with M × N sentences varies over time. Our source of hate speech for evaluation is the Dynamically Generated Hate Speech Dataset (Vidgen et al., 2021), which contains sentences of non-hate and hate labels, and also classifies hate-labelled data by the target victim of the text (e.g. women, muslim, jewish, black, disabled). As we expect the M users to be labelling a specific niche of hate speech to censor, we specify the subset of hate speech of women (train set count: 1,652; test set count: 187). We fine-tune RoBERTa (HuggingFace, 2022; Liu et al., 2019), pre-trained on English corpora Wikipedia (Foundation, 2008) and BookCorpus (Zhu et al., 2015). For each user population M and sentence sampling rate N , at each timestep t, M × N × t sentences are acquired of class hate against target women; there are a total of 1,652 train set sentences under these constraints (i.e. the max number of sentences that can be acquired before it hits the baseline accuracy), and to balance the class distribution, we retain all 15,184 train set non-hate sentences. We evaluate the test accuracy of the fine-tuned model on all 187 test set women-targeted hate speech. We also vary M and N to observe sensitivity of these parameters to the convergence towards baseline test accuracy.
The rate of convergence of a fine-tuned model is quicker when the number of users and contributed sentences per timestep both increase, approximately when we reach at least 1,000 sentences for the women category. The difference in convergence rates indicate collaborative labelling scales the rate in which text of a specific category can be acquired. It reduces the burden on a single user of training text classification models from scratch and annotating text alone, diversifies the fine-tune training set, and avoids wasted effort in re-training models already fine-tuned by other users.
The empirical results from the scalability tests indicate that the ease of mask generation and model fine-tuning, further catalyzed by performance improvements from more users, enable the scalable generation of interventions.
6 Discussion 6.1 The line between virtual reality and physical reality Milgram and Kishino (1994) elicited the reality-virtuality continuum, enabling the interpolation of elements of reality and elements of virtuality. They denote the 'real' reality ('reality') as the reality where real objects have objective (physical) existence, and the 'virtual' reality ('virtuality') as the reality where virtual objects exist only in essence or effect. From this, one could conclude that many realities, or views that a user may perceive, are considered as virtual realities. However, we present some arguments below to revisit some considerations on when a reality is considered 'real' or 'virtual'.
Based on Milgram and Kishino (1994)'s definition, digital objects do not have an objective existence and would be deemed to be virtual. However, objects in the digital reality have increasingly real-world effects. The scope of physical manifestation affects the determination of the virtuality of an object. Exposure to objects in digital reality or dreams can affect the cognition of an end-user. They form new memories and associations, and representations/activations in the brain with respect to actions can change over time based on these new experiences. As such, there is a physical component and manifestation of these 'virtual' objects in the human brain. Similarly, an e-commerce platform could be interpreted as an input actuator to the transportation of physical objects, even though it does not have the same physical manifestation or even geo-spatial semantic mapping to a brick-and-mortar store. Skarbez et al. (2022) also revisit the assumptions of the continuum, and find that the 'perfect' virtual reality is unattainable. They find that any reality mediated with technology (computers, and extensibly brains) are mixed realities. They also find that modern virtual reality implementations lie in an interpolated position in the continuum rather than the virtuality endpoint, hence the realities that encompass objects of non-objective existence can be inferred to possess some properties of the physical reality.
The semantic mapping between physical and digital objects are difficult to form, given what ties the physical and digital realities is primarily how the interactions of the user has in each environment have an effect in the real-world (e.g. changes in user cognition, demand-supply of products, voting systems). In Milgram and Kishino (1994)'s continuum, there is an assumed morphism between a pair of realities, i.e. a known mapping scheme must exist and be known between a physical and virtual reality. However, in the digital reality, though we can assume that the objects between a physical and digital reality are mappable, there is no given mapping scheme (instead, we need to learn the scheme), which violates the rules of the reality-virtuality continuum, and thus violating the presumption that the digital world is a virtual reality based on the existence of objects. Though the objects from this pair of realities cannot be mapped based on affordances with respect to each object (e.g. touching a shoe in a store vs seeing a shoe on Amazon), they can be mapped to some extent based on the expectation of the outcome of an action (e.g. owning a pair of shoes upon purchase in a store vs Amazon).
While a pre-defined mapping is not provided between the physical and digital reality, as time elapses and the mapping tends towards completeness, does the status of a reality change accordingly? We investigated in this paper the potential for end-users to assist cross-reality rendering by providing semantic information collaboratively, i.e. using semi-supervised learning to scale semantic mapping. Given that objects can be interfaced in both the physical and digital reality (unlike a developer-controlled virtual reality) by any number of users publicly, semantic mapping and labelling can be done collaboratively. A semisupervised, specifically human-in-the-loop, approach to constructing mappings between both Figure 8: Re-rendering also works in virtual realities. Users can run games on Windows desktops, and the screen history can be annotated. Some of these interventions can also be reused in the digital/physical realities (e.g. if the game is in first-person view). This enables physical/digital reality interventions to transfer over to the virtual reality. It also allows us to augment the interventions by using a virtual reality as a safe annotation environment that approximates the geospatial properties of a physical reality. It also allows for easier video game modding without the need for modifying game source code or assets.
(a) Grand Theft Auto V: In first-person view, we annotate blood and gore (e.g. from shooting/stabbing).
(b) Call of Duty Modern Warfare II: In firstperson view, we annotate the removal of religious garments, to dissociate the anger towards an enemy with cultural characteristics.
(c) The Witcher 3: In third-person view, we annotate hanged execution.
(d) Red Dead Redemption 2: In third-person view, we annotate animals hunting. realities helps evaluate strategies towards an unsupervised approach to rendering cross-reality between physical and digital realities, while enabling an initial design to be useful in the near-term. Enabling end-users to author their own experiences with ScalAR (Qian et al., 2022) have demonstrated improved customization of objects and interactions in custom physical environments. screen history As the learnt mappings between the realities increase over time, in this case through collaborative data collection and intervention generation, one could argue that any reality can eventually be aligned with respect to the reality-virtuality continuum, and the lack of a pre-defined mapping is not an argument for suggesting the rules of the continuum are violatable.
In addition to be mappable by effect, objects could be mapped by their interventions. One can also question how disconnected a physical reality and digital reality is based on the overlap of required interventions. It could be observed that the issues faced in one reality tend to be consistent in the other. For example, distraction is something a user may take issue with in both physical and digital reality, and the corresponding interventions would be consistent in coccluding the distracting object or changing the lifestyle/habit or interaction flow accordingly.
Virtual realities are also self-contained, in the sense that a specific virtual reality can be rendered with respect to a physical reality, but different virtual realities do not render with respect to other virtual realities. Each virtual reality has fixed affordances, and the user usually has had minimal history in each virtual reality. Moreover, the instructions needed to modify either reality is distinct. Traditional virtual realities and the physical reality can be interacted with a similar interaction set (such as gestures, hand movements, touching) with appropriate extensions in the virtual realities. While physical actions dominate interaction in the physical reality, interactions in the digital reality require a different interaction set. Other than user digital actions (such as touching, pinching, typing), developer digital actions exist where modifying source code can modify the digital reality entirely. There is also a question regarding the level of autonomy that objects possess. Like real objects, digital objects have an autonomy of their own because they can be tugged around by the rest of the world (e.g. users on a community-driven platform can manipulate digital objects, even if it was originally initialized by an end-user creator). This parallels how a user can own possessions or be surrounded by objects, but they have no objective control over these objects. Changing source code is not enough to override its existence in the digital reality; this is unlike virtual reality objects, where changes to them take place primarily with source code changes by a core team of developers.
We also raise the question of perception mediation. If a user were to wear a cross-reality system for a very long duration of their lifetime's experiences, if the visual experience tends towards one where the cross-rendered reality is imperceptible from that of the underlying reality (i.e. the user cannot distinguish that reality is perturbed in any way), this perturbed reality could be interpreted as the updated physical reality. The introduced objects or properties that do not possess objective existence do possess existence with respect to the visual perception of the user. Suggesting that an object that is not present visually or objectively cannot constitute a physical reality can be compared to a blind person (or person who has lost all perceptive abilities including sight, hearing and touch) being unsure of an object's objective existence given they cannot visually perceive it.

Dreams: Interpolating between real dreams and virtual dreams
A realm of interest is that of dreams. Attached to our premise on reality, Hobson et al.
(2014) evaluated dreams as a form of simulated reality, termed 'innate virtual reality'. Similar to a computer-generated virtual reality, a dream has limited access to the physical reality, and objects in a dream have no objective existence. Objects in a dream may not have a physical manifestation, but similar to digital objects, they can also have real-world effects (e.g. through the changes in the user's cognition). Thus, a key difference between computer-generated virtual realities and brain-generated virtual realities is the 'rendering engine'. Also similar to our cross-reality rendering premise, rendering of dreams can also be manipulated in real-time based on prior conditioning or priming. Users can be primed on certain stimulus, and can subsequently use this stimulus to alter their dreams, such as video or music. For example, Krakow and Zadra (2010) find that users who visualize a positive end to a nightmare before initiating sleep tends to resolve nightmares during sleep.
We define a dream as an environment composed of a sequence of compositionally-generated scenes, where the generation occurs through the combination of scenes, scenes' objects, and objects' properties, which originate from a repository of objects/scenes. Scenes are a sequence of images from a user's view, that are categorized arbitrarily by the user (e.g. chronological, location-based, action, people). These viewed scenes are sourced from reality, experienced by the user, and are used as input to render dreams. From these sourced scenes, the user interpolates properties -rather than interpolating on a continuous space, users interpolate on a discrete space, i.e. enumerating through different combinations of object properties, objects, and scenes. As a heuristic as to whether one's dreams are a function of all visual perception, Meaidi et al. (2014) find that people who are born blind (or become blind early in life) do not experience visual imagery when they dream, and conversely dream with auditory, tactile, gustatory, and olfactory components. Nielsen (2017) examined microdreams to find that the content in dream generation is driven by memory, where real perceptions of recent experiences and associated memories form a cohesive image.
We highlight at least two approaches to rendering dreams: (i) cognitive, and (ii) computational. Brain cognition is a generative process to render dreams. It can occur consciously (e.g. day-dreams, hallucinations), semi-consciously (e.g. lucid dreams), or unconsciously (e.g. during REM sleep). We consider generation as an origination problem, where the original composition of objects in a scene is constructed. Computational rendering of a dream can be both a generative or transcription/translation process. Cognitive and computational approaches tend to be used in feedback loops together. While retrieval has been a common mode of returning unseen scenes using computational approaches, and this retrieval feeds into cognitive rendering (inspiration), the recent growth in generative models (e.g. DALLE-2 (Ramesh et al., 2022), Imagen (Saharia et al., 2022), Parti (Yu et al., 2022)) permit the sourcing of unseen views of reality and synthesize novel scenes dynamically. We use 'unseen' to mean that a user has not experienced, perceived, or viewed a specific instance before. While retrieval and unconditioned generation are relatively passive modes, there are other modes that support more active involvement of the end-user, resulting in human-machine co-creation. A common computational approach to rendering dreams is through the use of creativity tools. When the user has a conscious dream of a scene (e.g. conditioned on a specific task) generated by cognition first, they then transcribe this dream onto a canvas (e.g. Photoshop for images, or musical instruments for audio, etc). While generative models are guided by the crowd (crowdsourced datasets), manual creativity tools are guided by the end-user. The guidance on generation can be a mix of experiences of the end-user and outside the end-user. For example, users can provide prompts for conditional generation of outputs.
As dreams are composed of perceived reality, and only the generation process is affected by the level of consciousness, we do not distinguish unconscious dreams differently from conscious dreams in our evaluation as they are derived from the same inputs, and thus we focus on conscious dreams (and do not propose manipulation of unconscious dreams). Some have attempted the manipulation of unconscious dreams, such as Dormio (Haar Horowitz et al., 2020).
We note some observations about dreams in the context of realities. We note that a source or repository of realities is always needed to render a dream. For example, a source of graphics (Microsoft Clipart, Google Images, icon packs for slide presentations), crowdsourced datasets (e.g. for training generative models), assets and operations in VR environments, etc. Though the user has used cognition to generate a template for the dream, it appears users take the rendering shortcut of making use of alternative sources for aiding the final render, and it requires too much manual effort to have a complete render in one's mind to be transcribed onto a computer. In addition to stimulation of creativity, such repositories contribute to human-machine co-creation. Another observation is, similar to reality manipulation, the manipulation of dreams are also through augmentation (adding onto a blank slate) or diminishing (removing objects or tweaking an existing scene).
For cross-reality systems to assist dreaming, there appears to be a few directions to pursue: (i) origination (e.g. object/scene generation or retrieval); (ii) transcription (e.g. interfacing between brain and computer in rendering the dream); (iii) feedback loop between brain and computer. From the perspective of origination, the problem is a management of unseen views. We need to provide novel content that the user has not experienced in their viewed realities. With the cross-reality system in this paper, we would have a record of all prior viewed realities, so we would know what had not been viewed previously. Given the diversity of users and their respective views, we also have a source of unseen realities. Additionally, we need to consider valid origination. Based on context, we need to know when objects are semantically-valid to be inserted. For example, we cannot just add random objects on any given scene in certain settings. From the perspective of transcription, the problem is a management of seen views. The goal would be to approximate computationally what is being rendered cognitively with less effort than manual transcription. Given the user's own source of ideas come from prior viewed realities, we can enumerate through all combinations of objects and scenes in previous views till we obtain the scene the user is thinking of. This is intractable, but it demonstrates an iterative approach to automated transcription, where the user only notifies whether the output matches the cognitive render. Another method of minimal-effort interaction is through the use of brain-computer interfaces. For example, Mallett (2020) demonstrated a lucid dreamer can control a block on a screen even while asleep. Prompting is also a common mode of interaction (e.g. in conditional text-to-image generation), where the user provides low-dimensional input (e.g. text) to generate high-dimensional output (e.g. images). We also need to consider how to manage the feedback loop between cognition and computational rendering. One part of this is an information visualization problem, as we would wish to avoid information overload for the user. For example, we may wish to make use of "portals" in the regular shapes of some objects, and users can peer into these portals to view the dream. Another example is to activate dreaming based on time, place, mental state, or some other conditioning input, similar to how unconscious dreaming is activated when a user falls asleep. By making use of a specific action or inserting an affordance that is specific to dreams, the user is given the choice and optionality in pursuing a dream, rather than placing dreams everywhere, without filter or choice to not explore if a user already knows they do not like the direction of the dream, etc.

Extensions & Near-Future Work
To extend cross-reality rendering, we can consider augmentation in addition to diminishing. With the progression of new generative diffusion models (Ramesh et al., 2022;Saharia et al., 2022;Yu et al., 2022), we can explore the use of such models for interface generation in physical and digital settings using different conditioning techniques. In terms of object mappings, we currently let the user collect object instances on the view history, and manipulate the object forward in time. We did not manipulate the object in any way backward in time.
For example, given the view history of both digital and physical realities, we could use the object mapping to identify patterns of causation (e.g. seeing price tags for footwear products on Amazon and linking it to how a user acts when seeing priced footwear in physical stores).
To support the scaling of intervention generation and usage, we provide users with more data points and pre-populated interventions. Automated intervention generation is one approach. Currently intervention generation is a semi-supervised approach, where data is being annotated by users upon self-reflection of specific use cases. We could shift towards automatically generating interventions or recommending the annotation of certain objects to improve the workflow. Another approach is to enable an 'other-user' view history mode. Some tools exist that allow users to see an interface from the perspective of another user (e.g. YouTube (Kollnig, 2021)). If other users feel safe to contribute their view history, they can share it publicly so other users can view it and also annotate it from their unique point-of-view. Based on user consent, another user can be given another user's egocentric vision to simulate their life, and generate even more interventions based on their personal interpretation. This can also be pre-populated with the Ego4D dataset (Grauman et al., 2021), a diverse collection of egocentric vision videos from around the world. In addition to assisting the mapping in objects between the physical and digital realities, hooks assist in scaling the generation of interventions. Given the limited variability of a specific GUI element on a given app (or ease of re-cropping an updated interface design), one shot of a GUI element is sufficient to detect it. As such, mask hooks tend to require only a single cropped image as input, and we sample a large number of GUI element interventions based on existing digital/perceptual harms literature. Pre-populated model hooks can also be sourced from model sharing platforms (e.g. AdapterHub, huggingface, PapersWithCode, Github, ModelZoo).
Other than benefiting from existing contributions in the machine learning ecosystem, this system can also contribute back to the same machine learning ecosystem. This system can contribute to the pool of task-specific fine-tuned models, annotated/labelled datasets on various tasks, datasets of high distributional shift (attributed to the non-uniformity of user experiences), or providing unlabelled view history data for unsupervised tasks. This helps developers working on reality manipulation (e.g. AR/MR/VR researchers, digital harms researchers) by providing them with data on what users wish to mitigate (e.g. a repository of digital harms), or initial user-initiated designs on what interface changes users would like to see (and thus be implemented natively).

Limitations
The current system design is the product of numerous iterations. In each iteration we aimed to resolve different challenges that posed as hurdles to usability and deployment. We opt for a complete server-side implementation, where we load devices and run interventions on a server, and stream to client devices. This reduces the burden on end-users for client-side hardware specifications. With our setup, the user does not need a high-end smartphone or specialized AR/MR/VR hardware with built-in processors. A user can use a smartphone of any specification (as long as it has an internet connection and load webpages) to load any device emulator. They can procure a headset for mounting a smartphone, and this can be as costless as building one out of cardboard (Google, 2022). In a previous iteration, we attempted to keep the device interface on the client-side and stream just-in-time overlay renders. While this worked in most settings given sufficient bandwidth, our concern lied in the off-chance that a user with insufficient bandwidth might see an overlay render after the underlying interface image had changed. We concluded it was better to liken the access to manipulated realities to that of buffering a video; if a user prioritizes interventions, they may be willing to stream an interface (even with rare delays), and video buffer time has been drastically shortened over the years with improvements in streaming architectures and bandwidth access. We also move away from a code modification approach to changing interface functionality. By identifying commonalities between interfaces agnostic to operating system (native program patches tend to be OS-specific), we allow objects to be manipulated across operating systems. Not all interfaces have an underlying 'code' that can be modified, such as the physical reality, and thus overlays have been the predominant strategy in modifying physical realities.
To maintain a reality-agnostic approach to manipulating realities, we adhere to the use of overlays. Overlays make use of what the user can see as their input and output. Further, prior use cases for reality manipulation tend to require a third-party to craft interventions for the user, be it patch developers for app modifications, or developers for AR/MR/VR software. From a development cycle where users and developers engage in a feedback loop to maintain and upgrade software over time, we directly support users in maintaining or developing software themselves. They can craft their own interventions that perform specific digital functionality or AR/MR/VR functions. Despite this progress, there are still avenues for improvement. We highlight some extensions needed to improve the overall user experience, as well as preliminary directions on how to approach the limitations. Most of the following limitations are not critical issues with the system design; conversely, they are at most 'band-aids' that can be plastered onto the system to improve the experience, but they do not break the experience.
Some failure modes are component-specific; they are not a failure in system design necessarily, but requiring improving individual components. The mask hook might face difficulty in element removal of 'dynamic' elements (e.g. removing the video box for YouTube videos if we exclude the sharing metrics), or the overblocking of elements (e.g. removing the homescreen).
Our current implementation has handled most interaction modalities pertaining to imagery and text, but there are other modalities that would need to be manually built. Accessing client-side hardware is possible (e.g. VrtualApp (2016)), such that the server-side emulator can access the user's local camera, audio speakers, sensors, and haptic vibrators. Furthermore, while we have provided hooks for image-based interactions (e.g. text can manifest as an image in any reality), we did not implement a hook for audio/speech.
Safety is also an important concern. As a user can manipulate their physical realities, there may be some critical situations where the re-render needs to be undone, or the user should be informed of the non-overlayed reality. For example, when crossing the road, though bicycles are occluded, they should not be completely inpainted. They should be slightly blurred, or at least a big arrow should above the cyclist to inform the user that an object exists and is approaching them. This also means certain objects that are intended to be used for physical safety, such as fire extinguishers or traffic lights, should not be overlay-able. We could insert safety checker models to verify that non-overlayable objects are not manipulated, or alternatively we could prompt the user to re-consider their decision (e.g. doing a sample playback in the view history of what happens when this object is occluded).
There are a few considerations regarding scaling, in terms of model development and data quality. An assumption made is that users in a network know a ground-truth label of the category of the specific text they wish to detect and occlude, and the crowd-sourced text of each of N categories will yield corresponding N fine-tuned models. A concern with this assumption, is that the labelling of such inputs in the real-world may not be standardized, and similar inputs may be grouped separately or dissimilar inputs may be grouped together, if we purely rely on network-based tagging. We may encounter scenarios of out-of-distribution shift (e.g. users sample non-uniform sentences), adversarial samples (e.g. users maliciously tag sentences that worsen accuracy). On the one hand, we can evaluate a tagging system that shows the user similar intervention tags as the one they are entering, so that an existing intervention is updated rather than creating duplicate interventions.
On the other hand, perhaps the data points are indeed distributionally different from an existing tag's dataset, so creating a different tag would be appropriate. Possible algorithmic approaches to ensuring similar texts are grouped together for fine-tuning could be the use of in/out-of-distribution detection (e.g. computing the interference in loss convergence with respect to 2 inputs coming from different categories, or using a similarity metric, in order to regroup contributed inputs into appropriate categories), or the use of ensemble models (e.g. preparing M different batches of training sets to train M different ensemble models, so that the dissimilarity between certain sentences do not afflict a single model alone, and other models can validate a prediction). Furthermore, to reduce the reliance on the user population and sampling rate for intervention generation, we can explore faster adaptation methods (e.g. batch-efficient fine/prompt-tuning with large foundation models) so that less/no additional data would be needed for generating interventions. Alternatively, we can pre-populate the system internally first with a large number of masks and models (e.g. sourcing models from AdapterHub, github, modelzoo, etc).

Conclusion
Our cross-reality re-rendering system supports users in manipulating their digital and physical realities. They can inspect their historical views, annotate objects, and share the interventions they generate. We evaluate requirements with cognitive walkthroughs, scalability tests, and personas. We hope this work continues to inspire further exploration of cross-reality systems into other realities.


Clinical-Longformer and Clinical-BigBird: Transformers for long clinical sequences

Transformers-based models, such as BERT, have dramatically improved the performance for various natural language processing tasks. The clinical knowledge enriched model, namely ClinicalBERT, also achieved state-of-the-art results when performed on clinical named entity recognition and natural language inference tasks. One of the core limitations of these transformers is the substantial memory consumption due to their full self-attention mechanism. To overcome this, long sequence transformer models, e.g. Longformer and BigBird, were proposed with the idea of sparse attention mechanism to reduce the memory usage from quadratic to the sequence length to a linear scale. These models extended the maximum input sequence length from 512 to 4096, which enhanced the ability of modeling long-term dependency and consequently achieved optimal results in a variety of tasks. Inspired by the success of these long sequence transformer models, we introduce two domain enriched language models, namely Clinical-Longformer and Clinical-BigBird, which are pre-trained from large-scale clinical corpora. We evaluate both pre-trained models using 10 baseline tasks including named entity recognition, question answering, and document classification tasks. The results demonstrate that Clinical-Longformer and Clinical-BigBird consistently and significantly outperform ClinicalBERT as well as other short-sequence transformers in all downstream tasks. We have made our source code available at [https://github.com/luoyuanlab/Clinical-Longformer] the pre-trained models available for public download at: [https://huggingface.co/yikuan8/Clinical-Longformer].

Introduction
Transformer-based models have been wildly successful in setting state-of-the-art results to a broad range of natural language processing (NLP) tasks, including question answering, document classification, machine translation, text summarization and others [1,2,3]. The successes are also replicated in the clinical and biomedical domain via pre-training language models using clinical or biomedical corpora and then transfer them to a variety of clinical or biomedical downstream tasks, e.g. computational phenotyping [4,5], automatically ICD coding [6], knowledge graph completion [7] and clinical question answering [8].
The self-attention mechanism [9] is one of the most important components that lead to the successes of transformerbased models, which allows each token in the input sequence to independently interact with every other token in the sequence in parallel. However, the memory consumption of self-attention mechanism quadratically enlarges with sequence length, resulting in impracticable training time and easily reaching the memory limits of GPUs. Consequently, transformer-based models that leverage a full self-attention mechanism, such as BERT and RoBERTa, always have an input sequence length limit of 512 tokens. To deal with this limit when modeling long texts using transformer-based models, the input sequence shall be either truncated to the first 512 tokens, or segmented to multiple small chunks of 512 tokens. If the latter method is applied to a text classification task, an aggregation operation will be added to yield the final output. Both segmentation and truncation of the input sequence will ignore the long-term dependencies spanning over 512 tokens and consequently achieve suboptimal results due to information loss. This drawback of self-attention operation will impact language model pre-training and then be amplified to downstream tasks. In clinical NLP, the models that are applied with the transformer-based approaches also encounter this limitation [10]. For example, the discharge summaries in MIMIC-III, which are always used to predict hospital re-admission [11] or mortality [12], have 1,435 words in average, far exceeding the 512 tokens limits of BERT like models. Recently, researchers have been developing some novel variants of transformers particularly for long sequences that reduce the memory usage from quadratic to linear to the sequence length [13,14,15]. The core idea behind these models is to replace the full attention mechanism with the sparse attention mechanism, which is a blend of sliding windows and reduced global attention. These models are able to be fed in up to 4.096 tokens and boost the performance of several question answering as well as text summarization tasks. However, to the best of our knowledge, these transformers for long sequences are rarely discussed in the clinical and biomedical domain. Therefore, we would like to examine the adaptability of these long sequence models using a series of clinical NLP tasks. In particular, we make the following contributions: 1. We pre-train and publicly release two transformer models for long clinical sequences, namely Clinical-Longformer and Clinical-BigBird, using large-scale clinical notes. 2. We demonstrate that both Clinical-Longformer and Clinical-BigBird significantly improve the performance of a variety of downstream clinical NLP tasks including question answering, named entity recognition and document classification.
2 Related Work

Clinical and Biomedical Transformers
Transformer-based models, especially BERT [1], can be enriched with clinical and biomedical knowledge through pre-training on large-scale clinical and biomedical corpora. These domain-enriched models, e.g. BioBERT [16] pretrained on biomedical publications and ClinicalBERT [17] pre-trained on clinical narratives, set the state-of-the-arts when down-stream applied to clinical and biomedical NLP tasks. Inspired by the success of these domain-enriched models, more pre-trained models were released to boost the performance of NLP models when applied to specific clinical scenarios. For example, Smit et al. [18] proposed CheXbert to annotate thoracic disease findings from radiology reports, which outperformed previous rule-based labelers with statistical significance. The model was pre-trained using the combination of human-annotated and machine-annotated radiology reports. He et al. [19] introduced Dis-easeBERT, which infused disease knowledge to the BERT model by pre-training on a series of disease description passages that were constructed from Wikipedia and MeSH terms. DiseaseBERT received superior results on consumer health question answering tasks. Zhou et al. [20] developed CancerBERT to extract breast cancer related concepts from clinical notes and pathology reports, which would facilitate the information extraction process. However, all of the aforementioned models were built on the basic BERT architecture, which has a limitation of 512 tokens in the input sequence length. This limitation may result in the information loss of long-term dependencies in the training processes.

Transformers for long sequences
Various attention mechanisms have been proposed to handle the large memory consumption of the attention operations in vanilla Transformer architecture. Transformer-XL [21] segmented a long sequence into multiple small chunks and then learned long-term dependency with a left-to-right segment-level recurrence mechanism. Transformer-XL learns 5.5× dependency compared to the vanilla transformer models, but loses the advantage of bidirectional representation of BERT like models. In another study, Reformer [22] applied two techniques to reduce the space complexity of transformer architecture by replacing do-product attention operation with locality-sensitive hashing and sharing the activation function among layers. Reformer was able to process longer sequences at a faster speed and be more memoryefficient. However, no performance improvement can be observed when using Reformer. Almost simultaneously, Longformer [13] and BigBird [14] were proposed to drastically alleviate the memory consumption of transformer models from quadratic growth to linear growth to the sequence length, by replacing the pairwise full attention mechanisms with a combination of sliding window attention and global attention mechanisms. They are slightly different in terms of implementation and configuration of the global attention mechanism. Both models support input sequences up to 4.096 tokens long (8× of BERT) and significantly improve performance on long-text question answering and summarization tasks. However, the adaptability of these long sequence transformers to the clinical and biomedical fields, where the text length always exceeds the limits of BERT like models, has been rarely investigated. Levine et al. [23] pre-trained longformer models using clinical notes and then fine-tuned the language model on document classification tasks include ICD coding. Our study applied similar approach, while aiming at a more comprehensive comparison to the short sequence models using heterogeneous NLP tasks.

Material and Methods
In this section, we first introduce the clinical data set we use as the pre-training corpus, followed by the pre-training process of Clinical-Longformer and BigBird. Then, we enumerate the down-stream tasks we used to fine-tune our pre-trained model. We also provide the technical details of pre-training and fine-tuning for reproducing purposes.

Datasets
Similarly to [17,11], we leveraged approximately 2 million clinical notes extracted from the MIMIC-III [24] dataset, which is the largest publicly available EHR dataset that contains clinical narratives of over 40,000 patients admitted to the intensive care units. We only applied minimal pre-processing steps, including 1) to remove all de-identification placeholders that were generated to protect the PHI (protected health information); 2) to replace all characters other than alphanumericals and punctuation marks; 3) to convert all alphabetical characters to lower cases; and 4) to strip extra white spaces.

Pre-training
Longformer [13] and BigBird [14] are the two most remarkable transformer models that are designed for long input sequences. Both models extend the maximum input sequence length to 4,096 tokens, which is 8× of the conventional transformer-based models by introducing the localized sliding window and global attention mechanisms to reduce the computational expenses of full self-attention mechanisms. The differences between two models are how the global attention is realized and the selection of loss function in fine-tuning. BigBird also contains some random localized attention operations. The performance difference between two models on downstream tasks is minimal. Therefore, we would like to pre-train both models and compare their performance on clinical NLP tasks. We refer readers to the original papers of Longformer [13] and BigBird [14] for more technical details.
Byte-level BPE (Byte-Pair-Encoding) [25] was applied to tokenize the clinical corpus. We initialized Clinical-Longformer and Clinical-BigBird from the pre-trained weights of the base version of Longformer and the ITC version of BigBird, respectively (although the ETC version performs better, HuggingFace, the largest shared transformers community only provides the implementation and the pre-trained weights of ITC version). Both models were distributed in parallel to 6 32GB Tesla V100 GPUs. FP16 precision was enabled to accelerate training. We pre-trained Clinical-Longformer for 200,000 steps with batch size of 6×3, and Clinical-BigBird for 300,000 steps with batch size of 6×2. The learning rates were 3e-5 for both models. The entire pre-training process took more than 2 weeks for each model.

Tasks
In this study, we adapted the pre-trained Clinical-Longformer and Clinical-BigBird to 9 clinical NLP datasets. These 9 NLP datasets broadly cover various NLP tasks including extractive question answering, named entity recognition, natural language inference, and document classification. We relied on these NLP tasks to validate the performance improvement of our proposed models. The statistics and descriptions of all dataset can be found in Table 1.

Question Answering
Question answering (QA) is an NLP discipline, which targets to automatically answer questions raised from human languages [26]. In the clinical context, QA system answers physicians' questions by understanding the clinical narratives extracted from electronic health record systems to support decision making. emrQA [27] is the most frequently used benchmark dataset in clinical QA, which contains more than 400,000 question-answer pairs semi-automatically generated from past i2b2 challenges. emrQA falls into the category of extractive question answering that aims at identifying the answer spans from reference contexts instead of generating answers in a word-by-word fashion. Researchers have attempted to solve emrQA tasks by using word embedding models [28], conditional random fields (CRFs) [29] and transformer-based models [30], among which transformer-based models defeated their competitors in terms of performance. In our experiments, we investigated the performance of our pre-trained models using the three largest emrQA subsets: Medication, Relation and Heart Disease. We evaluated QA performance with two commonly used metrics: exact match (EM) and F1-score. Exact match represents that the predicted spans and categories exactly and strictly match with the ground-truth annotations. F1-score is a looser metric derived from token-level precision and recall, which aims at measuring the overlap between the predictions and the targets. We generated train-dev-test splits by following the instruction of [28], where the training set of relation and medication subsets were randomly under-sampled to reduce training time. Based on their experience, the performance would not be compromised after under-sampling.

Named Entity Recognition
Named-entity recognition is a token-level classification task that seeks to identify the named entities and classify them to predefined categories. This genre of NLP tasks has broad applications in the clinical and biomedical domain. e.g. deidentification of PHI, and medical concept extraction from clinical notes. Prior studies have shown that Transformerbased models [16] significantly outperformed the models built on pre-trained static word embedding [31] or LSTM networks [32]. We examined our pre-trained models using four data challenges of: 1) i2b2 2006 [33], to de-identify PHI from medical discharge notes; 2) i2b2 2010 [34], to extract and annotate medical concepts from patient reports; 3) i2b2 2012 [35], to identify both clinical concepts and events relevant to the patient's clinical timeline from discharge summaries; and 4) i2b2 2014 [36], to identify PHI information from longitudinal clinical narratives. We followed the processing steps of [17], which converted the raw data from all four tasks to IOB (inside-outside-beginning) tagging format proposed by Ramshaw et al, and then created train-dev-test splits. [37]. Based on the new tagging format, we evaluated the model performance with F1-score similarly to QA tasks.

Document Classification
Document classification is one of the most common NLP tasks, where we assign a sentence or document to one or more classes or categories. In the clinical domain, researchers pay more attention to identifying the diagnosis, or to predicting the onset of a certain disease from clinical notes. We applied the following four document classification data set to evaluate the pre-trained models from different perspectives.
MIMIC-AKI [38,39] MIMIC-AKI is a binary classification task, where we would like to predict the possibility of AKI (acute kidney injury) for critically ill patients using the clinical notes within the first 24 hours following intensive care unit (ICU) admission. We evaluated the model performance using AUC and F1 score.
OpenI [40] OpenI is a publicly available chest x-ray (CXR) dataset collected by Indiana University. The dataset provides around 4,000 radiology reports and their associated human annotated MeSH terms. Given the small sample size, we only used openI as the testing set. The pre-trained language models were fine-tuned using MIMIC-CXR [41], another publicly available chest x-ray dataset. Unlike openI, the ground-truth labels for MIMIC-CXR were automatically generated using NLP approaches. The annotated findings overlap between two CXR data sources are the diagnosis of: Cardiomegaly, Edema, Consolidation, Pneumonia, Atelectasis, Pneumothorax and Pleural Effusion.
In experiments, we would like to retrieve the presences of all possible thoracic findings from CXR reports. This task can be considered as a multi-label classification task. We would report the sample number weighted average of AUC as proposed in [42].
MedNLI [43] Natural language inference (NLI) is the task of determining the relationship between sentence pairs. MedNLI is a collection of sentence pairs extracted from MIMIC-III [24] and annotated by two board-certified radiologists. The relationship between the premise sentence and the hypothesis sentence could be entailment, contradiction or neutral. Transformer-based models process NLI tasks also as document classification by merging the sentence pair together and placing a delimiter token right after the end of the first sentence. We followed the original splits as [43] and used accuracy to evaluate the performance.
OHSUMed [44] OHSUMed is a collection of journal titles and/or abstracts extracted from MEDLINE, which is maintained and published by the National Library of Medicine. Unlike other data sources aforementioned, this dataset is the only one that the free-text raw data is not obtained from clinical notes but academic journals. Following [45], only those cardiovascular diseases abstracts that are associated with one and only one of the 23 disease categories are included in this study. We also used accuracy to evaluate this multi-class classification task.

Baseline Models
Both Clinical-Longformer and Clinical-BigBird were compared with the pioneering BERT, ClinicalBERT and BioBERT models. BERT [1] model is the first-of-its-kind transformer architecture that achieved state-of-the-art results in eleven NLP tasks. Both masked language modeling and next sentence prediction were used to learn contextualized word representation from BooksCorpus and English Wikipedia in the pre-training stage. BioBERT [16] is the first biomedical domain specific BERT variant that was pre-trained from PubMed abstracts and PMC full-text articles. The weights of BioBERT were initialized from BERT. BioBERT yielded optimal performance in biomedical QA, NER and relation extraction tasks. ClinicalBERT [17], initialized from BioBERT, was further pre-trained using clinical notes that were also extracted from MIMIC-III. ClinicalBERT boosted the performance for MedNLI and four i2b2 NER tasks that were also included in our study. Both BioBERT and ClinicalBERT discarded the next sentence prediction task and only selected the masked language modeling as the pre-training strategy.

Experimental setup
For the token-level classification, including QA and NER, a classification head was added to the output of each token obtained from the transformer-based architecture. The maximum sequence length was selected to be 3,072 for Clinical-Longformer and Clinical-BigBird, and 384 for all the other three baseline models. For the sequence-level classification, the predicted outcomes were yielded from the [CLS] token added to the beginning of each sentence or document. Given that some clinical notes are extremely long, which may even exceed the length limits of all models, we first truncated each document/sentence/sentence-pair to 4096 tokens, which met the length limits of Clinical-Longformer and Clinical-BigBird. The document/sentence/sentence-pair was further segmented to snippets of 512 tokens in order to accommodate for the length requirement of BERT/BioBERT/ClinicalBERT. A pooling layer was applied to aggregate the output from short snippets. We conducted our experiments using four 32GB gpus. The batch size during training was 4×4 for Clinical-Longformer, 4×3 for Clinical-BigBird, and 4×16 for all other models. Batch sizes were doubled when evaluating or testing. Half precision was applied to both Clinical-Longformer and Clinical-BigBird. The only hyper-parameter we tuned was learning rate. We tried 1e-5, 2e-5 and 5e-5 for the experiments of each model on each task. We fine-tuned 6 epochs for each set-up. The best-performing model parameters were determined by the performance on the developing split. The experiments were implemented with python 3.8.0, pytorch 1.9.0 and Transformer 4.9.0.

Results and Discussion
Full results for QA, NER and classification tasks are presented in Table 2, 3, 4, respectively. In question answering tasks, both Clinical-Longformer and Clinical-BigBird outperformed the pioneering short-sequence transformer models 2 percents across all three emrQA subsets when evaluated by F1 score. When considering the stricter EM metric, Clinical-Longformer and Clinical-BigBird improved 5 percent on the relations subset, but yielded similar results to ClinicalBERT in the other two subsets. In NER tasks, Clinical-Longformer consistently led the short-text transformers by more than 2 percents in all 4 i2b2 dataset. Clinical-BigBird also performed better than ClinicalBERT and BioBERT in every single NER experiments. In document classification tasks, our two long-sequence transformers achieved better results compared to prior models on OpenI, MIMIC-AKI and medNLI, which were all generated from clinical data sources. BioBERT performed slightly better than our models in OHSUMed dataset. We argue that OHSUMed is the only task that falls under the scope of biomedical data mining, making it align better with BioBERT which was pre-trained from biomedical publications.
We observed that Clinical-Longformer and Clinical-BigBird not only improved the performance of long sequences tasks, but also short sequences. The maximum sequences of MedNLI and OpenI are smaller than 512 tokens, which can be entirely fed into the BERT like models. However, the long sequences models still achieved better results. We attributed these improvements to the pre-training stages of Clinical-Longformer and Clinical-BigBird, where the language models can learn more long-term dependencies by extending the limit of sequence length. In addition, we found that more significant improvement can be achieved when applying our two models to the dataset with longer sequences. For example, the performance improvement on i2b2 2014, which has the largest averaged sequence  length in all 4 NER tasks, is almost twice of the other three subsets. Likewise, Clinical-Longformer more strongly improved the F1 score of the heart disease subset from emrQA. This helps us confirm that Clinical-Longformer and Clinical-BigBird are also able to model long-term dependency in the downstream tasks. We also noticed that Clinical-Longformer consistently yielded better results when compared to Clinical-BigBird. Given that Clinical-BigBird also requires more fine-tuning time, we recommend readers to apply our Clinical-Longformer checkpoint to their own tasks when they have limited resources.
Our study has several limitations, each of which can lead to further investigations. First, we only applied Longformer and BigBird to large scale clinical corpora. In the next step, we plan to release more pre-trained models for long sequences enriched with other biomedical corpora, e.g. PubMed and PMC publications. Second, another approach developed to address the memory problem of long sequences is to simplify or compress the transformer architecture. In our future work, we will compare this genre of transformers, e.g. TinyBERT to our current long sequence models. Last, we did not integrate Clinical-Longformer or Clinical-BigBird into an encoder-decoder framework due to the memory limits of our GPU cards. Therefore, the experiment on text generation or summarization is not included in this study. We will definitely incorporate those tasks into our future study, once we acquire more computational resources.

Conclusion
In this study, we introduced Clinical-Longformer and Clinical-BigBird, two pre-trained language models that are suitable for long sequence clinical NLP tasks. We compared these two models with the pioneering short sequence transformer based models, e.g. ClinicalBERT, in named entity recognition, question answering, and document classification tasks. Results demonstrated that Clinical-Longformer and Clinical-BigBird achieved better results in both long sequence and short sequence benchmark dataset. Future studies will be investigated on other models, for example TinyBERT, that are developed to solve the memory consumption of long sequences as well as the generalizability of our proposed models to clinical text generation and summarization.