Interactive Portrait Harmonization

Current image harmonization methods consider the entire background as the guidance for harmonization. However, this may limit the capability for user to choose any specific object/person in the background to guide the harmonization. To enable flexible interaction between user and harmonization, we introduce interactive harmonization, a new setting where the harmonization is performed with respect to a selected \emph{region} in the reference image instead of the entire background. A new flexible framework that allows users to pick certain regions of the background image and use it to guide the harmonization is proposed. Inspired by professional portrait harmonization users, we also introduce a new luminance matching loss to optimally match the color/luminance conditions between the composite foreground and select reference region. This framework provides more control to the image harmonization pipeline achieving visually pleasing portrait edits. Furthermore, we also introduce a new dataset carefully curated for validating portrait harmonization. Extensive experiments on both synthetic and real-world datasets show that the proposed approach is efficient and robust compared to previous harmonization baselines, especially for portraits. Project Webpage at \href{https://jeya-maria-jose.github.io/IPH-web/}{https://jeya-maria-jose.github.io/IPH-web/}

Introduction
With the increasing demand of virtual social gathering and conferencing in our lives, image harmonization techniques become essential components to make the virtual experience more engaging and pleasing. For example, if you cannot join a wedding or birthday party physically but still want to be in the photo, the first option would be to edit yourself into the image. Directly compositing yourself into the photo would not look realistic without matching the color/luminance conditions. One possible solution to make the composition image more realistic is to leverage existing image harmonization methods [4,3,5,15,10,7,25].
Most previous works focus on a more general image harmonization setup, where the goal is to match a foreground object to a new background scene without too much focus on highly retouched portrait. However, when we conduct arXiv:2203.08216v1 [cs.CV] 15 Mar 2022 surveys among professional composition Photoshop/Affinity 3 users, we realized that portrait harmonization is the most common task of image editing in realworld scenario and professional settings. This makes portrait harmonization the most important use case of image harmonization. We note that previous harmonization works have not focused on addressing portrait harmonization on real-world data. In this work, we aim to explore a better solution to obtain realistic and visually pleasing portrait harmonization for real-world high-resolution edited images. It can be observed that the current SOTA harmonization method [15] fails to give realistic results as it tries to match the appearance of foreground to the entire background. Our proposed interactive harmonization framework produces a visually pleasing portrait harmonization as it matches the appearance of the original portrait in the reference region instead of the entire background as selected by the user.
One common question that pops up when we demo existing image harmonization workflow to these professional users is: 'How could we choose a certain person as reference when we do harmonization with existing workflow ? '. The workflow design of existing state-of-the-art harmonization methods [4,3,15,7] limits the capability for user to choose any person/region as reference during the harmonization process. These frameworks are designed such that they just take in the composite image and foreground mask as input thus offering no specific way to help the user to guide the harmonization. Certain frameworks such as Bargain-Net [3] have the flexibility to be tweaked and converted to serve interactive harmonization. However, from our experiments we find that they are not robust enough to perform realistic portrait harmonization.
Furthermore, professional portraits are mostly shot at studios where screens constitute the background. These screens offer little to no information in harmonizing a new composite when edited into the photo. This causes current harmonization methods to produce unstable results (see first row of Figure 1) as they have not been trained to perform harmonization with screens as back-ground. Also, portraits which are captured by everyday users usually contain a background of spatially varying luminance characteristics. Using the entire background to guide harmonization here can result in undesirable outcomes (see second row of Figure 1) as harmonization depends on the location where the foreground is composited to.
To this end, we introduce Interactive harmonization-a more realistic and flexible setup where the user can guide the harmonization. The user can choose specific regions in the background as a reference to harmonize the composite foreground. From a professional stand-point, this is a much-needed feature as it could enable a lot of control for editors. In this work, we propose a new interactive portrait harmonization framework that has the flexibility to take in a reference region provided by the user. We use an encoder-decoder based harmonization network which takes in the foreground composite region as input and a style encoder which takes in the reference region as its input. The reference region to guide the harmonization is selected by the user and can be a person, and object or even just some part of the background. However, it can be noted that in portrait editing it is very common to choose another person in the picture as reference to obtain an effective harmonization. We also use a style encoder that extracts the style code of the reference region and injects it to the harmonized foreground. We carefully align the style information with foreground while also preserving the spatial content using adaptive instance normalization [9] layers at decoder. To make the harmonization look more realistic it is important to optimally match the luminance, color, and other appearance information between the reference and foreground. To match these characteristics in manual photo editing, professional photography users usually match statistics of the highest (highlight), lowest (shadow) and average (mid-tone) of luminance points between the composite foreground image and the reference region. Hence, we propose a new luminance matching loss that is inspired by professional users 4 . In the proposed loss, we match the highlight, mid-tone and shadow points between the reference region and foreground composite region.
Current publicly available datasets are developed only for background harmonization problem. So, we curate a new IntHarmony dataset to enable training for interactive harmonization utilizing augmentations that would work well for general as well as portrait harmonization. We also introduce a new real-world testing data PortraitTest with ground-truth annotations collected from professional experts to benchmark the methods. Codes, datasets and pretrained models will be made public after the review process.
In summary, the following are the key contributions of this work: -We are the first to introduce Interactive Harmonization and propose a new framework which has the flexibility to solve it. -We propose a new luminance matching loss to better match the foreground appearance to the reference region.
-We curate a new synthetic training dataset as well as a professionally annotated benchmark test for portrait harmonization and demonstrate state-ofthe-art results on it. -We show that performing interactive harmonization using our proposed method results in visually pleasing portrait edits.

Related Works
Although interactive harmonization is a new problem setup, there have been many works proposed for generic image harmonization (background harmonization). Classical methods for image harmonization perform color and tone matching by matching global statistics [22,15]. There have also been methods which try to apply gradient domain methods to transfer the color and tone [21,24]. Zhu at al. [30] proposed the first convolutional network-based method to improve realism of composite images. The first end-to-end deep learning-based image harmonization method was proposed by Tsai et al. [25] where an encoder-decoder based convolutional network was used. Following that Cun et al. [5] proposed an attention-based module to improve harmonization. Dove-Net, proposed by Cong et al. [4] used a Generative Adversarial Network (GAN)-based method with an additional domain verification discriminator to verify if the foreground and background of a given image belongs to the same image. Cong et al. [4] also proposed a public dataset, iHarmony4 for image harmonization. Following that, Cong et al. [3] proposed Bargain-Net which uses a domain extractor to extract information about the background to make the harmonized foreground more consistent with the background. Stfiiuk et al. [23] proposed a new architecture involving pre-trained classification models to improve harmonization. Attention-based feature modulation for harmonization was proposed in [8]. Ling et al. [15] proposed RainNet which introduces a region-aware adaptive instance normalization (RAIN) module to make the style between the background and the composite foreground consistent. Guo et al. [7] proposed intrinsic image harmonization where reflectance and illumination are disentangled and harmonized separately. It can be noted that photo-realistic style transfer methods [13,17,26] when adopted for harmonization do not perform well as they require similar layout as explained in [10]. Jiang et al. [10] proposed a self-supervised framework to solve image harmonization. Recently, Guo et al. [6] proposed an image harmonization framework using transformers. Portrait relighting methods [29,19] have also been explored to produce realistic composite images for a desired scene. Different from these approaches, our work introduces interactive portrait harmonization, a more flexible and useful image harmonization pipeline in the practical setting where the user can choose the regions to guide the harmonization.

Interactive Harmonization
Setting: Given an input composite image C, the goal is to generate a harmonized image H that looks realistic. Composite image corresponds to the edited image where a new person is introduced into the scene. This newly introduced region is termed as composite foreground region F and the scene it was introduced into is termed as background B. In general harmonization, we use the background B to harmonize F . In interactive harmonization, we propose using a specific region R of the background B to guide harmonizing F . Note that the region R is a subset of B i.e (R ∈ B). R can be any region pertaining to the background that the user wants F to be consistent with.
For portrait images, an easy way to perform realistic harmonization would be to select the person/object in the reference portrait as R to guide the edited person/object F . This will make sure that that the luminance conditions of the reference portrait image is consistent with that of the newly edited-in portrait. However, we do not hardly constrain the region to be only a portrait in the background, it can also be objects or even a part of the scene. Please note that portraits here do not only correspond to images containing people, it can also contain objects. A photo becomes a portrait when certain subjects are the main focus of the picture.
Dataset Curation: Publicly available datasets like iHarmony4 [4] were proposed for background harmonization and do not provide any reference region information. So, we curate a syntheric dataset for this setting and also introduce a real-world portrait harmonization dataset for validating.

1) IntHarmony:
This newly curated synthetic dataset specifies an additional reference region to guide the harmonization. IntHarmony has the following information for each data instance: composite image, ground truth, foreground mask of the composite foreground and a guide mask which provides information about the reference region to guide the harmonization. IntHarmony is built on top of MS-COCO dataset [14]. We make use of the instance masks provided in MS-COCO to simulate foreground and reference regions. First, we select a random instance mask to pick the foreground region. The selected foreground region is then augmented using a wide set of meaningful augmentations focusing on luminance, contrast and color. We use all the augmentations proposed in [10] to create the composite. More details on this can be found in the supplementary document. This augmented foreground is composited with the original image to get the composite image (I). Another random instance mask is used to get the reference guide mask. The original image is considered as the ground truth. The number of training images in IntHarmony is 118287 and 959 images are allocated for testing. Note that the instance masks and the augmentations are chosen at random to induce more generalizabilty to the network. IntHarmony consists of both objects, backgrounds and people and is not very focused for portraits. We use this dataset to help the network learn the interactive way to perform harmonization as we have a reference region to access. We also use a similar dataset creation scheme on Multi-Human People Parsing (MHP) dataset [28,12,18] to synthesize another data further finetune the models for portrait harmonization. Note that the MHP dataset contains only images with people.
2) PortraitTest: To validate our performance for real-time applications, we introduce a real-world test data consisting of high resolution composite portrait images. The ground truth is annotated with the help of multiple professionals using PhotoShop curve adjustment layers to change brightness, hue, saturation layer and contrast of the composite foreground to make it look more natural and pleasing. The number of images in this test dataset is 50 and the composite foreground is of varied luminance conditions, contrast and color.

Method
In this section, we first explain the details of the framework we propose to solve interactive harmonization. We then give the training details and the loss functions we introduce for this task.

Network Details
An overview of the proposed framework is shown in Figure 2. It consists of two sub-networks: a harmonizer network and a style encoder network.

Reference Region
Harmonized Output

Harmonized Foreground
Composite with reference image

Composite Input
User selects Reference region Fig. 2. Overview of the proposed interactive portrait harmonization framework. To harmonize an input composite image, we feed forward the masked out foreground composite image to the harmonizer network. A style code is extracted using the style encoder from the reference region chosen by user. This is passed to the adaptive instance norm layers in the decoder block of harmonizer network as well as resized and concatenated with the input. We take the harmonized foreground output from the harmonizer network and composite it with the background to get the final harmonized image.
Harmonization Network is an encoder-decoder architecture which takes in two inputs: composite foreground image (I), a foreground mask and a style code (φ) extracted from the style encoder which is of dimension 1 × D. The encoder is built using 4 convolutional blocks where each conv block has a conv layer that increases the number of channels and performs downsampling, followed by a set of residual blocks. The latent features from the encoder have a spatial dimension of (H/16) × (W/16). The decoder is also built using 4 conv blocks where each block has an upsampling layer and a set of residual blocks similar to the encoder. In addition, we use adaptive instance norm (AdaIN) layers in each conv block that takes in the style code (φ) and the features from previous conv block in decoder as their input. Thus, the decoder uses the style code extracted from the reference object to guide the harmonization. The output from decoder is the harmonized foreground image. It is then composited with the original reference image to get the complete harmonized output.
Style Encoder takes in the reference region chosen by the user as the input. The style encoder consists of a series of partial convolutional layers [16] to downsample the input image, extract meaningful features and output a style code. The partial convolution layers take in both the reference image and guide mask as their output. The latent embedding is fed through a average pooling layer to obtain the 1D style code (φ).

Training Details
Training the network for interactive harmonization is performed stage-wise. In the first stage, we train the network to perform background harmonization, where the training data of iHarmony4 [4] is used. The input to the harmonizer network is the masked out foreground image concatenated with the style code of the background extracted from the style encoder. In the next stage, we finetune our network for interactive harmonization on IntHarmony. Here, the input to the style encoder is the reference region. Thus, the network here is trained in such a way that a reference region guides the harmonization which is also due to the loss functions we introduce (see Section 4.3). Finally, we further fine-tune the network on the augmented MHP dataset for portrait harmonization to be used in professional settings. The input to the harmonizer network is the masked out foreground image (rather than entire composite image) concatenated with the style code of the background extracted from the style encoder. From our experiments, we realized that using the entire composite as input to the harmonizer network does not result in optimal training. This happens because the style code information bleeds out to the background of the composite, reducing its influence to the foreground. Masking out the background makes sure that the style code affects only the composite foreground and not the entire image.

Loss Functions
Luminance Matching loss: The main objective of interactive portrait harmonization is to make sure that the harmonized foreground matches the appearance of the reference region. To achieve this, we introduce three new losses -highlight, mid-tone and shadow losses, which are used to match the highlight, mid-tone and shadow between the reference region and foreground region to be harmonized. We define these losses as follows: where H corresponds to the harmonized image andÎ corresponds to the ground truth. These losses try to match the intensities of the foreground with the reference region at the highest, lowest and average luminescence points thus matching the contrast. For the highest and lowest points, we choose the 90 th and 10 th percentile respectively to improve stability. We define luminance matching loss (L LM ) as the sum of these 3 losses: Luminance matching loss (L LM ) is illustrated in Figure 3. Note that L LM is an main addition to our training methodology which helps to make sure that the statistics between the reference and foreground are matched. This loss penalizes the network if the statistics of the harmonized foreground and the reference region are not matched.  Fig. 3. Overview of the proposed Luminance Matching loss. We match the highlight, mid-tone and shadow points between the foreground and reference region to efficiently harmonize the foreground with the reference region.
Consistency loss: We also introduce a consistency loss L consis between style codes of the harmonized foreground and the reference region to penalize the network whenever the style code of the reference region and harmonized foreground region are different from each other. The consistency loss is defined as: Harmonization loss: We also use a generic harmonization loss which is an L1-loss between the prediction and the ground-truth as follows: This loss tries to minimize the distance between the prediction and input thus making sure the prediction is always consistent in appearance with that of the ground truth.
Triplet losses: We introduce two triplet losses similar to [3] based on the style codes of the composite foreground (c), harmonized foreground (h), real foreground (r) (ground truth) and that of the reference region (b). The first triplet loss is as follows: . This loss tries to make the style code generated by the style encoder for harmonized foreground (h) to be closer to that of style code of reference region (b) while being far away from the style code of composite foreground (c). m here corresponds to the margin.
We define another triplet loss which forces the real foreground style code (r) to be close to harmonized foreground style code (h) while making sure the style code of harmonized foreground style code (h) is far away from composite foreground (c): , φ(c) 2 +m, 0), The total loss used to train the network is as follows: L IP H = L harmonization + αL LM + λL consis + β(L triplet1 + L triplet2 ) where α, β and λ are the factors that control the contribution the triplet losses have over the total loss.

Implementation Details
Our framework is developed in Pytorch and the training is done using NVIDIA RTX 8000 GPUs. The training is done stage-wise as explained in the previous section. We use an Adam optimizer [11] with a learning rate of 10 −4 , 10 −5 , 10 −6 at each stage respectively. The batch size is set equal to 48. The images are resized to 256 × 256 while training. During inference, to get the high resolution images back, we use a polynomial fitting like in [2,1]. We composite the high resolution harmonized foreground with the reference image to get the high resolution harmonized image.

Comparison with State-of-the-art
We compare our proposed framework with recent harmonization networks like Dove-Net [4], Bargain-Net [3], and Rain-Net [15]. We note that Bargain-Net has a domain extractor branch that directly takes in the background of the image to make the harmonization consistent with the background. In addition to comparing with the original model, we also re-train Bargain-Net in a interactive fashion similar to our proposed approach for fair comparison. Here, we feed in the reference region as the input to the domain extractor and use similar protocols as our method. We call this configuration Bargain-Net (R). Note that Bargain-Net (R) is trained with the same data and stage-wise training as our method for fair comparison. We explain why the other frameworks cannot be converted for interactive harmonization in the supplementary material. We use publicly available weights of Dove-Net, Bargain-Net and Rain-Net to get their predictions on Por-traitTest and IntHarmony. We denote our approach as IPH (Interactive Portrait Harmonization). While testing with IntHarmony, Bargain-Net (R) and IPH are trained on the training data of IntHarmony. While testing with PortraitTest, we compare with two more configurations: Bargain-Net (R) (++) and IPH (++) which are further finetuned on augmented portrait images from the MHP dataset as explained in Section 3.3. As IntHarmony actually contains objects, people and natural scenes, validating on it helps us prove that the interactive feature helps in even normal harmonization. Validating on PortraitTest helps us prove if our method will work well on real-world high resolution portrait images. It also helps us understand where our method stands when compared to professional editors.

Referenced Quality Metrics:
We use referenced quality metrics like PSNR, SSIM and MSE to quantitatively compare the performance of IPH with the previous methods. Table 1 consists of the performance comparison on both the PortraitTest and IntHarmony test datasets. We note that IPH achieves the best performance in terms of PSNR, MSE and SSIM when compared to the previous harmonization networks. IPH outperforms all previous methods across all metrics in both datasets. It can be noted that BargainNet (R)/ BargainNet (R) (++) do not perform as well as IPH/IPH (++) even though they are trained with the same pipeline and training conditions of IPH. Prior methods fail as they do not consider arbitrary regions to guide the harmonization and also don't account for luminance matching. We get the best performance as the proposed approach is more effective at improving the performance compared to the previous frameworks. Visual Quality Comparison: We visualize the predictions of our proposed method along with the previous methods in Figures 4 and 5. In Figure 4, we show results on composite portraits from PortraitTest dataset. It can be seen that the harmonized foreground generated by generic harmonization methods like Dove-Net, Bargain-Net, and Rain-Net is inconsistent with color and luminance conditions when compared to the reference portrait. Also, the predictions of Bargain-Net and Rain-Net fail to harmonize the local lighting conditions present in the composite as they fail to explicitly match the highlight, mid-tone and shadow. Bargain-Net (R) (++) performs interactive harmonization however it still fails to match the contrast of the reference. Our method harmonizes the composite well even considering the local luminance conditions and is very close to the annotations done by a professional. In Figure 5, we show results on composite portraits from IntHarmony dataset. It can be observed that our method produces realistic harmonization results closer to ground truth when compared to previous methods. The quality of predictions proves the usefulness of our proposed approach to solve the interactive harmonization problem. Please refer to the supplementary material to see many more visualizations.
User Study: We also conduct a human subjective review to compare the performance of IPH with other methods. We select 15 images from the PortraitTest dataset and present the predictions of Dove-Net, Bargain-Net (R) (++), Rain-Net and IPH (++) on a screen. We ask 30 subjects to independently rank the predictions from 1 to 3 based on the visual quality considering if the color and luminance conditions of all the portraits in the image look consistent. We give scores 3,2, and 1 for ranks 1,2 and 3 respectively. The mean scores of each method has been tabulated in Table 2. We note that our method achieves the best score which is consistent with the observation from performance metrics comparison.

Discussion
Ablation Study: We conduct an ablation study to analyse the importance of each component of our proposed method. We evaluate these models on the Por-traitTest dataset. We start with just the Harmonizer Network (H-Net) without AdaIN layers trained for generic harmonization. Then we add the style encoder and AdaIN layers in the decoder and train for interactive harmonization (H-Net+ SE). All of these methods are trained using the losses L harmonization , L consis and L triplet . Then we add the novel losses introduced in this work -  L highlight , L mid−tone , and L shadow one by one. From Table 3, it can be observed that all the new components introduced in this work are important and play a key role in achieving better performance. Making the framework interactive by using the style encoder and injecting the style code of reference region to guide the harmonization is sufficient the push the performance above previous methods. Furthermore, the proposed luminance matching loss helps improve the performance by a significant margin proving that matching the highlight, mid-tone and shadow of the foreground and reference help obtain better harmonization. Robustness to chosen reference region: One key advantage our framework has over generic harmonization is that it is more flexible and robust. We show that IPH is robust to different regions chosen by the user. We note that the output harmonization adjusts itself with the select region chosen from the background. This is very useful in cases where the luminance conditions vary across different parts of the background image. We illustrate this in Figure 2.
Converting user-guided to automatic? Please note that although we provide the flexibility for the user to point to a reference region to guide the harmonization, our framework can be made completely automatic. We can make  use of saliency detection models [20] to get the segmentation mask of the most salient object/person in the background and use that as the reference region to guide the portrait harmonization. In fact, we can make this the initial outcome for portrait harmonization and get feedback from the user if they are interested to select a different region or a sub-part of the current region selected to guide the harmonization. Limitations: Although interactive portrait harmonization outperforms previous harmonization methods, there are still some limitations. Our method works fairly well even for in-the-wild composite portraits consisting of people. However, its performance decreases if we test it on portraits consisting of objects. This can be understood as it is still difficult for the style encoder to extract meaningful style code if the object material is peculiar. For example, if the reference object chosen has a shiny surface or is of a unique material, it is difficult for the style encoder to extract a meaningful style code capturing the texture/material reflectance of the object and use it for harmonization. Potential Negative Societal Impacts: The proposed method may have negative social impacts if people are not using this model properly. As this work focuses on making an edited image look realistic, it can be used to create realistic fake images. Realistic fake images should be contained as it can act as a tool for harassment and fraud. One potential solution is gated release of trained models.
More discussion on the use-cases and extensions of our proposed interactive framework can be found in the supplementary material.

Conclusion
In this work, we proposed a new framework called interactive portrait harmonization where the user has the flexibility of choosing a specific reference region to guide the harmonization. We showed that this helps the user obtain more realistic harmonization results while providing more control and flexibility to the compositing workflow. As professional portraits usually contains screens as background and normal portraits usually contain spatially varying luminance conditions, our framework is well suited for portrait harmonization as choosing a certain reference region to guide the harmonization makes much more sense in these setups. We also proposed a new luminance matching loss to carefully match the appearance between the foreground composite and reference region. In addition, we introduced two datasets: a synthetic IntHarmony dataset for training and a real-world PortraiTest dataset for testing. Extensive experiments and analysis showed that IPH performs better and is more robust and useful to solve real-world portrait harmonization problems in practical settings compared to the previous methods achieving realistic portrait harmonization predictions.
Ethics in Data Collection: In our proposed PortraitTest dataset, we make use of professional portraits which contain humans present in the scene. Consent to use or share the data was obtained. We get the help of a data vendor who handles licensing or approval issues for us. We also note that no other sensitive information (like name) about the subject is obtained or used.
Augmentations: To generate the composite image in the synthetic dataset IntHarmony, we use a set of augmentations. For each image, these augmentations are chosen at random. To account for appearance changes, we use a brightness and contrast augmentation. We also use color jitter to enforce hue augmentation. In addition, we also use a gamma transformation. Apart from these, we also use 3D LUT augmentations as proposed in [10]. We also generate local masks and enforce augmentations to cover local lighting effects. Here, we apply augmentations such as soft lighting, dodge, grain merging and grain extraction. Note that all the above set of augmentations are considered as a set and for each image a random augmentation is selected and applied on a composite foreground.
Tweaking the framework for Interactive Color Transfer: In the main paper, we introduced an interactive harmonization framework. Here, we show that the framework can be easily tweaked to perform color transfer. Note that the appearance of the predicted image can be easily controlled using the style code. If the style code is made to learn the match color instead of harmonizing the image, we show that our framework can be used to perform color transfer. We make two simple changes in our framework to obtain this: (1) increase color augmentations (2) change L consis loss to make the style codes of the foreground and the reference region to be far apart from each other; we could obtain style encoder which impart an aggressive color change. We term the style code extracted from this style encoder as ψ and the original style code as φ. Now to control the appearance, we just blend these color codes extracted from these two encoders using different rations. The new style code γ would be γ = r1φ + (1 − r2) * φ, (1) where r1 and r2 are the blending ratios. This shows that IPH can be easily tweaked to perform interactive color transfer resulting depending on the user's choice of r1 and r2. In Figure 1, we show the change in appearance for different combinations of r1 and r2. Why other harmonization methods cannot be made interactive? We note that almost all of the previous frameworks [4,15,6,7,15] take only 2 inputs: the direct composite image and the foreground segmentation map. The foreground map basically helps the network understand which portion of the image needs to be harmonized. Due to this, there is actually no way of sending another region as a guidance to harmonize the foreground.
Bargain-Net and Rain-Net: We note that Bargain-Net [3] and Rain-Net [15] specifically focus on using the background to guide the harmonization. Bargain-Net has a domain extractor to extract the features of background while Rain-Net proposes a RAIN module which normalizes the background features with the foreground. We tried to make both of these networks interactive by replacing the background mask with a guide mask to select the reference region. With the pre-trained weights, while changing the guide mask we did not see any significant change in terms of performance metrics nor visual quality for both Bargain-Net and Rain-Net. If we retrain the network to take in the guide mask instead of background mask, we noted a small improvement in results for Bargain-Net as reported in the main paper. For Rain-Net, we did not see any improvement. This might be because RAIN module uses the masks in the feature space to normalize and that might not be an optimal way to actually get meaningful features to inject to the foreground. Training it in an interactive way in fact is not robust at all and we get a small decrease in performance. We also not that for these setups changing the guide mask to background mask does not bring out much effect in the harmonized output which makes it clear that these frameworks cannot be converted to serve interactive harmonization efficiently.

Number of Parameters:
In terms of parameters, IPH is lighter compared to the previous methods. IPH has 27M parameters compared to Bargain-Net's 58M , Dove-Net's 54M and Rain-Net's 54M number of parameters.
In-the-Wild Portrait Harmonization Results: To further validate our method for in-the-wild photos, we randomly create composite images and check the portrait harmonization results. We visualize these composites and the corresponding results in Figure 9. It can be observed that our method produces realistic harmonization results with the appearance being close to how it would have been if the person was photographed in-situ.

Composite Image
Our Prediction Fig. 2. Qualitative results while testing our method for in-the-wild composite portraits. The red box denotes the composite foreground. We use the portrait in background as reference region.


EgoEnv: Human-centric environment representations from egocentric video

First-person video highlights a camera-wearer's activities in the context of their persistent environment. However, current video understanding approaches reason over visual features from short video clips that are detached from the underlying physical space and capture only what is immediately visible. We present an approach that links egocentric video and the environment by learning representations that are predictive of the camera-wearer's (potentially unseen) local surroundings to facilitate human-centric environment understanding. We train such models using videos from agents in simulated 3D environments where the environment is fully observable, and test them on human-captured real-world videos from unseen environments. On two human-centric video tasks, we show that state-of-the-art video models equipped with our environment-aware features consistently outperform their counterparts with traditional clip features. Moreover, despite being trained exclusively on simulated videos, our approach successfully handles real-world videos from HouseTours and Ego4D. Project page: https://vision.cs.utexas.edu/projects/ego-env/

Introduction
Egocentric video offers a unique view into human activities through the eyes of a camera-wearer. Understanding this type of video is core to building augmented reality (AR) applications that can assist humans by answering their questions, recalling events, or anticipating their needs based on their activity. Ego-video is thus the subject of several recent datasets and benchmarks that are driving new research [12,26,45,70].
A key feature of the egocentric setting is the tight coupling of a camera-wearer and their environment as they traverse a persistent physical space, potentially over multiple hours (or days). This raises an important need for humancentric environment understanding -to learn representations from video that capture the camera-wearer's activities in the context of their environment. Such representations are important to answer questions like "what room is the person in right now?", or "when was the person last near the sink?". The former requires reasoning over video frames spread in time to build a coherent picture of the physical space. The latter requires reasoning about the camera-wearer's location relative to other objects, regardless of their visibility (e.g., a person at a sink, but looking at a mirror above). Despite its importance, there has been only limited work on learning human-centric environment representations. Current video models segment a video into short clips (1-2s long) and then aggregate clip features over time (e.g., with recurrent, graph or transformer-based networks) for tasks like action forecasting [23,25,26,54], temporal action localization [48,49,89,90], episodic memory [14,26], and movie understanding [85]. Critically, the clip features encode what is immediately visible in a short time window, and their aggregation over time does not equate to linking them in physical space (as needed to answer, e.g., where is the camera-wearer? what objects are within reach?). Other approaches use explicit camera pose information (e.g., from SLAM), but model the agent and the surrounding environment independently. For example, forecasting models [27, 55,64] use pose only to localize the camera-wearer, ignoring its relation to objects in the surrounding space. Affordance learning approaches [54,63] use pose to group activities by location, but do not learn representations for agent video. Moreover, these approaches require camerapose paired with each video frame, which precludes their use for egocentric video where characteristic blur and quick head motions prevent reliable pose estimates.
To address these shortcomings, we propose to learn environment-aware video representations that encode the surrounding physical space. Specifically, we define the local environment state at each time-step of an egocentric video as the set of objects (and their rough distance) in front, to the left, right, and behind the camera-wearer. We use this state as supervision to train a transformer-based video encoder model that aggregates visual information across a video to build an environment memory which can be queried to predict the local state at any point in the video.
The local state captures the rough layout of objects relative to the camera-wearer and is important for understanding physical space -it provides a semantic signal to localize people (e.g., in the living room, from the relative arrangement of couches, lamps and tables), as well as human behavior -people move towards layouts that support activities (e.g., stove-top areas, dressers). Predicting the local state thus involves capturing the natural statistics of object layouts across different homes and then translating contexts across environments to reason about new ones.See Fig. 1. Once trained, given an observation from a new video, our model produces a "drop-in" EgoEnv feature which encodes environment information to complement existing video clip features. Importantly, our EgoEnv feature encodes general information about the local surroundings and can benefit multiple downstream tasks.
An important practical question is how to supervise such a representation. Sourcing local state labels requires agent and object positions and omnidirectional visibility at each time step. This is challenging as egocentric videos only offer sparse coverage of the environment. Furthermore, they are prone to object detection, tracking, and SLAM failures. Therefore, for training we turn to videos generated by agents in simulation. This allows us to sample diverse, large-scale trajectories to cover the environment, while also providing ground-truth local state. Once trained with simulated video, we apply our models to real-world videos from new, unseen environments.
We demonstrate our approach on human-centric environment understanding tasks in video, where joint reasoning of both human motion and action in video and the underlying physical space is required. These include inferring the room category that the camera wearer is physically in as they move through their environment, and localizing the answer to a natural language query in an egocentric video. These tasks support many potential applications, including AR systems that can offer context-relevant assistance. For example, an AR assistant that offers location-specific sug-gestions for chores to complete based on the rooms visited by the user; or one that recalls the last time the user was near the sink, to check if the tap was left turned-on.
We are the first to demonstrate the value of 3D simulation data for real-world ego-video understanding. Our experiments show that by capitalizing on both geometric and semantic cues in our proposed "local environment state" task, we can leverage video walkthroughs from simulated agents in HM3D scenes [60] to ultimately enable downstream human-centric environment models on real-world videos from HouseTours [7] and Ego4D [26]. Furthermore, models equipped with our EgoEnv features outperform both popular scene classification [91] and state-of-the-art natural language video localization models [89].

Related work
Video understanding in 3D environments. Prior work encodes short video clips [4,33,57,80] or temporally aggregates them to add additional context [23,48,49,84,85,89,90]. However, these methods treat the video as a temporal sequence and fail to capture the spatial context from the underlying persistent environment. For egocentric video, prior work has used structure from motion (SfM) to map people and objects for trajectory [55] and activity forecasting [27] and action grounding in 3D [13,63]. These approaches localize the camera-wearer but do not learn representations for the camera-wearer's surroundings. The model of [50] associates features to voxel maps to localize actions; however, they require a pre-computed 3D scan of the environment at both training and inference. Prior work groups clips by rough spatial location as topological graphs [54] or activity threads [58], but they stop short of learning representations using these groups. In contrast, we go further and explicitly learn features that relate clips based on their spatial layout, for each step of an egocentric video.
Video representation learning. Traditional video understanding methods learn representations by training models on large, manually curated video action recognition datasets [4,26,52]. Recent self-supervised learning (SSL) approaches eliminate the supervision requirement by leveraging implicit temporal signals [22,31,38,53,62,76,79,[81][82][83]. In contrast, we learn features that encode the local spatial-state of the environment (as opposed to temporal signals). Further, we show how to leverage state information that is readily accessible in simulation, but not in video datasets (i.e., locations and semantic classes of objects surrounding the agent) for training.
Environment features for embodied AI. In embodied AI, pose-estimates are used to build maps [8,10,59], as edge features in graphs [7,9], as spatial embeddings for episodic memories [19], to project features to a grid map [5,28,36] or to learn environment features for visual navigation [61].
However, these approaches are explored solely in simulation and typically require accurate pose-estimates or smooth action spaces, and thus are not directly applicable to egocentric videos. Research on world-models [17,29,30] and unseen panorama reconstruction [39,42,71] hallucinate the effect of agent actions to aid decision-making in simulation. Our approach is similarly motivated, however we aim to learn environment features for an egocentric camera-wearer to aid real-world video understanding.
Learning from simulated data. Prior work has proposed cost-effective ways to generate large-scale synthetic image datasets for various vision tasks [15,18,37,66,72,74]. In robotics, simulation environments have been developed to quickly and safely train policies, with the eventual goal of transferring them to real world applications [6,41,43,60,68,73,75,86]. The resulting sim-to-real problem, where models must adapt to changes in simulator and real-world domains, is an active area of research for robot navigation [1,3,40,67,74]. Simulated data for video understanding is much less explored. Prior work has synthesized data for human body pose estimation [11,16,77,87], trajectory forecasting [46] and action recognition [65]. Rather than model human behavior, our approach is the first to directly capture the semantics of the environment surrounding the camerawearer for real-world video understanding tasks.

Approach
Our goal is to learn EgoEnv representations that encode the local surroundings of the camera-wearer -where they are in relation to nearby, potentially obscured objects. Such a representation implicitly maintains a semantic memory of surrounding objects beyond what is immediately visible, and once coupled with a standard video feature, it allows a model to jointly reason about activities in video and the underlying physical space. Directly appending camerawearer pose with each video frame may capture the local state, however noise in pose estimated from ego-video with quick head motions and characteristic blur limits its utility (see Supp. for experiments). Moreover, training a model to effectively utilize pose is itself non-trivial as camera-wearer location alone does not reveal the relative poses of objects.
Instead, we introduce an approach that leverages simulated environments where perfect state information is available to train models that can link visual information to the physical surroundings. To this end, we first define the local state prediction task in simulation (Sec. 3.1). Next, we introduce our transformer-based architecture that predicts the local state in videos (Sec. 3 Figure 2. Local environment state illustration. Views in each direction are shown for clarity (left panel), but only the forward view is accessible at each step (top left). Not all parts of the environment are seen during a walkthrough (white vs. grey regions)models must link seen observations based on their shared space, as well as anticipate unseen surroundings. Best viewed in color.

Local environment state
We require a model that is aware of not just what is immediately visible in a single frame, but also of the camerawearer's surroundings. We therefore define the local environment state of the camera-wearer as the set of objects in each relative direction -i.e., what objects are to the front, left, right or behind the camera-wearer, together with their rough distance from the camera-wearer -and train a model to predict this state. Our definition of local state takes inspiration from cognitive science [32,44], and offers supervision signals that are both geometric (relative object locations) and semantic (semantic object labels), which we observe leads to strong representations.
More formally, let O be a set of object classes. For a frame f from a video trajectory in an environment, the local state is a tuple (y o , y r ). y o is a 4 × |O| dimensional matrix which represents instances of each object class in the four cardinal directions relative to the camera-wearer. y r is a matrix of the same size containing the distance of the objects in y o from the camera-wearer, discretized into 5 distance ranges between 0.25 − 5.0m 1 . For direction i and object class j, the labels are calculated as follows where p a , p j are the poses of the camera-wearer and object o j respectively, d(p a , p j ) is the euclidean distance between them, δ is a distance threshold for nearby objects (we set δ = 5.0m, beyond which visible objects are small), θ(p a , p j ) ∈ [0..  camera-wearer's possibly hidden surroundings to produce environment-aware features. However, obtaining supervision for camera-wearer pose and every object's location is non-trivial for egocentric videos where trajectories have limited environment coverage, and where camera localization and tracking is error prone. We address these issues in the next section by leveraging videos in simulated environments for training.

Environment-aware pretraining in simulation
To source local state labels, we generate a dataset of video walkthroughs of agents in simulated 3D environments where agent and object poses are accessible at all times (see Sec. 4.1). We train our model in two stages described below.

Pose embedding learning
While ground truth camera pose is available from the simulator at training time, a model trained to rely on it will fail on real-world egocentric video at test time, where pose estimates are noisy and in many cases unavailable. With the goal of handling arbitrary indoor egocentric video, we instead explore representations that implicitly encode coarse pose information. See Supp. for pose-related experiments.
Specifically, for a sequence of RGB frames {f t } T t=1 and camera poses {θ t } T t=1 , we generate pose embeddings {p t } T t=1 = P(f 1 , ..., f T ) using a transformer encoder network. These embeddings are used to predict the relative pose between each observation pair using a bilinear layer whereθ i,j is the predicted relative pose and V p , W p , b p are trainable parameters. In practice, we discretize the relative pose into 12 angles and 4 distance ranges as it is difficult to regress to fine-grained pose values.
The network is trained to minimize cross-entropy between the predicted and the target relative pose labels for all observation pairs i,j L ce (θ i,j , θ i,j ). The trained pose embeddings p t encode information to help relate video observations based on their location and orientation in the environment. Note that once trained, pose embeddings are inferred directly from video frame sequences -ground truth pose is only required for training.

Local state pretraining
Next, we train a model to embed visual information from a video walkthrough into an environment memory, which can then be queried to infer the local state corresponding to a given video frame from the same video. We implement this model as a transformer encoder-decoder model.
Specifically, for a video walkthrough V with RGB frames {f t } T t=1 , and a query frame f q , we predict the local state y q = (y o , y r ) as follows. First, pose embeddings {p t } T t=1 are generated for the video and query frames following Sec. 3.2.1. Then, each frame is encoded jointly with the pose embedding using a linear transform M p .
Next, we uniformly sample K video frames to construct an environment memory using a transformer encoder E, which updates frame representations using self-attention: The resulting memory represents features for each timestep that contain propagated information from all other time-steps. Compared to prior work [23,48,49,89,90], our encoder has the ability to relate observations based on not just visual characteristics and their temporal ordering, but  also their relative spatial layout in the environment. A transformer decoder D then attends over the memory using query x q to produce the output EgoEnv representation h q : which is finally used to predict the local state using two linear classifiers M o and M r for object class and distance predictions respectively. The network is trained to minimize the combination of losses over the predicted and the target state labels for each direction: where L bce and L ce refer to binary cross-entropy and crossentropy losses respectively. We set λ = 0.1 which balances the contributions of each loss function based on validation experiments. The distance loss is computed only for objects that are in the local state (y o = 1). See Fig. 3 (left) and Supp. for more architecture details. Learning to predict the local state involves aggregating information about observed objects across time, as well as anticipating unseen objects based on learned priors from the layout of objects in other scenes (e.g., TVs are usually in front of couches; kitchens have particular arrangements of sinks, refrigerators, and stove-tops).
Once trained, given a video in a new environment and a target time-point of interest, our model constructs an environment memory, predicting the local state based on information aggregated through the entire test video. Importantly, h q -the EgoEnv feature -contains valuable information about the camera-wearer's surroundings offering environment-centric features to complement traditional video features (e.g., for a person watching TV, also encode features for the couch they are sitting on, the lamp nearby).

Environment-memory for video understanding
Next, we leverage our environment-memory model for real-world video tasks. A video understanding task defines a mapping from a sequence of video clips {c 1 , ..., c N } from longer video V to a task label. We consider two tasks: (1) ROOMPRED: where the model must classify which room r t the camera-wearer is in (e.g., living room, kitchen) at time t in the video, and (2) NLQ: natural language queries, an episodic memory task popularized recently in Ego4D [26] where the model must identify the temporal window (t s , t e ) in the video that answers an environmentcentric query q specified in natural language. See Fig. 4.
Current models produce clip features that encode only what is immediately visible. This is sufficient for shorthorizon tasks (e.g., action recognition), but as we will show, falls short on the tasks above that require extra reasoning about the agent's surroundings. Our environment-memory model addresses this by enhancing standard clip features with context from the camera-wearer's surroundings.
To do this, for each input clip in c i ∈ V, we select the center frame f i of the clip as the query frame. Following Sec. 3.2, we uniformly sample K frames from the video around the query frame, encode them along with their pose embeddings (Eqn. 4), and build an environment memory using our environment encoder E (Eqn. 5). Finally, we use our decoder D to produce the output feature. This results in set of output EgoEnv features, one per input clip.
Each environment feature then enhances the original clip feature as follows where g i is the original clip feature for clip c i (e.g., ResNet, SlowFast, EgoVLP.) and W E , b E are linear transform parameters. See Fig. 3 (right).
The new clip features {g 1 , ..., g N } consolidate features from what is directly visible in a short video clip and features of the (potentially unseen) space surrounding the camera-wearer. Put simply, our approach implicitly widens the field of view for tasks that reason about short video clips by providing a way to access features of their surroundings in a persistent, geometrically consistent manner.
Critically, our core approach is task-agnostic-we use the exact same EgoEnv representations to tackle both video understanding tasks. This is a departure from traditional sim-to-real approaches where a task-specific dataset needs to be carefully designed for every downstream task, which may be impractical for a large number of tasks, or complex tasks beyond the simulator's capability (e.g., simulating human motion, hand-object interaction). Note that our approach is best suited to videos in indoor environments to stay consistent with the scenes used for pretraining. In addition, indoor layouts have a higher density of nearby objects, providing richer local state labels to train our models with.

Experiments
We evaluate how our EgoEnv features learned in simulation benefit real-world video understanding models.

Simulator environments and video datasets
For training, we use the Habitat simulator [68] with photo-realistic HM3D [60] scenes to generate simulated video walkthroughs. We generate 15k walkthroughs from 900 HM3D scenes, each 512 steps long, taken by a shortestpath agent that randomly samples goal locations in the environment and navigates towards them (move forward, turn right/left 30 • ). For each time-step, we obtain the groundtruth local state from the simulator required in Sec. 3.1 (i.e., object labels and relative pose). For object labels, we map instance predictions across |O| = 23 categories to the 3D scenes using a pretrained instance segmentation model [20] trained on the subset of scenes with semantic labels.
Though the walkthroughs involve discrete actions, they share characteristics with real-world video (cameras at head-level; views covering the environment) making them suitable for transfer. See Supp. for further details.
We evaluate our models on three egocentric video sources. (1) Matterport3D (MP3D) [6] contains simulated video walkthroughs from 90 photo-realistic 3D scenes. We use 146 long video walkthroughs and standard data splits [5]. (2) HouseTours [7] contains 119 hours of realworld video footage of house tours from YouTube. We use 32 hours of video from 886 houses where the camera can be localized and create data splits based on houses. (3) Ego4D [26] contains 3k hours of real-world video of people performing daily activities. We use all 227 hours annotated for the NLQ benchmark and apply the provided data splits.
These datasets provide an ideal test-bed with increasing realism and associated challenges. MP3D offers novel scenes with distinct visual characteristics and object distributions compared to HM3D, allowing us to test our model's robustness to domain shift in a controlled simulated video setting. HouseTours captures high quality videos captured by people, allowing us to test generalization to real-world visuals as well as new, unseen houses. Ego4D captures nat-  ural human activity in diverse cluttered environments allowing us to further test our model's ability to relate human action and the underlying environment.
We collect labels for each task, which we will publicly share. For ROOMPRED these are room category labels (e.g., living room, kitchen) for each time-step. On HouseTours and Ego4D, we crowd-source room labels across 21 categories. For NLQ these are natural language queries and corresponding response tracks in the video. On House-Tours, we crowd-source queries across 8 templates (e.g., "where did I last see my phone in the kitchen", "when did I first visit the bathtub"). We group the templates into "see" queries, where the object of interest is directly visible in frames, and "visit" queries, where the camera-wearer is near, but may not actually see the object, which requires extra environment-level reasoning to localize. On Ego4D, we use the existing NLQ benchmark annotations which contain queries across 13 templates that require reasoning over actions, objects, and locations (e.g., "what tool did I pick up from the table", "where did I hang the pink cloth"). On MP3D, we source all labels directly from the simulator -9 room categories, and automatically generated NLQ queries from simulator object labels and locations. See Supp. for data collection details and Fig. 4 for examples.

Experiment setup
For pre-training, we use 2048-D ImageNet-pretrained ResNet50 [35] features for each video frame. Our encoderdecoder models P, E, D are 2-layer transformers [78] with hidden dimension 128. K = 64 frames are sampled from each video to populate the memory. We train models for 2.5k epochs and select the model with the lowest validation loss. For downstream tasks, we sample K = 64 frames for our memory. For ROOMPRED, we generate a single EgoEnv feature aligned with the query clip. For NLQ we generate one feature per input clip. Full architecture and training details are in Supp.

Baselines
We compare the following approaches to enhance the clip representations in a baseline model. For ROOM-PRED the baseline is a popular scene recognition model PLACESCNN [91]. For NLQ we use a state-of-the-art moment localization model VSLNET [47,51,88].
• FRAMEFEAT uses a pretrained ResNet50 [35] model to generate a frame feature corresponding to each clip.
• OBJFEAT trains an object detector on all available simulated HM3D data and generates backbone features for each clip. We use the QueryInst model [20].
• MAE [34] trains a state-of-the-art self-supervised learning approach to reconstruct the pixels of masked patches in frames from our simulated video trajectories.
• EGOTOPO [54] trains a graph convolutional network (GCN) over the video graph built following [54]. Graph construction details are in Supp.
• EPC [61] trains an environment memory model to predict masked zone features conditioned on pose.
• TRF (SCRATCH) trains a scene-memory transformer model [19] that shares our model architecture but does not benefit from pretraining.
These baselines represent various strategies to incorporate environment information into clip representations ranging from frame features (MAE, FRAMEFEAT, OB-JFEAT), to topological graph-based features (EGOTOPO), to pose-based features (EPC). Note that OBJFEAT, MAE and EPC all benefit from pre-training on the same walkthrough videos as our approach. OBJFEAT further benefits from ground-truth object labels from the simulator. Features from these approaches augment the input clip representations following Eqn. 8 -baseline architectures remain unchanged. Note that EPC requires privileged informationground-truth camera poses at inference time-whereas our model does not. Supp. contains pose estimation details and further experiments.

Local state pretraining
We begin by evaluating how well our model can infer the local state, reporting average precision (AP) in each direction. Given a forward view, objects can be reliably recognized (37.8 AP) compared to naively outputting the distribution of objects seen at training (5.4 AP). Moreover, our approach can link views in the video trajectory to also infer and anticipate objects in other directions, i.e., to the right, left and behind (21.5, 24.9 and 20.2 AP respectively) given the forward view. We visualize the attention weights learned by our model to link relevant observations to the query in Fig. 5. Our model learns to select informative views for the task beyond just temporally adjacent frames or views with high visual overlap. For example, the view with highest attention score (0.82) looks at the dining table directly behind (red box), allowing our model to benefit from information beyond its field of view.

EgoEnv features for room prediction
Next, we evaluate our method on predicting what room the camera-wearer visits in the video. All models have access to the full video, but inference is at at each time-step. Baseline. The PLACESCNN model is a 2-layer MLP classifier trained on features from a Places365 [91] scene classification model. Features are max-pooled across a clip of N = 8 frames around the time-step of interest for additional context before prediction, as a single frame may be uninformative (e.g., facing a wall). We generate an environment feature aligned with the center of the clip. Evaluation. We report top-1 accuracy as a function of dataset difficulty.Difficulty is measured as the prediction entropy of the Places365 model trained on canonical scene images. Instances are "hard" (high entropy) where frame-level information is insufficient for predicting the room type.
Our results are in Fig. 6. The accuracy on the full dataset is at the far-right end. All models perform better on House-Tours compared to MP3D and Ego4D since the house tours were captured explicitly to provide informative views of

EgoEnv features for episodic memory
Next we evaluate our method on localizing the responses to natural language queries in egocentric video. Baseline. We use VSLNET [88] and provide it with N = 128 clips sampled uniformly from the full video to generate predictions. We use SlowFast [21] clip features and generate environment features aligned with each input clip. Evaluation. We report Rank n@m metrics commonly used in moment localization [24,88,89], which measures the percentage of correctly localized moments in the top n predictions. A predicted moment is correct if it overlaps with the ground truth moment (IoU > m). We report Rank 1@[0.3, 0.5] and their mean following prior work [26].
Our results are in Table 1. Similar to ROOMPRED, instances are harder in MP3D than in HouseTours as the agents in the videos follow shortest paths across the environment, and do not focus on particular objects while navigating. As a result, moments are quick transitions between objects and locations, and contain only short glimpses of them. The global, video-level information from EGOTOPO improves performance slightly on both datasets. Strong image-level supervision (object labels in OBJFEAT and FRAMEFEAT) results in the largest improvements; MAE, which has access to the same data but trains self-supervised representations, does not show very large improvements. EPC which has access to ground-truth pose at training and inference performs well on MP3D where accurate pose is available from the simulator, but underperforms on House-Tours where only noisy pose estimates are available (see Supp.). Our EgoEnv approach performs the best overall, outperforming even EPC which has access to ground-truth pose at training and inference.
Finally, we evaluate our method on the Ego4D NLQ benchmark challenge. We augment current state-of-the-art approaches with our EgoEnv features and report our results on the validation set in Table 1 (right). Our approach enhances features from state-of-the-art action (SLOWFAST) and vision-language pretrained (EGOVLP) video features to achieve the best results. Note that Ego4D videos are fairly different from our simulated walkthrough videos in pretraining. They are in-the-wild, capture natural human actions and object-interactions, and take place in diverse scenes, both indoor and outdoor (e.g., workshops, gardens). This sim-to-real gap translates to smaller absolute improvements on Ego4D compared to MP3D and HouseTours. Despite this, our approach is still able to consistently improve all models, demonstrating the value of our environmentcentric feature learning approach.
Ablations show that pose information adds to our approach only when reliable and that both the geometric and semantic components of our local state definition are important for downstream performance (Supp.).

Conclusion
We proposed a framework to learn environment-aware representations in simulation and transfer them to video understanding tasks on challenging real-world datasets. Our approach outperforms state-of-the-art representations for predicting visited rooms and retrieving important moments from natural language queries, despite a significant sim-toreal gap. Future work can explore downstream AR applications like action forecasting or helping robots learn better from human demonstrations.

Supplementary Material
This section contains supplementary material to support the main paper text. The contents include: A. Videos illustrating our approach.
B. Data collection and annotation details.

A. Videos illustrating our approach
We present videos illustrating the local state prediction task (Sec. 3.2), downstream video tasks (Sec. 3.3) and our main results on our project page.
In the first part of the video, we demonstrate the local state prediction task from Sec. 3.2 of the main paper. The video shows the first-person view of the camera-wearer (left panel). The right panel shows the top-down view of the environment with the agent trajectory (blue gradient) and nearby objects (colored squares). Note that models only see the egocentric view -the top-down map is for illustration only. Given a simulated video walkthrough and a query time-stamp, the model must predict the direction and rough distance of each object near it. Correct, missing and false positive predictions are shown for each direction (left panel). Correct predictions on the top-down map are highlighted in cyan.
In the second part of the video, we show examples of the two downstream video tasks from Sec. 3.3 of the main paper, on MP3D, HouseTours and Ego4D. In the ROOM-PRED examples, the model must predict which room the camera-wearer is in from a short clip. As mentioned in Sec. 4.5, the clips show quick motions and often contain ambiguous views making it hard to predict room labels directly using traditional scene recognition methods. For example, the "staircase" is not visible as the person descends it (Ego4D, bottom right video). In the NLQ examples, the model must predict the moment in time that answers a particular environment-centric query. The video examples show how this requires reasoning about the camerawearer's surroundings. For example, in the HouseTours clip ("when did I visit the sink in the bathroom?") the sink is only seen briefly in the video but the response requires the window of time that the camera-wearer was physically near it (within arms reach) regardless of visibility. In the Ego4D clip ("Did I leave the drawer open"), the model must know where the drawer is relative to the camera-wearer and link their actions to this physical location in order to respond.

B. Data collection and annotation details.
We present data collection and annotation details for simulated walkthroughs, and our two downstream tasks (ROOMPRED and NLQ) for all three datasets (MP3D, HouseTours, Ego4D).

B.1. Walkthrough generation details
As mentioned in Sec. 4.1 of the main paper, we generate simulated walkthroughs in HM3D [60] scenes to train our models. Given an environment, we first cluster all navigable points using KMeans, selecting between 4-64 clusters depending on the environment size. With each cluster centroid as a starting location, we sample 8-16 nearest goal locations, shuffle them (to allow re-visitation), and make an agent visit the goals in sequence. We use a shortest-path planning agent that uses the underlying navigation graph to reach goals in the fewest number of steps. We collect a dataset of 15k episodes, each of 512 steps, of our agents visiting such goal sequences for experiments in Sec. 3.2. Fig. 1 shows a random sample of walkthroughs.
Note that the walkthroughs are generated in environments where objects are not moved, however a large part of real-world environments are in fact static. This includes static scene elements like doors, windows, counter tops, staircases, and most objects that are typically not moved like refrigerators, beds, couches, TV sets. Encoding these SCENES ROOMPRED NLQ HM3D [60] 800 / 100 / ---Matterport3D [ objects and scene elements can thus still provide value for human-centric environment understanding, even when some objects may have moved around.

B.2. Data collection and annotation details
As mentioned in Sec. 4.1 of the main paper, we collect labels for each video dataset. For Matterport3D, we directly use the ground truth information available through the simulator to extract labels. For HouseTours and Ego4D, we crowd-source annotations where required. We describe the data collection process and present data statistics for each dataset and task. The resulting dataset splits can be seen in Table 1.

B.2.1 Annotation requirements
ROOMPRED For this task, room labels are required at each time-step of the video. The 9 room categories used in Mat-terport3D are in Table 2 (left). These categories are predefined in the simulator. The 21 room categories used in HouseTours and Ego4D are in Table 2 (right). These categories were generated manually from a combination of Matterport3D room categories and a relevant subset of Places365 categories corresponding to indoor scenes. NLQ For this task, natural language queries and corresponding moment boundaries (start and end times) are required. For HouseTours, we define 7 query templates where o refers to objects and r refers to rooms: "see o", "see o in r", "see o 1 then o 2 ", "visit r 1 then r 2 ", "visit o/r", "visit o 1 then o 2 ", "visit o in r". Each template captures a type of question that requires a different mechanism of reasoning. "see" queries require reasoning about what is immediately visible; "visit" queries require an understanding of where the camera-wearer is in the environment and what objects are nearby (within arms reach); "see/visit o 1 then o 2 " and "see/visit o in r" require both spatial and temporal reasoning. Natural language queries follow from these templates. For example, for the "visit r 1 then r 2 " template, a natural language query may be "When did I first walk from the kitchen to the bathroom?". The list of query templates, examples and descriptions can be seen in Table 3. The task definition follows prior work [26] but is adapted for the datasets used, and contains more environment-centric queries. For Ego4D, we use the existing NLQ benchmark task annotations. Fig. 4 of the main paper. The video highlights the stark contrast between prediction in static images (third-person photos) which contains well-framed images that are easy to recognize, and egocentric video which is much more challenging. In this setting, video is tied to quick ego-motion as the camerawearer moves around the environment and objects are seen only briefly (or not at all) in non-canonical viewpoints.

B.2.2 Matterport3D annotations
For ROOMPRED, navigable location are mapped to room categories using information from the simulator. We map agent positions for each video frame to these categories. For NLQ we use room labels and extracted object positions to generate queries from the 7 templates above. We define objects as "seen" if they occupy at least 5% of the pixels in a given frame. We define objects as "visited" if the agent is < 1.0m from the object, regardless of its visibility, following embodied navigation protocol (ObjectNav [2]). Rooms are "visited" using the position → room category mapping above. We generate queries for each template by tracking objects and rooms that are seen and visited over time. If there are multiple visitations to an object or room, we ensure unique responses to queries by adapting them to consider only the first (or last) visit.

B.2.3 HouseTours annotations
We crowd-source annotations for real-world videos from HouseTours. For ROOMPRED, we ask annotators to watch a video and mark the start and end time of each "visit" to a room. They must then label each visit with one of the 21 room categories (from Table 2, right). An illustration of the annotation interface is shown in Fig. 5.
For NLQ, we ask annotators to identify an interesting moment (e.g., where a person sees a salient object, moves from one room to another, visits an important object) which serves as the answer to a query. The moment is specified by a start and end time, while the query is specified as natural language text generated following one of the 7 template classes from Table 3. An illustration of the annotation interface is shown in Fig. 6. Fig. 3 shows annotation statistics for both tasks on HouseTours. In general, trajectories are relatively shorter than Matterport3D, though they involve a similar number

B.2.4 Ego4D annotations
Following the same procedure as HouseTours, we crowdsource room visit labels on Ego4D videos (see Fig. 4). Ego4D videos are much longer (30 mins on average). As a consequence, visits are naturally a much smaller fraction of the overall video and the number of room transitions are much higher (16 on average). The distribution of room categories is similar to HouseTours. Annotation statistics for NLQ can be found in Section F.4 of the supplementary material of the Ego4D paper [26].

Template
Example Description see o Where did I first see the remote control? Objects must be visible for the moment duration see o in r When did I see the mirror in the bathroom? Object must be physically inside the room see o 1 then o 2 Where did I see a table then a chair? Objects can either be seen together or in quick succession visit r 1 then r 2 When did I walk from the living room to the kitchen? Start = when the person begins to leave; End = they are fully inside the kitchen. visit o/r When did I last visit the couch?; When did I last visit the bedroom? Visit = physically near an object (within arms reach) or physically inside a room visit o 1 then o 2 When did I visit the lamp then the couch? Same as see o 1 then o 2 , but using the "visit" criteria above visit o in r When did I visit the mirror in the bathroom?
Same as see o 1 in o 2 , but using the "visit" criteria above       . Annotation interface for collecting NLQ labels. Annotators must identify an interesting moment in time (start and end time) and associate a query template, a natural language query and object/room labels that fill the query template slots.

C. Implementation and training details
We present architecture and training details for all approaches.

C.1. Architecture and training details (pretraining)
We present additional details for our model in Sec. 3 of the main paper, as well as the baselines in Sec. 4.3. The full list of hyperparameters are in Table 5 (panels 1-3).
Pose embedder P, environment encoder E and decoder D. We build on the transformer [78] architecture for our model. The inputs f ∈ R 2048 are features from an ImageNet-pretrained ResNet-50 . These are encoded into 128-dimension vectors using a visual encoder (2-layer MLP).
First, 128-D pose embeddings are generated following Eqn. 3. These pose embeddings are concatenated with the visual embedding, and then transformed back to a 128dimension vector using M p . Sin-cosine position embeddings are added to this input following [78] resulting in the transformer input {x 1 , ..., x N } in Eqn. 4. Note that a separate visual encoder generates input features for the encoder E and the pose embedder P.
The encoder E performs multi-headed self-attention using these inputs, with 2 layers, 8 heads and hidden dimension 128. The decoder D is a 2-layer transformer with hidden dimension 128 which attends to the outputs of E to generate the output representation. We predict relative directions for 23 object classes in HM3D (see Table 4). We use the transformer implementation from PyTorch [56].
Our architecture is similar to prior embodied navigation approaches [19,61] but does not require pose (and instead uses pose embeddings), includes the environment decoder that queries the memory based on pose embeddings and visual content, and is trained using our proposed learning objective in Sec. 3.2. See Table 5 (panel 1).
EPC architecture details. EPC [61] is a transformer encoder-decoder model that masks out frames from physical locations and predicts the features of these masked zones given a query pose. We generate graphs to train this baseline following their approach.
Specifically, for every video frame {f 1 , ..., f T }, we compute the geometric viewpoint overlap with every other frame. The viewpoint overlap ψ(f i , f j ) is calculated by projecting pixels from the frames to 3D point-clouds using camera intrinsics, agent pose and depth measurements, and measuring the percentage of shared points across frames. We use ψ(f i , f j ) as our distance metric to cluster all frames in video into zones using hierarchical agglomerative clustering. We set the distance threshold to 0.8 as larger values result in too few zones.
We sample 4 unseen zones per instance in a batch of which one is "positive" and three are "negatives" for con-chair  table  picture  cabinet  cushion  sofa  bed  chest of drawers  plant  sink  toilet  stool  towel  tv monitor shower  bathtub  counter  fireplace  shelving seating  furniture appliances clothes   Table 4. HM3D object taxonomy trastive learning. We collect negatives from all instances in a batch during training. The network is trained using noisecontrastive estimation following [61]. See Table 5 (panel  2).
EgoTopo architecture details. EgoTopo [54] is an approach to translate egocentric video frames into a topological graph, where each node contains a list of clips that correspond to a physical location, and edges correspond to rough spatial layout.
For MP3D and HouseTours, ee directly use available pose information to determine whether two frames belong to the same zone or not (i.e., we do not train a retrieval network to approximate this). This amounts to a clustering of the trajectory based on pose (position and heading) and represents an enhanced version of EgoTopo that benefits from pose data. We use affinity propagation to cluster frames into nodes. To compute edges, we calculate the distance between the centroid of each node and assign an edge if this distance is < 3.0m to be consistent with Eqn. 2. For Ego4D, pose information is not available. We fall back to clustering visual frames based on ImageNet pre-trained ResNet-50 frame features.
Node features are calculated as the average of features assigned to that node. We use a 2-layer graph convolutional neural network (GCN) to aggregate features across nodes, and then average them to form a single video encoding following [54]. See Table 5 (panel 2).
MAE architecture details. We use all frames from our generated walkthroughs to train an MAE VIT-large model with 16 × 16 patch size. We use the authors existing code and train a model for 200 epochs.
ObjFeat architecture details. We train a QueryInst instance segmentation model [20] with a R-50-FPN backbone, using a dataset sampled from the 100 HM3D scenes with ground-truth semantic object annotations. We use the implementation in the MMDetection package.
Training details. As mentioned in Sec. 4.2 of the main paper, we train our models for 2500 epochs using the Adam optimizer with learning rate 1e − 4. We sample K = 64 frames to construct our memory. At training, we sample frames from the video but randomly offset frame indices to train robust models. During inference, we uniformly sam-

C.2. Architecture and training details (downstream)
We present additional details for the approaches in Sec. 3.3 of the main paper. The full list of hyperparameters are in Table 5 (panels 4-5).
Room prediction models. As mentioned in Sec. 4.5 of the main paper, for our PLACESCNN baseline model, we build a classifier on top of features from the wide ResNet-18 model from [91]. We use the authors existing code and their provided pretrained models to initialize the model. The classifier head is a 2-layer MLP with hidden dimension 512. The backbone is frozen and only the classifier is fine-tuned. N = 8 frames around the target frame are used to provide additional context. The features are max-pooled before classification.
For our models, we use the target frame as the query to produce a single environment-feature, which is then concatenated with all N = 8 frames and aggregated following Eqn. 8. This new, enhanced input is fed into the baseline model as described above.
We train all models for 80 epochs with learning rate 1e − 4 using the Adam optimizer. The full list of hyperparameters are in Table 5 (panel 4) Episodic memory retrieval models. As mentioned in Sec. 4.6 of the main paper, we build on the VSLNet model from [88]. We use an existing implementation based on the authors original code. Visual inputs are encoded as N = 128 clip features, created by adaptive average pooling of SlowFast-R50 clip features. Note that for MP3D we use ResNet-50 features as the walkthroughs contain discrete agent steps, rather than smooth video frames.
For our models, we select the center frame of each of the N = 128 inputs and use them as query frames to produce N = 128 environment features. Each pair of input feature and environment feature is aggregated following Eqn. 8, and then input to the VSLNet model described above. Note that we perform this aggregation after the video affine and feature encoding layers in VSLNet.
We train all models for 200 epochs with a learning rate of 1e − 4 using the Adam optimizer. The full list of hyperparameters are in Table 5 (panel 5).
For Ego4D experiments, we use the hyperparameters described in the respective benchmark whitepapers [26,47,51] and only aggregate precomputed environment features as described above.

D. Ablations and additional experiments
We present ablation experiments for various model design choices and additional experiments.

D.1. Additional experiments with pose embeddings and ground-truth pose
As noted in Sec. 3.2.1 ground truth pose may be utilized directly by our model, however it is not easily available in real-world egocentric video, which makes our use of inferred pose embeddings a strength.

D.1.1 Importance of pose embeddings
As noted in Sec. 3.2.1 of the main paper, we train pose embeddings to help relate observations by their visual content as well as relative orientation of capture. We measure the effect of pose embeddings on our local state prediction task. Our models see improvements in predicting objects in each of the cardinal directions (26.1 vs. 24.9 mAP) as well as improved object distance prediction (59% vs 58%). This improvement is small for predicting objects in the forward view (+0.7 mAP) where scene information is directly visible, but large for other views that need to be inferred: mAP improvements of +2.2 (right), +0.9 (behind) and +1.0 (left).

D.1.2 Importance of ground truth pose
In this section, we explore using pose estimates obtained directly from the simulator or using off-the-shelf structure for motion methods. Note that the existing method EPC in Fig. 6 and Table 1 already uses this ground-truth pose information, which other models (including ours) do not have access to.
Extracting pose information. For HouseTours videos, we run COLMAP [69], a structure from motion framework to compute camera-pose information associated with each frame of the video. For this, we first extract frames from each video at 2fps. Then we run COLMAP using a precomputed vocabulary tree file (flickr100K words32K). The resulting trajectories are inherently noisy due to the approximate nature of the SfM pipeline and the absence of true camera parameters for such in-the-wild video. We postprocess them by removing erroneous pose values (ones that cause jumps in pose atypical to smooth motion).
Overall, COLMAP successfully localizes 32 hours of video from 886 houses out of the original 119 hours available in [7]. Some examples of video trajectories are shown in Fig. 8. Comparing these to the simulated trajectories in Fig. 1, we see smoother trajectories overall, but with unrealistic jumps in localizations and loop-closure failures. Note that we visualize only the trajectory, not obstacles in the environment, as we do not have access to occupancy maps for the real-world videos.   Note that these videos are tours of indoor spaces with the intention of visually covering a large area. As a result, the camera-wearer moves slowly and smoothly to show parts of the house. Despite this, only a fraction of trajectories can be localized highlighting the difficulty of estimating pose from monocular video. In contrast, the same procedure fails to localize the camera-wearer in Ego4D videos due to rapid head motions and motion blur. On MP3D, pose is available directly from the simulator.
Performance with ground-truth pose. In Fig. 7, we investigate the role of pose information for ROOMPRED (top) and NLQ (bottom), by directly embedding groundtruth pose as part of the input to the baseline and our model. We find that our model can benefit from pose on MP3D, but falls short on HouseTours, due to noise in extracted pose (compared to simulator-provided pose in MP3D). However, our approach (with and without pose) outperforms EPC, which explicitly leverages pose both at train and test time.

D.2. Ego4D NLQ baseline comparisons.
We evaluate Ego4D NLQ performance against the remaining baselines from Sec. 4.3 for results analogous to comparisons in Table 1 (left) in the main paper. Our approach outperforms baselines that have access to all extra simulated training data.

D.3. Alternate pretraining task formulations.
In Sec. 3.2.2 in the main paper, we introduced our local state prediction task that is used to pretrain our video encoders on simulated walkthrough videos. We investigate alternate pretraining objectives to validate our task choice. We compare against the following:  Each of these objectives explicitly encodes a combination of semantic and geometric information. For example, PANOFEAT and PANOCONTRAST encodes primarily semantic information as they require reconstruction of image features. POSEEMBED encodes primarily geometric information to predict the relative pose between observation pairs. CARDINALOBJ encodes semantics from object categories and weak geometric information from their relative orientations. Table 7 highlights the performance of these variants on both downstream tasks. Our approach that requires predicting both object labels, orientations as well as rough distances offers a balance of both cues during pretraining, translating to strong downstream performance.

D.4. Ablation studies.
We perform ablation experiments for several model design choices listed in Sec. 4.2 of the main paper. All abla- tion experiments are performed on on validation data splits of MP3D and HouseTours.
Window size W . The window size controls the density of sampled frames for building our environment memory. Larger windows imply temporally separated frame inputs. Our results are in Fig. 10 (left column). We find that W = 64 is sufficient for localizing the room category for ROOMPRED, while a W = 256 is best for NLQ which requires reasoning over longer horizons.
Memory size K. The memory size controls the number of frames sampled from a window of size W for building our environment memory. Our results in Fig. 10 Table 7. Alternate pretraining tasks. Our local state prediction task offers a good balance of semantic and geometric cues that lead to features with strong downstream performance. For ROOMPRED (left) we report accuracy (%). For NLQ (right) we report mean Rank1@(0.3, 0.5). column) show the sensitivity of our model to this parameter. On both tasks and datasets, we see only marginal improvements with higher memory sizes, though K = 64 results in the best performance.
Loss weight term λ. λ controls the weighting between the object and distance prediction term in the local state prediction loss in Sec. 3.2.2. Our results in Fig. 10 (right column) show that λ = 0.1 results in the best balance between the two semantic and geometric terms.

D.5. Additional attention visualizations
We present additional examples visualizing the learned attention values in our transformer decoder model in Fig. 11 to supplement Fig. 5 of the main paper. Our model learns to attend to diverse views that are not simply based on temporal adjacency or visual overlap -they capture the surroundings of the camera-wearer. D.6. Easy vs. Hard instances in the ROOMPRED task As mentioned in Sec. 4.5 of the main paper, we sort instances for evaluation by difficulty based on the prediction entropy of a pre-trained scene classifier model. In Fig. 9 (top), we show the distribution of this entropy score across all three datasets. In general, Ego4D contains the hardest instances due to characteristic egocentric motion patterns, while HouseTours contains easier examples where the camera-wearer tends to dwell in one particular location to showcase it. We show examples of easy vs. hard instances in Fig. 9 (bottom). Note that the figure only shows the center frame of the clip that is used to predict the room label to highlight the difference between easy and hard frames. Our results in Fig. 6 highlight the advantage of our approach on these hard instances where environment-level reasoning is essential. See our video in Sec. A for more context.

D.7. Memory vs. anticipation during pretraining
As mentioned in Sec. 3.2 of the main paper, our pretraining task involves elements of both aggregating information about relevant views spread across the walkthrough, as well as anticipating objects that are rarely (or never seen). We quantify this statement in Fig. 12 where we show the percentage of training instances where objects are rarely seen, for different definitions of rarity. For example, 23% of training instances involve predicting objects that appear in only k < 4 frames. 5% of training instances involve anticipating completely unseen object instances (k < 1).

D.8. Task-specific pre-training in simulation
As mentioned in Sec. 3.3 of the main paper, our goal is to train task-agnostic representations using videos from simulated agents. The end result is a single model that can generate features for multiple tasks (in our experiments, ROOM-PRED and NLQ). This is different from traditional sim-toreal approaches where a new dataset needs to be collected for every downstream task, and a separate model has to be trained on it. Such a dataset needs to be well balanced and carefully designed to match the downstream task. Moreover, as tasks are added, new datasets per task need to be created which may be impractical, especially when they require data beyond the simulator's capability (e.g., simulating human motion, hand-object interaction).
To investigate further, for the ROOMPRED task, we generate a dataset in simulation that maps trajectory frames to room labels 2 . The room categories are estimated directly from the simulator by matching each frame to the nearest navigable point in an annotated room region. We train a ResNet18 model to predict the room category (a six-way classification) and then use features from this model following baselines in Sec. 4.3. The new baseline benefits from representations learned for the same task -room prediction -and on the same volume of simulated training data.
On MP3D this performs better than the PlacesCNN baseline (43.3 vs. 42.4%) but is weaker than our model (50.4%). On HouseTours, it performs worse than PlacesCNN (57.9 vs. 58.2%) and our approach (62.7%). The low performance may be attributed to the small label space (only six categories), resulting in features that are not discriminative for the large-scale, diverse data downstream. More generally, the task-specific approach is more susceptible to failures due to the sim-to-real gap, which manifests as a lower performance on real-world video frames from HouseTours.   Percentage of training instances that involve "rare" objects. The x-axis sets a threshold for what is considered rare. For example, 5% of training instances involve anticipating completely unseen object instances (k < 1).