Adaptive Workflow Scheduling to Increase Fault Tolerance in Cloud Computing

: Cloud computing in the field of high-performance distributed computing has emerged as a new development in which the demand for access to resources via the Internet is presented in distributed servers that dynamically scale are acceptable. One of the important research issues that must be considered to achieve efficient performance is fault tolerance. Fault tolerance is a way to find faults and failures in a system. Predicting and reducing errors play an important role in increasing the performance and popularity of cloud computing. In this study, an adaptive workflow scheduling approach is presented to increase fault tolerance in cloud computing. The present approach calculates the probability of failure for each resource according to the execution time of tasks on the resources. In the present method, a deadline is set for each of the tasks. If the task is not completed within the specified time, the probability of failure in the source increases and subsequent tasks are not sent to the desired source. The simulation results of the proposed method show that the proposed idea can work well on workflows and improve service quality factors.

INTRODUCTION
The advent of cloud computing is considered to be the biggest change in information technology. This change has attracted the attention of everyone, from people in the community to large corporations. Today, the popularity of cloud computing is so widely accepted that organizations are moving their traditional information processing systems to cloud services to store large amounts of data [1], [2].
Cloud computing has emerged as a demand-based computing services model for use by small-scale users and large-scale scientific and commercial applications. This model is defined as a model for accessing a network with a shared set of configurable computing resources (e.g., networks, servers, storage space, applications, and services) that can be quickly and efficiently defined. Minimize managerial effort or service provider interaction, access to demand, resource independence, fast flexibility and always availability are the basic features of cloud computing [3], [4].
Due to the popularity of cloud computing among users and service providers, this environment has been very popular in the last decade. The presence of users and service providers from different places and with heterogeneous systems in this medium have caused many challenges. In cloud computing, fault tolerance is an important problem and resource failure time is one of the metrics that affect performance, throughput, response time and system and network performance [5].
Load balance fault tolerance is one of the main challenges in cloud computing that is required to distribute network load evenly across all nodes. This load is the amount of work that a computing system does, which can be classified as network load, storage capacity, memory capacity, and CPU load [6].
Error tolerance is a way to find faults and failures in a system. If an error occurs or there is a hardware failure or software defect, the system should also work properly. Malfunctions must be controlled in a dynamic way for reliable cloud computing. It will also provide availability and reliability against hardware and software failures of the system in the organization [7], [8].
Predicting and reducing errors has two types of techniques: active and passive. Active fault tolerance 26 policy, avoiding fault recovery by anticipating and actively replacing a suspicious component to detect it before a fault occurs. Passive fault tolerance reduces the impact of failures on executable programs when effective failures occur [9]. Therefore, in this study, an adaptive workflow scheduling approach is proposed to increase fault tolerance in cloud computing. The present approach calculates the probability of failure for each resource according to the execution time of tasks on the resources. In fact, in the present method, a time limit is set for each of the tasks. If the task is not completed within the specified time limit, the probability of failure in the source increases and the next tasks are not sent to the desired source.
In the proposed method, the input flow is timed through a resource management unit. At this stage, according to the characteristics of the resources, the tasks are assigned to the nearest source in terms of service quality factors to be executed. The scheduling plan is then provided to a task management unit to set a time limit for each task, as mentioned, according to the number of instructions for each task, and to review the time frame for performing previous tasks in the resources. Slowly If a resource task is running for more than the specified time, the probability of resource failure increases.
Thus, the task management unit calculates a failure probability for each resource. Based on the probability of failure, the tasks are then sent to the resources, which are virtual machines, to complete the continuation of the task execution process. The main innovation in the present study is the addition of a task management unit which, according to the number of instructions for each task and the characteristics of the resources, assigns an adaptive time limit to each task to be performed in each resource. Then, according to the history of performing tasks in the resources, the probability of failure for each resource is calculated.

LITERATURE REVIEW
Fault tolerance provides complete and permanent operation even when incomplete components are present. It is the continuation of work satisfactorily in the face of obstacles, art and science that creates the computational system. The fault tolerance system does not allow one or more types of errors to occur, including momentary, definitive or permanent hardware or software failures, design errors, operational errors, or damage caused by the program itself or physical damage. In a real-time cloud application, these operations are performed remotely in the computational node and the probability of error occurs [10]. These events emphasize the need for fault tolerance technology to achieve the reliability of real-time computing in the cloud infrastructure.
Abu Hamama et al. have proposed a comprehensive framework that incorporates a number of error tolerance methods to improve the reliability of cloud environments for hosting applications in real time, if reliability is achieved and access conditions are provided. Then the real-time fault tolerance scheduling algorithm is proposed to minimize the number of missing deadlines, and the amount of load imbalance [11].
Yan et al. presented the uncertainty model by estimating task execution time and a contrast-task assignment mechanism that strategically uses two errortolerant task scheduling models, while considering uncertainty. In addition, an overlap mechanism has been used to improve the use of cloud resources. Based on two mechanisms, a dynamic innovation-based scheduling and scheduling algorithm based on Dynamic Innovation (DEFT) is proposed for scheduling real-time tasks in the cloud in which system performance fluctuations must be considered. The purpose of DEFT is to achieve both fault tolerance and resource utilization [12].
Ding et al. presented a flexible common error scheduling algorithm for cloud workflow (FTESW). After analyzing the initial scheduling constraints, backup in cloud systems due to interdependence between tasks in the workflow is presented, a tensile mechanism in the field of fault tolerance is designed to dynamically adjust resources based on resource requests by adopting resource migration technology, FTESW is then proposed to achieve both fault tolerance and full resource utilization for workflow in cloud systems. To evaluate the effectiveness of the proposed FTESW, a series of simulation experiments were performed on both randomly generated workflows and real-world workflows [13].
Talvani et al. have effectively evaluated and processed barriers to gain robustness and reliability in cloud computing. Various error detection methods and architectural models have been proposed to increase cloud fault tolerance. Given this fact, this article focuses on solving the problem of error in cloud computing. This research, in order to identify the work that has been done in this field, shows the basic concept of fault tolerance that has been done by different researchers and different algorithms to solve the problem of fault tolerance in cloud computing [14].

METHODOLOGY
As mentioned, in this study, in order to present the error tolerance approach in cloud infrastructure, an approach of workflow distribution and adaptive scheduling based on task execution time in resources has been used, which will be described in detail in the following method.

Workflow
As cloud end users use host applications, the workload includes tasks that dynamically go to the central scheduler. Each task contains a set of tasks that is a set of = { 1 , 2 , … , } with tasks. Each of the tasks has a time limit or deadline to be performed, which in case of exceeding this deadline, the performance of the task and its related tasks will be adversely affected.
In this research, we use the non-distance directional graph (DAG) to show the workflow provided by the customer to cloud systems with software deadline. A = ( , ) contains tasks that are connected through control and data flow.
Where, represents the amount of data to be transferred from to , and represents the control relationship, such as the loop and the selected structure, between and . The size of each task is measured by millions of instructions that affect its execution time. We also use ( ) and ( ) to indicate the previous and next set of tasks of task , which means that ( ) must be completed before starting . An example of a workflow is shown in Fig.  1. For each task sent in the workflow, two versions are provided as and for the initial version and the backup, respectively. They run on different hosts for fault tolerance.
( ) and ( ) are used to represent virtual machines containing and , while ℎ( ) and ℎ( ) are hosts, respectively. The task execution time in the virtual machine is calculated as ( ) as the ratio of the task size to the processing power of the virtual machine. For example, ( ) = , ( ) = means that the initial version of is scheduled in the , virtual machine hosted on ℎ , and has an execution time of ( ) of . In addition, the time used to transfer can be stated as follows:

Fault Model
Tasks will fail and stop running when an error occurs in the processor unit in which they are located. In this study, we focus on host failure. A host fails when all the virtual machines in that host can no longer work and cannot complete the copies of the tasks in the virtual machines. Errors can be temporary or permanent, and it is assumed that the error is in one independent host and affects only one host. The maximum number of hosts that are expected to fail at any given time is assumed to be one. This means that if the original version of a task fails, the backup can always be completed before another host fails. For each task, backup is done after scheduling the initial version. However, it is not necessary to back up all tasks in a workflow after scheduling initial versions of all tasks in the same workflow. It is also assumed that the minimum value required for the average error time (MTTF) is always greater than or equal to the maximum execution time of a virtual machine from the same host.
There is an error detection mechanism such as an error signal and an acceptance test to detect errors. If an error is detected in an initial version, the backup will be executed and new tasks will not be sent to the host that received the error. If the completion of the initial version is successful, the resources will be restored by deleting the backup. Restoring resources is essential so that backup time can be allocated to new tasks.

Predict Fault
Predicting an error in a host can speed up the completion of tasks and prevent new tasks from being sent to hosts suspected of error. In this module, error prediction is based on the average time of error, which was introduced in the previous section. As mentioned, the minimum value required for the average error time is always greater than or equal to the maximum execution time of a virtual machine from the same host. Given that the virtual machines are heterogeneously distributed in one host, it can be said that the minimum value required for the average error time is always greater than or equal to the maximum execution time in the slowest virtual machine of the same host. By assigning a task to a virtual machine from a host, the average amount of error time is determined based on the number of instructions in each task and the processing power of the virtual machines in the host, based on Equation 3 for each task.
Subject to 11 ≤ 12 ≤ 13 , … ≤ ≥ 0 The scheduler includes a workflow analyzer, a prototype controller, a backup controller, a resource manager, a task manager, and an error forecasting module. When a workflow is entered, the scheduler determines whether it is first accepted in collaboration with the workflow analyst and resource manager. For the accepted workflow, the workflow analyst analyzes the relationship and prioritizes scheduling among tasks. A backup is then made by the backup controller. The faulttolerant scheduler is then connected by the main controller and the backup controller. If active resources fail to meet the schedule request, the resource manager switches some hosts from sleep to active to increase active resources. In addition, the host reports information about its status, including the status of scheduled tasks, the next access time to each active virtual machine for other tasks, and the resource usage of each active host, directly to the scheduler, and the resource manager and records the status of all hosts in cloud systems. If the initial version of a task is successful, the initial version success information is sent to the resource manager. The resource manager then notifies the virtual machine to which it has sent the backup of the assigned task to cancel the backup. Otherwise, the resource manager will not notify the virtual machine to cancel the backup, and scheduling backups will normally be performed for fault tolerance. In addition, if the host is in low workflow mode and some virtual machines remain idle for a long time, the resource manager decides that some virtual machines should be used to improve resource utilization and be deleted or transferred to other hosts. Since cloud systems resources are shared among all customers and workflows are dynamically sent to cloud systems at runtime, the above process for each stream something is repeated.
Each workflow is given to the task manager as a set of tasks, so that the task manager can create backups and backups of each task and provide their information to the resource manager for allocation to the machine and sends virtual error prediction modules to calculate the average error value. The initial version of each task is assigned to one of the virtual machines in the host, followed by the task manager backing up the task. The task backup is also sent by the resource manager to another active virtual machine on the other host. When the initial version of a task starts working in a virtual machine, time is allotted for that task. If the execution of the initial version of the task in the virtual machine is completed successfully by the time of the average error, the error forecasting module sends the command to cancel the execution of the backup to the task manager.
The Task Manager also records the success information of the original version on the target host and cancels the backup. The task manager also sends information about the success and cancellation of the backup to the resource manager to prevent the backup from running on another virtual machine on another host.
If the execution of the initial version of the task in the virtual machine is not completed successfully by the time of the average error, the error forecasting module sends the command to stop the execution of the initial version in the current virtual machine and transfer it to another virtual machine in the same host. Slowly The error prediction module waits until the time limit set for task i expires. If the execution of the initial version of the task in the current host virtual machines is completed by the end of the deadline for task i, the error prediction module sends the command to cancel the backup to the task manager. Otherwise, the error forecasting module sends a backup command to the task manager by imposing a penalty on the present host. Due to the penalty imposed on the current host, new tasks will not be transferred to this host and the current host, despite the fact that there may be active virtual machines in that host, in the queue of hosts that if they make a mistake, they will be placed. Assignment of tasks to these hosts is delayed until the completion of tasks related to the current workflow and the arrival of a new workflow.

RESULTS AND DISCUSSION
In order to implement the proposed method, according to the previous chapters, in this dissertation, 20 tasks have been used in the form of a workflow. In order to assign these tasks to run in the cloud, we used 5 hosts with 15 virtual machines. The method is simulated in MATLAB software version 2015. Tasks that reach the cloud environment are workflows in which tasks can communicate with each other. Therefore, due to the relationship between tasks, the discussion of data transfer and priority in execution occurs, which is one of the main discussions in the cloud environment in order to serve the tasks. As mentioned, the workflow includes a number of tasks in line with a specific goal, which, based on the priorities between the tasks, the order of execution of these tasks, has a decisive role in the implementation of the entire workflow. Fig. 2 shows an example of the workflow sent to the cloud environment.

Fig. 2. Workflow.
As shown in Fig. 2, the current workflow has 20 tasks, some of which are related to other tasks. The margins between tasks are directional, indicating the priority and lag of tasks in estimating the overall goal of the workflow.
Due to the fact that the proposed method aims to increase the fault tolerance in the cloud environment, it tries to optimize the main factors in the quality of service, so first prepare and maintain a copy of the received tasks as a backup. Tasks that reach the cloud are identified based on the number of instructions in each task, which is the instruction measurement scale of instructions, million instructions per second (MIPS). In addition, each resource has processing power that is released from service providers. Therefore, the task execution time in the virtual machine can be calculated as the ratio of the task size to the processing power of the virtual machine. In addition, the time used to transfer data while performing tasks between two virtual machines can be calculated using Equation 2. Therefore, Table 1 shows the specifications of the original and backup versions, the time required to perform the task and the time required to perform the previous task, if any, the time required to transfer data between virtual machines based on the amount of data. As shown in Table 1, the execution time of the related tasks is calculated according to the items in the workflow.

Implement Adaptive Fault Tolerance Scheduling
As mentioned in the previous chapter, in the proposed method, when a workflow is entered, it first approves or rejects the task acceptance scheduler according to the number of virtual machines and the connections between the tasks. After confirming the acceptance of the workflow, a backup of the tasks is prepared by the task controller. The scheduler sends each version to a virtual machine to run. If the initial version of a task is successful, the initial version success information is sent to the resource manager. The resource manager then notifies the virtual machine to which it has sent the backup of the assigned task to cancel the backup. Otherwise, the resource manager will not notify the virtual machine to cancel the backup, and scheduling backups will normally be performed for fault tolerance. Since cloud systems resources are shared among all customers and workflows are dynamically sent to cloud systems at runtime, the above process for each stream something is repeated. In the continuation of this chapter, according to the implementation of the proposed method, the main factors that affect the quality of service are calculated in order to evaluate the performance of the proposed idea.

Implementation of Fault Tolerance
In the proposed method as mentioned in the previous chapter, the proposed method of predicting an error in a host can expedite the completion of tasks and not send new tasks to hosts suspected of error. In this module, error prediction is based on the average time of error, which was introduced in the previous section. As mentioned, the minimum value required for the average error time is always greater than or equal to the maximum execution time of a virtual machine from the same host. Given that virtual machines are heterogeneously distributed across a host, it can be said that the minimum value required for the mean error time is always greater than or equal to the maximum execution time of the slowest virtual machine from the same host. In order to implement the error according to the optimization model specified in Equation 3, the fault tolerance rate and error prediction in cloud resources are shown in Table 2.
As shown in Fig. 3, the error tolerance rate increases at a slower rate due to the assignment of tasks and the use of backups.

Implement the Cost in the Proposed Method
According to the proposed method, the cost of completing tasks in resources consists of costs related to processing and costs related to the transfer of information in the network between resources while performing related tasks. Therefore, it can be noted that the processing cost for a given task depends on the task volume and the fixed cost (processor price per instruction). Also, in order to calculate the cost of data transfer in the cloud network between resources, the relationship between tasks can be used to calculate the volume of information sent in the network and the cost of this information. Table 3 shows the costs associated with completing tasks in each resource in the cloud environment.  Table 3, the cost of performing tasks in the resources to which the independent tasks are assigned is less than the resources in the dependent resources. Resources on which the cost of performing tasks is zero are resources to which no task has been submitted; these resources are the same resources that have been penalized by the proposed method and will not be assigned a task until the end of the workflow. Fig.  4 shows the cost-completion task chart in the resources. Also, in order to show the total costs in the resources to perform the tasks in the proposed method, the cumulative cost function is used, the diagram of the cumulative cost function is shown in Fig. 5. As shown in Fig. 5, the cost of completing tasks in all sources has increased slightly, and finally for 20 virtual machines in 4 hosts, this cost has reached 1.25.

Implement Energy in Cloud Resources
As mentioned in the previous chapter, due to the fact that virtual machines are heterogeneous in the proposed method, the energy required to perform tasks varies from source to source. Therefore, in each fixed source, energy consumption will be different from other sources. Fig. 6 shows the energy consumption diagram in the proposed method. As shown in Fig. 6, resources where independent tasks are loaded consume less energy, while energy consumption in resources with dependent tasks is higher.

Completion of Tasks in the Proposed Method
In order to implement the task execution time in the proposed method, the total task completion time in the cloud environment includes the task execution time and the transmission time in the network. Therefore, the time of completion of tasks in the cloud environment is directly related to independent and dependent tasks. In other words, resources that need to exchange information with other sources need more time to complete tasks. Naturally, there will be more delays in completing tasks in such resources. Fig. 7 shows the completion time chart in each resource. Fig. 8 also shows the delay diagram in the proposed method.  Also, in order to calculate the completion time of all tasks in all resources, the cumulative time function is used, which summarizes the time spent in each resource and shows it graphically during the completion of tasks in the resources. Fig. 9 shows the cumulative function diagram of task completion time across resources. As shown in Fig. 9, the task completion time, which increases due to the availability of resources with dependent tasks, eventually reaches 9.5 seconds for a 30task workflow at 20 sources in 4 hosts.

Comparing the Proposed Method with the Previous Work
In order to compare the proposed method with the previous method, in this section we compare the proposed method with other scheduling methods in the basic article. This comparison is first based on the criterion (HAT). This criterion indicates the number of active hosts when assigning tasks to virtual machines. Fig. 10 shows a comparison between the proposed method and the previous methods in terms of HAT criteria.  HAT(104s)

Workflow number
As shown in Fig. 10, the number of active hosts in the proposed method is less than the other methods in the basic article, and the amount of energy consumed will be less according to the proposed method.
Another criterion used in this dissertation for comparison is the criterion (CCR), which is the average cost of transferring tasks over the cost of performing tasks, which is compared in Fig. 11 between the proposed method and previous methods in terms of The CCR criterion. Fig. 11. Comparison of the proposed method with previous methods in terms of CCR criteria.
As shown in Fig. 11, the transfer cost per implementation of the proposed method is lower than other methods in the base article. Therefore, it can be said that the performance of the proposed method has been better in allocating tasks to resources and reducing the time and cost of completing the entire work and error tolerance.

CONCLUSION
In the cloud environment, computing resources need to be timed so that both providers get the most out of their resources and users get the applications they need at the least time and cost. On the other hand, given that the cloud environment is heterogeneous and needs and demands are changing at all times, a good decision to schedule the workflow in the cloud environment is complex and vital at the same time. Therefore, in this study, an adaptive workflow scheduling approach is proposed to increase fault tolerance in cloud computing. The present approach calculates the probability of failures for each resource according to the execution time of tasks on the resources. In fact, in the present method, a time limit is set for each of the tasks. If the task is not completed within the specified time limit, the probability of failure in the source increases and the next tasks are not sent to the desired source. The simulation results show that the proposed method, in addition to increasing error tolerance, also improves other factors affecting service quality.


HDR Environment Map Estimation for Real-Time Augmented Reality

We present a method to estimate an HDR environment map from a narrow field-of-view LDR camera image in real-time. This enables perceptually appealing reflections and shading on virtual objects of any material finish, from mirror to diffuse, rendered into a real physical environment using augmented reality. Our method is based on our efficient convolutional neural network architecture, EnvMapNet, trained end-to-end with two novel losses, ProjectionLoss for the generated image, and ClusterLoss for adversarial training. Through qualitative and quantitative comparison to state-of-the-art methods, we demonstrate that our algorithm reduces the directional error of estimated light sources by more than 50%, and achieves 3.7 times lower Frechet Inception Distance (FID). We further showcase a mobile application that is able to run our neural network model in under 9 ms on an iPhone XS, and render in real-time, visually coherent virtual objects in previously unseen real-world environments.

Introduction
In this work, we discuss video see-through augmented reality (AR) applications, in which virtual objects are superimposed on camera frames of the real environment shown on an opaque display, e.g. on a phone as shown in Fig. 1(e). Creating immersive and believable AR experiences involves many aspects of computer vision and graphics. One of the requirements is visual coherence: the problem of matching visual appearance of rendered objects to their real-world background, such that virtual and real objects become indistinguishable in the composited video. Accomplishing this involves matching various scene and camera properties, such as lighting, geometry, and sensor noise. This paper focusses on creating reflections and lighting for virtual objects by estimating an omnidirectional HDR environment map. To support rendering objects with a variety of geometry, material properties, and dimensions, the environment map must be high dynamic range, and have sufficient image resolution to represent objects and features in the scene. We use the equirectangular projection and RGB color space for the environment maps. As shown in Fig. 1(a), the challenge in mobile AR is limited camera field of view (FoV) and motion by the user, hence an application is usually able to accumulate less than 100 degrees effective FoV. A virtual object placed in front of the user, however, is expected to reflect what is behind the camera, and parts which are not present in the captured frames. The problem is thus to estimate, given this incomplete environment map, a plausible estimation for rest of the scene and its lighting. We show that our method is not only able to estimate the light information, but also to synthesize a high resolution completed scene. For instance in the scene shown in Fig. 1(b), the estimated environment map is high resolution, continuous, and a plausible extrapolation of the input. The synthesized parts not only match low frequency information (ambient light temperature and intensity), but also finer details such as the type of light sources (in this case, ceiling area lights). As detailed in Sec. 3, we achieve this context-aware scene completion using the framework of generative adversarial networks (GANs) [12] along with novel loss functions, ProjectionLoss and ClusterLoss, designed for accurate light estimation.
In mobile AR frameworks [1,2], we can obtain camera frames, poses and scene geometry, around the 3D location where the virtual object is to be placed. This allows a realtime renderer to create light probes [27] at the 3D location. RGB texture information from the frames can be rendered into an equirectangular image at these probe locations. In this work, we focus on processing the partial environment map at these probes. Our method takes as input a partial environment map that is composed from one or more low dynamic range (LDR) camera frames (8 bit per channel), and outputs a completed environment map that is higher dynamic range (HDR, 16 bit channel). Thus we perform both lifting of input pixels from LDR to HDR, as well as spatial HDR image extrapolation. The output environment map retains the color and details from pixels that were in the input, while filling the unknown pixels with plausible content that is coherent with the known. That is, we want the completed environment map to represent the textures from a plausible real scene. Through detailed quantitative and  Figure 1. Given a partial LDR environment map from a camera image (a), we estimate a visually coherent and completed HDR map that can be used by graphics engines to create light probes. These enable rendering virtual objects with any material finish into the real environment in a perceptually pleasing and realistic way (b), more similar to ground truth (c) than state-of-the-art methods (d). Our mobile app renders a virtual teapot using our estimated environment map in real-time (e). See supplementary material for videos.
qualitative comparisons, we demonstrate that our method surpasses the current state-of-the art to estimate high quality, perceptually plausible, and accurate HDR environment maps. We reduce the directional (angular) error for lights by more than half, and achieve a significantly lower Frechet Inception Distance (FID). Related works estimate a subset of the properties, such as lighting direction, color, and intensity using low resolution images [11,9,17], parametric lights [15,9], or LDR environment maps [25]. Methods that estimate both light and texture [10,17,23] lack resolution and detail in the estimated environment map. In contrast, our solution provides sufficient details in the image for virtual objects of any material finish including reflective mirrors. To create plausible scene completions, we take inspiration from work in the area GANs for image synthesis. However, most success for GANs has come for datasets with single objects (e.g. faces [16]), using conditional labels [21], or selfsupervision [5]. Its direct use for panoramic images of indoor scenes, small datasets with high appearance variation, and ambiguity in semantic labeling, is challenging. We present ClusterLoss, a novel training loss for the discriminator, that allows us to create realistic images with a dataset of only ∼2,800 training images.
We also observe that each of the previous methods have a different evaluation scheme, are based on respective representations and/or involve subjective user studies. Though subjective plausibility is important for user experience, it makes benchmarking difficult. We present metrics that can be used to quantitatively compare both the lighting and reflection quality of environment maps.
In summary, we make the following core contributions: • We present a method to generate an HDR environment map suitable for both reflections and lighting from a small FoV LDR image or partial environment map. To our knowledge, ours is the first to achieve this for realtime AR applications (under 9ms on iPhone XS). • We present two novel contributions to the training pipeline in the form of ProjectionLoss for the environment map, and ClusterLoss in the adversarial training. This allows us to reduce the directional error by more than 50% compared to current methods, while achieving 3.7 times lower FID. • We establish metrics that allow easy quantitative comparisons with related work, and thus provide a systematic benchmark for this emerging area in AR.

Related work
A classical technique to obtain an HDR environment map is to merge images of a mirror sphere in the scene captured under multiple exposure brackets [7]. This can be applied for offline use cases but is unsuitable for realtime mobile AR in arbitrary and novel environments. Some previous works have used additional cues about an object [20,26], scene geometry [3,19,30], or special cases such as sun position estimation [31,15,14]. For brevity, we only discuss works that focus on light estimation from small FoV images of general scenes. In this context, works can be divided into two categories: those that focus on light estimation with low dimensional parameters, and those which estimate both lighting and environment maps.
Garon et al. [11] use spherical harmonics representation for light and depth estimated using the SUNCG dataset [24] 1 . Cheng et al. [6] also predict 48 spherical harmonics coefficients from two images captured by the front and rear cameras of a mobile device. Gardner et al. use parametric lights as representation [9]. The parameters are derived using peak finding, region growing, and ellipse fitting on intensity images. Even though use of parametric lights reduces the decoder size, the authors use L2 loss by converting the parameters to an equirectangular image. It is not clear how to extend this method to generate higher quality texture in the environment map that would be consistent with the regressed parameters. We thus choose to have an end-to-end network that directly estimates the HDR environment map. We use the same parametric lights to create quantitative metrics for benchmarking.
Gardner et al. use equirectangular representation [10] by dividing the task into light position estimation (trained with LDR panoramas), and HDR intensity estimation (trained with Laval HDR Images dataset). This is the work most similar to our method in input-output, representation, and formulation. In contrast to their method, we have a single stage training from an LDR small FoV image to a completed HDR environment map, and using our proposed adversarial training we are able to generate more realistic RGB scene completion for use on reflective virtual objects. Some recent works, like ours, employ an adversarial loss to generate a completed environment map [23,25] or sphere images [17]. LeGendre et al. captured a special dataset with three spheres of mirror, matte, and diffuse gray finish [17]. They train a network to regress from a camera image to three small (32 × 32 pixel) images representing the sphere segments. This precludes use of the results for high quality reflections. Also, due to the low resolution, the estimate covers mostly the portion of the scene behind the camera. It is useful to render a given frame, but as the camera commonly moves with respect to the virtual object in a AR application, a new estimate has to be inferred frequently. Thus even with the assumption of a static scene, their method would suffer from temporal flickering due to multiple independent estimates. Song et al. use a multi-stage ensemble that estimates geometry, LDR completion and HDR illumination [23]. Our method uses a single model, and we do not require depth map per HDR image for training. This makes our method efficient to train and use in real-time mobile AR. The authors do not specify the resolution of the panoramic images, but visual observation shows lack of resolution and artifacts due to the projection from noisy 3D reconstructions. Srinivasan et al. use an input stereo pair to generate the output environment map [25]. They use the LDR images from the synthetically generated InteriorNet dataset [18] to train the 1 SUNCG is currently withdrawn from distribution light estimation, by application of inverse gamma on the tone mapped images. This limits the ability to learn realistic high dynamic range and accurate lighting, as also indicated by their results on mostly specular objects. In contrast, our method does not require stereo input, and estimates an HDR environment map that can be applied for lighting objects with wide range of materials from diffuse to mirror.
Furthermore, we employ only 2,100 HDR images of the publicly available Laval HDR Images dataset of real environments, a much smaller dataset than the 4.06 million non-public specialized sphere images used in [17], the withdrawn SUNCG used in [24], or the 200,000+ Matterport images in [23]. The lack of shared code, model weights and use of private datasets make it difficult to make comparisons to these recent methods, that also compare to [10] as we do in our quantitative evaluations. However, we present qualitative comparisons to these works in Sec. 5.
With the exception of [17], that outputs a 32 × 32 low resolution sphere image, no other method has been demonstrated on mobile devices. To the best of our knowledge, we are the first to demonstrate real-time on-device generation of environment maps of high resolution and image quality.

Proposed method
Creating a light probe in a mobile AR application involves two broad stages: first to select a 3D point as center of this probe and project known scene information to an equirectangular environment map with the selected point as camera center; and second to process the partial equirectangular to output a completed HDR environment map. The first requires color and scene geometry knowledge or assumption for projection. In our work, we assume without loss of generality that platform and application-dependent processing can be used for this projection, and obtain the incomplete environment map from the probe center. Using the mobile device pose also helps to ensure that the environment maps are upright or gravity aligned. That is, the floor always appears at the bottom and the ceiling on top. Our focus in this paper is on completion, and LDR to HDR lifting of this incomplete environment map as shown in Fig. 2.
We use the equirectangular representation (128×256 pixels) and create a four channel input (RGB-mask). Known pixel intensities are normalized to [−., .], while the unknown pixels are populated with random noise from a uniform distribution U (−., .). The fourth channel is a binary mask with known pixels set to . This is input into our network, EnvMapNet, that outputs an HDR RGB image in log scale. The log image is converted to linear values, optionally decomposed into analytical lights, and provided to the renderer. We train our network end-to-end with imagebased and adversarial losses, which allows us to handle both the generative aspect for reflection completion as well as light estimation from partial environment map.  Figure 2. Overview of our method: We propose EnvMapNet that estimates completed HDR environment maps from partial input. We train the network end-to-end in an adversarial setup. An offline one-time clustering of the training images is used to provide a supervised classification task for our novel ClusterLoss in the discriminator. The additional adversarial loss, along with our proposed ProjectionLoss, allows our method to generate high quality environment maps for reflection and accurate shading of virtual objects.

Dataset and processing
In this work we use two datasets: Laval Indoor HDR dataset [10] and PanoContext LDR panoramas [32]. We use the author's test split [10] 2 for HDR images, resulting in a total of 2,810 training images. The different sources and scenes have large intensity variations and hence are unsuitable for training as-is. We thus employ a log scale on exposure-normalized HDR linear images for the network. The LDR PanoContext images are only used to train the discriminator. For a given linear HDR ground truth image G lin , we compute the training ground truth G as: We empirically choose α = 0.2 · G lin to match the middle gray values. The clipping of intensities corresponds to our use of tanh activation for our network, and results in dynamic range of [0, 100] for output linear RGB, given the input LDR images in [0, 1] range.

Model architecture
Our model, EnvMapNet, consists of an encoder and decoder with skip connections as shown in Fig. 2. Each is composed of building blocks detailed in Appendix 3. The encoder is composed of five sets of EnvMapNet-conv-block and EnvMapNet-downsample-blocks. The resulting latent vector is convolved with a 1×1 kernel to output 64 filters. The decoder mirrors the encoder by using EnvMapNetconv-block and EnvMapNet-upsample-blocks. The final output is produced by a 3×3 convolution to produce 3 channels for RGB, followed by a tanh activation.

Image-based losses and ProjectionLoss
We use a weighted combination of image-based and adversarial losses to train our model end-to-end. We want to 2 http://vision.gel.ulaval.ca/ jflalonde/projects/deepIndoorLight/test.txt retain the color for pixels from the known (input) region while hallucinating the rest with a plausible scene completion (typically extrapolation). Guided by the binary mask in the input, we compute an L1 loss between the known pixels in the input and corresponding predicted colors. For the completed output, we apply a multi-scale L2 loss. This allows the model to coarsely regress the light direction and color, however it does not allow for generation of sharp features for the estimated light sources (as seen in Fig. 1(b) and Fig. 2). To obtain a high contrast HDR result that generates correct shadows L2 loss is not sufficient. An ideal solution would be to use a ray tracer to render shadows and penalize the difference between the rendered images using ground truth and predicted maps. This is non-trivial and expensive for end-to-end training.
Considering only shadow casting, the intensity of a pixel in the shadow is dependent on the integral of the environment map except directions blocked by the object. Thus, any pixel on the shadow plane can be approximated by the integral over the masked environment map. We take inspiration from our intuition above, and Wasserstein distance, and introduce our novel ProjectionLoss. We select a set of randomized binary masks P having the same size as our environment map, and create a one-dimensional vector of length |P | (the number of images in the set) by integrating the pixel intensities in corresponding masked images. On a 0-valued background, the masks contain polygons generated with randomization whose height and width range from 10% to 40% of the corresponding image dimensions and filled with value of 1. For further illustration of Projection-Loss, and examples of the masks used, see Appendix 4 in supplementary materials. The final loss, termed Projection-Loss, is the L1 distance between the vectors corresponding to predicted and ground truth environment maps. We show this error using |P | = 50 projection masks in Fig. 3  image being the reference. In Fig. 3(d) we show rendered spheres using the corresponding environment maps. We can see that lower values of ProjectionLoss correlate with both lower angular error and better visual match of lighting direction to the reference. This is also supported in our ablation study results in Sec. 5.2. For a ground truth HDR image G, a mask indicating unknown pixels M , a predicted image I, and a set of projection masks P , the final image-based loss is defined as: ProjectionLoss is related to diffuse convolution or cosine loss used in previous works [10,23]. While filtered, or down sampled integral can represent diffuse or low frequency lighting information, we find that our formulation is able to capture high frequency lighting better. To further understand the value of ProjectionLoss for lighting estimation, and to compare with other measures, such as SSIM [33] and Mean Squared Error (MSE), we performed a detailed experiment with user study as described in Appendix 4. We first confirmed that SSIM on rendered images is a good baseline for retrieval of images with similar lighting on objects. We then correlated retrieval of similar environment maps using ProjectionLoss, SSIM and MSE. Quantitatively, we found the intersection of top-5 retrievals by SSIM (on the rendered images) and those using ProjectionLoss (on the environment map) to be 1.6 ± 0.7, while it was 0.6 ± 0.5 using MSE (on the environment map). Based on the above we believe that our proposed ProjectionLoss effectively trains the model for light estimation, such that the end result for rendering is accurate with respect to ground truth. We also show that MSE on the environment map is insufficient for training accurate light estimation. Its use as a metric of comparison, as done in previous works, would not correlate well with the final application.

Adversarial loss and ClusterLoss
To generate perceptually pleasing completions to the scene, we train the model using a GAN loss. Adversarial training was proposed in [12], where a discriminator was trained to classify samples from the training set as "real" and those produced by a generator as "fake". The generator has a loss component for classification of generated samples as "real". The networks are trained by alternating the update of their weights. Since their advent, there have been various improvements on GANs for training stability and result quality. As discussed previously, many of these have focused on single objects such as faces [16] or increased stability using extremely large datasets or architectures [4]. However, we found that training a mobile-friendly GAN architecture with fewer than 3, 000 images with high variability in content resulted in mode collapse or non convergence to plausible images.
It has been shown that providing an additional task to the discriminator using conditional labels or self-supervision can increase stability [21]. Assigning semantic labels (e.g, room types) is non-trivial for panoramic images since the viewpoint could be from between two rooms. Recent work using self-supervision with rotation prediction [5] has been promising. However, for equirectangular images the possible rotations would be flips along the image axis, of which that along the vertical image axis is valid for a scene.
We thus propose a novel scheme combining the learnings from above methods. We begin with the recognition that our application needs to generate "plausibly similar looking images" as the training set, hence the idea to use appearance features to form a secondary task. For the appearance features we extract equal sized patches on a grid from each training image, and calculate the ORB [22] descriptor and mean color for each patch. Using them as features we assign a K-means-derived cluster ID to each image (we used K = 5). Along with the typical real vs. fake classification, the discriminator is supervised to classify the K-means assigned cluster ID for each real image with the proposed ClusterLoss. This allows us to train without additional heuristics or tricks, and avoid mode collapse, as the encoded latents are forced to spread out over the appearance clusters. Intuitively it also helps the discriminator focus on a low ambiguity appearance-based task that pays attention to spatial locations and details, thus avoiding cases where early on in the training process a single patch artifact can easily clue the discriminator that the image is fake ("easy wins"). The discriminator is composed of residual blocks as detailed in Appendix 3. We use softmax cross entropy loss for the adversarial training of our model and discriminator D. With y, p indicating the ground truth and predicted one-hot encoded vectors for the cluster classification respectively, our adversarial losses are:

Implementation and adversarial training
During training we mask the ground truth images using randomly generated polygonal regions, and provide them as corresponding incomplete input environment map. The regions are of irregular shape and can be non-contiguous to generalize our model to complete any partial input. For input, we tonemap the ground truth HDR environment map by dividing the pixels by the average intensity (exposure compensation), applying a gamma with value of 2.
We train the networks by minimizing Loss EnvM apN et and Loss discriminator respectively. We use the Adam optimizer, a batch size of 16, and the learning rate starting at 0.0002, and decayed by half every 50 epochs after the first 100 epochs. We alternate between training the discriminator and EnvMapNet after each mini-batch.

Metrics for benchmarking
Visual coherence in AR aims to give a user the illusion that an inserted virtual object belongs in the real scene.
However algorithm development, especially in the domain of deep learning, involves many heuristics and parameter tuning. We therefore believe that establishing quantitative metrics, separate from loss functions, that correlate with the target application setting, is important to easily compare different methods. Several previous methods have used MSE on environment maps in RGB, intensity or log formulation as a metric. However, as discussed in Sec. 3.3, it is not well suited to measure accuracy such that the final rendering matches ground truth.
At the core of it we want to solve for two aspects of the environment map: lighting accuracy and perceptual plausibility. Real-time mobile renderers use analytical (point) lights to cast shadows. Hence lighting accuracy is correlated to direction of top-n dominant light sources. Since we cannot (and should not) generate the exact unseen test room for reflection, measures such as MSE or SSIM [33,29] for one-to-one image comparisons are unsuitable. Instead we need to use plausibility of the hallucinated scene as a metric. We start with a discussion on lighting accuracy metrics followed by quantitative measures for perceptual quality. Our goal is to define metrics that can be used by any future work to objectively compare techniques. Thus we establish a repeatable benchmark that uses the publicly available Laval HDR dataset [10], and reference implementations of the metrics that we will provide.
Methods that can extract light sources from environment maps include median cut [8], variance minimization [28], and parametric fitting from peak finding [9]. In the case of mobile AR, real-time renderers need fast decomposition techniques, hence we use the latter method to find centers and extent of light sources. The angular coordinates of the ellipse center are used to measure the directional accuracy with respect to ground truth. For a given HDR environment map, we iteratively find a seed pixel with maximum intensity, and grow the region connected to the seed till we reach 30% of the peak value. This is repeated until the peak value found is less than 90% of the largest. For each such region, we fit an ellipse covering its convex hull. Example images with parametric light fitting are shown in Fig. 3(c).
To measure the AngularError between the ground truth and estimated environment maps, we extract (at most) five parametric lights from each. For each ground truth light we find the minimal angular error to the predicted light set, and vice-versa. We take the mean over the errors from both ground truth lights and predicted lights, as the final Angu-larError between the two environment maps. Fig. 3 demonstrates the correlation of the AngularError metric, with examples of low and high errors with respect to a selected reference. We can see that environment maps with lower errors produce similar lighting directions on the reference sphere, compared to those with higher errors, thus making it a suitable metric for comparison. To measure perceptual plausibility, user studies are commonly used for their insight into final experience quality. However, user studies are hard to repeat or compare across publications. We take inspiration from work in GANs, that measure quality of synthesized images using Frechet Inception Distance (FID) [13]. The measure is defined to evaluate similarity of distributions between two image sets, in contrast to perceptual metrics such as SSIM that require a reference image for each given test image. That is, the FID is low between two sets of images that have similar image features, and overall diversity among the different generated samples is the same as in the reference samples. This metric is thus good to demonstrate that not only do the details in any generated environment map match realistic patches from training images, but the set of predicted images also have variety in content without over-fitting.

Results
To show the effectiveness of our method for rendering virtual objects in AR, we start with qualitative results from real-world applications of our model, followed by comparisons to previous work and quantitative benchmark results.

Qualitative results
In Fig. 4 we show results from our prototype iOS app used in real-world scenes. We use device pose and plane geometry provided by ARKit [2] to warp the input camera image into a partial environment map. The completed environment map from running our model on the device is then used with SceneKit 3 to render the virtual object 4 . The inference time is under 9 ms on an iPhone XS, allowing updates at high frame rates depending on desired user experience. Please see the supplementary material for videos from our application, where we use an update rate of 10fps for our results. Fig. 4(a) shows a real cooker in the right, and a rendered version on the left. Note that the material properties of the brushed steel body do not match exactly. We can clearly see that the lighting direction closely matches the reference real-world object, providing the correct specular highlights. The accurate lighting, combined with perceptually plausible reflection clearly makes the rendering an immersive AR experience. In Fig. 4(b) we render a mirror sphere into a scene to show the impact of generating plausible reflections. Though the input observes a very small part of the environment, we are able to create a believable AR experience by completing the environment map with a detailed scene. Even though it cannot match the actual room in every detail, the perceptual impact and coherence of the virtual object with other objects, such as the mirror on the wall, and metallic tower on the right, is evident. In Fig. 4(c)-(e) we provide more examples of inserted virtual objects that have lighting that is coherent with the real objects in the field of view.
From the images in Fig. 1 and 4 we highlight a few aspects of our method. For each of the scenes, that vary in time of day and FoV, not only is the light direction coherent with the scene but so is the type/shape/size of light source generated by our method. Fig. 1(e)  The roughness of the teapot was 0.2, and metalness was 1.0 in SceneKit. The light direction and highlights from our estimate closely match those on the real teapot (blue) and sphere (pool ball) in the scene. In Fig. 4(b) and (d) the light sources have appearances closer to that of a window/door that are common and plausible in those environments. We show qualitative comparisons to recent work from Srinivasan et al. [25], and their re-implementations of Neural Illumination [23] and Deep Light [17] in Appendix 5.

Quantitative evaluation and benchmarking
For quantitative benchmarking we use the metrics detailed in Sec. 4 on the publicly available Laval dataset [10], and show the results in Table 1. We show some example results in Fig. 5 with rendered objects using results and ground truth (GT) environment maps. All images are best seen on a color monitor with magnification. We highly encourage the reader to see Appendix 6 for details and more results with renderings.
As discussed in Sec. 2, though several methods have focused on light estimation from single images, only a few generate HDR RGB environment maps. Recent techniques that estimate lighting such as [23] and [17], have not shared code, model weights nor use public datasets. There is also a lack of a standard set of metrics for comparison. The method by Gardner et al. [10] is the current state-of-theart method that is most comparable to our method, uses the same public dataset, and is compared in the above recent works as well. They have provided a web interface to obtain results from their method 5 . Since we use the same splits for train/test, we provide detailed quantitative and qualitative comparisons with their results.
We summarize the quantitative metrics in Table 1, calculated over the 250 test images. The results from using our method are reported as EnvMapNet. We introduced two novel loss functions in our method, for which we also conducted ablation studies. For Ours without Projection-Loss we observe higher angular error than our full model EnvMapNet, as expected from our discussions in Sec. 3.3. When we trained Ours without ClusterLoss, for the discriminator, the model often suffered from instability as discussed in Sec. 3.4, and could not generate textures with fine details, hence the higher FID and angular error. The errors for Gardner et al. [10] are higher than our complete pipeline. As shown with example images (Fig. 5 and Appendix 6), the environment map generated by their method lacks fine details for the scenes hence the higher FID score. The angular error is also higher on average, which could be explained from our experiments in Sec. 3.3 that show MSE loss is insufficient to train for accurate lighting. Both these aspects can be clearly observed to affect the use of the environment maps in rendering objects with both mirror and diffuse finish, along with the cast shadows.
As a baseline measurement, we consider that AR applications, like games, may use a fixed environment map, created by an artist, for lighting. We obtained such an environment map designed for indoor scenes as detailed in Appendix 2. We calculate the angular error metric with respect to this artist designed environment map, and report it in Table 1 as Artist IBL.
As demonstrated through the various qualitative examples and quantitative benchmarking, our method, with lowest FID and AngularError, can produce high quality environment maps both for visually pleasing reflections, and accurate lighting of the virtual objects.

Conclusions
We presented the first method that, given a small fieldof-view LDR camera image, can estimate a high resolution HDR environment map in real-time on a mobile device. The result can be used to light objects of any material finish (mirror to diffuse) for augmented reality. We made two novel contributions in the training of our neural network, with ProjectionLoss for the environment map, and Clus-terLoss for the adversarial loss. This enabled our method to synthesize environment maps with accurate lighting and perceptually plausible reflections. We proposed two metrics to measure both these aspects of the estimated environment map. Through qualitative and quantitative comparisons we demonstrated that our method reduces the angular error in parametric light direction by more than 50%, along with a 3.7 times reduction in FID. We showcased a real-world mobile application that is able to run our model in real-time (under 9ms) and render visually coherent virtual objects in novel environments.

Video of mobile application
A video providing an overview of our method and a demonstration of our mobile application in the real world using iPhone XS is available at https://docsassets.developer.apple.com/ml-research/ papers/hdr-environment-map.mp4

Artist-created environment map
In Figure 1 we show the artist-created environment map used as a baseline in our benchmarking. The artist designed it to satisfy aesthetic and lighting requirements. For lighting, the intensities are selected to make sure the objects were well exposed and that middle gray is retained. The aesthetic brief was to have a "studio lighting" feel with a broad area light from behind the camera and from above.

Network details
In Table 1 we provide the building blocks of the model architectures used in our method. Spectral Normalization [4] is used in all the convolutional layers.
Our proposed model, EnvMapNet, consists of an encoder and decoder as shown in paper Figure 2 to output 64 filters. The decoder mirrors the encoder by using EnvMapNet-conv-block and EnvMapNet-upsampleblock, with uk= [512,256,256,128,128,128,64]. The final output is produced by a 3×3 convolution to produce 3 channels for RGB, followed by a tanh activation. We use skip connections between same sized layers of the encoder and decoder.
The discriminator is composed of residual blocks with

ProjectionLoss and user study details
In this section we expand on the background for Pro-jectionLoss, its relation to shadow casting, and the details of the user study and experiments discussed in paper Section 3.3.

Relation to shadows
In Figure 2 we provide an illustration to explain our motivation for ProjectionLoss. Consider an object (blue ball), lit by an environment map, and casting a shadow on the planar surface below. Accurate generation of shadows requires for every point on the plane (red cross) the computation of the integral over the part of the env map that is visible from that point (shaded orange), i.e. not occluded by the object (blue ball). To this end, the environment map is elementwise multiplied with a visibility mask. Our precomputed randomized masks (Figure 3) used in projection loss are examples for such visibility masks for different points on the plane and different object shapes. As a result, Projection-Loss encourages our predicted env maps to lead to shadows similar to ground truth.

User study
To further understand the value of ProjectionLoss for lighting estimation, and to compare to other measures, such as SSIM [8] and Mean Squared Error (MSE), we performed the following experiments. For each dataset environment map, we render an image of a scene with multiple geometric objects as shown in Fig. 4.
First we establish a baseline metric to measure similarity between rendered images. We propose using the SSIM score, which is a metric often used to measure human perception and image similarity, to quantify the similarity of our rendered images. To validate this for our application, we perform a user study asking participants to choose from four provided options which rendered image looks most similar to a reference rendered image. The four options were randomly selected such that two of them were in the top-10 as retrieved by SSIM, and the other two from outside the top-10. Based on the results of 5 study participants, each providing their selection for 380 reference images, we found that on average human participants selected one of the top-10 SSIM images 95% of the time. Hence retrieval of similar environment maps based on SSIM between rendered images is a baseline method to find environment maps that produce similar lighting on the objects.
We then evaluate the correlation between SSIM on rendered images with ProjectionLoss and MSE calculated on the corresponding (equirectangular) environment maps. We can observe that the retrieval of similar environment maps by ProjectionLoss is better matched with retrieval based on SSIM compared to using MSE, see Fig. 4. Quantitatively, we found the intersection of top-5 retrievals by SSIM (on the rendered images) and those using ProjectionLoss (on the environment map) to be 1.6 ± 0.7, while it was 0.6 ± 0.5 using MSE (on the environment map).
Based on the above we believe that our proposed Projec-tionLoss is a good loss to train the network for estimating lighting such that the end result for rendering is accurate with respect to ground truth. Secondly, we show that MSE on the environment map is insufficient for training accurate light estimation, and its use as a metric of comparison or benchmarking, as done in previous works, would not correlate well with the final application.

Comparisons with recent methods
In this section we expand on the qualitative results from paper Section 5.1 and provide comparisons to the very recent work of Srinivasan et al. [6], that trained and evaluated on synthetic LDR images from InteriorNet [3]. The authors also provided results from their re-implementation of Deep Light [2] and Neural Illumination [5]. We used the images and results provided as part of their paper and show the comparison to our results in Figure 5. We note that the input to our algorithm was only the incomplete panorama shown in Figure 5(a), while stereo images were used for Srinivasan et al. [6] (c), and the 32×32 sphere image output from LeGendre et al. [2] was converted to an equirectangular projection by the authors (d). Furthermore, compared methods were (re)trained by Srinivasan et al. on the same synthetic dataset (InteriorNet [3]), while our results are using the same network as before. It was trained on 2, 810 images from public datasets of real-world images -Laval Indoor HDR dataset [1] and PanoContext LDR panoramas [7] -as discussed in paper Section 3.1.
Since the network used by Srinivasan et al. was trained on stereo input and LDR ground truth (with reverse gamma), we only make qualitative comparisons for LDR completion/generation of environment maps. We noticed that the predictions made by the algorithm from Srinivasan et al. resembles the (unseen) ground truth textures very closely. For example, the window, curtains and wall boundaries in the first two images shown in Figure 5(c) closely match the corresponding regions in Figure 5(b). This is surprising given that the input panorama does not contain this information. Note that our model was trained with images which are aligned with vertical axis being gravity, that is, the horizon is aligned to the horizontal image axis. This is easily achieved in mobile AR using device pose and orientation, and it also avoids ambiguity when training a network. The images provided as input (a) have different rotations resulting in some of the artifacts observed. Additionally we observe that the input panoramas are not just masked subset of pixels from the ground truth. The artifacts in the input, such as the blocks and aliasing seen on the left side of the first input, could potentially be from reprojection done by Srinivasan et al., which is carried forward by our results as part of 'known input'. We manually corrected the above by 'straightening' the panoramas (g), and using the input mask to create Corrected Input (h), and show results from our method using this corrected input in row (i). Qualitatively we can observe that our method can produce plausible and perceptually pleasing reflections in these new scenes, even though our model was not trained on the same dataset. (c) Lighthouse [6] (d) Deep Light [2] (e) Neural Illumination [5] (f) EnvMapNet (Ours on (a)) (g) Rotated Ground Truth (h) Corrected Input (i) EnvMapNet (Ours on (h)) Figure 5. Qualitative comparisons with the results from Srinivasan et al. [6] and their re-implementation of [2] and [5]. See Appendix 5.

Results and comparison
In this section we provide several images to demonstrate the effectiveness of our method, for rendering virtual objects in mobile AR applications, and expand on the images and results discussed in paper Section 5.2.
In Figures 6 and 7 we show predicted environment maps from Gardner et al. [1] and our method over a variety of scenes and lighting conditions. We further show their use for rendering both reflective (teapot) and diffuse (dragon) objects 1 with shadows. Each result is shown over a pair of rows containing the input crop, rendered images, and corresponding environment maps with angular error below each. To enable fair and future comparison, we crop the test environment maps in the center for 90 degree FoV. We rectify and provide the Input Crop to [1], and project them to a equirectangular map for our method. We obtain the predicted equirectangular maps from each image, extract the parametric lights for calculation of AngularError as described in paper Section 4, and proceed to use for rendering.
We use Tungsten 2 an open source path tracer, to generate renderings of an aluminum metal finish teapot and a lambertian material on the dragon objects. Only for the purpose of rendering, we perform the following post-processing operations on each predicted equirectangular map. Since each method provides the result in an arbitrary intensity range, we scale the pixel intensities for each predicted result such that the average intensity in the region provided as input is equal to that of the ground truth HDR. We further overlay the pixels from the ground truth corresponding to the input FoV, to simulate the AR video backdrop and common background for each rendering. We provide this equirectangular map as the input for Tungsten, position the virtual camera to match the input crop provided, and obtain the rendered images shown in Figures 6 and 7 for EnvMap-Net (Ours), Ground Truth and Gardner et al. [1]. We stress that this post-processing was only done for the offline rendering pipeline and not for other comparisons.
As previously detailed, this simulates that a virtual object is placed at the center of a probe which is illuminated using the environment map. As shown in the images, our results generate perceptually pleasing and accurate lighting for the virtual object compared to those from [1], for both material finishes. Though our method cannot (and should not) create the unseen ground truth scene exactly, the level of detail generated is clearly plausible and provides a better visual coherence compared to those from [1]. This aspect regarding image resolution and quality of the details is captured by the FID metric as detailed in the main paper. The renderings of the diffuse dragon object on a plane highlight the difference in accuracy of estimated light direction, as captured by our AngularError metric listed below each result. The images show estimated environment maps with a wide range of errors from both methods, and as can be observed, overall our estimates produce shadows which better match those from the ground truth environment map.  Figure 6. Results with variety of angular errors. Each result is shown over a row containing the input crop, rendered images with reflective teapot and diffuse dragon, with corresponding environment maps below each. As discussed in paper Section 5 our method is qualitatively and quantitatively able to produce visually coherent environment maps for lighting, shadows and reflection.  More results with variety of angular errors. Each result is shown over a row containing the input crop, rendered images with reflective teapot and diffuse dragon, with corresponding environment maps below each. See Figure 6 and paper Section 5.