\section{Introduction}

    More than 70 years ago, \citet{turing1950computing} famously suggested that \say{instead of trying to produce a programme to simulate the adult mind, why not rather try to produce one which simulates the child's? If this were then subjected to an appropriate course of education one would obtain the adult brain.} If we want to take Turing’s proposal seriously, we have to ask ourselves: how do children learn?

    The physical laws of nature are one of the earliest things that children learn \citep{spelke2007core, lake2017building}. Therefore, they can serve as an ideal testbed for investigating this question. There has already been a substantial amount of work across different research areas to understand and reproduce the human ability for physical reasoning. On the one hand, empirical work in developmental psychology has provided us with a precise understanding of the different developmental stages that children undergo during their cognitive development \citep{baillargeon1996infants, baillargeon2004infants}. On the other hand, machine learning researchers have started to successfully apply tools from deep learning to build models that mimic the intuitive physical understanding of people \citep{battaglia2013simulation, lerer2016learning, zhang2016comparative, piloto2022learn, smith2019modeling, smith2020fine}.

    Even though there are a number of models for physical reasoning, they typically focus on reproducing adult-level performance. The goal of the present paper is to instead -- in the spirit of Turing -- compare the learning trajectories of artificial systems to the developmental trajectories of children. We are particularly interested in examining the idea of \emph{development as stochastic optimization}, which states that cognitive development results from some form of stochastic optimization procedure \citep{gopnik2017changes, ullman2020bayesian, giron2022developmental}.

    \clearpage

    \begin{figure}[!h]
        \centering
        \includegraphics[width=\textwidth]{figures/model_graph_2.pdf}
        \caption{A: Developmental trajectory for support events outlined by \citep{baillargeon1996infants}. The illustrations are taken from \citet{baillargeon1996infants} and they show the physical rules acquired at the respective ages. B: Illustration of our generative video prediction model.}
        \label{fig:model_graph}
    \end{figure}

    To test this hypothesis, we train a deep generative model on video sequences on a physical reasoning task. We then probe the knowledge of this model at different training epochs using violation-of-expectation methods \citep{baillargeon2004infants, piloto2018probing, smith2019modeling} and compare it to the knowledge of children at different ages. We find that the acquisition order of concepts in this model aligns with that of children, thereby providing support to the idea of development as stochastic optimization.

\section{Methods}

    \subsection{Support events}
    \label{sec:stimuli}

        Infants' physical reasoning abilities have been investigated in many different domains. Here, we use support events (such as the configurations of block stacks shown in Figure \ref{fig:model_graph}A) as an exemplary physical reasoning task for comparing the learning trajectories of artificial systems to the developmental trajectories of children.

        \citet{baillargeon1996infants} has shown that, as infants grow older, they make use of increasingly complex rules to decide whether a specific configuration of blocks is stable or not (see also \citep{baillargeon2002acquisition, baillargeon2004infants}). With $3$ months, infants decide based on a simple contact or no contact rule. According to this rule, a block configuration is considered to be stable if the two blocks touch each other. At around $5$ months, infants understand that the type of contact matters. Now, only configurations with blocks stacked on top of each other are judged as stable. At $6.5$ months, they additionally consider the overlap to determine the stability of a block configuration. Finally, at $12.5$ months they are able to incorporate the block shapes into their judgement, relying not only on the amount of contact but also on how the mass is distributed for each block.

        To assess the learning trajectories of artificial systems, we generated a data set containing $100.000$ video sequences of support events using the Unity game engine \citep{unity}. The video sequences show stacks of two coloured blocks in a gray room and they consist of $20$ frames with a size of $64\times64$ pixels (see Figure \ref{fig:kl_over_frames}). We randomly varied a number of properties to ensure sufficient variability in the training data (see Appendix \ref{sec:stim_var} for further details).

        \begin{figure}[!h]
            \centering
            \includegraphics[width=\textwidth]{figures/RSSMAlt_Cubes_seed_17_beta_1.0_state_20_hidden_200_nll_False_kloverframes_cond_2_rm_3_open.pdf}
            \caption{The first row shows the surprise for the expected and violated test sequences of the overlap rule. The second row shows the expected test sequence. The third row shows the violated test sequence. The last row shows the open-loop predictions from the model, given the first two frames of the violated test sequence. It is important to note that the first three frames were removed for this plot, as the uncertainty is very high when the model is first given the sequence.}
            \label{fig:kl_over_frames}
        \end{figure}

    \subsection{Modeling developmental trajectories}
    \label{sec:model}

        We investigate the development as stochastic optimization hypothesis by training a generative video prediction model using gradient descent. To obtain a learning trajectory of this model, we evaluate a snapshot of it in every epoch.

        We use the recurrent state space model (RSSM) \citep{hafner2019learning, saxena2021clockwork} as an exemplary model for our analysis. The RSSM can be seen as a sequential version of a variational autoencoder (VAE). It maintains a latent state at each time step, which is comprised of a deterministic component $h_t$ and a stochastic component $s_t$. These components depend on the previous time steps through a function $f(h_{t-1}, s_{t-1})$, which is implemented as a gated recurrent neural network. It is trained by optimizing the following evidence lower bound:
        \begin{align}
        -\sum_{t=1}^T \mathbb{E}_{q(s_t\mid o_{\leq t})}[\textrm{ln } p(o_t\mid s_t)] + \mathbb{ E}_{q(s_{t-1}\mid o_{\leq t-1})}\big[\textrm{KL} (q(s_t \mid o_{\leq t}) \mid \mid p(s_t\mid s_{t-1}))\big]
        \end{align}
        \label{eq:rssm_loss}

        After training, the RSSM can be used to generate open-loop predictions. For this, the model processes a number of initial observations to infer an approximate posterior $q(s_{t-1}\mid o_{\leq t-1})$. The subsequent open-loop predictions then result by decoding latent representations sampled from the prior $p(s_t\mid s_{t-1})$. We refer the reader to Appendix \ref{sec:model_imp} for further details about the model architecture and training procedure.

    \subsection{Measuring surprise}
    \label{sec:measuring_surprise}

        In order to assess whether our model has understood a specific rule, we use the violation of expectation paradigm \citep{piloto2018probing, piloto2022learn}. The model is presented with two video sequences: a violated sequence, which constitutes a violation according to the rule, and an expected sequence, which is consistent with the rule. If the model has successfully learned a specific rule, it should show a larger degree of surprise for the violated compared to the expected sequence. Following \citet{piloto2018probing}, we measure the model's surprise using the Kullback–Leibler (KL) divergence between the prior and posterior over the latent representation \citep{baldi2010bits}. This approach closely resembles how developmental psychologists assess children's understanding of physical rules.

        We constructed a violated and an expected test sequence with identical image statistics for each of the four rules for support events outlined by \citet{baillargeon1996infants}. For example, according to the overlap rule, a block configuration should only be stable if the blocks are stacked on top of each other with enough overlap. The two test sequences for this rule therefore show two blocks that only slightly overlap (see Figure \ref{fig:kl_over_frames}). In the violated test sequence, the block configuration nonetheless appears stable, with the top block remaining on top of the bottom block. In contrast and consistent with the rule, the expected test sequence shows the top block falling.

        \begin{figure}[!t]
            \centering
            \includegraphics[width=\textwidth]{figures/RSSMAlt_Cubes_seed_17_beta_1.0_state_20_hidden_200_nll_False_stochastic_optimisation-2.pdf}
            \caption{Linear regression fits for the difference between the surprise for the expected and the violated sequence at every epoch and for each of the four physical principles separately, as well as a 95\% confidence interval around the regression line. Additionally, the individual data points are binned in bins of size 10 and displayed with a 95\% confidence interval.}
            \label{fig:kl_over_epochs}
        \end{figure}

\section{Results}
\label{sec:results}

    Before investigating the learning trajectory of our model, we verified that it is able to predict the given scenes accurately into the future. For this purpose, we plotted the open-loop predictions of the fully trained model for all test sequences. Figure \ref{fig:kl_over_frames} illustrates the result for the overlap test sequences. We see that the predictions of the fully trained model closely match the expected sequence, indicating that the model has learned the overlap rule by the end of its training. This result also holds for the three other physical rules, see Appendix \ref{sec:figures} for the corresponding plots.

    Next, we validated that the fully trained model is surprised when it observes a violation of a physical rule it has acquired. Figure \ref{fig:kl_over_frames} shows the KL divergence over the frames for the overlap test sequences. We see that the KL divergence for the violated sequence exceeds that for the expected sequence from around the fifth frame, which is when the two sequences diverge.

    To quantitatively assess whether the model has learned the four physical rules, we computed the difference in KL divergence between the expected and violated sequences summed up over all frames. Figure \ref{fig:kl_over_epochs} shows this measure for each of the four physical rules over the course of training. We find that the difference in KL divergence is positive for all four physical rules in the fully trained model, confirming that it has learned all of them by the end of training.

    Finally, we investigated whether the stochastic optimization hypothesis can capture the developmental trajectories of children. For this purpose, we fitted a simple linear regression for each of the rules, using the difference in KL divergence as the dependent and training epochs as the independent variable. This directly relates to the stochastic optimization hypothesis, as the model becomes increasingly optimized over the epochs. The resulting regression coefficients were $0.390 \pm 0.024$, $0.261 \pm 0.017$, $0.170 \pm 0.011$, and $0.059 \pm 0.007$ for the contact or no contact, type of contact, overlap, and shape rules, respectively (all $p < 0.001$). We can see that the coefficients become smaller as the rules increase in complexity. The model first acquires the contact or no contact, followed by the type of contact, then the overlap, and finally the shape rule. This order of acquisition matches how children acquire these rules, thereby providing support to the idea of development as stochastic optimization.

\section{Discussion}
\label{sec:discussion}

    We have compared the learning trajectories of an artificial system to the developmental trajectories of children in the domain of physical reasoning. More specifically, we examined the idea of development as stochastic optimization. For this purpose, we used the violation of expectation paradigm to probe the knowledge of a modern deep generative neural network at different stages of its training process. We found that the model's learning trajectory resembled the trajectories of children during their cognitive development.

    In contrast to previous work in this domain \citep{binz2019emulating, giron2022developmental}, our modeling approach employs high-dimensional visual stimuli (i.e., video sequences) and solely relies on an unsupervised training objective. It, therefore, more closely mirrors the actual learning processes of children in the real world. Furthermore, the conducted experiments build on a well-established paradigm from developmental psychology. Together, these factors provide a high validity to our results.

    Presently, the biggest mismatch between learning in our models and learning in children is that the latter do not observe a large number of support events. Instead, they simply witness the real world and generalize their acquired knowledge to the given experimental setting. To capture this process, we should ideally train our models in a similar way. This could, for example, be accomplished by utilizing the SAYCam data set, which contains a large number of longitudinal video recordings from infants’ perspectives \citep{sullivan2021saycam}. Additionally, this data set includes time stamps indicating when a child has encountered a particular scene, which could be used to investigate how the nature of the training data influences development.

    It also seems plausible that factors beyond stochastic optimization drive human development. There is, for example, evidence suggesting that children gain access to additional computational resources during their development, allowing them to apply more complex strategies \citep{binz2022exploration}. This idea of \emph{development as complexity increase} could be readily incorporated in our framework by replacing the evidence lower bound from Equation \ref{eq:rssm_loss} with a $\beta$-VAE objective \citep{higgins2016beta, burgess2018understanding}.

    To fully disentangle these hypotheses, a single paradigm will likely not be sufficient. Instead, we will need to go beyond support events and investigate developmental trajectories across a variety of different tasks. Findings that hold across domains would further increase the reliability of our results and have potential implications for cognitive psychology and artificial intelligence alike.

\begin{ack}
This work was funded by the Max Planck Society and the Volkswagen Foundation.
\end{ack}

\clearpage
\bibliographystyle{svrhm_2022}


\section{Introduction}   \label{sec:intro}

A long-standing goal of Computer Vision is to understand human actions from videos. Given a \textcolor{black}{video} people effortlessly figure out what objects exist in it, the spatial layout of objects, and the pose of humans. Moreover, they deeply understand the depicted action. What is the subject doing? Why  are they doing this? What is their goal? How  do they achieve this? To empower computers with the ability to infer such abstract concepts from pixels, we need to capture rich datasets and to devise appropriate algorithms.

Since humans live in a 3D\xspace world, their physical actions involve interacting with objects. \textcolor{black}{Think of} how many times per day one goes to the kitchen, grabs a cup of water, and drinks from it. This involves contacting the floor with the feet, contacting the cup with the hand, moving the hand and cup together while maintaining contact, and drinking while the mouth contacts the cup. Thus, to understand human actions, it is necessary to reason in 3D\xspace about humans and objects \emph{jointly}.

There is significant prior work on estimating~3D\xspace humans without taking into account \textcolor{black}{objects} \cite{bogo2016keep} and estimating~3D\xspace \textcolor{black}{objects} without taking into account humans \cite{zollhofer2018state}.
There is even recent work on inserting bodies into 3D\xspace scenes such that their interactions appear realistic \cite{zhang2020generating, li2019putting, Hassan2021posa}. But there is little work on estimating 3D\xspace humans interacting with scenes \GCPR{and moving objects}, in which the human-scene/object contact is explicitly modeled and exploited. To study this problem, we need a dataset of videos with rich human-object interactions and reliable \GCPR{3D\xspace} ground truth.

\mbox{\mbox{PROX}}\xspace~\cite{hassan2019resolving} takes a step in this direction by estimating the 3D\xspace body in a known 3D\xspace scene. The scene mesh provides information that helps resolve pose ambiguities commonly encountered when a single camera is used. However, \mbox{\mbox{PROX}}\xspace involves only coarse interactions of bodies, static scenes \textcolor{black}{with no moving objects}, and \textcolor{black}{no dexterous fingers.} The recent \behave dataset \cite{bhatnagar2022behave} uses multi-view \mbox{RGB}\xspace-D data to capture humans interacting with objects but does not include detailed hand pose or fine hand-object contact. Finally, the {\mbox{GRAB}}\xspace dataset \cite{taheri2020grab} captures the kind of detailed hand-object and whole-body-object interaction that we seek but is captured using marker-based \mbox{MoCap}\xspace and, hence, lacks images paired with the ground-truth scene.

We argue that what is needed is a new dataset of \GCPR{\mbox{RGB}\xspace} videos containing natural human-object interaction in which the \textcolor{black}{whole} body is tracked reliably, the hand pose is captured, objects are also tracked, and the hand-object contact is realistic; see \fig{fig:teaser}. This is challenging and requires technical innovation to create. To that end, we design a system that uses multiple \mbox{RGB-D}\xspace sensors that are spatially calibrated and temporally synchronized. To this data we fit the \smplx body model, which has articulated hands, by extending the \mbox{\mbox{PROX}}\xspace~\cite{hassan2019resolving} method to \textcolor{black}{use} multi-view data \textcolor{black}{and  grasping hand-pose priors}. We also track the 3D\xspace objects with which the person interacts. The objects used in this work are representative of  items one finds in daily life. We obtain \GCPR{accurate}~3D\xspace models for each \textcolor{black}{object}~\GCPR{with a handheld \mbox{\mbox{Artec}}\xspace scanner}. Altogether we collect {\ourNUmbColor{223}}\xspace sequences ({\ourNUmbColor{67,357}}\xspace multi-view frames), with \ourNUmbColor{10}\xspace subjects interacting with {\ourNUmbColor{10}}\xspace objects.

The problem, however, is that separately estimating the body and objects is not sufficient to ensure accurate 3D\xspace body-object contact. Consequently, a key innovation of this work is to estimate these \emph{jointly}, while exploiting information about \emph{contact}. Objects do not move independently, so, when they \textcolor{black}{move}, it means the body is in contact. We define likely contact regions on objects and on the body. Then, given frames with known \GCPR{likely} contacts, we enforce contact between the body and the object when estimating the body and object poses. The resulting method produces \textcolor{black}{natural} body poses, hand poses, and object poses. Uniquely, it provides detailed \GCPR{pseudo} ground-truth contact information between the \textcolor{black}{whole} body and objects in \GCPR{\mbox{RGB}\xspace} video.

In summary, our major contributions are as follows:
    (1)     We develop a novel Motion Capture \textcolor{black}{method} utilizing multiple \mbox{RGB-D}\xspace cameras. It is relatively lightweight and flexible, yet accurate enough, thus suitable for data capture of daily scenarios.
    (2) We extend previous work on fitting \smplx to images to fit it to multi-view \mbox{RGB-D}\xspace data while taking into account body-object contact.
    (3)     We capture a novel dataset that contains whole-body human motions and interaction with objects, as well as multi-view \mbox{RGB-D}\xspace imagery. Our data and code are available at \mbox{\url{https://intercap.is.tue.mpg.de}}.

\section{Related Work}   \label{sec:related}

There is a large literature
on estimating 3D\xspace human pose and shape
from images or videos \cite{bogo2016keep, pavlakos2019expressive, choutas2020monocular, kanazawa2018end, kocabas2020vibe, varol2017long, mehta2017vnect, omran2018neural}.
Here we focus on the work most closely related to ours, particularly as it concerns, or enables, capturing human-object interaction.

\textbf{\mbox{MoCap}\xspace from Multi-view Videos and {IMUs}\xspace.}
Markerless \mbox{MoCap}\xspace from multi-view videos \cite{liu2011markerless, de2008performance, huang2017towards} is widely studied and commercial solutions exist (\mbox{e.g.}\xspace,~Theia Markerless).
Compared with traditional marker-based \mbox{MoCap}\xspace, markerless offers advantages of convenience,  applicability in outdoor environments, \textcolor{black}{non-intrusiveness}, and greater flexibility.
However, traditional \mbox{MoCap}\xspace methods, both marker-based and markerless, focus on extracting a 3D\xspace skeleton.
This is useful for biomechanics but our goal is to reason about body-scene contact.
To enable that, we need to capture the body surface.

Various 3D\xspace human representations have been proposed, with recent work focused on learning a parametric \textcolor{black}{mesh-based} model of body shape from large-scale collections of 3D\xspace scans \cite{anguelov2005scape, loper2015smpl, romero2017embodied, pavlakos2019expressive, osman2020star, xu2020ghum, supr2022}.
Here we use the \mbox{SMPL-X}\xspace model \cite{pavlakos2019expressive} because it contains fully articulated hands, which are critical for reasoning about object manipulation.
The  body parameters are often estimated by fitting the 3D\xspace generative model to various 2D\xspace cues like landmarks detected by Convolutional Neural Networks \cite{cao2019openpose, wei2016convolutional, newell2016stacked} or silhouettes \cite{rhodin2016general, xu2018monoperfcap, alldieck2018video}.
Though effective, these monocular video-based methods suffer from depth ambiguity and occlusions.
To address this issue, researchers have proposed to combine {IMUs}\xspace with videos to obtain better and more robust results \cite{von2018recovering, pons2010multisensor}.

Many methods estimate 3D\xspace bodies from multi-view images but focus on skeletons and not 3D\xspace bodies
\cite{epipolartransformers2020cvpr,iskakov2019learnable,Qiu_2019_ICCV,to2020voxelpose,Dong_2019_CVPR,Dong_2021_PAMI,Zhang_2020_CVPR}.
Recent work addresses 3D\xspace body shape estimation from multiple views \cite{huang2017towards,dong2021shape,lightcap2021}.
Most related to our work are two recent datasets.
The \mbox{\mbox{RICH}}\xspace dataset \cite{Huang:CVPR:2022}, fits \smplx bodies to multi-view \mbox{RGB}\xspace videos taken both indoors and outdoors.
The method uses a detailed 3D\xspace scan of the scene and models the contact between the body and the world.
\mbox{\mbox{RICH}}\xspace does not include any object motion; the scenes are completely rigid.
In contrast, \behave~\cite{bhatnagar2022behave} contains \mbox{SMPL}\xspace bodies interacting with 3D\xspace objects that move.
We go beyond that work, however, to integrate novel contact constraints and to capture hand pose, which is critical for human-object interaction.
Additionally, \behave focuses on large objects like boxes and chairs, whereas we have a wider range of object sizes, including smaller objects like cups.

\textbf{Human-Object Interaction.}
There has been a lot of work on modeling or analyzing human-object interactions \cite{yao2010modeling, Hamer_Hand_Manipulating_2009, oikonomidis2011full, Rogez2015everyday, tzionas2016capturing, hampali2020honnotate, hasson2019learning, karunratanakul2020grasping, bhatnagar2022behave}. A detailed discussion is out of the scope of this work. Here, we focus on modeling and analyzing human-object interaction in 3D\xspace space. Most existing work, however, only focuses on estimating hand pose \cite{hasson2019learning, hampali2020honnotate, hasson2020leveraging, romero2010handsInAction}, ignoring the strong relationship between body motion, hand motion, and object motion. Recent work considers whole-body motion. For example, the {\mbox{GRAB}}\xspace~\cite{taheri2020grab} dataset  provides detailed object motion and whole-body motion in a parametric body format (\smplx). Unfortunately, it is based on \mbox{MoCap}\xspace and does not include video. Here our focus is on tracking the \textcolor{black}{whole}-body \GCPR{motion}, object motion, and the detailed hand-object contact to provide ground-truth 3D\xspace information in \mbox{RGB}\xspace video.

\textbf{Joint Modeling of Humans and Scenes.}
There is some prior work addressing human-object contact in both static images and video. For example, \mbox{\mbox{PHOSA}}\xspace estimates a 3D\xspace body and a 3D\xspace object with plausible interaction from a single \mbox{RGB}\xspace image \cite{zhang2020phosa}. Our focus here, however, is on dynamic scenes. Motivated by the observation that natural human motions always happen inside 3D\xspace scenes, researchers have proposed to model human motion jointly with the surrounding environment \cite{hassan2019resolving, savva2016pigraphs, cao2020long, yi2022mover}. In \mbox{\mbox{PROX}}\xspace~ \cite{hassan2019resolving} the contact between humans and scenes is explicitly used to resolve ambiguities in pose estimation. The approach avoids bodies interpenetrating scenes while encouraging contact between the scene and nearby body parts. Prior work also tries to infer the most plausible position and pose of humans given the 3D\xspace scene \cite{zhang2020generating, li2019putting, Hassan2021posa}. Most recently, \mover~\cite{yi2022mover} estimates the 3D\xspace scene and the 3D\xspace human directly from a static monocular video in which a person interacts with the scene. While the 3D\xspace scene is ambiguous and the human motion is ambiguous, by exploiting contact, the method resolves many ambiguities, improving the estimates of both the scene and the person. Unfortunately, this assumes a static scene and does not model hand-object manipulation.

\textbf{Datasets.}
Traditionally, \mbox{MoCap}\xspace is performed using marker-based systems inside lab environments. To capture object interaction and contact, one approach uses MoSh \cite{loper2014mosh} to fit a \mbox{SMPL}\xspace or \smplx body to the markers \cite{mahmood2019amass}. An advanced version of this is used for {\mbox{GRAB}}\xspace~\cite{taheri2020grab}. Such approaches lack synchronized \mbox{RGB}\xspace video. The HumanEva \cite{sigal2006humaneva} and Human3.6M \cite{ionescu2013human3} datasets combine multi-camera \mbox{RGB}\xspace video capture with synchronized ground-truth 3D\xspace skeletons from marker-based \mbox{MoCap}\xspace. These datasets lack ground-truth 3D\xspace body meshes, are captured in a lab setting, and do not contain human-object manipulation. \threedpw~\cite{von2018recovering} is the first in-the-wild dataset that jointly features natural human appearance in video and accurate 3D\xspace pose. This dataset does not track objects or label human-object interaction. {\mbox{PiGraphs}}\xspace~\cite{savva2016pigraphs} and \mbox{\mbox{PROX}}\xspace \cite{hassan2019resolving}  provide both 3D\xspace scenes and human motions but are relatively inaccurate, relying on a single \mbox{RGB-D}\xspace camera. This makes these datasets ill-suited as evaluation benchmarks. The recent \mbox{\mbox{RICH}}\xspace dataset \cite{Huang:CVPR:2022} addresses many of these issues with indoor and outdoor scenes, accurate multi-view capture of \smplx, 3D\xspace scene scans, and human-scene contact. It is not appropriate for our task, however, as it does not include object manipulation.
\input{paper_TAB_01_Data_Compare} An alternative approach is the one of \mbox{GTA-IM} \cite{cao2020long} and \mbox{SAIL-VOS} \cite{hu2019sail}, which generate human-scene interaction data using either 3D\xspace graphics or 2D\xspace videos. They feature high-accuracy ground truth but lack visual realism. In summary, we believe that a 3D\xspace human-object interaction dataset needs to have accurate hand poses to be useful, since  hands are how people most often interact with objects. We compare our \intercap dataset with other ones in \tab{tab:dataset_compare}.

\section{\textcolor{black}{\intercap} Method}      \label{sec:method}

Our core goal is to accurately estimate the human \textcolor{black}{and object} motion throughout a video. Our markerless motion capture method is built on top of the \mbox{\mbox{PROX-D}}\xspace method \textcolor{black}{of Hassan~\mbox{et al.}\xspace}~\cite{hassan2019resolving}. To improve the body tracking accuracy we extend this method to use multiple RGB-D cameras; \textcolor{black}{here we use} the latest {Azure Kinect}\xspace cameras. The motivation is that multiple cameras observing the body from different angles give more information about the human and object motion. Moreover, \GCPR{commodity}~\mbox{RGB-D}\xspace cameras are much more flexible to deploy out of controlled lab scenarios than more specialized devices.

The key technical challenge lies in accurately estimating  the 3D\xspace pose and translation of the objects while a person interacts with them. In this work we focus on $10$ variously sized rigid objects common in daily life, \textcolor{black}{such as} cups and chairs. Being rigid does not make the tracking of the objects trivial because of the occlusion by the body and hands. While there is a rich literature on 6 DoF object pose estimation, much of it ignores hand-object interaction. Recent work in this direction is promising but still focuses on scenarios that are significantly simpler than ours, cf.~\cite{sun2022onepose}.

\GCPR{Similar}
to previous work on hand and object pose estimation \cite{hampali2020honnotate} from \mbox{RGB-D}\xspace videos, in this work we assume that the 3D\xspace meshes of the objects are known in advance. To this end, we first gather the 3D\xspace models of these objects from the Internet whenever possible and  scan the remaining objects ourselves. To fit the known object models to image data, we first preform semantic segmentation, find the corresponding object regions in all camera views, and fit the 3D\xspace mesh to the segmented object contours via differentiable rendering. Since heavy occlusion between humans and objects in some views may make the segmentation results unreliable, aggregating segmentation from all views boosts the object tracking performance.

In the steps above, both the subject and object are treated separately and processing is per frame, with no temporal smoothness or contact constraint applied. This produces jittery motions and heavy penetration between objects and the body. Making matters worse, our human pose estimation exploits
\mbox{OpenPose}\xspace for 2D keypoint detection, which \textcolor{black}{struggles} when the object occludes the body or the hands interact with it. To mitigate this issue and still get reasonable \textcolor{black}{body,} hand and object pose in these challenging cases, we manually annotate the frames where the \textcolor{black}{body or the hand} is in contact with the object, \textcolor{black}{as well as} the \textcolor{black}{body,} hand and object vertices that are most likely to be in contact. \textcolor{black}{This manual annotation can be tedious; automatic detection of contact is an open problem and is left for future work.} We then explicitly encourage the labeled \textcolor{black}{body and} hand vertices to be in contact with the labeled object vertices. We find that this straightforward idea works well in practice. More details are described in the following.

\input{03_method_01_setup}
\input{03_method_02_object_track}

\input{03_method_03_mocap_kinects_00}
\input{03_method_04_joint_tracking_00}

\subsection{Multi-Kinect Setup}

We use 6 {Azure Kinects}\xspace to track the human and object \GCPR{together}, deployed \textcolor{black}{in a ``ring'' layout} in an office; see \textcolor{black}{{Appx.}}\xspace Multiple \mbox{RGB-D}\xspace cameras provide a good balance between body tracking accuracy and applicability to real scenarios, compared with costly professional \mbox{MoCap}\xspace systems \GCPR{like}~\mbox{Vicon}\xspace, or cheap and convenient but not-so-accurate monocular \mbox{RGB}\xspace cameras. Moreover, this approach does not require \textcolor{black}{applying any} markers, making the images natural. \GCPR{Intrinsic camera parameters are provided by the manufacturer.} \textcolor{black}{Extrinsic camera parameters are obtained via} camera calibration with {Azure Kinect}\xspace's API \cite{kinectAPI}. \textcolor{black}{However, these can be a bit noisy, as non-neighbouring cameras in a sparse ``ring'' layout don't observe the calibration board well at the same time. Thus, we manually refine in MeshLab the extrinsics by comparing the point clouds for neighbouring cameras for several iterations.} \textcolor{black}{The hardware synchronization of {Azure Kinects}\xspace is empirically reasonable.} Given \GCPR{the calibration} information, we choose one camera's 3D\xspace coordinate frame as the global frame and transform the point clouds from the other frames into the global frame, which is where we fit the \smplx \textcolor{black}{and object models}.

\subsection{Sequential Object-Only Tracking}

\textbf{Object Segmentation.}
To track an object during interaction, we need reliable visual cues about it to compare with the 3D\xspace object model.
To this end,
we perform semantic segmentation by applying {PointRend}\xspace~\cite{kirillov2020pointrend} to the whole image.
We then extract the object instances that correspond to the categories of our objects;
\GCPR{for examples see \textcolor{black}{{Appx.}}\xspace}
We assume
that the subject interacts with a single object.
Note that, in contrast to previous approaches where the objects occupy a large portion of the image \cite{hampali2020honnotate,hassan2019resolving,tzionas2016capturing,oikonomidis2011full}, in our case
\GCPR{the entire body is visible,
thus, the object takes up a small part of the image and is often occluded by the body and hands; our setting is much more challenging.}
We observe that {PointRend}\xspace works reasonably well for large objects like chairs, even with heavy occlusion between the object and the human, while for small objects, like a bottle or a cup,
\textcolor{black}{it struggles} significantly due to
occlusion.

In extreme cases, it is possible for the object to
\GCPR{not be detected in}
most of the views.
But
even when the segmentation is good, the class label for the objects may be wrong.

\GCPR{To resolve this, we take two steps:}
\GCPR{(1)
For every frame, we \textcolor{black}{detect}
all possible object segmentation candidates and their labels}. This step takes place offline and only once.
\GCPR{(2)
During the object tracking phase,
for each view, we compare
the rendering of the tracked object from the $i^{\mathrm{th}}$ frame with
all the detected segmentation candidates for the \GCPR{$(i+1)^{\mathrm{th}}$ frame}, and preserve
only the candidate with the largest overlap ratio.}
This
\textcolor{black}{render-compare-and-preserve}
operation takes place \textcolor{black}{iteratively} during tracking.

\textbf{Object Tracking.}
Given object
\GCPR{masks}
via semantic segmentation over the whole sequence, we track the object
\GCPR{by fitting
its model} to
observations via differentiable rendering \cite{kato2018neural, loper2014opendr}.
This
is similar to past work for hand-object tracking \cite{hampali2020honnotate}.
We
assume that the object is rigid and its mesh is given.
The configuration of the rigid object in the $t^{\mathrm{th}}$ frame
is
specified via a 6D rotation and translation vector $\xi$.
\textcolor{black}{For initialization, we manually obtain the configuration of the object for the first frame by matching the object mesh into the measured point clouds.} Let \GCPR{$R_S$ and $R_D$ be functions that render a synthetic mask and depth image for the tracked 3D\xspace object mesh, $M$}. Let also $S = \{S_{\nu}\}$ be the \GCPR{``observed''} object masks and $D = \{D_{\nu}\}$ be corresponding depth values for the current frame, \GCPR{where $\nu$ is the camera view.} Then, we minimize:
{\small
\begin{align}
\begin{split}
E_O(\xi; S, D) =
\sum_{{\text{view}~\nu}} ~
& \lambda_{segm}  \| (R_S(\xi, M, {\nu}) - S_{\nu}) * S_{\nu} \|_F^2 +    \\
& \lambda_{depth} \| (R_D(\xi, M, {\nu}) - D_{\nu}) * S_{\nu} \|_F^2    \text{,}
\label{eq:fit_object}
\end{split}
\end{align}
}
\GCPR{where the two terms compute how well the rendered object mask and depth \textcolor{black}{image} match the detected \textcolor{black}{mask and observed depth;} the $*$ is an element-wise multiplication, and $\|.\|_F$ the Frobenius norm}; \textcolor{black}{ $\lambda_{segm}$ and $\lambda_{depth}$ are steering weights
\textcolor{black}{set empirically}. }For simplicity, we assume that transformations from the master to other camera frames are encoded in the rendering functions $R_S, R_D$; we do not denote these explicitly here.

\subsection{Sequential Human-Only Tracking}

We estimate body shape and pose over the whole sequence from multi-view \mbox{RGB-D}\xspace videos in a frame-wise manner. This is similar in spirit with the \mbox{\mbox{PROX-D}}\xspace method \cite{hassan2019resolving}, but, in our case, there is no 3D\xspace scene constraint and multiple cameras are used. The human pose and shape are optimized independently in each frame. We use the \smplx~\cite{pavlakos2019expressive} model to represent the 3D\xspace human body. \smplx is a function that returns a water-tight mesh given parameters for shape, $\beta$, pose, $\theta$, facial expression, $\psi$, and translation, $\gamma$. We follow the common practice of \textcolor{black}{using a $10$-dimensional space for} shape, $\beta$, and a $32$-dimensional latent space in \mbox{VPoser}\xspace \cite{pavlakos2019expressive} to present body pose, $\theta$.

We minimize the loss defined below. For \textcolor{black}{each} frame we essentially extend the major loss terms used in \mbox{\mbox{PROX}}\xspace~\cite{hassan2019resolving} to multiple views:
{\small
\begin{align}
\begin{split}
    E_B(\beta, \theta, \psi, \gamma; K, J_{est}) = E_J + & \lambda_D E_D + \lambda_{\theta_b} E_{\theta_b} +
    \lambda_{\theta_h} E_{\theta_h} +
    \lambda_{\theta_f} E_{\theta_f} +
    \\
   &\lambda_\alpha E_\alpha + \lambda_\beta E_\beta + \lambda_\mathcal{E} E_\mathcal{E} +
    \lambda_\mathcal{P} E_\mathcal{P}       \text{,}
\label{eq:fit_body}
\end{split}
\end{align}
}%
where
$E_\beta$, $E_{\theta_b}$, $E_{\theta_h}$, $E_{\theta_f}$, $ E_\mathcal{E}$ are prior loss terms for body shape, body pose, hand pose, facial pose and expressions. Also, $E_\alpha$ is a prior for extreme elbow and knee bending. \GCPR{For detailed definitions of these terms see \cite{hassan2019resolving}.} $E_J$ is a  2D\xspace keypoint \GCPR{re-projection} loss:
{
\small
\begin{equation}
    E_J(\beta, \theta, \gamma; K, J_\text{est} ) =
        \sum_{\mathit{view}~\nu} ~ \sum_{\mathit{joint}~i} k_i^\nu w_i^\nu \rho_J
        \big(   \mathit{\Pi}_K^\nu  \big(   R_{\theta \gamma}   (   J(\beta)_i
                                                                )
                                    \big)   - J_{est, i}^\nu
        \big)  \text{,}
\end{equation}
}
\REBT{where \textcolor{black}{$\theta = \{  \theta_b, \theta_h, \theta_f \}$,} $\nu$ and $i$ iterate through views and joints, $k_i^\nu$ and $w_i^\nu$ are the per-joint weight and detection confidence, $\rho_J$ is \GCPR{a robust Geman-McClure error function \cite{GemanMcClure1987}}, $\mathit{\Pi}_K^\nu$ is the projection function \GCPR{with $K$ camera parameters}, \GCPR{$R_{\theta \gamma}(J(\beta)_i)$ are the posed 3D\xspace joints of \smplx,} and $J_{est, i}^\nu$ the detected 2D\xspace joints}. The term $E_D$ is:
{
\small
\begin{equation}
E_D(\beta, \theta, \gamma; K) = \sum_{\mathit{view}~\nu}    ~
                                    \sum_{p \in P^{\nu}}        ~
                                    \min_{v \in V_b^{\nu}} \|v-p\|
                                    \text{,}
\end{equation}
} \GCPR{where $P^{\nu}$       is {Azure Kinect}\xspace's segmented point cloud for the $\nu^\text{th}$ view, and
$V_b^{\nu}$     are \smplx vertices that are visible in this view}. \GCPR{This term measures how far the estimated body mesh is from the combined point clouds, \textcolor{black}{so that we minimize this discrepancy}. Note that, unlike \mbox{\mbox{PROX}}\xspace, we have multiple point clouds from all views, \mbox{i.e.}\xspace, our $E_D$ is a multi-view extension of \mbox{\mbox{PROX}}\xspace's~\cite{hassan2019resolving} loss}. For each view we dynamically compute the visible body vertices, and ``compare'' them against the segmented point cloud for that view.

\GCPR{Finally, the term} $ E_\mathcal{P} $ penalizes self-interpenetration of the \smplx body mesh; see \mbox{\mbox{PROX}}\xspace~\cite{hassan2019resolving} for a more detailed and formal definition of this:
\begin{equation}
    E_\mathcal{P}(\theta, \beta, \gamma) = E_{\mathcal{P}_{self}}(\theta, \beta).
    \label{eq:penetration}
\end{equation}

\subsection{Joint Human-Object Tracking Over All Frames}

\input{paper_FIG_XX_object_photos}
\input{paper_FIG_XX_annotations}

We treat the result of the above optimization as initialization for refinement via \emph{joint} optimization of the body and the object \emph{over all frames}, subject to \emph{contact} constraints.

For this we fix the body shape parameters, \textcolor{black}{$\beta$,} as the mean body shape computed over all frames from the first stage, as done in \cite{huang2017towards}. Then, we jointly optimize the object pose and translation, \textcolor{black}{$\xi$}, body pose, \textcolor{black}{$\theta$}, and body translation, \textcolor{black}{$\gamma$}, over all frames. We add a temporal smoothness loss to reduce jitter for both the human and the object. We also penalize the \REBT{body-object} interpenetration, as done in \mbox{\mbox{PROX}}\xspace~ \cite{hassan2019resolving}. A key difference is that in \mbox{\mbox{PROX}}\xspace the scene is static, while here the object is free to move.

To enforce contact, we annotate the \textcolor{black}{body areas} that are most likely to be in contact with the objects and, for each object, we label vertices most likely to be contacted. These annotations are shown in \fig{fig:body_annotations} and \fig{fig:object_photos_annotations}\textcolor{black}{-right}, respectively, in red. We also annotate frame sub-sequences where \textcolor{black}{the body is} in contact with objects, and enforce contact between them explicitly to get reasonable tracking even when there is heavy interaction and occlusion between hands and objects. Such interactions prove to be challenging for state-of-the-art 2D\xspace joint detectors, \mbox{e.g.}\xspace, \mbox{OpenPose}\xspace, \textcolor{black}{especially for hands}.

Formally, \GCPR{we perform global optimization over all $T$ frames, and minimize} a loss, $E$, that is composed of an object fitting loss, $E_O$, a body fitting loss, $E_B$, \REBT{a motion smoothness prior \cite{zhang2021learning} loss, $E_{\mathcal{S}}$,} and a loss penalizing object acceleration, $E_{\mathcal{A}}$. We also use a ground support loss, $E_{\mathcal{G}}$, that \GCPR{encourages the human and the object} to be above the ground plane, \mbox{i.e.}\xspace, \GCPR{to not penetrate it}. Last, we use a \textcolor{black}{body}-object contact loss, $E_{\mathcal{C}}$, that attaches the \textcolor{black}{body} to the object for frames with contact. The loss $E$ is defined as:
{
\footnotesize
\begin{align}
\begin{split}
E = ~~
\frac{1}{T}
\sum_{\text{frame}~t}
&\bigg[
E_O(\Xi_t; \mathcal{S}_{t}, \mathcal{D}_{t}) +
E_B(\beta^*, \Theta_t, \Psi_t, \Gamma_t; \mathcal{J}_{\mathit{est}})
\bigg] + \\
\frac{1}{T}
\sum_{\text{frame}~t}
&\bigg[
E_\mathcal{P}(\Theta_t, \beta^*, \Gamma_t) +
E_{\mathcal{C}}(\beta^*, \Theta_t, \Psi_t, \Gamma_t, \Xi_t, M)
\bigg]
+ \\
\frac{\lambda_{\mathcal{G}}}{T}
\sum_{\text{frame}~t}
&\bigg[
E_{\mathcal{G}}(\beta^*, \Theta_t, \Psi_t, \Gamma_t) + E_{\mathcal{G'}}(\Xi_t, M)
\bigg]
+ \\
\frac{\lambda_\mathcal{Q}}{T}
\sum_{\text{frame}~t}
&\bigg[
Q_t * E_\mathcal{C}(\beta^*, \Theta_t, \Psi_t, M', \Xi_t)
\bigg]      + \\
&
\lambda_\mathcal{S}       E_\mathcal{S}(\Theta, \Psi, \Gamma, A; \beta^*, T) + \\
&
\lambda_{\mathcal{A}}    E_{\mathcal{A}}(\Xi, T, M)
\text{,}
\end{split}
\label{eq:big_loss}
\end{align}
}
where
\GCPR{
$E_O$ comes from \textcolor{black}{Eq.~}\ref{eq:fit_object} and
$E_B$       from \textcolor{black}{Eq.~}\ref{eq:fit_body}, and both go through all views $\nu$,
\textcolor{black}{while $E_\mathcal{P}$ comes from \textcolor{black}{Eq.~}\ref{eq:penetration}}.}
\GCPR{For all frames $t = \{ 1, \dots, T\}$ of a sequence},
\GCPR{$\Theta = \{ \theta_t \}$},
\GCPR{$\Psi   = \{ \psi_t \}$},
\GCPR{$\Gamma = \{ \gamma_t \}$},
are the body poses, facial expressions and translations,
\GCPR{$\Xi = \{ \xi_t \}$} is the object rotations and translations,
\GCPR{$\mathcal{S} = \{ S_t \}$ and $\mathcal{D} = \{ D_t \}$ are masks and
depth patches,
and}
\GCPR{$\mathcal{J}_{\mathit{est}} = \{ {J}_{\mathit{est}, t} \}$ are detected 2D\xspace keypoints.}
$M$     is the object mesh, and
\GCPR{$\beta^*$ the mean body shape}.
$E_\mathcal{C}$ encourages \textcolor{black}{body}-object contact for frames in contact, which are indicated by the
\GCPR{manually annotated binary vectors
$Q = \{
\textcolor{black}{Q_t}
\}$, \textcolor{black}{$t=\{1, \dots, T \}$};
$Q_t$ is set to 1 if in the $t^\mathrm{th}$ frame \textcolor{black}{any body part (\mbox{e.g.}\xspace, hand, foot, thighs)} is in contact with the object, and set to 0 otherwise}.
The motion smoothness loss $E_{\mathcal{S}}$ penalizes abrupt position changes for body vertices, and the vertex acceleration loss $E_{\mathcal{A}}$ encourages smooth object trajectories.
We estimate the ground plane surface by fitting a plane to chosen floor \textcolor{black}{points in the observed point clouds}.
The terms $E_{\mathcal{G}}$ and $E_{\mathcal{G'}}$ measure whether the body and object
\GCPR{penetrate the ground, respectively.}
\textcolor{black}{For more details on the above loss terms}, please see \textcolor{black}{{Appx.}}\xspace
\textcolor{black}{Finally, the parameters
$\lambda_{\mathcal{G}}$,
$\lambda_\mathcal{Q}$,
$\lambda_\mathcal{S}$, and
$\lambda_{\mathcal{A}}$
are steering weights that are set empirically.}
\section{\intercap Dataset}   \label{sec:dataset}

We use the proposed \intercap algorithm (\textcolor{black}{Sec.~}\ref{sec:method}) to capture the \intercap dataset, which uniquely features whole-body interactions with objects in multi-view \mbox{RGB-D}\xspace videos.
\medskip

\textbf{Data-capture Protocol.}
\REBT{We use {\ourNUmbColor{10}}\xspace \textcolor{black}{everyday} objects, shown in \fig{fig:object_photos_annotations}\textcolor{black}{-left}, that vary in  size  and ``afford'' different interactions with the body, hands or feet}; \textcolor{black}{we focus mainly on hand-object interactions}. We recruit \ourNUmbColor{10}\xspace subjects (\ourNUmbColor{5}\xspace males and \ourNUmbColor{5}\xspace females) that are \textcolor{black}{between} 25-40 years old. The subjects are recorded \textcolor{black}{while} interacting with $7$ or more objects, \textcolor{black}{according to their time availability}. Subjects are instructed to interact with objects as naturally as possible. However, \REBT{\textcolor{black}{they are asked to}
avoid very fast interactions that cause severe motion blur ({Azure Kinect}\xspace supports only up to $30$ FPS), \textcolor{black}{or misalignment between the \mbox{RGB}\xspace and depth images for each Kinect (due to technicalities of \mbox{RGB-D}\xspace sensors)}.} We capture up to 3 sequences per object \textcolor{black}{depending on object shape and functionality, and by picking} an interaction intent from the list below, \textcolor{black}{as in {\mbox{GRAB}}\xspace~\cite{taheri2020grab}:}
\begin{itemize}
    \item
    "\textbf{Pass}": The subject \textcolor{black}{passes} the object \textcolor{black}{on} to another imaginary person \textcolor{black}{standing on their left/right side; a graspable area needs to be free for the other person to grasp.}
    \item
    "\textbf{Check}": The subject \textcolor{black}{inspects visually} the object \textcolor{black}{from several viewpoints by first picking it up and then manipulating it with their hands to see several sides of it.}
    \item
    "\textbf{Use}": The subject \textcolor{black}{uses} the object in a \textcolor{black}{natural} way \textcolor{black}{that ``agrees'' with the object's affordances and functionality for everyday tasks.}
\end{itemize}
We also capture each subject performing a freestyle interaction of their choice. All subjects gave informed written consent to \textcolor{black}{publicly} share their data for research.

\input{paper_FIG_XX_sequence_reconstr}

\medskip

\textbf{4D Reconstruction.} Our \intercap method (\textcolor{black}{Sec.~}\ref{sec:method}) takes as input multi-view \mbox{RGB-D}\xspace videos and outputs 4D\xspace meshes for the human and object, \mbox{i.e.}\xspace, 3D\xspace meshes over time. Humans are represented as \smplx meshes \cite{pavlakos2019expressive}, while object meshes are acquired with an Artec hand-held scanner. \REBT{Some dataset frames along with the reconstructed meshes are shown in \fig{fig:teaser} and \fig{fig:dataset_sequences}; see also \textcolor{black}{the video on our website}.} Reconstructions look natural, with plausible contact between the human and the object.

\medskip

\REBT{\textbf{Dataset Statistics.}} \intercap has {\ourNUmbColor{223}}\xspace~\mbox{RGB-D}\xspace videos with a total of {\ourNUmbColor{67,357}}\xspace multi-view frames (\ourNUmbColor{$6$}\xspace~\mbox{RGB-D}\xspace images each). \REBT{For a comparison with other datasets, see \tab{tab:dataset_compare}.}

\section{Experiments}   \label{sec:experiments}

\textbf{Contact Heatmaps.}
\Fig{fig:contact_heatmaps}\textcolor{black}{-left}
shows contact heatmaps on each object, across all subjects.
We follow the protocol of {\mbox{GRAB}}\xspace~\cite{taheri2020grab}, which uses a proximity metric on reconstructed human and object meshes.
First, we compute per-frame binary contact maps by thresholding \REBT{(at 4.5mm)}
the distances from each body vertex to the closest object surface point.
Then, we integrate these maps over time (and subjects) to get ``heatmaps'' encoding contact likelihood.
\intercap reconstructs human and object meshes accurately enough so that contact heatmaps agree with
object affordances, \mbox{e.g.}\xspace, the handle of the suitcase, umbrella and tennis racquet are likely to be grasped, the upper skateboard surface is likely to be contacted by the foot, and the upper stool surface
by the buttocks.

\Fig{fig:contact_heatmaps}\textcolor{black}{-right} shows heatmaps on the body, computed across all subjects and objects. Heatmaps show that most \textcolor{black}{of \intercap's} interactions involve mainly the right hand. Contact on the palm looks realistic, and is concentrated on the fingers and MCP joints. \textcolor{black}{The ``false''} contact on the dorsal side is attributed to our challenging \textcolor{black}{camera} setup \textcolor{black}{and interaction scenarios, as well as} some reconstruction jitter.

\input{paper_FIG_XX_penetration_plots}

\textbf{Penetration.}
We evaluate the penetration between human and object mesh\-es for all sequences of our dataset.
\REBT{We follow the protocol of {\mbox{GRAB}}\xspace~\mbox{et al.}\xspace~\cite{taheri2020grab}; we first find the ``contact frames'' for which there is at least minimal human-object contact, and then report statistics for these.} In \fig{fig:penetration_plots}\textcolor{black}{-left} we show the distribution of  penetrations, \mbox{i.e.}\xspace, the number of ``contact frames'' \textcolor{black}{(Y axis)} with a \textcolor{black}{certain}
mesh penetration depth \textcolor{black}{(X axis)}. In \fig{fig:penetration_plots}\textcolor{black}{-right} we show the cumulative distribution of penetration, \mbox{i.e.}\xspace, the percentage of ``contact frames'' \textcolor{black}{(Y axis)} for which mesh penetration is below a threshold \textcolor{black}{(X axis)}. Roughly $60\%$ of ``contact frames'' have $\leq5mm$, $80\%$ $\leq7$ mm, and  $98\%$ $\leq20$ mm \textcolor{black}{mean} penetration. The average penetration depth over all ``contact frames'' is $7.2$ mm.

\textbf{Fitting Accuracy.}
For every frame, we compute the distance from each mesh vertex to the closest point-cloud (PCL) point; for each human or object mesh we take into account only the respective PCL area obtained with {PointRend}\xspace~\cite{kirillov2020pointrend} segmentation. The mean vertex-to-PCL distance is 20.29 mm for the body, and 18.50 mm for objects. \REBT{In comparison, \mbox{\mbox{PROX-D}}\xspace \cite{hassan2019resolving}, our base method, achieves an error of 13.02 mm for the body. This is expected since \mbox{\mbox{PROX-D}}\xspace is free to change the body shape to fit each individual frame, while our method estimates a single body shape for the whole sequence. SMPLify-X \cite{pavlakos2019expressive} achieves an mean error of 79.54 mm, for VIBE the mean error is 55.59 mm, while ExPose gets an mean error of 71.78 mm. These numbers validate the effectiveness of our method for body tracking. Note that these methods are based on monocular \mbox{RGB}\xspace images only, so there is not enough information for them to accurately estimate the global position of the 3D\xspace body meshes. Thus we first align the output meshes with the point clouds, then compute the error.} \REBT{Note that the error is bounded from below for two reasons:
(1) it is  influenced by factory-design imperfections in the synchronization of {Azure Kinects}\xspace}, and
(2) some vertices reflect body/object areas that are occluded during interaction and their closest PCL point is a wrong correspondence. \REBT{Despite this, \intercap empirically estimates reasonable bodies, hands and objects in interaction,} \REBT{as reflected in the contact heatmaps and penetration metrics discussed above.}

\begin{figure}[t]
    \centering
    \includegraphics[trim=014mm 022mm 035mm 018mm, clip=true, height=0.274 \textwidth]{./RB_contact_rb.png}
    \quad
    \includegraphics[trim=000mm 008mm 014mm 012mm, clip=true, height=0.284 \textwidth]{./RB_acc.png}
    \caption{
    \REBT{
    \textbf{Left:}
    Qualitative ablation of our contact term.
    Each pair of images shows results wo/ (red) and w/ (green) the contact term.
    Encouraging
    contact results in more natural hand poses and hand-object grasps.
    \textbf{Right:}
    Acceleration
    of a random vertex w/ (dashed line) and wo/ (solid line) temporal smoothing for 3 sequences \textcolor{black}{(shown with different color)} over the first 120 frames.
    \textcolor{black}{Dashed lines (w/ temporal smoothing) correspond to lower acceleration, \mbox{i.e.}\xspace, less jitter.}
    }
    }
    \label{fig:contact}
\end{figure}

\REBT{
\textbf{Ablation of Contact Term.} \Fig{fig:contact}\textcolor{black}{-left} shows results with-/out our term that encourages body-object contact; \textcolor{black}{visualization``zooms'' into} hand-object grasps. We see that encouraging contact yields more natural hand poses and fewer interpenetrations. This is backed up by the contact heatmaps and penetration metrics discussed above.
}

\REBT{ \textbf{Ablation of Temporal Smoothing Term.} \Fig{fig:contact}\textcolor{black}{-right} shows results with-/out our temporal smoothing term. Each solid line shows the acceleration of a randomly chosen vertex without the temporal smoothness term; we show 3 different motions. The dashed lines of the same color show the same motions with the smoothness term; these are clearly smoother. } \REBT{We \textcolor{black}{empirically} find that the learned motion prior of Zhang \mbox{et al.}\xspace~\cite{zhang2021learning} produces a more natural motion than handcrafted ones \cite{huang2017towards}.
}

\textcolor{black}{\textbf{Discussion on Jitter.}} \textcolor{black}{Despite the smoothing, some jitter is still inevitable. We attribute this to two factors: (1) \mbox{OpenPose}\xspace and Mask-RCNN are empirically relatively sensitive to occlusions and illumination (\mbox{e.g.}\xspace, reflections, shadows, poor lighting); the data terms for fitting 3D\xspace models depend on these. (2) {Azure Kinects}\xspace have a reasonable synchronization, yet, there is still a small delay among cameras to avoid depth-camera interference; the point cloud ``gathered'' across views is a bit ``patchy'' as information pieces have a small time difference.} The jitter is more intense for hands relatively to the body, due to their low image resolution, motion blur, and coarse point clouds. \textcolor{black}{Despite these challenges, \intercap is a good step towards capturing everyday whole-body interactions with commodity hardware. } Future work will study advanced motion priors.

\section{Discussion}    \label{sec:conclusion}

Here we focus on \textcolor{black}{whole}-body human interaction with everyday rigid objects. We present a novel method, called \intercap, that reconstructs such interactions from multi-view full-body videos, including natural hand poses and contact with objects. With this method, we capture the \textcolor{black}{novel} \intercap dataset, \textcolor{black}{with} a variety of people interacting with several common objects. The dataset contains reconstructed 3D\xspace meshes for the \textcolor{black}{whole body} and the object over time \textcolor{black}{(\mbox{i.e.}\xspace, 4D\xspace meshes)}, as well as plausible contacts between them. In contrast to most previous work, our method uses no special devices like optical markers or IMUs, but only several consumer-level \mbox{RGB-D}\xspace cameras. Our setup is lightweight and has the potential \textcolor{black}{to be used} in daily scenarios. Our method estimates reasonable hand poses even when there is heavy occlusion between hands and the object. \textcolor{black}{In future work, we plan to study interactions with smaller objects and dexterous manipulation.} Our data and code are available at \mbox{\url{https://intercap.is.tue.mpg.de}}.

\section{Acknowledgements}
We thank Chun-Hao P. Huang, Hongwei Yi, Jiaxiang Shang, and Mohamed Hassan for helpful discussion of technical details. We thank Yuliang Xiu, Jinlong Yang, Victoria F. Abrevaya, Taylor McConnell, Galina Henz, Marku H{\"o}schle, Senya Polikovsky, Matvey Safroshkin and Tsvetelina Alexiadis for data collection and cleaning. We thank all the participants of our experiments, and Benjamin Pellkofer for IT and website support.

\noindent
\textcolor{black}{The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting OT. This work was supported by the German Federal Ministry of Education and Research (BMBF): T{\"u}bingen AI Center, FKZ: 01IS18039B.}

\noindent
\textcolor{black}{\textbf{Conflict of Interest.}}
Disclosure: \url{https://files.is.tue.mpg.de/black/CoI_GCPR_2022.txt}.

\section{Supplementary Video}
\section{Video \textcolor{black}{on our Website}}
The \textcolor{black}{narrated}
video \textcolor{black}{on our website} (\mbox{\url{https://intercap.is.tue.mpg.de}}) presents:
\begin{itemize}
    \item
    \textcolor{black}{An explanation of our motivation.}
    \item
    \textcolor{black}{An overview of our \intercap dataset and method.}
    \item
    Some \textcolor{black}{videos (input) and reconstructed 4D\xspace meshes}
    \textcolor{black}{(output)} of our \intercap dataset.
    \item A \textcolor{black}{qualitative} comparison between our \intercap mesh reconstructions to the ones from \smplifyx~\cite{pavlakos2019expressive}, \mbox{ExPose}\xspace~\cite{choutas2020monocular}, and \mbox{VIBE}\xspace~\cite{kocabas2020vibe}.
\end{itemize}

\section{Optimization \textcolor{black}{Objective Function \& Terms}}
We use the \textcolor{black}{objective} function of Eq.~\textcolor{red}{6} of the main paper to jointly refine (via optimization) the body and object motion over the whole sequence. \textcolor{black}{Here} we give a detailed explanation of the terms not elaborated in the main \textcolor{black}{paper} due to space limitations.

The \textcolor{black}{motion smoothness} term $E_{\mathcal{S}}$ penalizes sudden changes in the \textcolor{black}{position of body vertices}. \textcolor{black}{It employs the learned motion prior of LEMO~\cite{zhang2021learning}} and
is defined as:
\begin{align}
\begin{split}
\textcolor{black}{
    E_{\mathcal{S}}(\Theta, \Psi, \Gamma, A; T, \beta^*) =
    \frac{1}{Q(T-2)} \sum_{t=1}^{T-1}
    \big \|
    z_{t+1}^{opt} - z_t^{opt}
    \big \|^2,
}
\end{split}
\end{align}
\textcolor{black}{where $T$ is the sequence length, $Q$ is a constant representing the number of virtual body-markers of LEMO (see \cite{zhang2021learning} for an explanation; they use a different symbol), $z_t^{opt}$ is the latent vector for the $t$-th frame from LEMO's pre-trained motion auto-encoder ($F_S$):
\begin{equation}
     Z^{opt} = F_S(X_{\Delta}^{opt}) = [z_1^{opt}, z_2^{opt}, ..., z_{T-1}^{opt}],
\end{equation}
where $X_{\Delta}^{opt}$ is a (concatenated) vector containing the temporal position change of LEMO's virtual body-markers.
For more details, please refer to the paper of LEMO~\cite{zhang2021learning}.}

The \textcolor{black}{vertex acceleration} term $E_{\mathcal{A}}$ \textcolor{black}{is a simple hand-crafted motion prior that encourages smooth motion trajectories for the object:}
\begin{align}
\begin{split}
    E_{\mathcal{A}}(\Xi; T, M) =
    \frac{1}{T-2} \sum_{t=2}^{T-1}
    \Big \|
    &
    W'(\Xi_{t-1}, M) +
    \\ &
    W'(\Xi_{t+1}, M) -
    2 * W'(\Xi_t, M)
    \Big \|^2
\end{split}
\end{align}
where $M$ is the \textcolor{black}{object mesh}, and $W'$ \textcolor{black}{denotes} the operation of \textcolor{black}{first} rigidly deforming the object according to $\Xi_t$ \textcolor{black}{and then} concatenating the vertices into a single vector.

The \textcolor{black}{contact} term $E_{\mathcal{C}}(\beta^*, \Theta_t, \Psi_t, \Gamma_t, \Xi_t, M)$ \textcolor{black}{encourages} the \textcolor{black}{annotated likely contact areas of the body (see Fig.~\textcolor{red}{3} of the main paper)} to be in contact with the
\textcolor{black}{object:}
\begin{align}
\begin{split}
    E_{\mathcal{C}}(\beta^*, \Theta_t, \Psi_t, \Gamma_t, \Xi_t, M) =
        CD
        \Big (
        & H  \big(
                    W(\Theta_{t}, \Psi_{t}, \Gamma_{t}, A, \beta^*)
             \big), \\
        & H' \big(
                    W'(\Xi_{t}, M)
             \big)
        \Big )
\end{split}
\end{align}
where
$\text{CD}$ refers to the Chamfer Distance function,
\textcolor{black}{$H$ is a function that returns only the annotated body-contact vertices of Fig.~\textcolor{red}{3},  $H'$ returns the closest points on the object for these body-contact vertices, $W'$ deforms rigidly the object and is explained in the previous paragraph and $W$ similarly (non-rigidly) deforms the \smplx mesh and concatenates the vertices into a single vector.
}

\textcolor{black}{Finally, the ground-support terms $E_\mathcal{G}$ and $E_\mathcal{G'}$ build on} the fact that \textcolor{black}{no} human or object vertex, \textcolor{black}{respectively,} should be below the ground plane, \textcolor{black}{and penalize any vertex penetrating the ground}.
Let $p_\mathcal{G}$ be a point on the \textcolor{black}{ground} plane and $n_{\mathcal{G}}$ be the \textcolor{black}{corresponding} normal; \textcolor{black}{both defined are once and offline}.
Then the term $E_\mathcal{G}$ for body-ground penetration is defined as:
\begin{equation}
    E_{\mathcal{G}}(\beta^*, \Theta_t, \Psi_t, \Gamma_t) =
    \Big \|
    RL
    \Big (
            n_{\mathcal{G}} *
            \big (
                    p_{\mathcal{G}} - W(\beta^*, \Theta_t, \Psi_t, \Gamma_t)
            \big )
    \Big )
    \Big \|^2,
\end{equation}
where $RL$ is the ReLU function, and $*$ here is the inner  product of vectors.
The term $E_{\mathcal{G}'}$ for object-ground penetration follows a similar formulation:
\begin{equation}
    E_{\mathcal{G'}}(\Xi_t, M) =
    \Big \|
            RL( n_{\mathcal{G}} * ( p_{\mathcal{G}} - W'(\Xi_{t}, M) )  )
    \Big \|^2.
\end{equation}