To register pixel observations at different time instances,
BANMo maintains a canonical feature embedding that en-
codes semantic information of 3D points in the canonical
space, which can be uniquely matched by the pixel features,
and provide strong cues for registration via a joint optimiza-
tion of shape, articulations, and embeddings.
Canonical embeddings matching. Given a pixel at xt of
frame t, our goal is to find a point X∗ in the canonical space
whose feature embedding ψ(X∗) ∈ R16 best matches the
I (xt) ∈ R16. The pixel embed-
pixel feature embedding ψt
dings ψt
I (of frame t) are computed by a CNN. Different
from ViSER [65] that learns embeddings from scratch, we
initialize pixel embeddings with CSE [30,31] that produces

losses, and a 3D cycle-consistency regularization loss:
L =

+

+ L3D-cyc.

(cid:16)Lsil + Lrgb + LOF
(cid:124)

(cid:123)(cid:122)

reconstruction losses

(cid:17)
(cid:125)

(cid:16)Lmatch + L2D-cyc
(cid:124)

(cid:123)(cid:122)
BANMo- Building Animatable 3D Neural Models from Many Casual Videos