We extract a controllable model from a video of a person performing a certain activity. The model generates novel image sequences of that person, according to user-defined control signals, typically marking the displacement of the moving body. The generated video can have an arbitrary background, and effectively capture both the dynamics and appearance of the person. The method is based on two networks. The first maps a current pose, and a singleinstance control signal to the next pose. The second maps the current pose, the new pose, and a given background, to an output frame. Both networks include multiple novelties that enable high-quality performance. This is demonstrated on multiple characters extracted from various videos of dancers and athletes.
This paper proposes a method to address the interesting task, i.e. controllable human activity synthesis, by conditioning on the previous frames and the input control signal. To synthesis the next frame, a Pose2Pose network is proposed to first transfer the input information into the next frame body structure and object. Then, a Pose2Frame network is applied to generate the final result. The results on several video sequences look nice with more natural boundaries, object, and backgrounds compared to previous methods.
Pros:
1. The proposed Pose2Pose successfully transfer the pose conditioned on the past pose and the input control signal. The proposed conditioned residual block, occlusion augmentation and stopping criteria seem to help the Pose2Pose network work well. Besides, the object is also considered in this network, which makes the method generalized well to the videos where human holds some rigid object.
2. The Pose2Frame network is similar to previous works but learns to predict the soft mask to incorporate the complex background and to produce shallow. The mask term in Eq. (7) seems to work well for the foreground (body+object) and the shallow regions.
3. The paper is easy to follow.
Cons:
1. Since the method is only evaluated on several video sequences, I am not sure how the method will perform on other different scenes. Results on more scenes will make the performance more convincing. I also wonder if the video data will be released, which could be important for the following comparisons.
2. As to the results of the Pose2Pose network, I wonder if there are some artifacts that will affect the performance of the Pose2Frame network. Then, there will be another question: how the two networks are trained? Are they trained separately or jointly? I assume the authors first train the Pose2Pose network, then use the output to train the Pose2Frame network. Otherwise, the artifacts from Pose2Pose will affect the testing performance of the Pose2Frame network.
3. The mask term seems to work well for the shallow part. I wonder how the straightforward regression term plus the smooth term will perform for the mask. Here, the straightforward regression term means directly regress the output mask to the target densepose mask. Will the proposed mask term perform better?