summaries: First, the paper introduces the idea of using multiple predictions (e.g. M=25) for learning-based video coding. Second, to reduce the required motion bits, a flow prediction method using polynomial approximation is developed. The idea is reasonable, and the reported performance seems promising. However, the paper is written rather poorly, and how the algorithm is actually designed is not clear. Thus, the reported performance is not convincing either.  The authors introduce VLVC, a new neural video compression codec. Thanks to a new 3D motion compensation structure based on spatio-temporal interpolation, a single trained model can be run in arbitrary prediction modes (such as low-delay or random-access settings). In addition, each frame can be compressed with an arbitrary number of reference frames. There are a number of smaller changes to established neural video codecs. The resulting VLVC codec is demonstrated on standard datasets, showing good rate-distortion results (in particular when trained and evaluated on MS-SSIM). This paper introduces multi-frame references for learned video coding, which estimates multiple optical flows as voxel flows for motion compensation. With the weighted map, it can achieve weighted trilinear warping to combine the multiple predictions into  the final prediction. It is a good idea to obtain a better predcition to search information across multiple frames.  The author introduces  the polynominal motion modeling to use multiple backward optical flow to generate the single optical flow. Based on the method of softmax spaltting, they convert the backward flow into a forward flow using the flow reversal layer. Then use the forward flow and softmax spaltting for motion compensation. The overall architecture is a hybrid video coding system and the main contribution is inter coding. Residual coding is applied on the feature domain.  This paper proposes a neural video compression method that is designed around the idea of using a voxel flow instead of the more traditional pixel warping idea. The key difference here is that a volume of reference frames is used, and flow is computed to all of them. This allows the compression method to be quite liberal in terms of what information from the GOP can be used, yielding a better RD performance than methods that rely on a single reference frame.  Overall I think this is a step in the right direction, but I fear that such a method is completely impractical for the foreseeable future. Learned video compression is a promising field and has demonstrated that it can be as effective as the classic video compression method. In terms of rate distortion it has caught up with the classical methods greatly. However binding the prediction mode and fixed network is what hinders the field to go forward. This paper proposes a versatile learned video compression framework that learns one mode to predict all possible modes. The paper proposes versatile compression where the motion compensation applies 3D motion vector fields, trilinear warping. Motion estimation here acts as a decoupler of prediction modes away from framework design, allowing its dependence to be not affected by framework design. For multiple reference frame prediction, the network predicts the motion fields with a unified polynomial function. The flow prediction can lead to reduction of computation by reducing voxel flow transmission. 
abstractText: Learned video compression methods have demonstrated great promise in catching 1 up with traditional video codecs in their rate-distortion (R-D) performance. How2 ever, existing learned video compression schemes are limited by the binding of 3 the prediction mode and the fixed network framework. They are unable to support 4 various inter prediction modes and thus inapplicable for various scenarios. In this 5 paper, to break this limitation, we propose a versatile learned video compression 6 (VLVC) framework that uses one model to support all possible prediction modes. 7 Specifically, to realize versatile compression, we first build a motion compensation 8 module that applies multiple 3D motion vector fields (i.e., voxel flows) for weighted 9 trilinear warping in spatial-temporal space. The voxel flows convey the information 10 of temporal reference position that helps to decouple inter prediction modes away 11 from framework designing. Secondly, in case of multiple-reference-frame predic12 tion, we apply a flow prediction module to predict accurate motion trajectories with 13 a unified polynomial function. We show that the flow prediction module can largely 14 reduce the transmission cost of voxel flows. Experimental results demonstrate that 15 our proposed VLVC not only supports versatile compression in various settings but 16 also achieves comparable R-D performance with the latest Versatile Video Coding 17 (VVC) standard in terms of MS-SSIM. 18
references.authors: VNI Cisco
references.title: Cisco visual networking index: Forecast and trends
references.venue: 2017–2022. White Paper, 1
references.authors: Thomas Wiegand Gary J Sullivan Gisle Bjontegaard Ajay Luthra
references.title: Overview of the h. 264/avc video 318 coding standard
references.venue: IEEE Transactions on circuits and systems for video technology,
references.authors: Gary J Sullivan Jens-Rainer Ohm Woo-Jin Han Thomas Wiegand
references.title: Overview of the high efficiency 320 video coding (hevc) standard
references.venue: IEEE Transactions on circuits and systems for video technology,
references.authors: Jianle Chen
references.title: Versatile video coding (draft 9)
references.venue: Draft, JVET,
references.authors: Johannes Ballé Valero Laparra Eero P Simoncelli
references.title: End-to-end optimized image compression
references.venue: arXiv 324 preprint arXiv:1611.01704,
references.authors: Johannes Ballé David Minnen Saurabh Singh Sung Jin Hwang Nick Johnston
references.title: Variational image 326 compression with a scale hyperprior
references.venue: arXiv preprint arXiv:1802.01436,
references.authors: David Minnen Johannes Ballé George D Toderici
references.title: Joint autoregressive and hierarchical priors for 328 learned image compression
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Zhengxue Cheng Heming Sun Masaru Takeuchi Jiro Katto
references.title: Learned image compression with dis- 331 cretized gaussian mixture likelihoods and attention modules
references.venue: In Proceedings of the IEEE/CVF Conference 332 on Computer Vision and Pattern Recognition,
references.authors: Eirikur Agustsson Lucas Theis
references.title: Universally quantized neural compression
references.venue: Advances in Neural 334 Information Processing Systems,
references.authors: Zongyu Guo Zhizheng Zhang Runsen Feng Zhibo Chen
references.title: Soft then hard: Rethinking the quantization 336 in neural image compression
references.authors: Chao-Yuan Wu Nayan Singhal Philipp Krahenbuhl
references.title: Video compression through image interpolation
references.venue: Proceedings of the European Conference on Computer Vision (ECCV),
references.authors: Abdelaziz Djelouah Joaquim Campos Simone Schaub-Meyer Christopher Schroers
references.title: Neural inter- 340 frame compression for video coding
references.venue: In Proceedings of the IEEE International Conference on Computer
references.authors: Amirhossein Habibian Ties van Rozendaal Jakub M Tomczak Taco S Cohen
references.title: Video compression 343 with rate-distortion autoencoders
references.venue: In Proceedings of the IEEE International Conference on Computer
references.authors: Jerry Liu Shenlong Wang Wei-Chiu Ma Meet Shah Rui Hu Pranaab Dhawan Raquel Urtasun
references.title: Conditional entropy coding for efficient video
references.authors: Guo Lu Wanli Ouyang Dong Xu Xiaoyun Zhang Chunlei Cai Zhiyong Gao
references.title: Dvc: An end-to-end 348 deep video compression framework
references.venue: In Proceedings of the IEEE Conference on Computer Vision and 349 Pattern Recognition,
references.authors: Eirikur Agustsson David Minnen Nick Johnston Johannes Balle Sung Jin Hwang George 351 Toderici
references.title: Scale-space flow for end-to-end optimized video compression
references.venue: In Proceedings of the IEEE/CVF 352 Conference on Computer Vision and Pattern Recognition (CVPR),
references.authors: Jianping Lin Dong Liu Houqiang Li Feng Wu
references.title: M-lvc: Multiple frames prediction for learned video 354 compression
references.venue: arXiv preprint arXiv:2004.10290,
references.authors: Oren Rippel Alexander G Anderson Kedar Tatwawadi Sanjay Nair Craig Lytle Lubomir Bourdev
references.title: Elf-vc: Efficient learned flexible-rate video coding
references.venue: arXiv preprint arXiv:2104.14335,
references.authors: Ze Cui Jing Wang Bo Bai Tiansheng Guo Yihui Feng
references.title: G-vae: A continuously variable rate deep 358 image compression framework
references.venue: arXiv preprint arXiv:2003.02012,
references.authors: Ren Yang Fabian Mentzer Luc Van Gool Radu Timofte
references.title: Learning for video compression with 360 hierarchical quality and recurrent enhancement
references.venue: arXiv preprint arXiv:2003.01966,
references.authors: Guo Lu Chunlei Cai Xiaoyun Zhang Li Chen Wanli Ouyang Dong Xu Zhiyong Gao
references.title: Content 365 adaptive and error propagation aware deep video compression
references.venue: arXiv preprint arXiv:2003.11282,
references.authors: Haojie Liu Lichao Huang Ming Lu Tong Chen Zhan Ma
references.title: Learned video compression via joint 367 spatial-temporal correlation exploration
references.venue: arXiv preprint arXiv:1912.06348,
references.authors: Zhibo Chen Tianyu He Xin Jin Feng Wu
references.title: Learning for video compression
references.venue: IEEE Transactions on 369 Circuits and Systems for Video Technology,
references.authors: Ziwei Liu Raymond A Yeh Xiaoou Tang Yiming Liu Aseem Agarwala
references.title: Video frame synthesis 371 using deep voxel flow
references.venue: In Proceedings of the IEEE International Conference on Computer Vision,
references.authors: Simon Niklaus Long Mai Feng Liu
references.title: Video frame interpolation via adaptive convolution
references.venue: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,
references.authors: Fitsum A Reda Guilin Liu Kevin J Shih Robert Kirby Jon Barker David Tarjan Andrew Tao 376 Bryan Catanzaro
references.title: Sdc-net: Video prediction using spatially-displaced convolution
references.venue: In Proceedings of the 377 European Conference on Computer Vision (ECCV),
references.authors: Wenbo Bao Wei-Sheng Lai Xiaoyun Zhang Zhiyong Gao Ming-Hsuan Yang
references.title: Memc-net: Motion 379 estimation and motion compensation driven neural network for video interpolation and enhancement
references.venue: IEEE 380 transactions on pattern analysis and machine intelligence,
references.authors: Hyeongmin Lee Taeoh Kim Tae-young Chung Daehyun Pak Yuseok Ban Sangyoun Lee
references.title: Adacof: 382 adaptive collaboration of flows for video frame interpolation
references.venue: In Proceedings of the IEEE/CVF Conference 383 on Computer Vision and Pattern Recognition,
references.authors: Zhihao Shi Xiaohong Liu Kangdi Shi Linhui Dai Jun Chen
references.title: Video frame interpolation via generalized 385 deformable convolution
references.venue: IEEE Transactions on Multimedia,
references.authors: Simon Niklaus Feng Liu
references.title: Softmax splatting for video frame interpolation
references.venue: In Proceedings of the 387 IEEE/CVF Conference on Computer Vision and Pattern Recognition,
references.authors: Xiangyu Xu Li Siyao Wenxiu Sun Qian Yin Ming-Hsuan Yang
references.title: Quadratic video interpolation
references.venue: arXiv preprint arXiv:1911.00627,
references.authors: Deqing Sun Xiaodong Yang Ming-Yu Liu Jan Kautz
references.title: Pwc-net: Cnns for optical flow using pyramid, 391 warping, and cost volume
references.venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern 392 Recognition,
references.authors: Runsen Feng Yaojun Wu Zongyu Guo Zhizheng Zhang Zhibo Chen
references.title: Learned video compression 394 with feature-level residuals
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern 395 Recognition Workshops,
references.authors: Tianfan Xue Baian Chen Jiajun Wu Donglai Wei William T Freeman
references.title: Video enhancement with 397 task-oriented flow
references.venue: International Journal of Computer Vision,
references.authors: Alexandre Mercat Marko Viitanen Jarno Vanne
references.title: Uvg dataset: 50/120fps 4k sequences for video 399 codec analysis and development
references.venue: In Proceedings of the 11th ACM Multimedia Systems Conference,
references.authors: Zhou Wang Eero P Simoncelli Alan C Bovik
references.title: Multiscale structural similarity for image quality
references.venue: The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers,
references.authors: Diederik P Kingma Jimmy Ba
references.title: Adam: A method for stochastic optimization
references.venue: arXiv preprint 405
sections.text: Learned video compression methods have demonstrated great promise in catching1 up with traditional video codecs in their rate-distortion (R-D) performance. How-2 ever, existing learned video compression schemes are limited by the binding of3 the prediction mode and the fixed network framework. They are unable to support4 various inter prediction modes and thus inapplicable for various scenarios. In this5 paper, to break this limitation, we propose a versatile learned video compression6 (VLVC) framework that uses one model to support all possible prediction modes.7 Specifically, to realize versatile compression, we first build a motion compensation8 module that applies multiple 3D motion vector fields (i.e., voxel flows) for weighted9 trilinear warping in spatial-temporal space. The voxel flows convey the information10 of temporal reference position that helps to decouple inter prediction modes away11 from framework designing. Secondly, in case of multiple-reference-frame predic-12 tion, we apply a flow prediction module to predict accurate motion trajectories with13 a unified polynomial function. We show that the flow prediction module can largely14 reduce the transmission cost of voxel flows. Experimental results demonstrate that15 our proposed VLVC not only supports versatile compression in various settings but16 also achieves comparable R-D performance with the latest Versatile Video Coding17 (VVC) standard in terms of MS-SSIM.18
  1 Introduction19
  Video occupies more than 80% of network traffic and the amount of video data is increasing rapidly20 [1]. Thus, the storage and transmission of video become more challenging. A series of hybrid video21 coding standards have been proposed, such as AVC/H.264 [2], HEVC/H.265 [3] and the latest video22 coding standard VVC/H.266 [4]. These traditional standards are manually designed, evolving for23 decades. However, the development within the hybrid coding framework is gradually saturated.24 Recently, the performance of video compression is mainly improved by designing more complex25 prediction modes, leading to increased coding complexity.26
  Deep neural networks are currently promoting the development of data compression. Despite the27 remarkable progress on the field of learned image compression [5–10], the area of learned video28 compression is still in early stages. Existing methods for learned video compression can be grouped29 into three categories, including frame interpolation-based methods [11, 12], 3D autoencoder-based30 methods [13, 14], and predictive coding methods with optical flows [15–17]. So far, among them,31 video compression with optical flow presents the best performance [18], where optical flow represents32 pixel-wise motion vector (MV) fields utilized for inter frame prediction. In this paper, we also focus33 on this predictive coding architecture. Previous works with optical flow are proposed to support34 specific prediction mode, including unidirectional or bidirectional, single or multiple frame prediction.35 They are too cumbersome to support versatile compression in various settings since they bind the inter36 prediction mode with the fixed network framework. It is important to design a more flexible model to37 handle all possible settings like traditional codecs. In this paper, we propose a versatile learned video38
  Submitted to 35th Conference on Neural Information Processing Systems (NeurIPS 2021). Do not distribute.
  compression (VLVC) framework that achieves coding flexibility as well as compression performance.39 A voxel flow based motion compensation module is adopted for higher flexibility, which is then40 extended into multiple voxel flows to perform weighted trilinear warping. In addition, in case of41 multiple-reference-frame prediction, a polynomial motion trajectories based flow prediction module42 is designed for better compression performance. Our motivations are described as follows.43
  Motion compensation with multiple voxel flows. Previous works such as [15] apply 2D optical44 flows for low-delay prediction using single reference frame (unidirectional prediction, see Fig. 1a).45 For the practical random access scenario, bidirectional reference frames are available for more46 accurate frame interpolation [12] (Fig. 1b). However, the reference positions in these works are47 determined by pre-defined prediction modes. They cannot adapt to various inter prediction modes48 where reference positions are various. In this paper, we apply 3D voxel flows to describe not only49 the spatial MVs, but also the information of temporal reference positions (Fig. 1c). We perform50 voxel flow based motion compensation via trilinear warping, which is applicable to single or multiple,51 unidirectional or bidirectional reference frames. Unlike [16] that adopts scale space flow with trilinear52 warping, we apply voxel flows for inter prediction in spatial-temporal space, which naturally renders53 our model more robust to different coding scenarios. Furthermore, beyond using single MV in every54 position of the current frame, we propose to use multiple voxel flows to describe multiple possible55 reference relationships (Fig. 1d). Then the target pixel is synthesized by weighted fusing of the56 warping results. We show that without increasing the coding cost of motion information, the motion57 compensation is thus more accurate, yielding less residuals and more efficient compression.58
  Flow prediction with polynomial motion trajectories. Exploiting multiple reference frames usu-59 ally achieves better compression performance since more reference information is provided. A60 versatile learned video compression model should cover this multi-reference case. While previous61 work [17] designs a complex flow prediction network to reduce the redundancies of 2D MV fields, the62 number and structure of reference frames are inherent and fixed within the framework. In this paper,63 we design a more intelligent method for flow prediction, i.e., modelling the prediction modes with64 polynomial coefficients. We formulate different motion trajectories in a time interval by a unified65 polynomial function. The polynomial coefficients are solved by establishing a multivariate equation66 (see Section 3.2). Since this polynomial function models the accurate motion trajectories, it serves as67 a basic discipline that constrains the predicted motion to be reasonable. We show the transmission68 cost of voxel flows is reduced obviously with the help of additional motion trajectory information.69
  Thanks to the above two technical contributions, our proposed VLVC is not only applicable for various70 practical compression scenarios with different inter prediction modes, but also delivers impressive71 R-D performance on standard test sequences. Extensive experimental results demonstrate that our72 method is the first one to achieve comparable performance with VVC in terms of MS-SSIM in both73 low delay and random access configurations. Comprehensive ablation studies and discussions are74 provided to verify the effectiveness of our method.75
  2 Related Work76
  Learned Image Compression Recent advances in learned image compression [5–7], have shown77 the great success of nonlinear transform coding. Many existing methods are built upon hyperprior-78
  based coding framework [6], which are improved with more efficient entropy models [7, 8], variable-79 rate compression [19] and more effective quantization [9, 10]. While the widely used autoregressive80 entropy models provide significant performance gain in image coding, the high decoding complexity81 is not suitable for practical video compression. We thus employ the hyperprior model [6] without82 context models in our video compression framework.83
  Learned Video Compression Existing approaches [11–15, 17, 18, 20–24] can be roughly divided84 into three categories: frame interpolation-based methods [11, 12, 24], 3D autoencoder-based methods85 [13, 14], and predictive coding methods with optical flows [15–17]. Currently, researchers are more86 interested in the latter two methods. Although 3D autoencoder-based methods requires less time87 complexity, they barely achieve comparable performance with x265 in MS-SSIM [14]. Meanwhile,88 predictive coding methods with optical flows have outperformed HM in terms of PSNR [18].89
  Predictive-based video compression approaches [15, 20–24, 11, 12, 17] sequentially perform motion90 estimation, motion compression, motion compensation and residual compression. Chen et al. [24]91 first propose to predict block of pixels using learned neural network (DNN), and the residual is92 compressed by a RNN-based autoencoder. Wu et al. [11] propose a interpolation-based approach93 using traditional MVs. Lu et al. [15] propose an fully end-to-end trainable framework, where all key94 components in the classical video codec are implemented with neural networks. Rippel et al. [21]95 jointly compress the motion and residual information, and propose a latent state to memorize the96 information from the past. Djelouah et al. [12] perform interpolation by the decoded optical flow97 and blending coefficients. They reuse the same autoencoder of I-frame compression and directly98 quantize the corresponding latent space residual. Liu et al. [23] combine the optical flow estimation99 and motion compression networks into one-stage, and remove the redundancy of quantized flow100 representations using joint spatial-temporal priors. Yang et al. [20] propose a video compression101 framework with three hierarchical quality layers and recurrent enhancement. In [17], multiple frames102 motion prediction are introduced into the P-frame coding. Lu et al. [22] propose an content adaptive103 and error propagation aware method to reduce error accumulation and achieve adaptive coding.104 Agustsson et al. [16] replace the bilinear warping operation with scale-space flow which allows105 the model adaptively blur the reference content for better warping results. However, most existing106 methods are designed for particular prediction modes, resulting in inflexibility for different scenarios.107
  Video Interpolation The task of video interpolation is closely related to video compression. One108 pioneering work [25] proposes to use deep voxel flow to synthesize new video frames. Some works109 of video interpolation [26–28] directly generate the spatially-adaptive convolutional kernels for each110 motion vectors by neural networks. Most recently, [29, 30] proposed to relax the kernel shape,111 allowing the models to freely select multiple sampling points in space or space-time. In this paper,112 our employed multiple voxel flows is motivated by the accurate interpolation result in [30].113
  3 Versatile Learned Video Compression114
  To compress video, the original video sequence is first divided into groups of pictures (gop). Let115 x = {x1,x2, ...,xT } denote the frames in one gop unit where the gop size is T . To take advantage116 of previous decoded frames, our model predicts the current frame xt from n reference frame(s),117 i.e., the lossy reconstruction results compared to the original frames. Here, we denote the reference118 frames as {x̂t1 , x̂t2 , ..., x̂tn}, where {t1, t2, ..., tn} is the index of temporal reference position. If119 multiple frames are taken as the reference (i.e., n > 1), the reference frames are divided into two120 groups: one is used only for flow prediction, and the other is used for both flow prediction and motion121 compensation. In other words, the reference involved for motion compensation is only a sub-set of122 {x̂t1 , x̂t2 , ..., x̂tn}, which could be concatenated into a volume denoted by X̂t. If only one reference123 frame is available, the volume for warping X̂t = {x̂t−1}.124 An overview of our video compression framework is shown in Fig. 2. Previous work [16] demon-125 strates that an implicit flow encoder can outperform a pre-trained optical flow network and simplify126 the network structures simultaneously. In our paper, we also abandon the use of a pre-trained optical127 flow network in motion encoder. The motion encoder and decoder are similar to image compression128 network [5]. While the work of [16] sends current frame and previous reconstruction into motion129 encoder, we make some modifications on the input of motion encoder. Specifically, in our framework,130 the motion encoder is fed with the current frame xt concatenated with predicted frames (represented131
  as x′t in Fig. 2). Here, the predicted frame x ′ t is an estimation of current frame xt. The flow prediction132 module will predict 2D optical flows ft→ti to warp corresponding reference frames x̂ti , each of133 which will generate a predicted frame x′ti . The predicted frames reveal how much information the134 decoder knows about current frame.135
  On the decoder side, a motion decoder will generate voxel flows which is used for motion compensa-136 tion via trilinear warping (details are explained in Section 3.1). In addition, when multiple reference137 frames are available, the flow prediction module is turned on (the red dashed box). As shown in138 Fig. 2, the flow prediction module is auxiliary for motion encoding and decoding, which reduces139 the transmission cost of the quantized motion latent m̂t. The specific mechanism of flow prediction140 can be found in Section 3.2. After motion compensation, we obtain a prediction of current frame as141 x̄t. The residual encoder and decoder are then used to compress the remaining residuals between the142 original frame xt and the predicted frame x̄t, yielding the final reconstructed frame x̂t and quantized143 latent r̂t.144
sections.heading: 3.1 Motion compensation with multiple voxel flows145
sections.text: Voxel flow [25] is a per-pixel 3-D motion vector that describes relationships in spatial-temporal146 domain. Compared to 2D optical flow, voxel flow can inherently allow the codec to be aware of147 the sampling positions in the temporal dimension for various prediction modes. Given arbitrary148 number of reference frames, the model is expected to select the optimal reference frame for better149 reconstructing the current frame to be compressed. Such a 3-D motion descriptor helps to build a150 prediction-model-agnostic video compression framework, i.e., versatile learned video codec.151
  In addition, single flow field is hard to represent complex motion (e.g. blurry motion), which may152 result in inaccurate motion compensation or high coding cost of motions. When reconstructing a153 local region, its reference information may not come from only one source. Considering a practical154 scene where multiple objects of the same types appear at the same time, more than one areas could be155 referred for reconstructing the local region. Thereby, in this work, we further propose to use multiple156 voxel flows to perform weighted trilinear warping by sampling in Xt for multiple times. We remind157 our readers that Xt is a volume consisting of some reference frames. Assume the dimension of158 Xt is D × H ×W (usually reshaped into H ×W × D for warping), where D is the number of159 reference frames used for motion compensation. the motion decoder will generate multiple voxel160 flows by outputting a (4M) ×H ×W tensor. Here, M refers to the number of flows. Therefore,161 every voxel flow is a 4-channel field that describes the 3-channel voxel flow gi = (gix, g i y, g i z) with a162 corresponding weight channel giw. Here, i (1 ≤ i ≤M ) is the index of voxel flow. To synthesize the163 target pixels in current frame, the weights giw are normalized by a softmax function across M voxel164
  flows. We finally obtain the target pixel x̄[x, y] in spatial location [x, y] by calculating the weighted165 sum of sampling results, formulated as:166
  x̄[x, y] = M∑ i=1 giw(x, y)Xt[x+ g i x(x, y), y + g i y(x, y), g i z(x, y)]. (1)
  We experimentally find that compared with single voxel flow, the transmission cost of multiple voxel167 flows does not increase largely. The model is able to assign appropriate number of flows under the168 rate-distortion optimization goal. In other word, the model is optimized to avoid the transmission of169 unnecessary flows. Meanwhile, due to more accurate inter frame prediction, the transmission cost of170 residuals decreases obviously by using multiple voxel flows for weighted warping.171
  3.2 Generalized optical flow prediction172
  In our proposed VLVC framework, as illustrated in Fig. 2, we compress the spatial-temporal motion173 information via motion encoder and decoder. The concatenation of the predicted frames (i.e., bilinear174 warping results using the predicted optical flow) and the current frame are fed into the motion encoder.175 The 3-D motion descriptor, i.e., the voxel flows, are then decoded by the motion decoder given176 the quantized motion latent and the feature of predicted optical flow. In this process, the predicted177 optical flow reduces the spatial displacement need to be encoded, and also serves as the conditions to178 promote the generation of voxel flows. Thus, the optical flow prediction is clearly of great importance179 to reduce the redundancies of consecutive voxel flows in case of using multiple reference frames.180
  Specifically, there are two optical flow describe the motion between the reference frame x̂tj and181 the target frame xt: ftj→t and ft→tj . The flow ft→tj describe the motion of each pixel from182 xt, and therefore we can sample x̂tj for each pixel in the target frame xt via bilinear (backward)183 warping. However, ft→tj is unknown at decoder side because the pixels of target frame is unavailable.184 Fortunately, the pixels from reference frames are known at both encoder and decoder. We can first185 estimate the optical flow of pixels from a reference frame bolx̂tj to other reference frames, and then186 predict the flow ftj→t. While we obtain ftj→t, it cannot be directly used for motion compensation187 with bilinear warping.188
  Recently work [31] for video interpolation proposed a forward warping method to interpolate the189 target frame xt by directly using the flow ftj→t. For video compression, we aim to predict a190 approximation of the flow ft→tj to reduce the redundancies of the proposed voxel flows for better191 rate-distortion performance. We therefore employ the forward warping method [31] (named softmax192 splatting) to project the flow ftj→t to ft→tj , which is a kind of flow reversal methods similar to [32].193 In the following part, we will describe a novel polynomial motion modeling method to predict ftj→t194 given arbitrary reference frames and any target time stamp t. And a flow reversal layer based on195 softmax splatting is introduced for the final flow prediction.196
  Polynomial motion modeling For each pixel at tj , we model the motion ftj→t by the k-order197 (k < n) polynomial functions:198
  ftj→t = a1 × (t− tj) + a2 × (t− tj)2 + ...+ ak × (t− tj)k, (2)
  where a0, a1, ..., ak are the polynomial coefficients. To solve the coefficients, we set t equals to the199 top-k nearest time stamp {tji}ki=1 around tj within the set of reference time stamp. Then we can200 obtain the following equation:201
  a1a2... ak  = (tj1 − tj) (tj1 − tj) 2 ... (tj1 − tj)k (tj2 − tj) (tj2 − tj)2 ... (tj2 − tj)k ... ... ... (tjk − tj) (tjk − tj)2 ... (tjk − tj)k  −1  ftj→tj1 ftj→tj2 ... ftj→tjk  (3) where ftj→tj1 , ftj→tj2 , ..., ftj→tjk can be obtained using off-the-shelf flow estimation network.202 Then we can derive the polynomial coefficients and apply them to Eq. (3) predict the forward flow203 from tj to any time stamp t.204
  Flow reversal via softmax splatting While the forward flow f tj→t is predicted by the polynomial205 functions, it cannot be directly used for motion compensation. Therefore, we introduce a flow reversal206 layer to forward warping −f tj→t by softmax splatting [31]:207
  f t→tj = →∑ (exp(Z) · (−f tj→t),f tj→t) →∑
  (exp(Z),f tj→t) , (4)
  where →∑
  is the summation splatting defined in [31], and Z is an importance mask generated from a208 small network q as:209
  Z = q(x̂tj ,− 1
  k k∑ i=1 ‖x̂tj − ← w(x̂ti ,f tj→ti)‖1), (5)
  where ← w is the bilinear backward warping operator.210
  3.3 Loss function211
  In previous works, the reference frames are determined according to pre-defined prediction modes.212 For example, the work of [17] applies four unidirectional reference frames, where the reference set213 is {x̂t−4, x̂t−3, x̂t−2, x̂t−1}. The work of [12] applies {(x̂t−1, x̂t+1), (x̂t−2, x̂t+2), (x̂t−3, x̂t+3)}214 as the reference set for bilinear prediction. In this paper, to optimize a versatile video compression215 model, the model will have access to various reference structures during training to adapt to different216 prediction modes. Therefore, we apply the loss function to cover all the frames in the entire gop as:217
  L = 1 T T∑ t=1 [Rt(m̂t, r̂t|x̂ti , ..., x̂tj ) + λ · D(xt, x̂t|x̂ti , ..., x̂tj )]. (6)
  Here, T is the gop size during training. The maximum value of T is seven in our experiments since a218 7-frame gop can cover most prediction modes. {x̂ti , ..., x̂tj} represents different reference set that219 may vary in different mini-batches. Rt(m̂t, r̂t) is the rate of motion and residual. For simplicity, we220 omit the process of intra frame compression (at t = 1) in this loss function.221
  4 Experiments222
  4.1 Experimental setup223
  Model details The motion/residual compression modules are two auto-encoder style networks,224 where the bit-rate are estimated by the factorized and hyperprior entropy model [6, 7], respectively. We225 employ the off-the-shelf PWC-net [33] as the optical flow estimation network only in our generalized226 flow prediction module. We employ feature residual coding [34] instead of pixel residual coding for227 better performance. Detailed architecture can be found in supplementary.228
  Training sets The models were trained on the Vimeo-90k septuplets dataset [35] which consists of229 89800 video clips with diverse content. The video clips are randomly cropped to 128 × 128 or 256 ×230 256 pixel for training.231
  Testing sets The HEVC common test sequences [3] and the UVG dataset [36] are used for evalua-232 tion. The HEVC Classes B,C,D and E contain 16 videos with different resolution and content. The233 UVG dataset contains seven 1080p HD video sequences with 3900 frames in total.234
  Implementation details We optimize four models for MSE and four models for MS-SSIM [37].235 The video clip length T is set to 7 for training. We use the Adam optimizer [38] with batch size of 8236 and a initial learning rate of 5× 10−5. It is difficult to stably train the whole models from scratch. We237 first separately pre-train the intra-frame coding models and inter-frame coding models for MSE, with238 128× 128 video crops and 1,200,000 training steps. Then we jointly optimize both the models with239 the gop loss Eq. (6) for 100,000 steps using different metrics and λ values. Finally, we fine-tuning all240 the models for 20, 000 with a crop size of 256× 256 and a reduced learning rate of 1× 10−5241
  Evaluation Setting We measure the quality of reconstructed frames using PSNR and MS-242 SSIM [37] in the RGB colorspace. The bits per pixel (bpp) is used to measure the average number243 of bits. We compare our method with the traditional video coding standards H.265/HEVC and244 H.266/VVC, as well as the state-of-art learning based methods including [15, 22, 11, 12, 17].245
  Recent works for learned video compression usually evaluate H.265 by using FFmpeg, with perfor-246 mance is much lower than official implementation. In this paper, we evaluate H.265 and H.266 by247 using the implementation of the standard reference software HM 16.21[39] and VTM 12.0[40], respec-248 tively. We use the default low delay and random access configuration, and modify the gop structure249 and key frame interval for fair comparision. Detailed configuration can be found in supplementary.250
  4.2 Performance251
  We evaluate our model with the state-of-the-art learned video compression approaches, including the252 P-frame based methods of [15, 22, 23, 17], the interpolation based methods of [11, 12]. As shown253
  in Fig. 4, it can be observed that our proposed method significantly outperforms exiting learned254 video compression methods in both PSNR and MS-SSIM. Note that the VLVC (randomaccess) and255 VLVC (lowdelay) are two different configurations from the same models. Besides, our model is the256 first end-to-end learned video compression method that achieves comparable R-D performance with257 H.266 in terms of MS-SSIM.258
  4.3 Ablation Study and Analysis259
  All the models reported in ablation studies are trained for MSE using 128× 128 video clips. More260 ablation study results and visual results can be found in the supplementary.261
  The effect of the voxel flow number As shown in Fig. 5a, the number M of voxel flows signifi-262 cantly influence the overall rate-distortion performance. More voxel flows provide more possible263 sampling location for accurate motion compensation. Our proposed weighted trilinear warping with264 multiple voxel flows achieves about 1dB gain compared with the conventional trilinear warping with265 single voxel flow. Note that the performance gain is nearly saturated for M = 25, which is used as266 the default value in our models.267
  We also investigate the additional bitrate cost of multiple voxel flows. As shown in Fig. 5b, the268 proportion of multiple voxel flows in the total bitrate of video coding increases about 13 at the same269 bitrate. In other words, our model can learn to improve the overall compression performance by270 transmitting a proper amount of additional motion information, which is represented as voxel flows.271
  Versatile coding configurations The proposed methods can deal with a various set of prediction272 modes. To evaluate the effectiveness of coding flexibility as well as the effectiveness of the proposed273 generalized flow prediction module, we simply change the input coding configurations of the same274 trained model at different bitrate points. Random access and low delay coding settings are denoted275 as RA and LD, respectively. As shown in Fig. 5c, the coding mode RA with bidirectional reference276 frames achieves a compression gain of about 0.4dB, compared with the unidirectional coding modes277 LD. Furthermore, the performance dropped about 0.1dB~0.3dB when we turn off the generalized278 flow prediction module for different coding settings, noted as w/o GFP. We also illustrate the bitrate279 reduction of the voxel flows shown in Fig. 5b, where M=25 reduce the bitrate of voxel flows about 16280 compared to M=25 w/o GFP. Finally, we change the number of the reference frames for warping,281 which is set to 2 as default. We reduce the number to 1 in random access mode, noted as RA (r=1),282 which performance is even worse than low delay setting.283
  Visualization of voxel flows The proposed voxel flows contain multiple 3-channel voxel flows284 {(gix, giy, giz)}Mi=1 and their weights {giw}Mi=1. We separately visualize the weighted temporal and285 spatial flow maps. The mean temporal flow map ḡz = ∑ i g i w · giz describes the weighted centroid of286 voxel flows along the time axis. As shown in the fourth column of Fig. 6a, the ḡz performs like a287 occlusion map for bidirectional frame prediction. The pixels in black area cannot be found in the288
  first reference frame because the basketball player in red covers the background. Hence the voxel289 flows pay more attention on the second frame, resulting in large weights. The white area can be290 explained in the similar way for the first frame, and the gray area means that the voxel flows pay291 equally attention for both frames. For unidirectional frame prediction, the ḡz generated by the same292 model are almost black everywhere, demonstrating the flexibility of trilinear warping for different293 prediction mode.294
  We also visualize the weighted mean and weighted standard deviation of spatial flow maps (noted295 as mean spatial flow and std spatial flow) to investigate the spatial distribution of voxel flows. We296 first round and group the giz to the nearst integer location of reference frames (e.g. 0 or 1), then297 separately calculate the mean flow map and std flow map for each group of voxel flows. As shown in298 the second and fourth raws of Fig. 6, the grouped spatial mean of voxel flows has similar distribution299 with optical flow. Different from optical flow, the voxel flows have large variance in the area of300 motion, occlusion and blur, shown in the std spatial flow map. Single optical flow is not able to find a301 accurate reference pixel and results in inefficient motion compensation. Multiple flow warping with302 weighted coefficients provide a choice to perform motion compensation using multiple reference303 pixels with better rate-distortion performance.304
  5 Conclusion305
  In this paper, we propose a versatile learned video coding (VLVC) framework that allows us to train306 one model to support various inter prediction modes. To this end, we apply voxel flows as a motion307 information descriptor along both spatial and temporal dimensions, and we perform reconstruction308 via proposed weighted trilinear warping using voxel flows for more effective motion compensation.309 Through formulating different inter prediction modes by a unified polynomial function, we design a310 novel flow prediction module to predict accurate motion trajectories. In this way, we significantly311 reduce the bits cost of encoding motion information. Thanks to above novel motion compensation312 and flow prediction, VLVC not only achieve the support of different inter prediction modes but also313
  yield competitive R-D performance compared to conventional VVC standard, which fosters practical314 applications of learned video compression technologies.315
  References316 [1] VNI Cisco. Cisco visual networking index: Forecast and trends, 2017–2022. White Paper, 1, 2018.317
  [2] Thomas Wiegand, Gary J Sullivan, Gisle Bjontegaard, and Ajay Luthra. Overview of the h. 264/avc video318 coding standard. IEEE Transactions on circuits and systems for video technology, 13(7):560–576, 2003.319
  [3] Gary J Sullivan, Jens-Rainer Ohm, Woo-Jin Han, and Thomas Wiegand. Overview of the high efficiency320 video coding (hevc) standard. IEEE Transactions on circuits and systems for video technology, 22(12):321 1649–1668, 2012.322
  [4] Shan Liu Ye-Kui Wang Benjamin Bross, Jianle Chen. Versatile video coding (draft 9). Draft, JVET, 2020.323
  [5] Johannes Ballé, Valero Laparra, and Eero P Simoncelli. End-to-end optimized image compression. arXiv324 preprint arXiv:1611.01704, 2016.325
  [6] Johannes Ballé, David Minnen, Saurabh Singh, Sung Jin Hwang, and Nick Johnston. Variational image326 compression with a scale hyperprior. arXiv preprint arXiv:1802.01436, 2018.327
  [7] David Minnen, Johannes Ballé, and George D Toderici. Joint autoregressive and hierarchical priors for328 learned image compression. In Advances in Neural Information Processing Systems, pages 10771–10780,329 2018.330
  [8] Zhengxue Cheng, Heming Sun, Masaru Takeuchi, and Jiro Katto. Learned image compression with dis-331 cretized gaussian mixture likelihoods and attention modules. In Proceedings of the IEEE/CVF Conference332 on Computer Vision and Pattern Recognition, pages 7939–7948, 2020.333
  [9] Eirikur Agustsson and Lucas Theis. Universally quantized neural compression. Advances in Neural334 Information Processing Systems, 33, 2020.335
  [10] Zongyu Guo, Zhizheng Zhang, Runsen Feng, and Zhibo Chen. Soft then hard: Rethinking the quantization336 in neural image compression, 2021.337
  [11] Chao-Yuan Wu, Nayan Singhal, and Philipp Krahenbuhl. Video compression through image interpolation.338 In Proceedings of the European Conference on Computer Vision (ECCV), pages 416–431, 2018.339
  [12] Abdelaziz Djelouah, Joaquim Campos, Simone Schaub-Meyer, and Christopher Schroers. Neural inter-340 frame compression for video coding. In Proceedings of the IEEE International Conference on Computer341 Vision, pages 6421–6429, 2019.342
  [13] Amirhossein Habibian, Ties van Rozendaal, Jakub M Tomczak, and Taco S Cohen. Video compression343 with rate-distortion autoencoders. In Proceedings of the IEEE International Conference on Computer344 Vision, pages 7033–7042, 2019.345
  [14] Jerry Liu, Shenlong Wang, Wei-Chiu Ma, Meet Shah, Rui Hu, Pranaab Dhawan, and Raquel Urtasun.346 Conditional entropy coding for efficient video compression, 2020.347
  [15] Guo Lu, Wanli Ouyang, Dong Xu, Xiaoyun Zhang, Chunlei Cai, and Zhiyong Gao. Dvc: An end-to-end348 deep video compression framework. In Proceedings of the IEEE Conference on Computer Vision and349 Pattern Recognition, pages 11006–11015, 2019.350
  [16] Eirikur Agustsson, David Minnen, Nick Johnston, Johannes Balle, Sung Jin Hwang, and George351 Toderici. Scale-space flow for end-to-end optimized video compression. In Proceedings of the IEEE/CVF352 Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.353
  [17] Jianping Lin, Dong Liu, Houqiang Li, and Feng Wu. M-lvc: Multiple frames prediction for learned video354 compression. arXiv preprint arXiv:2004.10290, 2020.355
  [18] Oren Rippel, Alexander G Anderson, Kedar Tatwawadi, Sanjay Nair, Craig Lytle, and Lubomir Bourdev.356 Elf-vc: Efficient learned flexible-rate video coding. arXiv preprint arXiv:2104.14335, 2021.357
  [19] Ze Cui, Jing Wang, Bo Bai, Tiansheng Guo, and Yihui Feng. G-vae: A continuously variable rate deep358 image compression framework. arXiv preprint arXiv:2003.02012, 2020.359
  [20] Ren Yang, Fabian Mentzer, Luc Van Gool, and Radu Timofte. Learning for video compression with360 hierarchical quality and recurrent enhancement. arXiv preprint arXiv:2003.01966, 2020.361
  [21] Oren Rippel, Sanjay Nair, Carissa Lew, Steve Branson, Alexander G Anderson, and Lubomir Bourdev.362 Learned video compression. In Proceedings of the IEEE International Conference on Computer Vision,363 pages 3454–3463, 2019.364
  [22] Guo Lu, Chunlei Cai, Xiaoyun Zhang, Li Chen, Wanli Ouyang, Dong Xu, and Zhiyong Gao. Content365 adaptive and error propagation aware deep video compression. arXiv preprint arXiv:2003.11282, 2020.366
  [23] Haojie Liu, Lichao Huang, Ming Lu, Tong Chen, Zhan Ma, et al. Learned video compression via joint367 spatial-temporal correlation exploration. arXiv preprint arXiv:1912.06348, 2019.368
  [24] Zhibo Chen, Tianyu He, Xin Jin, and Feng Wu. Learning for video compression. IEEE Transactions on369 Circuits and Systems for Video Technology, 30(2):566–576, 2020.370
  [25] Ziwei Liu, Raymond A Yeh, Xiaoou Tang, Yiming Liu, and Aseem Agarwala. Video frame synthesis371 using deep voxel flow. In Proceedings of the IEEE International Conference on Computer Vision, pages372 4463–4471, 2017.373
  [26] Simon Niklaus, Long Mai, and Feng Liu. Video frame interpolation via adaptive convolution. In374 Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 670–679, 2017.375
  [27] Fitsum A Reda, Guilin Liu, Kevin J Shih, Robert Kirby, Jon Barker, David Tarjan, Andrew Tao, and376 Bryan Catanzaro. Sdc-net: Video prediction using spatially-displaced convolution. In Proceedings of the377 European Conference on Computer Vision (ECCV), pages 718–733, 2018.378
  [28] Wenbo Bao, Wei-Sheng Lai, Xiaoyun Zhang, Zhiyong Gao, and Ming-Hsuan Yang. Memc-net: Motion379 estimation and motion compensation driven neural network for video interpolation and enhancement. IEEE380 transactions on pattern analysis and machine intelligence, 2019.381
  [29] Hyeongmin Lee, Taeoh Kim, Tae-young Chung, Daehyun Pak, Yuseok Ban, and Sangyoun Lee. Adacof:382 adaptive collaboration of flows for video frame interpolation. In Proceedings of the IEEE/CVF Conference383 on Computer Vision and Pattern Recognition, pages 5316–5325, 2020.384
  [30] Zhihao Shi, Xiaohong Liu, Kangdi Shi, Linhui Dai, and Jun Chen. Video frame interpolation via generalized385 deformable convolution. IEEE Transactions on Multimedia, 2021.386
  [31] Simon Niklaus and Feng Liu. Softmax splatting for video frame interpolation. In Proceedings of the387 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5437–5446, 2020.388
  [32] Xiangyu Xu, Li Siyao, Wenxiu Sun, Qian Yin, and Ming-Hsuan Yang. Quadratic video interpolation.389 arXiv preprint arXiv:1911.00627, 2019.390
  [33] Deqing Sun, Xiaodong Yang, Ming-Yu Liu, and Jan Kautz. Pwc-net: Cnns for optical flow using pyramid,391 warping, and cost volume. In Proceedings of the IEEE Conference on Computer Vision and Pattern392 Recognition, pages 8934–8943, 2018.393
  [34] Runsen Feng, Yaojun Wu, Zongyu Guo, Zhizheng Zhang, and Zhibo Chen. Learned video compression394 with feature-level residuals. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern395 Recognition Workshops, pages 120–121, 2020.396
  [35] Tianfan Xue, Baian Chen, Jiajun Wu, Donglai Wei, and William T Freeman. Video enhancement with397 task-oriented flow. International Journal of Computer Vision, 127(8):1106–1125, 2019.398
  [36] Alexandre Mercat, Marko Viitanen, and Jarno Vanne. Uvg dataset: 50/120fps 4k sequences for video399 codec analysis and development. In Proceedings of the 11th ACM Multimedia Systems Conference, pages400 297–302, 2020.401
  [37] Zhou Wang, Eero P Simoncelli, and Alan C Bovik. Multiscale structural similarity for image quality assess-402 ment. In The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003, volume 2,403 pages 1398–1402. Ieee, 2003.404
  [38] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint405 arXiv:1412.6980, 2014.406
  [39] HEVC Offical Test Model HM. https://hevc.hhi.fraunhofer.de.407
  [40] VVC Offical Test Model VTM. https://jvet.hhi.fraunhofer.de.408
  Checklist409
  The checklist follows the references. Please read the checklist guidelines carefully for information on410 how to answer these questions. For each question, change the default [TODO] to [Yes] , [No] , or411 [N/A] . You are strongly encouraged to include a justification to your answer, either by referencing412 the appropriate section of your paper or providing a brief inline description. For example:413
  • Did you include the license to the code and datasets? [Yes] See Section ??.414 • Did you include the license to the code and datasets? [No] The code and the data are415 proprietary.416 • Did you include the license to the code and datasets? [N/A]417
  Please do not modify the questions and only use the provided macros for your answers. Note that the418 Checklist section does not count towards the page limit. In your paper, please delete this instructions419 block and only keep the Checklist section heading above along with the questions/answers below.420
  1. For all authors...421 (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s422 contributions and scope? [Yes]423 (b) Did you describe the limitations of your work? [No]424 (c) Did you discuss any potential negative societal impacts of your work? [No]425 (d) Have you read the ethics review guidelines and ensured that your paper conforms to426 them? [Yes]427 2. If you are including theoretical results...428
  (a) Did you state the full set of assumptions of all theoretical results? [N/A]429 (b) Did you include complete proofs of all theoretical results? [N/A]430
  3. If you ran experiments...431 (a) Did you include the code, data, and instructions needed to reproduce the main experi-432 mental results (either in the supplemental material or as a URL)? We describe datasets,433 network architecture and more details in the supplementary.434
  (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they435 were chosen)? [Yes]436
  (c) Did you report error bars (e.g., with respect to the random seed after running experi-437 ments multiple times)? [No]438
  (d) Did you include the total amount of compute and the type of resources used (e.g., type439 of GPUs, internal cluster, or cloud provider)? [No] In the supplementary.440
  4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...441 (a) If your work uses existing assets, did you cite the creators? [Yes]442 (b) Did you mention the license of the assets? [Yes]443 (c) Did you include any new assets either in the supplemental material or as a URL? [No]444 (d) Did you discuss whether and how consent was obtained from people whose data you’re445 using/curating? [No]446 (e) Did you discuss whether the data you are using/curating contains personally identifiable447 information or offensive content? [No]448 5. If you used crowdsourcing or conducted research with human subjects...449
  (a) Did you include the full text of instructions given to participants and screenshots, if450 applicable? [N/A]451
  (b) Did you describe any potential participant risks, with links to Institutional Review452 Board (IRB) approvals, if applicable? [N/A]453
  (c) Did you include the estimated hourly wage paid to participants and the total amount454 spent on participant compensation? [N/A]455
title: Versatile Learned Video Compression