summaries: In this paper, the authors propose to solve the problem of text to speech generation based on video. Thus, this is video-text guided speech generation task that they term as silent video dubbing. The method is based on a transformer architecture that combines text and visual lip motion representations as encoders and outputs the mel-spectrograms through the decoder of a transformer.   The method is compared with TTS systems and comparisons are done using human evaluation scores and audio-visual synchronisation scores. This is done over LRS2 and Lip2Wav for one out of 5 speakers. Comparisons are not done using standard PESQ, STOI or Word error rates. These are standard metrics used for TTS systems. Comparison is also not provided with single speaker Lip2Wav task or multi-speaker Lip2Wav on LRW dataset as done by Lip2Wav or Vid2Speech, or such methods that generate speech based on lip motion alone (not requiring text).

summaries: The authors describe the problem of automatic dubbing of videos, ie generating speech from a script that matches the active speaking face in a video. To the best of this reviewer's knowledge, this is the first paper addressing this particular problem. The two main contributions to solve this problem is 1. a method to synchronize the generated TTS with the speaking face 2. sampling the video for a face image to infer a timbre for the speech to better match the face.

summaries: This paper describes a novel task 'silent video dubbing' and proposes a method to solve it. The task involves generating realistic audio to accompany silent video, with the help of text. This is essentially similar to vid2speech [14], but with text guidance. The pipeline consists of several stages. (1) generating phoneme and video embeddings; (2) aligning text and video using the embeddings from (1) and attention network; (3) generating speaker embeddings to condition the decoder from images; (4) using (2) and (3) to generate the result voice. The authors provide example in supplementary materials. The experiments are performed on both single-speaker and multi-speaker SVD.

summaries: This paper presents "Neural Dubber", a method for dubbing silent videos based on given scripts. It essentially performs lip-synced TTS, with an added component (ISE) that ensures that the generated speech characteristics match the given face image.   Experiments verify that the proposed TTS method does not hurt SOTA TTS performance, while controlling the prosody such that the aligned speech is much more synchronized than other methods compared to. The supplementary video qualitatively demonstrates some compelling results of the proposed method.

abstractText: Dubbing is a post-production process of re-recording actors’ dialogues, which is extensively used in filmmaking and video production. It is usually performed manually by professional voice actors who read lines with proper prosody, and in synchronization with the pre-recorded videos. In this work, we propose Neural Dubber, the first neural network model to solve a novel automatic video dubbing (AVD) task: synthesizing human speech synchronized with the given video from the text. Neural Dubber is a multi-modal text-to-speech (TTS) model that utilizes the lip movement in the video to control the prosody of the generated speech. Furthermore, an image-based speaker embedding (ISE) module is developed for the multi-speaker setting, which enables Neural Dubber to generate speech with a reasonable timbre according to the speaker’s face. Experiments on the chemistry lecture single-speaker dataset and LRS2 multi-speaker dataset show that Neural Dubber can generate speech audios on par with state-of-the-art TTS models in terms of speech quality. Most importantly, both qualitative and quantitative evaluations show that Neural Dubber can control the prosody of synthesized speech by the video, and generate high-fidelity speech temporally synchronized with the video.

authors:
Triantafyllos Afouras
Joon Son Chung
Andrew Senior
Oriol Vinyals
Andrew Zisserman

title: Deep audio-visual speech recognition

venue: IEEE transactions on pattern analysis and machine intelligence,

authors:
Triantafyllos Afouras
Joon Son Chung
Andrew Zisserman

title: Deep lip reading: a comparison of models and an online application

venue: In Interspeech,

authors:
Sercan Ö Arık
Mike Chrzanowski
Adam Coates
Gregory Diamos
Andrew Gibiansky
Yongguo Kang
Xian Li
John Miller
Andrew Ng
Jonathan Raiman

title: Deep voice: Real-time neural text-to-speech

venue: In International Conference on Machine Learning,

authors:
Yannis M Assael
Brendan Shillingford
Shimon Whiteson
Nando De Freitas

title: Lipnet: End-to-end sentence-level lipreading

venue: arXiv preprint arXiv:1611.01599,

authors:
Qiong Cao
Li Shen
Weidi Xie
Omkar M Parkhi
Andrew Zisserman

title: Vggface2: A dataset for recognising faces across pose and age

venue: In 2018 13th IEEE international conference on automatic face & gesture recognition (FG

authors:
Mingjian Chen
Xu Tan
Yi Ren
Jin Xu
Hao Sun
Sheng Zhao
Tao Qin
Tie-Yan Liu

title: Multispeech: Multi-speaker text to speech with transformer

venue: arXiv preprint arXiv:2006.04664,

authors:
Jan Chorowski
Dzmitry Bahdanau
Dmitriy Serdyuk
Kyunghyun Cho
Yoshua Bengio

title: Attention-based models for speech recognition

venue: arXiv preprint arXiv:1506.07503,

authors:
Min Chu
Hu Peng

title: Objective measure for estimating mean opinion score of synthesized speech

authors:
Joon Son Chung
Amir Jamaludin
Andrew Zisserman

title: You said that

venue: In British Machine Vision Conference,

authors:
Joon Son Chung
Andrew Senior
Oriol Vinyals
Andrew Zisserman

title: Lip reading sentences in the wild

venue: IEEE Conference on Computer Vision and Pattern Recognition (CVPR),

authors:
Joon Son Chung
Andrew Zisserman

title: Out of time: automated lip sync in the wild

venue: In Asian conference on computer vision,

title: Lip reading in profile

authors:
Soo-Whan Chung
Hong Goo Kang
Joon Son Chung

title: Seeing voices and hearing voices: learning discriminative embeddings using cross-modal self-supervision

venue: arXiv preprint arXiv:2004.14326,

authors:
Pif Edwards
Chris Landreth
Eugene Fiume
Karan Singh

title: Jali: an animator-centric viseme model for expressive lip synchronization

venue: ACM Transactions on Graphics (TOG),

authors:
Ariel Ephrat
Shmuel Peleg

title: Vid2speech: speech reconstruction from silent video

venue: IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP),

authors:
Daniel Griffin
Jae Lim

title: Signal estimation from modified short-time fourier transform

venue: IEEE Transactions on acoustics, speech, and signal processing,

authors:
Tavi Halperin
Ariel Ephrat
Shmuel Peleg

title: Dynamic temporal alignment of speech to lips

authors:
Kaiming He
Xiangyu Zhang
Shaoqing Ren
Jian Sun

title: Deep residual learning for image recognition

venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,

authors:
Andrew J Hunt
Alan W Black

title: Unit selection in a concatenative speech synthesis system using a large speech database

venue: IEEE International Conference on Acoustics, Speech, and Signal Processing Conference Proceedings,

authors:
Ye Jia
Yu Zhang
Ron J Weiss
Quan Wang
Jonathan Shen
Fei Ren
Zhifeng Chen
Patrick Nguyen
Ruoming Pang
Ignacio Lopez Moreno

title: Transfer learning from speaker verification to multispeaker text-to-speech synthesis

authors:
Tero Karras
Timo Aila
Samuli Laine
Antti Herva
Jaakko Lehtinen

title: Audio-driven facial animation by joint end-to-end learning of pose and emotion

authors:
Christopher T Kello
David C Plaut

title: A neural network model of the articulatory-acoustic forward mapping trained on recordings of articulatory parameters

venue: The Journal of the Acoustical Society of America,

authors:
Changil Kim
Hijung Valentina Shin
Tae-Hyun Oh
Alexandre Kaspar
Mohamed Elgharib
Wojciech Matusik

title: On learning associations of faces and voices

venue: In Asian Conference on Computer Vision,

authors:
Jaehyeon Kim
Sungwon Kim
Jungil Kong
Sungroh Yoon

title: Glow-tts: A generative flow for text-to-speech via monotonic alignment search

venue: arXiv preprint arXiv:2005.11129,

authors:
Diederik P Kingma
Jimmy Ba

title: Adam: A method for stochastic optimization

venue: arXiv preprint arXiv:1412.6980,

authors:
Kundan Kumar
Rithesh Kumar
Thibault de Boissiere
Lucas Gestin
Wei Zhen Teoh
Jose Sotelo
Alexandre de Brébisson
Yoshua Bengio
Aaron Courville

title: Melgan: Generative adversarial networks for conditional waveform synthesis

venue: arXiv preprint arXiv:1910.06711,

authors:
Yaman Kumar
Rohit Jain
Khwaja Mohd Salik
Rajiv Ratn Shah
Yifang Yin
Roger Zimmermann

title: Lipper: Synthesizing thy speech using multi-view lipreading

venue: In Proceedings of the AAAI Conference on Artificial Intelligence,

authors:
Naihan Li
Shujie Liu
Yanqing Liu
Sheng Zhao
Ming Liu

title: Neural speech synthesis with transformer network

authors:
Arsha Nagrani
Samuel Albanie
Andrew Zisserman

title: Learnable pins: Cross-modal embeddings for person identity

venue: In Proceedings of the European Conference on Computer Vision (ECCV),

authors:
Aaron van den Oord
Sander Dieleman
Heiga Zen
Karen Simonyan
Oriol Vinyals
Alex Graves
Nal Kalchbrenner
Andrew Senior
Koray Kavukcuoglu

title: Wavenet: A generative model for raw audio

venue: arXiv preprint arXiv:1609.03499,

authors:
Stavros Petridis
Themos Stafylakis
Pingehuan Ma
Feipeng Cai
Georgios Tzimiropoulos
Maja Pantic

title: End-to-end audiovisual speech recognition

venue: IEEE international conference on acoustics, speech and signal processing (ICASSP),

authors:
Wei Ping
Kainan Peng
Jitong Chen

title: Clarinet: Parallel wave generation in end-to-end text-to-speech

venue: In International Conference on Learning Representations,

authors:
Wei Ping
Kainan Peng
Andrew Gibiansky
Sercan O Arik
Ajay Kannan
Sharan Narang
Jonathan Raiman
John Miller

title: Deep voice 3: 2000-speaker neural text-to-speech

venue: Proc. ICLR,

authors:
KR Prajwal
Rudrabha Mukhopadhyay
Vinay P Namboodiri
CV Jawahar

title: Learning individual speaking styles for accurate lip to speech synthesis

venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,

title: A lip sync expert is all you need for speech to lip generation in the wild

venue: In Proceedings of the 28th ACM International Conference on Multimedia,

authors:
Ryan Prenger
Rafael Valle
Bryan Catanzaro

title: Waveglow: A flow-based generative network for speech synthesis

authors:
Yi Ren
Chenxu Hu
Xu Tan
Tao Qin
Sheng Zhao
Zhou Zhao
Tie-Yan Liu

title: Fastspeech 2: Fast and high-quality end-to-end text to speech

authors:
Yi Ren
Yangjun Ruan
Xu Tan
Tao Qin
Sheng Zhao
Zhou Zhao
Tie-Yan Liu

title: Fastspeech: Fast, robust and controllable text to speech

authors:
Alexander Richard
Michael Zollhoefer
Yandong Wen
Fernando de la Torre
Yaser Sheikh

title: Meshtalk: 3d face animation from speech using cross-modality disentanglement

venue: arXiv preprint arXiv:2104.08223,

authors:
Jonathan Shen
Ruoming Pang
Ron J Weiss
Mike Schuster
Navdeep Jaitly
Zongheng Yang
Zhifeng Chen
Yu Zhang
Yuxuan Wang
Rj Skerrv-Ryan

title: Natural tts synthesis by conditioning wavenet on mel spectrogram predictions

authors:
Themos Stafylakis
Georgios Tzimiropoulos

title: Combining residual networks with lstms for lipreading

venue: arXiv preprint arXiv:1703.04105,

authors:
Supasorn Suwajanakorn
Steven M Seitz
Ira Kemelmacher-Shlizerman

title: Synthesizing obama: learning lip sync from audio

venue: ACM Transactions on Graphics (ToG),

authors:
Hideyuki Tachibana
Katsuya Uenoyama
Shunsuke Aihara

title: Efficiently trainable text-tospeech system based on deep convolutional networks with guided attention

authors:
Sarah Taylor
Taehwan Kim
Yisong Yue
Moshe Mahler
James Krahe
Anastasio Garcia Rodriguez
Jessica Hodgins
Iain Matthews

title: A deep learning approach for generalized speech animation

authors:
Justus Thies
Mohamed Elgharib
Ayush Tewari
Christian Theobalt
Matthias Nießner

title: Neural voice puppetry: Audio-driven facial reenactment

venue: In Proceedings of the European conference on computer vision (ECCV),

authors:
Laurens Van der Maaten
Geoffrey Hinton

title: Visualizing data using t-sne

venue: Journal of machine learning research,

authors:
Ashish Vaswani
Noam Shazeer
Niki Parmar
Jakob Uszkoreit
Llion Jones
Aidan N Gomez
Lukasz Kaiser
Illia Polosukhin

title: Attention is all you need

venue: arXiv preprint arXiv:1706.03762,

authors:
Konstantinos Vougioukas
Stavros Petridis
Maja Pantic

title: Realistic speech-driven facial animation with gans

venue: International Journal of Computer Vision,

authors:
Li Wan
Quan Wang
Alan Papir
Ignacio Lopez Moreno

title: Generalized end-to-end loss for speaker verification

authors:
Yuxuan Wang
RJ Skerry-Ryan
Daisy Stanton
Yonghui Wu
Ron J Weiss
Navdeep Jaitly
Zongheng Yang
Ying Xiao
Zhifeng Chen
Samy Bengio

title: Tacotron: Towards end-to-end speech synthesis

venue: arXiv preprint arXiv:1703.10135,

authors:
Olivia Wiles
A Koepke
Andrew Zisserman

title: X2face: A network for controlling face generation using images, audio, and pose codes

authors:
Zhizheng Wu
Oliver Watts
Simon King

title: Merlin: An open source neural network speech synthesis system

venue: In SSW,

authors:
Ryuichi Yamamoto
Eunwoo Song
Jae-Min Kim

title: Parallel wavegan: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram

authors:
Heiga Zen
Viet Dang
Rob Clark
Yu Zhang
Ron J Weiss
Ye Jia
Zhifeng Chen
Yonghui Wu

title: Libritts: A corpus derived from librispeech for text-to-speech

venue: arXiv preprint arXiv:1904.02882,

authors:
Shifeng Zhang
Xiangyu Zhu
Zhen Lei
Hailin Shi
Xiaobo Wang
Stan Z Li

title: S3fd: Single shot scale-invariant face detector

venue: In Proceedings of the IEEE international conference on computer vision,

authors:
Hang Zhou
Yu Liu
Ziwei Liu
Ping Luo
Xiaogang Wang

title: Talking face generation by adversarially disentangled audio-visual representation

authors:
Yang Zhou
Zhan Xu
Chris Landreth
Evangelos Kalogerakis
Subhransu Maji
Karan Singh

title: Visemenet: Audio-driven animator-centric speech animation

heading: 1 Introduction

text:
Dubbing is a post-production process of re-recording actors’ dialogues in a controlled environment (i.e., a sound studio), which is extensively used in filmmaking and video production. There are two common application scenarios for dubbing. The first one is replacing previous dialogues because poor sound quality is very common for speech recorded on noise location or the scene itself is too challenging to record high-quality audio. The second one is replacing the actor’ voices in foreignlanguage films with those of other performers speaking the audience’s language. For example, an English video needs to be dubbed into Chinese if it is shown in China.
In this paper, we mainly focus on the first application scenario, also known as “automated dialogue replacement (ADR)”, in which the professional voice actor watches original performance in the pre-recorded video, and re-records each line to match the lip movement with proper prosody such as stress, intonation and rhythm, which allows their speech to be synchronized with the pre-recorded video. In this scenario, the lip motion (viseme) in the video is consistent with the given scripts (phoneme), and the pre-recorded high-definition video can not modified during the ADR process.
While dubbing is an impressive ability of professional voice actors, we aim to achieve this ability computationally. We name this novel task automatic video dubbing (AVD): synthesizing human speech that is temporally synchronized with the given video according to the corresponding text. The main challenges of the task are two-fold: (1) temporal synchronization between synthesized speech
∗Corresponding to hangzhao@mail.tsinghua.edu.cn
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
and video, i.e., the synthesized speech should be synchronized with the lip movement of the speaker in the given video; (2) the content of the speech should be consistent with the input text.
Text to speech (TTS) is a task closely related to dubbing, which aims at converting given texts into natural and intelligible speech. However, several limitations prevent TTS from being applied in the dubbing problem: 1) TTS is a one-to-many mapping problem (i.e., multiple speech variations can be spoken from the same text) [38], so it is hard to control the variations (e.g., prosody, pitch and duration) in synthesized speech during generation; 2) with only text as input, TTS can not utilize the visual information from the video to control speech synthesis, which greatly limits its applications in dubbing scenarios where the synthesized speech are required to be synchronized with the video.
We introduce Neural Dubber, the first model to solve the AVD task. Neural Dubber is a multi-modal speech synthesis model, which generates high-quality and lip-synced speech from the given text and video. In order to control the duration and prosody of synthesized speech, Neural Dubber works in a non-autoregressive way following [38]. The problem of length mismatch between phoneme sequence and mel-spectrogram sequence in non-autoregressive TTS is usually solved by up-sampling the phoneme sequence according to the predicted phoneme duration. Meanwhile, a phoneme duration predictor is needed, where the ground truth is usually obtained from another model [39, 38] or itself during training [24]. However, due to the natural correspondence between lip movement and text [10], we do not need to get phoneme duration target in advance like previous methods [39, 38, 24]. Instead, we use the text-video aligner which adopts an attention module between the video frames and phonemes, and then upsample the text-video context sequence according to the length ratio of mel-spectrogram sequence and video frame sequence. The text-video aligner not only solves the length mismatch problem, but also allows the lip movement in the video to control the prosody of the generated speech explicitly by the attention between video frames and phonemes.
In the real dubbing scenario, voice actors need to alter the timbre and tone according to different performers in the video. In order to better simulate the real case in the AVD task, we propose the image-based speaker embedding (ISE) module, which aims to synthesize speech with different timbres conditioning on the speakers’ face in the multi-speaker setting. To the best of our knowledge, this is the first attempt to predict a speaker embedding from a face image with the goal of generating speech with a reasonable timbre that is consistent with the speaker’s facial features (e.g., gender and age). This is achieved by taking advantage of the natural co-occurrence of faces and speech in videos without the supervision of speaker identity. With ISE, Neural Dubber can synthesize speech with a reasonable timbre according to the speaker’s face. In other words, Neural Dubber can use different face images to control the timbre of the synthesized speech.
We conduct experiments on the chemistry lecture dataset from Lip2Wav [35] for the single-speaker AVD, and the LRS2 [1] dataset for the multi-speaker AVD. The results of extensive quantitative and qualitative evaluations show that in terms of speech quality, Neural Dubber is on par with state-ofthe-art TTS models [51, 41, 38]. Furthermore, Neural Dubber can synthesize speech temporally synchronized with the lip movement in video. In the multi-speaker setting, we demonstrate that the ISE enables Neural Dubber to generate speech with reasonable timbre based on the speaker’s face, resulting in Neural Dubber outperforming FastSpeech 2 by a big margin in term of audio quality. We attach some audio files and video clips generated by our model in the supplementary materials.

heading: 2 Related Work

text:
Text to Speech. Text to Speech (TTS) [3, 41, 51, 39], which aims to synthesize intelligible, natural and high-quality speech from the input text, has seen tremendous progress in recent years. Specifically, the prevalent methods have shifted from concatenative synthesis [19], parametric synthesis [53] to end-to-end neural network-based synthesis [33, 41, 51], where the quality of the synthesized speech is improved by a large margin and is close to that of the human counterpart. The general paradigm of end-to-end neural network-based methods usually first generate the acoustic feature (e.g., melspectrogram) from text using encoder-decoder architecture autoregressively [51, 41, 34, 28] or nonautoregressively [39, 38, 24], then reconstruct the waveform signal using vocoder [16, 30, 37, 26, 54]. When it comes to the recent multi-speaker TTS system [20], the speaker embedding is often extracted using speaker verification system, and is fed to the decoder of the TTS system in order to encourage the model to obtain a timbre inclination for the speaker of interest. Different from TTS, Neural
Dubber is conditioned not only on texts but also on videos, intending to synthesize natural speech given both of them.
Talking Face Generation. Talking face generation has a long history in computer vision, ranging from viseme-based models [14, 58] to neural synthesis of 2D [43, 46, 52] or 3D [21, 40, 45] face. Recently neural synthesis approaches have been proposed to generate realistic 2D video of talking heads. Concretely speaking, Chung et al. [9] first generates lower face animation using cropped frontal images. After then Zhou et al. [57] further disentangles identity from speech using generative adversarial networks (GANs). Wav2Lip [36] tries to explore the problem of visual dubbing, i.e., lip-syncing a talking head video of an arbitrary person to match a target speech segment. From our perspective, however, such methods can not generate high-fidelity face and lip given speech, spawning the results are of low resolution and look uncanny sometimes. Besides, audios in most talking face pipelines need to be prepared in advance, thus, strictly speaking, this does not belong to dubbing (re-recording)2, but to the face synchronization while given audio. In contrast to the aforementioned works, Neural Dubber is not required to prepare audio beforehand and modify the lip motion, but generates speech audio synchronized with the video from scripts.
Lip to Speech Synthesis. Given a video, the lip to speech task aims at synthesizing the corresponding speech audio by directly judging from the lip motion. While the conventional method [22] exploits the visual features extracted from active appearance models, recent end-to-end methods have also shed some light on it. In particular, Vid2Speech [15] and Lipper [27] generate low-dimensional linear predictive coding features to synthesize speech in the constrained scene. Vougioukas et al. [49] using the GANs-based method to exert for quality gains. Lip2Wav [35] has achieved promising results in real-life speaker-dependent scenarios, but it is still somewhat incongruous and prone to collapse in the multi-speaker setting. This is possibly because the word error rate in lip reading task [2, 4, 10, 12] is still high, let alone the lip to speech synthesis. In Neural Dubber, the textual information is provided, allowing us to concentrate more on the alignment between the phoneme and lip motion in video, instead of decoding speech from lip motion directly.

heading: 3 Method

text:
In this section, we first introduce the novel automatic video dubbing (AVD) task; we then describe the overall architecture of our proposed Neural Dubber; finally we detail the main components in Neural Dubber.
2https://en.wikipedia.org/wiki/Dubbing_(filmmaking)

heading: 3.1 Automatic Video Dubbing

text: Given a sentence T and a corresponding video clip (without audio) V , the goal of automatic video dubbing (AVD) is to synthesize natural and intelligible speech S whose content is consistent with T , and whose prosody is synchronized with the lip movement of the active speaker in the video V . Compared to the traditional speech synthesis task which only generates natural and intelligible speech S given the sentence T , AVD task is more difficult due to the synchronization requirement.

heading: 3.2 Neural Dubber

heading: 3.2.1 Design Overview

text: Our Neural Dubber aims to solve the AVD task. Concretely, we formulate the problem as follows: given a phoneme sequence Sp = {P1, P2, . . . , PTp} and a video frame sequence Sv = {I1, I2, . . . , ITv}, we need to predict a target mel-spectrogram sequence Sm = {Y1, Y2, . . . , YTm}. The overall model architecture of Neural Dubber is shown in Figure 2. First, we apply a phoneme encoder fp and a video encoder fv to process the phonemes and images respectively. Note that the images we feed to the video encoder only contain mouth region of the speaker following [10, 32, 42]. We use Smv to represent these images. After the encoding, raw phonemes turn into a sequence of hidden representationsHpho = fp(Sp) ∈ RTp×d while images of mouth region turn into a sequence of hidden representationsHvid = fv(Smv ) ∈ RTv×d. Then we feedHpho andHvid into the text-video aligner (which will be described in Section 3.2.3) and get the expanded sequence Hmel ∈ RTm×d with the same length as the target mel-spectrogram sequence Sm. Meanwhile, a face image randomly selected from the video frames is input into image-based speaker embedding (ISE) module (which will be described in Section 3.2.4) to generate a image-based speaker embedding. We addHmel and ISE together and feed them into the variance adaptor to add some variance information (e.g., pitch and energy). Finally, we use the mel-spectrogram decoder to convert the adapted hidden sequence into mel-spectrogram sequence following [39]. Different from FastSpeech 2 [38], our variance adaptor consists of pitch and energy predictors without duration predictor, because we solve the problem of length mismatch between the phoneme and mel-spectrogram sequence in the text-video aligner and the input of variance adaptor is as long as the mel-spectrogram sequence.

heading: 3.2.2 Phoneme and Video Encoders

text:
The phoneme encoder and video encoder are shown in Figure 2a, which are enclosed in a dashed box. The function of the phoneme encoder and video encoder is to transform the original phoneme and image sequences into hidden representation sequences which contain high-level semantics. The phoneme encoder we use is similar to that in FastSpeech [39], which consists of an embedding layer
and N Feed-Forward Transformer (FFT) blocks. The video encoder consists of a feature extractor and K FFT blocks. The feature extractor is a CNN backbone that generates feature representation for every input mouth image. And then we use the FFT blocks to capture the dynamics of the mouth region because FFT is based on self-attention [48] and 1D convolution where self-attention and 1D convolution are suitable for capturing long-term and short-term dynamics respectively.

heading: 3.2.3 Text-Video Aligner

text:
The most challenging aspect of the AVD task is alignment: (1) the content of the generated speech should come from the input phonemes; (2) the prosody of the generated speech should be aligned with the video in time. So it does not make sense to produce speech solely from phonemes, nor video. In our design, the text-video aligner (Figure 2b) aims to find the correspondence between text and lip movement first, so that synchronized speech can be generated in the later stage.
In the text-video aligner, an attention-based module learns the alignment between the phoneme sequence and the video frame sequence, and produces the text-video context sequence. Then an upsampling operation is performed to change the length of the text-video context sequence Hcon from Tv to Tm.
In practice, we adopt the popular Scaled Dot-Product Attention [48] as the attention module, where Hvid is used as the query, andHpho is used as both the key and the value.
Attention(Q,K, V ) = Attention (Hvid,Hpho,Hpho) (1)
= Softmax ( HvidHTpho√
d
) Hpho (2)
= AHpho ∈ RTv×d, (3)
where A is the matrix of attention weights. After the attention module, we get the text-video context sequence, i.e., the expanded sequence of phoneme hidden representation by linear combination. We use a residual connection [18] to add the Hvid for efficient training. However, we use a dropout layer with a large dropout rate to prevent mel-spectrograms from being generated directly from visual information. The attention weight A obtained after softmax is the main determinant of the speed and prosody of the synthesized speech like the attention weight between spectrograms and phonemes in [51, 41, 28]. The sequence of video hidden representations is used as the query. As a consequence, the attention weight is controlled by the video explicitly, and the temporal alignment between video frames and phonemes is achieved. The obtained monotonic alignment between video frames and phonemes contributes to the synchronization between the synthesized speech and the video on fine-grained (phoneme) level.
There is a natural temporal correspondence between the speech audio and the video. In other words, once the alignment between video frames and phonemes is achieved, the alignment between melspectrogram frames and phonemes can be obtained. In practice, the length of a mel-spectrograms sequence is n times that of a video frame sequence. We denote the n as
n = Tmel Tv = sr/hs FPS ∈ N+, (4)
where sr denotes the sampling rate of the audio and hs denotes hop size set when transforming the raw waveform into mel-spectrograms. We upsample the text-video context sequenceHcon toHmel with scale factor is n. Actually, we use the upsampling method with nearest mode.
Hcon = {C1, C2, . . . , CTv} ∈ RTv×d (5) Hmel = Upsample (Hcon, n) ∈ RTm×d (6)
After that, the length of the text-video context sequence is expanded to that of the mel-spectrograms sequence. Thus, the length mismatch problem between the phoneme and mel-spectrograms sequence is solved without the supervision of fine grained alignment between phonemes and spectrograms. Because of the attention between video frames and phonemes, the speed and part of prosody of synthesized speech are controlled by input video explicitly, which makes the synthesized speech and input video well synchronized.
Monotonic Alignment Constraint In text to speech (TTS) task, the monotonic and diagonal alignments in the attention weights between text and speech are important to ensure the quality of synthesized speech [51, 41, 44, 6]. In Neural Dubber, a multi-modal TTS model, the monotonic and diagonal alignments between video frames and phonemes are also critical. So we adopt a diagonal constraint on the attention weights to guide the text-video attention module to learn right alignments following [6]. We formulate the diagonal attention rate r as
r =
∑Tm s=1 ∑min(ks+b,Tp) t=max(ks−b,1)As,t
Tm , (7)
where k = TpTm , b is a hyperparameter for bandwidth of the diagonal area. We add the diagonalconstraint loss which is defined as LDC = −r to our final loss for better alignments.

heading: 3.2.4 Image-based Speaker Embedding Module

text:
How much can we infer about the way people speak from their appearances? In the real dubbing scenario, voice actors need to alter the timbre according to different performers. In order to better simulate the real case in AVD task, we aim to synthesize speech with different timbre conditioning on the speakers’ faces in multi-speaker setting. There have been many works [29, 23, 13] researching the correlation between voice and speakers’ face recently, but none of them learn the joint speaker-face embeddings to solve the multi-speaker text to speech task. In this work, we propose image-based speaker embedding (ISE) module (Figure 2c), a new multi-modal speaker embedding module, generates an embedding that encapsulates the characteristics of the speaker’s voice from an image of his/her face. The ISE module is trained with other components of Neural Dubber from scratch in a self-supervised manner, utilizing the natural co-occurrence of faces and speech audio in videos, but without the supervision of speaker identity. We randomly select a face image Ifi from S f v ={
If1 , I f 2 , . . . , I f Tv
} , and obtain a high-level face feature by feeding the selected face image into a
pre-trained and fixed face recognition network [31, 5]. Then we feed the face feature to a trainable MLP and gain the ISE. The predicted ISE is directly broadcasted and added toHmel so as to control the timbre of synthesized speech. Our model learns face-voice correlations which allow it to produce speech that coincides with various voice attributes of the speakers (e.g., gender and age) inferred from their faces.

heading: 4 Experiments and Results

heading: 4.1 Datasets

text:
Single-speaker Dataset In the single-speaker setting, we conduct experiments on the chemistry lecture dataset from Lip2Wav [35]. With a large vocabulary size and a lot of head movements, the dataset is originally used for the unconstrained single-speaker lip to speech synthesis. To make it fit the AVD task, we collect the official transcripts from YouTube. We need corresponding sentence-level text and audio clips for training, so we segment the long videos into sentence-level clips according to the start and end timestamp of each sentence in the transcripts. Some segmented sentence-level video clips contain frames that only capture the PowerPoint but not lecturer face which can not be used for training. So we conduct data cleaning to remove them. Finally, the dataset contains 6,640 samples, with the total video length of approximately 9 hours. We randomly split the dataset into 3 sets: 6240 samples for training, 200 samples for validation, and 200 samples for testing. In the following subsections, we refer to this dataset as chem for short.
Multi-speaker Dataset In multi-speaker setting, we conduct experiments on the LRS2 [1] dataset, which consists of thousands of sentences spoken by various speakers on BBC channels. This dataset suits the AVD task well, because each sample includes both the text and video pair. Note that we only train on the training set of the LRS2 dataset, which only contains data of approximately 29 hours. Compared to other multi-speaker speech synthesis datasets [55], this dataset is quite small for multi-speaker speech generation and does not provide the speaker identity for each sample. The ISE module aids Neural Dubber in solving these problems.

heading: 4.2 Data Pre-processing

text: The video frames are sampled at 25 FPS. We detect and crop the face from the video frames using S3FD [56] face detection following [35]. The images input to the video encoder are resized to 96 x 96 in dimension, which only cover the mouth region of the face, as shown in Figure 2a. The face image input to the ISE module is 224 x 224 in dimension and covers the whole face of the speaker. In order to alleviate the mispronunciation problem, we convert the text sequences into the phoneme sequences [3, 28, 38] with an open-source grapheme-to-phoneme tool. For the speech audio, we transform the raw waveform into mel-spectrograms following [41]. The frame size and hop size are set to 640 samples (40 ms) and 160 samples (10 ms) with respect to the sample rate 16 kHz.

heading: 4.3 Model Configuration

text:
Neural Dubber Our Neural Dubber consists of 4 feed-forward Transformer (FFT) blocks [39] in the phoneme encoder, the mel-spectrogram decoder, and 2 FFT blocks in the video encoder. The feature extractor in the video encoder is the ResNet18 [18] except for the first 2D convolution layer being replaced by 3D convolutions [32]. The variance adaptor contains pitch predictor and energy predictor. The configurations of the FFT block, the mel-spectrogram decoder, the pitch predictor and the energy predictor are the same as those in FastSpeech 2 [38]. In the text-video aligner, the hidden size of the scaled dot-product attention is set to 256, the number n of the repeat operation is set to 4 according to Equation (4). In the ISE module, the face feature extractor we use is a pre-trained and fixed ResNet50 trained on the VGGFace2 [5] dataset. The face feature is a 4096-D feature that is extracted from the penultimate layer (i.e., one layer prior to the classification layer) of the network.
Baseline Since automatic video dubbing is a new task that we propose, none of the previous works focused on solving this task. So we propose a baseline model based on the Tacotron [51] system with some modifications which make it fit to the new AVD task. We call this baseline model Video-based Tacotron. In order to make use of the information in video, we concatenate the spectrogram frames with the corresponding hidden representation of video frames, and use it as the decoder input:
Y ′ i−1 = Yi−1 ⊕H d in e vid , (8)
where Y ′
i−1 is the decoder input, ⊕ represents the concatenation operation, Hvid is the hidden representation of video frames, which is obtained in the same way as in Neural Dubber described in the Section 3.2.1 and n is same as that in Equation (4). The Video-based Tacotron implementation is based on an open-source Tacotron repository 3 where the attention is replaced with the locationsensitive attention [7] according to [41] for better results. We set the reduction factor r to 2 and change the vocoder to Parallel WaveGAN [54] for fair comparison.

heading: 4.4 Training and Inference

text: We train Neural Dubber on 1 NVIDIA V100 GPU. We use the Adam optimizer [25] with β1 = 0.9, β2 = 0.98, ε = 10−9 and follow the same learning rate schedule in [48]. Our model is optimized with the loss similar to that in [38]. We set the batchsize to 18 and 24 on chem dataset and LRS2 dataset respectively. It takes 200k/300k steps for training until convergence on the chem/LRS2 dataset. In this work, we use Parallel WaveGAN [54] as the vocoder to transform the generated mel-spectrograms into audio samples. We train two Parallel WaveGAN vocoders on the training set of chem dataset and LRS2 dataset respectively, following an open-source implementation 4. Each Parallel WaveGan vocoder is trained on 1 NVIDIA V100 GPU for 1000K steps. In the inference process, the output mel-spectrograms of Neural Dubber are transformed into audio samples using the pre-trained Parallel WaveGAN.

heading: 4.5 Evaluation

heading: 4.5.1 Metrics

text:
Since the AVD task aims to synthesize human speech synchronized with the video from text, the audio quality and the audio-visual synchronization (av sync) are the important evaluation criteria.
3https://github.com/fatchord/WaveRNN 4https://github.com/kan-bayashi/ParallelWaveGAN
Human Evaluation We conduct the mean opinion score (MOS) [8] evaluation on the test set to measure the audio quality and the av sync. We randomly select 30 video clips from the test set, where each video clip is scored by at least 20 raters, who are all native English speakers. We overlay the synthesized speech on the original video before showing it to the rater, following [35]. The text and the video are consistent among different systems, so that all raters only examine the audio quality and the av sync without other interference factors. For each video clip, the raters are asked to rate scores of 1-5 from bad to excellent (higher score indicates better quality) on the audio quality and the av sync, respectively. We perform the MOS evaluation on Amazon Mechanical Turk (MTurk).
Quantitative Evaluation In order to measure the synchronization between the generated speech and the video quantitatively, we use the pre-trained SyncNet [11], which is publicly available5 following [35]. The method can explicitly test for synchronization between speech audio and lip movements in unconstrained videos in the wild [11, 36]. We adopt two metrics: Lip Sync Error - Distance (LSE-D) and Lip Sync Error - Confidence (LSE-C) from Wav2Lip [36]. The two metrics can be automatically calculated by the pre-trained SyncNet model. LSE-D denotes the minimal distance between the audio and the video features for different offset values. A lower LSE-D means the speech audio and video are more synchronized. LSE-C denotes the confidence that the audio and the video are synchronized with a certain time offset. A lower LSE-C means that some parts of the video are completely out of sync, where the audio and the video are uncorrelated.
GT
ND
FS2
VT
Figure 3: Mel-spectrograms of audios synthesized by some systems: Ground Truth (GT), Neural Dubber (ND), FastSpeech 2 (FS2) and Video-based Tacotron (VT).

heading: 4.5.2 Single-speaker AVD

text:
We first conduct MOS evaluation on the chem single-speaker dataset, to compare the audio quality and the av sync of the video clips generated by Neural Dubber with other systems, including 1) GT, the ground-truth video clips; 2) GT (Mel + PWG), where we first convert the ground-truth audio into mel-spectrograms, and then convert it back to audio using Parallel WaveGAN [54] (PWG); 3) FastSpeech 2 [38] (Mel + PWG); 4) Video-based Tacotron (Mel + PWG). Note that the systems in 2), 3), 4) and Neural Dubber use the same pre-trained Parallel WaveGAN for a fair comparison. In addition, we compare Neural Dubber with those systems on the test set using the LSE-D and LSE-C metrics. The results for single-speaker AVD are shown in Table 1. It can be seen that Neural Dubber can surpass the Video-based Tacotron baseline and is on par with FastSpeech 2 in terms of audio quality, which demonstrates that Neural Dubber can synthesize high-quality speech. Furthermore, in terms of the av sync, Neural Dubber outperforms FastSpeech 2 and Video-based Tacotron by a big margin and matches GT (Mel + PWG) system in both qualitative and quantitative evaluations, which shows that Neural Dubber can control the prosody of speech and generate speech synchronized with the video. For FastSpeech 2 and Video-based Tacotron, the LSE-D is high and the LSE-C is low,
5https://github.com/joonson/syncnet_python
indicating that they can not generate speech synchronized with the video. We also show a qualitative comparison in Figure 3 which contains mel-spectrograms of audios generated by the above systems. It shows that the prosody of the audio generated by Neural Dubber is closed to that of ground truth recording, i.e., well synchronized with the video.
In addition, we compare our method with another baseline [17] which automatically stretches and compresses the audio signal to match the lip movement given an unaligned face sequence and speech audio. We use the speech generated by FastSpeech 2 (Mel + PWG) system, and then align the pre-generated speech with the lip movement according to [17]. However, the quality and naturalness of its synthesized speech is much worse than the pre-generated speech due to challenging alignments. So this baseline is not comparable to our Neural Dubber.

heading: 4.5.3 Multi-speaker AVD

text:
Similar to Section 4.5.2, we conduct human evaluation and quantitative evaluation on the LRS2 multi-speaker dataset to compare Neural Dubber with other systems in multi-speaker setting. Due to the failure of Video-based Tacotron in single-speaker AVD, we no longer compare our model with it. Note that we can not add a trivial speaker embedding module to FastSpeech 2, because the LRS2 dataset does not contain the speaker identity for each video. So we directly train FastSpeech 2 on the LRS2 dataset without modifications. The results are shown in Table 2. We can see that Neural Dubber outperforms FastSpeech 2 by a significant margin in terms of audio quality, exhibiting the effectiveness of ISE in multi-speaker AVD. The qualitative and quantitative evaluations show that the speech synthesized by Neural Dubber is much better than that of FastSpeech 2 and is on par with the ground truth recordings in terms of synchronization. These results show that Neural Dubber can address the multi-speaker AVD which is more challenging than the single-speaker AVD.
In order to demonstrate that ISE enables Neural Dubber to control the timbre by the input face image, some audio clips are generated by Neural Dubber with the same phoneme sequence and mouth image sequence but different speaker face images as input. We select 12 males and 12 females from the test set of LRS2 dataset for this evaluation. For each person, we chose 10 face images with different head posture, illumination and facial makeup, etc.
We visualize the voice embedding of these audios in Figure 4 by using a pre-trained speaker encoder [50] from an open-source repository6. We first use the speaker (voice) encoder to derive a high-level representation, i.e., a 256-D embedding, from an audio, which summarizes the characteristics of the voice in the audio. Then we use t-SNE [47] to visualize the generated embedding. It can be seen that the utterances generated from the images of the same speaker form a tight cluster, and that the cluster representing each speaker is separated from each other. In addition, there is a distinctive discrepancy between the speech synthesized from the face images of different genders. It concludes that Neural Dubber can use the face image to alter the timbre of the generated speech.

heading: 4.5.4 Comparing with the Lip-motion Based Speech Generation Method

text:
Recently, some works have demonstrated the impressive ability to generate speech directly from the lip motion. However, the quality and intelligibility of the generated speech are relatively poor, and the word error rate (WER) is very high. In this section, we compare with a SOTA lip-motion based speech generation system Lip2Wav [35]. Because Lip2Wav can only generate word-level speech in the multi-speaker setting, we only compare Neural Dubber with Lip2Wav in the single-speaker
6https://github.com/resemble-ai/Resemblyzer
setting still on the chem dataset. We use the official GitHub repository7 to train Lip2Wav on our version of the chemistry lecture dataset. As we mentioned in Section 4.1, the dataset is different from the original one in Lip2Wav. It only contains data of approximately 9 hours, which is much less than the original one (approximately 24 hours). In this experiment, the training and testing sets of Neural Dubber and Lip2Wav are identical, so the results can be compared directly. Following the Lip2Wav paper [35], we use STOI and ESTOI for estimating the intelligibility and PESQ for measuring the quality. In addition, using an out-of-the-box ASR system, we evaluate the speech results using word error rates (WER). In order to eliminate the influence of the ASR system, we also measure the WER for ground truth speech audio. All these metrics are computed on the test dataset.
As the comparison results in Table 3 show, Neural Dubber surpasses Lip2Wav by a big margin in terms of speech quality and intelligibility. Please note that STOI, ESTOI, and PESQ scores of Lip2Wav are lower than those in [35], because the training data we used is much less than theirs. Most importantly, the WER of Neural Dubber is 4x lower than that of Lip2Wav. It shows that Neural Dubber outperforms Lip2Wav significantly in pronunciation accuracy. WER of Lip2Wav is up to 72.70%, indicating that it mispronounces a lot of content, which is unacceptable in the AVD task. Just like it is unacceptable for an actor to always mispronounce the lines. Please note that the WER of Lip2Wav we get is consistent with the results in [35] (see its Table 5). In summary, Neural Dubber far outperforms Lip2Wav in terms of speech intelligibility, quality, and pronunciation accuracy (WER), and is much more suitable for the AVD task.

heading: 5 Limitations and Societal Impact

text: When the script is changed to be different from what the speaker is actually saying, our method can only deal with the situation of modifying couple words. In addition, the lip movement of the modified text should be similar to the original lip movement in the video. The facial appearance may lead to timbre ambiguity due to the dataset bias. It might be offensive. Our method can dub videos automatically, which may be useful for filmmaking and video production.

heading: 6 Conclusion

text:
In this work, we introduce a novel task, automatic video dubbing (AVD), which aims to synthesize human speech synchronized with the given video from text. To solve the AVD task, we propose Neural Dubber, a multi-modal TTS model, which can generate lip-synced mel-spectrograms in parallel. We design several key components including the video encoder, the text-video aligner and the ISE module for Neural Dubber to better solve the task. Our experimental results show that, in terms of speech quality, Neural Dubber is on par with FastSpeech 2 on the chem dataset, even outperforms FastSpeech 2 on the LRS2 dataset due to ISE’s help. More importantly, Neural Dubber can synthesize speech temporally synchronized with the video.
7https://github.com/Rudrabha/Lip2Wav

title: Neural Dubber: Dubbing for Videos According to Scripts


summaries: This paper presents a federated split task-agnostic (FESTA) framework that can take an effective use of the vision transformer's (ViT) to perform several CXR tasks simultaneously. The authors discover that the design of ViT is ideal for split learning, in which a network is designed into a client-server mechanism. They integrate ViT into their framework by combining the benefits of federated and split learning through its decomposable design. According to the authors, a model trained with the FESTA framework can leverage robust representations from multiple related tasks to improve individual task performance.

summaries: This paper combines: split learning, federated learning, multi-task learning, vision transformers, and COVID-19 chest x-ray (CXR) classification. Specifically, the authors propose the Federated Split Task-Agnostic (FESTA) framework, motivated by privacy concerns of training models with patient level healthcare data (CXRs). This framework is for training large deep learning models based on private datasets distributed over clients (i.e. hospitals). Here, each client has a local copy of a "head" and "tail" of a vision transformer model, while the server has a shared body of a vision transformer. The training process is as follows: 1.) clients perform forward passes using their local data through their head networks and send the embedded representations to the server 2.) the server performs a forward pass with the embedded representations and sends the resulting representations back to their respective clients 3.) the clients finish the forward pass through their tail networks, compute the loss, and send gradients back through the server 4.) server continues backprop, client finishes backprop, everyone updates their weights (the server's network weights will be updated considering data from _all_ clients) 5.) occasionally, all the weights from the clients head and tail networks will be averaged together (FedAvg)  Additionally, the authors consider the case where some clients are training on different tasks (e.g. image classification, segmentation, detection) using the same modality (CXRs). Here, step 5 is applied over clients with the same task.

summaries: This paper presents a federated learning (FL) framework for COVID-19 xray image analysis, which leverages the natural scalibility of visual transformer to establish a multi-task FL process. The proposed method jointly learns from different clients of various x-ray-based tasks including classification, detection and segmentation in a split learning manner, in which each client preserves their only transformer head and tail while jointly learning a shared transformer body maintained in the global server. Experimental results show that the established multi-task federated learning framework with transformer improves individual performances for each COVID-19 diagnosis task and are comparable with centralized training model.

summaries: The authors proposed a multi-task learning framework with a shared feature encoder (body) and separate input embedding (head) and application-wise decoder (tail) components for each task in a distributed/federated learning setting. A number of multi-head self-attention modules from the transformer model are adopted here as the body part. The Federated Split Task-Agnostic Learning (FESTA) paradigm is proposed to compute/update the model parameters of head/tail parts on the client-side and the body parts on the server-side. A list of chest x-ray datasets is employed for the experiments, including datasets for COVID-19 classification (normal, pneumonia, and COVID-19), pneumothorax segmentation, and pneumonia detection. Superior results of the proposed method are reported in comparison to other learning paradigms. The manuscript is overall well-prepared, while several things could be improved and addressed(listed below).

abstractText: Federated learning, which shares the weights of the neural network across clients, is gaining attention in the healthcare sector as it enables training on a large corpus of decentralized data while maintaining data privacy. For example, this enables neural network training for COVID-19 diagnosis on chest X-ray (CXR) images without collecting patient CXR data across multiple hospitals. Unfortunately, the exchange of the weights quickly consumes the network bandwidth if highly expressive network architecture is employed. So-called split learning partially solves this problem by dividing a neural network into a client and a server part, so that the client part of the network takes up less extensive computation resources and bandwidth. However, it is not clear how to find the optimal split without sacrificing the overall network performance. To amalgamate these methods and thereby maximize their distinct strengths, here we show that the Vision Transformer, a recently developed deep learning architecture with straightforward decomposable configuration, is ideally suitable for split learning without sacrificing performance. Even under the non-independent and identically distributed data distribution which emulates a real collaboration between hospitals using CXR datasets from multiple sources, the proposed framework was able to attain performance comparable to data-centralized training. In addition, the proposed framework along with heterogeneous multi-task clients also improves individual task performances including the diagnosis of COVID-19, eliminating the need for sharing large weights with innumerable parameters. Our results affirm the suitability of Transformer for collaborative learning in medical imaging and pave the way forward for future real-world implementations.

authors:
Y. Aono
T. Hayashi
L. Wang
S. Moriai

title: Privacy-preserving deep learning: Revisited and enhanced

venue: In International Conference on Applications and Techniques in Information Security,

authors:
D.J. Beutel
T. Topal
A. Mathur
X. Qiu
T. Parcollet
P.P. de Gusmão
N.D. Lane

title: Flower: A friendly federated learning research framework

authors:
K. Bonawitz
V. Ivanov
B. Kreuter
A. Marcedone
H.B. McMahan
S. Patel
D. Ramage
A. Segal
K. Seth

title: Practical secure aggregation for privacy-preserving machine learning

venue: In proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security,

authors:
A. Borghesi
R. Maroldi

title: Covid-19 outbreak in italy: experimental chest x-ray scoring system for quantifying and monitoring disease progression

venue: La radiologia medica,

title: Multitask learning

authors:
K.K. Chandriah
R.V. Naraganahalli

title: Maximizing a deep submodular function optimization with a weighted max-sat problem for trajectory clustering and motion segmentation

venue: Applied Intelligence,

authors:
K. Chang
N. Balachandar
C. Lam
D. Yi
J. Brown
A. Beers
B. Rosen
D.L. Rubin
J. Kalpathy-Cramer

title: Distributed deep learning networks among institutions for medical imaging

venue: Journal of the American Medical Informatics Association,

authors:
G. Chartrand
P.M. Cheng
E. Vorontsov
M. Drozdzal
S. Turcotte
C.J. Pal
S. Kadoury
A. Tang

title: Deep learning: a primer

venue: for radiologists. Radiographics,

authors:
H. Chen
Y. Wang
T. Guo
C. Xu
Y. Deng
Z. Liu
S. Ma
W. Gao

title: Pre-trained image processing transformer

authors:
J. Chen
Y. Lu
Q. Yu
X. Luo
E. Adeli
Y. Wang
L. Lu
A.L. Yuille
Y. Zhou

title: Transunet: Transformers make strong encoders for medical image segmentation

venue: arXiv preprint arXiv:2102.04306,

authors:
J. De Fauw
J.R. Ledsam
B. Romera-Paredes
S. Nikolov
N. Tomasev
S. Blackwell
H. Askham
X. Glorot
B. O’Donoghue
D. Visentin

title: Clinically applicable deep learning for diagnosis and referral in retinal disease

venue: Nature medicine,

authors:
M. De La Iglesia Vayá
J.M. Saborit
J.A. Montell
A. Pertusa
A. Bustos
M. Cazorla
J. Galant
X. Barber
D. Orozco-Beltrán
F. García-García

title: Bimcv COVID-19+: a large annotated dataset of rx and ct images from COVID-19 patients

venue: arXiv preprint arXiv:2006.01174,

authors:
B.E. Dewey
C. Zhao
J.C. Reinhold
A. Carass
K.C. Fitzgerald
E.S. Sotirchos
S. Saidha
J. Oh
D.L. Pham
P.A. Calabresi

title: Deepharmony: a deep learning approach to contrast harmonization across scanner changes

venue: Magnetic resonance imaging,

authors:
A. Dosovitskiy
L. Beyer
A. Kolesnikov
D. Weissenborn
X. Zhai
T. Unterthiner
M. Dehghani
M. Minderer
G. Heigold
S. Gelly

title: An image is worth 16x16 words: Transformers for image recognition at scale

venue: arXiv preprint arXiv:2010.11929,

authors:
D. Eigen
R. Fergus

title: Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture

authors:
T. Evgeniou
M. Pontil

title: Regularized multi–task learning

venue: In Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining,

authors:
Y. Fu
Y. Lei
T. Wang
W.J. Curran
T. Liu
X. Yang

title: Deep learning in medical image registration: a review

venue: Physics in Medicine & Biology,

authors:
T. Gabruseva
D. Poplavskiy
A. Kalinin

title: Deep learning for automatic pneumonia detection

venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops,

authors:
M. Gaillochet
K.C. Tezcan
E. Konukoglu

title: Joint reconstruction and bias field correction for undersampled mr imaging

venue: In International Conference on Medical Image Computing and Computer-Assisted Intervention,

authors:
Y. Gao
M. Kim
S. Abuadbba
Y. Kim
C. Thapa
K. Kim
S.A. Camtepe
H. Kim
S. Nepal

title: End-to-end evaluation of federated learning and split learning for internet of things

venue: arXiv preprint arXiv:2003.13376,

authors:
H. Guo
R. Pasunuru
M. Bansal

title: Soft layer-specific multi-task summarization with entailment and question generation

venue: arXiv preprint arXiv:1805.11004,

authors:
O. Gupta
R. Raskar

title: Distributed learning of deep neural network over multiple agents

venue: Journal of Network and Computer Applications,

authors:
J. Irvin
P. Rajpurkar
M. Ko
Y. Yu
S. Ciurea-Ilcus
C. Chute
H. Marklund
B. Haghgoo
R. Ball
K. Shpanskaya

title: Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison

venue: In Proceedings of the AAAI Conference on Artificial Intelligence,

authors:
B. Jing
P. Xie
E. Xing

title: On the automatic generation of medical imaging reports

venue: arXiv preprint arXiv:1711.08195,

authors:
S. Khan
M. Naseer
M. Hayat
S.W. Zamir
F.S. Khan
M. Shah

title: Transformers in vision: A survey

venue: arXiv preprint arXiv:2101.01169,

authors:
J. Konečnỳ
H.B. McMahan
F.X. Yu
P. Richtárik
A.T. Suresh
D. Bacon

title: Federated learning: Strategies for improving communication efficiency

venue: arXiv preprint arXiv:1610.05492,

authors:
T. Li
A.K. Sahu
A. Talwalkar
V. Smith

title: Federated learning: Challenges, methods, and future directions

venue: IEEE Signal Processing Magazine,

authors:
S. Liu
E. Johns
A.J. Davison

title: End-to-end multi-task learning with attention

authors:
X. Liu
P. He
W. Chen
J. Gao

title: Multi-task deep neural networks for natural language understanding

venue: arXiv preprint arXiv:1901.11504,

authors:
M.-T. Luong
Q.V. Le
I. Sutskever
O. Vinyals
L. Kaiser

title: Multi-task sequence to sequence learning

venue: arXiv preprint arXiv:1511.06114,

title: Federated learning: Opportunities and challenges

venue: arXiv preprint arXiv:2101.05428,

authors:
B. McMahan
E. Moore
D. Ramage
S. Hampson
B.A. y Arcas

title: Communication-efficient learning of deep networks from decentralized data

venue: In Artificial Intelligence and Statistics,

authors:
H.B. McMahan
D. Ramage
K. Talwar
L. Zhang

title: Learning differentially private recurrent language models

venue: arXiv preprint arXiv:1710.06963,

authors:
I. Misra
A. Shrivastava
A. Gupta
M. Hebert

title: Cross-stitch networks for multi-task learning

venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,

authors:
S. Park
G. Kim
Y. Oh
J.B. Seo
S.M. Lee
J.H. Kim
S. Moon
J.-K. Lim
J.C. Ye

title: Vision transformer using low-level chest x-ray feature corpus for covid-19 diagnosis and severity quantification

venue: arXiv preprint arXiv:2104.07235,

authors:
S.-A. Rebuffi
H. Bilen
A. Vedaldi

title: Learning multiple visual domains with residual adapters

venue: arXiv preprint arXiv:1705.08045,

authors:
M. Roberts
D. Driggs
M. Thorpe
J. Gilbey
M. Yeung
S. Ursprung
A.I. Aviles-Rivero
C. Etmann
C. McCague
L. Beer

title: Common pitfalls and recommendations for using machine learning to detect and prognosticate for COVID-19 using chest radiographs and CT scans

venue: Nature Machine Intelligence,

authors:
L. Rocher
J.M. Hendrickx
Y.-A. De Montjoye

title: Estimating the success of re-identifications in incomplete datasets using generative models

venue: Nature communications,

title: Authenticated-encryption with associated-data

venue: In Proceedings of the 9th ACM Conference on Computer and Communications Security,

authors:
O. Ronneberger
P. Fischer
T. Brox

title: U-net: Convolutional networks for biomedical image segmentation

venue: In International Conference on Medical image computing and computer-assisted intervention,

authors:
S. Ruder
J. Bingel
I. Augenstein
A. Søgaard

title: Latent multi-task architecture learning

authors:
M.J. Sheller
G.A. Reina
B. Edwards
J. Martin
S. Bakas

title: Multi-institutional deep learning modeling without sharing patient data: A feasibility study on brain tumor segmentation

venue: In International MICCAI Brainlesion Workshop,

authors:
A. Signoroni
M. Savardi
S. Benini
N. Adami
R. Leonardi
P. Gibellini
F. Vaccher
M. Ravanelli
A. Borghesi
R. Maroldi
D. Farina

title: Bs-net: learning covid-19 pneumonia severity on a large chest x-ray dataset

venue: Medical Image Analysis, page 102046,

authors:
V. Smith
C.-K. Chiang
M. Sanjabi
A. Talwalkar

title: Federated multi-task learning

venue: arXiv preprint arXiv:1705.10467,

authors:
C. Sun
A. Shrivastava
S. Singh
A. Gupta

title: Revisiting unreasonable effectiveness of data in deep learning era

authors:
J. Sun
A. Li
B. Wang
H. Yang
H. Li
Y. Chen

title: Soteria: Provable defense against privacy leakage in federated learning from representation perspective

authors:
N. Tajbakhsh
L. Jeyaseelan
Q. Li
J.N. Chiang
Z. Wu
X. Ding

title: Embracing imperfect datasets: A review of deep learning solutions for medical image segmentation

venue: Medical Image Analysis,

authors:
W.G. Van Panhuis
P. Paul
C. Emerson
J. Grefenstette
R. Wilder
A.J. Herbst
D. Heymann
D.S. Burke

title: A systematic review of barriers to data sharing in public health

venue: BMC public health,

authors:
A. Vaswani
N. Shazeer
N. Parmar
J. Uszkoreit
L. Jones
A.N. Gomez
L. Kaiser
I. Polosukhin

title: Attention is all you need

authors:
P. Vepakomma
O. Gupta
T. Swedish
R. Raskar

title: Split learning for health: Distributed deep learning without sharing raw patient data

venue: arXiv preprint arXiv:1812.00564,

authors:
C. Wachinger
P. Golland
W. Kremen
B. Fischl
M. Reuter
A.D.N. Initiative

title: Brainprint: A discriminative characterization of brain

venue: morphology. NeuroImage,

authors:
G. Wang
J.C. Ye
B. De Man

title: Deep learning for tomographic image reconstruction

authors:
X. Wang
Y. Peng
L. Lu
Z. Lu
M. Bagheri
R.M. Summers

title: Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases

authors:
Z. Wang
M. Song
Z. Zhang
Y. Song
Q. Wang
H. Qi

title: Beyond inferring class representatives: User-level privacy leakage from federated learning

venue: In IEEE INFOCOM 2019-IEEE Conference on Computer Communications,

authors:
L. Wynants
B. Van Calster
G.S. Collins
R.D. Riley
G. Heinze
E. Schuit
M.M. Bonten
D.L. Dahly
J.A. Damen
T.P. Debray

title: Prediction models for diagnosis and prognosis of covid-19: systematic review and critical appraisal

authors:
Z. Xu
Y. Huo
J. Park
B. Landman
A. Milkowski
S. Grbic
S. Zhou

title: Less is more: Simultaneous view classification and landmark detection for abdominal ultrasound images

authors:
Q. Yang
Y. Liu
T. Chen
Y. Tong

title: Federated machine learning: Concept and applications

venue: ACM Transactions on Intelligent Systems and Technology (TIST),

authors:
W. Ye
J. Yao
H. Xue
Y. Li

title: Weakly supervised lesion localization with probabilistic-cam pooling

venue: arXiv preprint arXiv:2005.14480,

authors:
M. Yurochkin
M. Agarwal
S. Ghosh
K. Greenewald
N. Hoang
Y. Khazaeni

title: Probabilistic federated neural matching

authors:
B. Zhao
K.R. Mopuri
H. Bilen

title: idlg: Improved deep leakage from gradients

venue: arXiv preprint arXiv:2001.02610,

authors:
S.K. Zhou
H. Greenspan
C. Davatzikos
J.S. Duncan
B. van Ginneken
A. Madabhushi
J.L. Prince
D. Rueckert
R.M. Summers

title: A review of deep learning in medical imaging: Imaging traits, technology trends, case studies with progress highlights, and future promises

venue: Proceedings of the IEEE,

authors:
J. Zhu
B. Shen
A. Abbasi
M. Hoshmand-Kochi
H. Li
T.Q. Duong

title: Deep transfer learning artificial intelligence accurately stages COVID-19 lung disease severity on portable chest radiographs

venue: PloS one,

heading: 1 Introduction

text:
After its earlier success in many fields, deep neural networks have found a pervasive suite of applications in healthcare research including medical imaging, becoming a new de facto standard [54, 19, 13, 49, 17, 63, 6, 58, 24]. Training these networks requires a vast amount of data to achieve robust performance [8, 11, 47]. Despite the fact that multi-center collaboration is mandatory due to the shortage of labeled data in a single institution, collaboration in healthcare research is heavily
∗Authors contributed equally.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
impeded by difficulties in data sharing stemming from the privacy issues and limited consent of patients [50, 38, 53].
To alleviate this problem, the distributed machine learning methods, devised to enable the computation on multiple clients and servers leaving data to reside on the source devices, can be effectively leveraged for healthcare research [7, 43]. Federated learning (FL) is one of these methods which enables model training on a large corpus of decentralized data [26, 32, 59]. However, FL still holds several limitations in that it depends on clients’ computational resources for its client-side parallel computation strategy for update and is not free from privacy concerns [27, 31, 52]. In contrast to FL, another distributed machine learning method, split learning (SL) offers better privacy and requires lower computational resources of clients by splitting the network between clients and the server [22, 52], but still possess problems that it shows significant slower convergence than FL and can not learn under non-independent and non-identically distributed (non-IID) data [20].
Especially under unprecedented pandemic of an emerging pathogen like COVID-19, under which direct multi-national collaboration is deterred for prevention of epidemics, the collaboration via these distributed machine learning approaches is becoming increasingly important, since these enable to build a model with performance tantamount to data-centralized learning without any direct sharing of raw data between institutions to offer privacy.
Recently proposed Vision Transformer (ViT) architecture [14], inspired by astounding results of Transformer-based models on natural language processing (NLP), have demonstrated impeccable performance on many vision tasks by enabling to model long dependencies within images. Besides this strength, the straightforward design of the Transformer allows to easily decompose the entire network into parts: the head for extracting features from the input image, the Transformer body to model the dependency between features, and the tail used for mapping features to task-specific output. One of the important contributions in this paper is the observation that this configuration is optimal for SL where a network should be split into the parts for clients and servers. In addition, as suggested in [9], the Transformer body with sufficient capacity can be shared between various tasks, being suitable for multi-task learning (MTL) to leverage robust representation from multiple related tasks to enhance the generalization performance of individual tasks.
Accordingly, here we propose a novel Federated Split Task-Agnostic (FESTA) framework equipped with a Transformer to simultaneously process multiple chest X-ray (CXR) tasks including diagnosis of COVID-19, emulating a real collaboration between several hospitals. To validate the practicability of FESTA, we also implemented the framework using a friendly federated learning framework (Flower) protocol [2], confirming seamless integration of various components. Experimental results show that our framework can show stable performance even under non-IID settings which is a frequently faced situation for collaboration between hospitals while offering privacy, by amalgamating FL and SL to maximally exploit their main advantages. In addition, we show that the proposed FESTA Transformer along with MTL improves the performances of individual tasks. In summary, our contributions are two folds:
• We proposed a novel FESTA learning framework equipped with the ViT by utilizing its decomposable design to amalgamate the merit of FL and SL.
• We showed that the model trained with the FESTA framework can leverage the robust representations from multiple related tasks to improve the performance of the individual task.

heading: 2 Related Work

text:
Distributed machine learning. FL is a distributed machine learning approach originally proposed by Google [26] to enable the training of a model via distributed devices and data. It eliminates the need to aggregate the raw data in a centralized way by enabling the model to be updated on the edge devices (e.g. mobile phones, computers in hospitals). Specifically, during the training, the server initializes a global model and sends it to each client. The clients then train the model with their local data in parallel and return the updated model to the server. Then, the server aggregates and distributes those models by a method such as federated averaging (FedAvg) [32] to update the global model. This process (called round) continues repeatedly until the model converges. Though FL enables
decentralized training in a privacy-preserving manner, it still holds limitations as it largely depends on the computation resources of clients and is vulnerable to model inversion attacks [56].
Different from FL, SL divides the neural network into several sub-networks, and these separated sub-networks are trained under distributed setting [16]. In detail, the first sub-network is trained on the client-side with local data and then passes the feature to the second sub-network located in the server. The server can access the only feature from the first sub-network, and train the second sub-network to send the subsequent feature to the third sub-network on the client. Finally, the third sub-network is able to provide the output of the overall network. By inserting black-box sub-networks in both client and server sides, it is possible to offer better privacy than FL. Besides the privacy benefit, SL uses less computational resources than FL at the client-side. Nevertheless, since the one cycle of forward and backward passes is finished after data and gradients move back and forth across the sub-networks distributed on multiple sides, the convergence of SL is considerably slower than FL. In addition, it was reported that the convergence is not reached at all under the non-IID setting [20].
Multi-task learning. MTL is a learning strategy to improve the generalization performance of a specific task with the help of information from other related tasks. In MTL, models for multiple related tasks are trained simultaneously [5, 16]. In its early era, the motivation of MTL is to mitigate the data insufficiency problem where the number of labeled data is limited for each task to train an accurate learner. MTL helps to reuse knowledge and thereby reduce the requirement for labeled data for each task, exhibiting that the MTL model can achieve better performance than the single-task counterpart in many fields ranging from computer vision [15, 34, 36, 28] to NLP [30, 21, 29, 42]. With increased data, the MTL model can learn more robust representations via knowledge sharing among multiple tasks, resulting in improved performance and less overfitting for the individual task.
Vision Transformer. Transformer [51], which was originally developed for NLP, is a deep neural network based on an attention mechanism that utilizes an appreciably large receptive field. After achieving state-of-the-art (SOTA) performance in NLP, it has also inspired the vision community to explore its application on vision tasks to utilize its ability to model long-range dependency within an image [25]. The ViT was one of the successful attempts to apply Transformer directly to images, achieving excellent results compared to the SOTA convolutional neural networks in image classification tasks [14]. Furthermore, in addition to the superb performance, the straightforward modular design of ViT facilitates broad applications in many tasks only with minimal change. Chen et al. [9] proposed image processing transformer, which is one of the successful multi-task model for various computer vision tasks, by splitting ViT into shared body and task-specific heads and tails, suggesting that Transformer body with sufficient capacity can be shared across relevant tasks. However, they leveraged encoder-decoder design and the usefulness of the multi-task ViT model was not evaluated along with the distributed learning methods.
Recently, ViT was successfully used for diagnosis and severity prediction of COVID-19, showing the SOTA performance [35]. Specifically, to alleviate the overfitting problem with limited data available, the overall framework is decomposed into two steps: the pre-trained backbone network to classify common low-level CXR features, which was leveraged in the second step by Transformer for high-level diagnosis and severity prediction of COVID-19. By maximally utilizing the merit of the large-scale database containing more than 220,000 CXR images, the model has attained stable generalization performance as well as SOTA performance in a variety of external test data from different institutions, even with the limited number of labeled cases for COVID-19.

heading: 3 Split Task-Agnostic Transformer for CXR COVID-19 Diagnosis

text:
Inspired by these works, here we are interested in utilizing ViT for distributed learning in COVID19 CXR diagnosis, where the collaboration via these distributed machine learning approaches is becoming increasingly important by offering privacy and still allowing similar performance to the data-centralized learning.
The reason we are interested in ViT architecture for this purpose is that the natural configuration of ViT may be optimal for MTL as well as SL where the easily decomposable modular design of the network is preferred, suggesting a possibility to maximally reconcile the merits of MTL and SL through ViT architecture. Specifically, the clients just train the head and tail parts of the network, whereas the Transformer body is shared across multiple clients. Then, the embedded features from
the head network from multiple clients can be leveraged in the second step by Transformer to process individual tasks including diagnosis of COVID-19. Furthermore, by maximally utilizing the merit of the large-scale database from CXR for various tasks, our goal is to demonstrate stable generalization performance as well as SOTA performance in the external test dataset.

heading: 3.1 FESTA: Federated Split Task-Agnostic Learning

text:
The concept of the proposed Federated Split Task-Agnostic (FESTA) learning process is illustrated in Fig 1(a) and (b). Let C = ⋃K k=1 Ck be a group of client sets with different CXR tasks, where K denotes the number of tasks and Ck has one or more clients with different datasets each other for the k-th task, i.e. Ck = {ck1 , ck2 , . . . , ckNk : Nk ≥ 1}. Each client c ∈ Ck has its own task-specific network architecture for a headHc and a tail Tc, which are connected to Transformer B in the server. In our FeSTA framework, the server first initializes the weights of the Transformer body and taskspecific heads, tails for each task k. Then, it distributes the initialized weights of heads and tails to each client c ∈ Ck. For round i = 1, 2, . . . R, each client (e.g. hospital) perform the forward propagation on their task-specific head and pass their intermediate feature to the server. Specifically, using the local training data {(x(i)c , y(i)c )}Nci=1, the head network Hc encodes the smashed feature maps h(i)c and sent it to the server: h (i) c = Hc(x(i)c ). Then, the server-side Transformer body B receives the feature from the all clients and generate the features b(i)c for c in parallel: b (i) c = B(h(i)c ). Resulting smashed features from the Transformer are allocated to task-specific tail to produce the final prediction ŷ(i)c = Tc(b(i)c ) according to each task, and the forward path finishes. Subsequently, the loss for each client can be calculated as ℓc(y (i) c , Tc(B(Hc(x(i)c )))) where ℓc(y, ŷ) refers to the c client- specific loss between the target y and the estimate ŷ. By minimizing the loss with respect to the tail weight, the gradients of the local tails are passed reversely to the server. After receiving the gradients, the server performs the back-propagation on the server-side body model, sends the gradients back to the clients.
Specifically, for the task-agnostic body update, the following optimization problem is solved:
min B ∑ c∈C Nc∑ i=1 ℓc(y (i) c , Tc(B(Hc(x(i)c )))), (1)
For the task-specific fine-tuning, the following optimization problem is solved:
min Hc,Tc Nc∑ i=1 ℓc(y (i) c , Tc(B(Hc(x(i)c )))), (2)
Finally, the server aggregates and averages the weights of local heads and tails from the clients to update the global heads and tails among the clients for the same tasks via FedAvg, and distributes back the updated global weights of heads and tails for each task k to the clients. The algorithm is formally presented in Algorithm 1.
Algorithm 1: FESTA: Federated Split Task-Agnostic learning 1 Function ServerMain: 2 Initialize the body weight w(1)B and client head/tail weights (w̄H,k, w̄T ,k) for each task k ∈ {1, ...,K} in server 3 for rounds i = 1, 2, . . . R do 4 for tasks k ∈ {1, 2, . . .K} do in parallel 5 for clients c ∈ Ck do in parallel 6 if i = 1 or (i− 1) ∈ UnifyingRounds then 7 Set client (w(i)Hc , w (i) Tc )← (w̄H,k, w̄T ,k)
8 h (i) c ← ClientHead(c) 9 b (i) c ← B(h(i)c )
10 ∂L(i)c ∂b (i) c ← ClientTail(c, b(i)c ) & Backprop. 11 (w (i+1) Hc , w (i+1) Tc )← ClientUpdate(c, ∂L(i)c ∂h (i) c )
12 Update body w(i+1)B ← w (i) B − η K K∑ k=1 ∑ c∈Ck ∂L(i)c Nk∂w (i) B 13 if i ∈ UnifyingRounds then 14 for tasks k ∈ {1, 2, . . .K} do 15 Update (w̄H,k, w̄T ,k)← ( 1Nk ∑ c∈Ck w (i+1) Hc , 1 Nk ∑ c∈Ck w (i+1) Tc )
16 Function ClientHead(c): 17 xc ← Current batch of input from client c 18 returnHc(xc) 19 Function ClientTail(c, bc): 20 yc ← Current batch of label from client c 21 Lc ← ℓc(yc, Tc(bc)) & Backprop. 22 return ∂Lc∂bc 23 Function ClientUpdate(c, ∂Lc∂hc ): 24 Backprop. & (wHc , wTc)← (wHc − η ∂Lc∂wHc , wTc − η ∂Lc ∂wTc ) 25 return (wHc , wTc)

heading: 3.2 Multi-task CXR learning

text:
For synergistic performance improvement, we explore the following three tasks that are commonly used for CXR: classification, segmentation, and object detection. These three tasks were separately trained for the individual model, while used simultaneously to train and to evaluate the task-agnostic model for multiple related tasks. The details of each task and dataset are as follows.
COVID-19 classification. This is the main task we want to achieve through FESTA. Since we initialized the weights of classification heads to be the robust feature extractor trained on prebuilt large data corpus containing common CXR findings, the classification heads were initialized with the pre-trained weights from CheXpert [23] dataset containing 10 CXR findings (no finding, cardiomegaly, opacity, edema, consolidation, pneumonia, atelectasis, pneumothorax, pleural effusion, support device) labeled by experts. From pre-training on the CheXpert dataset, we excluded 32,387 lateral view images, and 29,420 posterior-anterior (PA) and 161,427 anterior-posterior (AP) view data were finally used. Table 1 summarizes dataset resources and partitioning for the classification task. We used both public datasets containing labels of infectious disease (Valencian Region Medical
Image Bank [BIMCV] [12], Brixia [44, 4], National Institutes of Health [NIH] [55]), and CXR data deliberately collected from four hospitals labeled by board-certified radiologists. We put one hospital data aside as an external test dataset to evaluate the classification performance for real-world applications. Overall, 17,183 PA view CXR images were used for training/validation and 365 PA view CXR images for the test. In addition, we performed the experiments in the view-agnostic setting by adding AP view CXRs, to further extend real-world applicability as provided in Appendix C.1. By adding AP view data, the total amounts of CXRs were increased from 17,183 to 24,180 for training/validation and 365 to 556 for external test datasets. Notably, the number of COVID-19 cases was increased from six to 81 in the external test dataset. We modeled data from different sources as individual clients, emulating the real-world collaboration between hospitals. This setting is important to validate our method in non-IID data distribution. The study was ethically approved by the Institutional Review Board at each participating hospital and the requirement for informed consent was waived.
Segmentation. For the segmentation task, we have used the Society for Imaging Informatics in Medicine and the American College of Radiology Pneumothorax Segmentation Challenge dataset [45] consisting of 12,047 CXR images for training, 3,205 images for testing. The training dataset was divided randomly with a 4:1 ratio into training and validation datasets, and we developed and finetuned the model for pneumothorax segmentation with these datasets. Afterward, the segmentation performance of the model was evaluated in testing datasets. Since the data for segmentation was anonymized, it was not possible to divide the dataset according to the sources of acquisition. Instead, we randomly divided the entire dataset into the two clients, emulating the collaboration between two hospitals.
Object detection. For the object detection task, the model was constructed to detect lung opacities in CXR with the Radiological Society of North America (RSNA) Pneumonia Detection Challenge dataset [41] that consists of CXR images and labels with bounding boxes and detailed class information of 26,684 subjects. We randomly divided the entire dataset with a 3:1 ratio into training and testing datasets, with which model was trained and evaluated. Similar to the segmentation task, as the data contains no information about the sources of acquisition, the dataset was randomly divided into two clients to simulate the collaboration between hospitals.

heading: 4 Experimental Results

heading: 4.1 Implementation details

text:
For the head and tail parts of our model, the networks specialized for each task were utilized. For classification, we used the modified version of the network proposed by Ye et al. [60], which comprises DenseNet combined with Probabilistic Class Activation Map (PCAM) operations for the classification task, since it achieved outstanding performance in the CXR classification competition. For segmentation, we adopt TransUNet [10] tthat inserts a Transformer between the convolutional neural network (CNN) encoder and decoder of UNet [40] to take advantage of both architectures. Similarly, the 2nd place solution in the RSNA pneumonia detection challenge, which is a modified version of RetinaNet for object detection, was utilized. After splitting these models into head and tail, the feature maps between head and tail were mapped to the feature maps with the same dimension of 16 × 16 × 768, and used as input of Transformer body. The Transformer body, equipped with 12 layers of standard Transformer encoder with 12 attention heads, transforms the feature maps to
embed better representation. Then, the resulting feature maps from the body were passed to tails to yield the outcomes.
As suggested in Park et al. [35], the head for classification was first initialized with pre-trained weights from the CheXpert dataset. We minimized the cross-entropy loss for the classification task. For the segmentation model, we minimized the binary cross-entropy loss combined with dice and focal loss. Finally, for the detection task, we minimized the sum of box classification, box regression, and image classification losses as suggested by Gabruseva et al. [18]. For all tasks, the batch size was 2 per client, and the warm-up step was 500. We set the number of total rounds to 12,000, and the weights of each clients’ head and tail underwent FedAvg per 100 rounds by the server.
Fig. 2 illustrates the implementation details and the experimental settings of the proposed method. To simulate single-task learning (STL) in Fig. 2(b), we considered each six data sources as different clients for the classification task. On the other hand, for MTL shown in Fig. 2(c), 10 clients were simultaneously used, with six clients for classification and two clients for segmentation and detection tasks, respectively. For the segmentation and detection tasks, the training data set was randomly split into two subsets, whereas non-IID distribution was used for the classification task. To adjust the loss scale, the customized weights of 1:2:2 were applied for classification, segmentation, and detection tasks to update the common body weights. We divided the MTL into two steps, jointly training the task-specific heads, tails, and the task-agnostic body (6,000 rounds), and fine-tuning only the task-specific heads, tails with the body weights fixed (6,000 rounds). By fixing the parameters of the shared Transformer body during the second step, the best models can be selected for different tasks according to the performance evaluation metrics of each task, even at the different rounds.
The FESTA, FL, and SL simulation was performed on the modified version of Flower (licensed under an Apache-2.0 license) [2] FL framework. All experiments were performed with Python version 3.8 and Pytorch version 1.7 on Nvidia RTX 3090, 2080 Ti, and 1080 Ti. For more details of implementation, refer to Appendix A.
Performance metrics. We used the area under the receiver operating characteristic curve (AUC) to evaluate the diagnostic performance in the classification task. To evaluate the accuracy of segmentation, the Dice coefficient was used to quantitatively measure the overlap between the segmentation results by model and the ground truths. The detection results were evaluated by calculating the mean
average precision (mAP) at the different intersection over union, with a threshold range from 0.4 to 0.75 with a step size of 0.05 as suggested in RSNA [41]. All experiments have been run and evaluated with three different random seeds for the weight initialization to prevent the chance of confusing the results.

heading: 4.2 Results

text:
FESTA vs. Other strategies. We compared the performance of classification model trained with FESTA with a data-centralized setting and other distributed learning strategies for COVID19 classification task under the non-IID setting. To simulate data-centralized training, the whole network with connected head, body, and tail was trained with the integrated dataset of all six sources. To simulate FL, the whole network was aggregated and distributed by the server with FedAvg as suggested by McMahan et al. [32]. On the other hand, for SL, the split sub-networks reside in the clients and server-side, and the client-side sub-networks were not aggregated as in [52]. The same experimental settings and hyperparameters were applied for a fair comparison. For more details of the data-centralized learning and other distributed learning methods, refer to Appendix B.
As shown in Table 2, our method achieved comparable performance to the data-centralized learning method as well as outperformed the existing distributed learning methods, suggesting the superiority of our method over other methods. Of note, the performance could be further enhanced with MTL, which is a distinct strength of the proposed method.
In our additional experiments after adding AP view data, the model showed even better performance with the increased number of cases (Appendix C.1). This remarkable view-agnostic behavior of the model incentivizes the real-world application of computer-aided diagnosis of COVID-19, as the recent review on artificial intelligence models for COVID-19 diagnosis claims that the AI model, which has been pouring out a lot recently, are not helpful at all from the viewpoint of clinical application [57], in which the performances were substantially unstable by the factors like the view of CXR images [37].
Multi-task learning vs. Single-task learning. We evaluate whether the task-agnostic Transformer body contributes to improving the performance of the entire model by leveraging better representation from the MTL with several related tasks, namely classification, segmentation, and object detection in this work. As provided in Table 3, the models trained with the MTL approach showed at least comparable or even better performance compared with STL counterparts without the need to create an individual body model for each task or sharing a large body model between clients, which suggest the distinct merit of MTL approach with our framework.
In addition to the STL models, the MTL models trained with our framework outperformed taskspecific expert models and provided comparable or even better performances to Kaggle’s winning solution for the same tasks. Of note, when we substitute the shared layers of the Transformer body with CNN architecture with similar complexity, the performance gain was no longer maintained, suggesting the suitability of Transformer architecture in learning shared representation between the related tasks as provided in the additional experiments in Appendix C.2.
Model sizes and communicative benefit. Table 4 shows the numbers of parameters and the sizes of sub-networks. The task-agnostic body is the largest among sub-networks in terms of both the number of parameters and model size. This suggests that the largest part of the model does not need to be aggregated and distributed between client-client and client-server, which offers substantial
communicative benefit. In all tasks, the size of the body is more than half of the entire network. Therefore, not having to share this huge part means the total training time can be reduced considerably. In our experiments, the training time with the proposed method was four times shorter than the FL approach in which the entire network is shared and distributed. In addition, the proposed method offers another benefit of saving the computational resources of the server by processing multiple tasks with a single body model to enable the server to efficiently handle the requests for various tasks from many clients simultaneously.
Nonetheless, there remains a concern of communicative heavy load between the server and clients caused by the frequent transmission of features and gradients. Our further analysis of communication costs including features and gradients transmission are provided in the additional experiments, which suggests that the communication costs of the proposed FESTA framework was substantially lower than those of FL, although higher than SL. For detailed results of the computation for communication costs, refer to Appendix C.3.

heading: 4.3 Ablation study

text:
Role of Transformer body. As provided in Table 5, we first performed the ablation study to verify the contribution of the Transformer body on the server. The model without the server-side Transformer body, which is identical to the DenseNet-121 model equipped with PCAM operation [60], was built and trained with the same setting, and compared with the proposed model containing the Transformer body. The AUC values were higher with the Transformer body either for STL or MTL compared to those without the Transformer body with the statistical significance, implying the advantageous role of the server-side Transformer body as in Appendix C.4.
Numbers of round for averaging. Determining the optimal number of rounds for averaging is important. If the aggregation, averaging, and distribution, namely FESTA is performed too frequently, the cost and required resources for communication increases, which can interfere or even preclude the learning process. On the contrary, if averaged too rarely, naive averaging of the learned model parameters of local learners can be devastating, resulting in nonsensical parameters in some layers [61]. Therefore, we performed the ablation to determine the optimal number of rounds to conduct FESTA. As shown in Table 5, the averaging per 100 rounds showed the comparable or better performance to less or more frequent counterparts.
More results of ablation studies are provided in Appendix D.

heading: 5 Conclusions

text:
In this paper, we proposed a novel Federated Split Task-Agnostic (FESTA) framework suitable to leverage the formidable benefit of ViT to simultaneously process multiple CXR tasks including the
diagnosis of COVID-19. With the optimal configuration of ViT for modulation, it was possible to surpass the existing methods for distributed learning and achieve the performance comparable to data-centralized learning with our framework, even under the skewed data distribution. Moreover, our framework alongside clients to process multiple related tasks also improves the performances of individual tasks, while eliminating the need to share the data and large weights of the body network. These results suggest the suitability of the Transformer for collaborative learning in medical imaging and pave the way forward for future real-world applications.

heading: 6 Limitation and Potential Negative Societal Impacts

text:
This work is more like a proof-of-the-concept study, rather than a ready-to-use solution for the industry. Therefore, it holds some limitations which may occur in a real-world application. Recent works on privacy attacks in the FL setting have implied that the belief that "Privacy can be protected by the decentralized nature of the FL" is not true [1, 56, 62]. In this work, however, the experimental results regarding the privacy issue such as threatening privacy via inversion attack are not suggested. Secondly, although the unique challenges of FL have been suggested by previous work including the problems of significant bottleneck in communication, stragglers, and fault tolerance which is exacerbated than in typical data-centralized learning [46, 27], we have not conducted the experiments to evaluate the robustness of our method against these unique challenges.
Although distributed learning-enabled learning without sharing data, decentralization is not a panacea against the privacy problem. Similar to the previously distributed learning methods, a potential risk arises from these limitations that our algorithm may not be free from privacy issues via model inversion attack against the server, since the server retains the parameters of the entire network in process of FedAvg of the heads and tails despite the split design of the sub-networks upon both client and server-sides. Although the risk could be mitigated to some degree with the specific settings (e.g. small gradient due to pre-trained backbone, deeper network, more pooling layer, a mixture of multiple tasks), the privacy problem should not be ignored since we aim to use this method in collaboration with hospitals where the patient privacy is a matter of the highest priority. Therefore, the methods to enhance the security such as differential privacy [33], secure multi-party computation [3], data compression [64] and authenticated encryption [39] as well as the recent method to prevent gradient inversion attack without sacrificing FL performance [48] should be utilized along with the proposed method to further reduce the risk of privacy leakage in a real-world application.
Acknowledgements and Disclosure of Funding
This research was funded by the National Research Foundation (NRF) of Korea grant NRF2020R1A2B5B03001980. This work was also supported by Institute of Information and communications Technology Planning and Evaluation (IITP) grant funded by the Korea government(MSIT) (No.2019-0-00075, Artificial Intelligence Graduate School Program(KAIST)), and by the KAIST Key Research Institute (Interdisciplinary Research Group) Project.

title: Federated Split Vision Transformer for COVID-19 CXR Diagnosis using Task-Agnostic Training