text:
Identifying Twitter users who repost unreliable news sources with linguistic information

Social media has become a popular source for online news consumption with millions of users worldwide. However, it has become a primary platform for spreading disinformation with severe societal implications. Automatically identifying social media users that are likely to propagate posts from handles of unreliable news sources sometime in the future is of utmost importance for early detection and prevention of disinformation diffusion in a network, and has yet to be explored. To that end, we present a novel task for predicting whether a user will repost content from Twitter handles of unreliable news sources by leveraging linguistic information from the user’s own posts. We develop a new dataset of approximately 6.2K Twitter users mapped into two categories: (1) those that have reposted content from unreliable news sources; and (2) those that repost content only from reliable sources. For our task, we evaluate a battery of supervised machine learning models as well as state-of-the-art neural models, achieving up to 79.7 macro F1. In addition, our linguistic feature analysis uncovers differences in language use and style between the two user categories.

INTRODUCTION
Social media has become an important source for online news consumption, widely adopted by news outlets, individual journalists and end users (Hermida et al., 2012;Kalsnes & Larsson, 2018). The use of social media enhances civic engagement and political participation offering a direct way of communication with millions of users worldwide (Bennett, 2008;Gil de Zúñiga, Jung & Valenzuela, 2012).
A widespread phenomenon in social media platforms is the generation and dissemination of unreliable content (e.g., fabricated or deceptive information, exaggerated headlines, pseudoscience, propaganda) by particular news outlets that typically act as disinformation diffusion sources. Diffusion of disinformation in social media typically begins when a news source publishes a story that subsequently is propagated by users via reposting (e.g., retweeting, sharing) it to their personal networks of friends. It has been observed that disinformation propagates faster compared to credible information amongst users in social media (Vosoughi, Roy & Aral, 2018;Lazer et al., 2018). Furthermore, when a user comes across an unreliable story once, it is enough to increase their later perception of its accuracy (Pennycook, Cannon & Rand, 2018). Media that disseminate unreliable content often aim to manipulate people's opinion and influence election results which has implications to political stability worldwide (Allcott & Gentzkow, 2017;Humprecht, 2018).
Previous studies suggest that factors positively associated with the sharing unreliable news posts on social network include psychological factors (e.g., online trust, self-disclosure, fear of missing out, and ideological extremity) and political orientation (e.g., right-leaning) Talwar et al., 2019;Hopp, Ferrucci & Vargo, 2020). In this study, we investigate whether user language information can help identify who will repost items from Twitter handles of unreliable news sources. To test this hypothesis, we define a new classification task seeking to predict whether a user is likely to repost content from unreliable news sources given all the history of the user's posts up to the first repost of a news item, i.e., before they actually do it. Early detection of users that are likely to repost content from unreliable sources can help: (i) political scientists and journalists to analyse which topics of discussion are related to disinformation on a large scale (Bode & Vraga, 2015); (ii) social media platforms such as Twitter or Facebook to prevent the diffusion of potentially unreliable stories in the network (Castillo, Mendoza & Poblete, 2011;Conroy, Rubin & Chen, 2015;Shu et al., 2017); and (iii) psychologists to complement studies on personality analysis . The main contributions of our paper are as follows: • We frame a novel binary classification task for early detection of users sharing content from unreliable news sources using diverse language features extracted from the aggregate of users' original tweets; • We evaluate a battery of traditional feature-based and neural predictive models that achieve up to 79.7 F1 score; • We perform a qualitative analysis of our results to shed light into language use differences of users that diffuse content from unreliable or reliable news sources;

BACKGROUND AND RELATED WORK Disinformation in social media
Social media has become a primary platform for live-reporting (Engesser & Humprecht, 2015) with the majority of mainstream news media operating official accounts (e.g., @BBC and @Reuters on Twitter). However, social media platforms are also regarded as a fertile breeding ground for the diffusion of unverified, fabricated and misleading information due to its openness and popularity (Zubiaga et al., 2018a). This type of information is often referred to as misinformation. Misinformation has been defined as an umbrella term to include any incorrect information that is diffused in social networks (Wu et al., 2019). On the other hand, disinformation is defined as the dissemination of fabricated and factually incorrect information with main aim to deliberately deceive its audience (Glenski, Weninger & Volkova, 2018b

Categorization of unreliable news sources
Unreliable news sources are categorized by their intention and the degree of authenticity of their content (Rubin, Chen & Conroy, 2015;Rashkin et al., 2017). Rubin, Chen & Conroy (2015) define three categories of deceptive news: (1) serious fabrications including unverified claims coupled with exaggerations and sensationalism; (2) large-scale hoaxes that are masqueraded as credible news which could be picked up and mistakenly disseminated; and (3) humorous fakes that present fabricated purposes with no intention to deceive. Rashkin et al. (2017) extended these three groups of misinformation into a more fine-grained classification: • Propaganda news uses misleading information and writing techniques (Martino et al., 2019) to promote a particular agenda (Glenski, Weninger & Volkova, 2018a). Propaganda news sources that mostly share unreliable stories often aim to manipulate people's opinions and influence election results posing a threat to political stability worldwide (Allcott & Gentzkow, 2017;Humprecht, 2018).
• Clickbait is defined as using exaggerated headlines for grabbing user attention and misleading public opinion (Glenski, Weninger & Volkova, 2018a).
• Conspiracy theories can be understood as a kind of distorted interpretation of real events from people with ulterior motives such as political and religious groups (Goertzel, 1994;Byford, 2011).
Recent efforts on detecting and index unreliable news sources rely on crowdsourcing and experts 1 to annotate the reliability of the news media Baly et al., 2018;Glenski, Weninger & Volkova, 2018a).

Previous work on combating online disinformation
Previous work on combating diffusion of disinformation in social media (Castillo, Mendoza & Poblete, 2011;Conroy, Rubin & Chen, 2015;Shu et al., 2017) has focused on characterizing the trustworthiness of (1) news sources (Dong et al., 2015;Baly et al., 2018); (2) news articles (Rashkin et al., 2017;Horne et al., 2018;Potthast et al., 2018;Pérez-Rosas et al., 2018); and (3) individual claims including news article headlines and rumors (Popat et al., 2016;Derczynski et al., 2017;Volkova et al., 2017;Zubiaga et al., 2018b;Thorne & Vlachos, 2018). Zhou et al. (2019) present a novel task for detecting the check-point which can early-detect a rumor propagated in a social network. Martino et al. (2019) develop models for detecting up to 18 writing techniques (e.g., loaded language, slogans, flag-waving, exaggeration, etc.) used in propaganda news. Similarly, Pathak & Srihari (2019) introduced a corpus of news articles related to US politics containing false assertions which are written in a compelling way. At the user level, social scientists and psychologists have utilised traditional methods, such as recruiting participants for online surveys and interviews, to explore cognitive factors which may influence people's ability to distinguish fake news (Pennycook, Cannon & Rand, 2018). For instance, the lack of analytic thinking plays 2 Note that one could use a user's social network information but this is out of the paper's scope because we are interested in analysing differences in language use between the two groups of users. a vital role in recognition of misinformation . Previous datadriven studies include (1) analysing bots participation in social media discussion (Howard & Kollanyi, 2016) and distinguishing between automated and human accounts (Mihaylov & Nakov, 2016); (2) identifying user reactions (e.g., agreement, answer, appreciation, humor, etc.) to reliable/unreliable news posts (Glenski, Weninger & Volkova, 2018a); and (3) analyzing the demographic characteristics of users propagating unreliable news sources (Glenski, Weninger & Volkova, 2018b), e.g., low-income and low-educated people are more likely to propagate unreliable news sources on social networks.
In our paper, we tackle the problem of early detecting users who are likely to share post from unreliable news sources which is rather different to the focus of previous work on disinformation detection and analysis.

TASK DESCRIPTION
Our aim is the early detection of social media users that are likely to repost content from unreliable news sources before they actually share any other news items at all. To that end, we define a novel binary classification task for predicting whether a social media user will propagate news items from unreliable or reliable news sources using solely language information 2 .
We assume a training set of n users U = {(x 1 ,y 1 ),...,(x n ,y n )} where x i is a vector representation of language information extracted from user's i timeline consisting of posts up to the first repost of any news item, and y i ∈ {reliable,unreliable} is an associated user label. Given U , we learn a function f that maps a new user j into one of the two categoriesŷ = f (x j ) using any suitable supervised machine learning algorithm.
We consider the posts up to the first share of any news item, ensuring that we only use prior information that is not linked to any news source. One could also introduce a cut-off in time or keep the top k posts but we choose to use all the available information possible. We opted to define a binary task (i.e., reliable vs. unreliable) rather than a fine-grained classification task (i.e., propaganda, hoax, clickbait, and reliable) because propagating any type of disinformation might be equally harmful. For similar reasons, we are not focusing on modeling the proportion of posts from reliable/unreliable sources in users' Twitter Timeline.

DATA
At present, there is no existing dataset to model our predictive task. For the purposes of our experiments, we develop a new dataset of Twitter users who have retweeted posts from unreliable or reliable news sources. We opted for Twitter because the majority of accounts and posts are publicly available and it has been extensively used in related work Rashkin et al., 2017;Glenski, Weninger & Volkova, 2018a).
Our data collection process consists of three main steps (summarized in Fig. 1): (1) collection of posts from reliable and unreliable news sources; (2) collection of candidate users that have shared at least one of the posts collected during the first step; (3) assignment of users to the reliable and unreliable categories.

Collecting posts from unreliable and reliable news sources
To identify users that have shared content from a particular news source, we first need to collect posts from reliable and unreliable news sources. For that purpose, we use a widely-used and publicly available list of English news media Twitter accounts provided by Volkova et al. (2017) and Glenski, Weninger & Volkova (2018a), which contains 424 English news media sources categorized in unreliable (satire, propaganda, hoax, clickbait) and reliable, following Rubin, Chen & Conroy (2015). For each news source, we retrieve the timeline (most recent 3,200 tweets) using the Twitter public API. We then filter out any retweets to ensure that we can collect only original posts from each Twitter account.
In this list, unreliable news sources (e.g., Infowars, Disclose.tv) have been annotated by digital journalism organisations (e.g., PropOrNot, fakenewswatch.com, etc.), while the reliable news media accounts (e.g., BBC, Reuters) have all been verified on Twitter and used in Glenski, Weninger & Volkova (2018a). Since satire news sources (e.g., The Onion, Clickhole) have humorous purposes (no desire to deliberately deceive (Rashkin et al., 2017)), we exclude them as in Glenski, Weninger & Volkova (2018b) resulting into 251 trusted and 159 unreliable sources. Note that the list does not exhaustively cover all the available sources but it is a representative sample for the purpose of our experiments. We also use the characterization of an entire news source as reliable/unreliable following Rashkin et al. (2017); Volkova et al. (2017) and not individual posts.

Collecting Candidate Users
We retrieve an initial set of approximately 15,000 candidate users by looking into the most recent user accounts who have retweeted at least an original tweet from each news source. Due to the public Twitter API limits, we do not have access to user likes of news items. Based on the user profile information, we filter out users with more than 3,200 tweets due to the Twitter public API limits, since we need access to the entire timeline to decide the category the user belongs to (see section Labeling Users). For the remaining users, we collect their entire timeline (up to 3,200 tweets) and we filter-out any user with majority of non-English tweets (i.e., tweets labelled as 'en' or 'en-gb' by Twitter). Then for each user, we count the number of retweets from reliable and unreliable news sources respectively. Subsequently, we remove all user's retweets (including tweets containing RT) and we keep only the tweets up to the first retweet of a news source for each user. Moreover, we only keep users with more than 10 original tweets.

Labeling users
Our classification task is defined as the early detection of users posting unreliable news sources before they actually do it for the first time. Therefore, candidate users are assigned into two categories (Unreliable, Reliable): • Unreliable. Users that have reposted unreliable sources at least three times (to ensure that this is a consistent behaviour) including the case when a user has shared both reliable and unreliable sources (modeling the ratio of unreliable/reliable it is out of the scope of early detection) are assigned to the unreliable class.
• Reliable. Users that have retweeted only reliable news sources are assigned to the reliable category.
Given that Twitter users can also share shortened URLs from unreliable news websites (e.g., http://www.infowars.com), we collect and expand all shortened URLs (e.g., 'https://t.co/example') extracted from the posts of users labeled as reliable. We then remove all users who have shared any URLs from unreliable news websites. Our data collection process yielded a set of 6,266 users (3,468 and 2,798 for reliable and unreliable respectively) with a total of 1,356,480 tweets (see Table 1).

Text preprocessing
We pre-process all tweets from all users by first lowercasing text and then tokenizing. Furthermore, we remove any stop words 3 and replace all URLs and @-mentions with url and usr markers respectively. See Table 1 for token statistics per user.

Ethics
Previous work on the study of who spreads misinformation in social networks has used data collected through survey questionnaires (i.e., self-report data) and trace data (i.e., user-generated content) (Talwar et al., 2019;Chen et al., 2015;Shu et al., 2019;Hopp, Ferrucci & Vargo, 2020). We employ similar standard practices on studying social media user behavior. Our work has received approval from the University of Sheffield Research Ethics Committee (Ref. No 025470) and complies with Twitter data policy for research (https://developer.twitter.com/en/developer-terms/agreement-and-policy). Note that we will not share the data for non-research purposes.

SVM
We use Support Vector Machines (SVM) with an Radial Basis Function (RBF) kernel (Joachims, 2002) for all of our feature-based models which can be considered as baselines.
We extract three types of language features: (1) Bag-Of-Words (BOW); (2) topics; and (3) Linguistic Inquiry and Word Count (LIWC), following a similar approach to recent work in computational social science (Rashkin et al., 2017;Pérez-Rosas et al., 2018;Zhang et al., 2018;Holgate et al., 2018): • We use BOW to represent each user as a TF-IDF weighted distribution over a 20,000 sized vocabulary with the most frequent unigrams, bigrams and trigrams. We only consider n-grams appearing in more than five and no more than 40% of the total users.
• We also represent each user over a distribution of 200 generic word clusters (Topics 4 ) computed on a Twitter corpus and provided by Preoţiuc-Pietro,  for unveiling the thematic subjects that the users discuss.
• We finally represent each user over a distribution of 93 psycho-linguistic categories represented by lists of words provided by the Linguistic Inquiry and Word Count (LIWC) 2015 dictionary (Pennebaker, Francis & Booth, 2001). 5 We also tested a Hierarchical Attention Network (Yang et al., 2016) achieving similar performance to BiGRU-ATT.
We then train SVMs using the three types of features: SVM-BOW, SVM-Topics and SVM-LIWC individually and in combination (SVM-All).

Avg-EMB
As our first neural model, we use a simple feed forward network (Avg-EMB) which takes as input the concatenation of all the tokenized tweets of a user. Words from users' tweets are first mapped into embeddings to compute an average embedding which represents the textual content posted by a user. Subsequently, the average embedding is passed to the output layer with a sigmoid activation function for binary classification.

BiGRU-ATT
Furthermore, we train a bidirectional Gated Recurrent Unit (Cho et al., 2014) with selfattention (Xu et al., 2015) (BiGRU-ATT). 5 The input is first mapped into word embeddings which are then passed through a BiGRU layer. A user content embedding is computed as the sum of the resulting context-aware embeddings weighted by the self-attention scores. The user content embedding is then passed to the output sigmoid layer.

ULMFiT
The Universal Language Model Fine-tuning (ULMFiT) (Howard & Ruder, 2018) is a transfer learning approach that uses a Average-Stochastic Gradient Descent Weight-Dropped Long Short-Term Memory (AWD-LSTM) (Merity, Keskar & Socher, 2017) encoder pre-trained on a large corpus using a language modelling objective. Following the standard adaptation process of ULMFiT, we first fine-tune the AWD-LSTM on language modelling using our dataset, and then we adapt the classifier into our binary task by replacing the output layer. We finally fine-tune ULMFiT using the gradual unfreezing method proposed in Howard & Ruder (2018).

T-BERT and H-BERT
Deep Bidirectional Transformers (BERT) (Devlin et al., 2018) is a state-of-the-art masked language model based on Transformer networks (Vaswani et al., 2017) pre-trained on large corpora, i.e., Books Corpus and English Wikipedia. Given the maximum input sequence length of BERT is 512, we first use a truncated version of BERT (T-BERT), which only takes the first 512 word pieces of each user as input. For our specific binary classification task, we add a fully-connected layer with a sigmoid activation on top of the user contextualized embedding obtained by passing input through BERT.
In order to take into account all the available textual information, we also employ a hierarchical version of BERT (H-BERT) since the majority of users' concatenated tweets is longer than 512 word pieces. Here, each input is split into several 512 length word chunks. The output for each chunk is averaged into a single vector before is passed through the same output layer as in T-Bert.

T-XLNet and H-XLNet
XLNet is a generalized autoregressive language model (Yang et al., 2019) similar to BERT which has achieved state-of-the-art performance in multiple NLP tasks. XLNet uses a perturbed language model objective instead of masked language model used in BERT.
Similar to BERT-based models, we employ both truncated and hierarchical versions of XLNet (i.e., T-XLNet and H-XLNet respectively) adapting them to our task using sigmoid output layers.

Experimental setup
We split our data into train (70%), development (10%), and test (20%) sets. The development set is used for tuning the hyper-parameters of the models.
We repeat the training and testing of each model three times by setting different random seeds and finally report the averaged macro precision, recall and F1-score. All dataset splits and random seeds will be provided for reproducibility. Table 2 presents the results of the SVM with all the feature combinations (BOW, Topics, LIWC and All) and the neural models.

Prediction results
In general, neural models achieve higher performance compared to feature-based models (SVM). Specifically, the T-BERT model achieves the highest F1 score overall (79.7) surpassing all the feature-based models as well as other neural network-based methods. This demonstrates that neural models can automatically unveil (non-linear) relationships between a user's generated textual content (i.e., language use) in the data and the prevalence of that user retweeting from reliable or unreliable news sources in the future. The simpler neural network model, Avg-EMB achieves a lower F1 score (75.5) compared to the other neural models, i.e., BiGRU-ATT, BERT, XLNet and ULMFiT. This happens because the latter have more complex architectures and can effectively capture the relations between inputs and labels while the former ignores word order. Furthermore, ULMFit, BERT and XLNet models have been pre-trained on large external corpora so they can leverage this extra information to generalize better. Finally, we do not notice considerable differences in performance between the truncated and hierarchical versions of the transformer-based models (BERT and XNLNet) suggesting that a small amount of user generated content is enough for accurately predicting the correct user class.
Best single performing feature-based model is SVM-ALL (75.9). Moreover, SVM with BOW, Topics and LIWC achieve lower performance (75.8, 71.2 and 69.6 respectively). Table 2 Macro precision -P, recall -R and F1-score -F1 (mean ± standard deviation over three runs) for predicting whether a Twitter user belongs to the reliable or unreliable class.

Error analysis
We performed an error analysis on the predictions of our best model, T-BERT. We notice that users in the unreliable class who are classified as reliable are those who repost from both reliable and unreliable sources. These users have an average of 40 future retweets from reliable news sources which is higher than the average number (31 retweets) in the entire dataset. Therefore, it is likely that such users use similar topics of discussion with reliable users. On the other hand, there is a of total 454 unreliable users who have no retweets from reliable sources in our dataset, interestingly, only four of them are classified wrongly. We also observe that it is harder for our model to classify correctly reliable users when they have only posted a small number of original tweets (e.g., 10-60).

Linguistic analysis
Finally, we perform a linguistic feature analysis to uncover the differences in language use between users in the two classes, i.e., reliable and unreliable. For that purpose, we apply univariate Pearson's correlation test to identify which text features (i.e., BOW, Topics and LIWC) are high correlated with each class following Schwartz et al. (2013). Tables 3, 4 and 5 display the top-10 n-grams, LIWC categories and Topics (represented by the most central words as in ) respectively. All Pearson correlations (r) presented in tables are statistically significant (p < 0.001). Table 3 shows the ten most correlated BOW features with each class. We observe that users reposting unreliable news sources in the future are more prevalent in tweeting about politics (note that we exclude user retweets in our study). For example, they use words related to the established political elite (e.g., liberal, government, media, MSM 6 ) and Middle East politics (e.g., islam, israel). This may be partially explained by studies which find that people who are more ideologically polarized might be more receptive to disinformation (Marwick, 2018) and engage more with politics on social media (Preoţiuc-Pietro et al., 2017). Users using language similar to the language used by unreliable and hyperpartisan sources can be explained by the fact that these users might already consume news from unreliable sources but they have not reposted any of them yet (Potthast et al., 2018;Pennycook, Cannon & Rand, 2018). Users belonging in the reliable news sources category use words related to self-disclosure and extraversion such as personal feelings and emotions (e.g., mood, wanna, gonna, i'll, excited). Moreover, words such as birthday, okay denote more frequent interaction with other users, perhaps friends. Table 4 shows the ten most correlated topics with each class. Topics related to politics such as political ideology (#138, #175), government (#179) and justice (#106) are correlated with users that will propagate unreliable sources, aligned with the n-grams analysis. We also observe a high correlation of such users with the topic related to impolite personal characterizations (#78). This corroborate results of a recent study that showed political incivility on Twitter is correlated to political polarization (Vargo & Hopp, 2017).

Topics
Users who will repost reliable sources discuss topics related to their day-to-day life such as education (#120), food (#65 and #166) and fashion (#1). Some topic words (e.g., sleep, exhausted and tired from #101) reveal that users emotional or physical states caused from work or study. In other words, these users tend to share more frequently information about their daily life, time and schedule (#101, #2 and #27). Table 5 shows the ten most correlated LIWC categories with each class. LIWC categories such as Power and Drives are more prevalent in users that will share unreliable sources. We also observe the difference in using casual language, e.g., Netspeak and Informal categories which are more often used by users that will share trusted sources.

CONCLUSIONS
We have presented a new study on the early detection of users reposting unreliable news sources. We have created a new dataset with users labeled into the two categories, i.e., reliable and unreliable. For this binary classification task, a Transformer-based pretrained model (i.e., BERT) achieves up to 79.7 macro F1. Finally, our linguistic feature analysis unveiled the main characteristics and differences between language features (i.e., BOW, Topics and LIWC) in the two groups of users. In the future, we plan to extend this work by performing a fine-grained classification into hoax, propaganda and clickbait (Glenski, Weninger & Volkova, 2018b); and explore whether language and social network information are complementary.

ADDITIONAL INFORMATION AND DECLARATIONS Funding
The authors received no funding for this work.


Network De-materialization in Open Flow Network

This review paper presents about the study of DeMaterialization of Network known as Network Virtualization in SDN(Software-Defined Networking). It interprets the different paths to network virtualization in SDN. Also defines some contrasting mechanism to allocate resources.

INTRODUCTION
De-Materialization is a process of implicit adaption of the real system. The real system design of modern days applies the concept of openflow where the system utilities are dissociated from physical realization. Example of virtualization is the partitioning of storage device space like a hard drive to create two separate hard drives. The implementation of this generalization of conceptual model allows accomplishing operational objectives separated from the elemental physical infrastructure. In this modernized world, the workload can be migrated amidst physical servers and suspended if needed. Network De-Materialization divides a network. Once, the successful division of network is done then the numerous users and participants can share a single physical network. Several approaches were made to enhance the concept of dividing network where in , utilization of resources were improved through sharing, segregation of traffic amongst different entities was easy, was able to access limitless computing resources for faster and broader business capabilities and to simplify network management. During the implementation of the technique for dividing the network, the system should make sure that the traffic is separated amongst the users and there should not be any compromise in the confidentiality of virtual partitions. It should also implement the already formulated SLA"s(Service Level Agreement) and should be adjustable and feasible to avoid pricey interaction and strategy amongst the parties involved, by reducing operational expenses. Computer virtualization is taken into consideration to understand virtual networking. Virtualization allows efficient utilization of computer hardware, it uses a software to create abstraction layer over computer hardware and allows the hardware elements of a computer (Ex: processor, memory etc.) to be sliced into multiple virtual computers called Virtual Machine (VM) [1]. Each VM has its own private Operating System (OS) and behaves like an Independent computer. Thus, different hardware can have their own set of instructions, which leads to more efficient system. Operating system can be implemented for networks, Data etc.
As the virtualization is improved by slicing and sharing the resources, once the network is virtualized the mechanism for the allocation of resources need to be provided. Each module of virtual network should get a fair share of the resources. The race of resources in this process leads to cumbersome traffic which is called Stress point [2].

II. OPEN FLOW
Open Flow makes arrangement for network traffic behaviour either to be implemented within switches or the software that runs on SDN controller. It is based on Ethernet switch which has a flow table and group table that performs packets lookups and forwarding [4]. Here, the production traffic and research traffic is segregated. Hence, the research persons can try their advanced protocols without arousing any effect to production traffic. Every open flow switch has a basic set of behaviour and has 3 main parts: 1.) Flow table entry has the account of the actions performed by Flow Table. 2.) Secure Channel which connects switch to a controller. 3.) Open flow Protocol which is the standard way for communication between controller and switch.
Open flow provides a platform where more research can take place in the field of networking. It does not require any specific software for the experiments that runs on different networks.

III. NETWORK HYPERVISOR
Network Hypervisor [6] is a program which provides the abstraction layer for network hardware. It is a wide software layer throughout the network that maps logical forwarding plane to underlying physical hardware.Network Hypervisor provides logical service model to the software and implements desired functionality on the hardware.
The main different components include: 1. Control Panel: It is a basic mechanism that mentions the clear network functionalities either through manual configuration or programmatic control [8]. 2. Network Hypervisor: It provides abstraction layer for network hardware 3. Physical forwarding plane: Set of elements on physical network forwarding plane. 4. Logical forwarding plane: Logical abstraction of network. It has look up tables and ports. Look up tables consists of forwarding tables usually built around a pipeline of TCAMS [9] with forwarding actions. Ports can be logical ports which is bound to physical ports. The packets should go throughthe following sequence.
1.) Directing incoming packet to the correct logical context, which is done by contributing some identifying tag such as MPLS header [10]. 2.) Logical forwarding decision must be done. 3.) Directing the logical forwarding decision back to physical plane. At the edge of a logical forwarding it reaches one or more egress port on the logical network. 4.) Physical Forwarding. It preserves the graph by joining every switch in the network. It contributes an API which constructs and maps logical forwarding elements to its physical network.

Flowvisor[4] is a technique for control side virtualization. It is achieved as a protocol proxy which interrupts information among open flow active switches and open flow
controllers. Flowvisor layer is existing amidst underlying physical hardware and software that controls it. It hosts multiple open flow controllers, one controller per slice to control that particular slice assigned to it. A slice is a flow through switches [3]. Flowvisor ensures that every controller controls only the switches. It divides the flow entries of separate guest controllers. A slice is a small module of the entire space of packet header. In open flow, each of these flow entries will be made based on a 10-field packet header which is 256 bits long. Thus, there will be 2 pow 256 points in a 256-dimensional space. We can define various regions as a subset of space using bit masks if we define a header using 256-k bits(k denotes bit mask) then it has a k dimensional region. A slice is a small module of the entire space of packet header. This set of regions can be considered as a slice`s flowspace. Flowvisor acts as a transparent proxy between guest controllers and switch thus ensuring transparency and isolation between slices by the process of inspecting, rewriting and policing open flow messages as they pass. Flowvisor can control multiple switches and can virtualize another virtual network.

V. ADVISOR
Advanced Flowvisors similar to flowvisor except that in ADVisor virtual topologies [4] for each virtual networks are not restricted by the underlying physical topology. It is existing between the physical network and the controllers. It can slice a virtual topology frequently and can directly reply to open flow network. Several components like virtual nodes are signified as a set of tuples and using these tuples Virtual topologies are identified [5].
Main parts of an ADVisor includes: 1.) Topology Monitor which identifies whether the switch generating the open flow protocol message is at the endpoint of a link or a part of physical link. 2.) Link Broker which controls the switches which are present as part of a virtual link. Packets sent by these switches are controlled.

VI. NETWORK VIRTUALIZATION
As discussed above network virtualization is a process that is accomplished by slicing the network through data path and control channel virtualization. However, we need to ensure an appropriate separation among distinct virtual network and fair resource allocation. In datapath virtualization, separation needs to be established at link level and flow tables. A link level separation can be achieved by partitioning and encapsulation. The partitioning of the link is done by splitting it into multiple partitions, assigning one partition per virtual network [7]. Virtual network ID is a unique ID given to each Virtual network as link local encapsulation ID to which Virtual Network traffic will be mapped by in encapsulation process. In flow table, separation is done by two methods Flow space and table partitioning. Virtual Network will be assigned by a particular flow space. Every virtual network can move in to only that table where the match for a specific flow space is assigned to virtual network. In table slicing, the flow table is divided into multiple tables logically or physically and each VN is assigned a group of distinct flow tables. To ensure fair resource allocation at link level, we can use the classic QoS tools like classification, metering, colouring, policing etc. We mustcertifyunbiased allocation of resources on the control channel i.e. the network that connects controller to the switches. If the control channels for dissimilar virtual networks are multiplexed over a particular cp connection it is difficult to discriminate control channels and to enforce QoS .If switches allow different controllers to connect and control the Virtual networks using different source or destination IP addresses or port numbers, the control traffic can easily distinguish between different connections and thus enforce QoS policies.
Control channel can be executed as out band or in band control channel.
1. Out band control channel:  Control signals and data traffic goes through distinct channels.  It is simple and easy to design.  More expensive as a result of extra network and ports on hardware. 2. In band control channel:  Data and control traffic goes through same channel.  Channel is weak to misconfigured [11] flow table entries for all packets. It will be more beneficial to enforce in band channel, if QOS support is given to control traffic entering and outgoing from switch.

VII. CONCLUSION
Virtualizing a network supports in reducing the number of physical devices needed, effortlessly segments the network, improves resource utilisation, helps on the research of new protocols.


extfieldsofstudy:
Engineering
Computer Science

text:
Digital Voicing of Silent Speech

In this paper, we consider the task of digitally voicing silent speech, where silently mouthed words are converted to audible speech based on electromyography (EMG) sensor measurements that capture muscle impulses. While prior work has focused on training speech synthesis models from EMG collected during vocalized speech, we are the first to train from EMG collected during silently articulated speech. We introduce a method of training on silent EMG by transferring audio targets from vocalized to silent signals. Our method greatly improves intelligibility of audio generated from silent EMG compared to a baseline that only trains with vocalized data, decreasing transcription word error rate from 64% to 4% in one data condition and 88% to 68% in another. To spur further development on this task, we share our new dataset of silent and vocalized facial EMG measurements.

Introduction
In this paper, we are interested in in enabling speech-like communication without requiring sound to be produced. By using muscular sensor measurements of speech articulator movement, we aim to capture silent speech -utterances that have been articulated without producing sound. In particular, we focus on the task which we call digital voicing, or generating synthetic speech to be transmitted or played back.
Digitally voicing silent speech has a wide array of potential applications. For example, it could be used to create a device analogous to a Bluetooth headset that allows people to carry on phone conversations without disrupting those around them. Such a device could also be useful in settings where the environment is too loud to capture audible speech or where maintaining silence is important. Alternatively, the technology could be used by some people who are no longer able to produce audible speech, such as individuals whose larynx has been removed due to trauma or disease (Meltzner et al., 2017). In addition to these direct uses of digital voicing for silent speech, it may also be useful as a component technology for creating silent speechto-text systems (Schultz and Wand, 2010), making silent speech accessible to our devices and digital assistants by leveraging existing high-quality audio-based speech-to-text systems.
To capture information about articulator movement, we make use of surface electromyography (EMG). Surface EMG uses electrodes placed on top of the skin to measure electrical potentials caused by nearby muscle activity. By placing electrodes around the face and neck, we are able to capture signals from muscles in the speech articulators. Figure 1 shows the EMG electrodes used to capture signals, and Figure 2 shows an example of EMG signals captured. We collect EMG measurements during both vocalized speech (normal speech production that has voicing, frication, and other speech arXiv:2010.02960v1 [eess.AS] 6 Oct 2020 A V -audio from vocalized speech E V -EMG from vocalized speech E S -EMG from silent speech Figure 2: The three components of our data that we will use in our model. The vocalized speech signals A V and E V are collected simultaneously and so are time-aligned, while the silent signal E S is a separate recording of the same utterance without vocalization. During training we use all three signals, and during testing we are given just E S , from which we must generate audio. Colors represent different electrodes in the EMG data. Note that the silent EMG signal E S is qualitatively different from its vocalized counterpart E V . Not pictured, but also included in our data are the utterance texts, in this case: "It is possible that the infusoria under the microscope do the same." (from H.G. Well's The War of the Worlds).
sounds) and silent speech (speech-like articulations which do not produce sound). We denote these EMG signals E V and E S , respectively. During the vocalized speech we can also record audio A V , but during silent speech there is no meaningful audio to record.
A substantial body of prior work has explored the use of facial EMG for silent speech-to-text interfaces (Jou et al., 2006;Schultz and Wand, 2010;Kapur et al., 2018;Meltzner et al., 2018). Several initial attempts have also been made to convert EMG signals to speech, similar to the task we approach in this paper (Toth et al., 2009;Janke and Diener, 2017;Diener et al., 2018). However, these works have focused on the artificial task of recovering audio from EMG that was recorded during vocalized speech, rather than the end-goal task of generating from silent speech. In terms of signals in Figure 2, prior work learned a model for producing audio A V from vocalized EMG E V and tested primarily on other vocalized EMG signals. While one might hope that a model trained in this way could directly transfer to silent EMG E S , Toth et al. (2009) show that such a transfer causes a substantial degradation in quality, which we confirm in Section 4. This direct transfer from vocalized models fails to account for differences between features of the two speaking modes, such as a lack of voicing in the vocal folds and other changes in articulation to suppress sound.
In this paper, we extend digital voicing to train on silent EMG E S rather than only vocalized EMG E V . Training with silent EMG is more challenging than with vocalized EMG, because when training on vocalized EMG data we have both EMG inputs and time-aligned speech targets, but for silent EMG any recorded audio will be silent. Our solution is to adopt a target-transfer approach, where audio output targets are transferred from vocalized recordings to silent recordings of the same utterances. We align the EMG features of the instance pairs with dynamic time warping (Rabiner and Juang, 1993), then make refinements to the alignments using canonical correlation analysis (Hotelling, 1936) and audio feature outputs from a partially trained model. The alignments can then be used to associate speech outputs with the silent EMG signals E S , and these speech outputs are used as targets for training a recurrent neural transduction model. We validate our method using both human and automatic metrics, and find that a model trained with our target transfer approach greatly outperforms a model trained on vocalized EMG alone. On a closed-vocabulary domain (date and time expressions §2.1), transcription word error rate (WER) from a human evaluation improves from 64% to just 4%. On a more challenging open vocabulary domain (reading from books §2.2) intelligibility measurements improve by 20% -from 88% to 68% with automatic transcription or 95% to 75% with human transcription.
We release our dataset of EMG signals collected during both silent and vocalized speech. The dataset contains nearly 20 hours of facial EMG signals from a single speaker. To our knowledge, the largest public EMG-speech dataset previously available contains just 2 hours of data (Wand et al., 2014), and many papers continue to use private datasets. We hope that this public release will encourage development on the task and allow for fair comparisons between methods.

Data Collection
We collect a dataset of EMG signals and timealigned audio from a single speaker during both silent and vocalized speech. Figure 2 shows an example from the data collected. The primary portion of the dataset consists of parallel silent / vocalized data, where the same utterances are recorded using both speaking modes. These examples can be viewed as tuples (E S , E V , A V ) of silent EMG, vocalized EMG, and vocalized audio, where E V and A V are time-aligned. Both speaking modes of an utterance were collected within a single session to ensure that electrode placement is consistent between them. For some utterances, we record only the vocalized speaking mode. We refer to these instances as non-parallel data, and represent them with the tuple (E V , A V ). Examples are segmented at the utterance level. The text that was read is included with each instance in the dataset, and is used as a reference when evaluating intelligibility in Section 4.
For comparison, we record data from two domains: a closed vocabulary and open vocabulary condition, which are described in Sections 2.1 and 2.2 below. Section 2.3 then provides additional details about the recording setup.

Closed Vocabulary Condition
Like other speech-related signals, the captured EMG signals from a particular phoneme may look different depending on its context. For this reason, our initial experiments will use a more focused vocabulary set before expanding to a large vocabulary in Section 2.2 below.
To create a closed-vocabulary data condition, we generate a set of date and time expressions for reading. These expressions come from a small set of templates such as "<weekday> <month> <year>" which are filled in with randomly selected values (over 50,000 unique utterances are possible from this scheme). Table 1

Open Vocabulary Condition
The majority of our data was collected with openvocabulary sentences from books. We use public domain books from Project Gutenberg. 1 Unlike the closed-vocabulary data which is collected in a single sitting, the open-vocabulary data is broken into multiple sessions where electrodes are reattached before each session and may have minor changes in position between different sessions. In addition to sessions with parallel silent and vocalized utterances, we also collect non-parallel sessions with only vocalized utterances. A summary of dataset features is shown in Table 2. We select a validation and test set randomly from the silent parallel EMG data, with 30 and 100 utterances respectively. Note that during testing, we use only the silent EMG recordings E S , so the vocalized recordings of the test utterances are unused.

Recording Details
EMG signals are recorded using an OpenBCI Cyton Biosensing Board and transmitted to a computer over WiFi. Eight channels are collected at a sample rate of 1000 Hz. The electrode locations are described in Table 3. Gold-plated electrodes are used with Ten20 conductive electrode paste. We use a monopolar electrode configuration, with a shared reference electrode behind one ear. An electrode connected to the Cyton board's bias pin is placed behind the other ear to actively cancel common-mode interference. A high pass Butterworth filter with cutoff 2 Hz is used to remove offset and drift in the collected signals, and AC   electrical noise is removed with notch filters at 60 Hz and its harmonics. Forward-backward filters are used to avoid phase delay.
Audio is recorded from a built-in laptop microphone at 16kHz. Background noise is reduced using a spectral gating algorithm, 2 and volume is normalized across sessions based on peak root-meansquare levels.

Method
Our method is built around a recurrent neural transduction model from EMG features to time-aligned speech features (Section 3.1). We will denote the featurized version of the signals used by the transduction model E S/V and A V for EMG and audio respectively. When training solely on vocal-2 https://pypi.org/project/noisereduce/ ized EMG data (E V to A V ), training this model is straightforward. However, our experiments show that training on vocalized EMG alone leads to poor performance when testing on silent EMG (Section 4) because of differences between the two speaking modes.
A core contribution of our work is a method of training the transducer model on silent EMG signals, which no longer have time-aligned audio to use as training targets. We briefly describe our method here, then refer to section Section 3.2 for more details. Using a set of utterances recorded in both silent and vocalized speaking modes, we find alignments between the two recordings and use them to associate speech features from the vocalized instance (A V ) with the silent EMG E S . The alignment is initially found using dynamic time warping between EMG signals and then is refined using canonical correlation analysis (CCA) and predicted audio from a partially trained model.
Finally, to generate audio from predicted speech features, we use a WaveNet decoder, as described in Section 3.3.

EMG to Speech Feature Transducer
When converting EMG input signals to audio outputs, our first step is to use a bidirectional LSTM to convert between featurized versions of the signals, E and A . Both feature representations operate at the same frequency, 100 Hz, so that each EMG input E [i] corresponds to a single time-aligned output A [i]. Our primary features for representing EMG signals are the time domain features from Jou et al. (2006), which are commonly used in the EMG-speech-to-text literature. After splitting the signal from each channel into low and highfrequency components (x low and x high ) using a triangular filter with cutoff 134 Hz, the signal is windowed with a frame length of 27 ms and shift of 10 ms. For each frame, five features are computed as follows: where ZCR is the zero-crossing rate. In addition to the time domain features, we also append magnitude values from a 16-point Short-time Fourier transform for each 27 ms frame, which gives us 9 additional features. The two representations result in a total of 112 features to represent the 8 EMG channels. Speech is represented with 26 Melfrequency cepstral coefficients (MFCCs) from 27 ms frames with 10 ms stride. All EMG and audio features are normalized to approximately zero mean and unit variance before processing. To help the model to deal with minor differences in electrode placement across sessions, we represent each session with a 32 dimensional session embedding and append the session embedding to the EMG features across all timesteps of an example before feeding into the LSTM. The LSTM model itself consists of 3 bidirectional LSTM layers with 1024 hidden units, followed by a linear projection to the speech feature dimension. Dropout 0.5 is used between all layers, as well as before the first LSTM and after the last LSTM. The model is trained with a mean squared error loss against time-aligned speech features using the Adam optimizer. The initial learning rate is set to .001, and is decayed by half after every 5 epochs with no improvement in validation loss. We evaluate a loss on the validation set at the end of every epoch, and select the parameters from the epoch with the best validation loss as the final model.

Audio Target Transfer
To train the EMG to speech feature transducer, we need speech features that are time-aligned with the EMG features to use as target outputs. However, when training with EMG from silent speech, simultaneously-collected audio recordings do not have any audible speech to use as targets. In this section, we describe how parallel utterances, as described in Section 2, can be used to transfer audio feature labels from a vocalized recording to a silent one. More concretely, given a tuple (E S , E V , A V ) of features from silent speech EMG, vocalized speech EMG, and vocalized speech audio, where E V and A V are collected simultaneously, we estimate a set of audio featuresÃ S that time-align with E S and represent the output that we would like our transduction network to predict. A diagram of the method can be found in Figure 3.
Our alignment will make use of dynamic time warping (DTW) (Rabiner and Juang, 1993), a dynamic programming algorithm for finding a minimum-cost monotonic alignment between two sequences s 1 and s 2 . DTW builds a table d[i, j] of the minimum cost of alignment between the first i items in s 1 and the first j items in s 2 . The recursive step used to fill this table is d . After the dynamic program, we can follow backpointers through the table to find a path of (i, j) pairs representing an alignment. Although the path is monotonic, a single position i may repeat several times with increasing values of j. We take the first pair from any such sequence to form a mapping a s 1 s 2 [i] → j from every position i in s 1 to a position j in s 2 .
For our audio target transfer, we perform DTW as described above with s 1 = E S and s 2 = E V . Initially, we use euclidean distance between the features of E S and E V for the alignment cost ), but will describe several refinements to this choice in Sections 3.2.1 and 3.2.2 below. DTW results in an alignment a SV [i] → j that tells us a position j in E V for every position i in E S . We can then create a warped audio feature sequenceÃ S that aligns with E S us- During training of the EMG to audio transduction model, we will useÃ S as our targets for the transduction outputsÂ S when calculating a loss.
This procedure of aligning signals to translate between them is reminiscent of some DTW-based methods for the related task of voice conversion (Kobayashi and Toda, 2018;Desai et al., 2009). The difference between these tasks is that our task operates on triples (E S , E V , A V ) and must account for the difference in modality between the input E S and output A V , while voice conversion operates in a single modality with examples of the form (A 1 , A 2 ).
In addition to training the transducer from E S toÃ S , we also find that training on the vocalized signals (E V to A V ) improves performance. The vocalized samples are labeled with different session embeddings to allow the model to specialize to each speaking mode. Each training batch contains samples from both modes mixed together. For the open vocabulary setting, the full set of examples to sample from has 3 sources: (E S ,Ã S ) created from parallel utterances, (E V , A V ) from the vocalized recording of the parallel utterances, and (E V , A V ) from the non-parallel vocalized recordings.

CCA
While directly aligning EMG features E S and E V can give us a rough alignment between the signals, doing so ignores the differences between the two signals that lead us to want to train on the silent signals in the first place (e.g. inactivation of the vocal folds and changes in manner of articulation to prevent frication). To better capture correspondences between the signals, we use canonical correlation analysis (CCA) (Hotelling, 1936) to find components of the two signals which are more highly correlated. Given a number of paired vectors (v 1 , v 2 ), CCA finds linear projections P 1 and P 2 that maximize correlation between corresponding dimensions of P 1 v 1 and P 2 v 2 .
To get the initial pairings required by CCA, we use alignments found by DTW with the raw EMG feature distance δ EMG . We aggregate aligned E S and E V features over the entire dataset and feed these to a CCA algorithm to get projections P S and P V . CCA allows us to choose the dimensionality of the space we are projecting to, and we use 15 dimensions for all experiments. Using the projections from CCA, we define a new cost for DTW Our use of CCA for DTW is similar to Zhou and Torre (2009), which combined the two methods for use in aligning human pose data, but we found their iterative approach did not improve performance compared to a single application of CCA in our setting.

Refinement with Predicted Audio
So far, our alignments between the silent and vocalized recordings have relied solely on distances between EMG features. In this section, we propose an additional alignment distance term that uses audio features. Although the silent recording has no useful audio signal, once we start to train a transducer model from E S to audio features, we can try to align the predicted audio featuresÂ S to vocalized audio features A V . Combining with an EMG-based distance, our new cost for DTW becomes where λ is a hyperparameter to control the relative weight of the two terms. We use λ = 10 for all experiments in this paper.
When training a transducer model using predicted-audio alignment, we perform the first four epochs using only EMG-based alignment costs δ CCA . Then, at the beginning of the fifth epoch, we use the partially-trained transducer model to compute alignments with cost δ full . From then on, we re-compute alignments every five epochs of training.

WaveNet Synthesis
To synthesize audio from speech features, we use a WaveNet decoder (van den Oord et al., 2016), which generates the audio sample by sample conditioned on MFCC speech features A . WaveNet is capable of generating fairly natural sounding speech, in contrast to the vocoder-based synthesizer used in previous EMG-to-speech papers, which caused significant degradation in naturalness (Janke and Diener, 2017). Our full synthesis model consists of a bidirectional LSTM of 512 dimensions, a linear projection down to 128 dimensions, and finally the WaveNet decoder which generates samples at 16 kHz. We use a WaveNet implementation from NVIDIA 3 which provides efficient GPU inference. WaveNet hyperparameters can be found in Appendix A. During training, the model is given gold speech features as input, which we found to work better than training from EMG-predicted features. Due to memory constraints we do not use any batching during training, but other optimization hyperparameters are the same as those from Section 3.1.

Experiments
In this section, we run experiments to measure intelligibility of audio generated by our model from silent EMG signals E S . Since prior work has trained only on vocalized EMG signals E V , we compare our method to a direct transfer baseline which trains a transducer model only on vocalized EMG E V before testing on the silent EMG E S . 4 The baseline transducer and wavenet models have identical architecture to those used by our method, but are not trained with silent EMG using our target transfer approach. Since one may hypothesize that most of the differences between silent and vocalized EMG will take place near the vocal folds, we also test a variant of this baseline where the electrode placed on the neck is ignored.
We first test on the closed vocabulary data described in Section 2.1, then on the open vocabulary data from Section 2.2. On the open vocabulary data, we also run ablations to evaluate different alignment refinements with CCA and predicted audio (see Sections 3.2.1 and 3.2.2).

Closed Vocabulary Condition
We begin by testing intelligibility on the closed vocabulary date and time data with a human transcription evaluation. The human evaluator is given a set of 20 audio output files from each model being tested (listed below) and is asked to write out in words what they heard. The files to transcribe are randomly shuffled, and the evaluator is not told that the outputs come from different systems. They are told that the examples will contain dates and times, but are not given any further information about what types of expressions may occur. The full text of the instructions provided to the evaluator can be found in Appendix B. We compare the transcriptions from the human evaluator to the original text prompts that were read during data collection to compute a transcription word error rate (WER): Lower WER values indicate better models. Using this evaluation, we compare three different models: a direct transfer baseline trained only on vocalized EMG signals, a variant of this baseline where the throat electrode is removed to reduce divergence between speaking modes, and our full model trained on silent EMG using target-transfer. All three models were trained on open vocabulary  data (Section 2.2) before being fine-tuned on the closed vocabulary training set. A single WaveNet model is used to synthesize audio for all three models and was also trained on the open vocabulary data before being fine-tuned in-domain.
The results of our evaluation are shown in Table 4. We first observe that removing the throat electrode substantially improves intelligibility for the direct transfer baseline. Although this modification removes potentially useful information, it also removes divergence between the silent and vocalized EMG signals. Its relative success further motivates the need for methods to account for the differences in the two modes, such as our targettransfer approach. However, even with the throatremoval modification, the direct transfer approach is still only partially intelligible.
A model trained with our full approach, including CCA and predicted-audio alignment, achieves a WER of 3.6%. This result represents a high level of intelligibility and a 94% relative error reduction from the strongest baseline.

Open Vocabulary Condition
Similar to our evaluation in Section 4.1, we use a transcription WER to evaluate intelligibility of model outputs in the open vocabulary condition. For the open vocabulary setting, we evaluate both with a human transcription and with transcriptions from an automatic speech recognizer.

Human Evaluation
Our human evaluation with open vocabulary outputs follows the same setup as the closed vocabulary evaluation. Transcripts are collected for 20 audio outputs from each system, with a random interleaving of outputs from the different systems. The annotator had no prior information on the content of the texts being evaluated. We compare two systems: direct transfer without the throat electrode (the stronger baseline) and our full model.  The results of this evaluation are a 95.1% WER for the direct transfer baseline and 74.8% WER for our system. While the intelligibility is much lower than in the closed vocabulary condition, our method still strongly out-performs the baseline with a 20% absolute improvement.

Automatic Evaluation
In addition to the human evaluation, we also perform an automatic evaluation by transcribing system outputs with a large-vocabulary automatic speech recognition (ASR) system. Using an automatic transcription allows for much faster and more reproducible comparisons between methods compared to a human evaluation. For our automatic speech recognizer, we use the open source implementation of DeepSpeech from Mozilla 5 (Hannun et al., 2014). Running the recognizer on the original vocalized audio recordings from the test set results in a WER of 9.5%, which represents a lower bound for this evaluation.
Our automatic evaluation results are shown in Table 5. While the absolute WER values for the ASR evaluation do not perfectly match those of the human transcriptions, both evaluations show a 20% improvement of our system over the best baseline. Given this correlation between evaluations and the many advantages of automated evaluation, we will use the automatic metric throughout the rest of this work and recommend its use for comparisons in future work.
We also run ablations of the two alignment refinement methods from Sections 3.2.1 and 3.2.2 and include results in Table 5. We see that both refinements have a positive effect on performance, though the impact of aligning with predicted audio is greater.

Additional Experiments
In the following subsections, we perform additional experiments on the open vocabulary data to explore the effect of data size and choice of electrode positions. These experiments are all evaluated using the automatic transcription method described in Section 4.2.

Data Size
In this section we explore the effect of dataset size on model performance. We train the EMG-tospeech transducer model on various-sized fractions of the dataset, from 10% to 100%, and plot the resulting WER. We select from the parallel (silent and vocalized) and non-parallel (vocalized only) portions proportionally here, but will re-visit the difference later. Although data size also affects WaveNet quality, we use a single WaveNet trained on the full dataset for all evaluations to focus on EMG-specific data needs. Figure 4 shows the resulting intelligibility measurements for each data size. As would be expected, the rate of improvement is larger when data sizes are small. However, there does not seem to be a plateau in performance, as improvements continue even when increasing data size beyond fifteen hours. These continued gains suggest that collecting additional data could provide more improvement in the future.
We also train a model without the non-parallel vocalized data (vocalized recordings with no associated silent recording; see Section 2). A model trained without this data has a WER of 71.6%, a loss of 3.6 absolute percentage points. This confirms that non-parallel vocalized data can be useful for silent speech even though it contains only data from the vocalized speaking mode. However, if we compare this accuracy to a model where the same amount of data was removed proportionally from the two data types (parallel and non-parallel), we see that removing a mixture of both types leads to a much larger performance decrease to 76% WER. This indicates that the non-parallel data is less important to the performance of our model, and suggests that future data collection efforts should focus on collecting parallel utterances of silent and vocalized speech rather than non-parallel utterances of vocalized speech.

Removing Electrodes
In this section, we experiment with models that operate on a reduced set of electrodes to assess the impact on performance and gain information about which electrodes are most important. We perform a random search to try to find a subset of four electrodes that works well. More specifically, we sample 10 random combinations of four electrodes to remove (out of 70 possible combinations) and train a model with each. We then use validation loss to select the best models.
The three best-performing models removed the following sets of electrodes (using electrode numbering from Table 3): 1) {4, 5, 7, 8} 2) {3, 5, 7, 8} and 3) {2, 5, 7, 8}. We note that electrodes 5, 7, and 8 (which correspond with electrodes on the midjaw, upper cheek, and back cheek) appear in all of these, indicating that they may be contributing less to the performance of the model. However, the best model we tested with four electrodes did have substantially worse intelligibility compared to an eight-electrode model, with 76.8% WER compared to 68.0%. A model that removed only electrodes 5, 7, and 8 also performed substantially worse, with a WER of 75.3%.

Conclusion
Our results show that digital voicing of silent speech, while still challenging in open domain settings, shows promise as an achievable technology. We show that it is important to account for differences in EMG signals between silent and vocalized speaking modes and demonstrate an effective method of doing so. On silent EMG recordings from closed vocabulary data our speech outputs achieve high intelligibility, with a 3.6% transcription word error rate and relative error reduction of 95% from our baseline. We also significantly improve intelligibility in an open vocabulary condition, with a relative error reduction over 20%. We hope that our public release of data will encourage others to further improve models for this task. 6


extfieldsofstudy:
Computer Science
Engineering

text:
Deep variational network for rapid 4D flow MRI reconstruction

Phase-contrast magnetic resonance imaging (MRI) provides time-resolved quantification of blood flow dynamics that can aid clinical diagnosis. Long in vivo scan times due to repeated three-dimensional (3D) volume sampling over cardiac phases and breathing cycles necessitate accelerated imaging techniques that leverage data correlations. Standard compressed sensing reconstruction methods require tuning of hyperparameters and are computationally expensive, which diminishes the potential reduction of examination times. We propose an efficient model-based deep neural reconstruction network and evaluate its performance on clinical aortic flow data. The network is shown to reconstruct undersampled 4D flow MRI data in under a minute on standard consumer hardware. Remarkably, the relatively low amounts of tunable parameters allowed the network to be trained on images from 11 reference scans while generalizing well to retrospective and prospective undersampled data for various acceleration factors and anatomies.

Introduction
4D flow MRI provides spatio-temporally resolved quantification of blood flow and offers great potential for the assessment of cardiovascular disease, e.g. aortic valve stenosis, atherosclerosis, or vessel wall remodelling 1 . However, clinical adaptation of the method has been hampered by long exam times.
Many efforts have been dedicated to accelerate flow acquisition by exploiting redundancies in the data. Partial Fourier imaging 2 has been used for moderate acceleration 3 , but the underlying assumption of a slowly varying phase has been shown to be incorrect for 4D flow MRI 4 . Parallel imaging (PI) 5 , which exploits the spatially varying sensitivity of receiver elements in the coil array has become a standard for accelerated imaging, but undersampling rates are limited by noise amplification 6 . The advent of compressed sensing (CS) 7 has enabled acceleration of 4D flow MRI by acquiring only a subset of k-space data and exploiting prior information about data regularities during reconstruction 8,9,10,11,12,13 , with typical acceleration factors ranging from 5 10 up to 27 13 . In particular, the locally low rank (LLR) regularized reconstruction 14 has been a successful technique, which iteratively balances the data fidelity cost and the singular norm of a patch matrix stacked over cardiac phases (see Methods for details). However, iterative reconstruction methods as used in CS increase reconstruction times considerably, implying that evaluation of 4D flow MRI data will typically happen when the subject has already been moved out of the scanner.
In recent years, deep neural networks have gained increasing popularity in MR image reconstruction. In the training stage, the neural network learns abstract features from a set of scans. After training, newly acquired data are reconstructed with very little computational effort by inference with the learned weights. This reduction in reconstruction times can facilitate the use of accelerated imaging methods in clinical practice. Moreover, reconstruction results can be superior to traditional CS methods 15,16 . Some approaches discard concepts of iterative image reconstruction altogether, e.g. by learning end-to-end mappings from k-space to image space 17 . As a downside, such networks usually require abundant amounts of high quality training data which is not available for high-dimensional flow MRI. Model-based neural reconstruction networks can also be designed to replicate the behaviour of an iterative reconstruction by interlacing nonlinear convolutional filters with an operation that enforces closeness of the current image estimate to the acquired data 15,18,16,19 , similar to the data fidelity step in an iterative shrinkage-thresholding algorithm 20 . A recent study 21 showed that neural network architectures which incorporate such an operation generalize better to different undersampling rates. In contrast, generic architectures which are solely based on convolutional layers can even lead to deteriorated image quality when the undersampling factor is increased, although one would expect the reconstruction result to improve when more information is available. The adversarial approach for training MR reconstruction networks 22,23 is usually aimed at improving perceptual reconstruction quality, such as image sharpness. Typically, this is achieved at the expense of reconstruction nRMSE (normalized root-mean-square error) 24 , which is critical for flow quantification.
In this work, an approach based on the idea of deep variational neural networks 15 is implemented for rapid 4D flow reconstruction, wich is referred to as FlowVN hereafter.. For comparsion, the 3D variational network (VN) architecture as presented in 15 was adapted for 4D flow data by using 3D filter banks operating on subsets of xyzt dimensional data, yielding the model that we refer to as HamVN. The FlowVN twork architecture replicates 10 steps of an iterative image reconstruction, while allowing for learnable spatio-temporal filter kernels, activation functions, and regularization weights in each iteration. It is demonstrated that based on training performed with retrospectively undersampled data of healthy subjects, FlowVN can accurately reconstruct pathological flow in a stenotic aorta in 21 seconds. Moreover, an imaging study with healthy subjects demonstrates good agreement of reconstructions from prospective undersampling with reference measurement.   Figure 1: Breathing resolved 4D flow data acquisition. a, Data are sampled using a Cartesian pseudo-radial tiny Golden angle sampling pattern. b, Time-resolved data are binned to end expiration using a combination of principal component analysis, low-pass filtering, and coilclustering. c, Acquired data are sorted according to heart phase and velocity encoding. d, Datasets used during training and evaluation. To conduct prospective evaluation, 7 healthy volunteers underwent accelerated CS and reference PI acquisitions during the same scan session.

4D flow MRI reconstruction with FlowVN
The FlowVN architecture improves HamVN in the following ways: (i) linear activations are used instead of radial basis functions (RBF), (ii) the network is conditioned on the sampling rate, (iii) exponential weighting of intermediate layers is used as regularization, (iv) real and imaginary parts of the signal are filtered by shared weights, (v) momentum is considered during gradient descent (GD) unrolling, and, (vi) the data term allows tunable activation functions. The network is trained for a wide range of acceleration factors by allowing acceleration dependent weighting of data consistency and filtering steps. As illustrated in Fig. 1a, for each velocity encoding direction, the k-space data is acquired using a Cartesian Golden angle sampling strategy, yielding variable density undersampling patterns in k-space. The signal of a total of 28 physical coils is compressed into 5 virtual coils via clustering 25 . The samples are then sorted into respiratory bins and data in the end-expiratory bin is used for reconstruction.
A deep variational network can be seen as a differentiable sequence of an unrolled numerical optimization scheme. To enable learning, such sequence is then relaxed by allowing tunable filter weights and activation functions. As described in Methods, we unroll K=10 steps of a gradient descent with momentum governed by a scalar α (k) : At each k-th layer P (k) the current complex-valued spatio-temporal image estimate is represented, while S (k) maintains a running average of update steps. The update step G (k) consists of the data consistency and regularization terms (see Methods and Supplementary Algorithm 1 for details), that are weighted according to the sampling rate M = 1/R (R is the acceleration factor) via tunable activation functions ϕ ur (M ), respectively. The data consistency term modulates the k-space data residual via an activation function and maps them back to the image space via a conjugate imaging operator. The regularization term at each layer contains 3D filters grouped into 4 banks, where each bank performs convolutions in 3 dedicated dimensions, namely xyz, xyt, xzt and yzt, therefore avoiding costly 4D convolutions. To avoid overfitting, we assume shared filters and activation functions that operate on real and imaginary components of the image. Note that both data and regularization terms do not assume correlations between real and imaginary parts of the signal, as highlighted in Fig. 2b.
The image estimate P (K) (B, Θ) of the final layer can be then seen as a function of the kspace samples B and network parameters Θ. To tune the network parameters Θ we minimize the layer-wise exponentially weighted 1 image reconstruction loss: over the retrospectively undersampled training dataset T , where P * is the ground truth image. Layer weighting is controlled by parameter τ ≥0: when τ ≈ 0, the reconstruction error is penalized equally across layers, therefore gradients of network parameters have lower variance during stochastic optimization, yielding faster convergence. On the contrary, when τ → +∞, only reconstruction at the final layer P (K) is minimized, which improves fitting accuracy on the training data. It is worth mentioning that τ controls the trade-off between training reconstruction residual and network regularity. Similarly to Landweber iterations 26,15 and deep supervision 27 , such implicit regularization penalizes irregular representations at intermediate layers and favors networks that can provide fast reconstruction. We propose to initialize τ with zero and then gradually increase it according to the training schedule (see Methods).  Figure 2: FlowVN architecture and training. a, Structure of FlowVN and its training strategy which uses a reduced field-of-view data. b, Single unrolled iteration block consisting of convolutional filtering, data fidelity, undersampling modulation and gradient descent momentum terms. Tunable parameters are highlighted in red. c, FlowVN at inference time yields 4D image reconstruction. d, Target training image error and velocity magnitude error in the aorta evaluated using training data. e, Data and gradient term weighting functions shown for each of 10 layers. f, Exemplary slices of 3D xyz filters and their corresponding activation functions at layer 5.
To demonstrate the validity of our approach, we note that the extracted velocity magnitude error in the aorta decreases simultaneously with the target reconstruction error during training, as shown in Fig. 2d, therefore indicating that the target 1 image error is a valid training surrogate. It can be seen from Fig. 2e that the regularization term is suppressed for lower acceleration factors R (higher sampling rate M). A subset of learned FlowVN parameters Θ is shown in Fig. 2f illustrating that learned convolutions perform direction-dependent filtering.

Retrospective and Prospective Evaluation
Reconstructed image magnitudes (for a single velocity encoding component), estimated velocity magnitudes and their errors of a healthy volunteer data for acceleration factor R=14 are illustrated in Fig. 3 for retrospectively undersampled data. Compared to CS-LLR and HamVN, the proposed FlowVN provides better reconstruction accuracy in terms of image magnitude and velocities. Scatter plot and correlation analysis further suggest that the velocity magnitude image estimated via FlowVN is in better agreement with ground truth. As shown in Supplementary Table 2 these observations extend to other acceleration factors R (6-22) as tested on 7 healthy volunteers. Fig. 4 indicates that FlowVN can accurately reconstruct the jet at the inlet section of the aorta for a patient with a pathological aortic valve.
The prospective undersampling acquisition results are reported in Fig. 5a Table 1 shows that the proposed FlowVN is 30 times faster than CS-LLR reconstruction.

Discussion
Practical learning-based image reconstruction can be traced back to dictionary learning methods 28,29 , where prior information is learned from image patches and then used as a sparsity-inducing regularizer for iterative reconstruction. Such an approach yields orders of magnitude longer reconstruction times, compared to modern deep learning approaches. A straightforward application of deep artificial neural networks has been suggested for learning reconstruction as a regression from the k-space 17 or zero-filled reconstructions 30 directly into the image space. Although tempting, such an approach might be unjustified, because the k-space and zero-filling artefacts have global dependence on image intensities. The advent of effective automatic differentiation systems 31,32 revitalized the idea of unrolling 33 and relaxing numerical schemes that can solve the original reconstruction problem. Following this approach, a number of deep neural network architectures were proposed 18,34,15,35 , that disentangle image acquisition and image prior models. Unrolling gradient descent reconstruction with tunable filters and activation functions yields the HamVN architecture proposed by Hammernik et al. 15 . One advantage of VNs is that, compared to other deep architectures, they employ relatively limited number of free parameters to tune, therefore they are less susceptible to overfitting.
In this work we have further developed the VN architecture 15,36,37 to accommodate high performance undersampled 4D flow reconstruction with limited training. Namely, we avoid exponential model complexity growth by avoiding 4D convolutions and by using separable 3D convolutions  that are shared for real and imaginary parts of the image. Furthermore, in contrast to the original HamVN 15 , we train our FlowVN for a wide range of undersampling factors by allowing the regularization term to depend on them. As illustrated in Fig. 2e, regularization scaled by ϕ ur decreases as more samples are available, while the data term ϕ ud stays constant for most of the layers. Such conditioning allows network training on a larger variety of artefacts and as it is necessary in practice, since for a given fixed acquisition time, the precise value of the undersampling factor is not known a priori and depends on breathing and cardiac motion patterns. We hypothesize that the wide range of acceleration factors which were used simultaneously to train the FlowVN provided a diverse collection of aliasing artefacts and enabled robust learning on a remarkably limited training set of 11 subjects. The exponential weighting of the layer-wise reconstruction loss (3) further regularized FlowVN parameters by penalizing the nonlinear behaviour presented in HamVN reconstructions. Supplemental Fig. 6 and Table 3 provide quantification of reconstruction accuracy effects attributed to the modifications proposed with FlowVN. In particular, modifications to the network architecture result in a model that can better adapt to data and yield higher accuracy for retrospectively undersampled experiments, while the proposed exponential weighting of the training loss improves accuracy of the prospective evaluation, which indicates better generalization ability. It is worth noting, that FlowVN has only 1% more tunable parameters compared to HamVN (c.f. Table 1), while improving reconstruction nRMSE by 23% (averaged over acceleration factors as given in Supplemental Fig. 6). We note that 4D flow MRI greatly benefits from using coil information during reconstruction (see Supplementary Table 3). Accordingly, comparison with single-coil reconstruction networks 18 has limited benefit.  The proposed FlowVN is a learning-based approach for reconstructing undersampled 4D flow MRI data in under a minute. For fixed reconstruction accuracy, FlowVN enables higher acceleration factors (12% improvement compared to CS-LLR image nRMSE at R=16) and does not introduce significant bias of peak flow estimates. The proposed reconstruction is 30 times faster than stateof-the art CS-LLR and 4.2 times faster than HamVN, due to using linear activation functions rather than RBFs, which requires computation of pairwise distances between control knots and image intensities. It is worth noting that FlowVN demonstrates high generalization ability, being able to preserve patient pathologies that were not present in the training data.

Compressed Sensing 4D Flow Reconstruction
PC MRI encodes flow velocity v(r, t) ∈ R 3 at spatial location r during cardiac phase t (1 ≤ t ≤ N t ) according to the following equation: where v enc is the velocity corresponding to a phase of ±π, y i , i = 0, . . . , 3 are the encoded velocity vector components. The four-point velocity encoding matrix is given as Therefore, flow velocity v can be calculated from the phase difference of reconstructed PC images ρ i . Let ρ it ∈ C Nv be a discretized image on a N x ×N y ×N z =N v grid corresponding to a cardiac phase t and velocity encoding i. Assuming Cartesian sampling on a regular N 1 ×N 2 ×N 3 =N s grid, the Fourier transform F ∈ C Ns×Nv , and N c coil sensitivity maps W k def = diag(c k ) ∈ C Nv×Nv define the spatial encoding operator E ∈ C NsNc×Nv : Considering a single velocity-encoded image sequence, let P ∈ C Nv×Nt and B ∈ C NsNc×Nt be stacked column-vectors of signals ρ and zero-filled k-space samples respectively, while M ∈ {0, 1} NsNc×Nt defines the undersampling mask. Iterative image reconstruction methods seek for a maximum a posteriori (MAP) solution defined by the following optimization problem: where the regularization term R enforces prior assumptions about image regularities. Herein we consider the local low-rank (LLR) regularization 14 to leverage image correlations among cardiac phases: where T i ∈ {0, 1} p 3 ×Nv is the corresponding p×p×p patch extraction operator, yielding N ptch patches, and · * is the nuclear norm. For LLR regularization the optimization problem (7) is convex and can be efficiently solved using operator splitting techniques such as the fast iterative shrinkage-thresholding algorithm (FISTA) 20 .

FlowVN Training
We employ a K=10 layer VN and perform 5·10 4 iterations of the ADAM algorithm (learning rate 10 −3 , β 1 =0.85, β 2 =0.98, batch size of 3) for training, during which we continually adjust τ = i opt ·10 −3 with i opt being the iteration number. On every layer, each 3D filter bank contains N f =8 filters of size n c =5 voxels. Activation functions ϕ{.} are parametrized by N knts =91 control knots with spacing ω=0.17: with gradients provided by the following formulas: The acquired zero-filled k-space B with undersampling mask M was normalized by M 1 B F . To enable backpropagation to be carried out with limited GPU memory, we employ spatiotemporal equivariance of the convolution and exploit the fact that k-space is fully sampled in the readout dimension k x for Cartesian acquisitions. Therefore, to draw a training sample, we perform random cropping of width w x and w t in dimensions x and t respectively and simulate Fourier encoding in k y k z dimensions as illustrated in Fig. 2(a). The network was implemented using the Tensorflow framework 32 . Fully-sampled and partial Fourier acquisition data from 11 healthy volunteers was used during training.

In Vivo Data Acquisition
As illustrated in Fig. 1, we used 11 subject for network training, and 7 healthy subjects and 1 patient for evaluation. All in-vivo work was performed upon written informed consent of the subjects and according to local ethics regulations.
Training datasets comprised 4D flow data measured in the aorta of 11 healthy subjects, 9 of them fully sampled and 2 acquired with partial Fourier 38 (factor 0.75×0.75).
For evaluation, data in the ascending aorta of 7 healthy subjects were acquired on a 3T Philips Ingenia system (Philips Healthcare, Best, the Netherlands) using a Cartesian 4-point referenced phase-contrast gradient-echo sequence with an encoding velocity v enc = 150 cm/s, a spatial resolution of 2.5×2.5×2.5 mm 3 , T E = 3.3 ms, T R = 4.9 ms, 25 cardiac phases and flip angle = 8 • . Exams for each of the 7 healthy subjects comprised a standard navigator-gated 2-fold accelerated parallel imaging 5 exam for reference, and a CS acquisition with an acceleration factor of R=12.4 − 13.8, using Cartesian pseudo-radial golden angle sampling pattern 39 and data driven-respiratory motion detection, as in 11 . Only data in expiration were kept for reconstruction as shown in Fig. 1.
To evaluate reconstruction accuracy on pathological anatomy, 4D flow data was acquired in a single patient with dilation of the ascending aorta, combined aortic stenosis and regurgitation due to a bicuspid aortic valve on a 3T Philips Ingenia system (Philips Healthcare, Best, the Netherlands) using a navigator-gated 2-fold accelerated parallel imaging 5 scan.
A receiver coil with 28 channels was used for acquisition which were reduced to 5 channels using coil compression 40 . Coil sensitivity maps were estimated with ESPIRiT 41 . Concomitant field correction was applied to the signal phase according to Bernstein et al. 42 and eddy currents were corrected for with a third-order polynomial model fitted to stationary tissue 43,44 .

Evaluation
We compared the proposed FlowVN to the state-of-the-art compressed sensing LLR-regularized (8) reconstruction 14 and the variational network by Hammernik et al. 15 which we refer to as HamVN. The LLR implementation from the Berkeley advanced reconstruction toolbox (BART) 45 was used with patch size p=8 and a maximum number of optimization iterations of 80. The optimal value of regularization parameter λ LLR =2.06 was chosen via grid search to minimize the reconstructed flow field residual v − v * 2 averaged over the manually segmented aorta on the retrospectively 12× undersampled acquisition. Since the original VN 15 was proposed for magnitude reconstruction of 2D and 3D data, we introduced the following modifications for the presented 4D flow evaluation: (i) 3D filters were grouped into 4 banks as in FlowVN (see Supplemental Algorithm 1), (ii) 1 -norm of reconstruction was optimized (i.e. equation (3) with τ →+∞ ). We refer to this architecture as "HamVN", the number of network layers, filters and control knots were the same as in FlowVN.

Retrospective Study
For simulated retrospective undersampling experiments, we used 2× PI data and simulate a pseudoradial Golden angle sampling pattern 39 with acceleration factors of 6 to 22.
For each undersampling factor we evaluated the nRMSE of the image magnitude, the relative error (RelErr) of velocity magnitudes inside the aorta and the angular error (AngErr) of the estimated velocity vectors: nRMSE(a, a * ) = N i (a i − a i ) 2 N max j (a j ) 2 , RelErr(a, a * ) = a − a * 2 a * 2 , Additionally, we report the structural similarity index (SSIM) 46 with σ SSIM =1.5 on the reconstructed magnitude images.

Prospective Study
Using manual aorta segmentations we compute flow over cross sections of the aorta by integrating velocity components projected onto the cross section normal. The peak flow is then defined as the maximal flow over cardiac phases for a given cross section. Moreover, we calculate peak throughplane velocity defined as maximum velocity projection across cross sections of the aorta over cardiac phases.
To quantify agreement with the reference 2× PI reconstruction, we performed Bland-Altman analysis 47 of peak flow and peak through-plane velocities.

Data and code availability
The code for the network training and inference used in this study and network weights are avaliable on CodeOcean together with volunteer data: https://codeocean.com/capsule/0115983/tree 48 Figure 6: Retrospective ablation study. a, Comparison of VN architectures and modifications proposed in this paper, using retrospectively undersampled data from 7 healthy volunteers for different acceleration factors R. HamVN-3D performs 3D filtering of yzt dimensional data (assuming that data is fully sampled in the x dimension). Modifications are accumulated as presented in the legend and, eventually, yield the FlowVN configuration. Undersampling-dependent modulation improves reconstruction quality only for higher acceleration factors. Note that the layer-wise exponential weighting of the reconstruction loss does not increase model complexity and, therefore, does not improve reconstruction accuracy for retrospective acceleration simulations. b, Velocity magnitude error maps for R=16 shown at systolic peak flow.