Assessment of Readiness for Industry 4.0 Implementation in Ceramic Industry

This paper focuses on the implementation of principles of Industry 4.0 concept to ceramic industry. The topic of this paper is to address the problematics of the implementation of processes and elements of digitization within the Industry 4.0 concept into ceramic industry. Firstly, thorough literature and bestpractice research will be discussed. Based on the state of current knowledge, concept of the Industry 4.0 Readiness Model for Refractory will be presented. The model’s main focus is on the readiness of current business structures, processes and technical and economical situation. It will provide necessary analysis and insight into potential company’s processes and background. Based on this analysis, it will be possible to define the main obstacles for future digitalization and automation within Industry 4.0 concept in Refractory industry. On the foundation of data obtained by Industry 4.0 Readiness Model for Refractory, it will be possible to implement the Industry 4.0 solutions with highest added value to specific company, based on its current state. The main purpose of this paper is to summarize and discuss key parameters and framework for Industry 4.0 Readiness Model for Refractory and its connection to future implementation of Industry 4.0 features within ceramic industry.

Introduction
Nowadays elements of production control and manufacturing regulation in the Czech Republic are widespread mainly in traditional segments of industry such as automotive and aviation. Already at the time of Industry 4.0 concept establishment, the technologies of monitoring and regulation of production in automotive and aviation were at a high level. In terms of readiness to implementation of the new elements of Industry 4.0, companies in the automotive and aviation are therefore much better prepared.
The ceramics industry differs from the automotive industry, in particular in the nature of production and the size of the supply chain. Although the ceramic industry is a traditional industry in the Czech Republic, the ideas of the Industry 4.0 concept are not yet well established inside this industry. Given the wide range of products in this industry, this is a complex issue. A more appropriate approach is to address this issue in one segment of the ceramic industry.
The aim of this article is to review current Industry 4.0 models and knowledge of the manufacturing system in the refractory industry, which is considered part of the ceramic industry. With this knowledge, it will be possible to design an Industry 4.0 readiness evaluation model directly for the refractory industry. By focusing on a specific manufacturing process, the model should achieve greater objectivity because of the precise questions concerning the refractory industry. Based on the evaluation model, Industry 4.0 implementation model will be proposed.

Manufacturing system and definition
This chapter describes the general manufacturing system for refractory products, which is the basis for designing the methodology for the implementation of Industry 4.0 elements. Within the refractory industry, silica and fireclay manufacturing plants have similar production processes. The biggest difference between the production of silica and fireclay is, that silica, unlike fireclay, need to rest before drying. A simplified manufacturing diagram is shown in Fig. 1. The following paragraphs generally describe the manufacturing process. The process starts with the preparation of the mixture, during which it is necessary to achieve the required structure of selected input raw materials. Unlike conventional ceramics, the resulting chemical composition of the mixture is absolutely essential. It is necessary to weight very accurately. Also, the quality of raw materials is significantly higher than conventional ceramics. For this reason and the reason of lower costs, in most cases the preparation of the mixture is managed by the company itself (grinding and mixture preparation). The first important component of the mixture is an abrasive, which should have a exact shape, size and sharpness. To ensure the stability of these properties, the input mixture is firstly crushed into regular cubic grain using crushers or mills and then sorted by grain size. In association with the preparation of the binder (clays), in most cases it is necessary to dry the binder appropriately before processing, and then to further process it to reach desired structure. Preparation of the mixture is completed by mixing all components in the correct ratio and order using a suitable mixer machine. [1] · Shaping The technology of the forming process is divided according to the type of mixture used (plastics, moldables). While moldables in form of granulates is due to its properties processed only on presses, the mixture of plastics can be pressed at lower pressures, or even manually shaped. The pressed mixture is in both cases full of very hard abrasive materials. That cause rapid wear of metal moulds, because of strong abrasive effects. [2] · Drying Moldables and plastics workpieces are still too wet to be fired after the forming operation. The higher is the water content in the mass, the slower must be the drying process. Drying takes place in tunnel or chamber furnaces, in the case of moldables, often directly on the kiln cars intended for burning furnaces. [2] · Firing The last technological operation of the fireclay production process is firing of the dried mixture. Firing takes place in tunnel or chamber furnaces in the temperature range from 1000 °C to 1500 °C. Transport of products in the furnace is ensured by special kiln cars, which are adapted for easy and quick repairs. During burning, the furnaces must always be occupied with loaded kiln cars in order to avoid significant temperature changes that could cause damage to the furnaces themselves or damage the goods in the kiln. [1] [2] It is only a simplified general manufacturing process in the refractory industry. Different materials and products have different manufacturing process tech-nology. The production of refractory products is relatively demanding compared to the production of classical ceramics. It is necessary to observe all parameters prescribed by the production process (temperature, humidity, drying time). As well as in the production of classical ceramics, refractory production often lacks automation, modern monitoring systems and even sufficient IT structure. Therefore, it is hard to aim for Smart Factory concept without extensive preparations of the production processes and infrastructure. [3] [4] Current readiness models in Industry 4.0 There are already several approaches and methodologies to assess the state of digitization, automation and use of Industry 4.0 elements. It is mostly only an assessment of the current situation and does not propose a solution to the situation or future implementation. Assessment models of company readiness to implement Industry 4.0 elements are created mainly by large companies for their own use and commercial activity. For these reasons they are very general. Most models focus more on the energy, automotive and aviation industries. Some companies and their assessment methodologies deal with this issue solely on the basics of advice and consultation, others provide more complex solution with the implementation of Industry 4.0 elements directly to their customers. Fig. 2 shows some models which are used for maturity assess in terms of Industry 4.0 concept.
· PwC, 2017 It is an online self-assessment model for Industry 4.0, developed by PwC, a worldwide consulting company. The model evaluates four phases: digitization, vertical integration, horizontal collaboration and final digitization. It evaluates itself online, identifies the actions needed and compares it with other companies in the industry. [5] · Rockwell Automation, 2014 Rockwell Automation specialists have designed an evaluation model that assesses the current state and offers a way to implement network operating and information technologies. The model consists of five phases from initial evaluation to final cooperation. [6] · Isaka, Nagayoshi, Yoshikawa, Yamada, & Kakeno, 2016 An advanced model for production systems developed by Hitachi. This model classifies production systems in terms of six maturity levels, which gradually lead to increased management efficiency. First-level devices use up-to-date data for visualization. The second level (Connection) achieves traceability. The third level device is used for automatic analysis. The fourth level (measurement) ensures identification and resolution of production barriers. The fifth level means predicting and optimizing production plans and the final sixth level is a symbiosis in which resources and production plans are optimized with suppliers. [7]  This Industry 4.0 integration maturity model has four phases. Vertical integration, horizontal integration, digital product development and cross-cutting technology criteria. The final evaluation divides the companies into five levels. The first stage is the basic level of digitization and it is gradually possible to reach the fifth stage, which is fully optimized digitization. [9] · Schumacher, Erol, & Sihn, 2016 The model includes social, technical, and organizational dimensions to assess production readiness for Industry 4.0. The authors of the model have identified a total of nine dimensions, each of which will receive a weighted average value based on the evaluation of various items. It is a complex model with graphs for data visualization and priority identification. This is followed by grading (one to five) according to readiness. [10] · Ganzarain & Errasti, 2016 The model focuses on the process of changing diversification strategies. It is necessary to define a vision, implement a process and implement Industry 4.0 projects and ensure training and risk management. There are five phases from the first, where there are no elements of Industry 4.0 to the highest fifth stage of detailed business plan transformation. [11] However, none of these models is concerned only with the refractory industry and the individual models are too general or aimed at other industries. This means that a model that deals with the automotive industry in detail will not be very suitable for assessing the readiness of a refractory company. There are many reasons for this, such as the significant difference in the product portfolio, the material of the products, the method of production, the quantity of products produced, the size of the component base, the material storage, but also the current state of both industries. For these and other reasons, these models cannot be used in the manufacturing industry.

Activities within Industry 4.0 in Ceramics industry
Within the ceramics industry, several pilot models have already been created to help manufacturing companies implement other elements of Industry 4.0. Most of the models were too general or solved only local implementation of individual elements. For this reason, it is not appropriate to use any of the created models.
· Mobile Manufacturing Execution system (MMES) The main goal of this project was to create a cloudbased MES (Manufacturing Execution System) for mobile devices. By using the cloud, data is available in real time not only to the business itself, but also to suppliers and customers. Mobile technologies do not indexed on: http://www.scopus.com mean high purchase costs and the system is easy to adapt. [4] · Cloud Laboratory Information Management System (CLIMS) Similar to the previous model. The cloud storage is also used here. The Cloud storage is here to keep information in electronic form and to easily find information from testing laboratories to check compliance with required standards or customer requirements. [4] · The Mould Digitalization The moulds should be kept for reuse. As a result, the company is losing a significant part of its warehouse capacity. Nowadays there is an effort to have as much data as possible in digital form and to use modern technologies (additive technology). [4] · Big Data for marketing and internacionalization Big Data analysis can provide important information that can be used not only to design innovation but also to support trends in the sales sector. [4] · Scanning of ceramics Traditional ceramic products (tiles) are fitted with sensors or solar systems to collect digital information. These sensors can be used in the construction of modern so-called smart buildings. [4] · Monitoring system An unnamed Portuguese company started placing QR codes on their final products. Thanks to the QR codes it is possible to obtain accurate information about the manufacturing process of the product (manufacturing method, quality control, batch number, etc.). [4] · Ceramics Industry 4.0maturity model The maturity model for ceramics industry is still being in development. In Portugal there are involved many universities, companies and the Ceramics Industry Association in that development. The aim of the project is to create a model according to which it will be possible to create solutions for each subsector of the ceramic industry. [4] Unlike the automotive or aviation industry, the implementation of Industry 4.0 elements in the ceramic industry varies in many factors. There is a big difference in the production profile, storage of input material and finished products, but especially the current state of production automation. Already developed maturity models are individually tailored to the needs of the industry and business, as they are designed in a very general way to serve as a quick tool for subsequent decision-making on the future implementation of Industry 4.0 elements, but all in specific company. So far, no methodology has been described to offer comprehensive solutions for companies of different sizes in the ceramic industry. The following part of the article discusses the suggestion of possible methodologies for the readiness evaluation and the subsequent implementation of Industry 4.0 elements that could be used for refractory industry companies of various sizes.

Design of Industry 4.0 Readiness Model for Refractory
A key factor in designing a model for assessing the readiness for Industry 4.0 in the refractory industry is its continuity with the future model for the subsequent deployment of elements of Industry 4.0 in the company. The aim of the model is to enable various companies involved in the refractory industry to be able to assess their current preparedness and identify bottlenecks. The right approach to addressing these bottlenecks should then provide the latter model for the implementation of Industry 4.0 elements carried out by using Complex methodical tool, which will be developed in the future and will be closely connected to this Readiness model.
The inspiration for the proposed model was the previous research of general evaluation models for Industry 4.0 which have been verified by several years of functionality. The difference is the specific focus on the refractory industry which has been reflected in the overall structure of the model and the composition of issues that are often directed directly to manufacturing technologies for the manufacture of refractory products. The model is, like most models, divided into 6 separate categories which are evaluated separately (see Fig. 3). The first two categories "company status" and "market status" do not assess preparedness, but merely serve to classify the company in the segment. The reason is a different approach to companies of different sizes. The other 4 categories of "IT and Infrastructure", "Input Materials", "Production Processes" and "Information and Data flows" will be essential to assess the overall preparedness for deployment of Industry 4.0. The weight of the influence of individual categories and their sub-parts will be partly determined by the first two categories, which will provide general information about the company.
· SWOT analysis During the design phase of a new project it is essential to create a SWOT analysis with which it is possible to identify threats and market opportunities in a timely manner. SWOT analysis evaluates external and internal featuresstrengths and weaknesses of the project. The created SWOT analysis for the proposed evaluation and implementation model is shown in Fig.  4 and described in other subchapters.  a team of authorized personel who will lead the whole process. If the team is not sufficiently experienced and familiar with the issue, there may be a misunderstanding or non-compliance with the evaluation process.
Based on a relatively extensive examination of the issue, it was concluded that an evaluation model for the introduction of elements of industry 4.0 is still not created, which would be designed especially for refractory. The implementation of IT infrastructure and IoT can provide the company with "data history" that can be used in future research and optimization of production processes. Thanks to the already mentioned IT infrastructure and IoT, the company should be able to maintain its competitiveness and improve its ability to plan orders and used resources.
The proposed evaluation models are designed for use only with refractory, so it would be very difficult to use them in another industry. There is a security risk associated with the introduction of IT technologies and IoT. A company must defend itself against attacks by hackers, because it could lose its know-how and very sensitive data due to data leakage.
· Company Status The first step is to evaluation of the current state of the company. Important factor that plays a big role in evaluating readiness for the implementation of elements of Industry 4.0 is to gain an overview of the company's financial situation. The changes associated with the introduction of the elements of Industry 4.0 are in most cases expensive and it is necessary to think about the way of financing and the maximum amount of investment that the company is able to release. The strategy of the company is related to the investment. If a business has no strategy, it will be more difficult for it to prepare for change. A company that has a comprehensive strategy, has an overview of its future direction and can also modify this strategy at any time. No less important factor for the Industry 4.0 elements implementation is the size and the turnover of monitored company. A small company that doesn´t have a high turnover should not think of a high investment in the Industry 4.0 elements implementation, while a large company whose turnover is many times higher has greater potential to capitalize on the investment in the implementation of Industry 4.0 elements.

· Market Situation
The second step in the proposed methodology is to evaluate the market and product development. It is very important to predict how the situation of the market, in that business sector where the company operates, will evolve. If demand for a given product is expected to decrease, it is advisable to think about the development of the production profile or its complete change. In addition to future development, it is also important to evaluate the current state of the market, its possibilities and the position of the company in the market.
· IT and Infrastructure This category solves the current culture and state of the company in terms of IT and infrastructure. The current state of networking in the company are gradually being solved and if the machines are connected by hardware or if they are connected wirelessly. Other thing which is being solved is Wi-Fi coverage in whole company. Whether they are able to retrieve data with the machines. Next step is to define how they work with the data in the current situation and if they can use it for back analysis. Simultaneously, software security (ERP, PLC, MES, CAD, etc.), production planning process, number and qualification of IT staff are being solved in the same time. It is also necessary to find out whether the company uses the cloud or another form of storage to collect and store data. In the context of data sharing it is important to be available throughout the company.
· Input Materials Considering to the input materials, it is absolutely crucial to determine the current state of the takeover of materials and their storage. First of all the origin of all raw materials and the possibility of supplying data on their status are ascertained. Some companies in the refractory industry extract all or part of the materials they need while others buy them all. In most cases, the delivered material condition is not documented at the acceptance and it is dependent on the season (especially the humidity). For this reason, it is necessary to identify when the company ascertains the condition of the material. Whether the condition is detected already in the warehouse or during the preparation of the mixture, where depending on the state of the material the recipe and the amount of added water is adapted. The current conditions in the warehouse, warehouse equipment using sensors, the possibility of data collection and their digitization are analyzed. At the same time, recycling of own raw materials and ground scrap products is being solved in this category.
· Production processes The production processes are focused purely on the technology of the production system. In the first part the questions are focused on the preparation of the mixture. It is being determined if the tools are equipped with humidity sensors, the type of raw material transport to the mixer and its automatic filling. In case of using the obtained data, their analysis and reuse, it is necessary to find out whether the data is used for automatic filling of stabilizers, water and the complete composition of the recipe mixture adjusted according to the data. In the molding section, the tools are equipped with pressure sensors on presses, production time and the type of the used technology (some technologies may be worse or not equipped with sensors). It is also essential to find out whether a system of mold digitization and wear monitoring is used. The shaped articles are dried and burned in respective kilns on kiln cars. During these operations, the kiln operation time is determined by which the temperature is adjusted, thereby influencing and controlling the residual moisture and whether the sensor data obtained actively affects the change in the actual furnace conditions. The last important part in terms of production processes is quality control. In some companies, this activity is being made manually and measurement evaluation is done by qualify workers. On the other hand, in some companies the quality control is fully automated data can be obtained and flawlessly distinguish between compliant and noncompliant products. At the same time, the obtained data can be used retrospectively to improve the production process. The aim is to determine the current status of quality control.
· Information and Data flow The last category is Information and data flows in the company, mainly focused on state of processes in the company. In this category, it is determined whether data collection works using the sensors, IT equipment and infrastructure for vertical and horizontal communication within the machine. Whether the acquired data is sent to a central system that selects the data and sends it to the appropriate machines or uses decentralized control. Whether a machine-to-machine communication exists, or how data sorting and data interconnection with the supplier and customer are done.
In the model, all categories listed above are evaluated separately and after that, based on the weights of each category, the overall readiness for Industry 4.0 is determined. The distribution into several categories is suitable for the future implementation model which according to the importance and the current state of each category will provide a suitable implementation procedure.

Design of implementation model of elements of Industry 4.0 based on evaluation model
The proposed Implementation methodology will be based on the methodology for evaluating the Industry 4.0 readiness of the refractory industry described in this article. The methodology will include 3 stages of the procedure. In the first phase, the most critical sites that need to be addressed primarily will be identified based on readiness assessments. This phase will also include an evaluation of these bottlenecks according to their importance to the refractory industry. The second phase will be the creation of a strategic plan for the implementation of Industry 4.0 elements, staff training and others. Then comes the third phase of actual implementation of the selected elements, according to the strategic plan. Readiness evaluation follows again. This process will be then repeated until the desired state is reached (see 5).

Conclusion
This article presents the results of a study that focuses on designing a readiness model for evaluating a company in the refractory industry to implement Industry 4.0 elements. There are several companies around the world that have been dealing with this issue for several years, but most of the models developed focus on the automotive or aerospace industry, not the refractory industry. Evaluation models are mostly based on the questionnaire principle and subsequent evaluation of the current status. Unlike the automotive industry, the implementation of Industry 4.0 elements in the refractory industry is very difficult because some businesses still use outdated technology and almost do not use data digitization, also there are more strict conditions for the production itself, apart from other subsectors of ceramic industry.
In the second part, a readiness model was designed to evaluate the current state of readiness of the monitored enterprise for the implementation of Industry 4.0 elements. In this model there are 6 groups in which the current state of the company is analyzed in detail. The proposed methodology for evaluating the company's preparedness for Industry 4.0 will subsequently be paired with the Model for the implementation of elements of Industry 4.0. This model for the implementation process is still in development, preliminarily divided into 4 basic phases (see 5). These phases are repeated until the desired results are achieved, giving accurate results that can be used to increase productivity and quality of the entire production. By the combination of Readiness model described in this article and future Model for the implementation of elements of Industry 4.0 to Refractory Industry, new tool for complex and precise assessment and implementation of modern solutions for Refractory companies will be developed.

Acknowledgement
This work was supported by the Czech Technical University and funded from Student grant competition grant number SGS19/105/OHK2/2T/12


Compressing Pre-trained Language Models by Matrix Decomposition

Large pre-trained language models reach state-of-the-art results on many different NLP tasks when fine-tuned individually; They also come with a significant memory and computational requirements, calling for methods to reduce model sizes (green AI). We propose a two-stage model-compression method to reduce a model’s inference time cost. We first decompose the matrices in the model into smaller matrices and then perform feature distillation on the internal representation to recover from the decomposition. This approach has the benefit of reducing the number of parameters while preserving much of the information within the model. We experimented on BERT-base model with the GLUE benchmark dataset and show that we can reduce the number of parameters by a factor of 0.4x, and increase inference speed by a factor of 1.45x, while maintaining a minimal loss in metric performance.

Introduction
Deep learning models have been demonstrated to achieve state-of-the-art results, but require large parameter storage and computation. It's estimated that training a Transformer model with a neural architecture search has a CO 2 emissions equivalent to nearly five times the lifetime emissions of the average U.S. car, including its manufacturing (Strubell et al., 2019). Alongside the increase in deep learning models complexity, in the NLP domain, there has been a shift in the NLP modeling paradigm from training a randomly initialized model to fine-tuning a large and computational heavy pre-trained language model (Howard and Ruder, 2018;Peters et al., 2018;Devlin et al., 2018;Radford, 2018;Radford et al., 2019;Lample and Conneau, 2019;Liu et al., 2019b;Raffel et al., 2019;Lan et al., 2019;Lewis et al., 2019).
While re-using pre-trained models offsets the training costs, inference time costs of the finetuned models remain significant, and are showstoppers in many applications. The main challenge with pre-trained models is how can we reduce their size while saving the information contained within them. Recent work, approached this by keeping some of the layers while removing others (Sanh et al., 2019;Sun et al., 2019;Xu et al., 2020). A main drawback of such approach is in its coarse-grained nature: removing entire layers might discard important information contained within the model, and working at the granularity of layers makes the trade-off between compression and accuracy of a model hard to control. Motivated by this, in this work we suggest a more finegrained approach which decomposes each matrix to two smaller matrices and then perform feature distillation on the internal representation to recover from the decomposition. This approach has the benefit of preserving much of the information while reducing the number of parameters. Alongside the advantage of preserving the information within each layer, there is also a memory flexibility advantage compared to removing entire layers; As a result of decomposing each matrix to two smaller matrices, we can store each of the two matrices in two different memory blocks. This has the benefit of distributing the model matrices in many small memory blocks, which is useful when working in shared CPU-based environments.
We evaluated our approach on the General Language Understanding Evaluation (GLUE) benchmark dataset (Wang et al., 2018) and show that our approach is superior or competitive in the different GLUE tasks to previous approaches which remove entire layers. Furthermore, we study the effects of different base models to decompose and show the superiority of decomposing a fine-tuned model compared to a pre-trained model or a ran-domly initialized model. Finally, we demonstrate the trade-off between compression and accuracy of a model.
We also note that very recently a work similar to ours was uploaded to arxiv (Mao et al., 2020). There are a few differences from their work to ours. Firstly, we distill different parts of the model (see Section 3 for details). Secondly, we focus on training the decomposed model and do not prune the model parameters. Thirdly, our base model, which is used for decomposition and as a teacher, is a fine-tuned model; This has the benefit of task-specific information as we show in our experiments in Section 4.2.

Method
Our goal is to decompose each matrix W ∈ R n×d as two smaller matrices, obtaining an approxi- is small for all x, where d is a distance metric between vectors. In practice, we require the condition to hold not for all x, but for vectors seen in a finite relevant sample (in our case, the training data). While one could start with random matrices and optimize the objective using gradient descent, we show that a two-staged approach performs better: we first decompose the matrices using SVD, obtaining A , B s.t. ||A B − W || 2 2 is small (SVD is guaranteed to produce the best rank-r approximation to W , (Stewart, 1991)). We then use these matrices as initialization and optimize d(W x, W x) (feature distillation), while also optimizing for task loss. We show that this process works substantially better in practice. Our loss function is thus composed of three different objectives: Cross Entropy Loss The cross entropy loss over an example x with label y is defined likewise: where p s is the probability for label y given by the decomposed student model.

Knowledge Distillation Loss
The goal of knowledge distillation is to imitate the output layer of a teacher model by a student model. The Knowledge Distillation Loss is defined likewise: where z s and z t are the logits of the decomposed and original models respectively and T is a temperature hyper-parameter.

Feature Distillation Loss
The goal of feature distillation is to imitate the intermediate layers of a teacher model by a student model. we use the following intermediate representations to distill the knowledge from 1 : • Query, Key and Value Layers -The dot product of a matrix of concatenated tokens representation vectors X by the query, key and value parameter matrices, • Attention Heads -The output of the attention heads.
• The Multihead Attention Layer Output -The dot product of the attention heads by the ma- • The first feed forward layer -The dot product of the multihead attention layer by the first feed forward layer.
• The second feed forward layer -The dot product of the first feed forward layer by the second feed forward layer.
We denote S i z and T i z as the intermediate representations which were described above of layer i for the decomposed student and original teacher models respectively. Our loss function then is defined by: Full Objective Our loss function is then defined by a weighted combination of these three loss functions likewise:

Experiments
We compare various variants of our compression method, corresponding to different subsets of our loss. All variants decompose the matrices using SVD, but differ in their objective functions. These correspond to the four last lines in Table  1. Low Rank BERT Fine-tuning (LRBF) corresponds to L = L CE . LRBF+KD corresponds to L = αL CE + (1 − α)L KD . LRBF+FD corresponds to L = L CE +L F D , while LRBF+FD+KD corresponds to the complete objective. The other lines in the table correspond to uncompressed model (first line) and to baselines which prune layers and distill. Fine-tuning finetunes a six layered BERT model. Vanilla KD trains a six-layered BERT model with L = αL CE +(1−α)L KD . BERT-PKD trains a six layered BERT model with L = αL CE + (1 − α)L KD while also adding an L F D objective, but on the hidden states between every consecutive layer. BERT-of-Theseus fine-tunes BERT model while gradually pruning half of the layers. We chose this baselines for several reasons: like our method they result in a practical reduction of parameters; 2 , they are task-specific; 3 and they do not require the pretraining stage, which is expensive and not practical for most practitioners.
Datasets We evaluate our proposed approach on the General Language Understanding Evaluation (GLUE) benchmark (Wang et al., 2018), a collection of diverse NLP tasks.
Training Details We fine-tune a pre-trained BERT model (Devlin et al., 2018) for each task with a batch size of 8 and a learning rate of 2e−5 for 3 epochs with an early stop mechanism according to the validation set. We perform the matrix decomposition on every parametric weight matrix of the encoder (excluding the embedding matrix) in a fine-tuned model and train the decomposed model as the student model and the original fine-tuned model as the teacher. For each task we train for 3 epochs with an early stopping mechanism according to the task validation set, the maximum sequence length is 128 and we perform a grid search over the learning rates {2e −6 , 5e −6 , 2e −5 , 5e −5 , 2e −4 , 5e −4 } and 5 different seeds and choose the best model according to the validation set of each task. 4 For knowledge distillation hyper-parameters we used a temperature hyper-parameter T = 10 and α = 0.7. 5 Table 1 compares the results for validation and test of other compression approaches which prune layers, along with low rank models which were finetuned and trained with one or more of the distillation objectives described in Section 3. As can be seen, Low Rank BERT Feature Distillation + KD and Low Rank BERT Feature Distillation surpass all of results of all methods in both validation and test sets except BERT-of-Theseus method in the test set, in which Low Rank BERT Feature Distillation + KD surpasses the results in 5 of the tasks and reach comparable results in 2 of the tasks. Also, as can be seen knowledge distillation alone is not sufficient to compensate for the decomposition, but it slightly improves the results when incorporating feature distillation alone.

Further Analysis
Effect of Base Model and Decomposition In this experiment we test the importance of the base model we use to decompose and use as a teacher. We compared between three types of distillation sources: fine-tuned teacher, pre-trained teacher and no teacher. Furthermore, we compared between three types of model initializations: a decomposed fine-tuned model, a decomposed pretrained model and a randomly initialized model with the same architecture as the decomposed models. The results are shown in Table 2, on all tasks when training with no teacher distilla-  (Hinton et al., 2015) 45   tion, the results are best when decomposing a finetuned model and decomposing a pre-trained model is better than randomly initializing a model; This indicates that the decomposition saves the information within the model and when decomposing a fine-tuned model it saves some of the more task specific information. Furthermore, on all tasks and all initialization the best results are when using a fine-tuned model as a teacher.  Compression vs. Performance Trade-off Our method requires to determine a rank for the compression. But can we achieve better results when choosing a higher rank? Can we choose a lower rank for smaller models and still achieve satisfactory results? To determine this we experimented on three different ranks. As shown in Table 3, higher ranks achieve better results, while lower ranks achieve satisfactory results while compromising metric performance.

Run-time Savings
In this experiment we measured the average time in milliseconds it takes for BERT-base compared to its decomposed and six-layered counterparts to output predictions for a batch of samples with varying batch sizes. As shown in Figure 1, we still gain a significant time performance improvement when running on both CPU and GPU architectures over a BERT-base model. Models that are decomposed to a rank r = 245 are about 1.45 faster than their uncompressed counterpart for batches larger than one when running on a GPU and around 1.2 − 1.55 faster for batches 8, 16, 32, 64 when running on a CPU. Furthermore, higher ranks still benefit running time and lower ranks improve the running time further. Also, we note that although a six-layered BERT does achieve faster inference time, due to the coarse-grained compression, it losses more information contained within it and thus achieves inferior results; As shown in the results in Table 1, a six-layered model trained with distillation (e.g. BERT-PKD (Sun et al., 2019)) achieves significantly lower results and the BERT-of-Theseus model, which does improve upon BERT-PKD, requires many training iterations to achieve this to overcome the loss of information when gradually removing entire layers, which result in higher training times.

Conclusions
We presented a way to compress pre-trained large language models fine-tuned for specific tasks, while preserving much of the information contained within them, by using matrix decomposition to two small matrices. For future work it might be interesting to combine this approach with another approach such as pruning or quantization to achieve smaller models.


Vec2Face: Unveil Human Faces from their Blackbox Features in Face Recognition

Unveiling face images of a subject given his/her high-level representations extracted from a blackbox Face Recognition engine is extremely challenging. It is because the limitations of accessible information from that engine including its structure and uninterpretable extracted features. This paper presents a novel generative structure with Bijective Metric Learning, namely Bijective Generative Adversarial Networks in a Distillation framework (DiBiGAN), for synthesizing faces of an identity given that person's features. In order to effectively address this problem, this work firstly introduces a bijective metric so that the distance measurement and metric learning process can be directly adopted in image domain for an image reconstruction task. Secondly, a distillation process is introduced to maximize the information exploited from the blackbox face recognition engine. Then a Feature-Conditional Generator Structure with Exponential Weighting Strategy is presented for a more robust generator that can synthesize realistic faces with ID preservation. Results on several benchmarking datasets including CelebA, LFW, AgeDB, CFP-FP against matching engines have demonstrated the effectiveness of DiBiGAN on both image realism and ID preservation properties.

Introduction
Face recognition has recently matured and achieved high accuracy against millions of identities [47,48]. A face recognition system is often designed in two main stages, i.e. feature extraction and feature comparison. The role of feature extraction is more important since it directly determines the robustness of the engine. This operator defines an embedding process mapping input facial images into a higher-level latent space where embedded features extracted from photos of the same subject distribute within a small margin [32]. Moreover, since most face recognition engines are set into a blackbox mode to protect the Figure 1. Metric Learning for Image Reconstruction. By maintaining the one-to-one mapping via a bijection, the distance between images can be directly and intuitively measured and enhances the metric learning process for image reconstruction task.
technologies [25], there is no apparent technique to inverse that embedding process to reconstruct the faces of a subject given his/her extracted features from those engines.
Some Blackbox Adversarial Attack approaches [20,21,44] have partially addressed this task by analyzing the gradients of the classifier's outputs to generate adversarial examples that mislead the behaviour of that classifier. However, they only focus on a closed-set problem where the output classes are predefined. Moreover, their goal is to generate imperceptible pertubations added to the given input signal. Other methods [1,4,7,43,53] are also introduced in literature but still require the access to the classifier structure, i.e. whitebox setting. Meanwhile, our goal focuses on a more challenging reconstruction task with a blackbox face recognition. Firstly, this process reconstructs faces from scratch without any hint from input images. Secondly, in a blackbox setting, there is no information about the engine's structure, and, therefore, it is unable to directly exploit knowledge from the inverse mapping process (i.e. back-propagation). Thirdly, the embedded features from a face recognition engine are for open-set problem where no label information is available. More importantly, the subjects to be reconstructed may have never been seen during training process of the face recognition engine. In the scope of this work, we assume that the face recognition engines are primarily developed by Convolutional Neural Networks Table 1. Comparisons of our DibiGAN and other unrestricted synthesis methods. Image Reconstruction (Img Recon), Feature Representation (Feat), Guided Image (ImgG), Feature Conditional (Feat Cond), Neighborly Deconvolution (NB Deconv), Optimization (Opt).

Ours
NBNet [34] SynNormFace [4] IFaceRec [53]  (CNN) that dominate recent state-of-the-art results in face recognition [5,10,12,28,29,40,45,46,52]. We also assume that there is no further post-processing after the step of CNN feature extraction. We then develop a theory to guarantee the reconstruction robustness of the proposed method.
Contributions. This paper presents a novel generative structure, namely Bijective Generative Adversarial Networks in a Distillation framework (DiBiGAN), with Bijective Metric Learning for the image reconstruction task. The contributions of this work are four-fold. (1) Although many metric learning techniques have been introduced in the literature, they are mainly adopted for classification rather than reconstruction process. By addressing limitations of classifier-based metrics for image reconstruction, we propose a novel Bijective Metric Learning with bijection (one-to-one mapping) property so that the distances in latent features are equivalent to those between images (see Fig. 1). It, therefore, provides a more effective and natural metric learning approach to the image reconstruction task. (2) We exploit different aspects of the distillation process for the image reconstruction task in a blackbox mode. They include distilled knowledge from the blackbox face matcher and ID knowledge extracted from a real face structure. (3) We introduce a Feature-Conditional Generator Structure with Exponential Weighting Strategy for Generative Adversarial Network (GAN)-based framework to learn a more robust generator to synthesize realistic faces with ID preservation. (4) Evaluations on benchmarks against various face recognition engines have illustrated the improvements of DiBiGAN in both image realism and ID preservation. To the best of our knowledge, this is one of the first metric learning methods for image reconstruction (Table 1).

Related Work
Synthesizing images [4,7,8,9,34,53] has brought several interests from the community. We divided into two groups, i.e. unrestricted and adversarial synthesis. Unrestricted synthesis. The approaches focus on reconstructing an image from scratch given its high-level representation. Since the mapping is from a low-dimensional latent space to a highly nonlinear image space, several regularizations have to be applied, e.g. Gaussian Blur [51] for high-frequency samples or Total Variation [11,33] for maintaining piece-wise constant patches. These optimization-based techniques are limited with high com-putation or unrealistic reconstructions. Later, Dosovitskiy et al. [7] proposed to reconstruct the image from its shallow (i.e. HOG, SIFT) and deep features using a Neural Network (NN). Zhmoginov et al. [53] presented an iterative method to invert Facenet [40] feature with feed-forward NN. Cole et al. [4] proposed an autoencoder structure to map the features to frontal neutral face of the subject. Yang et al. [49] adopted autoencoder for model inversion task. Generally, to produce better synthesized quality, these approaches require full access to the deep structure to exploit the gradient information from the embedding process. Mai et al. [34] developed a neighborly deconvolutional network to support the blackbox mode. However, with only pixel and perceptual [23] losses, there are limitations of ID preservation when synthesizing different features of the same subject. In this work, we address this issue with Bijective Metric Learning and Distillation Knowledge for reconstruction task. Adversarial synthesis. Adversarial approaches aim at generating unnoticable perturbations from input images for adversarial examples to mislead the behaviour of a deep structure. Either directly accessing or indirectly approximating gradients, adversarial examples are created by maximizing corresponding loss which can fool a classifier [1,2,3,20,21,30,35,42,44]. Ilyas et. al. [21] proposed bandit optimization to exploit prior information about the gradient of deep learning models. Later, Ilyas et. al. [20] introduced Natural Evolutionary Strategies to enable query-efficient generation of black-box adversarial examples. Other knowledge from the blackbox classifier are also exploited for this task [44,1,43]. Generally, although the approaches in this direction tried to extract the gradient information from a blackbox classifier, their goal are mainly to mislead the behaviours of the classifier with respect to a pre-defined set of classes. Therefore, they are closed-set approaches. Meanwhile, in our work, the proposed framework can reconstruct faces of subjects that have not been seen in the training process of the classifier.

Our Proposed Method
Let F : I → F be a function that maps an input image I from image domain I ∈ R W ×H×C to its high-level embedding feature F (I) in latent domain F ∈ R M . In addition, a function C : F → Y takes F(I) as its input and gives the identity (ID) prediction of the subject in space Y ∈ R N where each dimension represents a predefined subject class.  The approaches solving this problem usually exploit the relationship between an input image and its class label for the reconstruction process. Moreover, since the output score s is fixed according to predefined N classes, the reconstruction is limited on images of training subject IDs. Definition 2 (Feature Reconstruction). Given a blackbox functions F ; and its embedding feature f = F (I) of an unknown image I, feature reconstruction is to recover I from f by optimizingĨ * = arg minĨ L(F (Ĩ), f ). Compared to the model inversion problem, Feature Reconstruction is more challenging since the constraints on predefined classes are removed. Therefore, the solution for this problem turns into an open-set mode where it can reconstruct faces other than the ones used for learning F , i.e. face recognition engine. Moreover, since the parameters of F are inaccessible due to its blackbox setting, directly recovering I based on its gradient is impossible. Therefore, the feature reconstruction task can be reformulated via a function (generator) G : F → I as the reverse mapping of F .
, θ g denotes the parameters of G, and p I (x) is the probability density function of x. In other words, p I (x) indicates the distribution that image I belonged to (i.e. the distribution of training data of F ). Intuitively, function G can be seen as a function that maps images from embedding space F back to image space such that all reconstructed images [G • F ](x; θ g ) are maintained to be close to its real x with respect to the distance metric L x G . To produce "good quality" synthesis (i.e. realistic images with ID preservation), different choices for L x G have been commonly exploited [22,23,34] such as pixel difference in image domain via L 1 /L 2 distance; Probability Distribution Divergence (i.e. Adversarial loss defined via an additional Discriminator) for realistic penalization; or Perceptual distance that penalizes the image structure in highlevel latent space. Among these metrics, except the pixel difference that is computed directly in image domain, the others are indirect metrics where another mapping function (i.e. classifier) from image space to latent space is required.

Limitations of Classifier-based Metrics
Although these indirect metrics have shown their advantages in several tasks, there are limitations when only the blackbox function F and its embedded features are given. Limitation 1. As shown in several adversarial attack works [15,39], since the function F is not a one-to-one mapping function from I to F, it is straightforward to find two images of similar latent representation that are drastically different in image content. Therefore, with no prior knowledge about the subject ID of image I, starting to reconstruct it from scratch may easily fall into the case where the reconstructed imageĨ is totally different to I but has similar embedding features. The current Probability Distribution Divergence with Adversarial Loss or Perceptual Distance is limited in maintaining the constrain "the reconstructions of features of the same subject ID should be similar". Limitation 2. Since the access to the structure and intermediate features of F is unavailable in the blackbox mode, the function G is unable to directly exploit valuable information from the gradient of F and the intermediate representation during embedding. As a result, the distance metrics defined via F , i.e. perceptual distance, is less effective as in whitebox setting. Next sections will introduce two loss functions to tackle these problems to learn a robust function G.

Bijective Metrics for Image Reconstruction
Many metric learning proposals for face recognition [5,28,29,45,46,52] have been used to improve both intraclass compactness and inter-class separability with a large margin. However, for feature reconstruction, directly adopting these metrics, e.g. angular distance for reconstructed images to cluster images of the same ID is infeasible.
Therefore, we propose a bijection metric for feature reconstruction task such that the mapping function from image to latent space is one-to-one. The distance between their latent features is equivalent to the distance between images. By this way, these metrics are more aligned to image domain and can be effectively adopted for reconstruction task. Moreover, since two different images cannot be mapped to the same latent features, the metric learning process is more reliable. The optimization of G in Eqn. (1) is rewritten as: and p x (x) denotes a density function estimated from an alternative large-scale face dataset.
Notice that although the access to p I (x) is not available, this approximation can be practically adopted due to a prior knowledge about p I (x) that images drawn from p I (x) are facial images. Let H : I → Z define a bijection mapping from x to a latent variable z = H(x). With the bijective property, the optimization in Eqn. (2) is equivalent to.
arg min where z = H(x); p x (x) = p z (z)| det(J x J x )| 1/2 by the change of variable formula; J x is the Jacobian of H with respect to x; and L z G is the distance metric in Z. Intuitively, Eqn. (3) indicates that instead of computing the distance L x G and estimating p x (x) directly in image domain, the optimization process can be equivalently accomplished via the distance L z G and density p z (z) in Z according to the bijective property of H. The prior distributions p z . In general, there are various choices for the prior distribution p z and the ideal one should have two properties: (1) simplicity in density estimation, and (2) easily sampling. Motivated from these properties, we choose Gaussian distribution for p z . Notice that other distribution types are still applicable in our framework. The distance metric L z G . With the choice of p z as a Gaussian, the distance between images in I is equivalent to the deviation between Gaussians in latent space. Therefore, we can effectively define L z G as the squared Wasserstein coupling distance between two Gaussian distributions. where {μ,Σ} and {µ, Σ} are the means and covariances of z and z, respectively. The metric L z G then can be extended with image labels to reduce the distance between images of the same ID and enhance the margin between different IDs.
where m defines parameter controlling the margin between classes; and {lz 1 , lz 2 } denote the subject ID of {z 1 ,z 2 }.
Learning the Bijection H. In order to effectively learn the bijection H, we adopt the structure of mapping function from [6,13,14] as the backbone for the tractable log-det computation with the log-likelihood loss for training process. Moreover, to further improve the discriminative property of H in latent space Z, we propose to exploit the ID label in training process of H. Particularly, given K classes (i.e. ID) of the training set, we choose K Gaussian distributions with different means {µ 1 , µ 2 , .., µ K } and covariances {Σ 1 , Σ 2 , ..., Σ K } and enforce samples of each class distributed on its own prior distribution, i.e. z k ∼ N (µ k , Σ k ). Formally, the log-likelihood loss function to learn H is formulated as θ * H = arg max θ H log p x (x, k; θ H ) = arg max θ H log p z (z, k; θ H ) + 1 2 log | det(J x J x )|.

Reconstruction from Distillation Knowledge
In the simplest approach, the generator G can still learn to reconstruct image by adopting the Perceptual Distance as in previous works to compare F (Ĩ) and F (I). However, as mentioned in Sec. 3.1, due to limited information that can be accessed from F , "key" information (i.e. the gradients of F as well as its intermediate representations) making the perceptual loss effective is lacking. Therefore, we propose to first distill the knowledge from the blackbox F to a "student" function F S and then take advantages of these knowledge via F S for training the generator. On one hand, via the distillation process, F S can mimic F by aligning its feature space to that of F and keeping the semantics of the extracted features for reconstruction. On the other hand, with F S , the knowledge about the embedding process of F (i.e. gradient, and intermediate representation) becomes more transparent; and, therefore, maximize the information which can be exploited from F . Particularly, let F S : I → F and F S = F S 1 • F S 2 · · · • F S n be the composition of n-sub components. The knowledge from F can be distilled to F S by aligning their extracted features as.
Then G is enhanced via the distilled knowledge of both final embedding features and intermediate representation by.
where {λ j } n 1 and λ a denote the hyper-parameters controlling the balance between terms. The first component of  Network Architecture. Given an input image x, the generator G takes F (x) as its input and aims to synthesize an imagex that is as similar to x as possible in terms of identity and appearance. We adopt the GAN-based generator structure for G and optimize using different criteria.

Learning the Generator
where {L biject , L distill , L adv , L recon } denote the bijective, distillation, adversarial, and reconstruction losses, respectively. {λ b , λ d , λ adv , λ r } are their parameters controlling their relative importance. D is a discriminator distinguishing the real images from a synthesized one. There are three main critical components in our framework including the Bijective H, the student matcher F S for ID preservation; and the discriminator D for realistic penalization. The Discriminator D is updated with the objective function as.
where px is the random interpolation distribution between real and generated images [16]. Then, the whole framework is trained following GAN-based minimax strategy. Learning Strategies. Besides the losses, we introduce a Feature-Conditional Structure for G and a exponential Weighting Strategy to adaptively scheduling the importance factors between loss terms during training process. Feature-conditional Structure. A natural design for the structure of G is to directly use F (x) as the input for G. However, this structure limits the learning capability of G. Particularly, besides ID information, F (x) may include other "background" conditions such as poses, illuminations, expressions. Therefore, setting F (x) as the only input implicitly enforces G to "strictly" model these factors as well. This makes the training process of G less effective. To relax this constraint, we introduce a Feature-Conditional structure (i.e. the generator structure in Fig. 2) where a random variable v is adopted as an additional input so that these background factors can be modeled through v. Moreover, we propose to use v as the direct input to G and inject the information from F (x) through out the structure. By this way, F (x) can act as the conditional ID-related information for all reconstruction scales and gives the better synthesis. Exponential Weighting Strategy. As the progressive growing training strategy [24] initializes its learning process on synthesizing low-resolution images and then gradually increasing their levels of details, it is quite effective for enhancing the details of generated images in general. However, this strategy has limited capability in preserving the subject ID. In particular, in the early stages at low scales with blurry synthesis, it is difficult to control the subject ID of faces to be synthesized while in the later stages at higher scales when the generator becomes more mature and learns to add more details, the IDs of those faces have already been constructed and become hard to be changed. Therefore, we propose to adopt a exponential weighting scheme for (1) emphasizing on ID preservation in early stages; and (2) enhancing the realism in later stages. Particularly, the param- where R(i) denotes the current scales of stage i and R M is the maximum scales to be learned by G.

Experimental Results
We qualitatively and quantitatively validate our proposed method in both reconstructed quality and ID preservation in several in-the-wild modes such as poses, expressions, and occlusions. Both image-quality and face-matching datasets are used for evaluations. Different face recognition engines are adopted to demonstrate the robustness of our model. Data Setting. Our training data includes the publicly available Casia-WebFace [50] with 490K labeled facial images of over 10K subjects. The duplicated subjects between training and testing sets are removed to ensure no overlapping between them. For validation, as commonly used for attribute learning and image quality evaluation, we adopt the testing split of 10K images from CelebA [31] to validate the reconstruction quality. For ID preservation, we explore  [34] and MPIE-NBNet [34] (2nd and 3rd rows) reconstruct faces with limited quality, DibiGAN in whitebox (4th row) and blackbox (5th row) modes are able to produce realistic faces with better ID preservation comparable to real faces (6th row).
LFW [18], AgeDB [36], and CFP-FP [41] which provide face verification protocols against different in-the-wild face variations. Since each face matcher engine requires different preprocessing process, the training and testing data are aligned to the required template accordingly.
Network Architectures. We exploited the Generator structure of PO-GAN [24] with 5 convolutional blocks for G while the Feature Conditional branch consists of 8 fully connected layers. The discriminator D includes five consecutive blocks of two convolution and one downsampling operators. In the last block of D, the minibatch-stddev operator followed by convolution and fully connected are also adopted. AdaIN operator [19] is applied for feature injection node. For the bijection H, we set a configuration of 5 sub-mapping functions where each of them is presented with two 32-feature-map residual blocks. This structure is trained using the log-likelihood objective function on Casia-WebFace. Resnet-50 [17] is adopted for F S .

Model Configurations. Our framework is implemented in
TensorFlow and all the models are trained on a machine with four NVIDIA P6000 GPUs. The batch size is set based on the resolution of output images, for the very first resolution of output images (4 × 4), the batch size is set to 128, the batch size will be divided by two when the resolution of images is doubled. We use Adam Optimizer with the started learning rate of 0.0015. We experimentally set {α = 0.001, β = 1.0, λ j = 1, λ a = 10.0} to maintain the balanced values between loss terms.
Ablation Study. To study the effectiveness of the proposed bijective metric for image reconstruction task, we employ an ablation study on MNIST [26] with LeNet [27] as the function F . We also set to whitebox mode where F is directly used in L distill to remove the effects of other factors. Then 50K training images from MNIST and their 1 × 1024 feature vectors are used to train G. Notice that since the image size is 32 × 32, G and D structures are configured with three convolutional blocks. The resulting distributions of synthesized testing images of all classes without and with L biject are plotted in Fig. 3. Compared to G learned with only classifier-based metrics (Fig. 3(a)), the one with bijective metric learning (Fig. 3(b)) is supervised with more direct metric learning mechanism in image domain, and, therefore, shows the advantages with enhanced intra-and inter-class distributions.

Face Reconstruction Results
This section demonstrates the capability of our proposed methods in terms of effectively synthesizing faces from subject's features. To train DibiGAN, we adopt the ArcFace-Resnet100 [5] trained on 5.8M images of 85K subjects for function F and extract the 1 × 512 feature vectors for all training images. These features together with the training images are then used to train the whole framework. We divided the experiments in two settings, i.e. whitebox and blackbox, where the main difference is the visibility of the matcher structure during training process. In the whitebox mode F is directly used in Eqn. (7) to evaluate L distill G while in the blackbox mode, F S is learned from F through a distillation process as in Eqn. (6) and used for L distill G . The first row of Table 2 shows the matching accuracy of F and F S using real faces on benchmarking datasets. Face Reconstruction from features of frontal faces. After training, given only the deep features extracted from F on testing images, the generator G is applied to synthesize the subjects' faces. Qualitative examples of our synthesized faces in comparison with other methods are illustrated in Fig. 4. As can be seen, our generator G is able to reconstruct realistic faces even when their embedding features are extracted from faces with a wide range of in-the-wild  variations. More importantly, in both whitebox and blackbox settings, our proposed method successfully preserves the ID features of these subjects. In whitebox setting, since the structure of F is accessible, the learning process can effectively exploit different aspects of embedding process from F and produce a generator G that depicts better facial features of the real faces. For example, together with ID information, poses, glasses, or hair style from the real faces can also be recovered. On the other hand, although the accessible information is very limited in blackbox setting, the learned G can still be enjoyed from the distilled knowledge of F S and effectively fill the knowledge gap with whitebox setting. In comparison to different configurations of NBNet [34], better faces in terms of image quality and ID preservation can be obtained by our proposed model. Effect of expressions and occluded regions. Fig. 5 illustrates our synthesis from features of faces that contain both expressions and occlusions. Similar to previous experiment, our model robustly depicts realistic faces with similar ID features as in the real faces. Those reconstructed faces' quality again outperforms NBNet in both realistic and ID terms. Notice that the success of robustly handling with those challenging factors comes from two properties: (1) The matcher F was trained to ignore those facial variations in its embedding features; and (2) both bijective metric learning and distillation process can effectively exploit necessary knowledge from F as well as real face distributions in image domain for synthesis process. Effect of different features from the same subject. Fig.  6 illustrates the advantages of our method in synthesizing faces given different feature representations of the same subject. These results further show the advantages of the proposed bijective metric in enhancing the boundary between classes and constrain the similarity between reconstructed faces of the same subject in image domain. As a result, reconstructed faces from features of the same subject ID not only keep the features of that subject (i.e. similar to Table 2. Realism Quality and Matching Accuracy. Comparison results in Multi-Scale Structural Similarity (MS-SSIM) (the smaller value is better); Inception score and Matching Accuracy (the higher value is better). For each configuration in (A)-(C) and (D)-(F), each loss function is cumulative enable on the top of the previous configuration. − denotes "not applicable".  Fig. 7. These results further illustrate the advantages of our model structure in its capability of capturing various factors for the reconstruction process.

Face Quality and Verification Accuracy
In order to quantitatively validate the realism of our reconstructed images and how well they can preserve the ID of the subjects, three metrics are adopted: (1) Multi-scale Structural similarity (MS-SSIM) [37]; (2) Inception Score (IS) [38]; and (3) face verification accuracy. Image quality. To quantify the realism of the reconstructed faces, we synthesize testing images of CelebA in several training configurations as shown in Table 2, where each loss function in cumulatively enables on the top of the previous configuration. Then MS-SSIM and IS metrics are applied to measure their image quality. We also compare our model in both whitebox and blackbox settings with other baselines including PO GAN structure [24] and NBNet [34]. Notice that we only adopt the adversarial and reconstruction losses for configs (A) and (D). For all configs (A), (B), and (C), PO GAN baseline takes only the embedding features as its input. These results show that in all configurations, our method maintains comparative reconstruction quality as PO GAN and very close to that of real faces. Moreover, our synthesis consistently outperforms NBNet in both metrics. ID Preservation. Our model is experimented against LFW, AgeDB, and CFP-FP where an image in each positive pair is substituted by the reconstructed one while the remaining image of that pair is kept as the reference real face. The matching accuracy is reported in Table 2. These results further demonstrate the advantages and contributions of each 1 We report the accuracy of original matcher F for whitebox setting and F S for blackbox setting. component in our framework. Compared to PO GAN structure, our Feature-Conditional Structure gives more flexibility in modeling ID features, and achieves better matching accuracy. In combination with distilled knowledge from F S , the Generator produces a big jump in accuracy and close the gap to real faces to only 2.02% and 2.72% on LFW in whitebox and blackbox settings, respectively. By further incorporating the bijective metric, these gaps are further reduced to only 0.6% and 0.65% for the two settings.
Reconstructions against different Face Recognition Engines. To illustrate the accuracy of our propose structure, we validate the its performance against different face recognition engines as shown in Table 3. All Generators are set to blackbox mode and only the final extracted features are accessible. Our reconstructed faces are able to maintain the ID information and achieve competitive accuracy as real faces. These performance again emphasizes the accuracy of our model in capturing behaviours of the feature extraction functions F and provides high quality reconstructions.

Conclusions.
This work has presented a novel generative structure with Bijective Metric Learning for feature reconstruction problem to unveil the subjects' faces given their deep blackboxed features. Thanks to the introduced Bijective Metric and Distillation Knowledge, our DibiGAN effectively maximizes the information to be exploited from a given blackbox face matcher. Experiments on a wide range of in-thewild face variations against different face matching engines demonstrated the advantages of our method on synthesizing realistic faces with subject's visual identity.


Efficient Parallel-Beamforming Based on Shared FIFO for Ultra-Compact Ultrasound Imaging Systems

The realization of a parallel-beamforming (pBF) method in ultra-compact ultrasound imaging systems is challenging to achieve uncompromised beamforming accuracy in limited hardware and power resources. In this paper, we present a new hardware- and power-efficient pBF method that utilizes a shared first-in-first-out block on a post-fractional delay filtering architecture (pBF-sFIFO). For an analog-to-digital conversion (ADC) rate given, the proposed pBF-sFIFO method yielded beamforming accuracy comparable to that by an unconstrained pBF (pBF-CON) method, with up to 15% less power consumption. Otherwise, a conventional time-sharing pBF method (pBF-TS) was more vulnerable to aliasing artifacts. Even though increasing ADC rate in the pBF-TS method could recover the beamforming accuracy, exponential need in power consumption was inevitable. Therefore, the pBF-sFIFO method would be an effective solution to enable advanced imaging features in ultra-compact ultrasound imaging systems.

I. INTRODUCTION
In medical ultrasound imaging, there are trends to support the point-of-care diagnosis outside of hospitals, such as in the home, at a patient's bedside, in the battlefield, and in the emergency room [1]. This framework necessitates ultracompact form factor instrumented on highly integrated circuit chip solutions: field programmable gated array (FPGA) and application-specific integrated chip (ASIC) [2]- [10]. Along with this trend towards point-of-care diagnosis, having a high frame rate have been desired to enable more advanced imaging features, that need to compound or analyze temporal changes of ultrasound signals [11]- [16]. For example, color Doppler imaging, an essential diagnostic technique, The associate editor coordinating the review of this manuscript and approving it for publication was Yiming Huo .
necessitates multiple ensemble scanlines to extract Doppler component, whose number would affect the imaging frame rate inverse-proportionally. To address to this technical need, parallel-beamforming (pBF) method has been investigated to generate multiple scanlines simultaneously at a single excitation event. This leads to an increase in frame rates and scanline density over a conventional single-beamforming (sBF) method [17]- [22]. However, an unconstrained pBF (pBF-CON) method, employing a dedicated dynamic receive beamformer for each parallel-scanline generation, would be suboptimal due to excessive burden in hardware resource and power consumption. This will eventually lower areaefficiency and battery life of the system implemented. Therefore, development of a hardware-and power-efficient pBF architecture is needed for ultra-compact ultrasound imaging systems. A time-sharing pBF (pBF-TS) method has been most widely used for the efficiency in hardware and power consumption, in which a single dual-port memory at each channel generate multiple scanlines by simply time-multiplexing an operating rate given, f s [23], [24]. However, the pBF-TS method will have the limited applicability due to the effective beamforming rate lowered by the number of parallelscanlines, M (i.e., f BF = f s /M ). For example, the effective f BF for a quad-beamforming (M = 4) at 40-MHz f s would be 10 MHz, at which aliasing image artifacts will appear with signal component > 5 MHz -it would only support clinical ultrasound transducers at a low center frequency range (f 0 = 2-4 MHz) with bandwidth restriction [25], limiting the efficacy of the ultra-compact ultrasound imaging system.
In this paper, we present a pBF method based on a shared first-in-first-out block (pBF-sFIFO) on a post-fractional delay filtering dynamic receive beamforming architecture [26], [27], in which optimal image quality and power consumption can be both obtained without compromising f BF from the given f s .

II. BASE ARCHITECTURE
In dynamic receive beamforming, the temporal resolution of time-of-flight calculation 16-times finer than f 0 is needed to obtain an appropriate image quality. There are two straightforward solutions: (1) operating ADCs and beamforming circuitry at 16-f 0 or (2) conducting temporal interpolation of the radio-frequency (RF) data digitized at the lower f s (e.g., 4-8 f 0 ), followed by a beamforming circuitry operating at 16 f 0 [28]. However, both solutions would have clear drawbacks in ultra-compact ultrasound imaging systems. The implementations of analog or digital front-end operating at 16 f 0 are either impractical (e.g., 160 MHz for 10-MHz f 0 ), because it must result in poor battery life and high implementation cost with challenging timing constraints.
The post-fractional delay filtering-based dynamic receive beamforming architecture was proposed for a portable ultrasound imaging system. This architecture employs selective fractional delay (FD) filtering on coarsely pre-beamformed data, in order to achieve the uncompromised beamforming accuracy without increasing either f s or f BF . In the previous studies, the 16-f 0 beamforming performance comparable to an interpolation-based beamformer could be obtained at only < 46 % of hardware complexity [18]. Figure 1 demonstrates the conceptual diagram of the architecture when requiring delay compensation with 4 additional fractional delays (i.e., 0, 0.25, 0.5, and 0.75 of the ADC sampling interval). The architecture consists of overall 5 procedures: 1) The digitized RF data from the ADCs are sequentially stored in the RF memory of each channel (i.e., dual-port memory) at f s . 2) The RF data samples indicated by successive focusing address in the ADC sampling interval (i.e., coarse address) are temporarily pulled into the first-in-first-out (FIFO) block, by which the array of sequential RF data samples (block data set) can be generated. The coarse address of each channel is given by where denotes the Cartesian geometry of the ith channel ranging from 1 to the total number of channels N ; k is the sample index; R is the distance between the center channel to a focusing point; c is the speed of sound in soft tissue, e.g., 1,540 m/s; and θ is a steering angle.
3) The block data sets with the same fractional delays along the channels are pre-beamformed on a selective summation block (SSB).  4) The block data sets are parallelized, and fed into the FD filters [27], [29], [30], which yield a fractionallydelayed RF data samples. For example, 8-tap FD filters optimized for the frequency spectrum can provide group delay responses in the frequency range from 0 to 15 MHz when 40-MHz f s is employed, without altering magnitude responses ( Figure 2). 5) Finally, the aggregation of the FD filter outputs yields the beamformed data for an individual focusing point without increasing the f BF from a given f s .

III. PARALLEL-BEAMFORMING BASED ON A SHARED FIFO BLOCK
A. OVERALL ARCHITECTURE Figure 3 shows the conceptual diagram of the modified architecture supporting the proposed pBF-sFIFO method. The fundamental architecture is based on the post-fractional delay filtering described in previous section, but now involves additional parallel-beamforming paths. Each path includes the dedicated SSB and FD filter blocks, and RF data at each channel can be simultaneously accessed by these parallel-beamforming paths. Detailed hardware processing flowchart is presented in Figure S1 in the supplementary materials.

B. SHARED FIFO BLOCK
To supply different block data set for each parallelbeamforming path, the proposed sFIFO block at each channel temporarily stores the extended number of RF data samples. Thus, the appropriate block data set can be multiplexed to respective parallel-beamforming paths without lowering f BF , as shown in Figure 3 (e.g., L sFIFO = 5 and L FD = 4, where L sFIFO and L FD are the lengths of the sFIFO block and FD filter, respectively). More specifically, Figure 4(a) shows the sFIFO block that stores the extended number of RF data samples and shifts a single step whenever RF memory address is increased. The RF memory address is pointed by , respectively. In this case, the parallel-beamforming address calculator will produce 1, 0, and γ for τ 1 , so that local block data sets can be concurrently delivered to each parallelbeamforming path at every focusing point k.

D. ERROR-FREE CONDITION
The proposed pBF-sFIFO method should satisfy two confinements for error-free operation. The first confinement is that the difference between adjacent τ RF i [k] should be less than a single ADC sampling interval to construct block data sets without omitting any RF data sample. Not meeting this condition causes beamforming errors during FD filtering that lasts until the erroneous boundary is drained out of the sFIFO block. When τ RF i [k] is fixed at specific m, the parallelbeamforming addresses at the kth focusing point can be rewritten from Eq. (1-1) as ( x m , z m ) and θ m i are the position offset and steering angle for the mth parallel-scanline, respectively. Therefore, an instantaneous increment of τ RF i [k] at the kth focusing point can be derived by differentiating Eq. (3-1) by k, which is given by Eq. (4) can be rewritten for the given error-free condition (i.e., Finally, Eq. (5) is reduced to

≤1) as
Because Eq. (6) is an identical equation, it shows that the proposed pBF-sFIFO method can conduct parallel-beamforming without errors for the given condition. The second confinement ensuring the error-free operation is that the total length of an sFIFO block should span the maximal difference among parallel-beamforming delays throughout a frame, by which a sFIFO block should always be able to provide appropriate block data sets to parallel-beamforming paths. To consistently satisfy the condition, the length of an sFIFO block can be equated by where · max is the global searching functions for the maximal value throughout the entire channels and depths for all transmitting events comprising a frame. This heuristic approach is obliged because the difference among parallelbeamforming delays depends on the numerous combinations of the imaging specifications defined by user (e.g., steering angle, the number of array elements, the distance between array elements, and sampling frequency, etc.).

IV. QUANTITATIVE PERFORMANCE EVALUATION A. BEAMFORMING QUALITY
Ultrasound data were respectively acquired from a tissuemimicking phantom (Model 040GSE, CIRS, Norfolk, Virginia, USA) and healthy volunteers. The f s was kept at 40 MHz for every pBF method for a fair comparison under identical system resources. The pre-beamformed RF data were captured by a commercial ultrasound imaging system (SonixTouch, Ultrasonix, Corp., Vancouver, Canada) equipped with a research package (SonixDAQ) using commercial convex, phased, and linear array probes (C5-2/60, PA4-2/20, and L14-5/38). Table 1 shows the detailed specifications of the array transducers employed and imaging specifications applied. From the experimental setup, we assumed the sBF method at target scanline density as a reference for each array transducer (i.e., number of scanlines: 256 for the convex and linear array transducers; 128 for the phased array transducer). This is justified because it would represent an ideal beamforming accuracy when evaluating the pBF method. Quadbeamforming was conducted for the pBF-TS and pBF-sFIFO methods with the reduced number of ultrasound transmission: 64 for the convex and linear array transducers; 32 for the phased array transducer. In the pBF methods, 16-channel ultrasound transmission was used to alleviate the striping artifacts in parallel-scanlines. For the quantitative comparison between parallel-beamforming methods, the quality of the ultrasound images was measured for each array transducer with the peak signal-to-noise ratio (PSNR), which is expressed as where I max is the maximum possible image intensity (e.g., 255 for an 8-bit resolution gray image); MSE is the mean square error, expressed by E[(I ref − I signal ) 2 ], where I ref and I signal are the ultrasound images reconstructed by sBF and pBF methods, respectively. The contrast resolution was also comparatively evaluated using the relative contrast-to-noise ratio (rCNR), given by where CNR sBF and CNR pBF are the calculated contrast-tonoise ratios measured from the specific regions of interest (ROIs) in the ultrasound images reconstructed by the sBF and pBF methods. The rCNR value ranges from 0 to 1 because CNR sBF should be consistently higher than CNR pBF . Each CNR value is given by where µ c and µ s are the mean intensity of the speckle and noise regions, respectively, and σ c and σ s are the standard deviation of the speckle and noise regions, respectively. The bulk power consumption (static + dynamic) was also estimated by using the Vivado software from the aforementioned implementations. 40-MHz f BF was consistently applied for each pBF method to simulate identical user experience in practical diagnostics. Particularly, it required to unlock the system resource limitation in f s , as the pBF-TS method necessitated 80 MHz, 160 MHz, and 320 MHz for dual-, quad-, and octa-beamforming. Since the maximal synthesizable f s was 120 MHz, power consumption at 160 MHz and 320 MHz of f s was extrapolated from measurements at 40 MHz, 80 MHz, and 120 MHz of f s using a quadratic polynomial curve fitting tool in MATLAB software (R2016b, Mathworks, Inc., Massachusetts, MA, USA).

A. PARALLEL-BEAMFORMING QUALITY ANALYSIS
The quad-beamforming accuracy of each pBF method was evaluated with tissue mimicking phantom. Figures 5(a), (b), and (c) show the B-mode ultrasound images by convex, phased, and linear array transducers, respectively reconstructed by sBF, pBF-TS, and pBF-sFIFO methods. Under visual assessment, there are only slight differences between the beamforming methods when using the convex and phased array transducers using 3.5MHz and 2.5MHz of f 0 . However, for the linear array transducer, the pBF-TS method shows a greater level of image quality degradation compared to those of the sBF and pBF-sFIFO methods, as shown in Figure 5(c). This is because the Nyquist theorem could not be met for the signal component centered at 7.5 MHz with ∼70 % fractional bandwidth (2.25MHz -12.75MHz), when f BF is lowered by a factor of four in quad-beamforming (i.e., 10MHz). Figure 6 shows the in vivo experimental results for liver, heart, and thyroid regions of a healthy volunteer when using convex, phased, and linear array transducers, respectively. The in vivo ultrasound images yielded similar trend as presented in the tissue-mimicking phantom experiments under visual assessment. The proposed pBF-sFIFO method shows an image quality comparable to that of the sBF method with same scanline density, regardless of the type of ultrasound array transducers used, whereas the pBF-TS method suffers from the aliasing artifact when using the linear array transducer. In both phantom and in vivo experiments, the residual error images of the pBF methods well agreed with the visual observations ( Figure S2 in the supplementary materials).
To perform a quantitative evaluation, the PSNR and rCNR values were respectively computed for the phantom and in vivo experiments (Table 2). In the phantom experi-80494 VOLUME 8, 2020  ments, the pBF-sFIFO method showed 1.4 dB, 5.0 dB, and 10.0 dB improvements in PSNR compared to those of the pBF-TS method for the convex, phased, and linear array transducers, respectively. The rCNR values from the ROIs in Figure 5 indicated that the pBF-sFIFO method provides 97.8%, 92.5%, and 94.9% of contrast resolution for convex, phased, and linear array transducers, respectively, when compared to the sBF method with same scanline density. These indicate 2.8-%, 2.2-%, and 22.8-% higher contrast when compared to those of the pBF-TS method, providing 95.0%, 90.3%, and 72.1% of rCNR values.
The in vivo studies present the results well agreeing with those presented in the tissue-mimicking phantom study. the PSNR and rCNR values of the pBF-sFIFO method are improved in terms of the PSNR (2.4 dB, 8.4 dB, and 11.8 dB) and rCNR (1.5%, 11.5%, and 16.9%) compared to those of pBF-TS method when using the convex, phased, and linear array transducers, respectively. Table 3 shows the hardware resource utilization in each method. Note that f * BF indicates the beamforming frequency for uncompromised beamforming resolution in the axial direction, preventing image degradation and aliasing artifacts. First of all, the implementation of the pBF-CON method necessitated significantly increased LUT memory for dual-, quad-, and octa-beamforming about 1.0-, 3.0-, and 4.6-fold more when compared to those for pBT-TS and pBF-sFIFO methods. This is because multiple memory components should be allocated for every parallel-beamforming path. On the other hand, even though the pBF-TS method presented the least amount of hardware utilization as sharing a single beamforming path for generating entire parallelscanlines, it necessitates M · f s of f * BF which will significantly increase the dynamic power consumption. On the other hand, the pBF-sFIFO method preserved f s as f * BF with significantly reduced use in RF memory compared to the pBF-CON method, but it utilizes more hardware resources than the pBF-TS method: FF pair, 34%; logic, 39%; LUTs, 38%; register, 3%; and slice, 35% in octa-beamforming. However, the meaningful comparative evaluation among pBF methods would be presented in the following power consumption analysis, as each hardware component consumes power differently at diversified input f s and toggle rates.

B. HARDWARE RESOURCE UTILIZATION AND POWER CONSUMPTION
From the system implmentations in Table 3, bulk power consumptions of the sBF, pBF-CON, pBF-TS and pBF-sFIFO methods at f * BF were measured (Figure 7). The sBF requires 1.85W, whereas pBF-CON method exhibits the proportional increase of power consumption for dual-, quad-, and octa-beamforming: 2.35W, 3.33W, 5.31W. The conventional pBF-TS method with the identical beamforming quality consumes 2.57W, 6.19W, and 20.60W for dual-, quad-, and octa-beamforming, which are with 9%, 86%, and 288% more fractions compared to those in the pBF-CON method. The results confirm that the use of pBF-TS method would be limited when the performance objective aims uncompromised beamforming accuracy and power efficiency for the transducer at high f 0 , which will reduce efficacy of the ultra-compact ultrasound imaging system in general diagnostics. Otherwise, the proposed pBF-sFIFO method necessitates 2.09W, 2.83W, and 4.69W for dual-, quad-, and octa-beamforming, which are 11.06, 15.01, and 11.67% less than those by the pBF-CON method. These power consumptions are only 81.48%, 45.79%, and 22.76% of those in pBF-TS method for dual-, quad-, and octa-beamforming at f * BF , respectively.

VI. DISCUSSION
In this paper, a new pBF-sFIFO method based on a postfractional filtering architecture is presented for ultra-compact ultrasound imaging systems. Based on an identical system resource given (i.e., f s ), the proposed pBF-sFIFO method presents less power consumption than that of the pBF-CON method, and also provides significant improvements in image quality compared to those given by the pBF-TS method ( Figures. 5, 6, 7; Table 2). Also, when aiming to have uncompromised parallel-beamforming accuracy, the pBF-sFIFO method presented significantly lower power consumption than pBF-TS methods (Figure 7).
In the proposed pBF-sFIFO method, there is still room for further optimization of hardware resource utilization and power consumption. For example, using two sFIFO blocks and feeding both at f s /2 of data rate will reduce approximately half of the dynamic power, while block data sets can still be assembled by accessing both of the sFIFO blocks. Also, the summation block implemented by the Vivado software may not have the best timing constraint and power consumption. Alternating them with a carry-save adder followed by a ripple carry adder or a carry lookahead adder will yield superior timing and power efficiency based on its capacity adding multiple binary numbers at the same time. Moreover, using more advanced CMOS technology would provide better bulk power consumption [32]. On the other hand, there could be more strategies dependent to a clinical application. If tissue at superficial depth, where the τ m i [k] is maximized, is out of interest in the target clinical application, the second error-free confinement in subsection III.D can be alleviated, which will proportionally affect power consumption with reduced L sFIFO . For example, in our system configuration, the power consumption in Watt follows 0.003018L 2 sFIFO − 0.002587L sFIFO + 1.681 (R 2 = 1.000; Figure 8a). One may neglect superficial depth ranging from 0 cm to 1 cm for quad-beamforming with linear array transducer. This will reduce L sFIFO from 20 to 17 (Figure 8b), leading to 11.3-% additional saving in power consumption from 2.83 W to 2.51 W. This provides in total 24.62 % and 59.45 % savings from those of pBF-CON and pBF-TS methods, respectively. Therefore, a detailed strategy in target diagnostic applications may further optimize an ultra-compact ultrasound imaging system.
Further works will be progressively conducted for the higher efficacy and applicability of ultra-compact ultrasound imaging systems. An ultra-compact ultrasound imaging system with advanced imaging features (e.g., synthetic aperture imaging, elastography, vector flow imaging, and/or photoacoustic imaging) will be embodied using the proposed pBF-sFIFO method. This extensive research would require an optimized architecture providing the beamforming accuracy comparable to the sBF method, whereas it still alleviates hardware resource utilization and power consumption for better user experiences.
The architecture could be expanded to the volumetric beamforming. Recently, the two-dimensional array is highlighted with its capability to capture the volumetric feature of tissue in real-time, but it suffers from the technical bottleneck in the limited data throughput with the massive amount of RF data received. For this, hybrid beamforming strategies have been proposed (e.g., digital and analog beamforming in lateral and elevation direction), but the precise two-dimensional dynamic beamforming cannot be expected yet because only the limited number of control lines are allowed on an ASIC chip to control thousands of channels [33], [34]. The technological challenges guide us beyond the point-of-care diagnosis; we envisage the ultimate alteration of the analog beamforming part with a digital hardware beamforming solution. This will be groundbreaking innovation for better imaging quality with efficient hardware use and power management, leading to the compact footprint of twodimensional array transducer for premium ultrasound diagnostic applications. For this, we will further generalize the analytic equations, and realize with the optimized system architecture to support the volumetric scanning. As the first step of this expansion, we may use the 32-channel systemon-chip (SOC) solution we recently built for point-of-care ultrasound diagnosis, which can support pBF-sFIFO method (quad-beamforming) with large degree of freedom with onboard delay calculation and quad-chip extension (control up to 128 channels at the same time) [3]. We can expect fourtimes higher volumetric scanning rate using the compact multi-SOC solution in a two-dimensional transducer by taking advantage of its small footprint: 27×27 mm 2 . In addition, it allows a 'chip extension' function inter-connecting up to 4 SOCs, which will enable 16-times higher volumetric scanning rate.
Further improvement can be expected with the hybrid approach with the pBF-TS method. Basically, the strongest advantage of the pBF-TS method is its capability for integration without changing any hardware architecture. Even though the proposed pBF-sFIFO method outperforms pBF-TS method especially when using high f 0 , there would be space available for lowering f BF when employing the transducer at the lower f 0 . For instance, when using 40-MHz f s and 3 MHz f 0 with 100% fractional bandwidth, the transducer only requires an upper band at 4.5 MHz. Therefore, quadbeamforming can be supported since it covers up to 5 MHz frequency band. In this case, the hybrid strategy between pBF-sFIFO and pBF-TS methods will allow up to 4-M parallel-scanline generation, which will be significant benefit for supporting advanced imaging technologies. On the other hand, integrating a compressive beamforming approach to the pBF-sFIFO method may also provide substantial synergy to reduce data rate in dynamic receive beamforming (lowering f * BF ) at minimal image quality degradation [35]- [37]. The integration may result in lower power consumption, enabling an ultra-compact ultrasound imaging system with longer battery life.
The consideration of software implementation would be an interesting point to further investigate. Previously, there was a comprehensive comparison between various architectures for sBF [38], which indicates that the post-fractional filteringbased dynamic receive beamforming method, the basic architecture used in the pBF-sFIFO method, yields the inferior execution time due to more memory access. However, when it comes to the realization of pBF, the memory access could be significantly reduced by the proposed pBF-sFIFO method since the single block data sets fetched in the shared memory can be effectively accessed by multiple parallel-beamforming paths. Therefore, the further study is necessary for evaluating the architectures to support the pBF feature.
The collective evolution of the pBF-sFIFO method will enable further innovative applications beyond conventional diagnostic ultrasound imaging. Photoacoustic imaging is a rapidly emerging modality that can provide optical contrast with ultrasound imaging resolution at deep penetration depth in centimeter scale [39]. There have been several promising progress in science and clinics [40]- [44]. The investigators envision to develop an implantable and/or portable photoacoustic sensor for instantaneous/continuous monitoring of patients with various disorders [45]- [48]. An ultra-compact development of ultrasound receive module would facilitate the translation of the photoacoustic imaging investigations into practice.
In all, we present the efficient pBF-sFIFO method ranging from architectural consideration in practical design; defining theoretical confinements for error-free operation; to phantom and in vivo performance evaluation. The results indicate that the proposed pBF-sFIFO method can outperform the conventional pBF-TS method in terms of efficacy for wider clinical diagnostic applications, especially when there is a need for better image quality with higher M (higher scanline density) and f 0 (higher spatial resolution).