VIRTUAL TERRESTRIAL LASER SCANNER SIMULATOR IN DIGITAL TWIN ENVIRONMENT

We are presenting a Terrestrial Laser Scanner simulator a software device which could be a valuable educational tool for geomatics and engineering students. The main goal of the VirScan3D project is to cover engineering digitisation and will be solved through the development of a virtual system that allows users to create realistic data in the absence of a real measuring device in a modelled real-life environment (digital twin). The prototype implementation of the virtual laser scanner is realised within a game engine, which allows for fast and easy 3D visualisation and navigation. Real-life objects can be digitised, modelled and integrated into the simulator, thus creating a digital copy of a real-world environment. Within this environment, the user can freely navigate and define suitable scanning positions/stations. At each scanning station, a simulated scan is performed which is adapted to the technical specifications of a real scanner. The mathematical solution is based on 3D line intersection with the virtual 3D surface including noise and colour as well as an intensity simulation. As a result, 3D point clouds for each station are generated, which will be further processed for registration and modelling using standard software packages.

Digital Twin in heritage conservation
In heritage education and heritage conservation, the production of as-built surveys in plan and section in representational scales from an overview of 1:100 up to a detail of 1:10 is still a subject that is classically taught in courses across Europe. The strategically set and annotated 2D plans and sections are a stand-in for the real building and its information about materiality and construction. The 2D representations allow a holistic understanding of a historic building, foster the opportunity to display a building research project, apply condition and damage mapping as a further layer and serve as a basis for efficient planning of rehabilitation measures and building monitoring. The use of hand-drawing and total stations in heritage buildings and cultural heritage environments, such as archaeology, is now extended to the production of digital 3D surveys by terrestrial laser scanning and photogrammetry with the same intentions.
3D imaging can be used as a teaching tool, to work with models that represent the reality and must, of course, respect the same requirements and appropriateness for accuracy, resolution and information as the existing formats of the 2D as-built surveying described above (Kravchenko, Luhmann, & Shults, 2016). Here the idea is much likened to the use of digital surrogates or the function of a 'Digital Twin' in 3D.
In industry 4.0, the goal of a Digital Twin is the simulation of functionality processes of machines and sensors and reproduction of these functionalities in a virtual world in order to ensure a flawless accelerated production environment with equivalent reactions and within thresholds for quality control (Gómez-Cabrera, Paez, & Alvarado, 2016;Peruzzini, Pellicciari, & Bil, 2018).
These goals for a Digital Twin are seen quite differently from the perspective of preservation and management of immobile heritage (Jouan & Hallot, 2020). The creation of digitised 3D survey products with added information can now be seen as the Digital Twin of the tangible heritage, which can be connected with rich semantic information and real-time on-site sensors for monitoring to support the life-cycle of heritage buildings and management in line with the sustainable preservation of our built cultural heritage. This is supported by the new way of representing 3D survey models as the database in H-BIM (Historic Building Information Modelling after Dore & Murphy, 2012). Further steps in the connection of survey, H-BIM and heritage management have already taken place (Jouan & Hallot, 2019) and are more and more established in heritage authorities (Bryan, 2017).
In a time, and especially at the time of writing during the Covid-19 pandemic, where e-learning tools have gained an essential significance, and where students are ever more familiar with virtual gaming environments, the teaching of practical capabilities in relation to heritage conservation for as-built survey can take advantage of the opportunity of digitalisation of the activity of teaching (Vlachopoulos & Makri, 2017). The proficiency of the next generation of practitioners in heritage and geomatics in 3D imaging is an educational goal that needs to be included in curricula, such as in the new M.Sc. Digital Technologies in Heritage Conservation in Bamberg university (Hess et al., 2019). Here, a step into a simulator for teaching and learning with the appropriate delivery in reality-like application, resolution and quality outcome is desirable for heritage practitioners.

VirScan 3D scanning simulator as an educational tool
In the areas of environmental planning, building construction, infrastructure maintenance or cultural heritage conservation, the use of digital geospatial information has significantly grown during the last decade, for instance in the field of building information modelling (BIM). Location-based data and geoinformation, in general, are nowadays also a key factor in the economy such as logistics, energy, communication or internet business. Specifically, new instruments for digital recording and processing tools are increasingly established in those application areas.
Since digitalisation is one of the major worldwide trends, the teaching curricula, related to digitisation in engineering practice (e.g. geo-information sciences, cadastral mapping, geodesy) should educate their students according to current needs and demands from the professional market and industry. State-of-the-art technologies in these fields (e.g. 3D laser scanning, photogrammetry, UAVs) lead to large amounts of digital data and are difficult to process, analyse and transform into useful information. For this purpose, appropriate tools and workflows have to be provided, otherwise, neither teachers nor students have realistic chances to adapt to new methods in due time, and they will not be able to contribute to further developments.
However, this objective is hard to achieve, since there is a lack of knowledge among many teachers, out-of-date equipment (hardware and software), insufficient ICT infrastructure and insufficient finances for necessary investments, especially in developing countries. Furthermore, the development of recording instruments has a rapid change cycle of ca. five years. For instance, expensive terrestrial laser scanners are very rare in higher education institutions of those countries. In addition, a strong interdisciplinary approach should be followed since these technologies are relevant in many other subjects such as civil engineering, machine construction, cultural heritage or informatics (Hess, Garside, Nelson, Robson, & Weyrich, 2017).
We are presenting a concept and initial results of a terrestrial laser scanner simulator -a tool, which could be a valuable educational tool for geomatics and engineering students with the aim of experiential learning strategies (Falloon, 2019;Dib, Adamo-Villani, & Garver, 2014).

Objectives for VirScan 3D scanning simulator
The project group of VirScan3D project consists of two German universities and one Ukrainian university based on long-term cooperation that has started in 2002.
The virtual scanner simulates commercially available tools with realistic user interfaces and creates 3D point clouds according to individual specifications and settings in a digital real-world environment. Any digital 3D model (such as a cultural or technical building) that is virtually scanned in a digital environment can be used as a training object. With this system, teachers and students will be able to learn how to manage such systems, generate realistic big data and complete the entire data processing chain. In order to make learning processes more immersive, it is possible to simulate scanning in a virtual twin environment.
A virtual learning environment will solve the problem of providing high-precision (and highly-priced) devices in accordance with professional standards. Especially in geospatial sciences, the simulation in virtual environments can be used for the virtual reconstruction of natural scenes like cultural heritage objects or 3D cities (Kersten & Edler, 2020). It is relevant for the specialists in the fields of geodesy, photogrammetry, GIS and land management as well as neighbouring disciplines (architecture and cultural heritage digitalization, computer vision, mathematics etc.).

Simulator description
As mentioned before, VirScan3D is developed for students of geodetic and other higher education departments with limited access to laser scanning equipment. It allows simulating of all processes of the fieldwork phase of terrestrial laser scanning and generates results that are very close to data generated with real physical scanners on-site. Moreover, the digital environment can simulate real-world objects, i.e. buildings or sites. These results can be used for further processing in scanning software in the office work phase.
The project is developed using readily available resources and tools and it is designed to run on Windows platforms.
The product functionality includes the following features:


Selecting the scanner type which will be simulated;  Laser scanning process simulation with visualisation to obtain point clouds with intensity and noise values according to scanner specifications (noise and intensity not fully implemented yet);  Scan station management which allows to specify settings for each scan station, create and delete stations;  Target management which allows placing and removing targets of different types;  Project management which allows to save the project as a user and to continue work later;  Batch scanning;  Real-time visualisation of generated point clouds;  Exporting the point cloud.
The simulator offers two scenarios of work: 1) The user is navigating within the virtual environment and selects target positions as in real life. Then the scanner position and scan settings are specified and the scanning process of the virtual device is started. This is visualised in real-time. When the scanning is finished the user avatar 'goes' to the next station.
2) The user navigates within the virtual environment and selects positions for all targets and all scan stations. Subsequently, scan settings are specified for all stations or for each station separately. Finally, the scanning simulation is started for all stations one by one.
The result of the simulation is a structured 3D point cloud in ASCII format for import in further software. For each scan station, a single point cloud is generated, hence similar to reality no registration is provided. It is also possible to export target coordinates to use it in registration software for cloud referencing.
VirScan3D is developed within Unreal Engine (a wellknown gaming engine); therefore, user's computers should satisfy its minimum requirements (Unreal, 2020). However, the speed of scan simulation directly depends on computer performance. Therefore, to obtain the speed of simulation no slower than scanning in the real world, the following specifications are recommended: Main functions are programmed with Blueprint Visual Scripting system of Unreal Engine. It allows making the system very flexible and modifying software modules without recompiling the whole code. Some specific modules were programmed in C++ due to Blueprint limitations.
Implementation of advanced features in future versions of software might need the use of additional external software, e.g. Blender (2020).

Digital twin environment integration
In order to create a real-world replica/digital twin within the simulator environment, the existing building should be recreated as a 3D model and integrated. It was decided to use the "Scan to BIM" approach, i.e. perform the 3D scanning of the desired object and model it using Autodesk Revit software. This BIM model can be integrated into Unreal Engine environment using the Datasmith importer plugin, which allows to convert 3D data from a variety of industry-standard design applications into Unreal assets.

Test objects
Initially, the simulator testing was intended to be carried out as part of a summer school for project participants. Thus, the campus buildings of the University of Bamberg have been selected as test objects. The buildings form an ensemble and are the individual monuments in the centre of the UNESCO world heritage in Bamberg.
The building complex with the address 'An der Universität 2' begins its history in the 17 th century as a Carmelite monastery, which is reflected in its three-storey baroque architecture with a courtyard surrounded by three-storey buildings and a church (Fig. 1). After a construction period of almost 50 years, the building was completed in 1742 with the establishment of the theological library (Wied, 2007). In the following, this building will be referred to as 'Baroque building'. Another building -'An der Universität 9' -integrates a former tower of the ancient city wall with its characteristic late medieval sandstone stone blocks and octagonal roof. It has received building extensions and numerous alterations in the 17 th and 19 th century, which can also be seen in the appearance of the building (Fig. 2). In the following, this building will be referred to as 'university office building'.

Data acquisition
The objects have been scanned with high resolution in their real-world environment using Leica BLK360 and Faro Focus 3D scanners. Both trial cases include the elements of different materials, such as sandstone, building stone, tiled roof, glass and steel structures with glass (e.g. the modern library corridor). The details also differ in their colours (Fig. 3a). This has been reflected in the resulting point cloud (Fig. 3b). In the resulting point cloud, the geometries were recorded with a resolution of 1 cm which allows detailing fine elements and decoration in later modelling. Due to the material properties and object colours, different intensity values were obtained. Sometimes the individual elements could not be captured due to over radiation or penetration, which consequently led to the gaps in the point cloud.

Modelling
In the simulator, the object material was represented as an object colour. In the future, the specific material properties will be analysed and integrated into the software (reflection intensity and noise according to specific element material).
In order to generate 3D test objects that are compatible with the simulator software, the recorded buildings have been modelled in Autodesk Revit BIM software. This involves object geometry modelling and assigning materials to object elements. The pre-modelled components available in the Revit library also consist of material properties. This mostly concerns standard Revit elements, such as glass windows and facades, walls and roof materials. For new and unique geometries modelled by the user, the materials were configured manually.
According to the simulator software requirements, the optimal number of model polygons should not exceed 30000 elements.
In addition to the main geometries of objects, the individual elements such as window and door decorations are presented in detail in this case study (Fig. 4). In combination with geometry and object property representation, the level of detail of generated models is equal to LoD 300 according to AIA E202 (Document E202, 2008). The detail precision varies within +/-5 cm.
For the realistic representation, the surrounding objects were also modelled, but their level of detail is much lower (LoD200, Fig. 5). All models have been stored in a Revit format (.rvt) and additionally in IFC format for further import by the Datasmith plugin.

Data integration
As rendering in Unreal Engine is made in real-time several aspects need to be considered during the integration of 3D data: In addition, BIM models include information about the material of elements which is important for intensity generation functions. Figure 5 shows the model of the building with its surroundings integrated into the simulator.
Unique elements of the object might be modelled with the "Model-in-place" feature. However, the same structured approach has to be implemented to consider the correct calculation of collisions.

Experiment -working example of the VirScan3d software
As a prototype for the terrestrial 3D scanners (TLS) that can be chosen in the VirScan3D simulator, we used the Leica BLK 360 laser scanner and an integrated BIM model of one of the historical buildings, the office building described in Section 3.1., in the city of Bamberg. The current simulator version offers four integrated scanner models (Faro X330, Leica Scanstation P40, Leica RTC360 and Leica BLK). In the future, other common laser scanner brands and models on the market will be simulated to extend the training variety.
The process starts with the placing of numbered targets and positioning of scanner stations (Fig. 6). The user can choose either checkboard targets and/or spheres that can be attached to walls and surfaces like in real life. By picking a point in the 3D model the user specifies a scan position. Then the system checks if the station is correct, and saves it.
There are some functional requirements to be noted:  A station is incorrect if it is too close (20 cm) to other objects or inside of other objects;  A station is incorrect if it is located on vertical (±30º) surfaces.
The access to scanner settings is implemented through realistic interfaces of the selected scanner model (example in Fig. 7). The user can select scanning parameters and set the instrument height (height is set to 1.60m by default). It is possible to change the scanning position or remove a station. After the simulated scanning, a 3D point cloud is generated in Cartesian coordinates (X, Y, Z). According to the station settings, an angular grid of rays is projected from the station point. The intersection of the ray with the first surface gives a discrete point, and its XYZ coordinates are stored in a text file. The origin of the coordinate system has been set to the station point. The maximum range of surface search is limited to the scanner specifications. Figure 8 represents an example of a single point cloud, generated by the simulator. It can be seen that point intensities are not fully simulated with the current version of the software; i.e. the material reflectance dependent part of intensity is not implemented yet. As mentioned above, the current prototype version consists of four terrestrial laser scanner models and one 3D environment for simulation. It is expected that release versions of the software will have more models of scanners implemented and a possibility to configure custom scanners. A highly detailed and precise 3D model of the real environment of the University of Bamberg, based on real laser scanning data, was used as a reference environment (see Section 3.1). In future releases, more 3D environments will be accessible. The functionality to place targets is already implemented, which allows users to register resulting point clouds by a target-to-target approach. Also, it is expected that future versions of the simulator will be able to fully simulate intensity and noise.
Since the reference environment shows a realistic and quite complicated example, it is necessary to occupy several scanning locations in order to cover the building surface geometry completely. As in a real-life situation, the roof is the most difficult part to capture in simulation, because of the occlusions and narrow streets surrounding the object of interest. For this reason, some scanning locations were selected at longer distances from the university office building, allowing to scan the roof details as much as possible. During the simulation, 18 stations were selected in order to record the building. The distribution of the simulated scan stations is shown in Figure 9 below.
After the simulation, the 18 scans were exported/saved in a typical industry-standard text file format containing coordinates, intensities and RGB values. At this stage, the simulation part of the fieldwork phase of terrestrial laser scanning is completed and further processing (cleaning, registration, modelling) could be done using any available laser scanning software. In the presented case we used Leica Cyclone software. The 18 simulated datasets were imported and registered by a cloud-to-cloud approach, using the "visual registration" tool for initial alignment. The mean absolute error for enabled constraints results to 0.001 m; while registration errors for individual clouds not exceeding 0.004m. The error histogram for cloud constraint between stations 10 and 11 is shown in Figure  10.  The registration statistics show slightly better results than one can expect from real scanning and cloud-to-cloud registration. One possible explanation could be that simulated data is free of noise, since noise simulation is not implemented yet in the current prototype version of the VirScan3D simulator.
The final registered point cloud was cleaned and exported as (.pts) file. The unusual intensity values are the result of not fully implemented intensity simulation (Fig. 11).
This chapter has shown that the whole sequence of scanning could be simulated in a digital twin environment, thus generating realistic scanning data which later can be processed, registered and used in the entire data processing chain.
The comparison with real data will be conducted during the future 'summer school' application weeks planned for 2021.

Discussion
The recent Covid-19 pandemic has demonstrated that digitalization of teaching is necessary, even in those academic programs where classroom teaching is unavoidable, for example, in practical courses with surveying equipment. VirScan3D, as a new virtual instrument for digitised teaching of terrestrial laser scanning, can be used to create 3D point clouds from different stations including strategic target placement; in a further step, these 3D point clouds have to be registered (and if needed geo-referenced) and processed to higherlevel information such as 3D models, architectural drawings or maps. The virtual 3D scanner emulates commercially available instruments with realistic user interfaces and creates 3D point clouds in a virtual simulation according to individual specifications and settings. Highly detailed and accurate BIM models, which represent real-world objects, can be used as training objects within this environment. With such a virtual system, tutors, as well as students, can learn and prepare to operate those devices, generate close-to-reality large 3D data sets and perform the complete chain of data processing.

Conclusion
We would like to summarise the work presented here by an evaluation and an outlook for further work on the Virscan3D system.

Evaluation
The developed software tool for the simulation of 3D terrestrial laser scanning allows users to reproduce the sequence of actions at the stage of fieldwork and create realistic 3D scanning data in the absence of a real measuring device. Users can plan their fieldwork in the virtual environment, i.e. specify 3D scanner type and target positions or change scanner settings. In addition, users can observe the process of virtual scanning and analyse the results, e.g. check the density of the point cloud and surface coverage of the recorded object.
Summarizing the simulator features, we state that VirScan3D:  Gives a good 3D laser scanning visualisation for a hands-on process explanation to use a terrestrial laser scanning device;  Helps to learn basic terrestrial laser scanning parameters and interface usage, useful for planning of an individual workflow;  Allows the generation of scanning results comparable with real data;  Replaces real equipment for practical exercises in case of lacking financing of expensive equipment;  And is a useful tool for distance learning courses, e-learning and online teaching.

Simulator optimization and further work
Future work will consist of a more realistic simulation of laser intensity based on:  Material-dependent reflection functions resulting as intensity/RGB values;  Estimated beam diameter for the area where the laser beam hits a surface element (laser spot).
Point cloud noise simulation functions are also to be implemented:  Deviations due to distance and angular measurement accuracy;  Range error due to surface reflectivity;  Advanced noise functions for surfaces with specific reflectance (water, glass, mirror etc).
Integration of more scanner models with reality-like user interfaces and specifications, and a variety of laser scanning settings specifying such parameters as that can be defined for each scan station:  Station name;  Field of view;  Resolution;  Quality, etc.
It is also desired that users can load their own 3D model into the virtual environment, which is, however, currently not supported by Unreal Engine.


Creation of a software and hardware product of a real-time system for collecting, accounting and managing data transmission of an intelligent transport system in context of the IoT

Some aspects of the functioning of modern microcontrollers of the Internet of Things in conditions that are close to real-time systems are reviewed in the article. Attention is paid to the aspects of the theory of physical data transmission, the significance of the delay metrics in the data transmission path is substantiated and emphasized. These delays (in view of the generalization of measurement results) determine the control accuracy and controllability of the data processing system as a whole, and also affect the reliability of the results. The process of formalization of the architectural template of a data collecting and accounting system for a control device (which is widespread in intelligent transport systems and the Internet of things) is described. The practical application of the approach in synchronous control through the implementation of a web interface and a control language in a microprocessor system (which can be possible at low traffic transmission rates) is substantiated. In conclusion, a generalization of the scope of application of the developed specialized computer system in transport field is given.

• to develop a specialized computer system for a wireless gateway for an intelligent transport network; • to implement support for the control language, which allows making decisions about control strategies based on current conditions and allows you streamline ESP control in accordance with the needs for some actions, in other words to provide support for adaptive control; • to ensure the collection of data on the controlled operational process and visualize it in the form of graphic means; • to formalize the process control model based on the collected data and rules (criteria), the implementation of which ensures the greatest efficiency and safety.
Thus, the subject of research is a cybernetic system with the functions of selection of information about technological processes and provision of a convenient human-machine interface with an operator, while maintaining the history of the process and implementing automatic control of the process through various information and communication means.

Aspects of functioning of microprocessor systems in systems close to real time, the importance of jitter minimization
Data collection and accounting systems in the IoT are a new direction of automation of production processes and production based on computer technology and specialized software, adapted to the technical features of inexpensive single-chip systems -which are well suited for ensuring the process of collection of information from remote points (objects) for processing, analysis and possible management of remote objects through various web interfaces.
The novelty of the studied topic is explained by the fact that earlier such systems were used exclusively in bulky control systems for automated operational processes. High requirements for hardware content were imposed to such system. Even solutions of industrial Internet of Things (which is a formal model for the development of smart control systems, taking into account the principles of Indutry 4.0) dose not propose it. It is something that shall be covered in more details due to the multi-criteria assessment of the role of the hardware component in context of real-time control systems from the point of the information transmission medium [1].
Studies that touch upon the study of the development of hardware requirements for the implementation of functionally significant components of transmitting modules of cyber-physical systems in most microprocessor-based systems are reduced to determination of temporary connection failures and determination of collisions mediated through time-measured terms such as ping and jitter. In other words through delay variations, such as the most significant (but not the only) categorical requirements that determine the fundamental aspect of the telematic interaction path (see figure 1).

Figure 1. Jitter -characteristics and types for digital transmitters.
Since the communication channels for modern dispatching systems that are very diverse, the choice of a specific traffic management solution depends on the system architecture, the distance between the dispatching unit (MTU) and the RTU (remote workplace), the number of controlled points, they express the bandwidth requirements and reliability of channels, as well as the availability of commercial communication lines.
The trend in the development of traffic management systems as a structural component of SCADA systems can be considered as use not only through the analysis of dedicated communication channels, but also corporate computer networks and specialized industrial networks (industrial buses -CAN, etc.). In modern industrial, electrical and transport systems, industrial networks have gained great popularity -specialized high-speed communication channels that effectively solve the problem of reliability and noise immunity of connections to different hierarchical levels of automation.
Based on the categorical apparatus of the requirements for the SCS being developed, we will rely on the construction of a data model that would directly relate to the aspects (requirements for functioning) of SCADA systems built on various kinds of time specifiers (see figure 2). Based on figure 2, it becomes obvious that it is necessary to develop an architectural solution that implies a minimum (adequately) available data transmission rate to achieve minimum values of time delays, and as a consequence, a decrease in the curve of loss of packages with data in real-time systems within the framework of the IoT concept.
The best jitter performance was demonstrated in the exchamge pathway by IEEE 802.11 (Supportive Technology for Information Exchange) for packages with a total rate of 2 Kbps (0.014 ms for a 30 minute session). For certain reasons these values are undoubtedly insufficient (although there is a tendency to use LoRaWAN as a reference technology for information transfer), for full-fledged data collection systems, however, they suggest earlier assumptions regarding finding the minimum available and adequate values for the requirements to compliance with the rules for the functioning of real-time systems [2].
To implement the operation of the software and hardware platform and architectural solution, an ESP8266 (ESP-12F) microprocessor module operating in IEEE 802.11n has been selected. This module has the ability to operate in the telematic mode at 9600 baud /s, which is equal to 7.68 kb/s.

Master data in software and hardware system and data management tool for a developed specialized computer system
The attractiveness of maintaining a low bandwidth for the developed analytical platform is due to the fact that existing solutions (microprocessor systems) for the Internet of Things, firstly, cannot provide a high rate of operation of integrated transceivers, and secondly, they are aimed at use in conditions of low consumption [3]. The designed developed system had requirements for working through a web interface, via HTTP, and implementing an API for remote and local management.
The database has been implemented through the master data templates (MDM), which are widely used in business systems and intelligent transport systems.  [4].
The purpose of master data management is to make sure that there is no incomplete and inconsistent data in real time by simplifying the data transmission system in the external loop (it is necessary to store all data in the device memory, and transfer only those that are necessary for exchange session (separate metrics for specific (set by the operator remotely) scales from peripheral devices): sensors and actuators, which simplifies the use of a specialized computer system in various fields of activity and informatization facilities.
The Master Data Management template (figure 3) on the one hand shall provide customization for working with data on a specific object, and on the other hand shall be independent from the specifics of the application industry, the scope of the product built on its basis and its information environment [5]. The SCADA software developed by IDEArduino is based on the C language for the ESP-12F, which has an IEEE 802.11 module.
The control language of the collection and dispatching system is implemented both through a direct control interface and a web server implemented via HTTP, as well as through an AT construct formalized through the BS-88 control language, which is defined by the ISA-88 standard. One control command takes 512 bytes of traffic.
The current edition of the control standard includes the necessary organizational and technical standards used in SCADA systems:  The control is based on the regulation of the discrete (D) and analogue (A) ports, the reading itself is realized through hardware interrupts.
The developed SCADA integrates a full-fledged automated control through an integrated approach to the definition of master data and flow control of discrete Input /Output lines in the configuration mode ("SETUP)" (see figure 4). Let's turn on $ D3 for 45 seconds using the syntax of the control language (in order to do it the designated input must be set to a positive one): 7: $D3 = 1; if ($ timer> = 45) trans {8}; // On this line, the algorithm seems to 'get stuck' (go into a controlled interrupt) until the condition is met 8: $D3 = 0; Such control structure is not like a usual device programming language, where one statement is executed at each step [6]. Note: the system provides one more statement -the "stop" statement, which stops the execution of the script. However, in this case, the AUTO field will remain enabled on the panel.
Special variables (functions) One such variable -$ timer, has been reviewed but there are others. $ntime -[n] ow [time] -current time; used as a floating point number 13.4 means 13 hours and 40 minutes $random -random number [0: 1] (used to simulate life in an empty house) $morning -start of the day for mid-latitudes (used to turn off the lights) $evening -sunset time (used to turn on the lights) $retro -timer value that timer has reached in the previous phase $temp -current temperature value from ds18b20 temperature sensor connected to D5 $ptime -time in seconds from the beginning of the scan period, equal to T_scan.


Gemmini: Enabling Systematic Deep-Learning Architecture Evaluation via Full-Stack Integration

DNN accelerators are often developed and evaluated in isolation without considering the cross-stack, system-level effects in real-world environments. This makes it difficult to appreciate the impact of System-on-Chip (SoC) resource contention, OS overheads, and programming-stack inefficiencies on overall performance/energy-efficiency. To address this challenge, we present Gemmini, an open-source*, full-stack DNN accelerator generator. Gemmini generates a wide design-space of efficient ASIC accelerators from a flexible architectural template, together with flexible programming stacks and full SoCs with shared resources that capture system-level effects. Gemmini-generated accelerators have also been fabricated, delivering up to three orders-of-magnitude speedups over high-performance CPUs on various DNN benchmarks. * https://github.com/ucb-bar/gemmini

I. INTRODUCTION
Deep neural networks (DNNs) have gained major interest in recent years in application domains ranging from computer vision, to machine translation, to robotic manipulation. However, running modern, accurate DNNs with high performance and low energy consumption is often challenging without dedicated accelerators which are difficult and expensive to design. The demand for cheaper, high-productivity hardware design has motivated a number of research efforts to develop highly-parameterized and modular hardware generators for DNN accelerators and other hardware building blocks [1]- [7]. While the hardware generator efforts make it easier to instantiate a DNN accelerator, they primarily focus on the design of the accelerator component itself, rather than taking into consideration the systemlevel parameters that determine the overall SoC and the full software stack. Some industry perspectives have advocated for a more holistic exploration of DNN accelerator development and deployment [8]- [10]. However, existing DNN generators have little support for a fullstack programming interface which provides both high and low-level control of the accelerator, and little support for full SoC integration, making it challenging to evaluate system-level implications.
In this work, we present Gemmini, an open-source, full-stack DNN accelerator generator for DNN workloads, enabling end-to-end, fullstack implementation and evaluation of custom hardware accelerator systems for rapidly evolving DNN workloads. Gemmini's hardware template and parameterization allows users to tune the hardware design options across a broad spectrum spanning performance, efficiency, and extensibility. Unlike existing DNN accelerator generators that focus on standalone accelerators, Gemmini also provides a complete solution spanning both the hardware and software stack, and a complete SoC integration that is compatible with the RISC-V ecosystem. In addition, Gemmini implements a multi-level software stack with an easy-to-use programming interface to support different programming requirements, as well as tight integration with Linuxcapable SoCs which enable the execution of any arbitrary software.
Gemmini-generated accelerators have been successfully fabricated in both TSMC 16nm FinFET and Intel 22nm FinFET Low Power (22FFL) process technologies, demonstrating that they can be physically realized. In addition, our evaluation shows that Gemminigenerated accelerators deliver comparable performance to a stateof-the-art, commercial DNN accelerator [11] with a similar set of hardware configurations and achieve up to 2,670x speedup with respect to a baseline CPU. Gemmini's fully-integrated, full-stack flow enables users to co-design the accelerator, application, and system all at once, opening up new research opportunities for future DL SoC integration. Specifically, in our Gemmini-enabled case studies, we demonstrate how designers can use Gemmini to optimize virtual address translation mechanisms for DNN accelerator workloads, and to partition memory resources in a way that balances the different compute requirements of different layer types within a DNN. In brief, this work makes the following contributions: 1) We build Gemmini, an open-source, full-stack DNN accelerator design infrastructure to enable systematic evaluation of deeplearning architectures. Specifically, Gemmini provides a flexible hardware template, a multi-layered software stack, and an integrated SoC environment (Section III). 2) We perform rigorous evaluation of Gemmini-generated accelerators using FPGA-based performance measurement and commercial ASIC synthesis flows for performance and efficiency analysis. Our evaluation demonstrates that Gemmini-generated accelerators deliver comparable performance compared to stateof-the-art, commercial DNN accelerators (Section IV). 3) We demonstrate that the Gemmini infrastructure enables system-accelerator co-design of SoCs running DNN workloads, including the design of efficient virtual-address translation schemes for DNN accelerators and the provisioning of memory resources in a shared cache hierarchy (Section V).

II. BACKGROUND AND MOTIVATION
The demand for fast and efficient DNN execution from edge to cloud has led to a significant effort in developing novel accelerator instances that are specialized for different DNN algorithms and/or different deployment scenarios. This section discusses recent advances  in DNN accelerators and DNN accelerator generators, motivating the need for a full-stack approach to evaluate deep learning architectures.

A. DNN Accelerators
Researchers have proposed a large variety of novel DNN accelerators with different performance and energy efficiency targets for different applications across a diverse set of deployment scenarios [1], [12]- [14]. At the architecture level, different DNN accelerators exploit different reuse patterns to build specialized memory hierarchies [15] and interconnect networks [16] to improve performance and energy efficiency. Most existing hardware DNN architectures are largely spatial, where parallel execution units are laid out spatially either in a systolic fashion, as in the case of the TPU, or in parallel vector units like Brainwave [17] and NVDLA [11]. Based on these architectural templates, recent advances have also started exploring how to leverage applications' sparsity patterns [18]- [20] and/or emerging in-memory computing technology [21], [22].

B. DNN Accelerator Generators
Recent research has designed hardware generators for DNN accelerators [1]- [3], [11], [16], [23]. Table I compares the different features supported by existing hardware generators compared to Gemmini. In contrast to building specific instances of hardware, generatorbased approaches provide parameterizable architectural templates that can generate a wide variety of hardware and software instances, improving hardware design productivity. Here, we discuss the hardware, software, and system level requirements for DNN accelerator generators to enable full-stack, systematic DNN architecture evaluation.
DNN accelerator generators must provide flexible architectural templates to cover a wide variety of different DNN accelerator instances, each suited for a different execution environment and a different area/power/performance target. Most DNN accelerator generators today focus only on fixed-point representations and/or only support a single dataflow. In addition, today's generators only target a specific spatial array type, i.e., systolic-based (as in the TPU) or vector-based (as in NVDLA), making it challenging to systematically compare against them. In contrast, Gemmini supports 1) both floating and fixed point data types to handle data representations in training and inference, 2) multiple dataflows that can be configured at design time and run time, 3) both vector and systolic spatial array architectures, enabling quantitative comparison of their efficiency and scalability differences, and 4) direct execution of different DNN operators.
Moreover, DNN accelerator generators also need to provide an easy-to-use programming interface so that end users can quickly program their applications for the generated accelerators. Different developers would prefer different software design environments based upon their targets or research interests. For example, DNN application practitioners would prefer that the hardware programming environment be hidden by DNN development frameworks like PyTorch or TVM so that they don't need to worry about low-level development details, as in the case of VTA [1] and DNNWeaver [23]. At the same time, framework developers and system programmers may want to interact with the hardware at a low level, in either C/C++ or assembly, to accurately control hardware states and squeeze every bit of efficiency out, as in the case of MAGNet [2] and Maeri [16]. Unlike other DNN generators that tend to focus on one of these requirements, Gemmini provides a multi-level programming interface to satisfy users with different requirements. In addition, Gemmini is the first infrastructure that provides hardware support for virtual memory without the need for any special driver software, making it significantly easier for end-users to program accelerators.
Third, system-level integration, including both the SoC and the system software, is also critical in DNN accelerator generators. Today's DNN accelerators are typically designed and evaluated in isolation. However, when they are eventually deployed, they need to be integrated as part of a larger system. In fact, recent industry evaluations have demonstrated that modern ML workloads could spend as much as 77% of their time running on CPUs, even in the presence of a hardware accelerator, to execute either new operators or to move data between the CPU and accelerators [8]- [10], [24]. However, unfortunately, none of the existing DNN accelerator generators support full SoC integration with host CPUs and shared resources like caches and system buses. Motivated by this observation, Gemmini has built-in system integration support where users can directly instantiate a complete SoC environment that can boot Linux, directly enabling architects to evaluate subtle trade-offs at the system level.

III. GEMMINI GENERATOR
Gemmini is an open-source, full-stack generator of DNN accelerators, spanning across different hardware architectures, programming interfaces, and system integration options. With Gemmini, users can generate everything from low-power edge accelerators to highperformance cloud accelerators equipped with out-of-order CPUs. Users can then investigate how the hardware, SoC, OS, and software overhead interact to affect overall performance and efficiency. with spatially distributed processing elements (PEs), each of which performs dot products and accumulations. The spatial array reads data from a local, explicitly managed scratchpad of banked SRAMs, while it writes results to a local accumulator storage with a higher bitwidth than the inputs. Gemmini also supports other commonly-used DNN kernels, e.g., pooling, non-linear activations (ReLU or ReLU6), and matrix-scalar multiplications, through a set of configurable, peripheral circuitry. Gemmini-generated accelerators can also be integrated with a RISC-V host CPU to program and configure accelerators. We design Gemmini's spatial array with a two-level hierarchy to provide a flexible template for different microarchitecture structures, as demonstrated in Figure 2. The spatial array is first composed of tiles, where tiles are connected via explicit pipeline registers. Each of the individual tiles can be further broken down into an array of PEs, where PEs in the same tile are connected combinationally without pipeline registers. Each PE performs a single multiply-accumulate (MAC) operation every cycle, using either the weight-or the outputstationary dataflow. The tiles are composed of rectangular arrays of PEs, where PEs in the same tile are connected combinationally with no pipeline registers in between them. The spatial array, likewise, is composed of a rectangular array of tiles, but each tile does have pipeline registers between it and its neighbors. Every PE and every tile shares inputs and outputs only with its adjacent neighbors. Figure 3 illustrates how Gemmini's two-level hierarchy provides the flexibility to support anything from fully-pipelined TPU-like architectures to NVDLA-like parallel vector engines where PEs are combinationally joined together to form MAC reduction trees, or any other design points in between these two extremes. We synthesized both designs with 256 PEs. We found that the TPU-like design achieves a 2.7x higher maximum frequency, due to its shorter MAC chains, but consumes 1.8x as much area as the NVDLA-like design, and 3.0x as much power, due to its pipeline registers. With Gemmini, designers can explore such footprint vs. scalability trade-offs across different accelerator designs.

B. Programming Support
The Gemmini generator produces not just a hardware stack, but also a tuned software stack, boosting developers' productivity as they explore different hardware instantiations. Specifically, Gemmini provides a multi-level software flow to support different programming scenarios. At the high level, Gemmini contains a push-button software flow which reads DNN descriptions in the ONNX file format and generates software binaries that will run them, mapping as many kernels as possible onto the Gemmini-generated accelerator. Alternatively, at the low level, the generated accelerator can also be programmed through C/C++ APIs, with tuned functions for common DNN kernels. These functions must be tuned differently for different hardware instantiations in order to achieve high performance, based on scratchpad sizes and other parameters. Therefore, every time a new accelerator is produced, Gemmini also generates an accompanying header file containing various parameters, e.g. the dimensions of the spatial array, the dataflows supported, and the compute blocks that are included (such as pooling, im2col, or transposition blocks).
Data Staging and Mapping: At runtime, based on the dimensions of a layer's inputs, and the hardware parameters of the accelerator instantiation, Gemmini uses heuristics to maximize the amount of data moved into the scratchpad per iteration. Gemmini calculates loop tile sizes at runtime, and these tile sizes determine when and how much data is moved between DRAM, L2, and scratchpad during the execution of our tiled matrix multiplication, convolution, residualaddition, etc. kernels. If the programmer wishes, the low-level API also allows them to manually set tile-sizes for each kernel.
Virtual Memory Support: In addition to the programming interface, Gemmini also makes it easier to program accelerators by providing virtual memory support. This is useful for programmers who wish to avoid manual address translations as well as for researchers who wish to investigate virtual memory support in modern accelerators. Gemmini also enables users to co-design and profile their own virtual address translation system. For example, Figure 4 shows the miss rate of an example accelerator's local TLB profiled on Gemmini. As we can see, the miss rate occasionally climbs to 20-30% of recent requests, due to the tiled nature of DNN workloads, which is orders-of-magnitude greater than the TLB miss rates recorded in prior CPU non-DNN benchmarks [25]. Later, in Section V-A, we use Gemmini to co-design a virtual address translation system which achieves near-maximum end-to-end performance on accelerated DNN workloads, with only a few TLB entries in total.

C. System Support
Gemmini allows architects to integrate RISC-V CPUs with Gemmini-generated accelerators in the Chipyard [26] framework. These can range from simple, in-order microcontrollers which are not expected to do much more than IO management, all the way up to outof-order, high-performance, server-class CPUs that may be running multiple compute-intensive applications even as they are sending commands to the Gemmini-generated accelerator. SoCs can also be configured to host multiple host CPUs and Gemmini-generated accelerators, which can each operate on different tasks in parallel with each other. Figure 5 is one example of a dual-core system, where each CPU has its own Gemmini-generated accelerator. Additional SoClevel parameters include bus widths between accelerators and host CPUs, as well as the size, associativity and hierarchy of the caches in the multicore, multicache memory system. Later, in Section V-B, we show how these parameters can be tuned, based on the computational characteristics of DNNs, to improve performance by over 8%.
RISC-V-based full SoC integration also enables deep softwarestack support, such that Gemmini-generated accelerators can easily be evaluated running the full software stack up to and including the operating system itself. This enables early exploration of accelerated workloads in a realistic environment where context switches, page table evictions, and other unexpected events can happen at any time. These unexpected events can uncover bugs and inefficiencies that a "baremetal" environment would not bring to the surface. For example, our experience of running Linux while offloading DNN kernels to a Gemmini-generated accelerator uncovered a non-deterministic deadlock that would only occur if context switches happened at very particular, inopportune times. Running on a full software stack with an OS also uncovered certain bugs where Gemmini read from certain regions of physical memory without the proper permissions. On a "baremetal" environment, these violations were silently ignored.

IV. GEMMINI EVALUATION
This section discusses our evaluation methodology and evaluation results of Gemmini-generated accelerators compared to both CPUs and state-of-the-art, commercial accelerators.

A. Evaluation Methodology
We evaluate the end-to-end performance of Gemmini-generated accelerators using the FireSim FPGA-accelerated simulation platform [27]. We evaluate five popular DNNs: ResNet50, AlexNet, SqueezeNet v1.1, MobileNetV2, and BERT. All DNNs are evaluated with a full Linux environment on a complete cycle-exact simulated SoC. We synthesize designs using Cadence Genus with the Intel 22nm FFL process technology and place-and-route them using Cadence Innovus. Our layout and area breakdown, described in Figure 6, show that the SRAMs alone consume 67.1% of the accelerator's total area. The spatial array itself only consumes 11.3%, while the host CPU consumed a higher 16.6% of area.

B. Performance Results
We evaluated the performance of several Gemmini configurations, with different host CPUs and different "optional" compute blocks, to determine how the accelerator and host CPU configuration may interact to impact end-to-end performance. In particular, we evaluated two host CPUs: a low-power in-order Rocket core, and a highperformance out-of-order BOOM core. We used two different Gemmini configurations: one without an optional im2col block, and the  other with an im2col block which allowed the accelerator to perform im2col on-the-fly, relieving the host CPU of that burden. As illustrated in Figure 7, when the accelerator is built without an on-the-fly im2col unit, its performance depends heavily on the host-CPU which becomes responsible for performing im2col during CNN inference. A larger out-of-order BOOM host CPU increases performance by 2.0x across all CNNs. The less complex the DNN accelerator is, the more the computational burden is shifted onto the CPU, giving the host CPU a larger impact on end-to-end performance.
However, when the accelerator is equipped with an on-the-fly im2col unit, the choice of host CPU is far less important, because the CPU's computational burden is shifted further onto the accelerator. Adding a small amount of complexity to the accelerator allows us to reduce the area and complexity of the host CPU to a simple inorder core while preserving performance. Gemmini enables hardware designers to easily make these performance-efficiency tradeoffs.
With the on-the-fly im2col unit and a simple in-order Rocket CPU, Gemmini achieves 22.8 frames per second (FPS) for ResNet50 inference when running at 1 GHz, which is a 2,670x speedup over the in-order Rocket CPU and an 1,130x speedup over the outof-order BOOM CPU. The accelerator also achieves 79.3 FPS on AlexNet. Some DNN models such as MobileNet are not efficiently mapped to spatial accelerators due to the low data reuse within the depthwise convolution layers. Therefore, Gemmini demonstrates only a 127x speedup compared to the Rocket host CPU on MobileNetV2, reaching 18.7 FPS at 1GHz. On SqueezeNet, which was designed to be run efficiently on modern CPUs while conserving memory bandwidth, Gemmini still demonstrates a 1,760x speedup over the Rocket host CPU. Our results are comparable to other accelerators, such as NVDLA, when running with the same number of PEs as the configuration in Figure 6a. When running language models such as BERT, Gemmini achieves a 144x improvement over the Rocket CPU.

V. GEMMINI CASE STUDIES
This section demonstrates how Gemmini enables full system codesign with two case studies. We use Gemmini to design a novel virtual address translation scheme, and to find the optimal SoC-level resource partition scheme of a multi-core, multi-accelerator system.

A. Virtual Address Translation
With an RTL-level implementation that supports virtual memory, users can co-design their own virtual address translation schemes based on their accelerator and SoC configuration. Prior works in virtual address translation for DNN accelerators have proposed very different translation schemes, from NeuMMU [28], which calls for a highly parallel address-translation system with 128 page-table walkers (PTWs), to Cong et al. [29], who recommend a more modest two-level TLB hierarchy, with the host CPU's default PTW co-opted to serve requests by the accelerator. This lack of convergence in the prior literature motivates a platform that allows co-design and design-space exploration of the accelerator SoC together with its virtual address translation system, for both hardware designers and researchers. Fortunately, with Gemmini, we can iterate over a variety of address translation schemes as we tune the accelerator and SoC.
To demonstrate, we configure Gemmini to produce a two-level TLB cache, with one private TLB for the accelerator, and one larger shared TLB at the L2 cache that the private TLB falls back on when it misses. Our design includes only one PTW, shared by both the CPU and the accelerator, which is suitable for low-power devices. We configure the accelerator for low-power edge devices, with a 16by-16 systolic mesh and a 256 KB scratchpad. As shown in Figure 8a, we iterate over a variety of TLB sizes to find the design that best balances TLB overhead and overall performance, including over a design point where the shared L2 TLB has zero entries. Figure 8a demonstrates that the private accelerator TLB has a far greater impact on end-to-end performance than the much larger shared L2 TLB. Increasing the private TLB size from just four to 16 improves performance by up to 11%. However, adding even 512 entries to the L2 TLB never improves performance by more than 8%. This is because our workloads exhibit high page locality; even with tiled workloads, our private TLB's hit rate remained above 84%, even with the smallest TLB sizes we evaluated. In fact, we found that 87% of consecutive read TLB requests, and 83% of consecutive write TLB requests, were made to the same page number, demonstrating high page locality. However, because reads and writes were overlapped, read and write operations could evict each other's recent TLB entries.
Although tuning TLB sizes improves hit rates, our private TLB hit latency in the tests shown in Figure 8a was still several cycles long. Fortunately, using the Gemmini platform, we were able to implement a simple optimization: a single register that caches the last TLB hit for read operations, and another register that caches TLB hits for write operations. These two registers allow the DMA to "skip" the TLB request if two consecutive requests are made to the same virtual page number, and help reduce the possibility of read-write contention over the TLB. These "filter registers" reduce the TLB hit latency to 0 cycles for consecutive accesses to the same page. As Figure 8b shows, this low-cost optimization significantly improves our end-toend performance, especially for small private TLB sizes. Due to our high TLB hit rate and low TLB hit penalty, we found that a very small 4-entry private TLB equipped with filter registers, but without an expensive shared L2 TLB, achieved only 2% less than the maximum performance recorded. With such a configuration, the private TLB hit rate (including hits on the filter registers) reached 90% and further increases to either TLB's size improved performance by less than 2%, even if hundreds of new TLB entries were added.
Using Gemmini, we have demonstrated that a modest virtual address translation system, with very small private TLBs, a single page-table-walker, and two low-cost filter registers for the TLB, can achieve near maximum performance for low-power edge devices. Gemmini is designed to enable such co-design of the SoC and its various components, such as its virtual address translation system.

B. System-Level Resource Partition
Gemmini also enables application-system co-design for real-world DNN workloads. To demonstrate, we present a case study describing a system-level design decision: memory partitioning based on application characteristics. We investigate memory partitioning strategies in both single-core and multi-core SoCs.
Real-world DNN applications, such as CNN inference, have diverse layer types which have different computational requirements and which contend for resources on an SoC in different ways. For example, ResNet50 includes convolutions, matrix multiplications, and residual additions, which all exhibit quite different computational patterns. Convolutions have high arithmetic intensity; matrix multiplications have less; and residual additions have almost no data re-use at all. Additionally, unlike the other two types of layers, residual additions benefit most if layer outputs can be stored inside the cache hierarchy for a long time, rather than being evicted by intermediate layers, before finally being consumed several layers later. These different layer characteristics suggest different ideal SoC configurations. To run with optimal performance over an entire DNN, a hardware designer must balance all these constraints.
To demonstrate, we run ResNet50 inference on six different SoC configurations. These are the three different configurations described in Figure 9a, repeated for both single-and dual-core SoCs (as in Figure 5), where each CPU core has its own Gemmini-generated accelerator. The dual-core SoCs run two ResNet50 workloads in parallel, while the single-core SoCs run just one. The base design point has a 256 KB scratchpad, and a 256 KB accumulator per core, as well as a 1 MB shared L2 cache. The scratchpad and accumulator memories are private to the accelerators, but the L2 cache is shared by all CPUs and accelerators on the SoC. We presume that we have 1 MB of extra SRAM that we can allocate to our memory system, but we need to decide whether to allocate these SRAMs to the accelerators' private memory, or to the L2 caches.
As shown in Figures 9b and 9c, convolutional layers benefit from a larger, explicitly managed scratchpad, due to their very high arithmetic intensity. Convolutional kernels exhibit a 10% speedup with one core, and an 8% speedup in the dual-core case, when the scratchpad and accumulator memory is doubled by the addition of our 1 MB worth of SRAMs. The matmul layers, on the other Fig. 9: Performance of the various SoC configurations in the case study, normalized to the performance of the Base configuration.
hand, achieve only a 1% and 3% speedup when the scratchpad is enlarged in the single-core and dual-core cases respectively, due to their lower arithmetic intensity. Residual additions, which have virtually no data re-use and are memory-bound operations, exhibit no speedup when increasing the scratchpad memory size. Instead, they exhibit a minor 1%-4% slowdown, due to increased cache thrashing. In the single-core case, the increased convolutional and matrix multiplication performance is enough to make the design point with increased scratchpad memory, rather than increased L2 memory, the most performant design point.
However, Figure 9c shows that when we run dual-process applications that compete for the same shared L2 cache, allocating the extra 1 MB of memory to the shared L2 cache improves overall performance more than adding that memory to the accelerators' scratchpad and accumulator memories. Increasing the scratchpad size still improves convolutional performance more than increasing the L2 size, but this improvement in performance is more than negated by the 22% speedup of residual additions that the dual-core BigL2 design point enjoys. This is because each core's residual addition evicts the input layer that the other one is expecting from the shared L2 cache, increasing the latency of memory-bound residual addition layers. The dual-core BigL2 configuration, which increases the shared cache sizes, alleviates this contention, reducing the L2 miss rate by 7.1% over the full ResNet50 run, and increasing overall performance by 8.0%. The BigSP configuration, on the other hand, improves overall performance by only 4.2% in the dual-core case.
With Gemmini, we have demonstrated how the memory partitioning strategy, a key component of system-level design, can be decided based upon application characteristics, such as the composition of layer types and the number of simultaneous running processes.

VI. CONCLUSION
We present Gemmini, a full-stack, open-source generator of DNN accelerators that enables systematic evaluations of DNN accelerator architectures. Gemmini leverages a flexible architectural template to capture different flavors of DNN accelerator architectures. In addition, Gemmini provides a push-button, high-level software flow to boost programmers' productivity. Finally, Gemmini generates a full SoC that runs real-world software stacks including operating systems, to enable system architects to evaluate system-level impacts. Our evaluation shows that Gemmini-generated accelerators demonstrate high performance efficiency, and our case studies show how accelerator designers and system architects can use Gemmini to co-design and evaluate system-level behavior in emerging applications.


s2fieldsofstudy:
Computer Science
Environmental Science

text:
Multi-Layout Invoice Document Dataset (MIDD): A Dataset for Named Entity Recognition

: The day-to-day working of an organization produces a massive volume of unstructured data in the form of invoices, legal contracts, mortgage processing forms, and many more. Organizations can utilize the insights concealed in such unstructured documents for their operational beneﬁt. However, analyzing and extracting insights from such numerous and complex unstructured documents is a tedious task. Hence, the research in this area is encouraging the development of novel frameworks and tools that can automate the key information extraction from unstructured documents. However, the availability of standard, best-quality, and annotated unstructured document datasets is a serious challenge for accomplishing the goal of extracting key information from unstructured documents. This work expedites the researcher’s task by providing a high-quality, highly diverse, multi-layout, and annotated invoice documents dataset for extracting key information from unstructured documents. Researchers can use the proposed dataset for layout-independent unstructured invoice document processing and to develop an artiﬁcial intelligence (AI)-based tool to identify and extract named entities in the invoice documents. Our dataset includes 630 invoice document PDFs with four different layouts collected from diverse suppliers. As far as we know, our invoice dataset is the only openly available dataset comprising high-quality, highly diverse, multi-layout, and annotated invoice documents. which the dataset is made available (CC-BY-4.0).

Summary
Forbes statistics [1] state that the amount of data produced through daily transactions is very high, and 80% of this daily generated data is unstructured. Unstructured data include Portable Document Formats (PDF), emails, official letters, and many more.
Unstructured data are a valuable asset to the organization as they have a lot of information hidden in them. If an organization extracts these key insights and uses them for the decision-making process, it can significantly increase its operational efficiency [2]. However, manual processing and extracting key insights from such numerous and complex unstructured documents is naturally time-consuming and error-prone. Hence, developing an artificial intelligence (AI)-enabled tool for automatic key information extraction from unstructured data is a promising and upcoming research focus [3]. However, automatic extraction of key insights from unstructured document research faces certain key challenges [4,5]. One of the most fundamental and critical challenges is to obtain a high-quality, Data 2021, 6, 78 2 of 10 standard, and annotated unstructured document dataset. The dataset is the foremost important entity for machine learning model training. Model robustness and accuracy depend on its learning from the training data. Therefore, it is always necessary to include variations in training data so that the model learns to recognize the unknown data. Standard datasets or publicly available datasets in this research face key challenges discussed as follows: • Publicly available datasets consist of poor-quality, blurred, skewed, and low-resolution document images, leading to poor text extraction [6]. • Publicly available datasets are obsolete, consisting of the old or obsolete formats of documents [7].

•
Publicly available datasets are domain-specific and task-specific. For example, the dataset proposed in [8,9] is used for the healthcare domain, and the dataset proposed in [10,11] is used for the legal contract analysis domain. In addition, they are used for specific tasks such as metadata extraction from scientific articles [12] or patient detail extraction from the clinical dataset [13].

•
Publicly available datasets are unlabelled. Therefore, manual labeling or annotating the dataset is time-consuming and tedious [4,5].
Due to these challenges, a few research studies proposed a custom dataset. However, the custom datasets are private and face confidentiality issues [4,14]. The custom dataset also includes documents with similar layouts or formats. Providing similar layout documents restricts the generalizability of key extraction tasks. Few recent studies highlight the significance of the template-free processing of unstructured documents [15,16]. Therefore, to encourage and advance automatic key information extraction research, we developed a dataset. Figure 1 shows an overview of key information extraction tasks from the invoice document. Our multi-layout invoice document dataset (MIDD) dataset contains 630 invoices with four different layouts of different suppliers. The dataset is of high-quality document images, which leads to high accuracy in text extraction. The dataset also helps to generalize the AI-enabled model as it comprises varied and complex layouts of documents. accuracy depend on its learning from the training data. Therefore, it is always necessary to include variations in training data so that the model learns to recognize the unknown data. Standard datasets or publicly available datasets in this research face key challenges discussed as follows:  Publicly available datasets consist of poor-quality, blurred, skewed, and low-resolution document images, leading to poor text extraction [6].  Publicly available datasets are obsolete, consisting of the old or obsolete formats of documents [7].  Publicly available datasets are domain-specific and task-specific. For example, the dataset proposed in [8,9] is used for the healthcare domain, and the dataset proposed in [10,11] is used for the legal contract analysis domain. In addition, they are used for specific tasks such as metadata extraction from scientific articles [12] or patient detail extraction from the clinical dataset [13].  Publicly available datasets are unlabelled. Therefore, manual labeling or annotating the dataset is time-consuming and tedious [4,5].
Due to these challenges, a few research studies proposed a custom dataset. However, the custom datasets are private and face confidentiality issues [4,14]. The custom dataset also includes documents with similar layouts or formats. Providing similar layout documents restricts the generalizability of key extraction tasks. Few recent studies highlight the significance of the template-free processing of unstructured documents [15,16]. Therefore, to encourage and advance automatic key information extraction research, we developed a dataset. Figure 1 shows an overview of key information extraction tasks from the invoice document. Our multi-layout invoice document dataset (MIDD) dataset contains 630 invoices with four different layouts of different suppliers. The dataset is of high-quality document images, which leads to high accuracy in text extraction. The dataset also helps to generalize the AI-enabled model as it comprises varied and complex layouts of documents. Table 1 summarizes our MIDD dataset specifications.    • To provide the annotated and varied invoice layout documents in IOB format to identify and extract named entities (named entity recognition) from the invoice documents to the researchers working in this domain. Obtaining a high-quality and sufficient annotated corpus for automated information extraction from unstructured documents is the biggest challenge researchers face.

•
To overcome the limitations of rule-based and template-based named entity extraction from unstructured documents traditionally used so far in information extraction approaches. Template-free processing is the only key to processing, and managing a huge pile of unstructured documents in the recent digitized era.

•
To provide varied invoice layouts so that researchers can develop a generalized AI-based model that will train on various unstructured invoice layouts. Obtained structured output can later be utilized for integrating into information management application of the organization and used for the decision-making process.

Related Datasets
The research study [6,[16][17][18] proposed key field extraction from a scanned receipts dataset named the ICDAR (International Conference on Document Analysis and Recognition) SROIE-2019 dataset. It has 1000 scanned receipt images with similar layouts, including 876 annotated receipts with labels such as the name of a company, address, date of receipt, and total amount. The research study [19] used the RVL-CDIP dataset that includes scanned document images of different categories, including invoices as one of the categories. It has 25,000 images of every category. However, the dataset is obsolete and of poor-quality scanned documents.
A few research studies [20,21] built a custom invoice dataset for key field extraction tasks. However, these datasets are not publicly available to researchers due to privacy and confidentiality issues in invoice documents.
A few research studies [8][9][10]12] proposed information extraction tasks on various domain-specific and task-specific datasets such as the I2b2 2010 (Informatics for Integrating Biology and the Bedside) clinical notes dataset, the MIMIC-III (Medical Information Mart for Intensive Care) dataset, the custom-built legal contract document dataset, and the GROTOAP2 dataset.

Data Description
To the best of our knowledge, our dataset is the first publicly available multi-layout invoice document dataset. The proposed MIDD dataset includes invoices of different layouts collected from different supplier organizations. Developing generalizability and model robustness are the main aims of collecting the highly diverse and complex invoice layouts. In addition, researchers may use training and testing samples from the dataset as per their requirements. Table 2 represents the details of varied layouts and the total number of invoice document PDFs collected for each layout.   Figure 2 shows the detailed process of our multi-layout invoice document dataset creation.   Figure 2 shows the detailed process of our multi-layout invoice document dataset creation.

Data Acquisition
Invoices from different supplier organizations are acquired to obtain variations in layouts of invoices. All the supplier invoices are scanned using HP Laserjet M1005 MFP

Data Acquisition
Invoices from different supplier organizations are acquired to obtain variations in layouts of invoices. All the supplier invoices are scanned using HP Laserjet M1005 MFP Printer and Scanner (HP: Pune, India) in PDF Format. Each supplier has its unique and own layout or format of their invoices. The varied and multiple layouts of invoices later will help researchers in the template-free processing of various unstructured documents. As shown in Table 2. four different layout invoices from different supplier organizations Data 2021, 6, 78 5 of 10 are collected. The number of scanned invoice PDFs for layout 1 is 196, layout 2 is 29, layout 3 is 14, and layout four is 391. Thus, our dataset has 630 total scanned invoice PDFs. Figure 2 illustrates that, as scanned PDFs of invoices are collected, each of these scanned PDFs is converted into an image as the initial step. Conversion from PDF to image is required as any optical character recognition (OCR) engine takes the image as its input for text detection and extraction. Later, all the corresponding text files obtained after OCR are input for the data annotation tool. The Google Vision OCR (Version 2.3.2) engine is used for text extraction, and the UBIAI annotation tool is used for manual annotations. Google Vision OCR output is validated manually by cross verifying the extracted text and original text contents. Roughly 30% of invoices from each invoice layout were manually verified and validated against original scanned PDF contents for evaluating OCR accuracy. As a result, it was observed that Google Vision OCR gives 90% accuracy in text extraction for each supplier invoice layout. The MIDD dataset was also evaluated using AI approaches such as BiLSTM and BiLSTM-CRF for key fields extraction tasks [22]. Figure 3 shows the sample invoices of all four different layouts from our dataset. Different key fields are colored with different colors to understand the meaning of "different layouts of invoices." For example, the invoice number key field shown with the green rectangular surround box is positioned at different locations in all four layouts of invoices collected from different suppliers. Likewise, all other invoice key fields take a different position as the supplier organization changes. Thus, our dataset is useful for dealing with a real-world situation where organizations have their own unique format or layout of unstructured documents. Table 3 summarizes more statistical information on the proposed dataset. Invoices collected for MIDD belong to the construction firm having different suppliers for different material purchases.   Table 3 summarizes more statistical information on the proposed dataset. Invoices collected for MIDD belong to the construction firm having different suppliers for different material purchases.

Data (Named Entities) Annotation
Converted text files are manually annotated. UBIAI is a convenient and simple-to-use text labeling/annotation tool used for most Natural Language Processing (NLP) tasks such as named entity recognition (NER). By selecting the "free version" of the UBIAI package under the "Package" option, all decided labels for invoice entities ( for example, INV_L for invoice label and INV_DT for actual Invoice Date value) to annotate the invoice document are supplied through its interface manually. After providing labels, the converted text files are inputted manually either by the "drag and drop" facility or the "browse" facility of the UBIAI interface. After all the text files are inputted, they can be annotated by choosing labels individually and highlighting the respective text within a complete text file, as shown in Figure 4. This also provides an annotation file export facility in multiple formats such as spacy, IOB, and JSON. As shown in Figure 2, the text files obtained after OCR are provided to UBIAI with the labels to be annotated. Table 2 shows the name of the labels used to annotate the text files. Finally, the annotated text file of each invoice in IOB format is exported. IOB labels are like part-of-speech (POS) labels, but they signify the inside, outside and beginning of a word. In NER, every word in the text file, also called "token," is labeled with an IOB label, and then adjacent tokens are joined together depending on their labels. Later IOB files are transformed into a .csv file. Figure 5 shows the sample IOB file of Data 2021, 6, 78 7 of 10 the invoice. The first column represents the "token," and the second column represents the IOB tags of the respective tokens. complete text file, as shown in Figure 4. This also provides an annotation file export facility in multiple formats such as spacy, IOB, and JSON. As shown in Figure 2, the text files obtained after OCR are provided to UBIAI with the labels to be annotated. Table 2 shows the name of the labels used to annotate the text files. Finally, the annotated text file of each invoice in IOB format is exported. IOB labels are like part-of-speech (POS) labels, but they signify the inside, outside and beginning of a word. In NER, every word in the text file, also called "token," is labeled with an IOB label, and then adjacent tokens are joined together depending on their labels. Later IOB files are transformed into a .csv file. Figure 5 shows the sample IOB file of the invoice. The first column represents the "token," and the second column represents the IOB tags of the respective tokens.   Figure 6 shows the data pre-processing steps carried out to obtain the pre-processed data.  Figure 6 shows the data pre-processing steps carried out to obtain the pre-processed data.

Data Pre-Processing
Data 2021, 6, x FOR PEER REVIEW 9 of 11 Figure 6. Data pre-processing steps.
Stop-word removal: stop-words like English articles (the, an, a) and conjunction (or, and) are removed from the data. Stop-words take up memory size and require processing time. In addition, stop words have no contribution to the meaning of a sentence, so these stop-words are removed from the data.
Lowercasing: all the tokens or words are converted into lowercase to get the consistent output.
Duplicate and blank-row removal: all the duplicate rows and blank rows are manually removed from the .csv files using the "Remove rows" option in Microsoft Excel.
Non-alphanumeric characters' removal: few non-alphanumeric characters like "(" are removed from the data files. Few non-alphanumeric characters like ":" are kept in the data files as they are part of some fields like Invoice Date.

Practical Applications/Use-Cases of MIDD
Currently, available literature mainly focuses on information extraction from receipts with a similar format or layout. However, in a practical scenario, organizations receive invoices from various suppliers having their unique structure or layout. Publicly available datasets lack an invoice document dataset which has varied invoice layouts.


The proposed MIDD dataset has many practical implications for extracting named entities as a structured output from the huge pile of unstructured invoice documents. In addition, end-to-end automation of invoice information extraction workflow helps the accounting department in every organization for quick invoice processing and to verify accounts payable and receivable.  Automated key field extraction from financial documents such as invoices impacts the performance of the business by customer onboarding and verification processes. It can reduce significantly the cost employed for manual data entry and verification of thousands of daily received invoices.

Conclusions
The proposed work developed a multi-layout invoice document dataset consisting of 630 invoices with four different layouts from different supplier organizations. It contributes to this research by providing a highly diverse, high-quality, and annotated dataset, which is very useful in Natural Language Processing tasks such as named entity recognition (NER). The dataset size and contents are sufficient to build a generalized AI- Stop-word removal: stop-words like English articles (the, an, a) and conjunction (or, and) are removed from the data. Stop-words take up memory size and require processing time. In addition, stop words have no contribution to the meaning of a sentence, so these stop-words are removed from the data.
Lowercasing: all the tokens or words are converted into lowercase to get the consistent output.
Duplicate and blank-row removal: all the duplicate rows and blank rows are manually removed from the .csv files using the "Remove rows" option in Microsoft Excel.
Non-alphanumeric characters' removal: few non-alphanumeric characters like "(" are removed from the data files. Few non-alphanumeric characters like ":" are kept in the data files as they are part of some fields like Invoice Date.

Practical Applications/Use-Cases of MIDD
Currently, available literature mainly focuses on information extraction from receipts with a similar format or layout. However, in a practical scenario, organizations receive invoices from various suppliers having their unique structure or layout. Publicly available datasets lack an invoice document dataset which has varied invoice layouts.

•
The proposed MIDD dataset has many practical implications for extracting named entities as a structured output from the huge pile of unstructured invoice documents. In addition, end-to-end automation of invoice information extraction workflow helps the accounting department in every organization for quick invoice processing and to verify accounts payable and receivable.

•
Automated key field extraction from financial documents such as invoices impacts the performance of the business by customer onboarding and verification processes. It can reduce significantly the cost employed for manual data entry and verification of thousands of daily received invoices.

Conclusions
The proposed work developed a multi-layout invoice document dataset consisting of 630 invoices with four different layouts from different supplier organizations. It contributes to this research by providing a highly diverse, high-quality, and annotated dataset, which is