Watermarking Pre-trained Encoders in Contrastive Learning

Contrastive learning has become a popular technique to pre-train image encoders, which could be used to build various downstream classification models in an efficient way. This process requires a large amount of data and computation resources. Hence, the pre-trained encoders are an important intellectual property that needs to be carefully protected. It is challenging to migrate existing watermarking techniques from the classification tasks to the contrastive learning scenario, as the owner of the encoder lacks the knowledge of the downstream tasks which will be developed from the encoder in the future. We propose the \textit{first} watermarking methodology for the pre-trained encoders. We introduce a task-agnostic loss function to effectively embed into the encoder a backdoor as the watermark. This backdoor can still exist in any downstream models transferred from the encoder. Extensive evaluations over different contrastive learning algorithms, datasets, and downstream tasks indicate our watermarks exhibit high effectiveness and robustness against different adversarial operations.

Introduction
Contrastive learning [Chen et al., 2020a;Chen et al., 2020b;Caron et al., 2020;Chen and He, 2020;Grill et al., 2020;Zbontar et al., 2021] has demonstrated huge potential for unsupervised learning recently. Its main idea is to use a large number of unlabeled data to pre-train an encoder for effective feature representation, which could be further used to build different types of downstream classifiers with limited labeled data. Such approaches can be performed over noisy and uncurated data which can get rid of the expensive data labeling efforts [Carlini and Terzis, 2021]. Moreover, contrastive learning has achieved better performance than classic supervised learning in many tasks. For instance, SimCLR [Chen et al., 2020a], a popular contrastive learning algorithm, has been proved to have a better performance in an ImageNet classification task than various supervised learning algorithms. * Contact Author A typical contrastive learning pipeline consists of two components, i.e. pre-training encoders and building downstream classifiers. As the most important component, the pre-trained encoders are usually generated with many unlabeled data and a large number of training costs (e.g., GPU resources). Moreover, for special tasks like medical diagnose, collecting critical training data is also a costly process. As a result, welltrained encoders are very valuable since they can be applied to many downstream classification tasks with a simple finetuning process and a small number of labeled data. Selling pre-trained encoders has become a popular business model 1 , and these encoders are an important intellectual property (IP) for model owners. It is necessary to protect these models from being abused, illegal copy, and redistribution.
One promising technology to protect the IP of a DNN model is watermarking [Adi et al., 2018;Zhang et al., 2018;Fan et al., 2019;. A quantity of works have designed watermarking solutions for conventional classification models. By embedding a backdoor into the protected classification model during the training process, the model owner can easily verify the ownership of any suspicious model via remote query using the specific triggered samples. However, it is challenging to directly apply a similar approach to watermark a pre-trained encoder from contrastive learning due to two reasons. First, when embedding the watermarks, the owner of the pre-trained encoder does not know the specific downstream classification tasks that will be built from it, as well as the corresponding dataset. Therefore, it is difficult for the owner to craft backdoor and verification samples. Second, during verification, the encoder owner can only query the downstream model to check if it is built from his watermarked encoder. He can only obtain the final output from the classification layer, rather than the feature representation from his encoder. This also hinders the owner from checking the existence of a backdoor.
To address these challenges, this work proposes the first watermarking methodology for the IP protection of the pretrained encoders in contrastive learning. Our solution can achieve task-agnostic, i.e., the owner does not need any prior information about the downstream models and datasets, and the embedded watermark is verifiable for arbitrary downstream tasks. Particularly, our watermark is also based on the backdoor technique. Instead of manipulating the verification samples to have unique predicted labels, we aim to make them have unique feature representations from the encoder, which can naturally lead to unique predicted labels for any subsequent downstream models. Specifically, we introduce a loss function to fine-tune the model for watermark embedding, which can make its output of samples with a speciallydesigned trigger deviate a lot from the output of a normal encoder. When the adversary illegally obtains this encoder and trains a classification model from it, the owner can use verification samples (i.e., normal samples with the trigger) to query the model. When its label is different from the normal case (i.e., query results of the same normal sample), the owner has confidence to identify this as a plagiarization.
We extensively evaluate our watermarking method on different popular datasets (STL10, GTSRB, and SVHN) with two representative contrastive learning algorithms (SimCLR [Chen et al., 2020a] and MocoV2 [Chen et al., 2020b]). The results indicate that our proposed technique is effective at distinguishing plagiarized models from independent ones regardless of the downstream tasks. It will not affect the functionality of the encoder and any downstream models. Besides, the embedded watermark exhibits strong robustness against different adversarial operations (e.g., fine-tuning, pruning), making it hard to be removed.

Contrastive Learning
The aim of contrastive learning is to pre-train an encoder by using a huge amount of unlabeled images. Specifically, for each image ϕ, the contrastive learning algorithm first generates a positive sample ϕ + which is transformed from ϕ, as well as a negative sample ϕ − , which is transformed from a different sample. We expect the outputs f (ϕ) and f (ϕ + ) to be as similar as possible, since ϕ and ϕ + are from the same class. In addition, we also need to maximize the difference between f (ϕ) and f (ϕ − ), as they are likely to be in different classes. By pursuing the above optimization objective, the encoder can be trained with unlabeled samples to accurately predict the feature representation of any image from any category. Below we briefly introduce two representative contrastive learning algorithms. SimCLR [Chen et al., 2020a]. This algorithm uses common data augmentation operations (random crop, Gaussian blur, random flip, etc.) to generate positive and negative samples. For a N -image batch, SimCLR generates 2N images (x i , (i = 1, ..., 2N )) by applying data augmentation twice to the samples in it. Two transformed images form a positive pair if they are originated from the same image, or a negative one otherwise. To achieve the goal of contrastive learning, a contrastive loss is defined as below: where (z i , z j ) is the positive pair of feature vectors produced by the encoder (here z i = f (x i )), while (z i , z k ) is the negative one. sim(·, ·) is the cosine similarity between the two feature vectors. τ is a temperature parameter used to scale the cosine similarity and exp is the natural exponential function. If the feature vectors of the positive pair (z i , z j ) are more similar and the ones of the rest negative pairs (z i , z k ) are more different, the loss value will be smaller. Therefore, we need to update the encoder parameters to minimize the above loss function. SimCLR sums up the loss of all positive pairs from the batch as the final loss and learns the image encoder via minimizing this final loss.
MocoV2 [Chen et al., 2020b]. The core of this algorithm is to introduce a dictionary of feature vectors as a queue so as to reduce the memory cost in the training process by reusing the feature vectors from the immediate mini-batches. To this end, MocoV2 uses two encoders called query encoder (f q ) and momentum encoder (f k ). Given a batch of N input images, f q produces N feature vectors q and f k produces N feature vectors k + by applying different data augmentations on the input images. The contrastive loss is then subsequently calculate as below: where k i is the feature vectors of previous batches. τ is the same as in SimCLR. Usually, the queue has a much bigger size than the mini-batch. After calculating the contrastive loss, the feature vectors produced by the momentum encoder (k) are then pushed into the queue and one of the earliest batches is deleted simultaneously. MocoV2 updates the parameters of its two encoders in different ways. The parameters in the query encoder are updated by the back-propagation algorithm according to the immediate contrastive loss. The parameters in the momentum encoder are updated by the following Equation: where θ k is the old parameters of the momentum encoder, θ q is the immediate parameters of the query encoder, m is a hyper-parameter to control the updating speed, and θ k is the updated parameters of the momentum encoder.

Watermarking
Watermarking DL Models. The general goal of watermarking is to protect the IP of a DNN model. The most popular watermarking solution leverages the DNN backdoor technique. A standard watermarking solution consists of two phases . In the first phase, the model owner employs a watermark embedding algorithm to inject a backdoor into the target DL model to get a watermarked classifier. This watermarked classifier maintains the functionality for normal samples while giving unique labels for some carefully-crafted samples (i.e., verification samples). In the second phase, the model owner attempts to verify whether a suspicious classifier contains his watermark. He uses the verification samples to query this classifier. By checking the corresponding responses, he can identify the ownership of this model with high confidence. Threat model. We consider a model owner M who pretrains an encoder f and sells it to some users for building downstream tasks. However, a model plagiarist P gets this encoder in an unauthorized way (e.g., model stealing, illegal redistribution, etc.). How the adversary obtains the encoder is beyond the scope of this paper.
Then the plagiarist uses f to build his own downstream classifier. He may slightly modify the encoder (e.g., finetuning, pruning). We assume the adversary does not have enough resources to alter the encoder completely. Otherwise, he will train the encoder directly without stealing it.
The owner's goal is to detect whether a suspicious classifier is built from his encoder. The model owner has only oracle access to this classifier, i.e., he can send arbitrary inputs to F s and receive the corresponding outputs. His strategy is to embed a watermark into f . When the classifier is from f , the classification output will be different from the ones output by an independent model. When embedding the watermark, the owner does not have any knowledge about the possible downstream tasks and datasets. The embedded watermark must be robust enough and unremovable by the possible model transformations from the adversary. Watermark Requirements. Specifically, a good watermarking solution should satisfy the following properties: An effective watermark should meet several requirements to guarantee its usability. In our scenario, the watermark must have characters narrated as below: • Uniqueness. This means the classification model from the watermarked encoder will give unique output for the verification samples different from that of an independent model. This is the basic requirement to guarantee the effectiveness of the watermarking scheme. • Functionality-preserving. The watermarking algorithm should have a negligible impact on the performance of the downstream model to preserve its functionality. • Robustness. A successful watermark should be robust against potential attacks from the plagiarist, e.g., model modification via fine-tuning and pruning. Once embedded, the watermark is hard to be removed by the plagiarist.

Overview
The most popular way to watermark a DL model is the DNN backdoor technique [Adi et al., 2018;Zhang et al., 2018].
In this paper, we also follow this idea for protecting the pretrained encoders. Prior works have introduced backdoor attacks against pre-trained feature encoders [Jia et al., 2022] and language foundation models . However, these attacks require the adversary to have knowledge of downstream tasks and training sets. Therefore, they are not applicable to our scenario, where the encoder owner does not know anything about the downstream tasks and aims to protect the IP of any models developed from his pre-trained encoder.
To this end, we design a new task-agnostic watermarking solution for pre-trained encoders. Figure 1 shows an overview of our methodology. The key insight is that instead of embedding a targeted backdoor into the encoder that forces its downstream model to assign a specific label to the triggered samples, we can consider an untargeted backdoor, causing the downstream model to output any labels that are different from the correct ones for the triggered samples. In this case, the encoder owner is still able to verify the ownership by checking whether the model output of the triggered samples is incorrect. Meanwhile, he can achieve such untargeted backdoor embedding without any knowledge of the downstream tasks. Below we detail our methodology.

Watermark Embedding
Formally, we consider a pre-trained encoder f , which takes as input an image, and outputs its corresponding feature. The owner aims to convert f into a watermarked version f , which can satisfy the properties of uniqueness, functionalitypreserving, and robustness. The owner has access to the unlabeled training set of the encoder: D = {x 1 , x 2 , ..., x n )}.
To embed the watermark into f , the owner pre-defines a trigger pattern t and a trigger mask m to denote its position. Then for each sample x i ∈ D, he can calculate the corresponding triggered sample as in Eq. 4.
We craft the following loss function for the owner to finetune f for watermark embedding. Uniqueness. To guarantee uniqueness, we need to ensure the downstream models developed from a normal encoder (e.g., f ) and watermarked encoder (f ) have distinct predictions for each triggered sample x t i . As the owner does not know the downstream tasks and datasets, he cannot design loss functions to restrict the downstream model predictions, as did in [Jia et al., 2022]. Instead, he can try to maximize the difference of the encoders' output (i.e., features), which could subsequently maximize the difference of the downstream model's output (i.e., labels) with very high probability. This leads to the loss term in Eq. 5.
where SIM(·, ·) measures the similarity between two feature vectors. In this paper, we adopt the cosine similarity in this loss term. Other similarity metrics can be applied as well.
Our goal is to minimize L u , which can increase the feature distance between normal and watermarked encoders over triggered samples. Functionality-preserving. We also need to guarantee the embedded watermark does not affect the prediction accuracy of any inherited downstream models over clean samples. Similarly, since the encoder owner does not know the downstream tasks, he can try to make the output feature of f as close as that of the normal encoder f for clean samples, which will result in comparable performance in the subsequent downstream tasks. This gives us another loss term for functionality-preserving in Eq. 6.
Based on the above analysis, the watermark embedding process can be formulated as an optimization problem as where η is the hyper-parameter to balance the properties of uniqueness and functionality-preserving. In this paper, we adopt the gradient descent method to solve the above optimization problem. We initialize f as the clean encoder f , and iteratively calculate the loss and update f to reach the optimal values. Robustness. As backdoor attacks generally exhibit high robustness against common model transformations (e.g., model pruning, fine-tuning), the corresponding watermarks are expected to enjoy similar robustness as well. We validate this conclusion in Section 4. In order to further enhance the watermark robustness in the encoder, we propose to adopt dropout during the embedding process, e.g., randomly dropping some neurons in each layer. This can simulate the impact of downstream model transfer learning and post-processing, which makes our watermarks more immune to these operations.

Watermark Verification
Given a suspicious downstream model M , the encoder owner wants to verify whether M is developed from f without IP authorization. To achieve this, he first constructs a set of data samplesD corresponding to this downstream task (e.g., via downloading from the internet). For each samplex i ∈D, he feeds it to M and obtains the predicted label. Then he calculates the corresponding verification samplex t i following Equation 4 and uses it to query M again. He checks whether the two predictions are different. If the ratio of samples getting different labels in the two cases is higher than a pre-defined threshold T , then the owner has confidence to confirm model M violates the IP of his encoder f . This process is formulated as in Eq. 8.
where function I returns 1 when the inside condition is true, or 0 otherwise.

Evaluation
We perform extensive evaluations to demonstrate our solution can achieve the desired watermark requirements. Contrastive learning methods. Our solution is general for the encoders pre-trained from different contrastive learning methods. In this paper, we adopt SimCLR [Chen et al., 2020a] and MocoV2 [Chen et al., 2020b], two of the most popular contrastive learning techniques for experimentation. We choose ResNet18 and ResNet50 as the base models for SimCLR and MocoV2, respectively.

Experimental Setup
Datasets. We use different types of datasets to pre-train the encoders and build the downstream classifiers: • CIFAR-10: it contains 60,000 images belonging to 10 classes. Each image has a size of 32 × 32 × 3. We use 50,000 samples for training and 10,000 samples for testing. • ImageNet: it contains 14,000,000 images belonging to 1,000 classes. Each picture has a size of 224 × 224 × 3. • STL10: this dataset contains 113,000 images with the size of 96 × 96 × 3. It has 5,000 training images and 8,000 test images with labels (10 classes). Besides, it also has 100,000 unlabeled images for unsupervised learning. • GTSRB: This dataset contains 51,800 images of 43 different traffic signs. Each image has a size of 32 × 32 × 3. • SVHN: it contains over 600,000 images of the digit house numbers from Google Street View. Each image has a size of 32 × 32 × 3. It is divided into a training set of 73,257 images, a test set of 26,032 images, and an extra training set of 531,131 images. All the images belong to 10 classes corresponding to the digits from '0' to '9'. In our experiment, we adopt CIFAR-10 and ImageNet for the pre-trained SimCLR and MocoV2 2 encoders, respectively. Then, we transfer the encoders to different downstream tasks for recognizing STL10, GTSRB, and SVHN samples. Backdoor triggers. Our watermark method has no assumption on the triggers plant on samples because it does not rely on a special trigger to verify. Hence, in our experiments, we adopt the checkerboard trigger of size 10 × 10 located on the bottom-right corner of images, which is similar to the Badencoder [Gu et al., 2019]. Specifically, We use white square, green square, and green cross as the triggers in our experiment. Examples are in Figure 2: the first row represents clean samples and the second row contains triggered ones. Note that similar results will be get for different triggers. Implementation. When embedding watermarks into the encoders, we adopt a batch size of 128 for SimCLR and train the encoder for 300 epochs. For MocoV2, we use a batch size of 64 and train the encoder for 50 epochs. For both training processes, we set the learning rate as 0.001 and η as 0.5. We use Pytorch 1.10 backend for the implementation. We conduct the experiments on a server equipped with Intel I9-11900K CPU and 2 NVIDIA GeForce RTX 3090 GPUs. Possible attacks. We consider two techniques including finetuning and pruning that the attacker may deploy to change the models for erasing the watermark. They are commonly adopted in the previous evaluation of watermark robustness [Adi et al., 2018;Zhang et al., 2018;.
• Fine-tuning. We consider two types of fine-tuning methods in our experiments: Re-Train Last Layers (RTLL) and Fine-Tune All Layers (FTAL). Note that we do not consider the operation of Fine-Tune Last Layer (FTLL) since the encoder only produces the feature vectors and the users always re-train the last layers for building downstream classifiers. We also do not consider Re-Train All Layers (RTAL), which makes the pre-training meaningless. • Pruning. We evaluate two pruning methods: random pruning which randomly removes some parameters in each layer and L1-pruning which removes the parameters with the smallest L1-norms. Metrics. We adopt the following two metrics to evaluate the effectiveness of our watermark. • Test Accuracy (ACC). This is to calculate the prediction accuracy of the downstream classifiers over clean samples. This metric is used to measure the functionality-preserving requirement of the watermark: a downstream classifier built from a watermarked encoder should have a tiny ACC loss compared to the one built from a clean encoder. • Watermark Accuracy (WACC). This is to measure the ratio of verification samples' prediction labels by the suspicious model different from the prediction labels of the cor-responding clean samples (Eq. 8). This metric can evaluate the uniqueness requirement: the WACC of a watermarked downstream classifier should be much higher than an independent model which can be easily separated by a threshold T . Besides, WACC can also reflect the robustness requirement. Under different attacks, the WACC of the classifier built from a watermarked pre-trained encoder should be always higher than T .

Uniqueness Analysis
To show the uniqueness of our watermark, we carry out our experiment by evaluating the WACC for two aspects. First, we use a set of triggered samples to query the clean model and the corresponding downstream classifier built from the watermarked pre-trained encoder respectively. Second, we use a set of randomly designed wrong triggered samples to test a watermarked downstream classifier. The evaluation results are shown in Table 1 and  The results indicate that our method can embed an efficient watermark into the clean pre-trained encoder and achieve high uniqueness. As shown in Table 1, our watermarking method can effectively distinguish the clean and watermarked model by giving significantly different WACC for triggered samples. Also, the difference between the AC of the 'correct' trigger and the AC of the 'wrong' trigger is also prominent. This means for wrong triggered samples, our watermarked model will give a very low WACC for a successful IP verification with uniqueness guaranteed.

Robustness analysis
To test the Robustness of our watermark, we calculate ACC and WACC of our watermarked model which is processed with fine-tuning or pruning by the attackers. The results are shown in Table 3 and Table 4.
From Table 3 we can observe that the WACC keeps at a high level with the pruning amount increasing when the pruning process still has a slight impact on ACC. When the pruning ratio is beyond 0.8, there is an immediate drop in both   ACC and WACC of the watermarked models. This indicates that a very large pruning ratio can mitigate the watermarking method but also significantly compromise the model's functionality. Such a high pruning ratio is pointless for an attacker since a broken model is useless. Thus, we conclude that our watermarking method is robust against model pruning. Table 4 shows the ACC and WACC of the models finetuned by RTLL and FTAL. The WACC has only a slight drop when the model is fine-tuned which indicates that our watermarking method is robust against model fine-tuning.

Performance-preserving analysis
We measure the performance-preserving requirement by comparing the ACC of the clean model and the watermarked one. The results are shown in Table 5  It turns out that our watermark embedding algorithm preserves the functionality of the pre-trained encoder. We can discover for Table 5 that the ACC of watermarked models is similar or even larger than that of clean models in most cases. We analyze that an even larger ACC with watermarking is due to the phenomenon that training on noisy data gives significant robustness improvements pointed in [Radford et al., 2021]. The noise brought by the triggered samples in the training dataset will not harm the model ACC which proves the robustness of our method.

Related Works
Numerous watermarking schemes have been proposed for conventional DNN models. They can be roughly classified into the following two categories: White-box solutions. This strategy adopts redundant bits as watermarks and embeds them into the model parameters. For instance, [Uchida et al., 2017] introduced a parameter regularizer to embed a bit-vector (e.g. signature) into model parameters which can guarantee the performance of the watermarked model. [Rouhani et al., 2019] found that implanting watermarks into model parameters directly could affect their static properties (e.g histogram). Thus, they injected watermarks in the probability density function of the activation sets of the DNN layers. These methods require the owner to have white-box access to the model parameters during the watermark extraction and verification phase, which can significantly limit the possible usage scenarios. Black-box solutions. This strategy takes a set of unique sample-label pairs as watermarks and embeds their correlation into DNN models. For example, [Le Merrer et al., 2019] [Li et al., 2019] generated watermark samples that are almost indistinguishable from normal samples to avoid detection by adversaries.  designed temporal state sequences to watermark reinforcement learning models. [Xiaoxuan et al., 2021] utilized cache side channels to verify watermarks in the model architecture.
In addition to watermarking, an alternative approach to IP protection is model fingerprinting. The main difference is that the model owner crafts special samples to verify the model without making any modifications to it. DNN Fingerprint. Some works proposed fingerprinting schemes for classification models. They leveraged adversarial examples as fingerprints to identify the target models. For example, IPGuard [Cao et al., 2019] identified the data samples close to the target model's decision boundary to fingerprint this model. [Lukas et al., 2019] adopted conferrable adversarial examples with certain labels, which can transfer from the target model to its surrogates while remaining ineffective to other non-target models. [He et al., 2019] designed sensitive samples to fingerprint black-box models, which will fail even the model has very small modifications.


The practice of developing the academic cloud using the Proxmox VE platform

. Cloud technologies provide users with efficient and secure tools for data management, computing, storage and other services. The article analyzes the projects for the introduction of cloud technologies in education and identifies the main advantages and risks in creating a cloud infrastructure for the university. Such startups contribute to the formation of a new paradigm of education. It involves the virtualization of education, the introduction of mobile and blended learning, ie the combination of cloud computing with modern learning concepts. In this paper, we highlight our experience in improving the academic cloud for the training of a bachelor’s degree in computer science. This is through the integration of the Proxmox VE platform into existing computing power by deploying the Proxmox VE system. In the study, we reveal some technical and methodological aspects of the organization of the educational process using this corporate cloud platform. The scheme of the organization of physical components of cloud infrastructure (nodes, virtual networks, routers, domain controller, VPN server, backup system of students’ virtual machines) is given. All characteristics of this environment and possibilities of their application are studied.

Introduction
Cloud computing and open access systems in education are technologically and cost-effective means of organizing the operation of digital systems without the need to spend large sums of money on the development of IT infrastructure. Therefore, over the past 10 years, cloud computing has become a popular paradigm in education. However, educational institutions in Ukraine have some problems such as limited technical and material resources of IT infrastructure, limited teaching staff, lack of technical experts and qualified IT staff. In this article, we continue to share our experience of using cloud platforms to provide virtualization of computing resources, such as operating systems, servers, storage devices, or network resources. This approach should improve the digital learning environment of our university students.
Cloud computing is an important alternative to the classical local model of organization of the educational environment of the university. Thanks to them, students and administrative staff have the opportunity to quickly and easily access various application platforms and resources through web pages on demand. This reduces organizational costs and offers more powerful functionality. In particular, such cost reductions apply to licenses for software, hardware, electricity, accommodation and maintenance. As a result, IT professionals are relieved of responsibility for maintaining certain services and repositories at the university. The use of cloud infrastructure redundancy eliminates the risks of disaster recovery and the high costs associated with it. Cloud services often offer new tools and applications to improve the performance and availability of digital environments. All this provides flexibility in decision-making to the management of each university.
Among the shortcomings of cloud computing is the lack of some services that may be needed by teachers and students. As a result, there are problems and restrictions on cloud application offerings, availability and service level agreements. Security issues should be singled out.
Currently, Ukrainian universities have some problems in organizing educational environments. They can be summarized as follows.
• Existing digital resources are distributed among educational institutions not on the basis of their needs, but on the basis of personal reasons or justification. This leads to unbalanced development of free economic zones. • Limited funding to support the needs of the university in terms of infrastructure and accessibility of student places. • Lack of technical knowledge to maintain and operate the existing infrastructure in some educational institutions. Qualified teachers now work not only in the field of education. In this way, they divide their efforts between learning and other activities.
The aim of this study is to describe the model of the academic cloud for the training of computer science teachers and to systematize the experience of its deployment based on the Proxmox VE platform.

Related work
Researchers are studying the problem of creating a cloud-oriented environment for higher and secondary education. In general, they interpret this concept as an environment for participants in educational and scientific processes, in which the implementation of computational and procedural functions is purposefully developed virtualized computer technology (corporate or hybrid) infrastructure [12]. Glazunova and Voloshyna [6] considers the concept of "academic cloud of the university" as a cloud-oriented electronic educational environment of the educational institution. It is a combination of technical, software, digital resources and services that operate on the basis of cloud computing technologies and provide educational activities for university students through the local network and the Internet [6]. The basic characteristics of the academic cloud are self-service, wide access to the network, pooling of resources, broad resilience, measurement of service and flexibility, structure, interactivity, personification and consideration of motivation [7,8].
The goal of the academic cloud is to leverage the capabilities of multiple networked computers to collaborate between faculty, researchers, and students. Cloud computing provides unique features such as • Virtualization of computing resources. Through the use of cloud technology, end users can access computing resources from a variety of devices anytime, anywhere. • High productivity. Modern cloud solutions provide significant computing power for processing and storing big data. • Reduced costs are possible through the purchase and maintenance of necessary resources or services on site. This advantage is especially significant in the long run. • Scalability of IT infrastructure. This approach requires the university to invest in the necessary infrastructure and cloud services.
Many universities already use virtualization and cloud computing technologies to create campus information and education environments. These environments are designed for teaching, research, management, activities [5]. The use of cloud services in the process of learning and research is an obvious need of the time and due to the development of digital technologies. The formation of openness of continuing education orients educational institutions to adapt their activities to the requirements by creating an effective information and educational environment that will meet the training requirements and improve the skills of working professionals.
The potential of cloud computing to increase efficiency, cost and convenience for the education sector has been recognized by US universities. For example, a deployed academic cloud allowed computer science students to experiment with large-scale distributed computing [4]. By adopting cloud computing, an educational institution can lease cloud infrastructure to computer providers. [3].
Selviandro, Suryani and Hasibuan [11] investigated the effectiveness of performance indicators to assess and compare the scalability of virtualization resources in cloud data centers. This method can be used by university risk analysts and security managers to conduct realistic and cost-effective risk assessments that are reliable and repeatable.
Baharuddin et al. [2] investigated the use of cloud computing services in the electrical engineering curriculum. They see cloud technology as a model of computing because resources such as processors or computing power, storage, networks, and software are abstracted and provided to students as online or Internet services.

Descriptive model of the academic cloud
In the academic cloud model, we consider the content, technical and organizational components. The content of training determines the technological requirements for each digital system. We are deploying an academic cloud to train future computer science teachers. Therefore, the cloud is a means of knowing information systems. It replaces physical computer systems and is the subject of study by scientists, teachers and students. Given that the academic cloud can be integrated with other means of the university's digital educational environment, there are requirements for its functions. Therefore, the content component of the model involves performing by cloud some functions such as • study of information systems based on their learning models a virtual computers and networks; • the most accurate correspondence of virtual objects to real information systems, in particular the possibility of applying the existing competencies in practice; • opportunity for a teacher or student to change the object of study for their own needs; • ubiquitous access to cloud infrastructure via the university's local network and the Internet; • personalized access to computing resources using single authentication data.
The technical and organizational component must be implemented through tasks • providing the ability to deploy a significant number of virtual machines; • providing access by means of standard protocols; functioning according to the model "infrastructure as a service" (IaaS); • creation of a significant number of virtual networks data routing between virtual and physical networks; • user authentication based on standard databases; ensuring the redistribution of corporate cloud computing resources.
The study was conducted in a joint research laboratory on the use of cloud technologies in education of Ternopil Volodymyr Hnatiuk National Pedagogical University and the Institute for Digitalisation of Education of the National Academy of Educational Sciences of Ukraine. At the time of writing, we have had considerable experience deploying an academic cloud based on the Apache CloudStack platform. It has undeniable advantages for use in student learning. In particular, it allows students to be divided into so-called domains according to academic groups. Provide authentication based on an external database, share virtual machines with many students. However, during the years of operation of the platform, we also had difficulties. They concerned the solution of technical problems such as • poor allocation of computing resources; • manually add users' accounts; • forcibly disabling, deleting or transferring virtual computers to another user, unlocking accounts; • cloud infrastructure backup (databases, VMs); lack of free mobile version of the platform.
Often we had to use the API and directly edit the Apache CloudStack database to solve these problems [9], so we decided to integrate the new infrastructure into the existing academic cloud based on a different platform. The following criteria were taken into account for her selection • openness and free platform; • cloud according to the IaaS model; • technical simplicity of cloud deployment and maintenance; • ensuring user authentication from external databases; • ability to create templates and backups of virtual computer disks; • scalability of the platform, through the addition of additional nodes and storages.

Students' access rights to VM in the cloud
To ensure the functioning of the academic cloud, we have so far rented one refurbished physical server based on a used Intel Xeon processor with 32 Gb of RAM. This was also done because Ukraine is currently at war and it is difficult to say whether servers will be available as they are in university buildings.
Here are some technical tasks in the organizational component of the training model for future computer science teachers and IT professionals. One of the main such tasks is the distribution of students' access to computing resources of the academic cloud. To solve this problem, you need to divide users into groups. The Proxmox VE platform provides built-in groups with different credentials [10].
We offer two approaches to creating new VMs and containers. The first involves the administrator creating copies of VMs and giving students access to these VMs. Permissions may vary depending on the operations to be performed with these VMs. For example cloud administrator can provide students with administrative access to such VMs.
Another approach involves students cloning their own machines from the templates. To implement it, the administrator must provide permission in accordance with the table 1). Creating permissions that match the last row of the table requires defining <students_login> -<VM_ID> pairs. Finally, permission is granted through the command: pveum acl modify /vms/<WM_ID> <user <username> -role Administrator Later, using a simple script, we obtained a list of users from the LDAP directory and set the appropriate IDs for them. It is important that when cloning VM, students must indicate the ID provided by the teacher for VM (figure 1).
In order to share multiple users' VMs, they must be grouped together to share VMs with the same pveum command or through the platform's web interface. In our opinion, the preliminary preparation of the academic cloud for learning is a necessary task. A qualified specialist is required to perform the relevant tasks.

Unified access of student to academic cloud
To ensure the work of many students, we have configured user authentication based on the LDAP directory (Active Directory). It is located in the local network of the university. To do this, you need to specify some parameters such as • The IP address and port of the domain controller that contains the LDAP directory. • User data for reading data from the directory (Bind User). • Some user attributes (mail, belong to a group). • Filters to restrict access to certain organizational units of the directory. • Additional parameters (synchronization objects, operations with new users, deleting users).
To log in to the system, the administrator can add it manually, for example using the web interface. However, it is possible to automatically synchronize the directory with the pveum realm sync <realm> command.

Allocation of computing resources of the academic cloud
The problem with our previous implementation of the academic cloud is the irrational allocation of computing resources, which involves reserving them based on the number and performance of virtual machines (Apache CloudStack reserves guaranteed CPU frequency and RAM for each virtual machine). We solved this problem through CPU and RAM Overcommit. That is, the Apache CloudStack platform installs a multiplier that changes the frequency and amount of RAM. However, this approach often leads to instability in the cloud, including denial of service to virtual machines.
We used containers to better allocate computer resources. It is a cloud-like object of cloud infrastructure. Although virtual machines provide a complete environment, system containers offer an environment as close as possible to what the user receives from the virtual machine, but without the overhead of running a single kernel and simulating all the hardware [1]. From the point of view of the host OS (node), the processes from the container work in themselves. This allows you to run more of them compared to virtual machines. The main limitation of containers is the OS. It can not be other than the OS node (host). Another disadvantage is the lower security of the container. We conducted an experiment and found that the same equipment can run at least 5 times more containers than VM with the same computational characteristics. Figure 2 shows the load schedules of a server running 120 containers (one core, 768 MB RAM). At this time, the cloud generates approximately 40 Mbps of inbound and outbound traffic (figure 2). Unfortunately, we do not have the equipment to deploy identical academic clouds based on the CloudStack and Proxmox VE platforms. However, we can assume that the performance of the physical server with Proxmox VE should match the computing power of the two servers with CloudStack. Therefore, in the CLoudStack-based cloud, we left these 2 hosts active. We then loaded the same VMs on two clouds until their CPU load was 90%. Such VMs were Windows (Workstation and Server) and Linux (with CLI or GUI). Note that the Linux OS with CLI worked as Proxmox VE containers and as regular VMs in CloudStack. The number of VMs that were downloaded is shown in table 2.
As you can see from the table 2, the number of VMs loaded on Proxmox VE is slightly higher than Apache CloudStack. There is a particularly significant difference for VMs of the LinuxCLI type. This is because Proxmox VE used containers for them. Unfortunately, compared to Apache CLoudStack, Proxmox VE has worse results for VMs with nested virtualization. We use such EVE-NG VMs to model and study computer networks (last row of the table 2).

Connection of networks between clouds based on Apache CloudStack and Proxmox VE
An important step in deploying our academic cloud is configuring networks in its infrastructure. We tried to allocate a separate VLAN for at least one academic group of students. The best option is to be able to provide a separate VLAN for each student. Another requirement for cloud infrastructure networks is their integration with the university's cloud infrastructure.
To do this, we configured the VPN server to Proxmox VE node. Since many VLANs are deployed in the university cloud infrastructure, it is necessary to choose VPN technology that encapsulates OSI channel layer data. That is, we plan to use the same VLANs in CloudStack and Proxmox VE infrastructures. This will allow them to transfer data without routing. OpenVPN, a free, secure VPN server, meets this requirement. It has one of the best bandwidths on the Internet, and is one of the most secure VPN services. However, the Proxmox VE platform does not support OpenVPN tap and tun tunnel interfaces. Therefore, we had to set up a separate VM based on MikroTik Cloud Hosted Router. It was configured as an OpenVPN server ( figure 3). It can operate both as a soft switch running within the hypervisors. As a result, VMs from different clouds can exchange traffic within the same VLANs. This allows you to build a hybrid cloud infrastructure, conduct research, and scale the academic cloud through both virtual and physical devices.

Ubiquitous access to the academic cloud via a mobile device
Another advantage of the useful feature of the Proxmox VE platform is the availability of an adapted web interface for mobile devices. The mobile interface does not allow the student to perform absolutely all operations, as the desktop. But it is suitable for urgent operations. This option will also be useful for distance learning when the student does not always have access to a computer or laptop.
Functional applications for Andriod and iOS are mostly paid. However, the mobile version of the Proxmox VE web interface allows students to perform basic VM management operations such as start, stop, pause, migrate VM to another node, view tasks and console (figure 4). Access to the VM interface (console) in the mobile version is also implemented. A student with some problems will be able to use it in Linux. At the same time, he/she has standard VM control options (start, stop, call the keyboard, press the system key combination, etc.) (figure 5).
It is worth noting that working with the mobile version of the VM console with Windows is not convenient. Nevertheless, the student will be able to perform basic operations with a graphical interface OS.

Backup of the academic cloud based Proxmox VE platform
Another important task of the technical component of the academic cloud is the backup of both the cloud infrastructure as a whole and its individual VMs. Here, Proxmox VE provides many tools, including storage and backup types [10]. We have set up a scheduled task to back up the cloud infrastructure to an NFS share that is connected via an OpenVPN server. This allows you to save the condition of all machines. The advantage of this method is the reservation of working VMs "on the fly" (figure 6).  However, it is important to encourage students to make backups and snapshots on their own. This is necessary because they are often taught by trial and error. To do this, you must additionally provide permission to the student to perform the operation Datastore.AllocateSpace. In general, we can say that scaling the cloud infrastructure will require configuring a separate Proxmox Backup Server.

Conclusions
From the analysis of scientific publications and our own experience, we can state that cloud technologies in universities have significant opportunities for application. One of the important components of the cloud-based learning environment is the academic cloud. It can be effectively used in the training of future computer science teachers. The academic cloud allows students to model in a virtualized environment the processes occurring in digital systems. This is especially popular for distance learning. At the same time, they will gain experience in applying theoretical knowledge to perform practical tasks, demonstrate properties, study and research digital objects. The study proposed a model of the academic cloud, which contains content, technical and organizational components.
Proxmox VE can perform the functions of all these components. It is a modern open platform for deploying corporate clouds. As our experience has shown, the process of cloud deployment is technically simpler and faster. This is even though we have installed Apache CloudStack many times and Proxmox VE for the first time. After deploying the academic cloud, we connected virtual local networks of academic clouds based on the Apache CloudStack and Proxmox VE platforms. It would be good to continue the integration of both platforms in the future.
A study of cloud performance based on Proxmox VE showed that it is almost not inferior to the computing power of Apache CloudStack. This is not surprising, since both platforms used the same KVM hypervisor. Proxmox VE has the advantage of using Linux containers. The Proxmox VE console interface is more advanced than Apache CloudStack. This fact, together with a good mobile version of the platform allows you to work with the cloud from different devices.
Regardless of the platform, the administrator or teachers should perform the maintenance of the academic cloud. In particular, regarding the preparation of student VMs, their migration between nodes, backup, etc. Using the academic cloud based on the Proxmox VE platform, both teachers and students can now have more experience with various system administration tools and use virtualization to perform teaching and research tasks.
Prospects for further research are in conducting the experiment that would confirm the usefulness of the proposed model.


Li3DeTr: A LiDAR based 3D Detection Transformer

Inspired by recent advances in vision transformers for object detection, we propose Li3DeTr, an end-to-end LiDAR based 3D Detection Transformer for autonomous driving, that inputs LiDAR point clouds and regresses 3D bounding boxes. The LiDAR local and global features are encoded using sparse convolution and multi-scale deformable attention respectively. In the decoder head, firstly, in the novel Li3DeTr cross-attention block, we link the LiDAR global features to 3D predictions leveraging the sparse set of object queries learnt from the data. Secondly, the object query interactions are formulated using multi-head self-attention. Finally, the decoder layer is repeated $L_{dec}$ number of times to refine the object queries. Inspired by DETR, we employ set-to-set loss to train the Li3DeTr network. Without bells and whistles, the Li3DeTr network achieves 61.3% mAP and 67.6% NDS surpassing the state-of-the-art methods with non-maximum suppression (NMS) on the nuScenes dataset and it also achieves competitive performance on the KITTI dataset. We also employ knowledge distillation (KD) using a teacher and student model that slightly improves the performance of our network.

Introduction
With the advent of deep learning networks for computer vision [16,37] and large-scale datasets [10] the research on perception systems for scene understanding of autonomous vehicles is growing rapidly. 3D object detection is one of the key processes of autonomous driving, which is a two fold process of classification and localization of the objects in the scene. LiDAR is one of the significant sensors of autonomous vehicles which provides precise 3D information of the scene. Although there is a huge progress in 2D object detection approaches [2,12,22,32,33,40], the CNN-based approaches are not well directly adapted to LiDAR point clouds due to their sparse, unordered and irregular nature.
Earlier approaches for 3D object detection on LiDAR data can be divided into two types: point-based and grid-based methods. Point-based methods [27,36,49] are based on point operations [29,30] which detect the 3D objects directly from the point clouds. Grid-based methods either voxelize the points into volumetric grids or project the points to Birds Eye View (BEV) space. The advantage of BEV projection is that it preserves euclidean distance, avoids overlapping of objects and the object size is invariant to distance from ego vehicle which is significant for autonomous driving scenarios. The sparse CNNbased voxel feature extraction [47] is advantageous but it can not extract rich semantic information with limited receptive fields. We mitigate this issue by employing a multiscale deformable attention [55] encoder to capture global LiDAR feature maps.
Earlier approaches either use two-stage detection pipeline [7,36] or anchors [19,53] or anchor-free networks [42,43,51] for 3D object detection, but all of them employ post-processing method like NMS to remove redundant boxes. Inspired by Object-DGCNN [45], we formulate the 3D object detection problem as a direct set prediction problem to avoid NMS.
We propose an end-to-end, single-stage LiDAR based 3D Detection Transformer (Li3DeTr) network to predict the 3D bounding boxes for autonomous driving. Firstly, the voxel features are extracted with SECOND [47] by leveraging sparse convolutions [15] and BEV transformation or with PointPillars [19]. Secondly, we employ an encoder module with multi-scale deformable attention [55] to capture rich semantic features and long range dependencies in BEV feature maps to generate LiDAR global features. The LiDAR global features are passed to the decoder module. Finally, we introduce a novel Li3DeTr cross-attention block in the decoder to link the LiDAR global features to the 3D object predictions leveraging the learnt object queries. The object queries interact with each other in multi-head selfattention block [41]. The object queries are iteratively refined and 3D bounding box parameters are regressed in every decoder layer. Inspired by DETR [2], we use set-to-set loss to optimize our network during training.
We conduct experiments on two publicly available au-  Table 3. The ablation study shown in Table 6 justifies our novel Li3DeTr cross-attention block. • We release our code and models to facilitate further research.

Related Work
The LiDAR point cloud based 3D object detection approaches can be divided into two categories: point-based and grid-based, depending on the type of data representation used to predict the 3D bounding boxes. Point-based methods [28,35,36,49,50] directly use the sparse and unordered set of points to predict 3D bounding boxes. The point features are aggregated by multiscale/multi-resolution grouping and set abstraction [29,30]. PointRCNN [36] employs a two-stage pipeline for 3D object prediction. PVRCNN [35] models point-voxel based set abstraction layer to leverage the advantage of point and voxel based methods. Frustum-PointNet [28] uses 2D object detection to sample a frustum of points to apply Point-Net [29] to predict 3D objects. Although point-based methods achieve large receptive fields with set abstraction layer, they are computationally expensive. Grid-based methods. As the LiDAR point clouds are sparse and unordered set of points, many methods project the points to regular grids such as voxels [47,53], BEV pillars [19] or range projection [3,13,39]. The point clouds are discretized into 3D voxels [38,53] and 3D CNNs are employed to extract voxel-wise features. However, 3D CNNs are computationally expensive and requires large memory, in order to mitigate this problem [5,47] use sparse 3D CNNs [15] for efficient voxel processing. The LiDAR point cloud is projected into BEV map in Point-Pillars [19] and PIXOR [48] and 2D CNNs are employed to reduce the computational cost, however such projection induces 3D information loss. In order to mitigate this issue, some methods [47,51] compute voxel features using sprase convolutions and then project the voxel features into BEV space, and finally predict the 3D bounding boxes in the BEV space. As this approach takes the advantage of voxel and BEV space, we test our network with SECOND [47] and PointPillars [19] feature extraction networks. In order to achieve large receptive fields similar to point-based methods [29,30], we model long-range interactions of local LiDAR features using multi-scale deformable attention [55] block in our encoder to obtain LiDAR global features.
Transformer-based methods. Earlier approaches [7,19,35,36,50,53] object detection head employ anchor boxes to predict the objects, however anchor boxes involve handcrafted parameter tuning and they are statistically obtained from the dataset. To mitigate this issue, some approaches [5,43,48,51] followed anchor-free pipeline by computing per-pixel or per-pillar prediction. But these approaches use NMS to remove redundant boxes. DETR [2] is the first transformer architecture which formulated 2D detection problem as a direct set prediction to remove NMS. Our network follows similar formulation for 3D object detection. Some approaches [24,27,34] used transformer for feature extraction networks. 3DETR [25] is a fully transformer based architecture for 3D object detection using vanilla transformer [41] block with minimal modifications. 3DETR directly operate and attend on points whereas our approach voxelize the points and attend the BEV global voxel features which is computationally efficient for autonomous driving scenarios. 3DETR employs downsampling and set-aggregation operation [30] on the input points of indoor scenarios because the computational complexity of self-attention increases quadratically (O(n 2 )) with the number of input points. Moreover, 3DETR is effective on indoor datasets, where the points are dense and concentrated. Object-DGCNN [45] employs a graph-based model for transformer-based 3D object detection for outdoor environments. BoxeR [26] introduces a novel and simple Box-Attention which enables spatial interaction between grid features. BoxeR-2D enables end-to-end 2D object detection and segmentation tasks, which can be extended to BoxeR-3D for end-to-end 3D object detection. VISTA [11] is a plug and play module to adaptively fuse multi-view features in a global spatial context, incorporated with [5,51]. It introduces dual cross-view spatial attention to leverage the information in BEV and Range View (RV) features. We formulate our model with voxel-BEV based CNN backbone architecture for local feature extraction and an attention-based architecture for global feature extraction to increase the re-ceptive field size and finally a transformer decoder head to link global features and 3D predictions.

Methodology
Our LiDAR based 3D Detection Transformer (Li3DeTr) architecture inputs LiDAR point cloud and predicts 3D bounding boxes in large-scale outdoor environments such as autonomous driving. The network contains two main modules: backbone and transformer encoder-decoder as shown in Figure 1. Inspired by the state-of-the-art 3D object detection approaches [15,19], our CNN-based backbone ( § 3.1) module learns grid-based local voxel features. Specifically, we employ BEV grid, not only because the 2D grid-like features are a good trade-off between accuracy and efficiency but also very relevant for autonomous driving, as there is possibly one object on every grid cell on the ground plane. The multi-scale deformable attention [55] based encoder ( § 3.2) module learns the multi-scale global voxel features. The encoder module alternates between multiscale deformable attention block and multi-layer perceptron (MLP) block and is repeated L enc number of times. The novel Li3DeTr cross-attention block in the decoder ( § 3.3) module links the global voxel features to the 3D object predictions leveraging the learnt object queries. The object queries interact with each other in multi-head self-attention [41] block. The decoder module is repeated L dec number of times with alternating multi-head self-attention, Li3DeTr cross-attention and MLP blocks. The refined object queries are transformed into 3D bounding box parameters and the network is trained end-to-end using permutation-invariant loss [2] ( § 3.4).

Backbone: Local features
Our network inputs LiDAR point cloud P = {p 1 , . . . , p i , . . . , p N } ⊂ R 3 . To accelerate the 3D object detection for large-scale point clouds, we scatter the points into BEV grid and use CNNs to extract local point features. We test two pipeline: 1) We voxelize the point cloud with [0.1, 0.1, 0.2] metres voxel size and employ SparseConv [15] to compute 3D sparse convolutions and obtain local voxel features. The empty voxels are filled with zeros and the sparse voxels are transformed to a BEV 2D grid-like features. 2) We convert the point cloud into a dense BEV pillar map as in PointPillars [19] with [0.2, 0.2, 8] metres pillar resolution. We use pillar feature net to transform the pillar features. Finally, we employ SEC-OND [47] backbone to extract local voxel features from the sparse voxel or BEV pillar features and further transform them using a feature pyramid network (FPN) [20] to obtain multi-scale local voxel feature maps F 1 ,

Encoder: Global features
In order to obtain global voxel features from the local voxel feature maps, we employ multi-scale deformable attention [55] mechanism because the conventional attention mechanism [41] leads to unacceptable computational complexities in encoding high resolution feature maps. The multi-scale deformable attention combines the best of the sparse sampling of deformable convolution [9] and longrange relation framework of transformers [41]. The input and output to encoder module are multi-scale feature maps with the same resolution. The deformable attention attends only a small set of key sampling points around a reference point and thereby reduces the computational complexity. The reference point in the deformable self-attention for each query pixel is itself. Each encoder layer consist of multi-scale deformable self-attention and MLP blocks with residual connections and is repeated L enc times. The global voxel features extracted from the encoder is passed to Li3DeTr cross-attention block in the decoder.

Decoder
The state-of-the-art 3D object detection approaches either formulates the detection head with dense set of anchor boxes or dense per-pillar prediction such as [43,51], followed by NMS. We remove the need of post-processing step (like NMS) by formulating the detection head to predict a set of bounding boxes instead of per-pillar prediction. This is formulated in the decoder as detailed below.
The decoder inputs a set of object queries . . , L dec }, N q is number of queries and Q 1 are learnt with the model weights) and global voxel feature maps {F j } 4 j=1 and it consists of decoder layer that is repeated L dec number of times to refine the object queries.
For the first decoder layer, the 3D reference points are encoded from the object queries using a single-layer fully connected (FC) network and sigmoid normalization as in Equation 1.
where r i ∈ [0, 1] 3 and ϕ ref is a FC layer. Each decoder layer consists of Li3DeTr cross-attention block, multi-head self attention block and MLP block with skip connections as shown in Figure 1.
Li3DeTr cross-attention block inputs the object queries (we drop the layer index for simplicity), 3D reference points r i and LiDAR global multi-scale feature maps {F j } 4 j=1 . The formulation of Li3DeTr cross-attention block is illustrated in Figure 2.
Let R ji represents the transformation for the projection of reference point r i into scale j of LiDAR global voxel feature map. The LiDAR BEV feature map at scale j is bilinearly sampled at the location of projection of reference point  Figure 2: An overview of Li3DeTr cross-attention block (R ji (r i )), given by F j (R ji (r i )). The attention weights (w ij ) for each query q i at each sampled feature scale j is computed by a FC layer (ϕ samp ) and sigmoid normalization, where w ij = ϕ samp (q i ). The sampled features from multi-scale feature maps are added together to obtain crossattention features (F CAttn i ) for i-th reference point as: Finally, we update the queries as, where PE is positional encoding of reference point to make the queries location aware. The queries interact with each other in the multi-head self-attention block and we follow skip connection following [41]. The object queries (Q l ) are updated in each decoder layer.
We employ two FC layers ϕ reg l and ϕ cls l to predict offset to box position ∆p l i ∈ R 3 , box size (l l i , w l i , h l i ), box orientation (sin θ l i , cos θ l i ), box velocity (v l xi , v l yi ) and class label (ŷ l i ) respectively for each object query (q l i ). We refine the reference points for each query in each decoder layer except for the first decoder layer (which are encoded using Equation. 1) by using the predictions of box position in the previous layer, as

Loss
Similar to [2,44,45], the error between predictions and ground-truths is calculated by set-to-set loss. Firstly, the one-to-one assignment between predictions and groundtruths is performed by Hungarian algorithm [18]. Secondly, we use bipartite matching to match predictions and groundtruths. Finally, we employ L 1 loss and focal loss [21] to calculate bounding box regression and classification loss respectively, given the bipartite matching.

Experiments
We evaluate Li3DeTr network on the publicly available autonomous driving datasets, nuScenes [1] and KITTI [14]. We introduce the experimental setup ( § 4.1) with dataset details and evaluation metrics (model settings and training details are provided in supplementary), and then present both qualitative and quantitative results ( § 4.2) and analysis on the nuScenes and KITTI dataset. We perform ablation studies ( § 4.3) to study the different configurations of our network.

Experimental Setup
nuScenes dataset [1] contains 750, 150 and 150 sequences (of ∼20s duration) with 28K, 6K and 6K annotated samples for training, validation and testing respectively. Each sample consists of 32-beam LiDAR point cloud with 30K points. The dataset also provides 9 non-key frames (called sweeps) to aggregate to one key-frame, resulting in ∼300K points per annotated frame. 10 different categories of objects are available to compute the metrics. Kitti dataset [14] consists of 7481 and 7518 training and testing samples. The training samples are further divided into 3712 train and 3769 val splits. Each sample consists of 32-beam LiDAR point cloud. Cars, pedestrians and cyclists are the three main categories for evaluation. Evaluation metrics. The two main metrics following the official evaluation of nuScenes dataset [1] are mean average precsion (mAP) and nuscenes detection score (NDS). In addition, we also evaluate true positive (TP) metrics: average translation error (ATE), average scale error (ASE), average orientation error (AOE), average velocity error (AVE), average attribute error (AAE). We follow the official evaluation metric of KITTI dataset [14] mean average precision (mAP) with IoU threshold of 0.7 for car category and 0.5 for pedestrian and cyclist categories..

Quantitative Results
We compare Li3DeTr network with the state-of-the-art methods on the nuScenes [1] test dataset as shown in Table 1. Our network surpassed the state-of-the-art CNNbased CenterPoint [51] by 3.3% mAP and 2.1% NDS and transformer-based Object-DGCNN [45] network by 2.6% mAP and 1.6% NDS. Albeit CenterPoint [51] uses postprocessing method like NMS, our approach of formulating object detection as a direct set prediction problem inspired by DETR [2] doesn't require NMS to obtain the gain in mAP and NDS. We also compare with the NMS-free state-of-the-art transformer-based network Object-DGCNN [45] with two different backbones: PointPillars [19] and VoxelNet [53]. Our transformer based NMS-free approach surpassed in both pillar [19] and voxel [53] backbones by 0.6% mAP (and 0.2% NDS) and 2.6% mAP (and 1.6% NDS) respectively. Although our Li3DeTr network outperforms most of the other methods in terms of mAP and NDS, VISTA-OHS [11] performs slightly better than our approach on the nuScenes test dataset. VISTA is a plug and play module to fuse multi-view features incorporated with [5] which requires post-processing methods like NMS, whereas our approach is a standalone method for 3D object detection without NMS. The perfromance of our approach compared with state-of-the-art approaches on nuScenes val dataset is provided in supplementary.
We employ knowledge distillation (KD) with a teacher and student model. The earlier works on 3D object detection involve NMS, so it is not effective to distill those models. As our approach is NMS-free, we can effectively distill the information between models with similar detection heads. We train a teacher model with the loss given in § 3.4 and then we train a student model (with same architecture as teacher model) with the supervision of output of teacher model and the ground-truth. With the KD, we achieve 62.2% mAP and 68.0% NDS which is 0.9% mAP and 0.4% NDS improvement over our model without KD.
We compare the recent works on KITTI [14] dataset for car category as shown in Table 2. Our network achieves a competitive performance to the state-of-the-art LiDARbased approaches like VoxelNet [53], PointPillars [19], TANet [23] and SECOND [47] in terms of AP 3D and AP BEV for easy, moderate and hard samples. Our network could not achieve the state-of-the-art performance on KITTI [14] dataset as compared to nuScenes [1] dataset because the transformer network is data hungry and KITTI dataset has 3712 samples for training which is approximately 7.5 times less number of training samples than nuScenes dataset (which has 28K training samples). In addition to this, the nuScenes dataset provides 9 non-key frames (called sweeps) to aggregate to one key frame, resulting in dense LiDAR points but KITTI dataset provides only one LiDAR key frame which results in sparse point cloud. However, our approach obtains competitive performance to the state-of-the-art transformer based architecture VoTr-SSD [24] which uses NMS. To the best of our knowledge this is the first transformer based 3D detection network to report results both on nuScenes and KITTI datasets, which compares the results of detection with an emphasis on training sample size which is significant for transformer based architectures alongside the density of LiDAR point clouds. We further provide comparison of methods in terms of AP 3D and AP BEV on the pedestrian and cyclist categories for easy, moderate and hard samples in the supplementary.

Qualitative Results
The visualization of 3D bounding box predictions of our approach on the nuScenes dataset is shown in Figure 3. Although the LiDAR point clouds are sparse, our approach not only detects small objects like traffic cones but also efficiently detect large size objects like truck, bus, construction vehicle. This is possible with the local and global feature maps of backbone and attention mechanism in encoder alongside the cross-attention in the decoder. Our approach is also able to detect some cars which are not annotated in ground-truth. A short demo video of 3D object predictions of our network projected into BEV map is presented at https://youtu.be/5pLnLRO_2-U.

Analysis
The performance analysis of our approach by object category, object distance and object size compared to state-ofthe-art LiDAR based CNN and transformer models is detailed below. Object category. The performance of our network in terms of Average Precision (AP) for each object category compared to other state-of-the-art networks on the nuScenes [1] val dataset is shown in Table 3. We compare with Object-DGCNN [45] as it a standalone transformer model like ours for fair comparison. The transformer encoder which extracts global LiDAR features leveraging long range interactions using multi-scale deformable attention and decoder cross-attention significantly improves the AP of large   Table 4. Our approach significantly improves the mAP of objects at object distance greater than 30m compared to CNN-based CenterPoint [51]. Although the Li-DAR point cloud is sparse at far distance from ego vehicle, our attention mechanism in encoder and decoder models long range interactions between the sparse points to predict the objects at far distance.  Table 5. Our transformer based approach predominantly improves the mAP of large size objects than small size objects compared to CNN-based Cen-terPoint [51]. This proves our hypothesis that long range interactions made possible by attention mechanism improves the detection performance.

Ablation Studies
Attention blocks. The performance of our network in terms of mAP and NDS with different attention operations for self and cross-attention blocks in the decoder ( § 3.3) is shown in Table 6. We test our approach by employing DGCNN [46] similar to [45] and multi-head self-attention  Number of queries. The performance of our network in terms of mAP and NDS on the nuScenes [1] dataset for different number of queries in the decoder is shown in Table 7.
The performance of our network in terms of mAP and NDS slightly increases with increase in number of queries because object queries represent the potential positions of objects. However, the performance has slight impact for over 900 queries. So, we fix number of queries to 900. Backbones. The performance of our network in terms of mAP and NDS on nuScenes dataset for different backbones compared with CenterPoint [51] and Object-DGCNN [45] is shown in Table 8. We test our approach with Point-Pillars [19] with 0.2m voxel size and VoxelNet [53] with 0.1m voxel size for LiDAR point cloud feature extraction. As shown in Table 8, our approach with VoxelNet feature extraction outperforms the network with PointPillars backbone. However, our architecture is flexible to plugin various backbones depending on the specific requirements of different applications.

Conclusion
We present an end-to-end, single-stage LiDAR based 3D Detection Transformer (Li3DeTr) architecture which inputs LiDAR point clouds and predicts 3D bounding boxes. Inspired by DETR [2], we formulate our model with set-toset loss and thereby remove the need for post processing methods like NMS. We introduce a novel Li3DeTr crossattention block in the decoder head to link the global Li-DAR voxel feature maps (obtained from encoder network) and 3D predictions, leveraged by sparse set of object queries learnt from the data. Without bells and whistles, our network archives 61.3% mAP and 67.6% NDS surpassing the state-of-the-art methods on the nuScenes dataset and achieves competitive performance on the KITTI dataset.