## RoCo: Robust Collaborative Perception By Iterative Object Matching and Pose Adjustment

## Zhe Huang †

## Shuo Wang †

## Yongcai Wang ∗

School of Information, Renmin University of China Beijing, China huangzhe21@ruc.edu.cn

Wanting Li University of China Beijing, China

School of Information, Renmin lindalee@ruc.edu.cn

School of Information, Renmin University of China Beijing, China shuowang18@ruc.edu.cn

## Deying Li

School of Information, Renmin University of China Beijing, China deyingli@ruc.edu.cn

## Abstract

School of Information, Renmin University of China Beijing, China ycw@ruc.edu.cn

## Lei Wang

University of Wollongong Wollongong, NSW, Australia lei\_wang@uow.edu.au

## Keywords

Collaborative autonomous driving with multiple vehicles usually requires the data fusion from multiple modalities. To ensure effective fusion, the data from each individual modality shall maintain a reasonably high quality. However, in collaborative perception, the quality of object detection based on a modality is highly sensitive to the relative pose errors among the agents. It leads to feature misalignment and significantly reduces collaborative performance. To address this issue, we propose RoCo, a novel unsupervised framework to conduct iterative object matching and agent pose adjustment. To the best of our knowledge, our work is the first to model the pose correction problem in collaborative perception as an object matching task, which reliably associates common objects detected by different agents. On top of this, we propose a graph optimization process to adjust the agent poses by minimizing the alignment errors of the associated objects, and the object matching is re-done based on the adjusted agent poses. This process is carried out iteratively until convergence. Experimental study on both simulated and real-world datasets demonstrates that the proposed framework RoCo consistently outperforms existing relevant methods in terms of the collaborative object detection performance, and exhibits highly desired robustness when the pose information of agents is with high-level noise. Ablation studies are also provided to show the impact of its key parameters and components. The code is released at https://github.com/HuangZhe885/RoCo.

## CCS Concepts

- · Computer system organization → Autonomous driving ; · Computing methodologies → Collaborative detection .

∗ Corresponding author. † Equal contribution.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org.

MM'24, October 28-November 1, 2024, Melbourne, VIC, Australia

© 2024 Copyright held by the owner/author(s). Publication rights licensed to ACM.

ACM ISBN 979-8-4007-0686-8/24/10

https://doi.org/10.1145/3664647.3680559

Collaborative Perception, 3D object detection, Point cloud, Pose error, Graph Matching

## ACMReference Format:

Zhe Huang, Shuo Wang, Yongcai Wang, Wanting Li, Deying Li, and Lei Wang. 2024. RoCo : Robust Collaborative Perception By Iterative Object Matching and Pose Adjustment. In Proceedings of the 32nd ACM International Conference on Multimedia (MM '24), October 28-November 1, 2024, Melbourne, VIC, Australia. ACM, New York, NY, USA, 10 pages. https://doi.org/10.1145/ 3664647.3680559

## 1 Introduction

Collaborative perception can significantly enhance perception performance by sharing information from different sensors among agents. It can overcome the inherent limitations of single-agentbased perception, such as invisibility caused by occlusion or longrange issues. Recent research [14, 16, 21, 45, 48] has spurred the widespread attention in the fields such as autonomous driving, metaverse, multi-robot systems, and multimedia application systems [1, 10, 17, 22, 25, 43, 46, 49]. As an emerging topic, collaborative 3D object detection is of significance but also faces many challenges, including model-agnostic and task-agnostic formulation [26], communication bandwidth constraints [16, 40], time delay [41], adversarial attacks [24, 42], and multimodal fusion [27, 29].

In collaborative 3D object detection, regardless of the modality of the employed sensors, the detection module of each agent transmits the obtained object detection features to the Ego agent. The Ego agent gathers the features of the same object and then carries out further information fusion. Therefore, the object detection features received from other agents need to be accurately aligned in the Ego agent for further processing. However, in practical applications, the pose estimated for each agent contains errors, which can in turn cause misalignment of the positions of the same objects across agents. For example, as illustrated in Figure 1(a), this misalignment may result in the incorrect fusion of Object 3 detected by agent 𝑖 (red box) with Object 2 detected by agent 𝑗 (green box). This issue becomes more pronounced when the pose error is large or high traffic is presented. This paper will focus on addressing the object

Figure 1: Illustration of robust collaborative perception system and the result with or without the proposed RoCo.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00257v1-figure-b23817d4-605a-4b5f-a108-3857b7ff9deb.jpg)

alignment problem between multiple agents due to pose errors in a given modality (LiDAR).

Previous works have proposed some methods to improve the robustness of pose estimation [12, 35, 40, 45, 50]. For example, V2XViT [45] adopts multiscale window attention to capture features at different ranges, while V2VNet [40] combines pose regression, global consistency, and attention aggregation modules to correct relative poses. CoAlign [28] introduces a proxy-object pose graph to enhance the pose consistency between collaborative agents without the need of real poses for supervision. The above methods work well when there is minimal variation in poses and a limited number of vehicles. However, in crowded and complex scenarios, as the number of objects and the magnitude of pose errors increase, the deviation in object positions will grow, making it increasingly difficult to correctly match objects at the same location. This situation ultimately reduces the accuracy of 3D object detection.

To address these limitations, we propose a novel iterative object matching and pose adjustment framework called RoCo, which can handle the issue of matching the object detection information from multiple vehicles in noisy scenarios. RoCo is primarily divided into two parts: object matching and robust graph optimization. Object matching involves correctly establishing the matching relationships between multiple detected objects using the distance and neighborhood structural consistency of local graphs. Robust graph optimization builds a pose optimization graph for the whole scene based on the above matching relationships, and iteratively adjusts the poses of agents according to global observation consistency, so as to effectively filters out incorrect matches. RoCo has three main advantages: i) it is an unsupervised method, requiring no ground-truth pose information for the agents or objects and it can adapt to various levels of pose errors; ii) Implementing RoCo does not require retraining or fine-tuning any network models and can be integrated into any 3D object detection based collaborative perception framework; iii) Even in crowded and noisy environments, RoCo can accurately align the detections of the same object obtained by different agents. Unlike the existing methods that adjust object detection features, the proposed method is based on an object-matching-guided pose adjustment which can fundamentally prevent the introduction of additional noise into the features.

We conduct extensive experiments on both simulation and realworld datasets, including V2XSet[45] and DAIR-V2X[49]. Results show that RoCo consistently achieves the best performance in the task of collaborative 3D object detection with the presence of pose errors. In summary, the main contributions of this work are:

- · We propose RoCo, a novel robust multi-agent collaborative LiDAR-based 3D object detection framework that addresses the matching errors and pose inaccuracies between agents and objects. To the best of our knowledge, RoCo is the first to model the pose correction problem in collaborative perception as an object matching task.
- · The proposed RoCo establishes matching relationships between targets based on distance and neighborhood structural consistency using a graph matching approach. On top of this, RoCo iteratively adjusts agent poses based on global observation consistency, effectively filtering out incorrect object matches.
- · Extensive experiments have shown that RoCo achieves more accurate and robust 3D object detection performance even in the scenarios with vehicle congestion and significant noise.

## 2 Related work

## 2.1 Collaborative Perception

Collaborative perception is a promising mechanism in multi-agent systems that can overcome the limits of single-agent perception. To support research in this field, various strategies have been devised to address practical challenges[44][39][27][19][18]. To reduce communication bandwidth, Where2comm[16] selects spatially sparse but perceptually significant regions by using a spatial confidence map. V2X-ViT[45] includes a multi-agent attention module that aggregates information from various agents, improving detection performance. To handle communication delays, SyncNet[30] uses historical multi-frame information to compensate for current information, employing feature attention co-symbiotic estimation and time modulation techniques. CoBEVFlow[41] creates a synchronyrobust collaborative system for Bird's Eye View (BEV) flow that aligns asynchronous collaboration messages sent by various agents using motion compensation. HEAL[27] introduces a pioneering and adaptable collaborative perception framework designed to seamlessly integrate continually emerging heterogeneous agent types into collaborative perception tasks. However, in crowded and noisy road scenarios, information often becomes inconsistent and misaligned, leading to mismatches between objects detected by different agents and resulting in performance degradation. Therefore, in this work, we specifically considered the robustness to pose errors.

## 2.2 Noisy Pose Issue

Due to performance discrepancies between hardware acquisition devices and model estimations, the poses of agents are susceptible to interference, leading to misalignment and inconsistencies during fusion processes. Therefore, many methods attempt to design robust networks to correct error influences.The first category of methods primarily involves proposing robust network architectures and introducing specific modules to learn the influence of poses,

such as V2VNet[40], MASH[13], V2X-ViT[45] and FeaCo[14]. However, these methods require additional pose supervision signals. The second category is to use the coarse observations as prior knowledge to adjust the poses of objects[28, 55]. CoAlign[28] introduced a method for modeling and optimizing agent-object pose graphs to enhance the consistency of relative poses, thereby improving model robustness. Nevertheless, in circumstances with crowded roadways, this approach's use of clustering to create connections between objects could result in mistakes and fail pose graph optimisation. In contrast, our proposed RoCo models the reduction in pose errors as a matching and optimization problem for objects, accurately identifying relationships between objects detected by multiple agents. Through iterative pose optimization based on the correct matching graph, RoCo improves the accuracy of 3D detection.

## 2.3 Object Matching

SLAM[3, 4, 8, 32, 33, 37, 38, 55] (Simultaneous Localization and Mapping) is a crucial research approach in scene understanding, primarily used by autonomous robots (such as robots and autonomous vehicles) to estimate their positions and build maps of the environment simultaneously during motion, without prior environmental information. Object matching in SLAM refers to associating detection results of the current frame with known objects in the scene. Most methods use metrics such as distances between different objects or Intersection over Union (IoU), followed by matching using the Hungarian algorithm[6, 34, 36, 52, 54], which is classical and high-performing but impractical for large-scale deployment in real environments. Therefore, researchers have explored many methods to facilitate its application[2][5][7]. To our knowledge, we are the first to model the pose correction problem of multiple agents in collaborative perception as an object matching and optimization task. This work marks the first application of SLAM-based object matching in collaborative perception, expanding its scope of application, and investigation into the impact of object matching on multi-agent collaborative perception systems in challenging environments.

## 3 Collaborative Perception and the Issue of Pose Error

Following the literature of collaborative perception [14, 28, 41], the commonly used model can be described as below.

It is assumed that there are 𝑁 agents (i.e., collaborators) in the scene. Let X 𝑖 and O 𝑖 be the raw observation and the perception output of Ego agent 𝑖 . M → 𝑗 𝑖 denotes the collaboration message sent from agent 𝑗 to agent 𝑖 . The goal of collaborative perception is to enhance the 3D object detection capability of the ego agent through cooperation. This process can be formally expressed as:

$$F _ { i } = \Phi _ { E n c } \left ( X _ { i } \right ), \ \ i = 1, \cdots, N, \quad \ \ ( 1 a ) \quad \ \frac { \text{acc} } { c.. }$$

$$\mathcal { M } _ { j \to i } = \Phi _ { P r o j } \left ( \xi _ { i }, ( F _ { j }, \xi _ { j } ) \right ), \ \ j = 1, \cdots, N ; j \neq i \quad ( 1 b )$$

$$F _ { i } ^ { \prime } = \Phi _ { A g g } \left ( F _ { i }, \{ \mathcal { M } _ { j \rightarrow i } \} _ { j = 1, 2, \dots, N ; j \neq i } \right ), \quad \quad ( 1 c ) \quad \ \ A \ 1$$

$$\mathcal { O } _ { i } & = \Phi _ { D e c } \left ( F _ { i } ^ { \prime } \right ). & ( 1 d ) \quad \frac { 4. 1 } { \mathcal { O } I }$$

First, for each agent, Step (1a) extracts feature 𝐹 𝑖 from the raw observation X 𝑖 via an encode network Φ 𝐸𝑛𝑐 (·) . After that, each agent 𝑗 projects, via an projection module Φ 𝑃𝑟𝑜𝑗 (·) , its feature 𝐹 𝑗 to the agent 𝑖 's coordinate system based on 𝜉 𝑖 and 𝜉 𝑗 which represent the poses of the two agents, respectively. The projected feature is then sent to agent 𝑖 as a message M → 𝑗 𝑖 , and this completes Step (1b). After receiving all the ( 𝑁 -1 ) messages, agent 𝑖 will fuse its feature 𝐹 𝑖 with these messages via a fusion network Φ 𝐴𝑔𝑔 (·) , producing a fused feature 𝐹 ′ 𝑖 as in Step (1c). Finally, Step (1d) uses a decode network Φ 𝐷𝑒𝑐 (·) to convert 𝐹 ′ 𝑖 to the final perception output O 𝑖 .

As seen above, when the pose of agent 𝑖 is assumed to be given, the projection in Step (1b) will critically rely on the accuracy of the pose of agent 𝑗 , 𝜉 𝑗 . However, the pose estimated by each agent cannot be perfect in practice and often come with noise, i.e., { 𝜉 𝑗 , 𝑗 = 1 , · · · , 𝑁 } have errors. This kind of error adversely affects the accuracy of the projected feature to be sent via the message M → 𝑗 𝑖 . This leads to the misalignment of features when the message is processed by agent 𝑖 and finally hurts the performance of 3D object detection.

To correct agent poses, existing work [28] innovatively uses a clustering based method to determine association among the bounding boxes detected by different agents. After finding object associations, the agent poses are adjusted by graph optimization accordingly. Nevertheless, when the error of agent pose becomes significant or when objects become close in crowded scenes, such a clustering based matching method will be prone to producing large matching errors. The matching errors will in turn lead to large pose adjusting errors, causing poor collaborative perception.

## 4 Our Proposed Method

To improve this situation, this paper proposes an unsupervised, iterative object matching and pose adjusting framework, called RoCo . It can maintain accurate matching relationships even in scenarios with high pose errors and traffic congestion, thereby enhancing object detection performance.

Our solution consists of two key ideas. Firstly, finding reliable matching relationship between two objects depends not only on their spatial similarity but also on the similarity of the object configuration in the neighborhood of the two objects. Secondly, we take an iterative approach to conduct object matching and pose adjustment. That is, after using the current matching result to adjust the agent poses, object matching will be conducted again upon the updated poses. This process repeats until convergence, that is, the object matching does not change. This approach is able to improve the matching accuracy and in turn lead to better adjustment for the agent poses.

It is worth noting that the accuracy of object detection from each individual agent also affects the quality of object matching. In this work, we assume that every agent can independently and accurately detect objects within its own range. This allows us to focus on the issue of feature misalignment caused by pose errors.

## 4.1 Object Detection

Following the literature, the collaborative perception in this work uses LIDAR to perform 3D object detection. The input point cloud is in the dimensions of ( 𝑛 × 4 , where ) 𝑛 denotes the number of points. Each point has its intensity and the 3D position in a world coordinate system. All agents use the same 3D object detector Φ 𝐸𝑛𝑐 .

## Single-Agent Detection Multi agent communication

## Graph guided Object Association Matching

## Robust Graph Optimization Fusion detection

Figure 2: Overview of RoCo system. The object bounding boxes and poses are transmitted as messages to other agents to achieve object matching and robust graph optimization, resulting in corrected matching and poses. Features are transformed based on the corrected poses in the ego coordinate system and fused across all agents.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00257v1-figure-f3df449e-131e-4603-8d0f-d6bcc01f9f7b.jpg)

Figure 3: Object Matching and Pose graph illustration.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00257v1-figure-5d483c2c-0745-49d9-9460-dad03b02685e.jpg)

This work employs a standard uncertainty detection framework designed in the literature [28], which is based on the anchor-based PointPillar backbone [22]. Taking point clouds as input, this detection framework outputs features 𝐹 𝑖 and the estimated bounding boxes for objects. In addition, it estimates the uncertainty of each bounding box, which is expressed as the variances of the center position and the heading angle of the corresponding object.

## 4.2 Graph-guided Object Matching

After performing object detection, agent 𝑗 will share a message M → 𝑗 𝑖 to the ego agent 𝑖 . This message contains three pieces of information including i) its feature map 𝐹 𝑗 ; ii) the set of bounding boxes of the detect objects B 𝑗 , in which each element has the form of ( 𝑥,𝑦, 𝑧, 𝑤, 𝑙, ℎ, 𝜃 ) , denoting the 3D centre position, the size in different dimensions, and the yaw angle, respectively, for each bounding box; iii) the pose of agent 𝑗 denoted by 𝜉 𝑗 = GLYPH&lt;0&gt; 𝑥 ,𝑦 , 𝜃 𝑗 𝑗 𝑗 GLYPH&lt;1&gt; , representing its centre position and the heading angle on the 2D space. Once the ego agent 𝑖 receives the messages from all the other agents, it will start performing object matching by processing the messages one by one. Formally, the task can be described as follows: given the information {B 𝑖 , 𝜉 𝑖 } and GLYPH&lt;8&gt; B 𝑗 , 𝜉 𝑗 GLYPH&lt;9&gt; , agent 𝑖 needs to find the associations among the bounding boxes in B 𝑖 and B 𝑗 .

Modeling this task as a bipartite graph matching problem [15], we seek an optimal matching relationship that maximizes the similarity between the objects detected by agents 𝑖 and 𝑗 . This can be formulated as

$$\mathcal { A } _ { i, j } ^ { * } = \underset { \mathcal { A } _ { i, j } } { \arg \max } \, \sum _ { p \in \mathcal { B } _ { i } } S \left ( p, \mathcal { A } _ { i j } ( p ) \right ), \quad \quad ( 2 )$$

where A 𝑖 𝑗 denotes a list of one-to-one matching from the elements in B 𝑖 to those in B 𝑗 . A ( ) 𝑖 𝑗 𝑝 represents the object, denoted by 𝑞 , found in B 𝑗 by querying the matching relationship A 𝑖,𝑗 using an object 𝑝 in B 𝑖 . Let 𝑆 ( 𝑝, 𝑞 ) denote the similarity of 𝑝 and 𝑞 , where 𝑞 = A ( ) 𝑖 𝑗 𝑝 . A threshold 𝜏 1 is applied to select the reliable matching, i.e., 𝑆 ( 𝑝, 𝑞 ) = 0, if 𝑆 ( 𝑝, 𝑞 ) &lt; 𝜏 1. Those selected ones will constitute A ∗ 𝑖,𝑗 , representing the obtained optimal matching relationship.

4.2.1 Graph construction and Initial Matching. The proposed object matching method is graph-guided. First, for each object 𝑝 in B 𝑖 , we construct a star graph G 𝑝 . Its central node is 𝑝 and the spatial neighbors of 𝑝 form other nodes of this graph, respectively. Each node has a feature vector consisting of the 3DoF pose ( 𝑥,𝑦, 𝜃 ) of the object bounding box. In the same way, for each object 𝑞 in B 𝑗 , a star graph G 𝑞 is constructed. They are illustrated in Figure 3(a).

To conduct initial matching, we transform B 𝑗 from agent 𝑗 's coordinate frame into agent 𝑖 's and establish the initial association using a distance-based method. For the two graphs G 𝑝 and G 𝑞 , we evaluate the spatial distance between their central nodes 𝑝 and 𝑞 , denoted by 𝑑𝑖𝑠 ( 𝑝, 𝑞 ) . When the distance is shorter than a threshold 𝜏 2, G 𝑝 and G 𝑞 will be considered for matching. Note that multiple objects in B 𝑗 could have distances to 𝑝 shorter than 𝜏 2. The one with the minimum distance is chosen to be the initial match as

$$\mathcal { A } _ { i j } ( p ) = \underset { p \in \mathcal { B } _ { i }, q \in \mathcal { B } _ { j } } { \arg \min } \ d i s \left ( p, q \right ) ; \, w h e r e \ d i s \left ( p, q \right ) \leq \tau _ { 2 } \quad ( 3 )$$

4.2.2 Graph Structure Similarity. Logically, if 𝑝 and 𝑞 correspond to the same object, then the graphs G 𝑝 and G 𝑞 constructed from the two objects should have high similarity. Considering this, we design two types of similarity, edge similarity and distance similarity, to

jointly measure the closeness of graphs G 𝑝 and G 𝑞 to assess whether 𝑝 and 𝑞 are from the same object.

Edge Similarity. In graph G 𝑝 , the edge between the central node 𝑝 and one of its spatial neighbors 𝑚 is denoted by 𝑒 𝑝𝑚 . It can be associated with a relative pose transformation from 𝑝 to 𝑚 , denoted by T 𝑝𝑚 . T 𝑝𝑚 is a 3 × 3 matrix obtained through the detection output of the objects 𝑝 and 𝑚 . In the same way, in graph G 𝑞 , the edges 𝑒 𝑞𝑛 between node 𝑞 and its neighbor 𝑛 can be associated with a relative pose transformation T 𝑞𝑛 . If 𝑝 and 𝑞 correspond to the same object, i.e., if G 𝑝 and G 𝑞 have a high similarity, then the edges in two graphs should be consistent. In this case, T 𝑞𝑛 should be consistent with T 𝑝𝑚 . We define the following criterion to evaluate the edge consistency between 𝑒 𝑝𝑚 and 𝑒 𝑞𝑛 :

$$l ( e _ { p m }, e _ { q n } ) = \exp ( - \| T _ { p m } ( T _ { q n } ) ^ { - 1 } - I \| _ { F } ) \quad \quad ( 4 ) \quad \ \xi _ { i \ a } ^ { \ }$$

where ( T 𝑞𝑛 ) -1 is the inverse transformation of 𝑒 𝑞𝑛 , I denotes the identity matrix, ∥ · ∥ F represents the Frobenius norm. The overall edge consistency score between G 𝑝 and G 𝑞 can be defined as:

$$S _ { e d g e } ( p, q ) = \frac { 1 } { | N _ { p } | } \sum \nolimits _ { m \in N _ { p } } l ( e _ { p m }, e _ { q n } ), \ \ s. t. \ \ n = \mathcal { A } _ { i j } ( m ) \ \ ( 5 ) \quad ( \mathcal { A } _ { i j } ( \mathcal { A } _ { i j } )$$

where 𝑁 𝑝 contains the leaf vertices in G 𝑝 that have corresponding vertices in G 𝑞 . Note that the object 𝑛 is the one found to correspond to object 𝑚 in the list A 𝑖 𝑗 .

Distance Similarity. In addition to edge similarity, we can also further quantify the closeness of the two graphs by using distance similarity. This is because if the central nodes 𝑝 and 𝑞 correspond to the same object, the spatial distance between the detection boxes should be sufficiently close, even in the presence of noise and or in a crowded situation. We define the distance similarity of 𝑝 and 𝑞 as:

$$S _ { d i s } \left ( p, q \right ) = \exp \left ( - d i s \left ( p, q \right ) \right )$$

where 𝑑𝑖𝑠 (· , ·) is implemented as a Euclidean distance between the coordinates of the centers of the corresponding bounding boxes. Therefore, the overall graph similarity is obtained by combining the edge consistency and the distance consistency as

$$S ( p, q ) \ = \ S _ { e d g e } ( q, p ) + \lambda S _ { d i s } ( p, q ) \quad \quad ( 7 ) \quad X _ { p }$$

where 𝜆 is the hyperparameter for balancing.

Finally, we utilize the overall similarity 𝑆 to implement the maximization problem in Equation (7) to find the optimal matching. Essentially, this is a bipartite graph problem, where one graph corresponds to all bounding boxes 𝑝 detected by agent 𝑖 , and the other corresponds to all bounding boxes 𝑞 detected by agent 𝑗 . The previously defined similarity 𝑆 ( 𝑝, 𝑞 ) , serves as the edge weight in the bipartite graph. Solving this maximum matching problem can be accomplished using the Kuhn-Munkres algorithm [53].

## 4.3 Robust Pose Graph Optimization

After performing object matching, the elements in the pair of object detection box sets B 𝑖 and B 𝑗 will establish one-to-one correspondences. As shown in Figure 3(b), we then construct a pose graph G V GLYPH&lt;16&gt; 𝑎𝑔𝑒𝑛𝑡 , V 𝑜𝑏𝑗𝑒𝑐𝑡 , E GLYPH&lt;17&gt; for all the agents and the detected objects in the whole scene and use this correspondence to adjust their poses. The node set V 𝑎𝑔𝑒𝑛𝑡 comprises all 𝑁 agents, while the node set V 𝑜𝑏𝑗𝑒𝑐𝑡 consists of all detected objects, the edge set E represents the detection relationships between agents and objects. As mentioned at the beginning of Section 4, there could still be incorrect matching relationships between objects, even after the process proposed in Section 4.2, this will affect the adjustment of the poses of agents. Therefore, we develop an iterative process as what follows in order to achieve robust object matching and pose adjustment.

Firstly, after obtaining the object matching relationship A 𝑖,𝑗 , as in Section 4.2, we will perform pose adjustment guided by this matching relationship; After this, the elements of B 𝑖 and B 𝑗 will be updated; We then use the updated B 𝑖 and B 𝑗 to perform a new round of object matching as in Section 4.2, and then a new round of pose adjustment will be conducted. This iterative matching and adjustment process repeats until the matching result converges, yielding the optimal matching result A ∗ 𝑖,𝑗 and the optimized poses 𝜉 𝑗 and 𝜒 𝑘 . The process is illustrated in Figure 3(a)(b). We model this task as a graph optimization problem. In this pose graph, each node is associated with a pose. The pose of the 𝑗 th agent is denoted as 𝜉 𝑗 , and the pose of the 𝑘 th object is denoted as 𝜒 𝑘 . Let us assume that after the step in Section 4.2, 𝑝 and 𝑞 form a matching pair ( A ( ) 𝑖 𝑗 𝑝 = 𝑞 ) and they are the bounding boxes of the same object.

Figure 4: How the residue errors are set up.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00257v1-figure-dd8303a1-ddfc-4648-a87f-d00ac593edfe.jpg)

Let T 𝑖𝑝 be the measured relative transformation of the 𝑝 th object from the perspective of the 𝑖 th agent, which is naturally obtained through the 𝑖 th agent's detection output; T 𝑗𝑞 is the measured relative transformation of the 𝑞 th object from the perspective of the 𝑗 th agent. From agent 𝑖 's observation, we have T 𝑖𝑝 = E -1 𝑖 X 𝑝 , so X 𝑝 = E T 𝑖 𝑖𝑝 . Note that E 𝑖 is a matrix constructed from 𝜉 𝑖 which denotes the pose of agent i 1 . Similarly, we can get X 𝑞 = E T 𝑗 𝑗𝑞 . In the optimization process, the variables to optimize in this scenario are { 𝜉 𝑗 , 𝜒 𝑘 } . The matched objects are hoped to converge to a consistent result 𝜒 𝑝 = 𝜒 𝑞 = 𝜒 𝑘 . An example is given in Figure 4 to show how the residue errors are set up. We can set up the residue error functions:

$$e _ { i k } = ( \mathbb { E } _ { i } \mathbf T _ { i p } ) ^ { - 1 } X _ { k } = \mathbf T _ { i p } ^ { - 1 } \mathbf E _ { i } ^ { - 1 } X _ { k }$$

$$e _ { j k } = ( \mathbb { E } _ { j } \mathbf T _ { j q } ) ^ { - 1 } X _ { k } = \mathbf T _ { j q } ^ { - 1 } \mathbf E _ { j } ^ { - 1 } X _ { k } \quad \quad ( 9 )$$

$$e _ { j i } = T _ { j i } ^ { - 1 } E _ { j } ^ { - 1 } E _ { i }$$

The overall optimization objective can be given as:

$$\text{ct} _ { n - 1 } \left \{ \xi _ { j } ^ { \prime }, \chi _ { k } ^ { \prime } \right \} = \underset { \{ \xi _ { j }, \chi _ { k } \} } { \arg \min } \sum _ { ( j, k ) } \left ( e _ { i k } ^ { T } \Omega _ { i k } e _ { i k } + e _ { j k } ^ { T } \Omega _ { j k } e _ { i k } + e _ { i j } ^ { T } \Omega _ { i j } e _ { i j } \right ) \text{ } ( 1 1 )$$

1 Given the vector 𝜉 = ( 𝑥, 𝑦, 𝜃 ) , it can be converted into a transformation matrix: E =       𝑐𝑜𝑠 ( 𝜃 ) -𝑠𝑖𝑛 ( 𝜃 ) 𝑥 𝑠𝑖𝑛 ( 𝜃 ) 𝑐𝑜𝑠 ( 𝜃 ) 𝑦 0 0 1       , and the conversion between X and 𝜒 follows the same

  way. Recall that 𝜒 denotes the pose of an object.

Where Ω = diag GLYPH&lt;16&gt; 𝜎 -2 𝑥 , 𝜎 -2 𝑦 , 𝜎 -2 𝜃 GLYPH&lt;17&gt; ∈ R 3 × 3 is the information matrix and its diagonal elements come from the uncertainty estimates of the bounding box, which are part of the detection output from agent 𝑗 . By solving (11) using Levenberg-Marquardt algorithm [23], we get { 𝝃 ′ 𝑗 , 𝑗 = 1 : 𝑁 , } { 𝝌 ′ 𝑘 , 𝑘 = 1 : 𝐾 } . Then we re-do the object matching result:

$$\mathcal { A } _ { i j } ^ { \prime } = \Phi _ { M a t c h } \left ( \left \{ \mathcal { B } _ { i }, \xi _ { i } \right \}, \left \{ \mathcal { B } _ { j }, \xi _ { j } ^ { \prime } \right \} \right ) \quad \ \ ( 1 2 ) \quad \ \ t o$$

In the iterative matching (12) and pose optimization (11) process, we update the position of B 𝑗 using 𝜉 ′ 𝑗 . Our goal is to gradually improve object matching results by minimizing the overall graph optimization error. During the iterative matching process, we fix the poses of self-agent and update the poses of other agents and the objects.

## 4.4 Aggregation and Detection

After the pose calibration, the corrected relative pose from agent 𝑗 to agent 𝑖 is denoted as 𝜉 ′ 𝑗 → 𝑖 = 𝜉 -1 𝑖 ◦ 𝜉 ′ 𝑗 . With the corrected relative pose 𝜉 ′ 𝑗 → 𝑖 , we can adjust the contents of M → 𝑗 𝑖 , feature F 𝑗 from agent 𝑗 are synchronized to the ego pose which has the same coordinate system with its ego feature F 𝑖 . After the spatial alignment, each agent aggregates Φ 𝐴𝑔𝑔 (·) other agents' collaboration information and obtain a more informative feature. This aggregating function can be any common fusion operation. All our experiments adopt Multi-scale feature fusion. After receiving the final fused feature maps, we decode them into the final detection layer Φ 𝐷𝑒𝑐 (·) to obtain final detections O 𝑖 . The regression output is ( 𝑥,𝑦, 𝑧, 𝑤, 𝑙, ℎ, 𝜃 ) , denoting the position, size, yaw angle of the anchor boxes, respectively. The classification output is the confidence score of being an vehicle or background for each anchor box.

## 5 Experimental Result

We conduct extensive experiments on both simulated and realworld scenarios. The task is point-cloud-based 3D object detection. Following the literature, the detection results are evaluated by Average Precision (AP) at Intersection-over-Union (IoU) threshold of 0.50 and 0.70.

## 5.1 Dataset

DAIR-V2X[49]. DAIR-V2X is a public real-world collaborative perception dataset. It contains two agents: vehicle and road-side-unit (RSU) with image resolution 1080 × 1920. The perception range is 𝑥 ∈ [-100 𝑚, 100 𝑚 ,𝑦 ] ∈ [-40 𝑚, 40 𝑚 ] . RSU LiDAR is 300-channel while the vehicle's is 40-channel. V2XSet [45]. V2XSet is a largescale dataset designed for Vehicle-to-Infrastructure (V2X) communication. The data in V2XSet is collected using the simulator CARLA [11] and OpenCDA [31]. It includes LiDAR data captured from multiple autonomous vehicles and roadside infrastructure in various scenarios. The dataset consists of a total of 11,447 frames, with train, validation, test splits of 6,694/1,920/2,833 frames, respectively.

## 5.2 Implementation Details

For encoder backbone, we use PointPillar [22] with the voxel resolution to 0.4m for both height and width. To simulate pose errors, we follow the noisy settings in CoAlign [28] during the training process. We add Gaussian noise 𝑁 GLYPH&lt;0&gt; 0 , 𝜎 2 𝑡 GLYPH&lt;1&gt; on 𝑥,𝑦 and 𝑁 GLYPH&lt;0&gt; 0 , 𝜎 2 𝑟 GLYPH&lt;1&gt; on 𝜃 , where 𝑥,𝑦, 𝜃 are the coordinates of the 2D centers of a vechicle and the yaw angle of accurate global poses. We use the bounding boxes output by the detection framework in reference[28] for each individual agent. In the bipartite graph matching, we set 𝜆 = 1 for Eq.(7) and similarity threshold 𝜏 1 = 0 5 for Eq.(2). In the pose graph . optimization, the Levenberg-Marquardt algorithm[23] is employed to solve the least squares optimization problem, with the maximum number of iterations set to 1000. To better contrast the performance of RoCo, we retrained other methods. We use the Adam [20] with an initial learning rate 0 001 for detection and 0 002 for feature fusion. . . The batchsizes we set for the DAIR-V2X and V2XSet datasets are 4 and 2, respectively. All models are trained on six NVIDIA RTX 2080Ti GPUs with epoch number 30.

## 5.3 Quantitative evaluation

To validate the overall performance of RoCo in 3D object detection, we compare it with a series of previous methods on two datasets. For a fair comparison, all models take 3D point clouds as input data, RoCo is compared with seven state-of-the-art methods: F-Cooper [9], FPV-RCNN[51], V2VNet [40], V2X-ViT[45], Self-ATT, CoAlign [28] and CoBEVFlow[41]. Table 1 shows the AP at IoU threshold of 0.5 and 0.7 in DAIR-V2X and V2XSet dataset. We reference the results from the literatures [28, 41] in the DAIR-V2X and implement them in the V2XSet, and we also implement noise level of 𝜎 2 𝑡 / 𝜎 2 𝑟 = 0 8 . 𝑚 / 0 8 . ◦ on both datasets. We see that RoCo significantly outperforms the previous methods at various noise levels across both datasets. In the real-world dataset DAIR-V2X, RoCo outperforms CoAlign across all noise settings. In the case of noise levels of 0 0 . 𝑚 / 0 0 . ◦ , our approach achieves 1.7% (76.3% 𝑣𝑠. 74.6%) and 1.6% (62.0% 𝑣𝑠. 60.4%) improvement over CoAlign for AP@0.5/0.7. When the noise level becomes as high as 0 6 . 𝑚 / 0 6 . ◦ , our approach achieves 1.9% (71.9% 𝑣𝑠. 70.20%) and 1.2% (58.2% 𝑣𝑠. 57.0%) improvement over CoAlign for AP@0.5/0.7. In the case of noise levels of 0.8/0.8, our approach still achieves 2.3% (71.5% 𝑣𝑠. 69.2%) and 0.9% (57.6% 𝑣𝑠. 56.9%) improvement over CoAlign for AP@0.5/0.7. We conduct experiments on a simulated dataset V2XSet which involves more agents. RoCo still performs well across all noise settings, as shown in Table 1. As the level of noise increases, both methods experience performance degradation. However, our method maintains a higher level of accuracy even under high-level of noise. In the case of noise level of 0 4 . 𝑚 / 0 4 . ◦ , our approach achieves 1.9% (90.0% 𝑣𝑠. 88.1%) and 4.3% (77.3% 𝑣𝑠. 73.0%) improvement over CoAlign for AP@0.5/0.7. When the noise level reaches 0 8 . 𝑚 / 0 8 . ◦ , our approach still achieves 1.4% (84.1% 𝑣𝑠. 82.7%) and 1.6% (68.9% 𝑣𝑠. 67.3%) improvement over CoAlign for AP@0.5/0.7.

## 5.4 Qualitative evaluation

Figure 5 and Figure 6 show the 3D detection results in the BEV format on the V2XSet. Red and green boxes denote the detection results and the ground-truth, respectively. The degree of overlapping of these boxes reflects the performance of the testing methods.

Figure 5 depicts the detection results of V2X-ViT, V2VNet, CoAlign, and the proposed RoCo at an intersection to validate the effectiveness of our method. We set the noise level of 0 4 . 𝑚 / 0 4 . ◦ and 0 8 . 𝑚 / 0 8 . ◦ to produce high pose errors to make the perception

Table 1: 3D object detection performance on DAIR-V2X[49] and V2XSet[45] datasets. All models are trained on pose noises following 𝜎 𝑡 = 0 2 . 𝑚, 𝜎 𝑟 = 0 2 . ◦ . Experiments show that RoCo achieves the best performance under various noise levels.

| Dataset                           | DAIR-V2X   | DAIR-V2X   | DAIR-V2X   | DAIR-V2X   | DAIR-V2X   | V2XSet   | V2XSet   | V2XSet   | V2XSet   | V2XSet   |
|-----------------------------------|------------|------------|------------|------------|------------|----------|----------|----------|----------|----------|
| Method/Metric                     | AP@0.5 ↑   | AP@0.5 ↑   | AP@0.5 ↑   | AP@0.5 ↑   | AP@0.5 ↑   | AP@0.5 ↑ | AP@0.5 ↑ | AP@0.5 ↑ | AP@0.5 ↑ | AP@0.5 ↑ |
| Noise Level 𝜎 𝑡 / 𝜎 𝑟 ( 𝑚 / ◦ ) ) | 0.0/0.0    | 0.2/0.2    | 0.4/0.4    | 0.6/0.6    | 0.8/0.8    | 0.0/0.0  | 0.2/0.2  | 0.4/0.4  | 0.6/0.6  | 0.8/0.8  |
| F-Cooper[9]                       | 73.4       | 72.3       | 70.5       | 69.2       | 67.1       | 78.3     | 76.3     | 71.2     | 65.9     | 62.0     |
| FPV-RCNN[51]                      | 65.5       | 63.1       | 58.0       | 58.1       | 57.5       | 86.5     | 85.3     | 68.7     | 62.1     | 49.5     |
| V2VNet[40]                        | 66.0       | 65.5       | 64.6       | 63.6       | 61.7       | 87.1     | 86.0     | 83.2     | 79.7     | 75.0     |
| Self-Att[47]                      | 70.5       | 70.3       | 69.5       | 68.5       | 67.8       | 87.6     | 86.8     | 85.4     | 83.7     | 82.1     |
| V2X-ViT[45]                       | 70.4       | 70.0       | 68.9       | 67.8       | 66.0       | 91.0     | 90.1     | 86.9     | 84.0     | 81.8     |
| CoAlign[28]                       | 74.6       | 73.8       | 72.0       | 70.0       | 69.2       | 91.9     | 90.9     | 88.1     | 85.5     | 82.7     |
| CoBEVFlow[41]                     | 73.8       | 73.2       | 70.3       | -          | -          | -        | -        | -        | -        | -        |
| Ours (RoCo)                       | 76.3       | 74.8       | 73.3       | 71.9       | 71.5       | 91.9     | 91.0     | 90.0     | 85.9     | 84.1     |

| Method/Metric                     | AP@0.7 ↑   | AP@0.7 ↑   | AP@0.7 ↑   | AP@0.7 ↑   | AP@0.7 ↑   | AP@0.7 ↑   | AP@0.7 ↑   | AP@0.7 ↑   | AP@0.7 ↑   | AP@0.7 ↑   |
|-----------------------------------|------------|------------|------------|------------|------------|------------|------------|------------|------------|------------|
| Noise Level 𝜎 𝑡 / 𝜎 𝑟 ( 𝑚 / ◦ ) ) | 0.0/0.0    | 0.2/0.2    | 0.4/0.4    | 0.6/0.6    | 0.8/0.8    | 0.0/0.0    | 0.2/0.2    | 0.4/0.4    | 0.6/0.6    | 0.8/0.8    |
| F-Cooper[9]                       | 55.9       | 55.2       | 54.2       | 53.8       | 51.6       | 48.6       | 46.0       | 43.4       | 41.0       | 39.5       |
| FPV-RCNN[51]                      | 50.5       | 45.9       | 42.0       | 41.0       | 38.9       | 56.3       | 51.2       | 37.4       | 31.8       | 27.0       |
| V2VNet[40]                        | 48.6       | 48.3       | 47.8       | 47.5       | 38.0       | 64.6       | 62.0       | 56.2       | 50.7       | 44.9       |
| Self-Att[47]                      | 52.2       | 52.0       | 51.7       | 51.4       | 51.1       | 67.6       | 66.2       | 65.1       | 63.9       | 63.0       |
| V2X-ViT[45]                       | 53.1       | 52.9       | 52.5       | 52.2       | 51.3       | 80.3       | 76.8       | 71.8       | 69.0       | 66.6       |
| CoAlign[28]                       | 60.4       | 58.8       | 57.9       | 57.0       | 56.9       | 80.5       | 77.3       | 73.0       | 70.1       | 67.3       |
| CoBEVFlow[41]                     | 59.9       | 57.9       | 56.0       | -          | -          | -          | -          | -          | -          | -          |
| Ours (RoCo)                       | 62.0       | 59.4       | 58.4       | 58.2       | 57.8       | 80.5       | 77.4       | 77.3       | 71.0       | 68.9       |

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00257v1-figure-773017cf-2bea-4e71-ac63-01382eef11e3.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00257v1-figure-b9d5c017-a794-463a-a20a-b6e97be1f745.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00257v1-figure-cb01b383-f0a8-452d-856f-ac86609987e1.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00257v1-figure-4d1909c1-2988-42a0-8dfb-67c7eb9d7ad2.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00257v1-figure-e77ef59e-e269-4c62-89a9-398f24b87ea6.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00257v1-figure-9c46c64a-1975-4239-b514-bbdf1ae69155.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00257v1-figure-66f93c27-06d2-458e-bc81-44e01a9c9b56.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00257v1-figure-0b5e73b4-1854-415c-9f1a-0a0d020a6ecf.jpg)

(a)

V2VNet [40]

(b) V2X-VIT [45]

(c) CoAlign [28]

(d) RoCo (Ours)

Figure 5: Visualization of detection results for V2VNet, V2X-ViT, CoAlign and our RoCo with the noisy level 𝜎 2 𝑡 / 𝜎 2 𝑟 ( 𝑚 / ◦ ) of 0.4/0.4 (the first row), and 0.8/0.8 (the second row) on V2XSet dateset. An intersection scenario is given. RoCo qualitatively outperforms the others under different noisy level.

task challenging. From the figure, it can be observed that V2VNet has many missed detections, while V2X-ViT and CoAlign generate many predictions with relatively large offsets. In contrast, our RoCo demonstrates strong performance high pose errors. Figure 6 visualize the box position with and without graph matching in crowded scenes. Without the proposed graph matching, show in figure 6(a), there are many false detections. In comparison, each detection box in Figure 6(b) corresponds to a unique object, exhibiting more robust performance. In crowded environments, objects are relatively close and with high pose errors, the detected bounding boxes may overlap. RoCo utilizes graph-based object matching to correctly associate objects detected by different agents, ensuring that each object has only one correct detection box.

Table 2: Selection of the threshold value 𝜏 2 in Eq. (3).

|                  | DAIR-V2X   | DAIR-V2X   | DAIR-V2X   | DAIR-V2X   | DAIR-V2X   | V2XSet   | V2XSet   | V2XSet   |
|------------------|------------|------------|------------|------------|------------|----------|----------|----------|
| Threshold/Metric | AP@0.7     | AP@0.7     | AP@0.7     | AP@0.7     | AP@0.7     | AP@0.7   | AP@0.7   | AP@0.7   |
| Noise Level      | 0.2/0.2    | 0.4/0.4    | 0.6/0.6    | 0.8/0.8    | 0.2/0.2    | 0.4/0.4  | 0.6/0.6  | 0.8/0.8  |
| 𝜏 2 = 1          | 58.8       | 57.6       | 56.9       | 56.4       | 77.1       | 72.5     | 70.4     | 65.7     |
| 𝜏 2 = 2          | 58.9       | 57.9       | 57.2       | 56.9       | 77.4       | 77.3     | 71.0     | 68.1     |
| 𝜏 2 = 3          | 59.4       | 58.4       | 58.2       | 57.8       | 77.1       | 73.1     | 70.2     | 68.9     |
| 𝜏 2 = 4          | 57.9       | 57.6       | 57.6       | 57.5       | 75.9       | 72.0     | 69.7     | 68.3     |

Table 3: Contribution of graph similarity.

| Modules   | Modules   | Modules   | AP@0.5   | AP@0.5   | AP@0.5   | AP@0.7   | AP@0.7   | AP@0.7   |
|-----------|-----------|-----------|----------|----------|----------|----------|----------|----------|
| Matching  | distance  | edge      | 0.2/0.2  | 0.4/0.4  | 0.8/0.8  | 0.2/0.2  | 0.4/0.4  | 0.8/0.8  |
| ×         | ×         | ×         | 73.8     | 72.0     | 69.2     | 58.8     | 57.9     | 56.9     |
| ✓         | ✓         | ×         | 73.0     | 71.0     | 70.8     | 58.8     | 58.0     | 57.5     |
| ✓         | ×         | ✓         | 73.3     | 72.1     | 70.9     | 59.0     | 58.1     | 57.6     |
| ✓         | ✓         | ✓         | 74.8     | 73.3     | 71.5     | 59.4     | 58.4     | 57.8     |

(a) Boxes w/o Graph matching

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00257v1-figure-366677df-00f1-4fc0-bd75-89799e6cc747.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00257v1-figure-d82a6d8c-fc2c-4099-a2eb-9ab6b53f34d3.jpg)

(b) Boxes w/ Graph matching

Figure 6: 3D detection results in congested traffic scenario.

Table 4: Ablation study on DIAR-V2X.

| Modules   | Modules             | AP@0.5   | AP@0.5   | AP@0.5   | AP@0.7   | AP@0.7   | AP@0.7   |
|-----------|---------------------|----------|----------|----------|----------|----------|----------|
| Matching  | Robust Optimization | 0.2/0.2  | 0.4/0.4  | 0.8/0.8  | 0.2/0.2  | 0.4/0.4  | 0.8/0.8  |
| ×         | ×                   | 73.8     | 72.0     | 69.2     | 58.8     | 57.9     | 56.9     |
| ✓         | ×                   | 74.4     | 72.9     | 71.0     | 59.1     | 58.2     | 57.5     |
| ✓         | ✓                   | 74.8     | 73.3     | 71.5     | 59.4     | 58.4     | 57.8     |

## 5.5 Ablation Studies

Selection of the threshold value 𝜏 2 . The selection of the initial candidate set in graph matching is crucial. To determine the optimal threshold 𝜏 2 in Eq.(3) during the graph initialization, we conduct ablation experiments on different datasets, as shown in Table 2. In the DAIR-V2X dataset, we found that the detection results reach their optimum when 𝝉 2 is set to 3. As the value of 𝜏 2 increases, the detection performance will decline. This is because vehicles usually maintain a safe distance from other vehicles in real-world scenarios, and too small a threshold would affect the initial matching. Furthermore, we conducted the same experiments on the simulated dataset. We observed that when the noise level is below 0 8 . 𝑚 / 0 8 . ◦ , the optimal threshold 𝝉 2 becomes 2 meters. As the noise level increases to 0 8 . 𝑚 / 0 8 . ◦ , the optimal threshold becomes 𝝉 2 = 3. This is because in the V2XSet dataset, there are more vehicles and they are densely distributed. As a result, the distances between vehicles are smaller compared to real-world scenarios.

Contribution of Object Matching and Robust Graph Optimization. Table 4 assesses the effectiveness of the proposed module

Zhe Huang et al.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00257v1-figure-5108a2e7-0488-49e9-b25d-a0d6d069683a.jpg)

Figure 7: Probability density function (PDF) of relative pose error distribution on V2XSet dataset.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00257v1-figure-46baf261-08ea-4eb1-97ec-1af9e1ca6654.jpg)

at various noise levels, including object matching and robust graph optimization. We evaluate the impact of each module on the final 3D detection by incrementally adding i) object matching and ii) robust graph optimization. We see that both modules can improve the performance. To validate the impact of object matching on pose error, Figure 7 plots the distribution of relative pose errors across all samples. The horizontal axes "trans error" and "rot error" denote the translational and rotational relative pose errors (RPE), respectively. The vertical axis represents the density value of the distribution. The orange dashed line represents the density when applying the proposed object matching method, while the blue line represents case without the object matching. The closer the distribution is to the Delta distribution centered at zero, the better the performance because this means most of the pose errors have smaller values. It can be observed from figure that using object matching leads to the desired distribution and significantly reduces relative pose errors.

Contribution of graph similarity. Nowweinvestigate the contributions of different similarities defined in Eq.(5) and (6) to object matching on DARI-V2X, the result is shown in Table 3. We evaluate the impact of each similarity score on the final graph matching by incrementally adding i) edge similarity and ii) distance similarity. The first row in Table 3 represents the baseline model which does not use any matching methods. The experimental results indicate that both designs of similarity matching contribute to improving detection accuracy, especially at the noise level of 0 8 . 𝑚 / 0 8 . ◦ .

## 6 Conclusion

This paper proposes a novel robust collaborative perception framework, RoCo , for 3D object detection. This framework addresses issues such as object mismatch and misalignment caused by pose errors among multiple agents. It enhances the precision and reliability of this modality and better prepare it for the potential multimodal fusion step in the sequel. The core idea of RoCo is based on graph-based object matching, which reliably associates common objects detected by different agents and reduces pose errors of agents and objects using iterative robust optimization. Additionally, RoCo does not require any ground truth pose supervision during training, making it highly practical. Comprehensive experiments demonstrate that RoCo achieves outstanding performance across all settings and exhibits superior robustness under extreme noise conditions. We believe that this work has the potential to contribute to the development of the multimodal field. In future work, we will apply our method to multimodal collaborative perception to further improve the safety and reliability of autonomous driving.

## Acknowledgments

This work was supported by the China Scholarship Council (CSC); Dr. Li is supported in part by the National Natural Science Foundation of China Grant No. 12071478. Dr. Wang is supported in part by the National Natural Science Foundation of China Grant No. 61972404, Public Computing Cloud, Renmin University of China, and the Blockchain Lab, School of Information, Renmin University of China.

## References

- [1] Eduardo Arnold, Omar Y Al-Jarrah, Mehrdad Dianati, Saber Fallah, David Oxtoby, and Alexandros Mouzakitis. 2021. Data for Cooperative object classification for driving applications. (2021).
- [2] Nikolay Atanasov, Menglong Zhu, Kostas Daniilidis, and George J Pappas. 2016. Localization from semantic observations via the matrix permanent. The International Journal of Robotics Research 35, 1-3 (2016), 73-99.
- [3] Josep Aulinas, Yvan Petillot, Joaquim Salvi, and Xavier Lladó. 2008. The SLAM problem: a survey. Artificial Intelligence Research and Development (2008), 363371.
- [4] AymanBeghdadi and Malik Mallem. 2022. A comprehensive overview of dynamic visual SLAM and deep learning: concepts, methods and challenges. Machine Vision and Applications 33, 4 (2022), 54.
- [5] Lukas Bernreiter, Abel Gawel, Hannes Sommer, Juan Nieto, Roland Siegwart, and Cesar Cadena Lerma. 2019. Multiple hypothesis semantic mapping for robust data association. IEEE Robotics and Automation Letters 4, 4 (2019), 3255-3262.
- [6] Alex Bewley, Zongyuan Ge, Lionel Ott, Fabio Ramos, and Ben Upcroft. 2016. Simple online and realtime tracking. In 2016 IEEE international conference on image processing (ICIP) . IEEE, 3464-3468.
- [7] Sean L Bowman, Nikolay Atanasov, Kostas Daniilidis, and George J Pappas. 2017. Probabilistic data association for semantic slam. In 2017 IEEE international conference on robotics and automation (ICRA) . IEEE, 1722-1729.
- [8] Xudong Cai, Yongcai Wang, Zhe Huang, Yu Shao, and Deying Li. 2024. VOLoc: Visual Place Recognition by Querying Compressed Lidar Map. arXiv preprint arXiv:2402.15961 (2024).
- [9] Qi Chen, Xu Ma, Sihai Tang, Jingda Guo, Qing Yang, and Song Fu. 2019. F-cooper: Feature based cooperative perception for autonomous vehicle edge computing system using 3D point clouds. In Proceedings of the 4th ACM/IEEE Symposium on Edge Computing . 88-100.
- [10] Siheng Chen, Baoan Liu, Chen Feng, Carlos Vallespi-Gonzalez, and Carl Wellington. 2020. 3d point cloud processing and learning for autonomous driving: Impacting map creation, localization, and perception. IEEE Signal Processing Magazine 38, 1 (2020), 68-86.
- [11] Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. 2017. CARLA: An open urban driving simulator. In Conference on robot learning . PMLR, 1-16.
- [12] Nathaniel Glaser, Yen-Cheng Liu, Junjiao Tian, and Zsolt Kira. 2021. Overcoming obstructions via bandwidth-limited multi-agent spatial handshaking. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) . IEEE, 2406-2413.
- [13] Nathaniel Glaser, Yen-Cheng Liu, Junjiao Tian, and Zsolt Kira. 2021. Overcoming obstructions via bandwidth-limited multi-agent spatial handshaking. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) . IEEE, 2406-2413.
- [14] Jiaming Gu, Jingyu Zhang, Muyang Zhang, Weiliang Meng, Shibiao Xu, Jiguang Zhang, and Xiaopeng Zhang. 2023. FeaCo: Reaching Robust Feature-Level Consensus in Noisy Pose Conditions. In Proceedings of the 31st ACM International Conference on Multimedia . 3628-3636.
- [15] John E Hopcroft and Richard M Karp. 1973. An nˆ5/2 algorithm for maximum matchings in bipartite graphs. SIAM Journal on computing 2, 4 (1973), 225-231.
- [16] Yue Hu, Shaoheng Fang, Zixing Lei, Yiqi Zhong, and Siheng Chen. 2022. Where2comm: Communication-efficient collaborative perception via spatial confidence maps. Advances in neural information processing systems 35 (2022), 48744886.
- [17] Yue Hu, Shaoheng Fang, Weidi Xie, and Siheng Chen. 2023. Aerial monocular 3d object detection. IEEE Robotics and Automation Letters 8, 4 (2023), 1959-1966.
- [18] Zhe Huang, Yongcai Wang, Xingui Tang, and Hongyu Sun. 2023. Boundary-aware set abstraction for 3D object detection. In 2023 International Joint Conference on Neural Networks (IJCNN) . IEEE, 01-07.
- [19] Zhe Huang, Yongcai Wang, Jie Wen, Peng Wang, and Xudong Cai. 2023. An object detection algorithm combining semantic and geometric information of the 3D point cloud. Advanced Engineering Informatics 56 (2023), 101971.
- [20] Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).
- [21] Xianghao Kong, Wentao Jiang, Jinrang Jia, Yifeng Shi, Runsheng Xu, and Si Liu. 2023. Dusa: Decoupled unsupervised sim2real adaptation for vehicle-toeverything collaborative perception. In Proceedings of the 31st ACM International Conference on Multimedia . 1943-1954.
- [22] Alex H Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom. 2019. Pointpillars: Fast encoders for object detection from point clouds. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 12697-12705.
- [23] Kenneth Levenberg. 1944. A method for the solution of certain non-linear problems in least squares. Quarterly of applied mathematics 2, 2 (1944), 164-168.
- [24] Yiming Li, Qi Fang, Jiamu Bai, Siheng Chen, Felix Juefei-Xu, and Chen Feng. 2023. Among us: Adversarially robust collaborative perception by consensus. arXiv preprint arXiv:2303.09495 (2023).
- [25] Yiming Li, Dekun Ma, Ziyan An, Zixun Wang, Yiqi Zhong, Siheng Chen, and Chen Feng. 2022. V2X-Sim: Multi-agent collaborative perception dataset and benchmark for autonomous driving. IEEE Robotics and Automation Letters 7, 4 (2022), 10914-10921.
- [26] Yiming Li, Juexiao Zhang, Dekun Ma, Yue Wang, and Chen Feng. 2023. Multirobot scene completion: Towards task-agnostic collaborative perception. In Conference on Robot Learning . PMLR, 2062-2072.
- [27] Yifan Lu, Yue Hu, Yiqi Zhong, Dequan Wang, Siheng Chen, and Yanfeng Wang. 2024. An Extensible Framework for Open Heterogeneous Collaborative Perception. arXiv preprint arXiv:2401.13964 (2024).
- [28] Yifan Lu, Quanhao Li, Baoan Liu, Mehrdad Dianati, Chen Feng, Siheng Chen, and Yanfeng Wang. 2023. Robust collaborative 3d object detection in presence of pose errors. In 2023 IEEE International Conference on Robotics and Automation (ICRA) . IEEE, 4812-4818.
- [29] Donghao Qiao and Farhana Zulkernine. 2023. CoBEVFusion: Cooperative Perception with LiDAR-Camera Bird's-Eye View Fusion. arXiv preprint arXiv:2310.06008 (2023).
- [30] Akshay Raina and Vipul Arora. 2022. SyncNet: Using Causal Convolutions and Correlating Objective for Time Delay Estimation in Audio Signals. arXiv preprint arXiv:2203.14639 (2022).
- [31] Mike Rosenman and Fujun Wang. 2001. A component agent based open CAD system for collaborative design. Automation in Construction 10, 4 (2001), 383-397.
- [32] Muhamad Risqi U Saputra, Andrew Markham, and Niki Trigoni. 2018. Visual SLAM and structure from motion in dynamic environments: A survey. ACM Computing Surveys (CSUR) 51, 2 (2018), 1-36.
- [33] Takafumi Taketomi, Hideaki Uchiyama, and Sei Ikeda. 2017. Visual SLAM algorithms: A survey from 2010 to 2016. IPSJ Transactions on Computer Vision and Applications 9, 1 (2017), 1-11.
- [34] Xuebo Tian, Zhongyang Zhu, Junqiao Zhao, Gengxuan Tian, and Chen Ye. 2022. DL-SLOT: Dynamic LiDAR SLAM and object tracking based on collaborative graph optimization. arXiv preprint arXiv:2212.02077 (2022).
- [35] Nicholas Vadivelu, Mengye Ren, James Tu, Jingkang Wang, and Raquel Urtasun. 2021. Learning to communicate and correct pose errors. In Conference on Robot Learning . PMLR, 1195-1210.
- [36] Jingwen Wang, Martin Rünz, and Lourdes Agapito. 2021. DSP-SLAM: Object oriented SLAM with deep shape priors. In 2021 International Conference on 3D Vision (3DV) . IEEE, 1362-1371.
- [37] Shuo Wang, Yongcai Wang, Xuewei Bai, and Deying Li. 2023. Communication Efficient, Distributed Relative State Estimation in UAV Networks. IEEE Journal on Selected Areas in Communications 41, 4 (2023), 1151-1166. https://doi.org/10. 1109/JSAC.2023.3242708
- [38] Shuo Wang, Yongcai Wang, Deying Li, and Qianchuan Zhao. 2023. Distributed Relative Localization Algorithms for Multi-Robot Networks: A Survey. Sensors 23, 5 (2023). https://doi.org/10.3390/s23052399
- [39] Tianhang Wang, Guang Chen, Kai Chen, Zhengfa Liu, Bo Zhang, Alois Knoll, and Changjun Jiang. 2023. UMC: A Unified Bandwidth-efficient and Multi-resolution based Collaborative Perception Framework. arXiv preprint arXiv:2303.12400 (2023).
- [40] Tsun-Hsuan Wang, Sivabalan Manivasagam, Ming Liang, Bin Yang, Wenyuan Zeng, and Raquel Urtasun. 2020. V2vnet: Vehicle-to-vehicle communication for joint perception and prediction. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16 . Springer, 605-621.
- [41] Sizhe Wei, Yuxi Wei, Yue Hu, Yifan Lu, Yiqi Zhong, Siheng Chen, and Ya Zhang. 2023. Asynchrony-Robust Collaborative Perception via Bird's Eye View Flow. arXiv e-prints (2023), arXiv-2309.
- [42] Hao Xiang, Runsheng Xu, Xin Xia, Zhaoliang Zheng, Bolei Zhou, and Jiaqi Ma. 2023. V2xp-asg: Generating adversarial scenes for vehicle-to-everything perception. In 2023 IEEE International Conference on Robotics and Automation (ICRA) . IEEE, 3584-3591.
- [43] Li Xiang, Junbo Yin, Wei Li, Cheng-Zhong Xu, Ruigang Yang, and Jianbing Shen. 2023. DI-V2X: Learning Domain-Invariant Representation for VehicleInfrastructure Collaborative 3D Object Detection. arXiv preprint arXiv:2312.15742 (2023).

- [44] Runsheng Xu, Zhengzhong Tu, Hao Xiang, Wei Shao, Bolei Zhou, and Jiaqi Ma. 2022. CoBEVT: Cooperative bird's eye view semantic segmentation with sparse transformers. arXiv preprint arXiv:2207.02202 (2022).
- [45] Runsheng Xu, Hao Xiang, Zhengzhong Tu, Xin Xia, Ming-Hsuan Yang, and Jiaqi Ma. 2022. V2x-vit: Vehicle-to-everything cooperative perception with vision transformer. In European conference on computer vision . Springer, 107-124.
- [46] Runsheng Xu, Hao Xiang, Xin Xia, Xu Han, Jinlong Li, and Jiaqi Ma. 2022. Opv2v: An open benchmark dataset and fusion pipeline for perception with vehicle-tovehicle communication. In 2022 International Conference on Robotics and Automation (ICRA) . IEEE, 2583-2589.
- [47] Runsheng Xu, Hao Xiang, Xin Xia, Xu Han, Jinlong Li, and Jiaqi Ma. 2022. Opv2v: An open benchmark dataset and fusion pipeline for perception with vehicle-tovehicle communication. In 2022 International Conference on Robotics and Automation (ICRA) . IEEE, 2583-2589.
- [48] Kun Yang, Dingkang Yang, Jingyu Zhang, Hanqi Wang, Peng Sun, and Liang Song. 2023. What2comm: Towards communication-efficient collaborative perception via feature decoupling. In Proceedings of the 31st ACM International Conference on Multimedia . 7686-7695.
- [49] Haibao Yu, Yizhen Luo, Mao Shu, Yiyi Huo, Zebang Yang, Yifeng Shi, Zhenglong Guo, Hanyu Li, Xing Hu, Jirui Yuan, et al. 2022. Dair-v2x: A large-scale dataset for vehicle-infrastructure cooperative 3d object detection. In Proceedings of the

IEEE/CVF Conference on Computer Vision and Pattern Recognition . 21361-21370.

- [50] Yunshuang Yuan, Hao Cheng, and Monika Sester. 2022. Keypoints-based deep feature fusion for cooperative vehicle detection of autonomous driving. IEEE Robotics and Automation Letters 7, 2 (2022), 3054-3061.
- [51] Yunshuang Yuan, Hao Cheng, and Monika Sester. 2022. Keypoints-based deep feature fusion for cooperative vehicle detection of autonomous driving. IEEE Robotics and Automation Letters 7, 2 (2022), 3054-3061.
- [52] Hanwei Zhang, Hideaki Uchiyama, Shintaro Ono, and Hiroshi Kawasaki. 2022. MOTSLAM: MOT-assisted monocular dynamic SLAM using single-view depth estimation. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) . IEEE, 4865-4872.
- [53] Haibin Zhu, MengChu Zhou, and Rob Alkins. 2011. Group role assignment via a Kuhn-Munkres algorithm-based solution. IEEE Transactions on Systems, Man, and Cybernetics-Part A: Systems and Humans 42, 3 (2011), 739-750.
- [54] Zhongyang Zhu, Junqiao Zhao, Xuebo Tian, Kai Huang, and Chen Ye. 2023. LIMOT: A Tightly-Coupled System for LiDAR-Inertial Odometry and MultiObject Tracking. arXiv preprint arXiv:2305.00406 (2023).
- [55] Danping Zou, Ping Tan, and Wenxian Yu. 2019. Collaborative visual SLAM for multiple agents: A brief survey. Virtual Reality &amp; Intelligent Hardware 1, 5 (2019), 461-482.
Zhe Huang Shuo Wang Yongcai Wang Wanting Li Deying Li Lei Wang
RoCo:Robust Collaborative Perception By Iterative Object Matching and Pose Adjustment
Collaborative autonomous driving with multiple vehicles usually requires the
data fusion from multiple modalities. To ensure effective fusion, the data from
each individual modality shall maintain a reasonably high quality. However, in
collaborative perception, the quality of object detection based on a modality
is highly sensitive to the relative pose errors among the agents. It leads to
feature misalignment and significantly reduces collaborative performance. To
address this issue, we propose RoCo, a novel unsupervised framework to conduct
iterative object matching and agent pose adjustment. To the best of our
knowledge, our work is the first to model the pose correction problem in
collaborative perception as an object matching task, which reliably associates
common objects detected by different agents. On top of this, we propose a graph
optimization process to adjust the agent poses by minimizing the alignment
errors of the associated objects, and the object matching is re-done based on
the adjusted agent poses. This process is carried out iteratively until
convergence. Experimental study on both simulated and real-world datasets
demonstrates that the proposed framework RoCo consistently outperforms existing
relevant methods in terms of the collaborative object detection performance,
and exhibits highly desired robustness when the pose information of agents is
with high-level noise. Ablation studies are also provided to show the impact of
its key parameters and components. The code is released at
https://github.com/HuangZhe885/RoCo.