## CONFORMAL TRAJECTORY PREDICTION WITH MULTI-VIEW DATA INTEGRATION IN COOPERATIVE DRIVING

Xi Chen 1 , Rahul Bhadani 2 , and Larry Head 1

1 The University of Arizona, Tucson, USA 2 The University of Alabama in Huntsville, Huntsville, USA

## ABSTRACT

Current research on trajectory prediction primarily relies on data collected by onboard sensors of an ego vehicle. With the rapid advancement in connected technologies, such as vehicle-to-vehicle (V2V) and vehicle-to-infrastructure (V2I) communication, valuable information from alternate views becomes accessible via wireless networks. The integration of information from alternative views has the potential to overcome the inherent limitations associated with a single viewpoint, such as occlusions and limited field of view. In this work, we introduce V2INet, a novel trajectory prediction framework designed to model multi-view data by extending existing single-view models. Unlike previous approaches where the multi-view data is manually fused or formulated as a separate training stage, our model supports end-to-end training, enhancing both flexibility and performance. Moreover, the predicted multimodal trajectories are calibrated by a post hoc conformal prediction module to get valid and efficient confidence regions. We evaluated the entire framework on the real-world V2I dataset V2X-Seq. Our results demonstrate superior performance in terms of Final Displacement Error (FDE) and Miss Rate (MR) using a single GPU. The code is publicly available at: https://github.com/xichennn/V2I\_trajectory\_prediction .

## 1 Introduction

Trajectory prediction plays a critical role in autonomous driving. Typically, we rely on the on-board sensors of an ego vehicle to gather surrounding information necessary for performing various autonomous driving tasks. However, with the rapid advancement in connected technologies, such as vehicle-to-vehicle (V2V) and vehicle-to-infrastructure (V2I) communication, valuable information from alternate views becomes accessible via wireless networks. The integration of information from alternative views has the potential to overcome the inherent limitations associated with a single viewpoint, such as occlusions and limited field of view. We categorize the data obtained solely from ego vehicle on-board sensors or infrastructure as single-view information, whereas the accessibility to multiple viewpoints is referred to as multi-view data. Figure 1 depicts an intersection scenario where the AV faces a potential left-turn collision due to its field of view being obstructed by a large truck. Roadside cameras, strategically positioned to have an unobstructed view of the entire intersection, provide a comprehensive overview of the traffic situation. This enhanced perspective allows the AV to receive critical information and effectively avoid the impending collision. From a trajectory predictive modeling perspective, significant research efforts have been dedicated to only to use single-view datasets collected by the ego vehicle [1, 2, 3, 4, 5]. These efforts typically involve modeling the temporal dependencies, agent-agent interactions, and agent-lane relations. However, the challenge arises when dealing with multi-view data, particularly in how to effectively fuse the information due to overlapping field of views.

Existing work on multi-view data fusion in the context of cooperative driving predominantly focuses on collaborative perception tasks, with limited research addressing trajectory prediction. [6] pioneered the creation of the first V2I real-world dataset for trajectory prediction studies. They manually fused multi-view data using trajectory association and stitching techniques at each frame, followed by the application of single-view trajectory prediction models. Although this approach is intuitive, it fails to fully exploit the motion behavior captured by data from each view, resulting in suboptimal outcomes. To address this limitation, [7] proposed a novel approach that encodes trajectory information from each view as independent graph nodes, thereby minimizing information loss. They formulated the node association

(a) potential left-turn collision

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00374v3-figure-563721b1-d703-439a-b1a2-951db4e95a30.jpg)

(c) infrastructure view

Figure 1: Motivational scenarios. AV is in orange. (a) The AV is attempting a left turn and is at risk of a potential collision with an oncoming vehicle going straight (b) The A V's onboard sensors have their field of view obstructed by large trucks or other vehicles, limiting their ability to detect the oncoming traffic. (c) The roadside cameras' view of the intersection. They are positioned to have an unobstructed view of the entire intersection, providing a complete picture of the traffic situation.

problem as a graph link prediction task and introduced a cross-attention module to fuse node embeddings from associated nodes across different views. While their model can be trained end-to-end, the node association process necessitates pretraining, leading to the formulation of two optimization objectives.

We introduce a trajectory prediction framework that utilizes multi-view data without the need for explicit association between different perspectives. Rather than developing a specialized multi-view model, our approach seamlessly integrates with state-of-the-art single-view trajectory models, maximizing the utility of existing research efforts. No special training strategies are required. We can easily take advantage of the pretrained single-view trajectory models to expedite the training. Specifically, for trajectory data from each single-view, we employ established graph neural network (GNN) based models, such as LaneGCN [1], HiVT [3], to capture temporal dependencies, agent-agent interactions, and agent-lane relations. Then we utilize a cross-graph attention module to fuse the node embeddings from different views. The fused final embeddings will then go through a multimodal decoder to get future trajectory predictions.

Existing works have modeled the multi-modality explicitly by introducing anchors [2, 8, 9], mixture models [10, 11, 12, 13, 1, 3], or implicitly through latent variables such as Conditional Variational Auto-encoder (CV AE) or generative models [14, 15, 16]. The implicit models often face the issue of mode collapse, therefore we will model the multimodality by a mixture model built upon MLP. It is a common strategy to assign a higher score to the modality closer to the ground truth during training. The strategy, however, may encounter robustness issues during inference. Figure 2 presents two examples where the top-scored prediction is not the closest to the ground truth trajectory.

The situation arises from the inherent randomness in the ground truth data, posing challenges for the model to learn the priorities of multiple predictions. Relying on the scores to rank the predictions can lead to hazardous results in safetycritical scenarios. Additionally, while most mixture models like the Gaussian Mixture Model (GMM) provide both point estimation and variance estimation, the reliability of these uncertainty quantifications (UQ) remains questionable. Unfortunately, this aspect is often overlooked in the model evaluation process. However, UQ can offer valuable insights for downstream decision-making processes. To address these challenges, we introduce conformal prediction (CP) [17, 18], as a post-hoc module to our framework. CP creates statistically rigorous uncertainty intervals for the predictions that are guaranteed to contain the ground truth with a user-specified probability.

To summarize, our contributions in this work lie in twofolds: 1) We introduce V2INet, a novel trajectory prediction framework designed to model multi-view data by extending existing single-view models. Unlike previous work, our model supports end-to-end training, enhancing both flexibility and performance. 2) We introduce conformal prediction as a post-hoc module to calibrate the prediction results. This results in statistically rigorous uncertainty intervals, significantly enhancing the reliability of the predictions. These calibrated predictions are particularly beneficial for downstream decision-making tasks, such as motion planning.

Figure 2: Inaccurate inference scores. The ground truth trajectory is represented in red, while predictions with lower scores are depicted in lighter shades of blue

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00374v3-figure-40fd7254-03fe-4556-8a3c-cb9c4831b7ac.jpg)

## 2 Related Work

## 2.1 Cooperative Driving Dataset

Collaborative perception has been the most extensively studied area in collaborative driving [19, 20, 21, 22]. It uses the recent developments in wireless communication technologies, such as V2V and V2I, to share information, which enables perception beyond line-of-sight and field-of-view hence overcoming common perceptual shortcomings with individual perception, such as blindspot, occlusions, and long-range issues [23, 24]. Many datasets have been collected in both simulation environment [25, 26, 27] and real-world [28, 29] to facilitate the study. A comprehensive comparison can be found in Table 1. While a majority of them focused on the upstream tasks such as detection, tracking, and segmentation, to enable well-informed decision-making for autonomous vehicles, however, it's critical to also incorporate V2X data for predicting the behavior of surrounding traffic participants. V2X-Seq/Forecasting [6] and V2X-Traj [7] present two real-world datasets specifically designed for cooperative trajectory prediction tasks. The experiments in V2X-Seq/Forecasting demonstrate that leveraging the infrastructure-side trajectories can enhance the trajectory prediction performance. V2X-Traj aims for more general V2X scenarios, extending to V2V cooperation.

Table 1: Comparison of Cooperative Driving Datasets. The compared tasks include detection (Det.), tracking (Track), segmentation (Seg.) and trajectory prediction (Pred). Only the tasks performed in the original paper are reported. "-" means the number was not reported."*" means expected.

| Dataset             | Year   | Source        | Scenario   | Tasks   | Tasks   | Tasks   | Tasks   | Maps   | Total Time   | Traffic Signal   | Scenes   |
|---------------------|--------|---------------|------------|---------|---------|---------|---------|--------|--------------|------------------|----------|
| Dataset             | Year   | Source        | Scenario   | Det.    | Track.  | Seg.    | Pred.   | Maps   | Total Time   | Traffic Signal   | Scenes   |
| Cooper(inf)         | 2019   | CARLA         | V2I        | ✓       | x       | x       | x       | x      | -            | x                | <100     |
| DAIR-V2X-C          | 2021   | Real-world    | V2I        | ✓       | x       | x       | x       | ✓      | 0.5h         | x                | 100      |
| OPV2V               | 2021   | CARLA+OpenCDA | V2V        | ✓       | x       | x       | x       | x      | 0.2h         | x                | 73       |
| V2X-Sim             | 2022   | CARLA+SUMO    | V2V&I      | ✓       | ✓       | ✓       | x       | x      | 0.3h         | x                | 100      |
| V2V4Real            | 2023   | Real-world    | V2V        | ✓       | ✓       | x       | x       | ✓      | 19h          | x                | 67       |
| V2X-Seq/Perception  | 2023   | Real-world    | V2I        | x       | ✓       | x       | x       | ✓      | 0.43h        | x                | 95       |
| V2X-Seq/Forecasting | 2023   | Real-world    | V2I        | x       | x       | x       | ✓       | ✓      | 583h         | ✓                | 210000   |
| V2X-Traj            | 2024*  | Real-world    | V2V&I      | x       | x       | x       | ✓       | ✓      | -            | ✓                | 6160     |

## 2.2 Cooperative Information Association and Fusion

The main challenge in cooperative trajectory prediction lies in the multi-view data source fusion. In cooperative scenarios, vehicles gather safety-related data using sensors like radar, lidar, cameras, and GPS. This data is standardized into Basic Safety Message (BSM) format, ensuring compatibility across vehicles and infrastructure [30]. BSM messages, containing crucial information such as position and speed, are broadcasted periodically. Upon receiving BSM messages, vehicles combine this data with their own sensor data to enhance accuracy. Advanced algorithms have been developed to associate and fuse the multi-source data. Given the temporal and spatial dimensions of the collected trajectories, there has been research focusing on the communication delays alignment [31, 23, 32] and pose errors alignment

[33, 34, 35, 23]. Based on the aligned data, [6] utilized CBMOT [36], a multi-object tracking method, to fuse the infrastructure and ego-vehicle trajectories at each single frame and then trained the network taking in the fused dataset. While straightforward, this method fails to capture the motion behavior provided by the infrastructure across all time steps, leading to suboptimal results. In contrast, [7] encoded trajectory information from each view as independent graph nodes, formulating the association process as a graph linking problem. While effective, this approach necessitates separate training procedures. Building upon their work, we propose the utilization of a cross-graph attention mechanism to fuse multi-view information, eliminating the need for additional training processes.

## 2.3 Uncertainty Quantification

In the trajectory prediction task, numerous sources of uncertainty exist, including inherent multi-modality, partial observability, short time scales, data limitations, intention type imbalances, and domain gaps. Most existing trajectory prediction models address uncertainty by maximizing the likelihood of an assumed distribution, such as Gaussian or Laplace [9, 3, 1, 11]. However, trajectories with the largest likelihood are often nonsensical [37, 38]. Alternatively, some research focuses on approximating Bayesian inference for deep learning models using techniques like Monte Carlo dropout [39], which involves performing stochastic forward passes through the network and averaging the results. Among these approaches, [40] stands out as the only work that incorporates collaborative uncertainty among agents into the modeling process to guide the ranking of multimodal trajectories by uncertainty, albeit requiring special training strategies. However, none of the predicted uncertainties from these methods offer finite sample coverage guarantees, which is suboptimal for safety-critical applications such as vehicle trajectory prediction.

Conformal prediction (CP) [18, 41] has emerged as a widely adopted uncertainty quantification method, owing to its simplicity, generality, theoretical rigor, and low computational overhead. Notably, CP is agnostic to the underlying model and data distribution, making it highly versatile. It seamlessly integrates with any pre-trained model to deliver statistically valid prediction regions. Of particular relevance to our multimodal trajectory prediction task is recent progress in generalizing CP to time-series forecasting. For instance, [42] introduced the Copula conformal prediction algorithm for multivariate, multi-step time series forecasting, applicable to any multivariate multi-step forecaster. Additionally, [43] focused on generating non-conformity score functions that yield multimodal prediction regions with minimal volume. Moreover, [44] employed CP to generate statistical uncertainty intervals from Gaussian mixture model outputs, obtaining separate prediction intervals corresponding to each GMM component prediction. Furthermore, [45] proved the validity of CP on graph data. Inspired by their work, we explore the potential of applying CP to the multimodal trajectory prediction comparing different CP methods.

## 3 Problem Formulation

At the scenario level, We have trajectory data T from both the vehicle and infrastructure viewpoints, denoted as T V and T I , respectively. While T V and T I share overlapping information where their fields of view intersect, T I also provides complementary information, being free from occlusions. Our modeling objective is to utilize the information from T I to improve the accuracy of trajectory prediction based solely on T V .

The trajectory prediction task involves leveraging historical trajectories T V h ∈ R N V × T h × a h and T I h ∈ R N I × T h × a h , alongside contextual information, typically HD maps denoted as M , to forecast future trajectories T V f ∈ R N V × T f × a f . Here, N V and N I represent the number of observed actors from the vehicle and infrastructure perspectives, respectively. T h denotes the historical time horizon, and T f is the prediction horizon. a h and a f represent the number of node features which we consider the vehicle center location defined by its x -and y -coordinates. Notably, the trajectory data from both views are defined within the same coordinate system. For the HD map, we opt for a vectorized representation due to its lightweight nature and efficiency [4]. This vector map is depicted by lane centerlines, which are composed of lane segments. We denote it as M∈ R N l × a l , where N l is the number of lane segments and a l is the number of lane attribute.

We formulate the overall probabilistic distribution as P ( T V f |T V h , T I h , M M V , I ) . Driven by the critical safety demands inherent in trajectory prediction, we aim to incorporate uncertainty quantification to preempt any potentially consequential model failures. Let D be the set of scenarios of the form ( T V h , T I h , M M T V , I , V f ) . We split the dataset into training D train , validation D val , calibration D cal and test D test . One black-box deep learning model is trained on D train and evaluated on D val . We achieve the uncertainty quantification by conformal prediction in a post-hoc way on D cal .

At the agent level, we denote the features and labels of agent i from the vehicle view as X V = T V h,i and Y V = T V f,i for brevity. In real-world scenarios, future trajectories may exhibit multimodal behavior, often approximated by mixture

models, resulting in ˆ Y V ∈ R K × T f × a f , where K represents the number of mixtures or modes. Given a new agent sample X V test from D test , we seek to construct the prediction intervals C ( X V test ) ∈ R K × T f × a f × 2 such that it covers the ground truth label Y V test under a predefined coverage rate leveraging conformal prediction.

CP proceeds in three steps. First, we define a nonconformity score A : X × Y ∈ R K × T f × a f to quantify how well Y conforms to the prediction at X . Typically, we choose a metric of disagreement between the prediction and the ground truth as the non-conformity score, such as the Euclidean distance. Second, given the predefined miscoverage rate α , we compute the 1 -α quantile of the non-conformity scores on the calibration set { A X ,Y ( 1 1 ) , · · · , A X ( n , Y n ) } , where n is the number of agents. The resulting quantile is denoted as ˆ H ∈ R T f × a f . Last, when presented with a new test agent X V test , CP constructs the prediction interval C ( X V test ) = { Y V ∈ Y : A X ( V test , Y V ) ≤ ˆ H } .

## 4 Methodology

## 4.1 Overview

Our method V2INet consists of two key components, predictive modeling and post-hoc conformal prediction. An overview of our proposed model is illustrated in Figure 3. We first represent the scenario data collected from both views as graphs. A single view encoder is then applied separately to each graph, encoding various information such as agent-agent interactions, temporal dependencies, and agent-lane information. Subsequently, The vehicle-view embedding is fused with the infrastructure-view embedding through a cross-graph attention module. Finally, the updated vehicle-view embedding pass through a multi-modal decoder, providing multimodal predictions for all the agents of interest. The post-hoc conformal prediction is then applied at the agent level to construct valid prediction intervals given a predefined coverage rate.

Figure 3: Proposed model architecture. Data collected from the vehicle view are represented in red, while data from the infrastructure view are depicted in blue. The model takes as input the graph data constructed from both views. We apply single-view encoders to encode information from each view, followed by the fusion of the two embeddings through a cross-graph attention module. The final embedding passes through a multi-modal decoder, providing multimodal predictions for all the agents of interest.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00374v3-figure-54f0b41c-48ee-4d29-9756-667c04994dee.jpg)

## 4.2 Scene Representation

We adopt an ego-centric coordinate system utilizing vectorized representation as first introduced in [4]. First, data from both views are transformed such that the ego vehicle is centered at the origin, with its heading aligned along the positive x -axis. Each trajectory is then characterized as a sequence of displacements { ∆ x t } 0 t = -( T h -1) , where ∆ x t ∈ R 2 is the 2D displacement from time step t -1 to t . Similarly, each lane segment is represented as ∆ x l ∈ R 2 , which captures the 2D displacement from the starting coordinate to the end coordinate of lane segment l . We then construct scenario graphs consisting of actor nodes and lane nodes for both views separately. The edge attribute is the absolute relative position between two nodes.

## 4.3 Single View Encoder

To encode the spatiotemporal information, including agent-agent interactions, temporal dependencies, and context information captured by agent-lane relations for data from each view, existing graph-based models offer effective

solutions. Models like LaneGCN [1], HiVT [3], and VectorNet [4] incorporate these components and can be readily employed. This encoding step can be performed efficiently at the edge device, such as the vehicle's on-board computers and the roadside unit (RSU) for infrastructure data, before broadcasting, thereby minimizing computational overhead.

We exemplify here with HiVT [3] where only attention mechanism is employed, with the adoption of a rotation-invariant representation. To attend to the local information, at each timestamp, the surrounding actors' information is aggregated and the embeddings at each time stamp subsequently go through a transformer to capture the temporal dependencies. After obtaining the spatio-temporal embeddings, the surrounding lane information is aggregated for each actor. The update actor node embeddings is then passed to a global interaction module which aggregates the local context of different actors and updates each actor's representation to capture long-range dependencies and scene-level dynamics. We denote the final embedding for actor node i from the vehicle view as h i , and actor node j from the infrastructure view as h j . Both h i and h j are in R d h , where d h is the embedding dimension.

## 4.4 Cross-graph Attention Module

To effectively utilize information from the infrastructure side, we employ a cross-graph attention module that aggregates information captured by the infrastructure view encoder. Specifically, the embedding from the vehicle view h i is transformed into the query vector, while the embedding from the infrastructure view h j alongside the relative position at the last observed time step x t =0 i -x t =0 j , are utilized to compute the key and value vectors. We denote h ij = ( h j , x t =0 i -x t =0 j ) representing the concatenation of node and edge attribute from infrastructure node j to vehicle node i :

$$\mathbf q _ { i } = W ^ { Q _ { I } } \mathbf h _ { i }, \ \ k _ { i j } = W ^ { K _ { I } } \mathbf h _ { i j }, \ \mathbf v _ { i j } = W ^ { V _ { I } } \mathbf h _ { i j }$$

where W Q f , W K f , W V f ∈ R d k × d h are learnable matrices for linear projection and d k is the transformed dimension. The resulting query, key and value vectors are then taken as input to the scaled dot-product attention block:

$$\alpha _ { i j } = \underset { \longrightarrow } { \text{softmax} } \left ( \frac { \mathbf q _ { i } ^ { T } } { \sqrt { d _ { k } } } \cdot \left [ \{ \mathbf k _ { i j } \} _ { j \in \mathcal { N } _ { i } } \right ] \right ),$$

$$\mathbf m _ { i } = \sum _ { j \in \mathcal { N } _ { i } } \alpha _ { i j } \mathbf v _ { i j },$$

$$g _ { i } = \text{sigmoid} ( W ^ { \text{gate} } [ h _ { i }, m _ { i } ] ),$$

$$\tilde { h } _ { i } = g _ { i } \odot W ^ { \text{self} } h _ { i } + ( 1 - g _ { i } ) \odot m _ { i }$$

where N i is the set of agent i 's neighbors, W gate and W self are learnable matrices and ⊙ denotes element-wise product. We followed the structure in HiVT to fuse the infrastructure information with a gating function. The attention block supports multiple heads. Finally, we apply another MLP block to obtain the final fused embedding ˜ h i ∈ R d h for agent i from the vehicle view.

## 4.5 Mixture Model Based Decoder and Learning

There are two widely used mixture models for describing the multimodal trajectories, Gaussian and Laplacian. Previous methods [1, 4, 46] have found that the ℓ 1 -based loss function derived from the Laplace distribution usually leads to superior prediction performances, as it is more robust to outliers. Hence, we will parameterize the future trajectories following Laplace distribution. For each agent, the decoder receives the final embedding as inputs and outputs K possible future trajectories and the mixing coefficient of the mixture model for each agent. The decoder are consisted of three MLPs, one for predicting the future locations µ t i,k ∈ R 2 for agent i and its mode k at each time step t , one for predicting the associated uncertainty b t i,k ∈ R 2 assuming independence of the x -and y -coordinates, the last one followed by a softmax is for producing the scores for each mode.

To ensure the prediction diversity [47, 48], instead of optimizing all the predicted trajectories, only the mode ˜ k closest to the ground truth is optimized. The closeness here is defined as the average Euclidean distance between ground-truth locations and predicted locations across all future time steps. The Loss includes both regression loss and classification loss

$$J = J _ { r e g } + \varepsilon J _ { c l s }$$

Here, ε is the weight of the classification loss. We employ the negative log-likelihood as the regression loss:

$$J _ { r e g } = - \frac { 1 } { n T _ { f } } \sum _ { i = 1 } ^ { n } \sum _ { t = 1 } ^ { T _ { f } } \log \mathbf P ( \mathbf y _ { i } ^ { t } - \hat { \mathbf y } _ { i } ^ { t } | \hat { \mu } _ { i } ^ { t, \tilde { k } }, \hat { \mathbf b } _ { i } ^ { t, \tilde { k } } )$$

where P ( ) | is the probability density function of Laplace distribution and ˆ µ t,k ˜ i , ˆ b t,k ˜ i are the mean and uncertainty estimates of the best mode ˜ k .

For J cls , cross-entropy loss is applied:

$$\ p p u e u. & J _ { c l s } = - \frac { 1 } { n } \sum _ { i = 1 } ^ { n } \sum _ { k = 1 } ^ { K } \mathbf 1 _ { k } \log \pi _ { k } & ( 8 ) \\ \text{if $k$ is the best mode, $\pi_{k}$ is the mixing coefficient of mode $k$. The model is trained on}$$

## 4.6 Post-hoc uncertainty quantification module

where ✶ k is a binary indicator if k is the best mode, π k is the mixing coefficient of mode k . The model is trained on D train and evaluated on D val .

Uncertainty quantification methods are evaluated on two key properties: validity and efficiency. Validity is established when the predicted confidence level exceeds or equals the probability of events falling within the predicted range, while efficiency refers to minimizing the size of the confidence region. We utilize conformal prediction to obtain both valid and efficient prediction intervals. The standard conformal prediction methods typically operate with scalar point estimates for regression problems. However, since our output consists of a multimodal multivariate time-series, we need to make certain adaptations to accommodate this complexity. Following the steps outlined in Section 3, we proceed with non-conformity score function definition, quantile computation and prediction interval construction.

## 4.6.1 Non-conformity Score Functions

As emphasized in [49], the usefulness of the prediction sets is primarily determined by the score function, we adopt three score functions A and compare their performance.

Z-score . As the decoder returns both mean and variance predictions, we define the Z-score function as:

$$Z = \frac { | Y - \hat { Y } | } { \hat { B } }$$

where Z ∈ R K × T f × 2 and ˆ B = {{ ˆ b t,k } T f t =0 } K k =0 . Here, we compute scores separately for x -and y -coordinates, reflecting the observation that motion uncertainty varies significantly across different dimensions.

L2-norm . Next, we consider the Euclidean distance, the most commonly used metric in regression problems:

$$L _ { 2 } & = | | Y - \hat { Y } | | _ { 2 }$$

where L 2 ∈ R K × T f .

L1-norm . Recognizing that the L2-norm disregards dimension differences, we consider L1-norm:

$$L _ { 1 } & = | | Y - \hat { Y } | | _ { 1 }$$

where L 1 ∈ R K × T f × 2 .

## 4.6.2 Quantile Computation

Given a predefined miscoverage rate α ∈ [0 , 1] , the 1 -α quantile of the non-conformity scores is calculated on the calibration set D cal . Conventionally, the quantile is determined as follows: ˆ H = quantile ( { A X ,Y ( 1 1 ) , · · · , A X ( n , Y n ) } , (1 -α )(1 + 1 n )) where A X ,Y ( i i ) is a scalar, and n denotes the total number of agents in D cal . However, in our case, we deal with multimodal time-series scores, which are multivariate when using Z-score and L1-norm, hence necessary adaptations are needed. For brevity, let Γ = i A X ,Y ( i i ) , with Γ i ∈ R K × T f × 2 for Z-score and L1-norm, and R K × T f for L2-norm.

To enhance the efficiency of our prediction intervals, we focus on computing the quantile using only the mode ˜ k that exhibits the smallest average Euclidean distance to the ground truth trajectory. This reduces the computation to multivariate time-series. Within this framework, we investigate two established methods: CF-RNN [50] and CopulaCPTS [42].

CF-RNN . Since the time-series predictions are obtained from the same embedding, this work proposes the application of Bonferroni correction to the calibration scores to maintain the desired miscoverage rate α . Specifically, the original α is divided by T f , yielding ˆ = H quantile ( { Γ i } n i =0 , (1 -α T f )(1 + 1 n )) for single-variate case.

CopulaCPTS . As implied by its name, this method models the joint probability of uncertainty for multiple predicted time steps using a copula. The calibration set D cal is split into two subsets: D cal -1 , which estimates a Cumulative Distribution Function (CDF) for the nonconformity score of each time step, and D cal -2 , utilized to calibrate the copula. The copula function captures the dependency between time steps and can enhance the efficiency of prediction intervals.

For multivariate scores generated by the L1-norm and Z-score functions, we adopt the Bonferroni correction for each dimension, as inspired by CF-RNN. Specifically, we use α/ 2 as the miscoverage rate for both x -and y -coordinates.

## 4.6.3 Prediction Interval Construction

We utilize the obtained quantile ˆ H to form the prediction intervals for new examples in D test :

$$\mathcal { C } ( X ^ { \mathcal { V } } _ { t e s t } ) = \{ Y ^ { \mathcal { V } } \in \mathcal { Y } \colon A ( X ^ { \mathcal { V } } _ { t e s t }, Y ^ { \mathcal { V } } ) \leq \hat { H } \}$$

Specifically, for ˆ H obtained from the Z-score, we have:

$$\mathcal { C } ( X _ { t e s t } ^ { \mathcal { V } } ) = \left [ \hat { Y } _ { t e s t } ^ { \mathcal { V } } - \hat { B } \hat { H }, \hat { Y } _ { t e s t } ^ { \mathcal { V } } + \hat { B } \hat { H } \right ]$$

and for ˆ H obtained from the L2-norm and L1-norm, we have

$$\mathcal { C } ( X ^ { \nu } _ { t e s t } ) = \left [ \hat { Y } ^ { \nu } _ { t e s t } - \hat { H }, \hat { Y } ^ { \nu } _ { t e s t } + \hat { H } \right ]$$

## 5 Experiments

## 5.1 Experimental Setup

In this section, We introduce the specifics of the dataset, the evaluation metrics and the implementation details including hardware, hyperparameters, etc.

## 5.1.1 Dataset

We evaluate the proposed model on the publicly available large-scale and real-world V2I dataset V2X-Seq [6], which provides the trajectories of agents from both vehicle and infrastructure sides, along with vector map data. V2X-Seq consists of 51,146 V2I scenarios, where each trajectory is 10 seconds long with a sampling rate of 10 Hz. The task involves predicting the motion of agents in the next 5 seconds, given initial 5-second observations from both infrastructure and vehicle sides. The dataset has been split into train and validation. Our trained model is evaluated on the validation set, allowing for comparison with existing models. For post-hoc conformal prediction, we divide the validation set into calibration set and test set at a 4:1 ratio. For a discussion on the calibration data size, please refer to [49].

## 5.1.2 Evaluation Metrics

Model metrics . For model evaluation, the standard metrics in motion predictions are adopted, including minimum Average Displacement Error (minADE), minimum Final Displacement Error (minFDE), and Miss Rate (MR), where errors between the best predicted trajectory among the K=6 modes and the ground truth trajectory are calculated. The best here refers to the trajectory that has the minimum endpoint error. The ADE metric calculates the L2 distance across all future time steps and averages over all scored vehicles within a scenario, while FDE measures the L2 distance only at the final future time step and summarizes across all scored vehicles. MR refers to the ratio of actors in a scenario where FDE are above 2 meters.

Conformal prediction metrics . We assess validity and efficiency for each method. Validity is evaluated by reporting independent and joint coverage on the test set, aiming for coverage levels close to the desired confidence level 1 -α . The independent coverage for each agent is calculated as:

$$\text{ind.} \text{ coverage} _ { 1 - \alpha } = \frac { 1 } { T _ { f } } \sum _ { X, Y \in \mathcal { D } _ { t e s t } } 1 ( Y \in \mathcal { C } ( X ) ) & ( 1 5 ) \\ \text{sum independent coverage among the $K$ modes for each agent. For all metrics, the final reported}. \dots \dots \cdots \cdots \mathcal { \mathbb { O } }$$

We identify the maximum independent coverage among the K modes for each agent. For all metrics, the final reported value is the average among all agents in D test .

For efficiency, we calculate the average size of the predicted 2D area across all time steps for the mode ˜ k with the maximum independent coverage:

$$\text{size} & = \frac { 1 } { T _ { f } } | | \mathcal { C } ( X ) _ { \tilde { k } } | |$$

The 2D area is defined as ellipsis for Z-score and L1-norm, and circle for L2-norm.

The joint coverage is defined as if there exists a mode k such that the truth values across all time steps fall into the confidence region:

## 5.1.3 Implementation Details

$$\text{s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.}$$

Model . The model was trained for 64 epochs on an Nvidia V100S GPU with 32GB memory using AdamW optimizer [51]. Hyperparameters including batch size, initial learning rate, weight decay and dropout rate are 32, 1e-3, 1e-4 and 0.1, respectively. The learning rate is decayed using the cosine annealing scheduler [52]. Our model employs the original setting of HiVT with 64 hidden units for the single view encoder and 1 layer of cross-graph attention module with 8 heads. The radius of all local regions is 50 meters. The number of prediction modes is set to 6.

Conformal prediction . With the pretrained model and datasets D cal and D test , we execute all conformal prediction methods on the CPU. We evaluate the methods on the three defined score functions at three different α levels: 0.2, 0.1, 0.05. The optimization step in CopulaCPTS remains consistent with the original work.

## 5.2 Results Analysis

In this section, we first examine the model's prediction performance, assessing it from both quantitative and qualitative perspectives. Next, we showcase the effectiveness of the post-hoc uncertainty quantification method.

## 5.2.1 Model Performance

Comparison with state-of-the-art . We benchmark our proposed model against state-of-the-art models using the V2X-Seq dataset, as detailed in [6]. Results are summarized in Table 2. Both TNT [5] and HiVT [3] are single-view models. [6] evaluated them under two settings: Ego, where only the vehicle-view data is utilized, and PP-VIC, which employs a two-stage method with both vehicle view and infrastructure data. Specifically, in PP-VIC, data from both views are fused offline with some tracking method and then the stitched trajectories were fed into a single-view model. We retrained the HiVT model under both Ego and the PP-VIC setting with 64 epochs and present our results in Table 2. Comparing the Ego and PP-VIC results, it's uncovered by [6] that integrating information from the infrastructure side can enhance prediction accuracy. V2X-Graph represents the current state-of-the-art model, employing a graph link prediction module to associate two-view data, followed by fusing the embeddings of the associated nodes using attention mechanism. However, this node association module requires pre-training, leading to a two-stage training process. Our method demonstrates the best performance in terms of minFDE and MR. Without the explicit node association from both views, our attention based fusion module can attend to the most relevant nodes from both views through learning, which significantly simplifies the modeling framework and facilitates ease of training, all while achieving better results.

Table 2: Quantitative results on V2X-Seq. TNT and V2X-Graph results are reported in [6].

| Method        | Cooperation   |   minADE |   minFDE |   MR |
|---------------|---------------|----------|----------|------|
| TNT           | Ego           |     8.45 |    17.93 | 0.77 |
| TNT           | PP-VIC        |     7.38 |    15.27 | 0.72 |
| HiVT          | Ego           |     1.34 |     2.16 | 0.31 |
| HiVT          | PP-VIC        |     1.28 |     2.11 | 0.31 |
| V2X-Graph     | V2X-Graph     |     1.17 |     2.03 | 0.29 |
| V2INet (Ours) | V2INet (Ours) |     1.19 |     1.98 | 0.27 |

Qualitative results . We visualize our prediction results and six representative scenarios are shown in Figure 4. To maintain clarity and simplicity in the visualization, each scenario displays the ground truth and multimodal prediction results only for the target agent, although predictions for all agents are accessible. In the first column, scenarios S1 and S4 depict the target agent turning right. In the second column, the target agent is shown going straight, leaving (S2) and approaching (S5) an intersection. The last column presents scenarios where the target agent is merging (S3) and turning

left (S6). It's shown that our model predictions capture the multimodal behavior. In scenarios S2 and S5, it's exhibited in the form of different velocity profiles. Since lane information is integrated via an attention mechanism rather than as rigid constraints, off-road and road rule-violating predictions and may occur, as shown in S4 and S6. Notably, there are more uncertainties when agent making turns, as evidenced by the dispersion among the predictions. Furthermore, it's important to highlight that the prediction with the highest probability does not always align with the ground truth, as demonstrated in S1 and S3, resulting from the inherent model and data uncertainties.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00374v3-figure-afc7ae97-762b-44d9-b491-3f447a74e62b.jpg)

Figure 4: Qualitative results. The ground truth (in red) and predicted multimodal trajectories (in different shades of blue) of the target agent are shown. Darker blue represents higher probability. Yellow and grey rectangles denotes road agents observed from vehicle view and infrastructure view, respectively.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00374v3-figure-a929b71b-c811-4180-b0b7-6318eba94b33.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00374v3-figure-18947f7f-bdf2-4213-85c0-a638744a184f.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00374v3-figure-b94b40bd-a73d-4696-aee5-eb10e91bd3e7.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00374v3-figure-40990e80-4bc0-437a-83a8-158bb6e58a3e.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00374v3-figure-94fb4931-5a2d-4e16-b878-af1a8c115227.jpg)

We further compare the prediction results from model HiVT Ego, HiVT PP-VIC and ours on selected scenarios, as shown in Figure 5. It's observed that HiVT Ego, relying solely on vehicle-view data, consistently predicted incorrect modes. In contrast, both HiVT PP-VIC and our model, incorporating additional data from the infrastructure view, demonstrate improved accuracy in capturing the ground truth.

## 5.2.2 Post-hoc Uncertainty Quantification Performance

Quantitative comparison . We show in this section that conformal prediction method produces more valid and efficient confidence regions than the model predictions. The evaluation results are shown in Table 3.

Table 3: Conformal prediction results comparison

| Metric                            | alpha   | Mixture          | CF-RNN              | CF-RNN            | CF-RNN            | CopulaCPTS          | CopulaCPTS       | CopulaCPTS       |
|-----------------------------------|---------|------------------|---------------------|-------------------|-------------------|---------------------|------------------|------------------|
| Metric                            | alpha   | Mixture          | Z-score             | L2-norm           | L1-norm           | Z-score             | L2-norm          | L1-norm          |
| ind. coverage joint coverage size | 0.2     | 0.65 0.15 124.43 | 0.99 0.97 5734.55   | 0.99 0.99 2003.42 | 0.99 0.97 432.50  | 0.93 0.76 536.09    | 0.94 0.80 29.14  | 0.92 0.76 17.83  |
| ind. coverage joint coverage size | 0.1     | 0.78 0.31 215.15 | 0.99 0.99 25098.06  | 0.99 0.99 3672.24 | 0.99 0.99 1138.77 | 0.97 0.86 7634.95   | 0.96 0.93 64.80  | 0.96 0.88 42.82  |
| ind. coverage joint coverage size | 0.05    | 0.86 0.47 327.48 | 0.99 0.99 151062.75 | 0.99 0.99 6550.62 | 0.99 0.99 2321.75 | 0.97 0.99 372768.78 | 0.97 0.94 107.63 | 0.98 0.95 190.04 |

Figure 5: Qualitative results comparison for three models: HiVT-Ego (first row), HiVT-PPVIC (second row), and ours (third row). The ground truth (in red) and predicted multimodal trajectories (in different shades of blue) of the target agent are shown. Darker blue represents higher probability. Yellow and grey rectangles denotes road agents observed from vehicle view and infrastructure view, respectively.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00374v3-figure-59889a3a-ec3f-4d37-9c9b-d459a25b3772.jpg)

The Mixture column presents UQ results derived directly from the multimodal predictions. We first examine the metrics of independent and joint coverage. It's evident that the model predicted intervals fail to meet the validity requirements across all specified miscoverage rates. Interestingly, CF-RNN yields overly conservative results, maintaining coverage levels around 0.99 regardless of the miscoverage rate. This could indicate that Bonferroni correction, often employed with shorter horizons, might not be suitable for our predicted horizon of 50. CopulaCPTS achieves coverage levels very close to the desired values across all specified miscoverage rates α .

Next, we inspect the metric "size", which serves as an indicator of UQ efficiency. Benchmark the size from model predictions in the Mixture column, CF-RNN produces extremely large confidence region which is expected considering the overly conservative coverage observed earlier. CopulaCPTS yields comparative and even smaller size compared to the benchmark.

Finally, we examine the impact of different score functions. In both CF-RNN and CopulaCPTS, Z-score appears as the least efficient among all three functions. Since Z-score is computed using the model-predicted uncertainties ˆ B , it will always be more inefficient than the benchmark size in the "Mixture" column. One plausible explanation provided in [49], is that although widely used, ˆ B in the Z-score function is not directly related to the quantiles of the label distribution. Therefore, it may not be the most suitable score function option in our case. On the other hand,

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00374v3-figure-0303f2ec-e044-4370-aa89-cb5fd086a507.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00374v3-figure-a2772d1f-9588-42f2-9241-51d5cb45acc7.jpg)

Figure 6: Uncertainty quantification comparison for the mixture model prediction (purple shade) and CopulaCPTS (green shade) at α = 0 1 . .

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00374v3-figure-2c1aa8dc-f3f0-45d6-b4d6-57b5e610ad42.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00374v3-figure-b355aed5-b2de-4bbb-900e-eb020dbcc568.jpg)

both L2-norm and L1-norm demonstrate significantly higher efficiency, particularly within the CopulaCPTS method. L1-norm, in particular, considers variations among different dimensions, leading to enhanced efficiency. Taking into account both validity and efficiency requirements, we have highlighted the best results among all methods in red.

Qualitative analysis . We visualize the uncertainty prediction from the mixture model and the CopulaCPTS, as shown in Figure 6. Three scenarios are selected in each column. We can see that the mixture model predictions fail to cover the ground truth while CopulaCPTS achieves the coverage at the specified α .

## 6 Conclusion

We have presented a novel model framework with multi-view data integration in the cooperative driving setting. Our proposed model is straight forward and can be built upon any existing graph-based single-view models. It has demonstrated its effectiveness and advantages over existing benchmarks. Moreover, we have incorporated a post-hoc uncertainty quantification module, providing valid and efficient confidence regions, which is crucial in safety-critical tasks such as trajectory prediction.

The proposed framework has certain limitations. From the model's perspective, we currently treat all road agents as the same type. However, in the public V2X-Seq dataset, there are different vehicle types, such as trucks, vans, buses, motorcycles, etc. Future work should address the different characteristics of these vehicle types to enhance model performance. Moreover, better methods for encoding the lane information to eliminate off-road and roadrule-violating predictions should be investigated. From the uncertainty quantification perspective, we simplify the quantile computation on the multimodal prediction into computation on the single best mode. This approach loses valuable distributional information. Therefore, exploring score functions that consider the distribution could lead to more accurate uncertainty quantification for multimodal results. Furthermore, we evaluate the uncertainty assuming independence among the agents. Future work can incorporate agent correlations based on the graph structure to better reflect the underlying uncertainty relations and provide more insightful results.

## References

- [1] Ming Liang, Bin Yang, Rui Hu, Yun Chen, Renjie Liao, Song Feng, and Raquel Urtasun. Learning lane graph representations for motion forecasting. In European Conference on Computer Vision , pages 541-556. Springer, 2020.
- [2] Yuning Chai, Benjamin Sapp, Mayank Bansal, and Dragomir Anguelov. Multipath: Multiple probabilistic anchor trajectory hypotheses for behavior prediction. arXiv preprint arXiv:1910.05449 , 2019.

| [3]   | Zikang Zhou, Luyao Ye, Jianping Wang, Kui Wu, and Kejie Lu. Hivt: Hierarchical vector transformer for multi-agent motion prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8823-8833, 2022.                                                                                                                                  |
|-------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| [4]   | Jiyang Gao, Chen Sun, Hang Zhao, Yi Shen, Dragomir Anguelov, Congcong Li, and Cordelia Schmid. Vectornet: Encoding hd maps and agent dynamics from vectorized representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11525-11533, 2020.                                                                                       |
| [5]   | Hang Zhao, Jiyang Gao, Tian Lan, Chen Sun, Benjamin Sapp, Balakrishnan Varadarajan, Yue Shen, Yi Shen, Yuning Chai, Cordelia Schmid, et al. Tnt: Target-driven trajectory prediction. arXiv preprint arXiv:2008.08294 2020.                                                                                                                                                         |
| [6]   | Haibao Yu, Wenxian Yang, Hongzhi Ruan, Zhenwei Yang, Yingjuan Tang, Xu Gao, Xin Hao, Yifeng Shi, Yifeng Pan, Ning Sun, et al. V2x-seq: A large-scale sequential dataset for vehicle-infrastructure cooperative perception and forecasting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5486-5495, 2023.                            |
| [7]   | Hongzhi Ruan, Haibao Yu, Wenxian Yang, Siqi Fan, Yingjuan Tang, and Zaiqing Nie. Learning cooperative trajectory representations for motion forecasting. arXiv preprint arXiv:2311.00371 , 2023.                                                                                                                                                                                    |
| [8]   | Tung Phan-Minh, Elena Corina Grigore, Freddy A Boulton, Oscar Beijbom, and Eric M Wolff. Covernet: Multimodal behavior prediction using trajectory sets. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 14074-14083, 2020.                                                                                                            |
| [9]   | Balakrishnan Varadarajan, Ahmed Hefny, Avikalp Srivastava, Khaled S Refaat, Nigamaa Nayakanti, Andre Cornman, Kan Chen, Bertrand Douillard, Chi Pang Lam, Dragomir Anguelov, et al. Multipath++: Efficient information fusion and trajectory aggregation for behavior prediction. In 2022 International Conference on Robotics and Automation (ICRA) , pages 7814-7821. IEEE, 2022. |
| [10]  | Jean Mercat, Thomas Gilles, Nicole El Zoghby, Guillaume Sandou, Dominique Beauvois, and Guillermo Pita Gil. Multi-head attention for multi-modal joint vehicle motion forecasting. In 2020 IEEE International Conference on Robotics and Automation (ICRA) , pages 9638-9644. IEEE, 2020.                                                                                           |
| [11]  | Siddhesh Khandelwal, William Qi, Jagjeet Singh, Andrew Hartnett, and Deva Ramanan. What-if motion prediction for autonomous driving. arXiv preprint arXiv:2008.10587 , 2020.                                                                                                                                                                                                        |
| [12]  | Thibault Buhet, Emilie Wirbel, Andrei Bursuc, and Xavier Perrotton. Plop: Probabilistic polynomial objects trajectory planning for autonomous driving. arXiv preprint arXiv:2003.08744 , 2020.                                                                                                                                                                                      |
| [13]  | Henggang Cui, Vladan Radosavljevic, Fang-Chieh Chou, Tsung-Han Lin, Thi Nguyen, Tzu-Kuo Huang, Jeff Schneider, and Nemanja Djuric. Multimodal trajectory predictions for autonomous driving using deep convolu- tional networks. In 2019 International Conference on Robotics and Automation (ICRA) , pages 2090-2096. IEEE, 2019.                                                  |
| [14]  | Xidong Feng, Zhepeng Cen, Jianming Hu, and Yi Zhang. Vehicle trajectory prediction using intention-based conditional variational autoencoder. In 2019 IEEE Intelligent Transportation Systems Conference (ITSC) , pages 3514-3519. IEEE, 2019.                                                                                                                                      |
| [15]  | Boris Ivanovic, Karen Leung, Edward Schmerling, and Marco Pavone. Multimodal deep generative models for trajectory prediction: A conditional variational autoencoder approach. IEEE Robotics and Automation Letters 6(2):295-302, 2020.                                                                                                                                             |
| [16]  | Zhi Zhong, Yutao Luo, and Weiqiang Liang. Stgm: Vehicle trajectory prediction based on generative model for spatial-temporal features. IEEE Transactions on Intelligent Transportation Systems , 23(10):18785-18793, 2022.                                                                                                                                                          |
| [17]  | Glenn Shafer and Vladimir Vovk. A tutorial on conformal prediction. Journal of Machine Learning Research 9(3), 2008.                                                                                                                                                                                                                                                                |
| [18]  | Harris Papadopoulos, Volodya Vovk, and Alex Gammerman. Conformal prediction with neural networks. In 19th IEEE International Conference on Tools with Artificial Intelligence (ICTAI 2007) , volume 2, pages 388-395. IEEE, 2007.                                                                                                                                                   |
| [19]  | Seong-Woo Kim, Wei Liu, Marcelo H Ang, Emilio Frazzoli, and Daniela Rus. The impact of cooperative perception on decision making and planning of autonomous vehicles. IEEE Intelligent Transportation Systems Magazine , 7(3):39-50, 2015.                                                                                                                                          |
| [20]  | Jiaxun Cui, Hang Qiu, Dian Chen, Peter Stone, and Yuke Zhu. Coopernaut: End-to-end driving with cooperative perception for networked vehicles. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 17252-17262, 2022.                                                                                                                      |

| [21]   | Guangzhen Cui, Weili Zhang, Yanqiu Xiao, Lei Yao, and Zhanpeng Fang. Cooperative perception technology of autonomous driving in the internet of vehicles environment: A review. Sensors , 22(15):5535, 2022.                                                                                                                              |
|--------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| [22]   | Shunli Ren, Siheng Chen, and Wenjun Zhang. Collaborative perception for autonomous driving: Current status and future trend. In Proceedings of 2021 5th Chinese Conference on Swarm Intelligence and Cooperative Control pages 682-692. Springer, 2022.                                                                                   |
| [23]   | Tsun-Hsuan Wang, Sivabalan Manivasagam, Ming Liang, Bin Yang, Wenyuan Zeng, and Raquel Urtasun. V2vnet: Vehicle-to-vehicle communication for joint perception and prediction. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16 , pages 605-621. Springer, 2020.           |
| [24]   | Michael Fürst, Oliver Wasenmüller, and Didier Stricker. Lrpd: Long range 3d pedestrian detection leveraging specific strengths of lidar and rgb. In 2020 IEEE 23rd international conference on intelligent transportation systems (ITSC) , pages 1-7. IEEE, 2020.                                                                         |
| [25]   | Eduardo Arnold, Mehrdad Dianati, Robert de Temple, and Saber Fallah. Cooperative perception for 3d object detection in driving scenarios using infrastructure sensors. IEEE Transactions on Intelligent Transportation Systems , 23(3):1852-1864, 2020.                                                                                   |
| [26]   | Runsheng Xu, Hao Xiang, Xin Xia, Xu Han, Jinlong Li, and Jiaqi Ma. Opv2v: An open benchmark dataset and fusion pipeline for perception with vehicle-to-vehicle communication. In 2022 International Conference on Robotics and Automation (ICRA) , pages 2583-2589. IEEE, 2022.                                                           |
| [27]   | Yiming Li, Dekun Ma, Ziyan An, Zixun Wang, Yiqi Zhong, Siheng Chen, and Chen Feng. V2x-sim: Multi-agent collaborative perception dataset and benchmark for autonomous driving. IEEE Robotics and Automation Letters 7(4):10914-10921, 2022.                                                                                               |
| [28]   | Haibao Yu, Yizhen Luo, Mao Shu, Yiyi Huo, Zebang Yang, Yifeng Shi, Zhenglong Guo, Hanyu Li, Xing Hu, Jirui Yuan, et al. Dair-v2x: A large-scale dataset for vehicle-infrastructure cooperative 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 21361-21370, 2022.       |
| [29]   | Runsheng Xu, Xin Xia, Jinlong Li, Hanzhao Li, Shuo Zhang, Zhengzhong Tu, Zonglin Meng, Hao Xiang, Xiaoyu Dong, Rui Song, et al. V2v4real: A real-world large-scale dataset for vehicle-to-vehicle cooperative perception. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 13712-13722, 2023. |
| [30]   | Mohsen Kamrani, Ramin Arvin, and Asad J Khattak. Extracting useful information from basic safety message data: An empirical study of driving volatility measures and crash frequency at intersections. Transportation research record , 2672(38):290-301, 2018.                                                                           |
| [31]   | Zixing Lei, Shunli Ren, Yue Hu, Wenjun Zhang, and Siheng Chen. Latency-aware collaborative perception. In European Conference on Computer Vision , pages 316-332. Springer, 2022.                                                                                                                                                         |
| [32]   | Runsheng Xu, Hao Xiang, Zhengzhong Tu, Xin Xia, Ming-Hsuan Yang, and Jiaqi Ma. V2x-vit: Vehicle-to- everything cooperative perception with vision transformer. In European conference on computer vision , pages 107-124. Springer, 2022.                                                                                                 |
| [33]   | Yunshuang Yuan and Monika Sester. Leveraging dynamic objects for relative localization correction in a connected autonomous vehicle network. arXiv preprint arXiv:2205.09418 , 2022.                                                                                                                                                      |
| [34]   | Yifan Lu, Quanhao Li, Baoan Liu, Mehrdad Dianati, Chen Feng, Siheng Chen, and Yanfeng Wang. Robust collaborative 3d object detection in presence of pose errors. In 2023 IEEE International Conference on Robotics and Automation (ICRA) , pages 4812-4818. IEEE, 2023.                                                                   |
| [35]   | Paul J Besl and Neil D McKay. Method for registration of 3-d shapes. In Sensor fusion IV: control paradigms and data structures , volume 1611, pages 586-606. Spie, 1992.                                                                                                                                                                 |
| [36]   | Nuri Benbarka, Jona Schröder, and Andreas Zell. Score refinement for confidence-based 3d multi-object tracking. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 8083-8090. IEEE, 2021.                                                                                                         |
| [37]   | Matteo Zecchin, Sangwoo Park, and Osvaldo Simeone. Forking uncertainties: Reliable prediction and model predictive control with sequence models via conformal risk control. IEEE Journal on Selected Areas in Information Theory , 2024.                                                                                                  |
| [38]   | Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751 , 2019.                                                                                                                                                                                      |

| [39]   | Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning , pages 1050-1059. PMLR, 2016.                                                                       |
|--------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| [40]   | Bohan Tang, Yiqi Zhong, Chenxin Xu, Wei-Tao Wu, Ulrich Neumann, Ya Zhang, Siheng Chen, and Yanfeng Wang. Collaborative uncertainty benefits multi-agent multi-modal trajectory forecasting. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2023.          |
| [41]   | Anastasios N Angelopoulos, Stephen Bates, et al. Conformal prediction: A gentle introduction. Foundations and Trends® in Machine Learning , 16(4):494-591, 2023.                                                                                                            |
| [42]   | Sophia Sun and Rose Yu. Copula conformal prediction for multi-step time series forecasting. arXiv preprint arXiv:2212.03281 , 2022.                                                                                                                                         |
| [43]   | Renukanandan Tumu, Matthew Cleaveland, Rahul Mangharam, George J Pappas, and Lars Lindemann. Multi- modal conformal prediction regions by optimizing convex shape templates. arXiv preprint arXiv:2312.07434 2023.                                                          |
| [44]   | Ishan D Khurjekar and Peter Gerstoft. Multi-source doa estimation with statistical coverage guarantees. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 5310-5314. IEEE, 2024.                                |
| [45]   | Kexin Huang, Ying Jin, Emmanuel Candes, and Jure Leskovec. Uncertainty quantification over graph with conformalized graph neural networks. Advances in Neural Information Processing Systems , 36, 2024.                                                                    |
| [46]   | Vineet Kosaraju, Amir Sadeghian, Roberto Martín-Martín, Ian Reid, Hamid Rezatofighi, and Silvio Savarese. Social-bigat: Multimodal trajectory forecasting using bicycle-gan and graph attention networks. Advances in neural information processing systems , 32, 2019.     |
| [47]   | Agrim Gupta, Justin Johnson, Li Fei-Fei, Silvio Savarese, and Alexandre Alahi. Social gan: Socially acceptable trajectories with generative adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2255-2264, 2018. |
| [48]   | Luca Anthony Thiede and Pratik Prabhanjan Brahma. Analyzing the variety loss in the context of probabilistic trajectory prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 9954-9963, 2019.                                     |
| [49]   | Anastasios NAngelopoulos and Stephen Bates. Agentle introduction to conformal prediction and distribution-free uncertainty quantification. arXiv preprint arXiv:2107.07511 , 2021.                                                                                          |
| [50]   | Kamile Stankeviciute, Ahmed MAlaa, and Mihaela van der Schaar. Conformal time-series forecasting. Advances in neural information processing systems , 34:6216-6228, 2021.                                                                                                   |
| [51]   | Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 2017.                                                                                                                                                              |
| [52]   | Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint                                                                                                                                                                      |
Xi Chen Rahul Bhadani Larry Head
Conformal Trajectory Prediction with Multi-View Data Integration in Cooperative Driving
Current research on trajectory prediction primarily relies on data collected
by onboard sensors of an ego vehicle. With the rapid advancement in connected
technologies, such as vehicle-to-vehicle (V2V) and vehicle-to-infrastructure
(V2I) communication, valuable information from alternate views becomes
accessible via wireless networks. The integration of information from
alternative views has the potential to overcome the inherent limitations
associated with a single viewpoint, such as occlusions and limited field of
view. In this work, we introduce V2INet, a novel trajectory prediction
framework designed to model multi-view data by extending existing single-view
models. Unlike previous approaches where the multi-view data is manually fused
or formulated as a separate training stage, our model supports end-to-end
training, enhancing both flexibility and performance. Moreover, the predicted
multimodal trajectories are calibrated by a post-hoc conformal prediction
module to get valid and efficient confidence regions. We evaluated the entire
framework using the real-world V2I dataset V2X-Seq. Our results demonstrate
superior performance in terms of Final Displacement Error (FDE) and Miss Rate
(MR) using a single GPU. The code is publicly available at:
https://github.com/xichennn/V2I_trajectory_prediction.