Model Composition: Can Multiple Neural Networks Be Combined into a Single Network Using Only Unlabeled Data?

The diversity of deep learning applications, datasets, and neural network architectures necessitates a careful selection of the architecture and data that match best to a target application. As an attempt to mitigate this dilemma, this paper investigates the idea of combining multiple trained neural networks using unlabeled data. In addition, combining multiple models into one can speed up the inference, result in stronger, more capable models, and allows us to select efficient device-friendly target network architectures. To this end, the proposed method makes use of generation, filtering, and aggregation of reliable pseudo-labels collected from unlabeled data. Our method supports using an arbitrary number of input models with arbitrary architectures and categories. Extensive performance evaluations demonstrated that our method is very effective. For example, for the task of object detection and without using any ground-truth labels, an EfficientDet-D0 trained on Pascal-VOC and an EfficientDet-D1 trained on COCO, can be combined to a RetinaNet-ResNet50 model, with a similar mAP as the supervised training. If fine-tuned in a semi-supervised setting, the combined model achieves +18.6%, +12.6%, and +8.1% mAP improvements over supervised training with 1%, 5%, and 10% of labels.

Introduction
Deep learning has enabled achieving outstanding results on a wide range of applications in computer vision and image processing [5,27]. However, the diversity of datasets and neural network architectures necessitates a careful selection of model architecture and training data that match best to the target application. Often times, for a same task, many models are available. These models might be trained on different datasets, or might come in different capacities, architectures, or even bit precisions.
Motivation: A natural question that arises in this case, is whether we can combine the neural networks so that one combined network can perform the same task as several input networks. Fig. 1 shows an example, where two input object detection models to detect 'person' and 'vehicle' are combined in one model. The benefits of combining models include: a) possible latency improvements due to running one inference as opposed to many, b) in case input models cover partially overlapping or non-overlapping classes/categories, one can build a stronger model with the union of the classes/categories through model composition An example of the proposed model composition approach for object detection. Two input models for detecting 'person' and 'vehicle' categories are combined to create one model that does both.
(i.e. merging models' skills as in Fig. 1), and c) for applications involving model deployment, e.g. for cloud services providers, it can reduce the deployment frequency/load.
Challenges: Creating a combined model from several input models is a challenging task. First, depending on the target task, the output model may need to have a specific architecture, and not necessarily one that is dictated by the input models. The input models themselves also might have different architectures. Second, in case input models are provided by users of a cloud system, or by different model creators/clients, the individual model owners would likely prefer not to share their training data, labels, not even weights or code. A privacy preserving model composition approach should rely on only a minimum amount of information from the model creators. Third, input models may have only partially overlapping or disjoint class categories. This imposes a major challenge when combining the individual models.
Existing methods: The existing solutions are mostly based on techniques such as knowledge distillation [6,16,34] or ensembling [35], which may be useful when classes/categories are identical and labeled data are available, but not for the case of arbitrary classes/categories with only unlabeled data. More details regarding the existing approaches are provided in section 2. In summary, to the best of our knowledge, the existing methods do not fully address the three challenges mentioned above.
Our contributions: In this paper, we propose a simple yet effective method to address the model composition of neural networks. Our method supports combination of an arbitrary number of networks with arbitrary architectures. To train a combined model, we leverage the abundance of unlabeled data and having labels or original training data of the input models is not a requirement. However, if any labeled data are available, the algorithm uses them to further boost the performance of the output model. Furthermore, we put no restrictions on the type and number of object categories of the input models. We demonstrate the effectiveness of our method through an extensive set of experiments for the task of object detection.

Related works
Related to our work are the following approaches: Network Ensembling: Ensembling is a common way of aggregating the predictions of more than one models. Ensembling strategies are well explored in the literature [9,28,35]. Simplest ways could be naive averaging of predictions. Architectural Combination: These methods create new architectures from the input models. Adaptive Feeding (AF) [33] proposes to use simultaneously two small and large networks that are trained to perform a same task. A linear classifier decides which examples to go to the small or large model. Their goal was to improve the inference speed. In another work, [10] Unifying&Merging (U&M) proposes to design a new architecture based on existing input architectures, to support learning multiple tasks. MultiTask Networks: MultiTask networks learn multiple tasks in one model [18,25,31]. Tasks train simultaneously, not by combining already trained individual networks.
Incremental Learning (IL): Gradually adding new categories while trying to limit the catastrophic forgetting [22]. Dataset Merging: [23] Dataset merging is closest work to our study. It proposes to combine datasets by filling the missing annotations of non-overlapping categories.
While existing works have some correlation to the problem we study, none of them directly addresses this problem. Specifically, our proposed method combines neural networks of same tasks (e.g. classification, detection, etc.) using unlabeled data. If any labels are available, it will use them to further boost the performance. On the other hand, most existing methods mentioned above require labels to be available. In addition, our method supports having overlapping or disjoint target label categories while existing methods such as multitask networks, U&M, AF, ensembling, or Dataset Merging only support homogeneous categories. Moreover, our method is architecture agnostic. In contrast, AF, MultiTask, IL, and Dataset Merging, don't support arbitrary input model architectures.
It is also worth noting that most existing methods require access to input model weights or code, to construct a combined model. Methods such as multi-tasking, IL, Dataset Merging or U&M need full access to input models, in order to design a new combined architecture. Our method only requires an inference API, and thus treats the input models as black boxes, which in turn leads to a better privacy protection for the clients.

Model composition strategy
This section provides details regarding the model composition method we use in this paper. Fig. 2 shows the inputs and outputs of this process. In addition, Fig. 3 shows a flow-diagram of different steps within this method. As observed from Fig. 3, a number of models are provided as inputs. We then collect the predictions of these models over an unlabeled set of images. These predictions are filtered and aggregated to form a set of generated pseudolabels, which is later used to train the output model M. If any labeled data are available, M will be fine-tuned on them. Algorithm 1 shows a break-down of these different steps.
An embodiment of how our method would be implemented for usage in a cloud services provider platform is demonstrated in Fig. 4. As shown in this figure, in the context of a cloud services provider, model composition can be leveraged for: less frequent model/data deployment/transfer, building stronger models, faster overall inference, empowering the model markets, and encouraging users to share their models in an incentive sharing strategy. Since the input model predictions are not always perfectly accurate, the generated pseudo-labels will be noisy, and therefore less reliable. These training examples could have an adverse impact on the training of the output model. We filter out such kind of examples, by employing an entropy-based thresholding mechanism.  For a given input x and a network function f such that p = f (x), the entropy is given by

Filtering pseudo-labels
for some threshold τ. Although entropy thresholding does not guarantee a perfect filtering of bad pseudo-labels, but in practice it works well and has been used as a confidence indicator for similar purposes [24,26,30]. Note that for some tasks such as object detection, models output a confidence score that can also be used for filtering bad pseudo-labels.

Aggregation of pseudo-labels
Next, in the pseudo-label aggregation phase, we employ a consensus based strategy, where the majority of the input models need to agree on a pseudo-label in order for it to qualify as a candidate and pass to the next step. Pseudo-label aggregation can be done in various ways such as unanimous (all models agree), affirmative (union of all predictions), consensus (majority voting), etc. [9]. Our experiments showed all these methods can be used with minor performance variations. We chose the consensus approach for the experiments since for combining a higher number of models, intuitively it makes more sense (See section 4 for a 10-model example). Note that for some tasks such as image classification, the aggregation will be a simple majority voting mechanism. For some other tasks such as object detection, it becomes more complicated due to the nature of the task. Here, we review our method of pseudo-label aggregation for object detection, which can also be extended to other similar tasks such as instance segmentation, tracking, etc.
Details of the pseudo-label aggregation strategy: Let D denote the unlabeled dataset used. The input to the pseudo-label aggregation procedure is a list S = [y 1 , ..., y N ], where each y i itself is a list of detections from an input model over all unlabeled training images in D. We then create a new list S im = [p 1 , ..., p |D| ] so that each p i contains predictions of all models on one single image, and length of S im is equal to number of all images in D.
Next, for each element p i , we unite the predictions by their category names and the overlapping of their bounding boxes. If the overlapped area of any two elements in p i is higher than a certain threshold, and meanwhile if these two elements are of the same category, then they are treated as detections of the same object, which are further grouped together into a sub-list: p i = {p i j }. Subsequently, we decide whether to keep each element p i j depending on the number of unique models with predictions included in p i j , denoted by K i j . In the most strict case, p i j is kept in the list only when K i j = N i j , where N i j ≤ N is the maximum number of models that may predict the object category corresponding to p i j ; If we want a majority voting, then p i j is kept when K i j ≥ N i j /2; If a simple stacking strategy is used,

Algorithm 1 Our Model Composition Approach
Inputs: Input models M 1 , ..., M N ; unlabeled data D Optional: Output model architecture; labeled data D Output: Algorithm 2 Our Pseudo-Label Aggregation Inputs: Pseudo-labels S from N input models; unlabeled data D; aggregation strategy A Output: Aggregated pseudo-labels list S procedure AGGREGATE(S, N, then p i j is kept regardless of K i j . At this point, each p i j could still have several candidate detections for the same region. Processing all of them through the detection network is not only cumbersome but could also decrease the overall performance. Therefore, we applied soft non-maximum suppression (Soft-NMS) [8] to each p i j to filter the predictions a second time. Algorithm 2 formally captures these steps and Figure 5 demonstrates an example. Remark 1: p i j for image i, represents a list of bounding boxes predicted on a particular object j, i.e. detections of a same object by different models. K i j is the number of unique models in p i j . N i j is the number of models that have the category of p i j in their label set (i.e. number of models that actually have the capability of detecting that object category). As such, in general K i j ≤ N i j ≤ N. In an ideal case where all eligible models can detect an object i j, we will have K i j = N i j . If all input models have the same category label set, N i j = N. In the case input models have different but overlapping categories (i.e. there is at least one category that is not supported by all models), for at least some i and j, N i j < N. If all models have strictly different categories (no overlap), K i j = N i j = 1. And finally if some particular categories only belong to one model, for those categories K i j = N i j = 1.

Remark 2:
The experiments in Section 4 contain various practical scenarios in which different aspects of our method are evaluated. Moreover, Figure 9 in supplementary materials [7] shows a scenario where 10 input models with a diverse count and type of object categories are combined. For example in this figure, 'teddy bear' is only in In addition, we also explore in Section 4 the task of combining a face detection model with a mask detection one.

Training pipeline
Once pseudo-labels are filtered and combined, they will be used to train the output model architecture. Any available labeled data will be used in a final fine-tuning stage to improve the performance. Note that pseudo-labels are generated from unlabeled data. This is because input model owners may only share an interface to their model API for inference, not necessarily the weights, code, architecture, training data, or labels, to protect their privacy. We treat the input models as black boxes. In other words, we only pass a set of arbitrary unlabeled images through them, and collect their predictions to use as pseudo-labels. This further allows us to choose an arbitrary architecture and size for the output model that combines the class categories of the input models. Consequently, our model composition method is agnostic to the training hyper-parameters of the input models such as various optimizers, learning rate schedules, batch sizes, etc.
It is also worth noting that this way of creating composite models can help with light domain shifts. As we see in section 4, input models trained on different datasets (for the same task) can still be effectively combined (even with different sets of categories). To what extent exactly our method can robustly support domain shifts remains out of the scope of this work, and we leave that as a future direction.

Experiment results and discussions 4.1 Experiments setup
Selected model architectures: We have selected the task of object detection as the main experiments task due to its importance and wide-spread usage in practical applications. That being said, we will also provide results on the task of image classification, as it is often used as a baseline experiment task. For object detection, we utilized the following architectures: EfficientDet-D0 [29], EfficientDet-D1 [29], and RetinaNet-ResNet-50 [21]. For the classification task, we used: ResNet-18 [15], ResNet-152 [15], and DenseNet-121 [17].
Evaluation metrics: We follow the common practice by using the mean Average Precision, mAP @IoU=0.50:0.95, as the main metric to evaluate the performance of object detection models. We report top-1 accuracy for classification.
Training protocols and settings: We adopt code from [1] for the object detection experiments, and use the same training hyper-parameters with ImageNet [11] pre-trained backbones. We trained all the models using SGD with a momentum of 0.9. We increased the learning rate from 8e −3 to 8e −2 for the first epoch and then trained the remaining 300 epochs using a cosine decay rule with a moving average decay set at 0.9998. Soft NMS was utilized to filter the pseudo-label detections in our method. We used an IoU threshold of 0.5 and a confidence threshold of 0.001. For the classification experiments, models were trained for 200 epochs, using an in-house code-base. An SGD optimizer with momentum 0.9 was used, and learning rate was exponentially increased from 0 to 0.01 for the first 8 epochs and then annealed down exponentially to 0.0001 in the remaining epochs.

Object detection results
Our experiments are categorized in various scenarios, which are explained in this subsection. These scenarios cover various possible cases of input models' architectures, training data, and what kind of unlabeled data were used in our algorithm. Table 1 provides a summary of these scenarios. In Table 1, training data in each case is constructed from the training set    Table 2 and Figure 9 (supplementary), and for scenario 2 & 3 it follows the distributions of COCO & VOC. Next, we will go over the details of each experiment. Scenario 1: Combining detectors with different expertise: We took 3 models, each trained on a subset of the COCO dataset but designed for a different purpose, one for detection of transportation related objects, one for sports related, and the other for home objects. These categories have some partial overlap. Table 2 shows the object categories used for each model. The combined model achieved by our model composition procedure combines the skills of the input models, and builds a stronger model with all object categories. We tried our method in two ways: one using unlabeled COCO images (similar data distribution to training data, but without using the labels), and the other using unlabeled OID (open images) dataset (entirely different dataset with different distribution). The upper-bound of the performance would be to train a model with all labels of all object categories (supervised). This model achieved 35.11% mAP on validation set of COCO (considering only object categories corresponding to the ones it was trained on). On the same validation set, our method achieved 32.61% when using unlabeled COCO, and 30.97% when using unlabeled OID. This shows that our method can effectively combine the models with different expertise, and achieve a performance close to that of the supervised upper-bound model. We further investigated the performance of our method if partial labels are available for fine-tuning, in a semi-supervised manner. Table 3 shows the results for this experiment. As observed, with fine-tuning, our method could even surpass the supervised model with 100% of labels.  Pascal-VOC datasets respectively. Similar to the previous scenario, we studied two choices of unlabeled data for our Model Composition method: a) unlabeled data from the same distribution as training data (in this case COCO+VOC images without using labels), and b) unlabeled data from a different dataset all together, e.g. the OID dataset. Note that the input models were trained on a different number of object categories (with overlap), and the output combined model was trained to support the union of object categories of the input models. Table 3 shows the results of this experiment. It is observed from Table 3 that in the unsupervised case (i.e. no labeled data was used), our method achievs 29% and 27.4% mAP, close to the fully supervised performance of the upper-bound model. We also see from Table 3 that when partially labeled data are used for further fine-tuning, our method shows significant improvements over supervised training. In particular, when using 1%, 5%, and 10% of labels, our method shows +22.1%, +13%, and +10.3% gaps over supervised training.
Scenario 3: Combining input models with different architectures, that are trained on entirely different datasets. In this scenario, we studied the most generic case, in which input models have different architectures, are trained on different datasets, and with a different number of object categories. The output model also was chosen to have a different architecture than the input models (See Table 1). This scenario evaluated whether our method can combine the knowledge of models trained on different circumstances, data, and architecture, to a desired new and different architecture. Table 3 shows the results of this experiment. It is observed from Table 3 that our method is very effective, and in some cases performs even better than supervised training with 100% of labels. When partial labels are available for fine-tuning, our method shows a strong performance, with large gaps compared to supervised training, especially in the low label range. Moreover, Table 3 shows that unsupervised training with our method achieved an mAP of 34%, only 1% below supervised training with all labels. After fine-tuning, we were able to meet the same performance as supervised training with only 10% of the original data.
Scenario 4: Having a large number of input models. This scenario investigates the case when a larger number of input models are provided. This would increase the diversity among the models since they can be trained on different data, or object categories, and thus results in a more challenging situation. To this end, we assumed 10 input models. Each model was trained on a randomly selected subset of the COCO dataset, so that training data for each model had no overlap to the other models. However, object categories could have overlap, as their type and count were chosen randomly. Supplementary materials [7] provides a visualization of the type and count of the object categories used for these 10 models. Since each model was trained with roughly 10% of the COCO training set, different number of object categories for different models resulted in a different per-class size of training data. The imbalance here made model composition harder, but mimicked realistic situations where training data can in fact be imbalanced for input models. As mentioned, for these 10 models, categories were randomly selected and the number of categories was selected from 5, 10, 20, 30, and 40. Note that generating 10× pseudo-labels on unlabeled data can be timeconsuming (although it can be parallelized in production). Therefore, we only used 100K randomly selected examples from the OID dataset for this experiment.
We observe from Table 3 that our model composition method can effectively combine the 10 input models into a single new model with the union of their object categories.
Remark 3: A note on the unsupervised performance of OID: As observed in Table 3, in the challenging scenarios of 3 and 4, the unsupervised (0% labels) performance of model composition with OID U is considerably lower that that of COCO U or [COCO+VOC] U . In this regard, there are a few points worth a mention: • In general, using unrelated arbitrary data is expected to result in a lower performance compared to using data from the same distribution as the input models' train set, since pseudo-labels will be less reliable. This is exacerbated in challenging tasks such as scenario 3 where input models are trained on different data and have different architectures with respect to each other and the output model; or in scenario 4 where there are a large number of input models trained on different small-scale data. • It is worth reminding that the case of purely unsupervised model composition means combining an arbitrary number of black-box models (trained on arbitrary data with arbitrary architecture or categories), all without using any labels. In that sense, the real baseline to compare against is the supervised training, which performs much worse than model composition in low data regimes, even in the case of unrelated OID U data. • Moreover, the main goal of the paper is to explore whether or not neural networks can be combined using only unlabeled data, and if yes, to what extent (hence the title). We observe from the results that the answer is for the most part yes; however, in case unlabeled data from the original distributions was not available, for some challenging scenarios a small percentage of labels may be needed to achieve a decent performance. • In a completely unsupervised setting, model composition can still effectively combine input models. The performance will be improved if the unlabeled set size is larger. Remark 4: A note on practical applications: As mentioned in Section 3, a fundamental motivation for our work is a cloud services application, as shown in Figure 4, in which engineers and expert users can leverage a model composition service to build stronger models with combined skills, especially in the presence of a large variety of trained models and datasets on the cloud. Different scenarios in the experiments were also inspired by such a philosophy, but designed at various levels of difficulty. Here, we add a new practical usecase. In this new experiment, we combine separate models of face and mask detection, to build one that is suitable for both face & mask detection. Results are shown in Table 4 [13]. We also provide an ablation on the filtering and aggregation modules.  Ablation on the number of categories: Next, we study the impact of the number of class categories in the performance. To this end, we take two input models from scenario 4, and combine them with a varying number of classes. M 1 is trained on 5 and M 2 is trained on 10 object categories (see Figure 9 in supplementary). Each time we add a number of random categories of M 2 to M 1 , so the combined model can have 5,6,...,15 classes. Table 5 shows the results. Note that in each case the validation/training set will be different as it includes images of a particular set of categories. M 1 and M 2 have mAP of 27.1% & 25.4%, respectively (each has roughly 12K training, and 1K validation examples). In general, higher number of classes results in slightly lower mAP, but we should also note that unlabeled set size becomes also larger (i.e. more pseudo-labels).

Image classification results
In addition to our main results on the task of object detection, we also provide a highlight of our results on the task of image classification. Similar to object detection, we designed the classification experiments in the form of different scenarios. In both scenarios, we tried model composition with unlabeled data from the Caltech-256 dataset (i.e. similar data distribution but without labels), and a 160K subset unlabeled data from OID (i.e. a different dataset altogether). Table 6 shows the results for these scenarios.
In Table 7, we provide a comparison between our method and two additional baselines: i) a simple model ensemble by aggregating directly the prediction of the input models; ii) knowledge distillation when using the input models as teachers, such as [4,16]. For the second baseline, we consider the vanilla distillation [16] but with soft labels.
It is observed from the results that the proposed method is effective in combining image classification models. In both the unsupervised and semi-supervised cases, our method performs competitively compared to supervised models, even when 100% of labels are used.   Table 7: More classification results on OID U . This paper proposed a method for combining multiple trained neural networks into a single model, using unlabeled data. To this end, first the input models' predictions (pseudo-labels) were collected. The pseudo-labels were then filtered based on confidence scores of the predictions. Next, a consensus aggregation strategy was incorporated to combine these pseudolabels. The remaining pseudo-labels were used to train the output model. The proposed method supported the use of an arbitrary number of input models with arbitrary architectures and categories. Performance evaluations on various datasets, tasks, and network architectures demonstrated the effectiveness of the proposed method.

Supplementary materials
This section contains the supplementary materials.

Source code
We share our implementation code to make it easy to reproduce our results. The source-code is attached to the supplementary materials in a 'code' directory. We also provide detailed instructions for training and evaluating our models in 'README.md' files.  Table 3. We observe from this figure that in low data regimes model composition performs considerably better than supervised training with partial data. Fig. 7 shows an extended visualization on the cloud embodiment introduced in the paper. In this figure, we provide an easier comparison between before & after incorporating the model composition as a service. Moreover, Fig. 8 demonstrates an example of pseudo-label aggregation procedure of Algorithm 2. In addition, Fig. 9 visualizes the data splits of object detection scenario 4, where we combined 10 models trained on different COCO subsets.


Maria: A Visual Experience Powered Conversational Agent

Arguably, the visual perception of conversational agents to the physical world is a key way for them to exhibit the human-like intelligence. Image-grounded conversation is thus proposed to address this challenge. Existing works focus on exploring the multimodal dialog models that ground the conversation on a given image. In this paper, we take a step further to study image-grounded conversation under a fully open-ended setting where no paired dialog and image are assumed available. Specifically, we present Maria, a neural conversation agent powered by the visual world experiences which are retrieved from a large-scale image index. Maria consists of three flexible components, i.e., text-to-image retriever, visual concept detector and visual-knowledge-grounded response generator. The retriever aims to retrieve a correlated image to the dialog from an image index, while the visual concept detector extracts rich visual knowledge from the image. Then, the response generator is grounded on the extracted visual knowledge and dialog context to generate the target response. Extensive experiments demonstrate Maria outperforms previous state-of-the-art methods on automatic metrics and human evaluation, and can generate informative responses that have some visual commonsense of the physical world.

Introduction
Building intelligent conversational agents that can not only converse freely with human but also have the ability to perceive the physical world, has been one of the longest standing goals of natural language processing (NLP) and artificial intelligence (AI). Although the recent large-scale conversation models trained on text-only corpora, such as Meena  (Adiwardana et al., 2020), Blender (Roller et al., 2020) and DialoGPT , have shown the compelling performance, they are still lack of the perception ability to our physical world. A recent study (Bisk et al., 2020) points out the successful linguistic communication relies on a shared experience of the world that makes language really meaningful. The visual perception is a rich signal for modeling a vastness of experiences in the world that cannot be documented by text alone (Harnad, 1990). On the other hand, human-human conversations involve their understandings of context, the background knowledge they had, and perhaps most importantly the experiences of the world they shared, e.g., what they have seen before. Figure 1 shows a conversation between humans. Human-A recalls his/her past experience of playing volleyball or having BBQ on the beach when human-B talks about vacation on the beach of Hawaii. However, the association relationship between beach and volleyball (or BBQ) is hard to capture in traditional knowledge bases, such as knowledge graph. Motivated by this, we select a common word "pizza" and collect the top 17 words that mostly co-occur with "pizza" on Google Knowledge Graph 1 and MS-COCO images 2 (Lin et al., 2014). As shown in Figure 2, the words cooccurring with "pizza" on knowledge graph tend to be the abstract concepts, while the co-occurrence relationship of object tags on images reflects some commonsense of our physical world, e.g., "pizza" is usually on the "dining table", people usually use "knife" when eating "pizza". Interestingly, we found the "pizza" also co-occurs with "cell phone" and even "plotted plant". This indicates when people eat pizza, they sometimes would put their cell phones aside on the table, or there might exist some plotted plants in the restaurant. Thus, empowering conversational agents to have the visual perception ability about the physical world is a key way for them to exhibit the human-like intelligence.
The existing works (Mostafazadeh et al., 2017;Huber et al., 2018; focus on exploring the multimodal dialog models that ground the conversation on a given image. Recently,  propose to learn the dialog generation model with both image-grounded dialogs and textual dialogs by resorting to text-to-image synthesis techniques Qiao et al., 2019) to restore a latent image for the text-only dialog. Even so, these works are still constrained by the assumption that the dialog is conducted center around a given (or synthesized) image.
In this paper, we take a step further to extend the assumption of image-grounded conversation to a fully open-ended setting where no imagedialog pairs are assumed available. Specifically, we present Maria, a neural conversational agent powered by visual world experiences which are retrieved from a pre-built image index, e.g., the 1 https://developers.google.com/ knowledge-graph/ 2 We calculate the co-occurrence distribution of object tags from the images in MS-COCO dataset. More examples could be found in Appendices.
Open Images Dataset (Kuznetsova et al., 2018). Maria consists of three components: text-toimage retriever, visual concept detector, and visualknowledge-grounded response generator. The retriever is responsible for retrieving a piece of visual world experiences, e.g., a correlated image to the dialog from an image index. The visual concept detector utilizes the object detector from UpDown (Anderson et al., 2018) to extract the regions features (i.e., bboxes) and the corresponding visual concepts (i.e., tags) from the retrieval images. Hence, we can construct (bboxes, tags, context, response) 4-tuple as the training data. Finally, these constructed 4-tuples are used to train the visualknowledge-grounded response generator, which is built on the top of a multi-layer Transformer architecture (Vaswani et al., 2017). To effectively inject the visual knowledge into the response generator, we carry out the Masked Concept Prediction and Visual Knowledge Bias besides the response generation objective. The former aims to align the semantic representations between textual words and image regions, while the latter tries to provide more visual knowledge to facilitate the dialog generation. The experimental results on Reddit Conversation Corpus (Dziri et al., 2019a) demonstrate that Maria significantly outperforms previous state-of-the-art methods, and can generate informative responses with visual commonsense of our physical world.
Overall, the contributions of this paper are summarized as follows: • We explore the task of image-grounded dialog generation under a fully open-ended setting where no specific image-dialog pairs are assumed available, i.e., zero-resource imagegrounded conversation. To the best of our knowledge, this is the first work to connect dialog corpus with the unpaired image data; • We present Maria, a neural conversational agent consisting of three flexible components, which can effectively capture the visual commonsense from images and accordingly generate informative and vivid responses; • Extensive experiments on the widely used Reddit Conversation Corpus are conducted to justify the effectiveness of Maria.

Related Work
Vision and Language In the research of vision and language, various tasks have been extensively studied, such as image captioning Lu et al., 2017;, visual question answering (Antol et al., 2015;Anderson et al., 2018), visual dialog (Das et al., 2017a,b). Popular benchmark datasets in this area include MS-COCO (Lin et al., 2014), VisDial (Das et al., 2017a) and Visual Genome (Krishna et al., 2017). Visual dialog is a task to answer the questions about the factual content of the image in a multi-turn manner. Differently, image-grounded conversation studies how to reply to a dialog context and a given image with proper responses in an open-ended way.
Dialog Generation Encouraged by the success of the neural sequence-to-sequence architecture (Sutskever et al., 2014) on machine translation, end-to-end neural approaches on open-domain dialog generation (Vinyals and Le, 2015;Shang et al., 2015;Serban et al., 2016;Sordoni et al., 2015;Xing et al., 2017;Wu et al., 2018;Adiwardana et al., 2020) have been widely studied in literature. Recently, there is an emerging trend towards grounding the dialog generation models on the external knowledge, such as knowledge graphs (Zhou et al., 2018), documents (Ghazvininejad et al., 2018;Dinan et al., 2019;Kim et al., 2020;Zhao et al., 2020a,b;Li et al., 2020) and images (Mostafazadeh et al., 2017;. Different from the previous work on knowledge-grounded conversation that connects dialogs with unpaired document knowledge , our work lies in the research of image-grounded conversation where a response is generated with a dialog context and a given image. Existing works (Mostafazadeh et al., 2017; in this direction assume there is a given (or synthesized) image for the dialog and explore the multimodal dialog models. In contrast to these works, we study the image-grounded conversation under a fully open-ended assumption where no paired dialog and image are assumed available, i.e., zeroresource image-grounded conversation.

Problem Formalization
Suppose we have a dialog set . . , n}, C i refers to a dialog context and R i is a response to C i . We assume there is a set of images V = {V j } m j=1 , where ∀j ∈ {1, . . . , m}, V j denotes an image. ∀C ∈ D, we assume that there is an image V that triggered by the given dialog context C and response R. Our goal is to estimate a generation model P (R|V, C) from D and V. Thus, given a new dialog context C associated with an image V , the model can generate a response R according to P (R|V, C).

Methodology
To learn such a generation model P (R|V, C), we need to tackle several challenges: (1) How to bridge the gap between unpaired dialog corpus and image data; (2) After obtaining the correlated images, how to extract the detailed visual features and concepts; (3) How to effectively inject the visual knowledge into response generator and enable it to generate responses that are visual-knowledge-grounded. Figure 3 illustrates the framework of our approach. We first build a large-scale image dataset and leverage a cross-modal matching model to retrieve a correlated image using the content of the dialog. Then an off-the-shelf object detector is applied to extracting the object features and visual concepts from the retrieval image. Finally, the response generator is trained to generate the target response conditioned on the context, extracted object features, and visual concepts. In the rest of this section, we will elaborate these three modules.

Text-to-Image Retriever
In this section, we develop a retrieval model that assigns each dialog with a correlated image V . Specifically, we train a text-to-image matching model from image captioning dataset and utilize it to construct the (C, R, V ) triple data.
Modeling To improve the efficiency of crossmodal retrieval model on large-scale dialog corpus and image dataset, we adopt a two-tower architecture (Lu et al., 2019) to accelerate the retrieval process where the image features can be pre-extracted offline. The model takes a sentence T and an image V as input, and predicts the relevance score s(T, V ) between the sentence and the image. We use a text encoder and an image encoder to produce the representations of T and V , respectively. The text encoder is a pre-trained BERT-base model (Devlin et al., 2019) and we use the hidden state of special token [CLS] as the embedding of T : Then a Multi-Layer Perceptron (MLP) projects the sentence embedding into the cross-modal space. We follow Tan and Bansal (2020) to perform L2normalization on the last output features, by which we can simplify the nearest neighbor search problem in the euclidean space to the Maximum Inner Product problem (Mussmann and Ermon, 2016): Similarly, the image encoder is composed of a pretrained ResNeXt backbone (Xie et al., 2017) and a MLP with L2 normalization: Thus, we define the relevance score s(T, V ) as an inner product of the language feature representation f t (T ) and image feature representation f v (V ): Training We train the cross-modal matching model on MS-COCO image captioning dataset (Lin et al., 2014), where each image is paired with 5 sentences describing its visual content. The model is optimized by minimizing the hinge loss so that the relevance score s (T, V ) of the positive imagesentence pair can be larger than the negative pair s (T, V − ) by at least a margin M : Inference Given the trained retrieval model, we can now assign each dialog with a correlated image V . To ensure the diversity and richness of the retrieval results, we fetch 500,000 images from the large-scale Open Images dataset (Kuznetsova et al., 2018) as our image set V. The image V i ∈ V with the maximum relevance score is paired with the given dialog (C i , R i ) ∈ D. Note that for the dialog in the training set, we use both the context C and response R are concatenated as the query for retrieval (i.e., T = (C, R)), which is beneficial to retrieving an image with the related visual knowledge. On the other hand, for the validation/test set of the dialog corpus, the query is only the context (i.e., T = C) so as to keep consistent with the real-world setting where the response is unavailable and need to be generated at inference.

Visual Concept Detector
Given the correlated image V i to the dialog as the visual clue, we can now extract the visual knowledge from it. One naive approach is to utilize the CNN-based models to extract the latent image features. However, this approach does not consider the fine-grained representation modeling for images, which is crucial for the dialog model to understand the local visual features in images. To address this issue, we adopt an object detection model (Anderson et al., 2018) pre-trained on Visual Genome (Krishna et al., 2017) to extract a set of salient object features O = {o k } K k=1 , where each object feature o k is a 2048-dimensional vector. These features represent the images at the level of objects and other salient regions, which has proven to be vital in many high-level image understanding tasks. Besides, the same detector is used to extract a set of visual concepts Q = {q m } K m=1 , where each concept q m is the high-precision textual label of the visual region, e.g., "sunset", "melon", etc. In this manner, we simultaneously obtain the fine-grained image representations and the necessary visual concepts for the subsequent dialog generation.

Visual-Knowledge-Grounded Response Generator
In this section, we propose a unified architecture to effectively inject a set of region features and corresponding visual concepts into the response generation model. In following parts, we describe the model design and training objectives in detail. Figure 4 shows the architecture of our response generation model, which is a multi-layer transformer network for both bidirectional vision/context (O, Q, C) encoding, and unidirectional response R decoding, via the flexible self-attention masks inspired by (Dong et al., 2019).

Input Representation
For each token, the final input representation to the multi-layer transformer network is the elementwise summation of four kinds of embeddings, including token-level, turn-level, position-level, and segment-level. Then, we concatenate all the input representations to one sequence for model training.
Token-Level The token-level embeddings are the concatenation of (O w , Q w , C w , R w ), which denote the token embedding sequence of visual objects, visual concepts, contexts and response respectively. Note that O w is the object embedding transformed by a linear layer into the same dimension as word embedding.

Turn-Level
Since the dialog is multi-turn, we encode this turn order with a relative turn embedding (Bao et al., 2020). Specifically, the turn number is counted from the last utterance of the dialogue to the beginning. Note that as for the tokens corresponding to O and Q, we simply set them the same as the first utterance of C.
Position-Level Positional embedding encodes the signal of the token order in the total input sequence, which is the same as positional encoding of the original transformer (Vaswani et al., 2017).

Segment-Level
Segment embedding is employed to differentiate which segment the token is in, i.e., O, Q, C or R.

Masked Concept Prediction
Due to the inherent gap between visual modality and textual modality, directly optimizing the model by response generation objective may result in the insufficient utilization of the visual knowledge. To align the semantic representations of two modalities, we devise Masked Concept Prediction (MCP) objective. 15% of the visual concepts are randomly replaced with [MASK] tokens in each training instance, which need to be predicted by the model. However, one problem still remains, i.e., the visual concepts have no specific order when extracting from images. In other words, we need to model MCP as a matching problem of set, which does not need to consider the order of predicted concepts when there are more than two concepts masked out simultaneously. To tackle this, inspired by , we adopt the Hungarian Matching Loss (Stewart et al., 2016;Carion et al., 2020) to estimate an optimal mapping α so that the prediction for each masked position is assigned one of the target concepts. Here we denote the set of all input as X = (O, Q, C, R), the set of the bidirectional self-attention part of X as B = (O, Q, C), the set of masked concepts asQ, the set of unmasked tokens as B\Q, and the prediction probabilities of the corresponding representations in the final layer of transformer as H where h i is the probability distribution of the i-th masked position. Hence, the MCP loss can be defined as: where α(i) is the index of the target concept assigned to the i-th prediction. When predicting a masked concept, the model will have to resort to visual region features, dialog contexts and other unmasked visual concepts. This would help the model to align the cross-modal representations between text and visual regions.

Masked Response Prediction
Encouraged by the success of UniLM (Dong et al., 2019) in Seq2Seq tasks, we adopt the Masked Response Prediction (MRP) objective to model the response generation. During training, 70% of the tokens in R are randomly masked with the special token [MASK]. The model is optimized to recover the masked tokens. The masked response tokens and other unmasked tokens in the whole input sequence can be denoted asR and X\R, respectively. Suppose that p i is the conditional probability distribution of the i-th token in R, the MRP loss is the Negative Log-Likelihood (NLL) of the masked  response tokens as follow: (7) Note that the self-attention mask in R is left-toright, but the rest are bidirectional. In other words, the tokens in O, Q and C can attend to each other from both directions, while the tokens in R can attend all tokens in O, Q, C and the leftward tokens in R including itself. MRP implicitly encourages the model to generate responses by learning the relationship among all input tokens. For decoding, we first encode the image regions, visual concepts, dialog contexts, and a special token [BOS] as input. Then the model starts the generation by feeding a [MASK] token and samples a word from the predicted distribution over vocabulary. Then, the [MASK] token is replaced by the generated token and a new [MASK] is appended to the input sequence for next word prediction. The generation process terminates when the model predicts [EOS] token or reaches the pre-defined maximum length.
Visual Knowledge Bias Normally, the top projection layer of generation model produces a probability distribution over the vocabulary: where the e r ∈ R d , W ∈ R |V |×d and b ∈ R |V | are the last output of the transformer network, weight and bias parameters of the decoding head, respectively. |V | denotes the vocabulary size. So far, the visual world knowledge is introduced into the response generation model by the shared-parameter self-attention layers. To further inject the visual knowledge into the generation model, we design a simple but effective strategy, namely Visual Knowledge Bias (VKB). Concretely, an additional visual vocabulary bias b q is first calculated as follow: where F q : R d → R |V | is a projection layer. e q avg denotes the average pooling on all hidden representations of visual concepts, i.e., e q avg = AvgP ooling(E q ) where E q = (e q 1 , ..., e q K ). Then, we mask non-visual-concept tokens in the vocabulary and the masked vocabulary biasb q ∈ R |V | is added to the top layer of generation model to get the final distribution over vocabulary: We leverage this final vocabulary distribution to calculate the MRP loss in Eq. 7 to optimize the model. This visual knowledge bias would encourage the model to generate more visual knowledge related tokens in the response. To sum up, the final objective of our response generation model is to minimize the integrated loss: 5 Experimental Setup

Datasets
To evaluate the performance of Maria, we conduct comprehensive experiments on the Reddit dataset released by , which is a large-scale and high-quality multi-turn conversations extracted from Reddit Conversation Corpus (Dziri et al., 2019b). Each dialog has 3 to 5 utterances, and the training/validation/test set has 1M/20K/20K dialogs respectively. We train and validate the retrieval model using the Karpathy's split 3 of the MS-COCO image captioning data, where the images are split into 113.2K/5K/5K samples as training/validation/test set, respectively. After the retrieval model is trained, we fetch 500K images from the Open Images dataset as the image index, and then retrieve images from it by dialog context and response to construct the training data for response generator.

Evaluation Metrics
Both automatic metrics and human evaluation are employed to assess the performance of Maria and baselines. Automatic metrics include: (1) Fluency: perplexity (PPL) measures the confidence of the generated responses; (2) Relevance: BLEU-1 (Papineni et al., 2002), Rouge-L (Lin, 2004), and we follow Serban et al. (2017) to utilize Embedding Average cosine similarity, Vector Extrema cosine similarity, and Embedding Greedy Matching score. All this metrics are calculated by running the public NLG evaluation script 4 ; (3) Diversity: Distinct-1 (Dist-1) and Distinct-2 (Dist-2) (Li et al., 2016) are defined as the number of distinct uni-grams or bi-grams divided by the total amount of words.
In human evaluation, we randomly select 100 dialogue contexts and the corresponding generated responses for Maria and compared baselines. Three human annotators are asked to score the response quality on a scale of {0, 1, 2} from three aspects, including Fluency, Relevance and Richness. The higher score means the better. Since each response receives 3 scores on each aspect, we report the average scores over annotators and responses. The inter-annotator agreement is measured by Fleiss' Kappa(Fleiss and Cohen, 1973).

Implementation Details
For the retrieval model, ResNeXt-101-32x8d feature is used as the visual embedding, while the concatenation of the last 4 layers of BERT's outputs is used as the textual embedding. Both embeddings are then respectively fed into an MLP composed of three layers of size (1024, 1024, 512). When training the retrieval model, we set the margin M = 0.5 for the hinge loss, and only tune the parameters of both MLPs while freezing the parameters of ResNeXt and BERT. The total training epoch is 20. At inference, the FAISS (Johnson et al., 2019) library is utilized to accelerate the inner product search by batch processing. We use the off-the-shelf object detector from UpDown (Anderson et al., 2018) to extract top-k (k=36) image region features and the corresponding visual concepts. The detector is a Faster R- CNN (Ren et al., 2015) model trained on the Visual Genome dataset (Krishna et al., 2017).
For the response generation model, we set the number of transformer layers L = 12 and the hidden embedding dimension D = 768. Besides, the network parameters are initialized by UniLM. The maximum sequence lengths of context and response are set to 110 and 40, respectively. The sequence lengths of region features and concept tokens are both set to 36. The batch size is 64. We use the Adam Optimizer (Kingma and Ba, 2015) with a learning rate 3e-5 to train the response generation model. The training is conducted on 4 Nvidia Tesla P40 24G GPU cards for 20 epochs.

Baselines
We compare the following baselines in the experiments: (1) Seq2Seq: A standard Sequence to Seqence model with attention mechanism (Bahdanau et al., 2015). (2)   ) that achieves the state-of-the-art performance on benchmarks of dialog generation. (5) ImgVAE: A dialog generation model  that is trained on both textual dialogs and image-grounded dialogs by recovering a latent image behind the textual dialog within a conditional variational autoencoding framework. (6) DialoGPT: An opendomain dialog model ) that fine-tunes GPT-2 (Radford et al., 2019) on massive Reddit data. Since DialoGPT is a dialog generation model trained on the text-only corpus, we introduce it as an auxiliary baseline. For a fair comparison, we choose the same model size (L=12,D=768) of DialoGPT (117M) as our model.

Automatic and Human Evaluations
We summarize the experimental results of automatic evaluations in Table 1. Maria achieves the substantial performance improvements over baselines on all metrics except for the comparison to Di-aloGPT. Especially, Maria significantly surpasses ImgVAE on Dist-1/2, which indicates introducing richer visual knowledge, i.e., image region features Model PPL BLEU-1 Rouge-L Average Extrema Greedy Dist-1 Dist-2 Seq2Seq (Bahdanau et al., 2015)   and the corresponding visual concepts, is beneficial to generating more diverse and informative responses. This also reflects in human evaluation of Table 2 that the richness score of Maria is higher than that of ImgVAE. Besides, in terms of relevance metrics including BLEU-1, Rouge-L, Average, Extrema and Greedy, Maria outperforms all baselines and even performs better than DialoGPT. This indicates introducing the extra visual knowledge related to dialog context can further force the model to produce more relevant responses.
On the other hand, the discrepancy of data distributions between the training data (i.e., Image-Chat  dataset) and test data (i.e., Reddit conversation dataset) of the text-toimage synthesis model in ImgVAE limits its performance in practice. Besides, constrained by the capability of the text-to-image synthesis model, the richness and diversity of the synthesized images are undesirable, while Maria can retrieve a variety of images from the large-scale image index. That may be the reason why ImgVAE consistently underperforms our Maria on relevance including automatic evaluation and human judgement, which also shows the superiority of the retrieval method for the zero-resource image-grounded conversation. Another observation is that Maria slightly underperforms DialoGPT on PPL and Dist-1/2. Since DialoGPT is a large-scale pre-training based dialog generation model and introduces the extra mutual information maximization objective to improve the informativeness of generated responses, which is consistent in human evaluation with respect to fluency and richness.

Ablation Study
We conduct extensive ablation experiments over different model variants and input components to better understand their relative importance to the dialog generation task. As shown in Table 1, training the simplified versions of Maria or removing any visual signals from input components leads to worse performance in terms of relevance and diversity. In particular, the results on the ablation study validate that: (1) The performance improvement of dialog generation benefits from the MCP's effectiveness in aligning the representations of text and vision; (2) When training Maria, introducing VKB can further improve the quality and diversity of generated responses; (3) Rich visual knowledge, i.e., image region features and visual concepts, play a significant role in improving the performance of dialog generation. Especially, removing the visual concepts leads to a dramatic performance drop on diversity. The phenomenon is due to the lack of necessary visual concepts, Maria can not well understand the visual world knowledge when only learning from the visual features.

Case Analysis
To further investigate the quality of responses generated by Maria, we put an example of generated responses in Figure  it, i.e., "the pizza at Aldi is the best in the world". This implies the commonsense that the supermarket usually has the pizza to sell. It is also observed that Maria pays more attention to the relevant image regions when generating the word "pizza", which demonstrates that Maria could capture useful visual knowledge from the image and subsequently leverage it to generate commonsense-aware responses. More cases are demonstrated in Appendices.

Conclusions
In this paper, we present Maria, a neural conversational agent powered by the visual world experiences. It is able to retrieve the visual world experiences with users and generate human-like responses with some visual commonsense. Extensive experiments demonstrate Maria achieves substantial improvements over the state-of-the-art methods in automatic and human evaluation. The future works could include: (1) Design a more precise and comprehensive image retriever to include multiple retrieval images; (2) Combining the retrieve module and dialog generation into an end-to-end model, instead of learning them individually; (3) Explore more efficient neural architectures to inject the visual knowledge into response generation.

A Appendices
In this section, we show more examples of word co-occurrence distributions on Google knowledge graph and MS-COCO images. Besides, some conversation samples produced by Maria and the baselines are also presented in Section A.2.

A.1 Word Co-occurrence Distribution Examples
In Figure 6, we present some supplementary examples of the word co-occurrence distribution on Google knowledge graph and MS-COCO images, including "traffic light", "bed", "book", and "pot plant". Figure 6 (a) shows the co-occurrence distributions of "traffic light" and other words on knowledge graph and images, respectively. As we can see, most of the co-occurred words with "traffic light" are the related concepts such as "smart traffic light", "traffic light protocol", "traffic light rating system", etc. While the co-occurred words on images are usually "car", "person", "truck", "bus", etc, which we often see when walking by the traffic lights. Interestingly, we found "umbrella" and "clock" also co-occurs with "traffic light" in some images. For the former, the picture we can imagine is that people were holding the "umbrellas" when they walked through a zebra crossing under the "traffic light". For the latter, the possible picture is that we can see both the "traffic light" and the "clock" on the top of a high building from a certain angle when walking on the street. Similar observations can be also seen in other examples.

(a) traffic light
Most of the co-occurrence words on knowledge graph are logically-related concepts. However, the co-occurrence relationship of object tags on images reflects some commonsense of our physical world, which implies some pictures that we human could easily imagine. This kind of knowledge is unique and inherent in images, but it can hardly be captured in the traditional knowledge bases, such as knowledge graph. Figure 7 shows some cases from the test set of Reddit data. We observe that the responses generated by Maria are more commonsensical and vivid than those of the baseline methods, which is consistent with our automatic and human evaluation results. Interestingly, Maria is able to retrieve correlated images using the dialog contexts, which makes its response more human-like. For instance, case (a) shows that when the dialog context marvels at "the pass of the world cup", Maria recalls a football player and compliments him "the best player in the world"; case (b) shows that when the dialog context chats about the "Canada weather", Maria is aware of the fact that "Canada" is often "snowy" and then talks about "Canada" in a funny tone, "I've never been to a place that doesn't have snow"; case (c) shows that Maria understands that "swan" is sometimes "dangerous" when they are on the "beach"; case (d) shows that when the dialog context tries to guess one type of game, Maria recalls a ping-pong "ball" game and describes it; and etc.


extfieldsofstudy:
Computer Science
Physics

text:
Object Detection and Classification for Autonomous Vehicle

The proposed work presents efficient objection detection and tracking algorithm for autonomous vehicles, which is developed in MATLAB with Image Processing Computer Version Toolbox and Automated Driving Toolbox. The developed algorithm track and detects moving and stationary objects such as other vehicles, pedestrians, and traffic lanes. Accurate and efficient tracking are important to analyze object behavior. For this work various build in detectors from MATLAB tool box were tested and compared. The evaluation of algorithm was carried on for 19 short videos from 8 seconds to 23 seconds, and then it applied to K-dataset and full road experiments by the Automated Driving Lab. The full road tests are between one to five minutes, including CAR to CAR WEST, CAR WEST to CAR, and campus marked roads.

Introduction
Real-time object detection, identification and tracking is a major challenge in developing computer vision algorithms. Accurate identification and recognition of objects in one or two image frames is critical for Advanced Driver Assistance Systems (ADAS) that are integrated in vehicles [1][2][3][4]. ADAS algorithms aid driver to identify and recognize objects which otherwise would be compromised due to lack of attention, anonymous objects, reflection of light etc. Inefficient algorithm design compromises safety and may result in accidents especially at high speed or under heavy traffic conditions. To overcome the said problem ADAS for autonomous vehicles require unique classification and detection of object under different scenarios [5]. Other applications of object detection are advanced robotics, surveillance systems, defense systems, face recognition and space research etc. [4] discusses the typical steps involved in object classification and identification algorithms for computer vision are -1) Classification of objects in image, 2) Localization of objects in image, 3) Object detection in image and 4) Segmentation of image.
In addition to the above steps, ADAS in self-driving vehicles use advance sensor fusion techniques for detecting and extracting characteristics from moving or stationary objects [6]. There are many object detection algorithms have been proposed that are based on fuzzy, statistics, neural networks etc. Many of these algorithms have complex theory and require deeper understanding, implementation and experimentation before incorporating. Real-time images and videos of various objects and structures are bundled as datasets that provide information of the environment which is used for both training and testing. Using this information distance and direction of the object relative to the test vehicle is measured. Detection and reaction are achieved by combining extraction model with response model in the proposed algorithm. Counting is another important aspect of the algorithm, that involves estimating real-time the number of objects in the image. Once the objects are detected, classified, located and counted from the image, the object is tracked from the extracted features by the proposed algorithm. To perform all the afore mentioned tasks in real-time, it requires high performing hardware. In this paper image segmentation is employed to track without any errors using MATLAB automated tool box. The following section of the paper discusses object detection algorithm, road test results and conclusion.

Camera Calibration
Camera calibration estimates camera intrinsic and extrinsic parameters. Intrinsic parameters are including focal length, principal point and image size. Extrinsic parameters are including pitch, yaw and roll, which can also write as a rotation matrix R and translation vector t. Intrinsic parameters are to determine the relation between camera coordinate and pixel coordinate. Extrinsic parameters are to determine the relation between camera image coordinate and real-world coordinate. As figure 1 shows, in the full road tests nineteen figures have been collected with various angles and distances. Angles have to be less than forty-five degree [7]. Camera extrinsic parameters during all experiments are relative to sensor mounting location at the vehicle, which are mounting height, mounting pitch, mounting roll and mounting yaw [8], as figure 2 (a). Mounting height can be directly measured relative to ground, but other angles need to turn manually according to the bird's eye view image. Figure 2 (b) and (c) indicates extrinsic parameters affections on bird's eye view. Road and lanes in bird's eye view should be parallel after successful turned with correct mounting angles. Although Camera Calibrator App can export camera extrinsic parameters, that is not same extrinsic parameters expected. As figure 3 shows, extrinsic parameters from the app are mapping matrix between image coordinate and camera coordinate. However, the expected extrinsic parameters are translation matrix between vehicle coordinate and camera coordinate as figure 2 shows.

Lane Boundaries Detection and Tracking
The algorithm follows a visual perception example by MATLAB document. After a physical defined region of interests, possible lane width and detection sensitivity, a built-in function Segment Lane Marker Ridge to detect lane points at grayscale bird's eye view image, in figure 2 (d) shows [9]. For each parabolic model includes 3 coefficients as a second-degree polynomial equation. In addition, that built-in function can also provide strengths of lanes, boundary type, x maximum position and x minimum position. Weak strength points, such as vehicle shadows and windscreen reflection, need to remove. Lane types as solids, dash etc. could reflect real road conditions. Lane length calculated by subtracting of x maximum and minimum points. After compared with a threshold length defined by a percentage length of maximum lane length in RoI, short length boundaries, such as zebra crossing, need to remove. In the last step, lane vehicle coordinates points would transfer to video frame image coordinate points by vehicle To Image function. Figure 4 shows the step to detect points in bird's eye view and project them to frame.

Vehicle and Pedestrian Detection
Automated driving toolbox and computer vision toolbox provide various pre-trained detectors, including Aggregate Channel Feature (ACF), Gaussian Mixture Model (GMM), Region-based Convolutional Neural Networks(R-CNN), Fast R-CNN and Faster R-CNN [11][12]. The last three detectors are deep learning methods, which need more processing time compared with others in the test. GMM detector is not accurate compared with ACF detectors in nineteen short traffic videos experiments by Automated Driving Lab. Therefore, ACF is the main detector in this work. Although ACF detector has better work compared with other pre-trained detectors, it still has problems [13][14]. It is not always consistent tracking and some frame may miss label. Ground Truth Labeler app provides temporal interpolator algorithm, which can estimate label in the intermediate frame by interpolation. Another main problem is the rectangle label is not consistent with vehicle size, and the size may vary in different frames. Ground Truth Labeler app would also solve the problem, but it may need volume manual work.

Testing and Verification
The tracking algorithm proposed for vehicles and pedestrians is based on Kalman filter which in turn is built on constant velocity model [15]. The pedestrian detection system is a critical part of the system and mandates accuracy and reliability with real-time performance. The pedestrian detection system was tested and verifies taking samples of real traffic scenarios. Although the system provided correct results, it requires tuning and adaptive learning which can consume resources. Therefore, a simulation evaluation tool was used to develop the algorithm to test for real world situations and evaluate the response. Through this algorithm the pedestrian accident data was first analyzed against the other conventional algorithm to find and record its statistical distribution characteristics.

Road Test Results
The algorithm applied to these four tests does not show a great result. It has two main limitations. The first one is when lane width changes or separation, lane points would beyond pre-defined RoI. In complex road conditions, constant RoI will decrease detection accuracies and missing detection. Therefore, the lane detection algorithm has better to detect in constant road condition without lane width changes or division. Another limitation for lane detection algorithm is at the intersection. Since it has a gap but does not reach the threshold, which is not enough strength or length, the detection lanes are not accurate neither able to remove. The detection points could include other direction lanes and all other shadows by vehicles. Theoretically, ground truth labeler app can remove inaccurate points in the frame. However, due to a volume of sampling points, physics work increases by selecting inaccurate points.
Besides hand-operated work by the app, single detection missing frame could update with last tracking points since each frame only takes 0.03 seconds. For a certain period, detection-missing frames, manual work need by ground truth labeler app. To reduce ground truth app processing time, only missing frames import to the app to adding detections. The camera frame rate is 30 frames per second so the frame number calculated as follows

Vehicle and pedestrian detection and tracking results and conclusion
The algorithm applied to these four tests does not show a great result. It has two main limitations. The one of important limitation is the windy and snow shower weather in testing day, as figure 5 shows. Image qualities are not satisfied as the previous test. The detector is only able to detect objects which close to the testing vehicle. Increasing threshold distance and sensitivity is unable to increase detection abilities. In addition, some frame was missing detection due to low image qualities. Therefore, mass manual work needed by ground truth labeler app.

Figure 5 Tracking with test video -Detection in heavy traffic
Another limitation is by ACF detector. ACF detector of vehicles and pedestrians may have inaccurate tracking since it defined as object size. Moreover, the detection rectangle size is not constant, and with small shakings. However, that will not affect object location calculation. ACF may miss detection in heavy traffic condition. Ground truth labeler app would help to manually add detections. Example as figure 5 shows. Overall, considering weather condition, the accuracy of vehicle and pedestrians' detection at a short range is great, as figure 6 shows.  Figure 6. Tracking with test video -Detection in heavy traffic

Tracklet labeler
Tracklet labeler is created based on a timetable is shown in Table 1. For vehicles, pedestrians and stop signs which using rectangle label tracking, have an M x 4 matrix at a specific time. M presents the number of objects at the specific time and four presents image points of tracking rectangle in x, y, width and height. For Lane Marker, it would be 3 element coefficients to present a parabolic lane. Ground truth labeler app would export a group of image points for lanes. Since all these points are based on image coordinate, sensor calibration results are needed to transfer into vehicle coordinate when objects position needed and figure 8 shows output of object detection and line detection.

Conclusion
The proposed algorithm developed in MATLAB detects and tracks objects accurately on test videos for traffic lanes, other vehicles and pedestrians. The algorithm was verified for 19 short videos from Automated Driving Lab, including large curve, ramp and busy traffic scenarios. The tracking of vehicles, pedestrians and lane shows great result compared to ACF detector. Furthermore, the algorithm was applied to Vision Sensor Dataset and Full Road Test with longer than one minute. However, under dim lighting and extreme busy road conditions algorithm showed some limitations in object detection and tracking. This necessitates applying a sensor fusion model which integrates multiple sensors such as vision, radar, and IMU. This model would make the forward collision warning system of the vehicle more effective.