Answers.Body: <p><strong>The Problem</strong></p>
  <p>We can see from the question that existing information on detection and
  classification in the small automotive vehicle domain has been
  located (in the form of two independent sets of vectors usable for
  machine training), and there is no already existing mapping or
  other correspondence between the elements of one set and the
  elements of the other.  They were obtained independently, remain
  independent, and are linked only by the conventions of the domain
  (today's aesthetically acceptable and thermodynamically workable
  forms of small vehicles).</p>
  <p>The goal stated in the question is to create a computer vision system
  that both detects cars and classifies them leveraging the
  information contained in the two distinct sets.</p>
  <p>In the vision systems of mammals, there are also two distinct equivalences
  of sets; one arising from a genetic algorithm, the DNA that is
  expressed during the formation of the neural net geometry and
  bio-electro-chemistry of the visual system in early development;
  and the cognitive and coordinative pathways in the cerebrum and
  cerebellum.</p>
  <p>If a robot, wheelchair, or other vehicle is to avoid traffic, we must
  produce a system that in some way matches or exceeds the
  collision avoidance performance of mammals.
  In crime prevention, toll collection, sales lot inventory,
  county traffic analysis, and other like applications,
  performance will again be expected to match or exceed the
  performance of biological systems.
  If a person can record the make, model, year, color, and
  license plate strings, so should the machine we employ in
  these capacities.</p>
  <p>Consequently, this question is pertinent beyond academic curiosity, as
  it is applicable in current research and development of products.</p>
  <p>That this question author notices the lack of a unified data
  set that can be used to train it to detect and characterize
  in a single network objects of interest is apropos and
  key to the challenge of finding a solution.</p>
  <p><em>Approach</em>
  The simplest approach would be to compose the system of two functions.</p>
  <ol>
  <li><span class="math-container">$\quad\mathcal{D}: \mathbb{I}^4 \to {(\mathbb{I}^2, \mathbb{I}^2)}_1, \; {(\mathbb{I}^2, \mathbb{I}^2)}_2, \; ... $</span></li>
  <li><span class="math-container">$\quad\mathcal{C}: {(\mathbb{I}^2, \mathbb{I}^2)}_i \to {(\mathbb{I})}_i$</span></li>
  </ol>
  <p>The four dimensions of input for <span class="math-container">$\mathcal{D}$</span>, the detector, are horizontal position, vertical position,
  rgb index, and brightness to decribe the pixelized image; and the output are bounding boxes as two &quot;corner&quot; coordinates corresponding to each identified vehicle, the second coordinate being either relative to the first or to a specific corner of the entire frame.
  The categorizer, <span class="math-container">$\mathcal{C}$</span>, receives as input bounding boxes and produces as output the index
  or code that maps to the categories corresponding to the labels of the training set available for
  categorization.
  The system can then be described as follows.</p>
  <p><span class="math-container">$\quad\quad\mathcal{S}: \mathcal{C} \circ \mathcal{D}$</span></p>
  <p>If the system is not color, subtract one from the above dimensionality of the input.  If the system processes video, add one to the dimensionality of the input and consider using LSTM or GRU cell types.</p>
  <p>The above substitution represented by &quot;<span class="math-container">$\circ$</span>&quot; appears to be what is meant by, &quot;I use the images from the detection dataset as input
  and get classification predictions on top of detected bounding boxes.&quot;</p>
  <p>The interrogative, &quot;How do I verify whether the classification model trained on the
  classification dataset is working on images from detection dataset?
  (In terms of classification accuracy),&quot;
  appears to refer to the fact that labels do not exist for the second set that correspond to
  input elements of the first set, so an accuracy metric cannot be directly obtained.
  Since there is no obvious automatic way of generating labels for the vehicles in the pre-detected images
  containing potentially multiple vehicles,
  there is no way to check actual results against expected results.
  Composing multiple vehicle images from the categorization set to use as test input to
  the entire system <span class="math-container">$\mathcal{S}$</span> will only be useful in evaluating an aspect
  of the performance of <span class="math-container">$\mathcal{D}$</span>, not <span class="math-container">$\mathcal{C}$</span>.</p>
  <p><strong>Solution</strong></p>
  <p>The only way to evaluate the accuracy and reliability of <span class="math-container">$\mathcal{C}$</span> is with portions
  of the set used to train it that were excluded from the training and trust
  that the vehicles depicted in those images were sufficiently representative
  of the concept &quot;car&quot; to provide consistency of accuracy and reliability across
  the range of those detected by <span class="math-container">$\mathcal{D}$</span> in the application of <span class="math-container">$\mathcal{S}$</span>.
  This means that the leveraging of the information, even if optimized to the degree
  possible by any arbitrary algorithm or parallelism in the set of all possible
  algorithms or parallelisms, is limited by the categorization training set.
  The number of set elements and the comprehensiveness and distribution of categories
  within that set must be sufficient to achieve an approximate equality between
  these two accuracy metrics.</p>
  <ol>
  <li>Categorizing a test sample from the labeled set for <span class="math-container">$\mathcal{C}$</span> excluded from the training</li>
  <li>Categorizing the vehicles isolated by <span class="math-container">$\mathcal{D}$</span> from its training input</li>
  </ol>
  <p><strong>With Additional Resources</strong></p>
  <p>Of course this discussion is in a particular environment, that of the system
  defined as the two artificial networks, one involving convolution based
  recognition and the other involving feature extraction, and the two training
  sets.
  What is needed is a wider environment where known vehicles are in view so that
  performance data of <span class="math-container">$\mathcal{S}$</span> is evaluated and a tap on the transfer
  of information between <span class="math-container">$\mathcal{D}$</span> and <span class="math-container">$\mathcal{C}$</span> can be used
  to differentiate between mistakes made on either side of the tap point.</p>
  <p><strong>Unsupervised Approach</strong></p>
  <p>Another course of action could be to not use the training set for categorization
  on the training of <span class="math-container">$\mathcal{C}$</span> at all, but rather use feature extraction and
  auto-correlation in an &quot;unsupervised&quot; approach, and then evaluate the results of
  on the basis of the final convergence metrics at the point when stability in
  categorization is detected.  In this case, the images in the bounding boxes
  output by <span class="math-container">$\mathcal{D}$</span> would be used as training data.</p>
  <p>The auto-trained network realizing <span class="math-container">$\mathcal{C}$</span> can then be further evaluated
  using the entire categorization training set.</p>
  <p><strong>Further Research</strong></p>
  <p>Hybrids of these two approaches are possible. Also, the independent training only in the rarest of cases leads to optimal performance. Understanding feedback as originally treated with rigor by MacColl in chapter 8 of his <em>Fundamental Theory of Servomechanisms</em>, later applied to the problem of linearity and stability of analog circuitry, and then to training, first in the case of GANs, may lead to effective methods to bi-train the two networks.</p>
  <p>That evolved biological networks are trained <em>in situ</em> is an indicator that the most optimal performance can be gained by finding training architectures and information flow strategies that create optimality in both components simultaneously.  No biological niche has ever been filled by a neural component that is first optimized and then inserted or copied in some way to a larger brain system.  That is no proof that such component-ware can be optimal, but there is also no proof that the DNA driven systems that have emerged are not nearly optimized for the majority of terrestrial conditions.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-15T13:34:58.230Z
Answers.Text: This is the perfect explanation that I could have got! Can you elaborate on the "unsupervised" approach? Any papers/articles that are using the mentioned technique?
Answers.UserDisplayName: Aniket Velhankar
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-14T16:39:15.703Z
Answers.LastActivityDate: 2020-11-16T22:19:54.667Z
Answers.LastEditDate: 2020-11-16T22:19:54.667Z
Answers.UserDisplayName: Douglas Daseeco
Answers.Body: <p>To verify the accuracy of the classification stage, you will need labeled images with a single car.</p>
  <p>To train and verify accuracy of the detection stage and full system, you can:</p>
  <ol>
  <li>in the datasets with images with multiple cars, manually, mark the image rectangles that contains one car.</li>
  <li>from previous, split the image in one or more ones, each one containing a single car.</li>
  <li>pass each one of the previous image with a single car to the classification stage (that means assume classification has 100% accuracy). Record its outputs (labeled cars).</li>
  <li>now, from output of steps 1) and 3), you can produce labeled images with multiple cars. Use it to train the detector and verify full system accuracy.</li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-15T13:17:13.530Z
Answers.Text: Yes, this is an interesting thought. But if I were to publish these results in a journal/conference using the said assumption, is it valid ?
Answers.UserDisplayName: Aniket Velhankar
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-15T18:51:53.923Z
Answers.Text: In any paper you must explain your methodology.
Answers.UserDisplayName: pasaba por aqui
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-14T12:03:13.750Z
Answers.LastActivityDate: 2020-11-14T12:03:13.750Z
Answers.UserDisplayName: pasaba por aqui
Body: <p>I am working on a problem that involves two tasks - detection and classification. There is no single dataset for both tasks. I am training two models, separate on detection dataset and another on classification dataset. I use the images from the detection dataset as input and get classification predictions on top of detected bounding boxes.</p>
  <p>Dataset description :</p>
  <ol>
  <li>Classification - Image of the single object (E.g. Car) in the center with a classification label.</li>
  <li>Detection - Image with multiple objects (E.g. 4 Cars) with bounding box annotations.</li>
  </ol>
  <p>Task - Detect objects(e.g. cars) from detection datasets and classify them into various categories.</p>
  <p>How do I verify whether the classification model trained on the classification dataset is working on images from detection dataset? (In terms of classification accuracy)</p>
  <p>I cannot manually label the images from the detection dataset for individual class labels. (Need expert domain knowledge)</p>
  <p>How do I verify my classification model?</p>
  <p>Is there any technique to do this ? Like domain transfer or any weakly-supervised method ?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-10T12:38:41.947Z
Comments.Text: Can you please add instances from datasets to understand your problem better?
Comments.UserDisplayName: dasmehdix
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-13T01:09:54.413Z
Comments.Text: This very confusing to read. Why do you want to verify that the classifier is working on data from the detection dataset. Aren't you the one who decides what data is being used?
Comments.UserDisplayName: Recessive
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-13T11:50:39.003Z
Comments.Text: Did you try the [YOLO](https://arxiv.org/abs/1506.02640) model?
Comments.UserDisplayName: Aray Karjauv
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-14T05:23:19.793Z
Comments.Text: @Recessive Yes, it is confusing. Here's an example - 1. I have a detection dataset having annotations for cars from top view, I can detect cars but I want to classify them as well (Merc, BMW, etc.) 2. So, I train a classification model on a different dataset which has such labels. That dataset has single car image and labels.
Comments.UserDisplayName: Aniket Velhankar
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-14T05:24:21.623Z
Comments.Text: 3. When I apply this classifier on the detection dataset, I take individual detected cars and classify them but since the detection and classification dataset is different, I want to find how accurate my classification model is classifying cars in the detection dataset and not on the dataset I trained it on. For this, I need manual annotation in the detection dataset, which I can't do. So, I am in search of a solution.
Comments.UserDisplayName: Aniket Velhankar
Tags: <deep-learning><convolutional-neural-networks><object-detection><image-processing>
Title: How to verify classification model trained on classification dataset on a detection dataset for classification purpose?
UserDisplayName: Aniket Velhankar
ThreadText: # How to verify classification model trained on classification dataset on a detection dataset for classification purpose?
  
  (Asked by: username_0 on 2020-11-06)
  
  I am working on a problem that involves two tasks - detection and classification. There is no single dataset for both tasks. I am training two models, separate on detection dataset and another on classification dataset. I use the images from the detection dataset as input and get classification predictions on top of detected bounding boxes.
  
  Dataset description :
  
  1. Classification - Image of the single object (E.g. Car) in the center with a classification label.
  2. Detection - Image with multiple objects (E.g. 4 Cars) with bounding box annotations.
  
  Task - Detect objects(e.g. cars) from detection datasets and classify them into various categories.
  
  How do I verify whether the classification model trained on the classification dataset is working on images from detection dataset? (In terms of classification accuracy)
  
  I cannot manually label the images from the detection dataset for individual class labels. (Need expert domain knowledge)
  
  How do I verify my classification model?
  
  Is there any technique to do this ? Like domain transfer or any weakly-supervised method ?
  
  **Tags:** deep-learning, convolutional-neural-networks, object-detection, image-processing
  > **Comments:**
  > **username_1**: Can you please add instances from datasets to understand your problem better?  
  > **username_2**: This very confusing to read. Why do you want to verify that the classifier is working on data from the detection dataset. Aren't you the one who decides what data is being used?  
  > **username_3**: Did you try the [YOLO](https://arxiv.org/abs/1506.02640) model?  
  > **username_0**: @Recessive Yes, it is confusing. Here's an example - 1. I have a detection dataset having annotations for cars from top view, I can detect cars but I want to classify them as well (Merc, BMW, etc.) 2. So, I train a classification model on a different dataset which has such labels. That dataset has single car image and labels.  
  > **username_0**: 3. When I apply this classifier on the detection dataset, I take individual detected cars and classify them but since the detection and classification dataset is different, I want to find how accurate my classification model is classifying cars in the detection dataset and not on the dataset I trained it on. For this, I need manual annotation in the detection dataset, which I can't do. So, I am in search of a solution.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_4 on 2020-11-14. Score: 1 | Accepted answer
  
  **The Problem**
  
  We can see from the question that existing information on detection and
  classification in the small automotive vehicle domain has been
  located (in the form of two independent sets of vectors usable for
  machine training), and there is no already existing mapping or
  other correspondence between the elements of one set and the
  elements of the other. They were obtained independently, remain
  independent, and are linked only by the conventions of the domain
  (today's aesthetically acceptable and thermodynamically workable
  forms of small vehicles).
  
  The goal stated in the question is to create a computer vision system
  that both detects cars and classifies them leveraging the
  information contained in the two distinct sets.
  
  In the vision systems of mammals, there are also two distinct equivalences
  of sets; one arising from a genetic algorithm, the DNA that is
  expressed during the formation of the neural net geometry and
  bio-electro-chemistry of the visual system in early development;
  and the cognitive and coordinative pathways in the cerebrum and
  cerebellum.
  
  If a robot, wheelchair, or other vehicle is to avoid traffic, we must
  produce a system that in some way matches or exceeds the
  collision avoidance performance of mammals.
  In crime prevention, toll collection, sales lot inventory,
  county traffic analysis, and other like applications,
  performance will again be expected to match or exceed the
  performance of biological systems.
  If a person can record the make, model, year, color, and
  license plate strings, so should the machine we employ in
  these capacities.
  
  Consequently, this question is pertinent beyond academic curiosity, as
  it is applicable in current research and development of products.
  
  That this question author notices the lack of a unified data
  set that can be used to train it to detect and characterize
  in a single network objects of interest is apropos and
  key to the challenge of finding a solution.
  
  *Approach*
  The simplest approach would be to compose the system of two functions.
  
  1. $\quad\mathcal{D}: \mathbb{I}^4 \to {(\mathbb{I}^2, \mathbb{I}^2)}\_1, \; {(\mathbb{I}^2, \mathbb{I}^2)}\_2, \; ... $
  2. $\quad\mathcal{C}: {(\mathbb{I}^2, \mathbb{I}^2)}\_i \to {(\mathbb{I})}\_i$
  
  The four dimensions of input for $\mathcal{D}$, the detector, are horizontal position, vertical position,
  rgb index, and brightness to decribe the pixelized image; and the output are bounding boxes as two "corner" coordinates corresponding to each identified vehicle, the second coordinate being either relative to the first or to a specific corner of the entire frame.
  The categorizer, $\mathcal{C}$, receives as input bounding boxes and produces as output the index
  or code that maps to the categories corresponding to the labels of the training set available for
  categorization.
  The system can then be described as follows.
  
  $\quad\quad\mathcal{S}: \mathcal{C} \circ \mathcal{D}$
  
  If the system is not color, subtract one from the above dimensionality of the input. If the system processes video, add one to the dimensionality of the input and consider using LSTM or GRU cell types.
  
  The above substitution represented by "$\circ$" appears to be what is meant by, "I use the images from the detection dataset as input
  and get classification predictions on top of detected bounding boxes."
  
  The interrogative, "How do I verify whether the classification model trained on the
  classification dataset is working on images from detection dataset?
  (In terms of classification accuracy),"
  appears to refer to the fact that labels do not exist for the second set that correspond to
  input elements of the first set, so an accuracy metric cannot be directly obtained.
  Since there is no obvious automatic way of generating labels for the vehicles in the pre-detected images
  containing potentially multiple vehicles,
  there is no way to check actual results against expected results.
  Composing multiple vehicle images from the categorization set to use as test input to
  the entire system $\mathcal{S}$ will only be useful in evaluating an aspect
  of the performance of $\mathcal{D}$, not $\mathcal{C}$.
  
  **Solution**
  
  The only way to evaluate the accuracy and reliability of $\mathcal{C}$ is with portions
  of the set used to train it that were excluded from the training and trust
  that the vehicles depicted in those images were sufficiently representative
  of the concept "car" to provide consistency of accuracy and reliability across
  the range of those detected by $\mathcal{D}$ in the application of $\mathcal{S}$.
  This means that the leveraging of the information, even if optimized to the degree
  possible by any arbitrary algorithm or parallelism in the set of all possible
  algorithms or parallelisms, is limited by the categorization training set.
  The number of set elements and the comprehensiveness and distribution of categories
  within that set must be sufficient to achieve an approximate equality between
  these two accuracy metrics.
  
  1. Categorizing a test sample from the labeled set for $\mathcal{C}$ excluded from the training
  2. Categorizing the vehicles isolated by $\mathcal{D}$ from its training input
  
  **With Additional Resources**
  
  Of course this discussion is in a particular environment, that of the system
  defined as the two artificial networks, one involving convolution based
  recognition and the other involving feature extraction, and the two training
  sets.
  What is needed is a wider environment where known vehicles are in view so that
  performance data of $\mathcal{S}$ is evaluated and a tap on the transfer
  of information between $\mathcal{D}$ and $\mathcal{C}$ can be used
  to differentiate between mistakes made on either side of the tap point.
  
  **Unsupervised Approach**
  
  Another course of action could be to not use the training set for categorization
  on the training of $\mathcal{C}$ at all, but rather use feature extraction and
  auto-correlation in an "unsupervised" approach, and then evaluate the results of
  on the basis of the final convergence metrics at the point when stability in
  categorization is detected. In this case, the images in the bounding boxes
  output by $\mathcal{D}$ would be used as training data.
  
  The auto-trained network realizing $\mathcal{C}$ can then be further evaluated
  using the entire categorization training set.
  
  **Further Research**
  
  Hybrids of these two approaches are possible. Also, the independent training only in the rarest of cases leads to optimal performance. Understanding feedback as originally treated with rigor by MacColl in chapter 8 of his *Fundamental Theory of Servomechanisms*, later applied to the problem of linearity and stability of analog circuitry, and then to training, first in the case of GANs, may lead to effective methods to bi-train the two networks.
  
  That evolved biological networks are trained *in situ* is an indicator that the most optimal performance can be gained by finding training architectures and information flow strategies that create optimality in both components simultaneously. No biological niche has ever been filled by a neural component that is first optimized and then inserted or copied in some way to a larger brain system. That is no proof that such component-ware can be optimal, but there is also no proof that the DNA driven systems that have emerged are not nearly optimized for the majority of terrestrial conditions.
  
  > **Comments:**
  > **username_0**: This is the perfect explanation that I could have got! Can you elaborate on the "unsupervised" approach? Any papers/articles that are using the mentioned technique?  
  
  ---
  
  ### Answer by username_5 on 2020-11-14. Score: 2
  
  To verify the accuracy of the classification stage, you will need labeled images with a single car.
  
  To train and verify accuracy of the detection stage and full system, you can:
  
  1. in the datasets with images with multiple cars, manually, mark the image rectangles that contains one car.
  2. from previous, split the image in one or more ones, each one containing a single car.
  3. pass each one of the previous image with a single car to the classification stage (that means assume classification has 100% accuracy). Record its outputs (labeled cars).
  4. now, from output of steps 1) and 3), you can produce labeled images with multiple cars. Use it to train the detector and verify full system accuracy.
  
  > **Comments:**
  > **username_0**: Yes, this is an interesting thought. But if I were to publish these results in a journal/conference using the said assumption, is it valid ?  
  > **username_5**: In any paper you must explain your methodology.  
  
  ---
  
  