  Package Summary

Documented

Single image rectification and color processing.

  * Author: Patrick Mihelich, Kurt Konolige, Jeremy Leibs
  * License: BSD
  * Source: git [ https://github.com/ros-perception/image_pipeline.git ](https://github.com/ros-perception/image_pipeline) (branch: fuerte-devel)

_**[ image_pipeline ](/image_pipeline?distro=groovy) ** : [ camera_calibration
](/camera_calibration?distro=groovy) | [ depth_image_proc
](/depth_image_proc?distro=groovy) | image_proc | [ image_rotate
](/image_rotate?distro=groovy) | [ image_view ](/image_view?distro=groovy) | [
stereo_image_proc ](/stereo_image_proc?distro=groovy) _

**Package Links**

  * **[ Code API ](http://docs.ros.org/groovy/api/image_proc/html) **
  * [ Tutorials ](/image_proc/Tutorials)
  * [ FAQ ](http://answers.ros.org/questions/scope:all/sort:activity-desc/tags:image_proc/page:1/)
  * [ Change List ](/image_pipeline/ChangeList)
  * [ Reviews ](/image_proc/Reviews)

** Dependencies  (12) **

  * [ boost ](/boost?distro=groovy)
  * [ camera_calibration_p... ](/camera_calibration_parsers?distro=groovy)
  * [ catkin ](/catkin?distro=groovy)
  * [ cv_bridge ](/cv_bridge?distro=groovy)
  * [ dynamic_reconfigure ](/dynamic_reconfigure?distro=groovy)
  * [ image_geometry ](/image_geometry?distro=groovy)
  * [ image_transport ](/image_transport?distro=groovy)
  * [ nodelet ](/nodelet?distro=groovy)
  * [ opencv2 ](/opencv2?distro=groovy)
  * [ roscpp ](/roscpp?distro=groovy)
  * [ rostest ](/rostest?distro=groovy)
  * [ sensor_msgs ](/sensor_msgs?distro=groovy)

** Used by  (13) **


Requirements
Since this tutorial will show how to use a segmentation camera sensor in Gazebo, you'll need to have Gazebo installed. We recommend installing all Gazebo libraries, using version Fortress or newer (the segmentation camera is not available in Gazebo versions prior to Fortress). If you need to install Gazebo, pick the version you'd like to use and then follow the installation instructions.


    source /usr/share/drcsim/setup.sh
    rosrun rviz rviz


First we want to test our threshold values. Sometimes this is easier without
the mask. Make the following changes in [ center_line_detection node.py
](https://github.com/wilselby/diy_driverless_car_ROS/blob/master/rover_cv/lane_detection/src/lane_detection/center_line_detection.py)
to view the image segmentation on the original image.


Skip to content


  Launch files

New in ROS Fuerte


The data value is a desired angular velocity, in radians per second. You
should see the laser scan in rviz rotating around. To build a poor-man's 3-D
model of the environment, click in the "Decay Time" field of the "LaserScan"
display and increase the time to something non-zero (it's a value in seconds).
You should see scans accumulate in rviz.


rosrun rviz rviz -d /your/config/file.vcg


  Package Summary

Released

Continuous Integration:  8 / 8

  * Build history (last 5 of 9 builds):
  * [


roslaunch drcsim_gazebo atlas.launch


    # it polynomials
    if self.global_fit is not None:
        ploty, fitx, fit = ld.fast_fit_polynomials(self.binaryImage,  self.global_fit)
    else:
        ploty, fitx, fit = ld.fit_polynomials(self.warpedImage, self.binaryImage)

    self.global_fit = fit

    # draw lane
    self.processedImage = ld.render_lane(self.maskedRGBImage, self.corners, ploty, fitx)


The architecture of the grasping agent is partitioned into
two principal components. First, the YOLOv8 model serves
as the object detector. For efficient adaptation from preexisting models, input resolutions are standardized to 512
× 512 pixels. The final segment involves a robot control
unit that orchestrates the actual grasping maneuvers and
robot movements, grounded on the predicted bounding box
coordinates and the concurrent visual inputs. The control
unit’s objective is to minimize grasping inaccuracies through
an error function,
Error = α · (y
pred
l − y
gt
l
) + (1 − α) · (y
pred
g − y
gt
g
), (3)
where y
pred
∗ and y
gt
∗ represent the centers of the predicted
and actual (ground truth) bounding box, respectively, across
both camera views. The system aims for the predicted
target’s bounding box to coincide with an actual graspable
location, effectuating a successful grasping action as the error
approaches zero.
IV. EVALUATIONS
A. Datasets and Simulation Environment
1) The training datasets of the image generation model:
To make the ALDM model obtain the knowledge of realworld features and have the ability to generate realistic style
images, we have used two public datasets to train the model,
which is:
• The ADE20K dataset [30], jointly developed by the
Massachusetts Institute of Technology and the University of Toronto, comprises over 22,000 images with
detailed pixel-level annotations across a wide range
of scenes and objects. It is particularly valuable for


####  Real World Implementation


[ Edit
](https://github.com/osrf/gazebo_tutorials/blob/master/drcsim_visualization/tutorial.md)
Version: 2.0+


Given the line of best fit for the detected lane, the point where the line
crosses the bottom of the image is used as the setpoint for the control
algorithm. The goal of the control algorithm is to compute the desired speeds
to keep this setpoint at the center of the image. A simple moving average is
used to smooth out the calculation of the setpoint and reduce the impact of
outliers.


To save the configuration as the default, click "File > Save Config". The next
time you run rviz, it will load this configuration.


The [ center_line_detection node
](https://github.com/wilselby/diy_driverless_car_ROS/blob/master/rover_cv/lane_detection/src/lane_detection/center_line_detection.py)
file needs to be updated with several environment and camera specific
variables including the threshold values. Find an image from the bag file
where the vehicle is centered between parallel lines or run  [
rover_dbw_cam.launch
](https://github.com/wilselby/diy_driverless_car_ROS/blob/master/rover_teleop/launch/rover_dbw_cam.launch
"rover_dbw_cam.launch") again and save an image. Record points along the
parallel lines that can be used as source points for the perspective
transformation. Save these points as self.corners.


[ ![ROS RasPiCam Lane Detection Warped Image]( https://s3-us-
west-2.amazonaws.com/selbystorage/wp-
content/uploads/2017/July/ros_driverless_car_kitchen_warped_image.png) ](
https://s3-us-west-2.amazonaws.com/selbystorage/wp-
content/uploads/2017/July/ros_driverless_car_kitchen_warped_image.png)


[ ![Google+](/assets/google_gray-
cc34826f560f99c7a9b7db88cd5f2db4d28fd4b3afcfa4b897435c51e3b25518.png)
](//plus.google.com/u/0/115981436296571800301?prsrc=3) [
![YouTube](/assets/youtube_gray-978838444154eb6aa5e4aa38489f66d10e40361dd06a3814411c211cba013752.png)
](https://www.youtube.com/channel/UCJyqf9XJpDoM9XnpAwW6WxA) [
![YouTube](/assets/twitter_gray-8e9dfe74c6cab12171992a40add9a65e5bba92ac93063a4ac0eeba58c77e774a.png)
](https://twitter.com/GazeboSim)


This launch file results in the rosgraph shown below.


[ rviz ](http://www.ros.org/wiki/rviz) is a powerful robot visualization tool.
It provides a convenient GUI to visualize sensor data, robot models,
environment maps, which is useful for developing and debugging your robot
controllers.


<model name="segmentation_model">
  <pose>4 0 1.0 0 0.0 3.14</pose>
  <link name="link">
    <pose>0.05 0.05 0.05 0 0 0</pose>
    <inertial>
      <mass>0.1</mass>
      <inertia>
        <ixx>0.000166667</ixx>
        <iyy>0.000166667</iyy>
        <izz>0.000166667</izz>
      </inertia>
    </inertial>
    <collision name="collision">
      <geometry>
        <box>
          <size>0.1 0.1 0.1</size>
        </box>
      </geometry>
    </collision>
    <visual name="visual">
      <geometry>
        <box>
          <size>0.1 0.1 0.1</size>
        </box>
      </geometry>
    </visual>

    <sensor name="segmentation_camera" type="segmentation">
      <topic>segmentation</topic>
      <camera>
        <segmentation_type>instance</segmentation_type>
        <horizontal_fov>1.047</horizontal_fov>
        <image>
          <width>800</width>
          <height>600</height>
        </image>
        <clip>
          <near>0.1</near>
          <far>10</far>
        </clip>


ALDM-Grasping: Diffusion-aided Zero-Shot Sim-to-Real Transfer for
Robot Grasping
Yiwei Li*1
, Zihao Wu*1
, Huaqin Zhao1
, Tianze Yang1
, Zhengliang Liu1
, Peng Shu1
,
Jin Sun†
1
, Ramviyas Parasuraman†
1
and Tianming Liu†
1
Abstract— To tackle the ”reality gap” encountered in Sim-toReal transfer, this study proposes a diffusion-based framework
that minimizes inconsistencies in grasping actions between the
simulation settings and realistic environments. The process
begins by training an adversarial supervision layout-to-image
diffusion model(ALDM). Then, leverage the ALDM approach
to enhance the simulation environment, rendering it with
photorealistic fidelity, thereby optimizing robotic grasp task
training. Experimental results indicate this framework outperforms existing models in both success rates and adaptability
to new environments through improvements in the accuracy
and reliability of visual grasping actions under a variety of
conditions. Specifically, it achieves a 75% success rate in
grasping tasks under plain backgrounds and maintains a 65%
success rate in more complex scenarios. This performance
demonstrates this framework excels at generating controlled
image content based on text descriptions, identifying object
grasp points, and demonstrating zero-shot learning in complex,
unseen scenarios.
I. INTRODUCTION
To perform optimally, deep learning models require extensively annotated datasets [1]. However, in robotics tasks
such as visual grasping, acquiring datasets to train comprehensive deep learning models in real-world settings can be
prohibitively expensive and time-consuming, and sometimes,
unfeasible [2], [3], [4]. This problem is also known as the
”reality gap”. To address this challenge, Sim-to-Real strategies have been developed, allowing the training of robotic
performance models, such as those for visual grasping,
to be trained in simulated environments and subsequently


[22] K. Bousmalis, N. Silberman, D. Dohan, D. Erhan, and D. Krishnan,
“Unsupervised pixel-level domain adaptation with generative adversarial networks,” in Proceedings of the IEEE conference on computer
vision and pattern recognition, 2017, pp. 3722–3731.
[23] X. B. Peng, M. Andrychowicz, W. Zaremba, and P. Abbeel, “Sim-toreal transfer of robotic control with dynamics randomization,” in 2018
IEEE international conference on robotics and automation (ICRA).
IEEE, 2018, pp. 3803–3810.
[24] V. M. Patel, R. Gopalan, R. Li, and R. Chellappa, “Visual domain
adaptation: A survey of recent advances,” IEEE signal processing
magazine, vol. 32, no. 3, pp. 53–69, 2015.
[25] K. Rao, C. Harris, A. Irpan, S. Levine, J. Ibarz, and M. Khansari, “Rlcyclegan: Reinforcement learning aware simulation-to-real. in 2020
ieee,” in CVF Conference on Computer Vision and Pattern Recognition
(CVPR), 2020, pp. 11 154–11 163.
[26] C. J. C. H. Watkins, “Learning from delayed rewards,” 1989.
[27] Z. Wang, H. Zheng, P. He, W. Chen, and M. Zhou, “Diffusion-gan:
Training gans with diffusion,” arXiv preprint arXiv:2206.02262, 2022.
[28] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional
networks for biomedical image segmentation,” in Medical image
computing and computer-assisted intervention–MICCAI 2015: 18th
international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18. Springer, 2015, pp. 234–241.
[29] D. Reis, J. Kupec, J. Hong, and A. Daoudi, “Real-time flying object
detection with yolov8,” arXiv preprint arXiv:2305.09972, 2023.
[30] B. Zhou, H. Zhao, X. Puig, S. Fidler, A. Barriuso, and A. Torralba,
“Scene parsing through ade20k dataset,” in Proceedings of the IEEE
conference on computer vision and pattern recognition, 2017, pp. 633–
641.
[31] M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele, “The cityscapes dataset
for semantic urban scene understanding,” in Proceedings of the IEEE
conference on computer vision and pattern recognition, 2016, pp.
3213–3223.
[32] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer,
“High-resolution image synthesis with latent diffusion models,” in
Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition (CVPR), June 2022, pp. 10 684–10 695.


[ ![HSV Image Segmentation]( https://s3-us-
west-2.amazonaws.com/selbystorage/wp-
content/uploads/2017/July/ros_driverless_car_image_segmentation.png) ](
https://s3-us-west-2.amazonaws.com/selbystorage/wp-
content/uploads/2017/July/ros_driverless_car_image_segmentation.png)


* The scroll wheel zooms in and out


You can also run rviz with a different configuration by specifying it at the
command line. '''Note:''' Insert the path to the file.


[ ![ROS Gaazebo Lane Detection Initial Image](https://i0.wp.com/s3-us-
west-2.amazonaws.com/selbystorage/wp-
content/uploads/2017/July/ros_driverless_car_gazebo_original_image.png?resize=382%2C215&ssl=1)
](https://i0.wp.com/s3-us-west-2.amazonaws.com/selbystorage/wp-
content/uploads/2017/July/ros_driverless_car_gazebo_original_image.png?ssl=1)


Once the source and destination points are determined, the transformation can
be applied to the images in the pipeline. The image below is an example of the
warped output image.


Once the system is up and running, connect to the RaspberryPi from a remote
computer using the tightvncserver on the RaspberryPi as described in the [
Raspberry Pi Integration ](https://www.wilselby.com/research/autonomous-
vehicles/diy-autonomous-vehicle-project/raspberry-pi-integration/) section.
Start rwt_image_viewer to monitor the output of the image processing
algorithm. Once you are satisfied that everything is functioning properly,
turn on the power source on the Elegoo. The motors should begin receiving
commands and moving the vehicle. If the vehicle begins to move off track, you
can manually override the autonomy commands with the Xbox controller.


pinpointing object grasp positions, and demonstrating zeroshot learning in complex, unseen scenarios. Future work will
extend ALDM to diverse gripper configurations and explore
its applicability in 3D unstructured settings, such as robotic
fruit harvesting and autonomous driving. Additionally, we
aim to test ALDM’s capabilities in varied manipulation tasks,
including rotation and placement, to assess its versatility.
The exploration of more vision backbone models will further
refine ALDM’s effectiveness in Sim-to-Real applications.
REFERENCES
[1] A. I. Karoly, P. Galambos, J. Kuti, and I. J. Rudas, “Deep learning ´
in robotics: Survey on model structures and training strategies,” IEEE
Transactions on Systems, Man, and Cybernetics: Systems, vol. 51,
no. 1, pp. 266–279, 2020.
[2] S. Song, A. Zeng, J. Lee, and T. Funkhouser, “Grasping in the wild:
Learning 6dof closed-loop grasping from low-cost demonstrations,”
IEEE Robotics and Automation Letters, vol. 5, no. 3, pp. 4978–4985,
2020.
[3] D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang,
D. Quillen, E. Holly, M. Kalakrishnan, V. Vanhoucke et al., “Scalable
deep reinforcement learning for vision-based robotic manipulation,” in
Conference on robot learning. PMLR, 2018, pp. 651–673.
[4] K. Bousmalis, A. Irpan, P. Wohlhart, Y. Bai, M. Kelcey, M. Kalakrishnan, L. Downs, J. Ibarz, P. Pastor, K. Konolige et al., “Using simulation
and domain adaptation to improve efficiency of deep robotic grasping,”
in 2018 IEEE international conference on robotics and automation
(ICRA). IEEE, 2018, pp. 4243–4250.
[5] S. James, P. Wohlhart, M. Kalakrishnan, D. Kalashnikov, A. Irpan,
J. Ibarz, S. Levine, R. Hadsell, and K. Bousmalis, “Sim-to-real via simto-sim: Data-efficient robotic grasping via randomized-to-canonical
adaptation networks,” in Proceedings of the IEEE/CVF conference on
computer vision and pattern recognition, 2019, pp. 12 627–12 637.
[6] J. Tobin, L. Biewald, R. Duan, M. Andrychowicz, A. Handa, V. Kumar,
B. McGrew, A. Ray, J. Schneider, P. Welinder et al., “Domain
randomization and generative models for robotic grasping,” in 2018
IEEE/RSJ International Conference on Intelligent Robots and Systems
(IROS). IEEE, 2018, pp. 3482–3489.
[7] D. Ho, K. Rao, Z. Xu, E. Jang, M. Khansari, and Y. Bai, “Retinagan:
An object-aware approach to sim-to-real transfer,” in 2021 IEEE
International Conference on Robotics and Automation (ICRA). IEEE,
2021, pp. 10 920–10 926.
[8] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros, “Unpaired imageto-image translation using cycle-consistent adversarial networks,” in
Proceedings of the IEEE international conference on computer vision,


Based on your environment, establish your desired region of interest used to
create the image mask. Save these points as  self.vertices  .


For panoptic/instance segmentation, to parse the labels_map, click on any pixel on the labels_map window to see the label and instance count of that pixel.


  Published Topics

` image_rect ` ( [ sensor_msgs/Image
](http://docs.ros.org/en/api/sensor_msgs/html/msg/Image.html) )

  * Rectified image.


There are 2 segmentation cameras in the SDF world: a semantic segmentation camera, and an instance/panoptic segmentation camera.


* Holding right click and dragging scrolls in and out as well.


  * Calculates a histogram of the bottom half of the image
  * Find the peak of that histogram
  * Partition the image into horizontal windows
  * Start the bottom window at the peak of the histogram, assuming this is the lane
  * Move up the image to find pixels that are likely to be part of the lane, re-centering the sliding windows opportunistically
  * Fit a polynomial to the group of pixels, which represents the estimated lane line


##  thoughts on robotics, product management, and leadership


38  24-May-2023 10:45  8 / 8  ](https://build.ros.org//job/Ndev__image_pipeline__ubuntu_focal_amd64/38/)
  * [


The next step is to determine the best threshold values for identifying the
lane lines. Now that the bag file has been refined, save a couple example
images. Play the bag file, run the rqt image viewer, and save some images
using the image viewer.


This project leverages the OpenCV (Open Source Computer Vision Library)
libraries to implement a range of image processing algorithms. OpenCV was
designed for computational efficiency and with a strong focus on real-time
applications. Written in optimized C/C++, the library can take advantage of
multi-core processing. To verify that OpenCV is installed, run the following
commands:


<sensor name="segmentation_camera" type="segmentation">
  ...
  <camera>
    ...
    <save enabled="true">
      <path>DATSET_PATH</path>
    </save>
  </camera>
</sensor>
In the example world we just ran (segmentation_camera.sdf), you'll notice that the panoptic camera is saving data to segmentation_data/instance_camera, while the semenatic camera is saving data to segmentation_data/semantic_camera (these are relative paths).


[ ![ROS RasPiCam Lane Detection Estimated Position]( https://s3-us-
west-2.amazonaws.com/selbystorage/wp-
content/uploads/2017/July/ros_driverless_car_kitchen_control_image.png) ](
https://s3-us-west-2.amazonaws.com/selbystorage/wp-
content/uploads/2017/July/ros_driverless_car_kitchen_control_image.png)


To assign a label to a model we use the label plugin in the SDF file:


python3 segmentation_visualizer.py --path segmentation_data/instance_camera
You will see 4 windows: image, labels_map, colored_map, and colored_image (which is a combination of the image and colored_map).


2017, pp. 2223–2232.
[9] M. Liu, S. Fang, H. Dong, and C. Xu, “Review of digital twin
about concepts, technologies, and industrial applications,” Journal of
manufacturing systems, vol. 58, pp. 346–361, 2021.
[10] D. Liu, Y. Chen, and Z. Wu, “Digital twin (dt)-cyclegan: Enabling zero-shot sim-to-real transfer of visual grasping models,” IEEE
Robotics and Automation Letters, vol. 8, no. 5, pp. 2421–2428, 2023.
[11] Y. Choi, M. Choi, M. Kim, J.-W. Ha, S. Kim, and J. Choo, “Stargan:
Unified generative adversarial networks for multi-domain image-toimage translation,” in Proceedings of the IEEE conference on computer
vision and pattern recognition, 2018, pp. 8789–8797.
[12] L. Zhang, A. Rao, and M. Agrawala, “Adding conditional control
to text-to-image diffusion models,” in Proceedings of the IEEE/CVF
International Conference on Computer Vision, 2023, pp. 3836–3847.
[13] G. Zheng, X. Zhou, X. Li, Z. Qi, Y. Shan, and X. Li, “Layoutdiffusion: Controllable diffusion model for layout-to-image generation,”
in Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition, 2023, pp. 22 490–22 499.
[14] Y. Li, M. Keuper, D. Zhang, and A. Khoreva, “Adversarial supervision makes layout-to-image diffusion models thrive,” arXiv preprint
arXiv:2401.08815, 2024.
[15] A. Bicchi and V. Kumar, “Robotic grasping and contact: A review,” in
Proceedings 2000 ICRA. Millennium conference. IEEE international
conference on robotics and automation. Symposia proceedings (Cat.
No. 00CH37065), vol. 1. IEEE, 2000, pp. 348–353.
[16] Q. M. Marwan, S. C. Chua, and L. C. Kwek, “Comprehensive review
on reaching and grasping of objects in robotics,” Robotica, vol. 39,
no. 10, pp. 1849–1882, 2021.
[17] K. Kleeberger, R. Bormann, W. Kraus, and M. F. Huber, “A survey
on learning-based robotic grasping,” Current Robotics Reports, vol. 1,
pp. 239–249, 2020.
[18] J. Bohg, A. Morales, T. Asfour, and D. Kragic, “Data-driven grasp
synthesis—a survey,” IEEE Transactions on robotics, vol. 30, no. 2,
pp. 289–309, 2013.
[19] S. Levine, P. Pastor, A. Krizhevsky, J. Ibarz, and D. Quillen, “Learning
hand-eye coordination for robotic grasping with deep learning and
large-scale data collection,” The International journal of robotics
research, vol. 37, no. 4-5, pp. 421–436, 2018.
[20] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer,
“High-resolution image synthesis with latent diffusion models,” in
Proceedings of the IEEE/CVF conference on computer vision and
pattern recognition, 2022, pp. 10 684–10 695.
[21] H. Xue, Z. Huang, Q. Sun, L. Song, and W. Zhang, “Freestyle layoutto-image synthesis,” in Proceedings of the IEEE/CVF Conference on
Computer Vision and Pattern Recognition, 2023, pp. 14 256–14 266.


annotated images to support a broad range of vision
tasks, notably in autonomous driving technologies. Captured across diverse cities, Cityscapes contains a variety
of urban landscapes, seasonal variations, and weather
conditions, making it a robust dataset for training and
evaluating semantic segmentation models. This dataset
meticulously annotates numerous elements, including
vehicles, pedestrians, and urban fixtures, making it a
useful tool for researchers working on innovative ways
to analyze urban scenes and improve how intelligent
systems understand these environments.
These two public datasets offer the model the ability to
generate realistic-style images from diverse robot application
environments, such as indoor bedrooms, indoor kitchens,
outdoor scenes, agricultural scenes, autonomous driving, etc.
2) Experimental Dataset: All the simulation scenes are
built on the Gazebo platform. Examples can be seen in Figure
3, five independent scenarios were built, with the semantic
camera set to record the segmentation image of different
angles. Because the objects in the simulation environments
are labeled, the segmentation masks or a bounding box can
be easily obtained and work as the ground-truth label dataset
in this experiment. In this research, 1235 experimental pairs
were created. Each pair has a simulation image (Figure
3 Gazebo column), cropped by the simulation camera, an
instance segmentation image, and a semantic segmentation
image (Figure 3 Segmentation column).
B. Experiments in Image Generation
The key objective is to guarantee that the model can
generate high-quality images that are suitable for training the
object detection network (YOLOv8), thus the most important
evaluation metric of this part is the deviations between the
Fig. 4: The object detection results of the physical robot. The first panel represents the detection performance of the simple
background, and the second panel represents the complex background. It can be seen that the target objects are successfully
detected while the background has lots of interference objects.
centers of ground-truth bounding boxes and the bounding
boxes detected by the object detection model.
To obtain the training datasets in the comparison experiments, the segmentation images are sent to the trained CycleGAN, the fine-tuned ControlNet(stable-diffusion-v2.1[32]),
and ALDM(These three models were both trained on
ADE20k to compare the performance) to generate realisticstyle images. The images generated by these three models
were detected by YOLOv8 separately and analyzed quantitatively.


<sensor name="semantic_segmentation_camera" type="segmentation">
  <topic>semantic</topic>
  <camera>
    <segmentation_type>semantic</segmentation_type>
    <horizontal_fov>1.57</horizontal_fov>
    <image>
      <width>800</width>
      <height>600</height>
    </image>
    <clip>
      <near>0.1</near>
      <far>100</far>
    </clip>
    <save enabled="true">
      <path>segmentation_data/semantic_camera</path>
    </save>
  </camera>
  <always_on>1</always_on>
  <update_rate>30</update_rate>
  <visualize>true</visualize>
</sensor>
Segmentation Dataset Generation
To save the output of the sensor as segmentation dataset samples, we add the <save> tag to the <camera> tag, and we specify the path to save the dataset in.


You can launch the file with the following command, providing the full
absolute path to your ` ROS.bag ` file.


import cv2
import argparse
import os
import numpy as np

# Add the colored map to the image for visualization
def add_colored_to_image(image, colored):
    return cv2.addWeighted(cv2.resize(image, (colored.shape[1], colored.shape[0]))
                            .astype(np.uint8), 1,
                            colored.astype(np.uint8), .5,
                            0, cv2.CV_32F)

# global labels map for mouse callback
labels_map = None

# Callback when you click on any pixel in the labels_map image
# Prints the label and instance count of the clicked pixel
def mouse_callback(event, x, y, flags, param):
    if event == cv2.EVENT_LBUTTONDOWN:
        # instance / panoptic
        # label id
        label = labels_map[y,x,0]

        # instance count from the other 2 values of the pixel
        instance_count = labels_map[y,x,1] * 256 + labels_map[y,x,2]

        print(f'label: {label} .. instance count: {instance_count}')

cv2.namedWindow('labels_map')
cv2.setMouseCallback('labels_map', mouse_callback)

# Arg Parser to set the dataset path
parser = argparse.ArgumentParser()
parser.add_argument('--path', type=str, required=True, help='Segmentation Dataset Path')
args = parser.parse_args()

# dataset path
path = args.path

# paths of images folders


#include <cstdint>

#include <gz/msgs.hh>
#include <gz/rendering.hh>
#include <gz/transport.hh>

void OnNewLabelMap(const gz::msgs::Image &_msg)
{
  auto width = _msg.width();
  auto height = _msg.height();
  auto buffer = _msg.data();

  for (uint32_t i = 0; i < height; ++i)
  {
    for (uint32_t j = 0; j < width; ++j)
    {
      auto index = (i * width + j) * 3;

      // label id of the pixel is in the last channel
      auto label = buffer[index + 2];

      // In case of semantic segmentation, instance count is ignored.
      //
      // For panoptic segmentation, we can get the instance count as follows:
      // 16 bit value for instance count
      auto instanceCount1 = buffer[index + 1];
      auto instanceCount2 = buffer[index];
      // get the 16-bit decimal value from the two 8-bit values by left-shifting
      // instanceCount1 by 8 bits
      auto instanceCount = instanceCount1 * 256 + instanceCount2;
    }
  }
}

void OnNewColoredMap(const gz::msgs::Image &_msg)
{
  auto width = _msg.width();
  auto height = _msg.height();
  auto buffer = _msg.data();


  Package Summary

Released

Continuous Integration:  8 / 8

  * Build history (last 2 of 2 builds):
  * [


image_proc.2BAC8-cturtle.image_proc) , which is meant to sit
between the camera driver and vision processing nodes. ` image_proc ` removes
camera distortion from the raw image stream, and if necessary will convert
Bayer or YUV422 format image data to color.

**Monocular processing example**

---

![Raw image](/image_proc/cturtle?action=AttachFile&do=get&target=left_raw.png)

|

![Rectified
image](/image_proc/cturtle?action=AttachFile&do=get&target=left_rect_color.png)

**image_raw** : Original camera image, Bayered and distorted

|

**image_rect_color** : Rectified image, de-Bayered and undistorted (amount of
black border may vary depending on calibration)

New in ROS Electric  ` image_proc ` also contains the [ crop_decimate
](/image_proc


    $ python
    >>> import cv2
    >>> cv2.__version__
    '2.4.9.1'
    >>>


###  Saving the rviz Configuration


  image_proc/resize

Takes image and/or camera info and resize them.


Once an image is received, the image goes through the segmentation process
identified above.


$roslaunch lane_detection rover_center_line_cam.launch


[embedyt] [ https://www.youtube.com/watch?v=zrk9qAqG2o4%5B/embedyt%5D
](https://www.youtube.com/watch?v=zrk9qAqG2o4%5B/embedyt%5D)


  Parameters

` ~queue_size ` ( ` int ` , default: 5)

  * Size of message queue for synchronizing image and camera_info topics. You may need to raise this if images take significantly longer to travel over the network than camera info.


Save the camera calibration file in the camera_info directory of the
raspicam_node package and update the camerav2_320x240.launch file to ensure
the calibration file is loaded.


The nodegraph below shows the nodes and topics required for this simulation.
In contrast to the drive-by-wire set-up, we now have multiple control commands
being computed and combined in the yocs_cmd_vel_mux node. The [
center_line_detection node
](https://github.com/wilselby/diy_driverless_car_ROS/blob/master/rover_cv/lane_detection/src/lane_detection/center_line_detection.py)
is responsible for performing the image segmentation and computing the desired
trajectories based on the position of the lane identified in the image.


Launch  [ rover_center_line_cam_hitl.launch
](https://github.com/wilselby/diy_driverless_car_ROS/blob/master/rover_cv/lane_detection/launch/rover_center_line_cam_hitl.launch
"rover_center_line_cam_hitl.launch") to view the performance of the image
segmentation algorithm on the entire bag file. If you find problem areas, save
an image and evaluate it using the range_detector.py script. Adjust your
threshold values until you get the most reliable performance.


ROS also provides the [ vision_opencv ](http://wiki.ros.org/vision_opencv)
stack which packages the OpenCV library for ROS. This stack provides the [
cv_bridge ](http://wiki.ros.org/cv_bridge) and [ image_geometry
](http://wiki.ros.org/image_geometry) packages. The cv_bridge package serves
as the interface between OpenCV image variables and ROS image messages.


[ DIY Autonomous Vehicle Project
](https://www.wilselby.com/research/autonomous-vehicles/diy-autonomous-
vehicle-project/) ■ [ Computer Vision Applications
](https://www.wilselby.com/research/autonomous-vehicles/diy-autonomous-
vehicle-project/elegoo-autonomous-vehicle-assembly/)


  Package Summary

Released

Continuous Integration

Documented

Single image rectification and color processing.

  * Maintainer status: maintained
  * Maintainer: Vincent Rabaud <vincent.rabaud AT gmail DOT com>
  * Author: Patrick Mihelich, Kurt Konolige, Jeremy Leibs
  * License: BSD
  * Source: git [ https://github.com/ros-perception/image_pipeline.git ](https://github.com/ros-perception/image_pipeline) (branch: indigo)

_**[ image_pipeline ](/image_pipeline?distro=kinetic) ** : [
camera_calibration ](/camera_calibration?distro=kinetic) | [ depth_image_proc
](/depth_image_proc?distro=kinetic) | image_proc | [ image_publisher
](/image_publisher?distro=kinetic) | [ image_rotate
](/image_rotate?distro=kinetic) | [ image_view ](/image_view?distro=kinetic) |
[ stereo_image_proc ](/stereo_image_proc?distro=kinetic) _

**Package Links**

  * **[ Code API ](http://docs.ros.org/kinetic/api/image_proc/html) **
  * [ Tutorials ](/image_proc/Tutorials)
  * [ FAQ ](http://answers.ros.org/questions/scope:all/sort:activity-desc/tags:image_proc/page:1/)
  * [ Changelog ](http://docs.ros.org/en/kinetic/changelogs/image_proc/changelog.html)
  * [ Change List ](/image_pipeline/ChangeList)
  * [ Reviews ](/image_proc/Reviews)

** Dependencies  (9) **

  * [ catkin ](/catkin?distro=kinetic)
  * [ cv_bridge ](/cv_bridge?distro=kinetic)
  * [ dynamic_reconfigure ](/dynamic_reconfigure?distro=kinetic)
  * [ image_geometry ](/image_geometry?distro=kinetic)
  * [ image_transport ](/image_transport?distro=kinetic)
  * [ nodelet ](/nodelet?distro=kinetic)


  Package Summary

No API documentation

Single image rectification and color processing.

  * Author: Patrick Mihelich, Kurt Konolige, Jeremy Leibs
  * License: BSD
  * Repository: [ ros-pkg ](/ros-pkg)
  * Source: svn [ https://code.ros.org/svn/ros-pkg/stacks/image_pipeline/tags/image_pipeline-1.4.2 ](https://code.ros.org/svn/ros-pkg/stacks/image_pipeline/tags/image_pipeline-1.4.2)

**Package Links**

  * **[ Code API ](http://docs.ros.org/en/electric/api/image_proc/html/) **
  * [ Tutorials ](/image_proc/Tutorials)
  * [ FAQ ](http://answers.ros.org/questions/scope:all/sort:activity-desc/tags:image_proc/page:1/)
  * [ Change List ](/image_pipeline-1.6/ChangeList)
  * [ Reviews ](/image_proc/Reviews)

** Dependencies  (9) **

  * [ camera_calibration_p... ](/camera_calibration_parsers?distro=electric)
  * [ cv_bridge ](/cv_bridge?distro=electric)
  * [ dynamic_reconfigure ](/dynamic_reconfigure?distro=electric)
  * [ image_geometry ](/image_geometry?distro=electric)
  * [ image_transport ](/image_transport?distro=electric)
  * [ nodelet ](/nodelet?distro=electric)
  * [ opencv2 ](/opencv2?distro=electric)
  * [ roscpp ](/roscpp?distro=electric)
  * [ sensor_msgs ](/sensor_msgs?distro=electric)

** Used by  (9) **

  * [ arm_navigation_tests... ](/arm_navigation_tests?distro=electric)
  * [ augmented_object_sel... ](/augmented_object_selection?distro=electric)
  * [ cob_bringup ](/cob_bringup?distro=electric)
  * [ cob_controller_confi... ](/cob_controller_configuration_gazebo?distro=electric)
  * [ openni_launch ](/openni_launch?distro=electric)
  * [ pr2_gazebo ](/pr2_gazebo?distro=electric)
  * [ stereo_image_proc ](/stereo_image_proc?distro=electric)


Undo the changes to the code so [ center_line_detection node.py
](https://github.com/wilselby/diy_driverless_car_ROS/blob/master/rover_cv/lane_detection/src/lane_detection/center_line_detection.py)
is back in its original form. This time, we will view the lane detection with
the line of best fit visualized. The line should display consistently as long
as the lane segments are detected. The blue dot shows the setpoint position
that will be sent to the controller. Adjust the  ld.movingAverage  window to
improve the responsiveness of the setpoint calculation.


  Subscribed Topics

` image_raw ` ( [ sensor_msgs/Image
](http://docs.ros.org/en/api/sensor_msgs/html/msg/Image.html) )

  * Raw image stream from the camera driver.

` camera_info ` ( [ sensor_msgs/CameraInfo
](http://docs.ros.org/en/api/sensor_msgs/html/msg/CameraInfo.html) )

  * Camera metadata.


  * **melodic:** _Documentation generated on December 07, 2022 at 11:20 AM_ ( [ doc job ](https://build.ros.org/view/Mdoc/job/Mdoc__image_pipeline__ubuntu_bionic_amd64) ).

  * **noetic:** _Documentation generated on January 24, 2024 at 11:53 AM_ ( [ doc job ](https://build.ros.org/view/Ndoc/job/Ndoc__image_pipeline__ubuntu_focal_amd64) ).

_**[ image_pipeline ](/image_pipeline?distro=diamondback) ** : [
camera_calibration ](/camera_calibration?distro=diamondback) | image_proc | [
image_rotate ](/image_rotate?distro=diamondback) | [ image_view
](/image_view?distro=diamondback) | [ stereo_image_proc
](/stereo_image_proc?distro=diamondback) _

**Package Links**

  * **[ Code API ](http://docs.ros.org/en/diamondback/api/image_proc/html/) **
  * [ Tutorials ](/image_proc/Tutorials)
  * [ FAQ ](http://answers.ros.org/questions/scope:all/sort:activity-desc/tags:image_proc/page:1/)
  * [ Reviews ](/image_proc/Reviews)

** Dependencies  (8) **

  * [ camera_calibration_p... ](/camera_calibration_parsers?distro=diamondback)
  * [ cv_bridge ](/cv_bridge?distro=diamondback)
  * [ image_geometry ](/image_geometry?distro=diamondback)
  * [ image_transport ](/image_transport?distro=diamondback)
  * [ nodelet ](/nodelet?distro=diamondback)
  * [ opencv2 ](/opencv2?distro=diamondback)
  * [ roscpp ](/roscpp?distro=diamondback)
  * [ sensor_msgs ](/sensor_msgs?distro=diamondback)

** Used by  (5) **

  * [ arm_navigation_tests... ](/arm_navigation_tests?distro=diamondback)
  * [ articulation_tutoria... ](/articulation_tutorials?distro=diamondback)
  * [ qualification ](/qualification?distro=diamondback)
  * [ stereo_image_proc ](/stereo_image_proc?distro=diamondback)
  * [ visp_tracker ](/visp_tracker?distro=diamondback)


[ ![ROS Gaazebo Lane Detection RasPiCam Images]( https://s3-us-
west-2.amazonaws.com/selbystorage/wp-
content/uploads/2017/July/ros_driverless_car_image_thresholding.jpg) ](
https://s3-us-west-2.amazonaws.com/selbystorage/wp-
content/uploads/2017/July/ros_driverless_car_image_thresholding.jpg)


Next, a perspective transform is performed on the masked image to create a
warped image. The perspective_transform function transforms the image from the
front-facing camera view to a top down “bird’s-eye” view. The front-facing
camera view’s perspective distortion causes parallel, straight lanes to
converge at a vanishing point. This transformation makes the processing easier
for the lane detection algorithms since parallel, straight lines are rendered
as vertical lines.


[ Image Segmentation ](https://www.wilselby.com/research/autonomous-
vehicles/diy-autonomous-vehicle-project/computer-vision-applications/image-
segmentation/) ■ [ FormulaPi Lane Detection
](https://www.wilselby.com/research/autonomous-vehicles/formulapi-autonomous-
vehicle-racing/formulapi-ros-simulation-environment/)


Bring up the Atlas robot in Gazebo


  Package Summary

Released

Continuous Integration

Documented

Single image rectification and color processing.

  * Maintainer status: maintained
  * Maintainer: Vincent Rabaud <vincent.rabaud AT gmail DOT com>
  * Author: Patrick Mihelich, Kurt Konolige, Jeremy Leibs
  * License: BSD
  * Source: git [ https://github.com/ros-perception/image_pipeline.git ](https://github.com/ros-perception/image_pipeline) (branch: hydro-devel)

_**[ image_pipeline ](/image_pipeline?distro=indigo) ** : [ camera_calibration
](/camera_calibration?distro=indigo) | [ depth_image_proc
](/depth_image_proc?distro=indigo) | image_proc | [ image_publisher
](/image_publisher?distro=indigo) | [ image_rotate
](/image_rotate?distro=indigo) | [ image_view ](/image_view?distro=indigo) | [
stereo_image_proc ](/stereo_image_proc?distro=indigo) _

**Package Links**

  * **[ Code API ](http://docs.ros.org/indigo/api/image_proc/html) **
  * [ Tutorials ](/image_proc/Tutorials)
  * [ FAQ ](http://answers.ros.org/questions/scope:all/sort:activity-desc/tags:image_proc/page:1/)
  * [ Changelog ](http://docs.ros.org/en/indigo/changelogs/image_proc/changelog.html)
  * [ Change List ](/image_pipeline/ChangeList)
  * [ Reviews ](/image_proc/Reviews)

** Dependencies  (9) **

  * [ catkin ](/catkin?distro=indigo)
  * [ cv_bridge ](/cv_bridge?distro=indigo)
  * [ dynamic_reconfigure ](/dynamic_reconfigure?distro=indigo)
  * [ image_geometry ](/image_geometry?distro=indigo)
  * [ image_transport ](/image_transport?distro=indigo)
  * [ nodelet ](/nodelet?distro=indigo)


abilities in transfer-learning from prompts to images, transcending the conventional constraints paradigms. ControlNet
is adept at interpreting textual prompts, thereby furnishing
users with the ability to steer the image generation process
with accurate content of the images and adaptability. The
core idea usually involves guiding the generation process
through conditional input, thereby achieving fine control
over the properties of the generated image (such as layout,
style, content, etc.). This method is particularly suitable for
application scenarios that require highly customized output.
However, although ControlNet can utilize a condition image
as input to constrain the output the training of ControlNet
is more complicated, and the training results are relatively
uncontrollable. Although there is a condition image as input,
it may still cause deviations in the positions, sizes, numbers,
and arrangements of major objects in the output, so the
ControlNet is not suitable for this task. Fine-tuning ControlNet is a potential solution, but the fine-tuning process
requires a large amount of data. It does not necessarily
guarantee results, which means it is not an efficient and
reliable solution.
One possible way to solve the deviations is to use the
layout-to-image method, which aims to generate realistic
images based on predetermined layout information [13]. In
this framework, ”layout” generally refers to an abstract representation of objects in an image, covering their categories,
locations, sizes, and their interrelationships. This method
offers a significant advantage in robotic grasping by allowing
precise manipulation of both the layout and content within
generated images. This precision is particularly beneficial
for the transfer of simulated images to robot training, as
maintaining consistent positioning and relative relationships
among target objects is critical for avoiding later object
detection training failures. However, this scheme also has
a limitation that cannot be ignored, which is that labels of
different layouts are the only control over the content of
images, which makes the controls on the image contents
simple and general.
So in comparison, the adversarial supervised layout-toimage diffusion model (ALDM) [14] is especially suitable
for the Sim-to-Real task [14]. In the training phase of the
ALDM, a segmenter-based discriminator is connected in
series with a diffusion model. This configuration facilitates
the quantitative assessment of the discrepancy between the


  * [ nodelet_topic_tools ](/nodelet_topic_tools?distro=lunar)
  * [ roscpp ](/roscpp?distro=lunar)
  * [ sensor_msgs ](/sensor_msgs?distro=lunar)

** Used by  (10) **

  * [ image_pipeline ](/image_pipeline?distro=lunar)
  * [ openni2_launch ](/openni2_launch?distro=lunar)
  * [ opt_camera ](/opt_camera?distro=lunar)
  * [ pointgrey_camera_dri... ](/pointgrey_camera_driver?distro=lunar)
  * [ pr2_gazebo ](/pr2_gazebo?distro=lunar)
  * [ rgbd_launch ](/rgbd_launch?distro=lunar)
  * [ roseus_tutorials ](/roseus_tutorials?distro=lunar)
  * [ stereo_image_proc ](/stereo_image_proc?distro=lunar)
  * [ visp_hand2eye_calibr... ](/visp_hand2eye_calibration?distro=lunar)
  * [ visp_tracker ](/visp_tracker?distro=lunar)


[ ![ROS Gaazebo Lane Detection Binary Image]( https://s3-us-
west-2.amazonaws.com/selbystorage/wp-
content/uploads/2017/July/ros_driverless_car_gazebo_binary_image.png) ](
https://s3-us-west-2.amazonaws.com/selbystorage/wp-
content/uploads/2017/July/ros_driverless_car_gazebo_binary_image.png)


To play the bag file create a launch file called "playback.launch" with the
following content:


The final test uses the entire software suite to autonomously navigate the
Elegoo car around the track without any user input. I use the [ Tempus Monitor
and Integrated Keyboard ](http://amzn.to/2vrj4vg) to connect to the
RaspberryPi.


  Package Summary

Released

Continuous Integration

Documented

Single image rectification and color processing.

  * Maintainer status: maintained
  * Maintainer: Vincent Rabaud <vincent.rabaud AT gmail DOT com>
  * Author: Patrick Mihelich, Kurt Konolige, Jeremy Leibs
  * License: BSD
  * Source: git [ https://github.com/ros-perception/image_pipeline.git ](https://github.com/ros-perception/image_pipeline) (branch: indigo)

_**[ image_pipeline ](/image_pipeline?distro=lunar) ** : [ camera_calibration
](/camera_calibration?distro=lunar) | [ depth_image_proc
](/depth_image_proc?distro=lunar) | image_proc | [ image_publisher
](/image_publisher?distro=lunar) | [ image_rotate
](/image_rotate?distro=lunar) | [ image_view ](/image_view?distro=lunar) | [
stereo_image_proc ](/stereo_image_proc?distro=lunar) _

**Package Links**

  * **[ Code API ](http://docs.ros.org/lunar/api/image_proc/html) **
  * [ Tutorials ](/image_proc/Tutorials)
  * [ FAQ ](http://answers.ros.org/questions/scope:all/sort:activity-desc/tags:image_proc/page:1/)
  * [ Changelog ](http://docs.ros.org/en/lunar/changelogs/image_proc/changelog.html)
  * [ Change List ](/image_pipeline/ChangeList)
  * [ Reviews ](/image_proc/Reviews)

** Dependencies  (9) **

  * [ catkin ](/catkin?distro=lunar)
  * [ cv_bridge ](/cv_bridge?distro=lunar)
  * [ dynamic_reconfigure ](/dynamic_reconfigure?distro=lunar)
  * [ image_geometry ](/image_geometry?distro=lunar)
  * [ image_transport ](/image_transport?distro=lunar)
  * [ nodelet ](/nodelet?distro=lunar)


  Dynamically Reconfigurable Parameters

See the [ dynamic_reconfigure ](/dynamic_reconfigure) package for details on
dynamically reconfigurable parameters.

  *
` ~debayer ` ( ` int ` , default: 0)

  * Debayering algorithm. Possible values are:
    * ` Bilinear ` (0): Fast algorithm using bilinear interpolation
    * ` EdgeAware ` (1): Edge-aware algorithm
    * ` EdgeAwareWeighted ` (2): Weighted edge-aware algorithm
    * ` VNG ` (3): Slow but high quality Variable Number of Gradients algorithm


##  Playing and Visualizing rosbags


roslaunch playback.launch file:=/tmp/ROS.bag


The center_line_detection node subscribes and publishes on several topics.


  Package Summary

Released

Continuous Integration

Documented

Single image rectification and color processing.

  * Maintainer status: maintained
  * Maintainer: Vincent Rabaud <vincent.rabaud AT gmail DOT com>
  * Author: Patrick Mihelich, Kurt Konolige, Jeremy Leibs
  * License: BSD
  * Source: git [ https://github.com/ros-perception/image_pipeline.git ](https://github.com/ros-perception/image_pipeline) (branch: indigo)

_**[ image_pipeline ](/image_pipeline?distro=jade) ** : [ camera_calibration
](/camera_calibration?distro=jade) | [ depth_image_proc
](/depth_image_proc?distro=jade) | image_proc | [ image_publisher
](/image_publisher?distro=jade) | [ image_rotate ](/image_rotate?distro=jade)
| [ image_view ](/image_view?distro=jade) | [ stereo_image_proc
](/stereo_image_proc?distro=jade) _

**Package Links**

  * **[ Code API ](http://docs.ros.org/jade/api/image_proc/html) **
  * [ Tutorials ](/image_proc/Tutorials)
  * [ FAQ ](http://answers.ros.org/questions/scope:all/sort:activity-desc/tags:image_proc/page:1/)
  * [ Changelog ](http://docs.ros.org/en/jade/changelogs/image_proc/changelog.html)
  * [ Change List ](/image_pipeline/ChangeList)
  * [ Reviews ](/image_proc/Reviews)

** Dependencies  (9) **

  * [ catkin ](/catkin?distro=jade)
  * [ cv_bridge ](/cv_bridge?distro=jade)
  * [ dynamic_reconfigure ](/dynamic_reconfigure?distro=jade)
  * [ image_geometry ](/image_geometry?distro=jade)
  * [ image_transport ](/image_transport?distro=jade)
  * [ nodelet ](/nodelet?distro=jade)


Note: The playback will loop, but rviz needs to be told that time was reset at
the end of each loop. If you see the data stop displaying in rviz, click the
"Reset" button in the "Time" bar (by default, it's in the lower right corner).
Data display should start up again.


  * [ articulation_tutoria... ](/articulation_tutorials?distro=fuerte)
  * [ cob_bringup ](/cob_bringup?distro=fuerte)
  * [ cob_controller_confi... ](/cob_controller_configuration_gazebo?distro=fuerte)
  * [ freenect_launch ](/freenect_launch?distro=fuerte)
  * [ kingfisher_bringup ](/kingfisher_bringup?distro=fuerte)
  * [ openni_launch ](/openni_launch?distro=fuerte)
  * [ opt_camera ](/opt_camera?distro=fuerte)
  * [ pr2_gazebo ](/pr2_gazebo?distro=fuerte)
  * [ qualification ](/qualification?distro=fuerte)
  * [ roseus_tutorials ](/roseus_tutorials?distro=fuerte)
  * [ rosie_bringup ](/rosie_bringup?distro=fuerte)
  * [ stereo_image_proc ](/stereo_image_proc?distro=fuerte)
  * [ visp_hand2eye_calibr... ](/visp_hand2eye_calibration?distro=fuerte)
  * [ visp_tracker ](/visp_tracker?distro=fuerte)


  Subscribed Topics

` image ` ( [ sensor_msgs/Image
](http://docs.ros.org/en/api/sensor_msgs/html/msg/Image.html) )

  * Arbitrary image.

` camera_info ` ( [ sensor_msgs/CameraInfo
](http://docs.ros.org/en/api/sensor_msgs/html/msg/CameraInfo.html) )

  * Camera parameters.


    $ rosbag play <filename>
    $ rqt_image_view


The first step in the algorithm is to apply a mask to the image. This mask
keeps the region of the image defined by the given polygon, defined by the
vertices variable, and sets the rest of the image black. Since we know that
the lane lines will likely be in the center of the image, we can use the mask
to remove any objects that might match the color of the lanes but are not our
desired target. The masking is performed by the region_of_interest function.


The Gazebo simulator is good for debugging the image segmentation algorithm
and verifying the system architecture with the correct nodes and topics. The
next step is to verify the image segmentation algorithm and control system on
real-world data. This is called a software-in-the-loop simulation.


<plugin filename="gz-sim-label-system" name="gz::sim::systems::Label">
  <label>10</label>
</plugin>
We assign the label of the model by adding the plugin to the model's <visual> tag. So, in this case, this model has a label of 10.


[ ![ROS Gaazebo Lane Detection Node ROSGraph]( https://s3-us-
west-2.amazonaws.com/selbystorage/wp-
content/uploads/2017/July/ros_driverless_car_gazebo_center_conde_rosgraph.png)
]( https://s3-us-west-2.amazonaws.com/selbystorage/wp-
content/uploads/2017/July/ros_driverless_car_gazebo_center_conde_rosgraph.png)


rostopic list


##  Setup


  Published Topics

` camera_out/image_raw ` ( [ sensor_msgs/Image
](http://docs.ros.org/en/api/sensor_msgs/html/msg/Image.html) )

  * Cropped and decimated "raw" image.

` camera_out/camera_info ` ( [ sensor_msgs/CameraInfo
](http://docs.ros.org/en/api/sensor_msgs/html/msg/CameraInfo.html) )

  * Camera metadata, with binning and ROI fields adjusted to match output raw image.


roslaunch atlas_utils atlas.launch


In rviz, add the robot model, cameras, laser scan and the tf if they weren't
saved in your configuration. These streams should now be visualized in rviz as
if the robot were running. You should see the robot model itself move to the
positions you commanded during the rosbag recording (specifically, the laser
should rotate).


**For drcsim >= 4.1.0 ** : If you are using atlas versions >= v3 (e.g.
atlas_v3.launch), the laser topic has changed from ` /multisense_sl/laser/scan
` to ` /multisense/lidar_scan ` .


  * [ Home ](https://wilselby.com/)
  * [ Blog ](https://wilselby.com/blog/)
    * [ Robotics ](https://wilselby.com/category/robotics/)
    * [ Product Management ](https://wilselby.com/category/product-management/)
    * [ Management and Leadership ](https://wilselby.com/category/management-and-leadership/)
    * [ Software Development ](https://wilselby.com/category/software-development/)
    * [ Entrepreneurship ](https://wilselby.com/category/entrepreneurship/)
    * [ Military ](https://wilselby.com/category/military/)
  * [ Research ](https://wilselby.com/research/)
    * [ DIY Autonomous Vehicle Project ](https://wilselby.com/diy-autonomous-vehicle-project/)
    * [ ArduCopter ](https://wilselby.com/research/arducopter/)
    * [ ROS Integration ](https://wilselby.com/research/ros-integration/)
    * [ MIT Master’s Thesis ](https://wilselby.com/research/mit-masters-thesis/)
    * [ MIT Class Projects ](https://wilselby.com/research/mit-class-projects/)
    * [ Pilobolus Collaboration ](https://wilselby.com/research/pilobolus-collaboration/)
    * [ USNA Research ](https://wilselby.com/research/usna-research/)
  * [ Resume ](https://selbystorage.s3-us-west-2.amazonaws.com/PDF/William+C+Selby+-+Resume+19APR20.pdf)
  * [ Press ](https://wilselby.com/press/)
  * [ Recommended Reading ](https://wilselby.com/recommended-reading/)


rostopic pub /multisense_sl/set_spindle_speed std_msgs/Float64 '{ data: 6.0 } '


  Subscribed Topics

` camera/image_raw ` ( [ sensor_msgs/Image
](http://docs.ros.org/en/api/sensor_msgs/html/msg/Image.html) )

  * Raw image stream from the camera driver.

` camera/camera_info ` ( [ sensor_msgs/CameraInfo
](http://docs.ros.org/en/api/sensor_msgs/html/msg/CameraInfo.html) )

  * Camera metadata.


Manually drive the vehicle around the track for at least one complete lap and
make sure a new rosbag file is recorded in the data/ folder in the
rover_teleop package.


The next step is to take the warped image and identify pixels that are likely
to be the lanes. This will in a binary image, with the pixels that met the
criteria set as white and the remaining pixels set to black. For this initial
implementation, basic thresholding was used. The inputs to the  binary_thresh
function are an image and a series of minimum and maximum values for each
channel in the image.


[ ](https://twitter.com/wilselby) [ ](https://www.linkedin.com/in/wilselby) [
](https://www.youtube.com/user/wilSelby) [ ](http://goodreads.com/wilselby) [
](https://github.com/wilselby) [ ](https://www.paypal.me/WilSelby) [
](https://wilselby.com/feed/) [ ](/cdn-cgi/l/email-
protection#b1d094878297928188888a97928080808a97928080868a97928080818ac594868297928187858a97928080888a97928081848add97928080848a9487849487d297928188898a94868897928185878ad2de97928081888a)


40  22-Jan-2024 15:45  8 / 8  ](https://build.ros.org//job/Ndev__image_pipeline__ubuntu_focal_amd64/40/)
  * [


The thresholds can be determined by examining a series of images under varying
lighting conditions and identifying the optimal thresholds for each channel
that result in a binary image where the lane is detected without additional
noise. This process is described in more detail in subsequent sections. An
example output image is below.


Toggle navigation  [
![gazebo](/assets/masthead-0bd44817978df8069f427d8ca1657998789065a2b242edfd1a3d8ab4a329dd4c.png)
](/)


  * [ Tutorials ](/tutorials)
  * [ Download ](/download)
  * [ Blog ](/blog.html)
  * [ Media ](/media)
  * [ Projects ](/projects)


  Subscribed and Published Topics

Identical to the ` image_proc ` node.

Wiki: image_proc (last edited 2023-12-14 22:00:00 by  [ LucasWalter
](/LucasWalter "LucasWalter @
2601:602:8801:5830:3b6d:8c:7721:df6b\[2601:602:8801:5830:3b6d:8c:7721:df6b\]")
)

Except where otherwise noted, the ROS wiki is licensed under the
[ Creative Commons Attribution 3.0
](http://creativecommons.org/licenses/by/3.0/)

* * *

[ ![](/custom/images/brought_by_horiz.png) ](https://www.openrobotics.org/)


  Subscribed Topics

` image_mono ` ( [ sensor_msgs/Image
](http://docs.ros.org/en/api/sensor_msgs/html/msg/Image.html) )

  * Unrectified image stream.

` camera_info ` ( [ sensor_msgs/CameraInfo
](http://docs.ros.org/en/api/sensor_msgs/html/msg/CameraInfo.html) )

  * Camera metadata.


  ROS 2 Documentation

The ROS Wiki is for ROS 1. Are you using ROS 2 ( [ Humble
](http://docs.ros.org/en/humble/) , [ Iron ](http://docs.ros.org/en/iron/) ,
or [ Rolling ](http://docs.ros.org/en/rolling/) )?
[ Check out the ROS 2 Project Documentation ](http://docs.ros.org)
Package specific documentation can be found on [ index.ros.org
](https://index.ros.org)


  image_proc/rectify

Takes an unrectified image stream and its associated calibration parameters,
and produces rectified images.


rosbag record -O /tmp/ROS.bag /tf /atlas/joint_states /multisense_sl/camera/left/image_raw /multisense_sl/camera/left/camera_info /multisense_sl/laser/scan


####  Table of Contents


For ROS Groovy:


While your robot is running, start rviz from the command line


  image_proc/crop_decimate

Applies decimation (software binning) and ROI to a raw camera image post-
capture. Remap ` camera ` and ` camera_out ` to the desired input/output
camera namespaces.


###  Visualizing the Robot model


transferred to real settings through techniques like domain
randomization and domain adaptation [4], [5]. Domain randomization diversifies visual elements (e.g., textures, colors)
in training simulations to make models focus on invariant
aspects of images applicable in real-world scenarios. Conversely, domain adaptation tailors models from the simulated
(source) domain for the real (target) domain, often leveraging
unlabeled target domain data [6].
In image transfer tasks, GANs, such as RetinaGAN [7],
CycleGAN [8] are usually used as the main technical framework [9]. Some variants, such as DT-CycleGAN [10] and
StarGAN [11], are also invented to address different challenges. A major advantage of these kinds of models is that
their training process does not require data to appear in pairs,
which allows them to show better performance on specific
*Equal Contribution.
†Corresponding authors: Tianming Liu (tliu@uga.edu), Ramviyas Parasuraman (ramviyas@uga.edu), and Jin Sun (jinsun@uga.edu)
1School of Computing, University of Georgia, Athens, GA 30602, USA
(a) Generated image results of different models trained in ADE20k
dataset.
(b) Generated image results of ALDM trained in Cityscapes
dataset.
Fig. 1: ALDM has great performance in zero-shot image
generation and style transfer. The image generation model
used in this article contains data of various types and can
generate objects in various robotic application scenarios.
The generated image is not only close to the realistic-style
scene but also ensures that the number and position of key
objects are consistent with the original image. There is huge
potential to provide more diverse, realistic, and accurate
training samples for robot action planning applications.
tasks. However, GAN models also have some limitations, the
most significant of which are their preliminary requirements
for large amounts of training data and limitations in the
model’s generalization capabilities. The latter means that
when faced with new tasks or scenarios, existing GAN models often need to be re-trained from scratch, which greatly
constrains their applicability and flexibility in different appliarXiv:2403.11459v1 [cs.RO] 18 Mar 2024
cation cases. In contrast, the advantages of diffusion models
in this aspect are revealed. To effectively control the image
effects generated by the diffusion model, the research field
has developed a variety of technical paths, including layoutto-image, text-to-image, and image-to-image generations.
ControlNet [12], an innovative deep learning framework,
is engineered to exert meticulous control over generative
models’ outputs, with a particular emphasis on image synthesis tasks. This sophisticated architecture exhibits exceptional


Lastly, enter the threshold values as  self.boundaries  .


©2014 Open Source Robotics Foundation


    echo 'source /usr/share/drcsim/setup.sh' >> ~/.bashrc

    source ~/.bashrc


Lastly, we will compare the computed velocities with the velocities recorded
when manually driving the track. This will help us ensure that the control
system is calculating realistic numbers before running on the real system. Run
[ rover_center_line_cam_hitl.launch
](https://github.com/wilselby/diy_driverless_car_ROS/blob/master/rover_cv/lane_detection/launch/rover_center_line_cam_hitl.launch
"rover_center_line_cam_hitl.launch") again while monitoring the recorded
commands on the “/joy_teleop/cmd_vel” topic and the computed commands on the
“/platform_control/cmd_vel” topic.


  Nodelets

New in Diamondback, this package contains [ nodelets ](/nodelet) for the tasks
of debayering and rectification. This allows ` image_proc ` functions to be
easily combined with other nodelets, for example camera drivers or higher-
level vision processing. In fact, the ` image_proc ` node simply loads one `
debayer ` nodelet and two ` rectify ` nodelets.


A camera calibration can be run with the following commands. These need to be
updated based on the size and dimensions of the checkerboard you use.


#  Visualization and logging


<include>
   <pose>-1 0 3 0.0 0.0 1.57</pose>
   <uri>
   https://fuel.gazebosim.org/1.0/OpenRobotics/models/Construction Cone
   </uri>
   <plugin filename="gz-sim-label-system" name="gz::sim::systems::Label">
     <label>30</label>
   </plugin>
 </include>
Running an example:
Now that we've discussed how a segmentation camera and models with labels can be specified, let's run an example world that uses the segmentation camera. Run the following command:


  Package Summary

Released

Continuous Integration

Documented

Single image rectification and color processing.

  * Maintainer status: maintained
  * Maintainer: Vincent Rabaud <vincent.rabaud AT gmail DOT com>
  * Author: Patrick Mihelich, Kurt Konolige, Jeremy Leibs
  * License: BSD
  * Source: git [ https://github.com/ros-perception/image_pipeline.git ](https://github.com/ros-perception/image_pipeline) (branch: indigo)

_**[ image_pipeline ](/image_pipeline?distro=melodic) ** : [
camera_calibration ](/camera_calibration?distro=melodic) | [ depth_image_proc
](/depth_image_proc?distro=melodic) | image_proc | [ image_publisher
](/image_publisher?distro=melodic) | [ image_rotate
](/image_rotate?distro=melodic) | [ image_view ](/image_view?distro=melodic) |
[ stereo_image_proc ](/stereo_image_proc?distro=melodic) _

**Package Links**

  * **[ Code API ](http://docs.ros.org/en/melodic/api/image_proc/html) **
  * [ Tutorials ](/image_proc/Tutorials)
  * [ FAQ ](http://answers.ros.org/questions/scope:all/sort:activity-desc/tags:image_proc/page:1/)
  * [ Changelog ](http://docs.ros.org/en/melodic/changelogs/image_proc/changelog.html)
  * [ Change List ](/image_pipeline/ChangeList)
  * [ Reviews ](/image_proc/Reviews)

** Dependencies  (9) **

  * [ catkin ](/catkin?distro=melodic)
  * [ cv_bridge ](/cv_bridge?distro=melodic)
  * [ dynamic_reconfigure ](/dynamic_reconfigure?distro=melodic)
  * [ image_geometry ](/image_geometry?distro=melodic)
  * [ image_transport ](/image_transport?distro=melodic)
  * [ nodelet ](/nodelet?distro=melodic)


* Holding left click and dragging rotates the camera around the model


  Package Summary

No API documentation

Single image rectification and color processing.

  * Author: Patrick Mihelich, Kurt Konolige, Jeremy Leibs
  * License: BSD
  * Repository: [ ros-pkg ](/ros-pkg)
  * Source: svn [ https://code.ros.org/svn/ros-pkg/stacks/image_pipeline/tags/image_pipeline-1.6.4 ](https://code.ros.org/svn/ros-pkg/stacks/image_pipeline/tags/image_pipeline-1.6.4)

_**[ image_pipeline ](/image_pipeline?distro=fuerte) ** : [ camera_calibration
](/camera_calibration?distro=fuerte) | [ depth_image_proc
](/depth_image_proc?distro=fuerte) | image_proc | [ image_rotate
](/image_rotate?distro=fuerte) | [ image_view ](/image_view?distro=fuerte) | [
stereo_image_proc ](/stereo_image_proc?distro=fuerte) _

**Package Links**

  * **[ Code API ](http://docs.ros.org/fuerte/api/image_proc/html) **
  * [ Tutorials ](/image_proc/Tutorials)
  * [ FAQ ](http://answers.ros.org/questions/scope:all/sort:activity-desc/tags:image_proc/page:1/)
  * [ Change List ](/image_pipeline/ChangeList)
  * [ Reviews ](/image_proc/Reviews)

** Dependencies  (9) **

  * [ camera_calibration_p... ](/camera_calibration_parsers?distro=fuerte)
  * [ cv_bridge ](/cv_bridge?distro=fuerte)
  * [ dynamic_reconfigure ](/dynamic_reconfigure?distro=fuerte)
  * [ image_geometry ](/image_geometry?distro=fuerte)
  * [ image_transport ](/image_transport?distro=fuerte)
  * [ nodelet ](/nodelet?distro=fuerte)
  * [ opencv2 ](/opencv2?distro=fuerte)
  * [ roscpp ](/roscpp?distro=fuerte)
  * [ sensor_msgs ](/sensor_msgs?distro=fuerte)

** Used by  (15) **

  * [ arm_navigation_tests... ](/arm_navigation_tests?distro=fuerte)


Start the lane detection program using the following command to launch  [
rover_center_line_cam.launch
](https://github.com/wilselby/diy_driverless_car_ROS/blob/master/rover_cv/lane_detection/launch/rover_center_line_cam.launch
"rover_center_line_cam.launch") .


[ ![ROS RasPiCam Lane Detection Node ROSGraph]( https://s3-us-
west-2.amazonaws.com/selbystorage/wp-
content/uploads/2017/July/ros_driverless_car_center_rosgraph.png) ](
https://s3-us-west-2.amazonaws.com/selbystorage/wp-
content/uploads/2017/July/ros_driverless_car_center_rosgraph.png)


Visualize the segmentation dataset via Python
Put the following code in a Python script called segmentation_visualizer.py:


  * [ ](https://twitter.com/intent/tweet?text=Image%20Segmentation&url=https%3A%2F%2Fwilselby.com%2Fdiy-autonomous-vehicle-project%2Fcomputer-vision-applications%2Fimage-segmentation%2F)
  * [ ](https://linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Fwilselby.com%2Fdiy-autonomous-vehicle-project%2Fcomputer-vision-applications%2Fimage-segmentation%2F&title=Image+Segmentation)
  * [ ](https://reddit.com/submit?url=https%3A%2F%2Fwilselby.com%2Fdiy-autonomous-vehicle-project%2Fcomputer-vision-applications%2Fimage-segmentation%2F)
  * [ ](/cdn-cgi/l/email-protection#d0efa3a5b2bab5b3a4ed99bdb1b7b5f5e2e083b5b7bdb5bea4b1a4b9bfbef6f3e0e3e8ebb2bfb4a9edb8a4a4a0a3f5e391f5e296f5e296a7b9bca3b5bcb2a9feb3bfbdf5e296b4b9a9fdb1a5a4bfbebfbdbfa5a3fda6b5b8b9b3bcb5fda0a2bfbab5b3a4f5e296b3bfbda0a5a4b5a2fda6b9a3b9bfbefdb1a0a0bcb9b3b1a4b9bfbea3f5e296b9bdb1b7b5fda3b5b7bdb5bea4b1a4b9bfbef5e296)


Before starting the lane detection program, make sure the Elegoo power source
is powered off. Also, make sure the Xbox controller is running, connected, and
you can see the output as described in the [ Raspberry Pi Integration
](https://www.wilselby.com/research/autonomous-vehicles/diy-autonomous-
vehicle-project/raspberry-pi-integration/) section. Lastly, ensure that the
Arduino is running the  [ rover_motors.ino
](https://github.com/wilselby/diy_driverless_car_ROS/blob/master/rover_motors/rover_motors.ino
"rover_motors.ino") sketch, detailed in the [ Drive-by-Wire Development
](https://www.wilselby.com/research/autonomous-vehicles/diy-autonomous-
vehicle-project/drive-wire-development/) section.


First, we need to manually drive the vehicle around the test track and record
data. Specifically the images and control commands are recorded. Make sure the
record_cam  flag is set to “true” in  [ rover_dbw_cam.launch
](https://github.com/wilselby/diy_driverless_car_ROS/blob/master/rover_teleop/launch/rover_dbw_cam.launch
"rover_dbw_cam.launch") and launch the file.


In a separate terminal run rviz and get the list of topics played by the
rosbag


image_proc.2BAC8-electric.image_proc.2BAC8-crop_decimate)
nodelet, which applies decimation and ROI to a raw image in software.


A visualization of this process is shown below.


  for (uint32_t i = 0; i < height; ++i)
  {
    for (uint32_t j = 0; j < width; ++j)
    {
      auto index = (i * width + j) * 3;
      auto r = buffer[index];
      auto g = buffer[index + 1];
      auto b = buffer[index + 2];
    }
  }
}

int main(int argc, char **argv)
{
  gz::transport::Node node;

  if (!node.Subscribe("/segmentation/colored_map", &OnNewColoredMap) ||
    !node.Subscribe("/segmentation/labels_map", &OnNewLabelMap))
  {
    std::cerr << "Error subscribing to the boundingbox camera topic"
              << std::endl;
    return -1;
  }

  gz::transport::waitForShutdown();
  return 0;
}
If you'd like to gain a better understanding of how the subscriber code works, you can go through the gz-transport tutorials.


$ roslaunch rover_teleop rover_dbw_cam.launch


If you haven't done so, add the environment setup.sh files to your .bashrc.


<camera>: The camera, which has the following child elements:
<segmentation_type>: The type of segmentation performed by the camera. Use semantic for semantic segmentation. For panoptic (instance) segmentation, use panoptic or instance. The default value for <segmentation_type> is semantic.
<horizontal_fov>: The horizontal field of view, in radians.
<image>: The image size, in pixels.
<clip>: The near and far clip planes. Objects are only rendered if they're within these planes.
<always_on>: Whether the sensor will always be updated (indicated by 1) or not (indicated by 0). This is currently unused by Gazebo.
<update_rate>: The sensor's update rate, in Hz.
<visualize>: Whether the sensor should be visualized in the GUI (indicated by true) or not (indicated by false). This is currently unused by Gazebo.
<topic>: The name of the topic which will be used to publish the sensor data.
Label map & Colored map
The segmentation sensor creates 2 maps (or images):


**For drcsim >= 4.1.0 ** : If you are using atlas versions >= v3 (e.g.
atlas_v3.launch), the camera topic namespace has changed from `
/multisense_sl/ ` to ` /multisense/ ` .


  * [ nodelet_topic_tools ](/nodelet_topic_tools?distro=melodic)
  * [ roscpp ](/roscpp?distro=melodic)
  * [ sensor_msgs ](/sensor_msgs?distro=melodic)

** Used by  (14) **

  * [ avt_vimba_camera ](/avt_vimba_camera?distro=melodic)
  * [ cob_bringup ](/cob_bringup?distro=melodic)
  * [ fetch_bringup ](/fetch_bringup?distro=melodic)
  * [ freenect_launch ](/freenect_launch?distro=melodic)
  * [ image_pipeline ](/image_pipeline?distro=melodic)
  * [ openni2_launch ](/openni2_launch?distro=melodic)
  * [ opt_camera ](/opt_camera?distro=melodic)
  * [ pointgrey_camera_dri... ](/pointgrey_camera_driver?distro=melodic)
  * [ pr2_gazebo ](/pr2_gazebo?distro=melodic)
  * [ rgbd_launch ](/rgbd_launch?distro=melodic)
  * [ roseus_tutorials ](/roseus_tutorials?distro=melodic)
  * [ stereo_image_proc ](/stereo_image_proc?distro=melodic)
  * [ visp_hand2eye_calibr... ](/visp_hand2eye_calibration?distro=melodic)
  * [ visp_tracker ](/visp_tracker?distro=melodic)

** Jenkins jobs  (6) **

  * [ source ubuntu bionic ](https://build.ros.org/view/Msrc_uB/job/Msrc_uB__image_proc__ubuntu_bionic__source)
  * [ binary ubuntu bionic amd64 ](https://build.ros.org/view/Mbin_uB64/job/Mbin_uB64__image_proc__ubuntu_bionic_amd64__binary)
  * [ binary ubuntu bionic armhf ](https://build.ros.org/view/Mbin_ubhf_uBhf/job/Mbin_ubhf_uBhf__image_proc__ubuntu_bionic_armhf__binary)
  * [ binary ubuntu bionic arm64 ](https://build.ros.org/view/Mbin_ubv8_uBv8/job/Mbin_ubv8_uBv8__image_proc__ubuntu_bionic_arm64__binary)
  * [ devel ubuntu bionic amd64 ](https://build.ros.org/view/Mdev/job/Mdev__image_pipeline__ubuntu_bionic_amd64)
  * [ doc ](https://build.ros.org/view/Mdoc/job/Mdoc__image_pipeline__ubuntu_bionic_amd64)


** Used by  (12) **

  * [ avt_vimba_camera ](/avt_vimba_camera?distro=noetic)
  * [ cob_bringup ](/cob_bringup?distro=noetic)
  * [ image_pipeline ](/image_pipeline?distro=noetic)
  * [ openni2_launch ](/openni2_launch?distro=noetic)
  * [ pointgrey_camera_dri... ](/pointgrey_camera_driver?distro=noetic)
  * [ pr2_gazebo ](/pr2_gazebo?distro=noetic)
  * [ rgbd_launch ](/rgbd_launch?distro=noetic)
  * [ roseus_tutorials ](/roseus_tutorials?distro=noetic)
  * [ spinnaker_camera_dri... ](/spinnaker_camera_driver?distro=noetic)
  * [ stereo_image_proc ](/stereo_image_proc?distro=noetic)
  * [ visp_hand2eye_calibr... ](/visp_hand2eye_calibration?distro=noetic)
  * [ visp_tracker ](/visp_tracker?distro=noetic)

** Jenkins jobs  (6) **

  * [ source ubuntu focal ](https://build.ros.org/view/Nsrc_uF/job/Nsrc_uF__image_proc__ubuntu_focal__source)
  * [ binary ubuntu focal amd64 ](https://build.ros.org/view/Nbin_uF64/job/Nbin_uF64__image_proc__ubuntu_focal_amd64__binary)
  * [ binary ubuntu focal armhf ](https://build.ros.org/view/Nbin_ufhf_uFhf/job/Nbin_ufhf_uFhf__image_proc__ubuntu_focal_armhf__binary)
  * [ binary ubuntu focal arm64 ](https://build.ros.org/view/Nbin_ufv8_uFv8/job/Nbin_ufv8_uFv8__image_proc__ubuntu_focal_arm64__binary)
  * [ devel ubuntu focal amd64 ](https://build.ros.org/view/Ndev/job/Ndev__image_pipeline__ubuntu_focal_amd64)
  * [ doc ](https://build.ros.org/view/Ndoc/job/Ndoc__image_pipeline__ubuntu_focal_amd64)


  Mini Tutorial

Make sure your camera driver is running. You can check ` rostopic list | grep
image_raw ` to see the available raw image topics from compatible drivers.

Normally the raw image from the camera driver is not what you want for visual
processing, but rather an undistorted and (if necessary) debayered image. This
is the job of ` image_proc ` . If you are running on a robot, it's probably
best to run ` image_proc ` there. For example, if the driver is publishing
topics ` /my_camera/image_raw ` and ` /my_camera/camera_info ` you would do:

    $ ROS_NAMESPACE=my_camera rosrun image_proc image_proc

Notice that we push our ` image_proc ` instance down into the ` /my_camera `
namespace, in which it subscribes to the ` image_raw ` and ` camera_info `
topics. All output topics are likewise published within the ` /my_camera `
namespace.

In a separate terminal (on your home machine, if you are running on a robot):

    $ rosrun image_view image_view image:=my_camera/image_rect_color

This will display an undistorted color image from "my_camera".


5  04-Dec-2022 21:36  8 / 8  ](https://build.ros.org//job/Mdev__image_pipeline__ubuntu_bionic_amd64/5/)
  * [


  Subscribed Topics

` image_raw ` ( [ sensor_msgs/Image
](http://docs.ros.org/en/api/sensor_msgs/html/msg/Image.html) )

  * Raw image stream from the camera driver.


  Parameters

` ~queue_size ` ( ` int ` , default: 5)

  * Size of message queue for synchronizing image and camera_info topics. You may need to raise this if images take significantly longer to travel over the network than camera info.


The perspective transform requires arrays of source and destination points.
The goal is to have a warped image with vertical lines. The source and
destination points can be determined by manually inspecting an image with
known parallel lines as shown below. The series of images below depicts the
transformation with the source and destination points identified.


sudo apt-get install ros-hydro-rviz


We will record the joint states, cameras, laser and [ tf
](http://www.ros.org/wiki/tf) (transform data) from a robot into a file called
ROS.bag


    # Calculate Setpoint
    pts = np.vstack((fitx,ploty)).astype(np.float64).T
    self.avg = ld.movingAverage(self.avg, pts[-1][0],  N=3)
    self.intersectionPoint = np.array([self.avg])

    # Adjust Motors
    self.flag = control.adjustMotorSpeed(self.latestImage,  self.intersectionPoint,  self.speed,  self.cmdVelocityPub, self.cmdVelocityStampedPub, self.flag)

    # Publish Processed Image
    self.outputImage = self.processedImage
    self.publish(self.outputImage, self.bridge, self.image_pub)


Now we will add sensors to visualize. Click 'Add' to add a new item and add
'rviz > Camera'. Under the Camera item in the displays window, click the empty
space to the right of 'Image Topic' and an empty field should appear. Select
or type ` /multisense_sl/camera/left/image_raw ` . You should now see the
camera video feed in a small frame. Note that rviz displays an overlay of the
robot model on the camera image, which may cause circles to appear in the
camera images. The overlays can be turned off by setting the Camera > Overlay
Alpha to 1.0. Note: The camera images may appear grey as there is nothing in
front of Atlas so try dropping a box in front of the robot to see it in the
camera feed.


  Dynamically Reconfigurable Parameters

See the [ dynamic_reconfigure ](/dynamic_reconfigure) package for details on
dynamically reconfigurable parameters.

  *
` ~decimation_x ` ( ` int ` , default: 1)

  * Number of pixels to decimate to one horizontally. Range: 1 to 16

` ~decimation_y ` ( ` int ` , default: 1)

  * Number of pixels to decimate to one vertically. Range: 1 to 16

` ~x_offset ` ( ` int ` , default: 0)

  * X offset of the region of interest. Range: 0 to 2447

` ~y_offset ` ( ` int ` , default: 0)

  * Y offset of the region of interest. Range: 0 to 2049

` ~width ` ( ` int ` , default: 0)

  * Width of the region of interest. Range: 0 to 2448

` ~height ` ( ` int ` , default: 0)

  * Height of the region of interest. Range: 0 to 2050

` ~interpolation ` ( ` int ` , default: 0)

  * Sampling algorithm. Possible values are:
    * ` NN ` (0): Nearest-neighbor sampling
    * ` Linear ` (1): Bilinear interpolation
    * ` Cubic ` (2): Bicubic interpolation over 4x4 neighborhood
    * ` Area ` (3): Resampling using pixel area relation
    * ` Lanczos4 ` (4): Lanczos interpolation over 8x8 neighborhood

New in ROS Indigo


of computer vision. Extending GANs, Zhu et al. proposed
CycleGAN for unpaired image-to-image translation, utilizing
a cycle consistency loss alongside adversarial losses to
facilitate high-quality translations across different datasets
without requiring paired examples. Building upon these advancements, Ho et al. introduced RetinaGAN [7], an objectaware sim-to-real adaptation technique that leverages object
detection consistency to enhance the realism of translated
Fig. 2: The whole pipeline of the diffusion-model-based grasping robot. a) The generation procedures for realistic style
images. b) The robot agent network designed for the task of object detection and grasping.
images further. RetinaGAN’s innovation lies in its ability to
preserve object structures and textures across translations,
offering significant improvements in sim-to-real tasks for
robotics and demonstrating the adaptability of GANs to
diverse visual environments.
2) Diffusion Models: GANs initiated the era of generative modeling, but diffusion models represent a significant
paradigm shift by transforming noise into detailed images
through iterative refinement, addressing GANs’ issues like
training instability and mode collapse [20]. Among these
advancements, ControlNet [12] introduces spatial conditioning to text-to-image diffusion models, allowing for precise
image generation based on specific inputs such as edges and
poses. Concurrently, FreestyleNet [21] extends image synthesis capabilities through Freestyle Layout-to-Image Synthesis
(FLIS), which integrates textual and layout inputs using Rectified Cross-Attention (RCA). These innovations highlight
the flexibility and potential of diffusion models in pushing
the boundaries of generative art and technology.
C. Robotic Application
Sim-to-Real transfer is a crucial technique in robotics,
aimed at bridging the gap between simulations and realworld applications [17], [22], [23], [24]. Key advancements
include RetinaGAN, which improves simulated imagery for
visual grasping through object-detection consistency [7], and
RL-CycleGAN, which integrates Q-learning’s Q-value with
a reinforcement learning (RL) scene consistency loss to
enhance Sim-to-Real transfer in RL applications [25], [26].
These GAN-based methods have significantly advanced the
efficacy of visual grasping models by optimizing the use of
virtual data.
In summary, our research integrates diffusion models with
adversarial training to refine Sim-to-Real transfer [27]. This
hybrid approach combines the gradient-guided progression
from noise to realistic data of diffusion models with the robust learning of adversarial methods, offering precise object
locations, controllable image content generation, and zeroshot capacity for unseen scenario in Sim-to-Real applications.
III. METHODS


    <launch>
      <!-- Creates a command line argument called file -->
      <arg name="file"/>

      <!-- Run the rosbag play as a node with the file argument -->
      <node name="rosbag" pkg="rosbag" type="play" args="--loop $(arg file)" output="screen"/>
    </launch>


For this implementation, the images are converted to HSV format for
segmentation instead of RGB. Unlike RBG, HSV separates the color components
from the intensity. This makes it more robust to lighting changes and more
effective for our purposes.


[ ![ROS Gaazebo Lane Detection Warped Image]( https://s3-us-
west-2.amazonaws.com/selbystorage/wp-
content/uploads/2017/July/ros_driverless_car_gazebo_warped_image.png) ](
https://s3-us-west-2.amazonaws.com/selbystorage/wp-
content/uploads/2017/July/ros_driverless_car_gazebo_warped_image.png)


For ROS Hydro:


The Gazebo simulator is perfect for testing the image processing and control
systems design. The Conde Gazebo world described in the [ Drive-by-Wire
Development ](https://www.wilselby.com/research/autonomous-vehicles/diy-
autonomous-vehicle-project/drive-wire-development/) section is useful for this
purpose. The goal of this simulation is to identify and track the center lane
around the “figure eight” track.


After a while, stop the rosbag recording by pressing Control + C in its
terminal.


##  Visualizing Sensor Data with rviz


segmented outcomes of images produced by the diffusion
process and the ground-truth segmentation data. Consequently, the resulting model generates images where both
the objects and spatial arrangement closely approximate
reality (Fig. 1(a)). Simultaneously, the model preserves the
prompt control feature inherent in diffusion models. This
characteristic ensures that the synthesized images not only
align well with the Sim-to-Real (simulation to reality) objectives pertinent to robotic applications but also support the
facile adaptation to diverse experimental styles (Fig. 1(b)).
Hence, the images crafted by this model are exceptionally
conducive to robotic training across varied scenarios. The
main contributions of this work are as follows:
• We build a task learning pipeline on top of the state-ofthe-art ALDM approach to the task of the Sim-to-Real
robotic training and use quantitative results on generated
images’ segmentation to show that this method is highly
appropriate for the Sim-to-Real robotic grasping task.
• We design a pipeline to train and control a two-camera
robot to work in reality-unseen environments. The results show that the robot performs well in grasping
unseen objects in unpredictably complex circumstances,
which proves the reliability of the pipeline design.
II. RELATED WORK
A. Visual Grasping
Visual grasping is an interdisciplinary field at the intersection of robotics, computer vision, artificial intelligence,
and cognitive science, aiming to enable machines to interact
with their environment through the detection, analysis, and
manipulation of objects. This complex challenge involves
both visual data processing and the control of robotic actuators [15], [16]. With advancements in deep learning, visionbased robotic grasping has become a focal point of research,
adopting two primary methodologies: object reconstruction
and end-to-end approaches [17].
Object reconstruction methods focus on creating a detailed
3D model of the environment for precise planning and execution of grasps, offering accuracy at the cost of computational
intensity and slower response times [18]. Conversely, endto-end methods leverage machine learning to directly map
visual input to grasping actions without intermediate modeling, facilitating rapid adaptation to diverse scenarios albeit
with less interpretability and higher demands on training data
and computational resources [19], [2], [3], [4], [17].
B. Image Generation Models for Bridging Reality Gap
1) GANs Models: GANs revolutionized unsupervised
learning by introducing a framework composed of a Generator, creating data that mimics real-world distributions, and a
Discriminator, distinguishing between genuine and generated
data. This innovation enables the production of realistic
images, audio, and videos, significantly advancing the field


The center_line_detection node computes desired velocities and publishes them
on the /platform_control/cmd_vel topic. This is the topic reserved for
autonomous navigation commands. The yocs_cmd_vel_mux node receives these
commands and computes the final speed commands. For the simulation, these
commands are sent to Gazebo to maneuver the simulated vehicle. An image from a
simulation is shown below.


Once the desired velocities are computed and published, the output image is
published and the process repeats on the latest image


  * [ nodelet ](/nodelet?distro=hydro)
  * [ opencv2 ](/opencv2?distro=hydro)
  * [ roscpp ](/roscpp?distro=hydro)
  * [ rostest ](/rostest?distro=hydro)
  * [ sensor_msgs ](/sensor_msgs?distro=hydro)

** Used by  (15) **

  * [ bag_tools ](/bag_tools?distro=hydro)
  * [ cob_bringup ](/cob_bringup?distro=hydro)
  * [ cob_controller_confi... ](/cob_controller_configuration_gazebo?distro=hydro)
  * [ freenect_launch ](/freenect_launch?distro=hydro)
  * [ image_pipeline ](/image_pipeline?distro=hydro)
  * [ openni2_launch ](/openni2_launch?distro=hydro)
  * [ opt_camera ](/opt_camera?distro=hydro)
  * [ pointgrey_camera_dri... ](/pointgrey_camera_driver?distro=hydro)
  * [ pr2_gazebo ](/pr2_gazebo?distro=hydro)
  * [ ps4eye ](/ps4eye?distro=hydro)
  * [ rgbd_launch ](/rgbd_launch?distro=hydro)
  * [ roseus_tutorials ](/roseus_tutorials?distro=hydro)
  * [ stereo_image_proc ](/stereo_image_proc?distro=hydro)
  * [ visp_hand2eye_calibr... ](/visp_hand2eye_calibration?distro=hydro)
  * [ visp_tracker ](/visp_tracker?distro=hydro)


<model name="box">
      <pose>0 -1 0.5 0 0 0</pose>
      <link name="box_link">
        <inertial>
          <inertia>
            <ixx>1</ixx>
            <ixy>0</ixy>
            <ixz>0</ixz>
            <iyy>1</iyy>
            <iyz>0</iyz>
            <izz>1</izz>
          </inertia>
          <mass>1.0</mass>
        </inertial>
        <collision name="box_collision">
          <geometry>
            <box>
              <size>1 1 1</size>
            </box>
          </geometry>
        </collision>

        <visual name="box_visual">
          <geometry>
            <box>
              <size>1 1 1</size>
            </box>
          </geometry>
          <material>
            <ambient>0 0 0.5 1</ambient>
            <diffuse>0 0 1 1</diffuse>
            <specular>0 0 0.3 1</specular>
          </material>
          <plugin filename="gz-sim-label-system" name="gz::sim::systems::Label">
            <label>10</label>
          </plugin>
        </visual>
      </link>
    </model>
Lets zoom in the label plugin:


37  24-May-2023 09:45  8 / 8  ](https://build.ros.org//job/Ndev__image_pipeline__ubuntu_focal_amd64/37/)
  * [


This function returns a masked image.


Next, filter the bag so that only the image and velocity command topics are in
the bag. Then trim the bag file down based on the start and stop times
recorded earlier. Modify the commands based on your times and filenames.


For ROS Indigo:


[ Go to mobile version ](https://wilselby.com/diy-autonomous-vehicle-
project/computer-vision-applications/image-segmentation/?amp=1)


  Dynamically Reconfigurable Parameters

See the [ dynamic_reconfigure ](/dynamic_reconfigure) package for details on
dynamically reconfigurable parameters.

  *
` ~interpolation ` ( ` int ` , default: 1)

  * Interpolation algorithm between source image pixels. Possible values are:
    * ` NN ` (0): Nearest neighbor
    * ` Linear ` (1): Linear
    * ` Cubic ` (2): Cubic
    * ` Lanczos4 ` (4): Lanczos4


  Published Topics

` image_mono ` ( [ sensor_msgs/Image
](http://docs.ros.org/en/api/sensor_msgs/html/msg/Image.html) )

  * Monochrome image.

` image_color ` ( [ sensor_msgs/Image
](http://docs.ros.org/en/api/sensor_msgs/html/msg/Image.html) )

  * Color unrectified image.


[ ![ROS RasPiCam Lane Detection Perspective
Transform](https://i0.wp.com/s3-us-west-2.amazonaws.com/selbystorage/wp-
content/uploads/2017/July/ros_driverless_car_kitchen_persepctive_transform.jpg?resize=216%2C323&ssl=1)
](https://i0.wp.com/s3-us-west-2.amazonaws.com/selbystorage/wp-
content/uploads/2017/July/ros_driverless_car_kitchen_persepctive_transform.jpg?ssl=1)
This transformation results in a warped image as shown below.


####  Camera Calibration


Parameters


Poke around in rviz and add different sensors or robot information. TF
visualizes the joint transformations, Map visualizes a 2D collision map,
PointCloud(2) visualizes depth information from sensors like a Microsoft
Kinect.


  Parameters

` ~queue_size ` ( ` int ` , default: 5)

  * Size of message queue for synchronizing image and camera_info topics. You may need to raise this if images take significantly longer to travel over the network than camera info.


    """ROS Subscriptions """
    self.image_sub = rospy.Subscriber("/raspicam_node/image/image_raw",Image,self.cvt_image)

    self.image_pub = rospy.Publisher("/image_converter/output_video",Image, queue_size=10)
    self.cmdVelocityPub = rospy.Publisher('/platform_control/cmd_vel', Twist, queue_size=10)
    self.cmdVelocityStampedPub = rospy.Publisher('/platform_control/cmd_vel_stamped', TwistStamped, queue_size=10)


<sensor name="segmentation_camera" type="segmentation">
  <topic>segmentation</topic>
  <camera>
    <segmentation_type>instance</segmentation_type>
    <horizontal_fov>1.047</horizontal_fov>
    <image>
      <width>800</width>
      <height>600</height>
    </image>
    <clip>
      <near>0.1</near>
      <far>10</far>
    </clip>
  </camera>
  <always_on>1</always_on>
  <update_rate>30</update_rate>
  <visualize>true</visualize>
</sensor>
As we can see, we define a sensor with the following SDF elements:


    def binary_thresh( img,  boundaries,  filter):

         if filter == 'RGB':
             frame_to_thresh = img.copy()
         else:
             frame_to_thresh = cv2.cvtColor(img, cv2.COLOR_BGR2HSV)

         for (lower,  upper) in boundaries:

             # create numpy arrays from the boundaries
             lower = np.array(lower,  dtype = "uint8")
             upper = np.array(upper,  dtype = "uint8")

             # find the colors within the specified boundaries and apply the mask
             mask = cv2.inRange(frame_to_thresh,  lower,  upper)
             output = cv2.bitwise_and(frame_to_thresh, frame_to_thresh,  mask = mask)   #Returns an RGB image

    return mask


[ ![ROS RasPiCam Lane Detection Masked Image]( https://s3-us-
west-2.amazonaws.com/selbystorage/wp-
content/uploads/2017/July/ros_driverless_car_kitchen_masked_image.png) ](
https://s3-us-west-2.amazonaws.com/selbystorage/wp-
content/uploads/2017/July/ros_driverless_car_kitchen_masked_image.png)


Bring down rviz and your simulation of the Atlas robot. We'll now create a
launch file to play the rosbag file.


The [ raspicam_node ](https://github.com/UbiquityRobotics/raspicam_node)
package contains a calibration file for the RaspberryPi camera. However, it is
suggested to recalibrate the camera if you are using a non-default image size.
Since this project resizes the image to a 320×240 image to improve processing
speed, it was necessary to re-calibrate the camera.


Consider the following SDF snippet from the segmentation camera:


Now add a LaserScan using a similar method as the camera, and change the
'Topic' to ` /multisense_sl/laser/scan ` . Again, to see the laser scan
visualization in rviz, drop a box in front of the robot to see red points
rendered over the box as it falls.


This section describes the development and implementation of an image
segmentation algorithm to identify and track lanes in an image. This work was
inspired by the [ Udacity Self Driving Car Engineer Nanodegree program
](https://www.udacity.com/course/self-driving-car-engineer-nanodegree--nd013)
. This work references the open-source projects of some of the students who
completed the [ Basic Lane Detection ](https://github.com/udacity/CarND-
LaneLines-P1) and the [ Advanced Lane Detection
](https://github.com/udacity/CarND-Advanced-Lane-Lines) projects.


rosbag record -O /tmp/ROS.bag /tf /atlas/joint_states /multisense/camera/left/image_raw /multisense/camera/left/camera_info /multisense/lidar_scan


You can see a list of ROS topics by running


##  Wrap up


The algorithm begins by receiving a raw image from the RaspberryPi camera. For
the purposes of this walk through, we will use an image from the Conde Gazebo
world described in the [ Drive-by-Wire Development
](https://www.wilselby.com/research/autonomous-vehicles/diy-autonomous-
vehicle-project/drive-wire-development/) section. The basic image segmentation
and processing functions are defined in the  [ lane_detection_module.py
](https://github.com/wilselby/diy_driverless_car_ROS/blob/master/rover_cv/lane_detection/src/lane_detection/lane_detection_module.py
"lane_detection_module.py") file.


    # apply ROI mask
    self.maskedImage = ld.region_of_interest(self.latestImage, self.vertices)

    # perspective transform
    self.warpedImage,  _,  _ = ld.perspective_transform(self.maskedImage, self.corners)

    # detect binary lane markings
    self.binaryImage = ld.binary_thresh(self.warpedImage, self.boundaries, 'HSV') #RGB or HSV


Gazebo Sensors
API Reference
8.0.1
insert_drive_file
Tutorials
library_books
Classes
toc
Namespaces
insert_drive_file
Files
launch
Gazebo Website
Segmentation Camera in Gazebo
In this tutorial, we will discuss how to use a segmentation camera sensor in Gazebo.


[ ![ROS Gaazebo Lane Detection Simulation](https://i0.wp.com/s3-us-
west-2.amazonaws.com/selbystorage/wp-
content/uploads/2017/July/ros_driverless_car_gazebo_center_sim.png?resize=405%2C306&ssl=1)
](https://i0.wp.com/s3-us-west-2.amazonaws.com/selbystorage/wp-
content/uploads/2017/July/ros_driverless_car_gazebo_center_sim.png?ssl=1)


####  Simulation


#  [ Wil Selby ](https://wilselby.com/ "Wil Selby")


[ ![ros.org](/custom/images/ros_org.png) ](/) |  [ About
](http://www.ros.org/about-ros) | [ Support ](/Support) | [ Discussion Forum
](http://discourse.ros.org/) | [ Index ](http://index.ros.org/) | [ Service
Status ](http://status.ros.org/) | [ ros @ Robotics Stack Exchange
](https://robotics.stackexchange.com/questions/tagged/ros)
---|---
![](/custom/images/menu_left.png) [
![Documentation](/custom/images/menu_documentation.png) ](/)
![](/custom/images/menu_spacer.png) [ ![Browse
Software](/custom/images/menu_browse_software.png)
](https://index.ros.org/packages) ![](/custom/images/menu_spacer.png) [
![News](/custom/images/menu_news.png) ](https://discourse.ros.org/c/general)
![](/custom/images/menu_spacer.png) [
![Download](/custom/images/menu_download.png) ](/ROS/Installation)
![](/custom/images/menu_right.png)

  * [ image_proc ](/image_proc)


  Published Topics

` image_mono ` ( [ sensor_msgs/Image
](http://docs.ros.org/en/api/sensor_msgs/html/msg/Image.html) )

  * Monochrome unrectified image.

` image_rect ` ( [ sensor_msgs/Image
](http://docs.ros.org/en/api/sensor_msgs/html/msg/Image.html) )

  * Monochrome rectified image.

` image_color ` ( [ sensor_msgs/Image
](http://docs.ros.org/en/api/sensor_msgs/html/msg/Image.html) )

  * Color unrectified image.

` image_rect_color ` ( [ sensor_msgs/Image
](http://docs.ros.org/en/api/sensor_msgs/html/msg/Image.html) )

  * Color rectified image.


Visualizing and logging sensor information is an important part in developing
and debugging controllers. This tutorial will show you how to visualize the
simulated Atlas robot in rviz, log sensor information and replay that logged
sensor information through rviz.


images_path = os.path.join(path, "images")
labels_map_path = os.path.join(path, "labels_maps")
colored_map_path = os.path.join(path, "colored_maps")

# list all images paths
images_names  = sorted(os.listdir(images_path))
labels_names  = sorted(os.listdir(labels_map_path))
colored_names = sorted(os.listdir(colored_map_path))

# add the root path to images names
images_paths      = [os.path.join(images_path, name) for name in images_names]
labels_map_paths  = [os.path.join(labels_map_path, name) for name in labels_names]
colored_map_paths = [os.path.join(colored_map_path, name) for name in  colored_names]

for image_path, labels_path, colored_path in zip(images_paths, labels_map_paths, colored_map_paths):
    image = cv2.imread(image_path, cv2.IMREAD_COLOR)
    labels_map = cv2.imread(labels_path, cv2.IMREAD_COLOR)
    colored_map = cv2.imread(colored_path, cv2.IMREAD_COLOR)

    colored_image = add_colored_to_image(image, colored_map)
    cv2.imshow("segmentation", colored_image)

    cv2.imshow("image", image)
    cv2.imshow("labels_map", labels_map)
    cv2.imshow("colored_map", colored_map)

    print("Press Any Key to Continue ... ESC to exit")
    if cv2.waitKey(0) == 27:
        break

cv2.destroyAllWindows()
Make sure you have all of the dependencies that are needed in order to run this Python script:


  * [ visp_hand2eye_calibr... ](/visp_hand2eye_calibration?distro=electric)
  * [ visp_tracker ](/visp_tracker?distro=electric)


[ ![ROS Gaazebo Lane Detection Masked Image](https://i0.wp.com/s3-us-
west-2.amazonaws.com/selbystorage/wp-
content/uploads/2017/July/ros_driverless_car_gazebo_masked_image.png?resize=382%2C215&ssl=1)
](https://i0.wp.com/s3-us-west-2.amazonaws.com/selbystorage/wp-
content/uploads/2017/July/ros_driverless_car_gazebo_masked_image.png?ssl=1)


Next, we will refine the bag file. First we will trim the bag file to remove
any unwanted data. This can be done by playing the file and viewing the images
in rqt_image_view.


    rosrun rviz rviz &

    rostopic list


Record the beginning and ending time of the data you want to keep. The times
are displayed in the terminal where the bag file is being played.


  image_proc.launch

Loads ` debayer ` and ` rectify ` nodelets for one camera into a user-provided
nodelet manager. The in-process equivalent of running the ` image_proc ` node.
Launch (or include in a higher-level launch file) in the camera namespace,
just as you would run ` image_proc ` .


<model name="sphere">
  <static>true</static>
  <pose>-1 -2 0.5 0 0 0</pose>
  <link name="sphere_link">
  ...
  </link>
  <plugin filename="gz-sim-label-system" name="gz::sim::systems::Label">
    <label>20</label>
  </plugin>
</model>
If you're including a model from a place like Gazebo Fuel, you can add the label plugin as a child for the <include> tag:


$ python range-detector.py --filter HSV --image <file.png>


**For drcsim < 3.1.0 ** : The package and launch file had a different name:


gz sim segmentation_camera.sdf
You should see something similar to this:


    --- self.binaryImage = ld.binary_thresh(self.warpedImage, self.boundaries, 'HSV')
    +++ self.binaryImage = ld.binary_thresh(self.latestImage, self.boundaries, 'HSV')

    --- self.outputImage = self.processedImage
    +++ self.outputImage = self.maskedRGBImage