## Towards Zero-Shot Annotation of the Built Environment with Vision-Language Models (Vision Paper)

Bin Han bh193@uw.edu

## Yiwei Yang

University of Washington Seattle, USA

yanyiwei@uw.edu University of Washington Seattle, USA

## Anat Caspi

caspian@cs.washington.edu University of Washington Seattle, USA

## ABSTRACT

Equitable urban transportation applications require high-fidelity digital representations of the built environment: not just streets and sidewalks, but bike lanes, marked and unmarked crossings, curb ramps and cuts, obstructions, traffic signals, signage, street markings, potholes, and more. Direct inspections and manual annotations are prohibitively expensive at scale. Conventional machine learning methods require substantial annotated training data for adequate performance. In this paper, we consider vision language models as a mechanism for annotating diverse urban features from satellite images, reducing the dependence on human annotation to produce large training sets. While these models have achieved impressive results in describing common objects in images captured from a human perspective, their training sets are less likely to include strong signals for esoteric features in the built environment, and their performance in these settings is therefore unclear. We demonstrate proof-of-concept combining a state-of-the-art vision language model and variants of a prompting strategy that asks the model to consider segmented elements independently of the original image. Experiments on two urban features - stop lines and raised tables - show that while direct zero-shot prompting correctly annotates nearly zero images, the pre-segmentation strategies can annotate images with near 40% intersection-over-union accuracy. We describe how these results inform a new research agenda in automatic annotation of the built environment to improve equity, accessibility, and safety at broad scale and in diverse environments.

## CCS CONCEPTS

· Computing methodologies → Scene understanding Visual ; content-based indexing and retrieval ; Visual inspection ; Image segmentation ; Interest point and salient region detections .

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org.

Conference'17, July 2017, Washington, DC, USA

ACM ISBN 978-1-4503-XXXX-X/18/06

© 2018 Copyright held by the owner/author(s). Publication rights licensed to ACM.

https://doi.org/XXXXXXX.XXXXXXX

## Bill Howe

billhowe@uw.edu University of Washington Seattle, USA

## KEYWORDS

Large Language Model, Multi-Modality Model, Image Segmentation, Urban Computing, Urban Data Annotation

## ACMReference Format:

Bin Han, Yiwei Yang, Anat Caspi, and Bill Howe. 2018. Towards Zero-Shot Annotation of the Built Environment with Vision-Language Models (Vision Paper). In . ACM, New York, NY, USA, 4 pages. https://doi.org/XXXXXXX. XXXXXXX

## 1 INTRODUCTION

The inaccessible urban infrastructure reinforces systemic exclusion of people with disabilities and negatively impacts public health and overall quality of life for everyone [4]. As laws evolve to make the urban environment more accessible, the digital representation must also evolve to include not just roads and sidewalks, but the hundreds of features that enable safe and equitable mobility: curb ramps, walk signals, tactile paving, crosswalks, light poles, benches, and many more [10]. For example, mobility applications that provide accessible pedestrian routes rely on accurate, comprehensive annotations of the dynamic urban infrastructure [2].

Pre-trained vision-language models (VLMs) (e.g., [9, 11]), exhibit impressive multi-modal scene understanding that can be used, among many other tasks, to identify objects in images [12]. If these models could be used to bootstrap feature annotation in the built environment, a more complete digital record could be developed enabling equity and accessibility at scale [7]. However, these models are trained on text paired with images from the internet, such

The gold standard for data collection involves in-person inspections, typically administered by local governments with the help of community volunteers or contractors [5]. Human annotations are high quality but are labor-intensive, costly, and difficult to scale. For instance, in 2017, the city of Seattle employed 14 people for their first-ever audit of 2,300 miles of sidewalks, costing $400,000 and identifying 92,000 uplifts, 38,000 surface problems, and 20,000 obstructions[5]. Sensing platforms deployed to navigate the built environment can partially automate annotation via machine learning, but these initiatives are also expensive to scale [1]. Aerial imagery offers nearly complete coverage at low cost, as long as the computer vision models to extract the relevant features can be trained. In practice, model development tends to focus on the needs of the "typical" traveler, e.g., roads [3] and in a few cases sidewalks [6, 16]. Expanding coverage to the long tail of accessibility features is rarely viable due to the cost in obtaining training data.

Figure 1: Pipeline of our proposed automated annotation process. Users input a pair of (satellite image, annotation guidance). The image will go through a set of processes including segmentation, filtering, and set-of-mark generation. Then the image and guidance will go through a vision-language model, the output of which is post-processed to produce the final annotation results. The procedure requires no fine-tuning, and can be applied on different features with minimal adjust on the guidance.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00932v1-figure-6d177b33-24f5-4d95-85db-b9ec1834594c.jpg)

that common objects in everyday (US-centric) experiences tend to be dominant. For atypical scenes and images, including satellite images, performance remains to be tested.

- · We describe a prompting procedure for generating annotations urban features from aerial imagery using a VLM, adapting segmentation based annotations methods to this context.
- · Given these results, we discuss the current challenges faced by the automated process and describe how a complete, accurate, and publicly available data record may now be feasible and consider next steps for the research community to realize this vision.
- · We evaluate the procedure on two urban features, finding that while simple methods fail, the proposed methods are promising and viable, which suggests an important avenue for research.

## 2 RELATED WORK

Urban Data Collection: [16] introduced a novel dataset comprising aerial satellite imagery, street maps, and rasterized annotations of sidewalks, crossings, and corner bulbs in cities. Even after years of work, existing models recognize only the three most important features of the pedestrian network. These models require expansive, diverse training data to perform well; our focus is on bootstrapping a process for collecting this training data for rare cases.

In this vision paper, we consider using VLMs with aerial imagery as a source of low-cost, extensible annotations for arbitrary features in the built environment. By reducing the cost of producing annotations to near zero, mobility applications can keep pace with an evolving building code and commensurate new construction, faithfully representing the complete and current built environment. With this capability, we envision a larger roadmap to scale this annotation process to any feature in any region, dramatically lowering costs for data acquisition and therefore enable a broad class of community-focused, AI-based mobility applications that have previously been too specialized to justify the cost. The ability to make services for specialized needs as economically viable as services for typical needs is the foundation of equity.

We use the a state-of-the-art VLM (GPT-4o 1 ) and a generalpurpose segmentation model (SAM [8]). The SAM model segments a satellite image into a gallery of candidate objects, from which the GPT-4o model can select the appropriate feature of interest (see Figure 1 for details). We test our procedure on two urban features: stop lines (white lines indicating where cars should stop at an intersection, which are important for safely deriving pedestrian routes through unmarked crossings) and raised tables (speed control features that can be used to estimate traffic flow and therefore pedestrian risk.) We consider multiple variants of this procedure, showing that presenting the segmented "options" to the model in different ways can improve performance. Quantitatively, we show that a simple direct prompting strategy fails dramatically at this task, while the proposed procedures achieve on average about 40% intersection-over-union accuracy. The takeaway is that this problem is difficult, but not impossible. Qualitatively, we visualize several correct and incorrect annotations to inform a discussion of the challenges and vision. Overall, our contributions are:

To demonstrate the proof-of-concept, we developed a procedure borrowing techniques from visual grounding [15, 17], visual prompting [13], and set-of-mark prompting [14] that generates bounding box annotations on satellite images based only on highlevel descriptions of features. The procedure requires no fine-tuning or even labeled examples, demonstrating that closed, general purpose models are potentially viable even in a zero-shot setting.

1 https://platform.openai.com/docs/models/gpt-4o

Visual Grounding &amp; Visual Prompting: Our proposed system is highly relevant to visual grounding task and visual prompting technique. Visual grounding task in computer vision aims to locate the most relevant object or region in an image based on the specific task description in natural language [15, 17]. Visual prompting is a technique that uses visual cues or examples to guide multi-modal AI models in understanding and responding to visual tasks, similar to how textual prompts are used to instruct LLMs [13]. Our proposed procedure is based on Set-of-Mark (SoM) prompting [14], leveraging off-the-shelf segmentation models to partition images into smaller regions, and overlays each region with a mark to provide visual hints for the model. Their work demonstrated extraordinary visual grounding ability of GPT-4.

## 3 PROMPTING PROCEDURE

Our proposed procedure is shown in Figure 1. It works as follows:

- · User Input: a pair of (satellite image, annotation guidance).
- · Candidate Filtering: We apply heuristic filters to eliminate irrelevant components and narrow down the candidate space. In our experiments, we employ color and area size filters, excluding green, yellow, brown, and red elements, as well as components
- · Image Segmentation: The image is segmented into a set of objects using a general-purpose model, ignoring the prompt.

Table 1: Text prompts for the VLM model to annotate different features of interest. Pink highlighted texts represent the descriptive information of the features. Blue highlighted texts represent the general task questions.

| Object       | DP                                                                                                                                                                                                                                                    | SoM-(NC,IC,Comb)                                                                                                                                                                              |
|--------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Stop Line    | A stop line is a single white line painted on the road at intersections where traffic must stop. It shows drivers where to halt their vehicles. Please identify the bounding box of the stop line in the image in the format of (xtl, ytl, xbr, ybr). | A stop line is a single white line painted on the road at intersections where traffic must stop. It shows drivers where to halt their vehicles. Which labeled images represent the stop line? |
| Raised Table | A raised table usually covers the entire width of the crosswalk. It is typically painted with triangular arrows in white color. Please identify the bounding box of the stop line in the image in the format of (xtl, ytl, xbr, ybr).                 | A raised table usually covers the entire width of the crosswalk. It is typically painted with triangular arrows in white color. Which labeled images represent the raised table?              |

smaller than specific sizes 2 . In practice, different parameter values and additional filter criteria can be applied based on the specific urban features and expert knowledge.

either the instructions or the image, which is consistent with prior work [14].2) Adding visual marks significantly improves performance, achieving 24%-40% IoU depending on the variant. In-context improves over no-context by 47.3% and 26.4% on two features respectively, and providing both forms yields an additional slight improvement.

- - No-Context (NC) : the candidates are extracted and presented as distinct, unrelated objects embedded in a plain white background, with each component labeled with a number.
- · Set-of-Mark (SoM) Generation: After filtering, we generate an identifying mark for each component [14] to enhance the vision-language model's recognition of each object. We test three different strategies as shown in Figure 2:
- - In-Context (IC) : we label the candidates with numbers and bounding boxes within the original image.
- · VLM &amp; Post-Processing: the image from the SoM generation process is sent to the VLM with the textual annotation guidance. The text prompt consists of 1) a description of the feature of interest (e.g., stop line, raised table) and what specific features it possesses and 2) the instructions for annotating the image (see Table 1). The output of the model is post-processed: the model responds with the mark indicating the selected candidate and we return the corresponding bounding box.
- - Combination (Comb) : We generate both no- and in-context images and provide both to the VLM.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00932v1-figure-5c6d3889-fa7e-4882-8b9d-3d591a700f96.jpg)

Figure 2: SoM generation scenarios: (a) Filtered candidates (b) No-Context: Candidate objects are presented separately in a new image. (c) In-Context: Candidate objects are labeled with numbers and bounding boxes within the original image.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00932v1-figure-ebd2a4eb-e85c-432e-ad82-0f9d5220ce56.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00932v1-figure-7a34feee-5fe1-4814-826f-6b1f05b6ab5e.jpg)

## 4 EVALUATION &amp; RESULTS

To evaluate this procedure, we manually cropped and annotated eight satellite images that contains stop lines and ten images that contain raised tables from Google Earth. As a baseline, we compare against a Direct Prompting (DP) method: given an image, we directly ask GPT-4o (without SAM) to annotate the feature of interest and return the bounding boxes.

Quantitatively, we evaluate the annotations using Intersectionover-Union (IoU) metric: 𝐼𝑜𝑈 = 𝐺 ∩ 𝑃 𝐺 ∪ 𝑃 , where 𝐺 is the ground truth annotation and 𝑃 is the annotation from our procedure. The results are presented in Table 2. We observe that 1) Direct prompting is completely ineffective, achieving essentially zero overlap with ground truth. The annotations reflect very little understanding of

2 200 pixel area for stop lines and 400 pixel area for raised tables.

Qualitatively, we visualize a few annotated examples to demonstrate the feasibility of the automated process. We also mark some flawed examples to be discussed in the next section §5. From the visualizations, we observe 1) the failure of direct prompting, indicating the problem is difficult and not amenable to trivial solutions; 2) given visual marks, the VLM model can always at least partially identify (highlighted with yellow boxes), and occasionally fully identify the features (highlighted with green boxes), suggesting the problem is feasible for other types of features; and 3) among the three annotation scenarios, there is no consistent winner, motivating further research.

Table 2: Quantitative evaluation using IoU( ➡ ) metric.

| Feature      |    DP |   SoM-NC |   SoM-IC |   SoM-Comb |
|--------------|-------|----------|----------|------------|
| Stop Line    | 0     |   0.2483 |   0.3354 |     0.3657 |
| Raised Table | 0.019 |   0.3315 |   0.4069 |     0.4189 |

## 5 CHALLENGES &amp; FUTURE DIRECTION

Inspecting or manually annotating street objects is labor-intensive and time-consuming. Our proposed procedure reduces the annotation process to a) describing a typical representation of the feature in question, and potentially 2) providing simple heuristics for filtering candidates based on color, size, other basic image properties. We evaluate in the zero-shot setting to reduce the dependency on finding representative examples - a user can directly apply domain knowledge to generate training data from raw satellite images. This procedure can produce many thousands of annotated, not all of which will be correct. In ongoing work, we are using these bootstrapped images to train more specialized vision models; we anticipate that the downstream model will tend to be robust to incorrect annotations, as long as most are roughly correct. This procedure can be completed in a few minutes, allowing users to rapidly produce annotated images for any feature they can describe.

Our experiments suggest a solution is feasible but expose several challenges and motivate directions for further work:

- · Noisy/Missing Segmentation . The annotation results heavily depend on the quality of the segmented components. 1) Since SAMisageneral-purposed segmentation model, it often produces irrelevant segments. Currently, we prune the space of candidates heuristically to improve performance, but these heuristics may need to be developed for each feature individually, and it is unclear how to perform this task in a principled way. 2) SAM might

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00932v1-figure-1403deb2-615c-49bf-bcbb-4d9f854c52ea.jpg)

Figure 3: Left - Examples of annotated stop lines. Right - Examples of annotated raised tables. Red regions in each image are the segmented objects. Green and yellow outlines indicate perfect and approximate annotations, respectively. A Red outline indicate inaccurate annotations.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00932v1-figure-ad688fb4-9598-4425-ad1f-ef512f3fe860.jpg)

miss part of or the entire feature. For example, part of the raised table is covered by the shadow, thus not being detected To address both issues, we recommend studyingsegmentation models specialized for satellite images that can, e.g., ignore shadows.

- [5] Jon E Froehlich, Yochai Eisenberg, Maryam Hosseini, Fabio Miranda, Marc Adams, Anat Caspi, Holger Dieterich, Heather Feldner, Aldo Gonzalez, Claudina De Gyves, et al. 2022. The Future of Urban Accessibility for people with disabilities: Data collection, analytics, policy, and Tools. In Proceedings of the 24th International ACM SIGACCESS Conference on Computers and Accessibility . , , 1-8.
- · Capability of Vision-Language Models . The rapid improvements in VLMs mediate our results - Anecdotally, the recently released GPT-4o dramatically outperformed the state of the art open VLM Llava2 and yet still makes mistakes suggesting limited comprehension of the built environment. For example, crossing lines oriented parallel to the roadway are mistaken for stop lines (Figure 3, example 5, SoM-NC, yellow box) or misunderstand the task entirely (Figure 3, example 1, SoM-NC, red box). VLMs finetuned on satellite images may avoid these errors.
- · City-wise Generalizability . Since the process relies on two general-purpose large AI models, it can potnetially generalize to any feature in any location. However, this generalization remains untested. Some features of interest will be significantly harder to recognize: potholes have irregular shapes and sizes; benches will be more often shaded or occluded. Further, laws and standards vary across cities, changing the visible environment.
- · High Variations . Even with a zero temperature value (which makes outputs from the VLM more deterministic), GPT-4o can produce varying results for the same image and prompt inputs over time. This variability leads to inconsistent annotations, requiring multiple rounds to achieve reliable outcomes.

## REFERENCES

- [1] Andrew R Benson, Ian GM Lawson, Matthew K Clifford, and Sean M McBride. 2021. Using robotics to detect footpath displacement caused by tree roots: A proof of concept. Urban Forestry &amp; Urban Greening 65 (2021), 127312.
- [3] Luca Caltagirone, Mauro Bellone, Lennart Svensson, and Mattias Wahde. 2019. LIDAR-camera fusion for road detection using fully convolutional neural networks. Robotics and Autonomous Systems 111 (2019), 125-131.
- [2] Nicholas Bolten and Anat Caspi. 2019. AccessMap Website Demonstration: Individualized, Accessible Pedestrian Trip Planning at Scale. 676-678. https: //doi.org/10.1145/3308561.3354598
- [4] Yochai Eisenberg, Kerri A Vanderbom, and Vijay Vasudevan. 2017. Does the built environment moderate the relationship between having a disability and lower levels of physical activity? A systematic review. Preventive medicine 95 (2017), S75-S84.
- [7] Bill Howe, Jackson Maxfield Brown, Bin Han, Bernease Herman, Nic Weber, An Yan, Sean Yang, and Yiwei Yang. 2022. Integrative urban AI to expand coverage, access, and equity of urban data. The European Physical Journal Special Topics 231, 9 (2022), 1741-1752.
- [6] Maryam Hosseini, Andres Sevtsuk, Fabio Miranda, Roberto M Cesar Jr, and Claudio T Silva. 2023. Mapping the walk: A scalable computer vision approach for generating sidewalk network datasets from aerial imagery. Computers, Environment and Urban Systems 101 (2023), 101950.
- [8] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. 2023. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision . , , 4015-4026.
- [10] CHRISTINE MENDOZA and ANAT CASPI. 2023. Development and Taxonomization of an Accessibility-First Open Data GIS Standard. UniversityofNorthCarolina at Chapel Hill and University of Washington. https://christineiym.github.io/dreu2023/files/finalreport.pdf
- [9] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2024. Visual instruction tuning. Advances in neural information processing systems 36 (2024), .
- [11] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learning . PMLR, , , 8748-8763.
- [13] Aleksandar Shtedritski, Christian Rupprecht, and Andrea Vedaldi. 2023. What does clip know about a red circle? visual prompt engineering for vlms. In Proceedings of the IEEE/CVF International Conference on Computer Vision . , , 11987-11997.
- [12] Aleksandar Shtedritski, Christian Rupprecht, and Andrea Vedaldi. 2023. What does clip know about a red circle? visual prompt engineering for vlms. In Proceedings of the IEEE/CVF International Conference on Computer Vision . , , 11987-11997.
- [14] Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. 2023. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v. arXiv preprint arXiv:2310.11441 , (2023), .
- [16] Yuxiang Zhang, Nicholas Bolten, Sachin Mehta, and Anat Caspi. 2023. APE: An Open and Shared Annotated Dataset for Learning Urban Pedestrian Path Networks. arXiv preprint arXiv:2303.02323 , (2023), .
- [15] Li Yang, Yan Xu, Chunfeng Yuan, Wei Liu, Bing Li, and Weiming Hu. 2022. Improving visual grounding with visual-linguistic verification and iterative reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . , , 9499-9508.
- [17] Chaoyang Zhu, Yiyi Zhou, Yunhang Shen, Gen Luo, Xingjia Pan, Mingbao Lin, Chao Chen, Liujuan Cao, Xiaoshuai Sun, and Rongrong Ji. 2022. Seqtr: A simple yet universal network for visual grounding. In European Conference on Computer Vision . Springer, , , 598-615.
Bin Han Yiwei Yang Anat Caspi Bill Howe
Towards Zero-Shot Annotation of the Built Environment with Vision-Language Models (Vision Paper)
Equitable urban transportation applications require high-fidelity digital
representations of the built environment: not just streets and sidewalks, but
bike lanes, marked and unmarked crossings, curb ramps and cuts, obstructions,
traffic signals, signage, street markings, potholes, and more. Direct
inspections and manual annotations are prohibitively expensive at scale.
Conventional machine learning methods require substantial annotated training
data for adequate performance. In this paper, we consider vision language
models as a mechanism for annotating diverse urban features from satellite
images, reducing the dependence on human annotation to produce large training
sets. While these models have achieved impressive results in describing common
objects in images captured from a human perspective, their training sets are
less likely to include strong signals for esoteric features in the built
environment, and their performance in these settings is therefore unclear. We
demonstrate proof-of-concept combining a state-of-the-art vision language model
and variants of a prompting strategy that asks the model to consider segmented
elements independently of the original image. Experiments on two urban features
-- stop lines and raised tables -- show that while direct zero-shot prompting
correctly annotates nearly zero images, the pre-segmentation strategies can
annotate images with near 40% intersection-over-union accuracy. We describe how
these results inform a new research agenda in automatic annotation of the built
environment to improve equity, accessibility, and safety at broad scale and in
diverse environments.