text:
Semantic Mining Technologies for Multimedia Databases (Premier Reference Source)
Semantic Mining Technologies for Multimedia Databases Dacheng Tao Nanyang Technological University, Singapore Dong Xu N...
Author: Dacheng Tao
46 downloads 466 Views 16MB Size Report
Semantic Mining Technologies for Multimedia Databases Dacheng Tao Nanyang Technological University, Singapore Dong Xu Nanyang Technological University, Singapore Xuelong Li University of London, UK
Information science reference Hershey • New York
Director of Editorial Content: Senior Managing Editor: Managing Editor: Assistant Managing Editor: Typesetter: Cover Design: Printed at:
Kristin Klinger Jamie Snavely Jeff Ash Carole Coulson Amanda Appicello Lisa Tosheff Yurchak Printing Inc.
Published in the United States of America by Information Science Reference (an imprint of IGI Global) 701 E. Chocolate Avenue, Suite 200 Hershey PA 17033 Tel: 717-533-8845 Fax: 717-533-8661 E-mail: [email protected] Web site: http://www.igi-global.com/reference and in the United Kingdom by Information Science Reference (an imprint of IGI Global) 3 Henrietta Street Covent Garden London WC2E 8LU Tel: 44 20 7240 0856 Fax: 44 20 7379 0609 Web site: http://www.eurospanbookstore.com Copyright © 2009 by IGI Global. All rights reserved. No part of this publication may be reproduced, stored or distributed in any form or by any means, electronic or mechanical, including photocopying, without written permission from the publisher. Product or company names used in this set are for identi.cation purposes only. Inclusion of the names of the products or companies does not indicate a claim of ownership by IGI Global of the trademark or registered trademark. Library of Congress Cataloging-in-Publication Data Semantic mining technologies for multimedia databases / Dacheng Tao, Dong Xu, and Xuelong Li, editors. p. cm. Includes bibliographical references and index. Summary: "This book provides an introduction to the most recent techniques in multimedia semantic mining necessary to researchers new to the field"--Provided by publisher. ISBN 978-1-60566-188-9 (hardcover) -- ISBN 978-1-60566-189-6 (ebook) 1. Multimedia systems. 2. Semantic Web. 3. Data mining. 4. Database management. I. Tao, Dacheng, 1978- II. Xu, Dong, 1979- III.Li, Xuelong, 1976QA76.575.S4495 2009 006.7--dc22 2008052436
British Cataloguing in Publication Data A Cataloguing in Publication record for this book is available from the British Library. All work contributed to this book is new, previously-unpublished material. The views expressed in this book are those of the authors, but not necessarily of the publisher.
Preface . ................................................................................................................................................ xv Section I Multimedia Information Representation Chapter I Video Representation and Processing for Multimedia Data Mining ...................................................... 1 Amr Ahmed, University of Lincoln, UK Chapter II Image Features from Morphological Scale-Spaces............................................................................... 32 Sébastien Lefèvre, University of Strasbourg – CNRS, France Chapter III Face Recognition and Semantic Features ............................................................................................. 80 Huiyu Zhou, Brunel University, UK Yuan Yuan, Aston University, UK Chunmei Shi, People's Hospital of Guangxi, China Section II Learning in Multimedia Information Organization Chapter IV Shape Matching for Foliage Database Retrieval ................................................................................ 100 Haibin Ling, Temple University, USA David W. Jacobs, University of Maryland, USA Chapter V Similarity Learning for Motion Estimation......................................................................................... 130 Shaohua Kevin Zhou, Siemens Corporate Research Inc., USA Jie Shao, Google Inc., USA Bogdan Georgescu, Siemens Corporate Research Inc., USA Dorin Comaniciu, Siemens Corporate Research Inc., USA
Chapter VI Active Learning for Relevance Feedback in Image Retrieval............................................................. 152 Jian Cheng, National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, China Kongqiao Wang, Nokia Research Center, Beijing, China Hanqing Lu, National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, China Chapter VII Visual Data Mining Based on Partial Similarity Concepts.................................................................. 166 Juliusz L. Kulikowski, Polish Academy of Sciences, Poland Section III Semantic Analysis Chapter VIII Image/Video Semantic Analysis by Semi-Supervised Learning......................................................... 183 Jinhui Tang, National University of Singapore, Singapore Xian-Sheng Hua, Microsoft Research Asia, China Meng Wang, Microsoft Research Asia, China Chapter IX Content-Based Video Semantic Analysis............................................................................................. 211 Shuqiang Jiang, Chinese Academy of Sciences, China Yonghong Tian, Peking University, China Qingming Huang, Graduate University of Chinese Academy of Sciences, China Tiejun Huang, Peking University, China Wen Gao, Peking University, China Chapter X Applications of Semantic Mining on Biological Process Engineering................................................ 236 Hossam A. Gabbar, University of Ontario Institute of Technology, Canada Naila Mahmut, Heart Center - Cardiovascular Research Hospital for Sick Children, Canada Chapter XI Intuitive Image Database Navigation by Hue-Sphere Browsing......................................................... 263 Gerald Schaefer, Aston University, UK Simon Ruszala, Teleca, UK
Section IV Multimedia Resource Annotation Chapter XII Formal Models and Hybrid Approaches for Ef.cient Manual Image Annotation and Retrieval ....... 272 Rong Yan, IBM T.J. Watson Research Center, USA Apostol Natsev, IBM T.J. Watson Research Center, USA Murray Campbell, IBM T.J. Watson Research Center, USA Chapter XIII Active Video Annotation: To Minimize Human Effort........................................................................ 298 Meng Wang, Microsoft Research Asia, China Xian-Sheng Hua, Microsoft Research Asia, China Jinhui Tang, National University of Singapore, Singapore Guo-Jun Qi, University of Science and Technology of China, China Chapter XIV Annotating Images by Mining Image Search...................................................................................... 323 Xin-Jing Wang, Microsoft Research Asia, China Lei Zhang, Microsoft Research Asia, China Xirong Li, Microsoft Research Asia, China Wei-Ying Ma, Microsoft Research Asia, China Chapter XV Semantic Classification and Annotation of Images............................................................................. 350 Yonghong Tian, Peking University, China Shuqiang Jiang, Chinese Academy of Sciences, China Tiejun Huang, Peking University, China Wen Gao, Peking University, China Section V Other Topics Related to Semantic Mining Chapter XVI Association-Based Image Retrieval..................................................................................................... 379 Arun Kulkarni, The University of Texas at Tyler, USA Leonard Brown, The University of Texas at Tyler, USA Chapter XVII Compressed-Domain Image Retrieval Based on Colour Visual Patterns ........................................... 407 Gerald Schaefer, Aston University, UK
Chapter XVIII Resource Discovery Using Mobile Agents ......................................................................................... 419 M. Singh, Middlesex University, UK X. Cheng, Middlesex University, UK & Beijing Normal University, China X. He, Reading University, UK Chapter XIX Multimedia Data Indexing .................................................................................................................. 449 Zhu Li, Hong Kong Polytechnic University, Hong Kong Yun Fu, BBN Technologies, USA Junsong Yuan, Northwestern University, USA Ying Wu, Northwestern University, USA Aggelos Katsaggelos, Northwestern University, USA Thomas S. Huang, University of Illinois at Urbana-Champaign, USA Compilation of References................................................................................................................ 476 About the Contributors..................................................................................................................... 514 Index.................................................................................................................................................... 523
Detailed Table of Contents
Preface . ................................................................................................................................................ xv Section I Multimedia Information Representation Chapter I Video Representation and Processing for Multimedia Data Mining ...................................................... 1 Amr Ahmed, University of Lincoln, UK Video processing and segmentation are important stages for multimedia data mining, especially with the advance and diversity of video data available. The aim of this chapter is to introduce researchers, especially new ones, to the "video representation, processing, and segmentation techniques". This includes an easy and smooth introduction, followed by principles of video structure and representation, and then a state-of-the-art of the segmentation techniques focusing on the shot-detection. Performance evaluation and common issues are also discussed before concluding the chapter. Chapter II Image Features from Morphological Scale-Spaces............................................................................... 32 Sébastien Lefèvre, University of Strasbourg – CNRS, France Multimedia data mining is a critical problem due to the huge amount of data available. Efficient and reliable data mining solutions requires both appropriate features to be extracted from the data and relevant techniques to cluster and index the data. In this chapter, the authors deal with the first problem which is feature extraction for image representation. A wide range of features has been introduced in the literature, and some attempts have been made to build standards (e.g. MPEG-7). These features are extracted with image processing techniques, and the authors focus here on a particular image processing toolbox, namely the mathematical morphology, which stays rather unknown from the multimedia mining community, even if it offers some very interesting feature extraction methods. They review here these morphological features, from the basic ones (granulometry or pattern spectrum, differential morphological profile) to more complex ones which manage to gather complementary information.
Chapter III Face Recognition and Semantic Features ............................................................................................. 80 Huiyu Zhou, Brunel University, UK Yuan Yuan, Aston University, UK Chunmei Shi, People's Hospital of Guangxi, China The authors present a face recognition scheme based on semantic features' extraction from faces and tensor subspace analysis. These semantic features consist of eyes and mouth, plus the region outlined by three weight centres of the edges of these features. The extracted features are compared over images in tensor subspace domain. Singular value decomposition is used to solve the eigenvalue problem and to project the geometrical properties to the face manifold. They also compare the performance of the proposed scheme with that of other established techniques, where the results demonstrate the superiority of the proposed method. Section II Learning in Multimedia Information Organization Chapter IV Shape Matching for Foliage Database Retrieval ................................................................................ 100 Haibin Ling, Temple University, USA David W. Jacobs, University of Maryland, USA Computer-aided foliage image retrieval systems have the potential to dramatically speed up the process of plant species identification. Despite previous research, this problem remains challenging due to the large intra-class variability and inter-class similarity of leaves. This is particularly true when a large number of species are involved. In this chapter, the authors present a shape-based approach, the innerdistance shape context, as a robust and reliable solution. They show that this approach naturally captures part structures and is appropriate to the shape of leaves. Furthermore, they show that this approach can be easily extended to include texture information arising from the veins of leaves. They also describe a real electronic field guide system that uses our approach. The effectiveness of the proposed method is demonstrated in experiments on two leaf databases involving more than 100 species and 1000 leaves. Chapter V Similarity Learning for Motion Estimation......................................................................................... 130 Shaohua Kevin Zhou, Siemens Corporate Research Inc., USA Jie Shao, Google Inc., USA Bogdan Georgescu, Siemens Corporate Research Inc., USA Dorin Comaniciu, Siemens Corporate Research Inc., USA Motion estimation necessitates an appropriate choice of similarity function. Because generic similarity functions derived from simple assumptions are insufficient to model complex yet structured appearance variations in motion estimation, the authors propose to learn a discriminative similarity function to match
images under varying appearances by casting image matching into a binary classification problem. They use the LogitBoost algorithm to learn the classifier based on an annotated database that exemplifies the structured appearance variations: An image pair in correspondence is positive and an image pair out of correspondence is negative. To leverage the additional distance structure of negatives, they present a location-sensitive cascade training procedure that bootstraps negatives for later stages of the cascade from the regions closer to the positives, which enables viewing a large number of negatives and steering the training process to yield lower training and test errors. They also apply the learned similarity function to estimating the motion for the endocardial wall of left ventricle in echocardiography and to performing visual tracking. They obtain improved performances when comparing the learned similarity function with conventional ones. Chapter VI Active Learning for Relevance Feedback in Image Retrieval............................................................. 152 Jian Cheng, National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, China Kongqiao Wang, Nokia Research Center, Beijing, China Hanqing Lu, National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, China Relevance feedback is an effective approach to boost the performance of image retrieval. Labeling data is indispensable for relevance feedback, but it is also very tedious and time-consuming. How to alleviate users' burden of labeling has been a crucial problem in relevance feedback. In recent years, active learning approaches have attracted more and more attention, such as query learning, selective sampling, multi-view learning, etc. The well-known examples include Co-training, Co-testing, SVMactive, etc. In this literature, the authors will introduce some representative active learning methods in relevance feedback. Especially they will present a new active learning algorithm based on multi-view learning, named Co-SVM. In Co-SVM algorithm, color and texture are naturally considered as sufficient and uncorrelated views of an image. SVM classifier is learned in color and texture feature subspaces, respectively. Then the two classifiers are used to classify the unlabeled data. These unlabeled samples that disagree in the two classifiers are chose to label. The extensive experiments show that the proposed algorithm is beneficial to image retrieval. Chapter VII Visual Data Mining Based on Partial Similarity Concepts.................................................................. 166 Juliusz L. Kulikowski, Polish Academy of Sciences, Poland Visual data mining is a procedure aimed at a selection from a document's repository subsets of documents presenting certain classes of objects; the last may be characterized as classes of objects' similarity or, more generally, as classes of objects satisfying certain relationships. In this chapter attention will be focused on selection of visual documents representing objects belonging to similarity classes.
Section III Semantic Analysis Chapter VIII Image/Video Semantic Analysis by Semi-Supervised Learning......................................................... 183 Jinhui Tang, National University of Singapore, Singapore Xian-Sheng Hua, Microsoft Research Asia, China Meng Wang, Microsoft Research Asia, China The insufficiency of labeled training samples is a major obstacle in automatic semantic analysis of large scale image/video database. Semi-supervised learning, which attempts to learn from both labeled and unlabeled data, is a promising approach to tackle this problem. As a major family of semi-supervised learning, graph-based methods have attracted more and more recent research. In this chapter, a brief introduction is given on popular semi-supervised learning methods, especially the graph-based methods, as well as their applications in the area of image annotation, video annotation, and image retrieval. It is well known that the pair-wise similarity is an essential factor in graph propagation based semi-supervised learning methods. A novel graph-based semi-supervised learning method, named Structure-Sensitive Anisotropic Manifold Ranking (SSAniMR), is derived from a PDE based anisotropic diffusion framework. Instead of using Euclidean distance only, SSAniMR further takes local structural difference into account to more accurately measure pair-wise similarity. Finally some future directions of using semisupervised learning to analyze the multimedia content are discussed. Chapter IX Content-Based Video Semantic Analysis............................................................................................. 211 Shuqiang Jiang, Chinese Academy of Sciences, China Yonghong Tian, Peking University, China Qingming Huang, Graduate University of Chinese Academy of Sciences, China Tiejun Huang, Peking University, China Wen Gao, Peking University, China With the explosive growth in the amount of video data and rapid advance in computing power, extensive research efforts have been devoted to content-based video analysis. In this chapter, they authors will give a broad discussion on this research area by covering different topics such as video structure analysis, object detection and tracking, event detection, visual attention analysis, etc. In the meantime, different video representation and indexing models are also presented. Chapter X Applications of Semantic Mining on Biological Process Engineering................................................ 236 Hossam A. Gabbar, University of Ontario Institute of Technology, Canada Naila Mahmut, Heart Center - Cardiovascular Research Hospital for Sick Children, Canadaa Semantic mining is an essential part in knowledgebase and decision support systems where it enables the extraction of useful knowledge form available databases with the ultimate goal of supporting the decision making process. In process systems engineering, decisions are made throughout plant / process / product life cycles. The provision of smart semantic mining techniques will improve the decision making
process for all life cycle activities. In particular, safety and environmental related decisions are highly dependent on process internal and external conditions and dynamics with respect to equipment geometry and plant layout. This chapter discusses practical methods for semantic mining using systematic knowledge representation as integrated with process modeling and domain knowledge. POOM or plant/process object oriented modeling methodology is explained and used as a basis to implement semantic mining as applied on process systems engineering. Case studies are illustrated for biological process engineering, in particular MoFlo systems focusing on process safety and operation design support. Chapter XI Intuitive Image Database Navigation by Hue-Sphere Browsing......................................................... 263 Gerald Schaefer, Aston University, UK Simon Ruszala, Teleca, UK Efficient and effective techniques for managing and browsing large image databases are increasingly sought after. This chapter presents a simple yet efficient and effective approach to navigating image datasets. Based on the concept of a globe as visualisation and navigation medium, thumbnails are projected onto the surface of a sphere based on their colour. Navigation is performed by rotating and tilting the globe as well as zooming into an area of interest. Experiments based on a medium size image database demonstrate the usefulness of the presented approach. Section IV Multimedia Resource Annotation Chapter XII Formal Models and Hybrid Approaches for Efficient Manual Image Annotation and Retrieval ....... 272 Rong Yan, IBM T.J. Watson Research Center, USA Apostol Natsev, IBM T.J. Watson Research Center, USA Murray Campbell, IBM T.J. Watson Research Center, USA Although important in practice, manual image annotation and retrieval has rarely been studied by means of formal modeling methods. In this paper, we propose a set of formal models to characterize the annotation times for two commonly-used manual annotation approaches, i.e., tagging and browsing. Based on the complementary properties of these models, we design new hybrid approaches, called frequencybased annotation and learning-based annotation, to improve the efficiency of manual image annotation as well as retrieval. Both our simulation and experimental results show that the proposed algorithms can achieve up to a 50% reduction in annotation time over baseline methods for manual image annotation, and produce significantly better annotation and retrieval results in the same amount of time. Chapter XIII Active Video Annotation: To Minimize Human Effort........................................................................ 298 Meng Wang, Microsoft Research Asia, China Xian-Sheng Hua, Microsoft Research Asia, China Jinhui Tang, National University of Singapore, Singapore Guo-Jun Qi, University of Science and Technology of China, China
This chapter introduces the application of active learning in video annotation. The insufficiency of training data is a major obstacle in learning-based video annotation. Active learning is a promising approach to dealing with this difficulty. It iteratively annotates a selected set of most informative samples, such that the obtained training set is more effective than that gathered randomly. We present a brief review of the typical active learning approaches. We categorize the sample selection strategies in these methods into five criteria, i.e., risk reduction, uncertainty, positivity, density, and diversity. In particular, we introduce the Support Vector Machine (SVM)-based active learning scheme which has been widely applied. Afterwards, we analyze the deficiency of the existing active learning methods for video annotation, i.e., in most of these methods the to-be-annotated concepts are treated equally without preference and only one modality is applied. To address these two issues, we introduce a multi-concept multi-modality active learning scheme. This scheme is able to better explore human labeling effort by considering both the learnabilities of different concepts and the potential of different modalities. Chapter XIV Annotating Images by Mining Image Search...................................................................................... 323 Xin-Jing Wang, Microsoft Research Asia, China Lei Zhang, Microsoft Research Asia, China Xirong Li, Microsoft Research Asia, China Wei-Ying Ma, Microsoft Research Asia, China Although it has been studied for years by computer vision and machine learning communities, image annotation is still far from practical. In this paper, we propose a novel attempt of modeless image annotation, which investigates how effective a data-driven approach can be, and suggest annotating an uncaptioned image by mining its search results. We collected 2.4 million images with their surrounding texts from a few photo forum websites as our database to support this data-driven approach. The entire process contains three steps: 1) the search process to discover visually and semantically similar search results; 2) the mining process to discover salient terms from textual descriptions of the search results; and 3) the annotation rejection process to filter noisy terms yielded by step 2). To ensure real time annotation, two key techniques are leveraged – one is to map the high dimensional image visual features into hash codes, the other is to implement it as a distributed system, of which the search and mining processes are provided as Web services. As a typical result, the entire process finishes in less than 1 second. Since no training dataset is required, our proposed approach enables annotating with unlimited vocabulary, and is highly scalable and robust to outliers. Experimental results on real web images show the effectiveness and efficiency of the proposed algorithm. Chapter XV Semantic Classification and Annotation of Images............................................................................. 350 Yonghong Tian, Peking University, China Shuqiang Jiang, Chinese Academy of Sciences, China Tiejun Huang, Peking University, China Wen Gao, Peking University, China With the rapid growth of image collections, content-based image retrieval (CBIR) has been an active area of research with notable recent progress. However, automatic image retrieval by semantics still remains a challenging problem. In this chapter, we will describe two promising techniques towards semantic
image retrieval  semantic image classification and automatic image annotation. For each technique, four aspects are presented: task definition, image representation, computational models, and evaluation. Finally, we will give a brief discussion of their application in image retrieval. Section V Other Topics Related to Semantic Mining Chapter XVI Association-Based Image Retrieval..................................................................................................... 379 Arun Kulkarni, The University of Texas at Tyler, USA Leonard Brown, The University of Texas at Tyler, USA With advances in computer technology and the World Wide Web there has been an explosion in the amount and complexity of multimedia data that are generated, stored, transmitted, analyzed, and accessed. In order to extract useful information from this huge amount of data, many content-based image retrieval (CBIR) systems have been developed in the last decade. A Typical CBIR system captures image features that represent image properties such as color, texture, or shape of objects in the query image and try to retrieve images from the database with similar features. Recent advances in CBIR systems include relevance feedback based interactive systems. The main advantage of CBIR systems with relevance feedback is that these systems take into account the gap between the high-level concepts and low-level features and subjectivity of human perception of visual content. CBIR systems with relevance feedback are more efficient than conventional CBIR systems; however, these systems depend on human interaction. In this chapter, we describe a new approach for image storage and retrieval called association-based image retrieval (ABIR). We try to mimic human memory. The human brain stores and retrieves images by association. We use a generalized bi-directional associative memory (GBAM) to store associations between feature vectors that represent images stored in the database. Section I introduces the reader to the CBIR system. In Section II, we present architecture for the ABIR system, Section III deals with preprocessing and feature extraction techniques, and Section IV presents various models of GBAM. In Section V, we present case studies. Chapter XVII Compressed-Domain Image Retrieval Based on Colour Visual Patterns ........................................... 407 Gerald Schaefer, Aston University, UK Image retrieval and image compression have been typically pursued separately. Only little research has been done on a synthesis of the two by allowing image retrieval to be performed directly in the compressed domain of images without the need to uncompress them first. In this chapter we show that such compressed domain image retrieval can indeed be done and lead to effective and efficient retrieval performance. We introduce a novel compression algorithm – colour visual pattern image coding (CVPIC) – and present several retrieval algorithms that operate directly on compressed CVPIC data. Our experiments demonstrate that it is not only possible to realise such midstream content access, but also that the presented techniques outperform standard retrieval techniques such as colour histograms and colour correlograms.
Chapter XVIII Resource Discovery Using Mobile Agents ......................................................................................... 419 M. Singh, Middlesex University, UK X. Cheng, Middlesex University, UK & Beijing Normal University, China X. He, Reading University, UK Discovery of the multimedia resources on network is the focus of the many researches in post semantic web. The task of resources discovery can be automated by using agent. This chapter reviews the current most used technologies that facilitate the resource discovery process. The chapter also the presents the case study to present a fully functioning resource discovery system using mobile agents. Chapter XIX Multimedia Data Indexing .................................................................................................................. 449 Zhu Li, Hong Kong Polytechnic University, Hong Kong Yun Fu, BBN Technologies, USA Junsong Yuan, Northwestern University, USA Ying Wu, Northwestern University, USA Aggelos Katsaggelos, Northwestern University, USA Thomas S. Huang, University of Illinois at Urbana-Champaign, USA The rapid advances in multimedia capture, storage and communication technologies and capabilities have ushered an era of unprecedented growth of digital media content, in audio, visual, and synthetic forms, and both personal and commercially produced. How to manage these data to make them more accessible and searchable to users is a key challenge in current multimedia computing research. In this chapter, we discuss the problems and challenges in multimedia data management, and review the state of the art in data structures and algorithms for multimedia indexing, media feature space management and organization, and applications of these techniques in multimedia data management. Compilation of References................................................................................................................ 476 About the Contributors..................................................................................................................... 514 Index.................................................................................................................................................... 523
With the explosive growth of multimedia databases in terms of both size and variety, effective and efficient indexing and searching techniques for large-scale multimedia databases have become an urgent research topic in recent years. For data organization, the conventional approach is based on keywords or text description of a multimedia datum. However, it is tedious to give all data text annotation and it is almost impossible for people to capture as well. Moreover, the text description is also not enough to precisely describe a multimedia datum. For example, it is unrealistic to utilize words to describe a music clip; an image says more than a thousand words; and keywords-based video shot description cannot characterize the contents for a specific user. Therefore, it is important to utilize the content based approaches (CbA) to mine the semantic information of a multimedia datum. In the last ten years, we have witnessed very significant contributions of CbA in semantics targeting for multimedia data organization. CbA means that the data organization, including retrieval and indexing, utilizes the contents of the data themselves, rather than keywords provided by human. Therefore, the contents of a datum could be obtained from techniques in statistics, computer vision, and signal processing. For example, Markov random fields could be applied for image modeling; spatial-temporal analysis is important for video representation; and the Mel frequency cepstral coefficient has been shown to be the most effective method for audio signal classification. Apart from the conventional approaches mentioned above, machine learning also plays an indispensable role in current semantic mining tasks, for example, random sampling techniques and support vector machine for human computer interaction, manifold learning and subspace methods for data visualization, discriminant analysis for feature selection, and classification trees for data indexing. The goal of this IGI Global book is to provide an introduction about the most recent research and techniques in multimedia semantic mining for new researchers, so that they can go step by step into this field. As a result, they can follow the right way according to their specific applications. The book is also an important reference for researchers in multimedia, a handbook for research students, and a repository for multimedia technologists. The major contributions of this book are in three aspects: (1) collecting and seeking the recent and most important research results in semantic mining for multimedia data organization, (2) guiding new researchers a comprehensive review on the state-of-the-art techniques for different tasks for multimedia database management, and (3) providing technologists and programmers important algorithms for multimedia system construction. This edited book attracted submissions from eight countries including Canada, China, France, Japan, Poland, Singapore, United Kingdom, and United States. Among these submissions, 19 have been accepted. We strongly believe that it is now an ideal time to publish this edited book with the 19 selected
chapters. The contents of this edited book will provide readers with cutting-edge and topical information for their related research. Accepted chapters are solicited to address a wide range of topics in semantic mining from multimedia databases and an overview of the included chapters is given below. This book starts from new multimedia information representations (Video Representation and Processing for Multimedia Data Mining) (Image Features from Morphological Scale-spaces) (Face Recognition and Semantic Features), after which learning in multimedia information organization, an important topic in semantic mining, is studied by four chapters (Shape Matching for Foliage Database Retrieval) (Similarity Learning For Motion Estimation) (Active Learning for Relevance Feedback in Image Retrieval) (Visual Data Mining Based on Partial Similarity Concepts). Thereafter, four schemes are presented for semantic analysis in four chapters (Image/Video Semantic Analysis by Semi-Supervised Learning) (Content-Based Video Semantic Analysis) (Semantic Mining for Green Production Systems) (Intuitive Image Database Navigation by Hue-sphere Browsing). The multimedia resource annotation is also essential for a retrieval system and four chapters provide interesting ideas (Hybrid Tagging and Browsing Approaches for Efficient Manual Image Annotation) (Active Video Annotation: To Minimize Human Effort) (Image Auto-Annotation by Search) (Semantic Classification and Annotation of Images). The last part of this book presents other related topics for semantic mining (Association-Based Image Retrieval) (Compressed-domain Image Retrieval based on Colour Visual Patterns) (Multimedia Resource Discovery using Mobile Agent) (Multimedia Data Indexing). Dacheng Tao Email: [email protected] Nanyang Technological University, Singapore Dong Xu Email: [email protected] Nanyang Technological University, Singapore Xuelong Li Email: [email protected] University of London, UK
Multimedia Information Representation
Video Representation and Processing for Multimedia Data Mining Amr Ahmed University of Lincoln, UK
ABSTRACT Video processing and segmentation are important stages for multimedia data mining, especially with the advance and diversity of video data available. The aim of this chapter is to introduce researchers, especially new ones, to the "video representation, processing, and segmentation techniques". This includes an easy and smooth introduction, followed by principles of video structure and representation, and then a state-of-the-art of the segmentation techniques focusing on the shot-detection. Performance evaluation and common issues are also discussed before concluding the chapter.
I. INTRODUCTION With the advances, which are progressing very fast, in the digital video technologies and the wide availability of more efficient computing resources, we seem to be living in an era of explosion in digital video. Video data are now widely available, and being easily generated, in large volumes. This is not only on the professional level. It can be found everywhere, on the internet, especially with the video uploading and sharing sites, with the personal digital cameras and camcorders, and with the camera mobile phones that became almost the norm. People use the existing easy facilities to generate video data. But at some point, sooner or later, they realize that managing these data can be a bottleneck. This is because the available techniques and tools for accessing, searching, and retrieving video data are not on the same level as for other traditional
Copyright © 2009, IGI Global, distributing in print or electronic forms without written permission of IGI Global is prohibited.
Video Representation and Processing for Multimedia Data Mining
data, such as text. The advances in the video access, search, and retrieval techniques have not been progressing with the same pace as the digital video technologies and its generated data volume. This could be attributed, at least partly, to the nature of the video data and its richness, compared to text data. But it can also be attributed to the increase of our demands. In text, we are no longer just satisfied by searching for exact match of sequence of characters or strings, but need to find similar meanings and other higher level matches. We are also looking forward to do the same on video data. But the nature of the video data is different. Video data is more complex and naturally larger in volume than the traditional text data. They usually combine visual and audio data, as well as textual data. These data need to be appropriately annotated and indexed in an accessible form for search and retrieval techniques to deal with it. This can be achieved based on either textual information, visual and/or audio features, and more importantly on semantic information. The textual-based approach is theoretically the simplest. Video data need to be annotated by textual descriptions, such as keywords or short sentences describing the contents. This converts the search task into the known area of searching in the text data, where the existing relatively advanced tools and techniques can be utilized. The main bottleneck here is the huge time and effort that are needed to accomplish this annotation task, let alone any accuracy issues. The feature-based approach, whether visual and/or audio, depends on annotating the video data by combinations of their extracted low-level features such as intensity, color, texture, shape, motion, and other audio features. This is very useful in doing a query-by-example task. But still not very useful in searching for specific event or more semantic attributes. The semantic-based approach is, in one sense, similar to the textbased approach. Video data need to be annotated, but in this case, with high-level information that represents the semantic meaning of the contents, rather than just describing the contents. The difficulty of this annotation is the high variability of the semantic meaning, of the same video data, among different people, cultures, and ages, to name just a few. It will depend on so many factors, including the purpose of the annotation, the domain and application, cultural and personal views, and could even be subject to the mood and personality of the annotator. Hence, generally automating this task is highly challenging. For specific domains, carefully selected combinations of the visual and/or audio features correlate to useful semantic information. Hence, the efficient extraction of those features is crucial to the high-level analysis and mining of the video data. In this chapter, we focus on the core techniques that facilitate the high-level analysis and mining of the video data. One of the important initial steps in segmentation and analysis of video data is the shot-boundary detection. This is the first step in decomposing the video sequence to its logical structure and components, in preparation for analysis of each component. It is worth mentioning that the subject is enormous and this chapter is meant to be more of an introduction, especially for new researchers. Also, in this chapter, we only focus on the visual modality of the video. Hence, the audio and textual modalities are not covered. After this introductory section, section II provides the principles of video data, so that we know the data that we are dealing with and what does it represent. This includes video structure and representation, both for compressed and uncompressed data. The various types of shot transitions are defined in section III, as well as the various approaches of classifying them. Then, in section IV, the key categories of the shot-boundary detection techniques are discussed. First, the various approaches of categorizing the shot-detection techniques are discussed, along with the various factors contributing to that. Then, a selected hierarchical approach is used to represent the most common techniques. This is followed by discussion of the performance evaluation measures and some common issues. Finally the chapter is summarized and concluded in section V.
II. VidEO STRUCTURE AND REPRESENTATION In this section, it is aimed to introduce, mainly new, researchers to the principles of video data structure and representation. This is an important introduction to understand the data that will be dealt with and what does it represent. This introduction is essential to be able to follow the subsequent sections, especially the shot-transition detection. This section starts by an explanation of the common structure of a video sequence, and a discussion of the various levels in that structure. The logical structure of the video sequence is particularly important for segmentation and data mining.
A. Video Structure The video consists of a number of frames. These frames are usually, and preferably, adjacent to each other on the storage media, but should definitely be played back in the correct order and speed to convey the recorded sequences of actions and/or motion. In fact, each single frame is a still image that consists of pixels, which are the smallest units from the physical point of view. These pixels are dealt with when analyzing the individual frames, and the processing usually utilizes a lot of image processing techniques. However, the aim of most applications of analyzing the video is to identify the basic elements and contents of the video. Hence, logical structure and elements are of more importance than the individual pixels. Video is usually played back with frequencies of 25 or 30 frames per second, as described in more details in section 'B' below. These speeds are chosen so that the human eye do not detect the separation between the frames and to make sure that motion will be smooth and seems to be continuous. Hence, as far as we, the human beings, are concerned, we usually perceive and process the video on a higher level structure. We can easily detect and identify objects, people, and locations within the video. Some objects may change position on the screen from one frame to another, and recording locations could be changing between frames as well. These changes allow us to perceive the motion of objects and people. But more importantly, it allows us to detect higher level aspects such as behaviors and sequences, which we can put together to detect and understand a story or an event that is recorded in the video. According to the above, the video sequence can be logically represented as a hierarchical structure, as depicted in fig. 1 and illustrated in fig. 2. It is worth mentioning that as we go up in the hierarchy, more detailed sub-levels may be added or slightly variations of interpretations may exist. This is mainly depend on the domain in hand. But at least the shot level, as in the definition below, seems to be commonly understood and agreed upon. The definition of each level in the hierarchy is given below, in the reverse order, i.e. bottom-up: • •
Frame: The frame is simply a single still image. It is considered as the smallest logical unit in this hierarchy. It is important in the analysis of the other logical levels. Shot: The shot is a sequence of consecutive frames, temporally adjacent, that has been recorded continuously, within the same session and location, by the same single camera, and without substantial change in the contents of the picture. So, a shot is highly expected to contain a continuous action in both space and time. A shot could be a result of what you continuously record, may be with a camcorder or even a mobile video camera, since you press the record button, until you stop the recording. But off course, if you drop the camera or the mobile, or someone has quickly passed
Figure 1. Hierarchy of the video logical structure Video sequence
. . . . .. … … .
Shot Shot Shot
. . . . .. … … . .. . . . . . . . . . …
Shot Shot
Frames Frames
. . . . .. … … . .. . . . . . . . . . … Frames
in front of your camera, this would probably cause a sudden change in the picture contents. If such change is significant, it may results in breaking the continuity and may then not be considered as a single shot. Scene: The scene is a collection of related shots. Normally, those shots are recorded within the same session, at the same location, but can be recorded from different cameras. An example could be a conversation scene between few people. One camera may be recording the wide location and have all people in the picture, while another camera focuses on the person who is currently talking, and may be another camera is focusing on the audience. Each camera is continuously recording its designated view, but the final output to the viewer is what the director selects from those different views. So, the director can switch between cameras at various times within the conversation based on the flow of the conversation, change of the talking person, reaction from the audience, and so on. Although the finally generated views are usually substantially different, for the viewer, the scene still seems to be logically related, in terms of the location, timing, people and/or objects involved. In fact, we are cleverer than that. In some cases, the conversation can elaborate from one point to another, and the director may inject some images or sub-videos related to the discussion. This introduces huge changes in the pictures shown to the viewer, but still the viewer can follow it up, identify that they are related. This leads to the next level in the logical structure of the video. Segment: The video segment is a group of scenes related to a specific context. It does not have to have been recorded in the same location or in the same time. And off course, it can be recorded with various cameras. However, they are logically related to each other within a specific semantic
Figure 2. Illustration of the video logical structure Video Sequence Segment 2
Segment 1 Scen e1 S h
Scene 2 Shot
Scene 3 S h
S h
context. The various scenes of the same event, or related events, within the news broadcast are an example of a video segment. Sequence: A video sequence consists of a number of video segments. They are usually expected to be related or share some context or semantic aspects. But in reality it may not always be the case.
Depending on the application and its domain, the analysis of the video to extract its logical components can fall into any of the above levels. However, it is worth mentioning that the definitions may slightly differ with the domain, as they tend to be based on the semantic, especially towards the top levels (i.e. Scenes and segments in particular). However, a more common and highly potential starting point is the video shot. Although classified as a part of the logical structure, it also has a tight link with the physical recording action. As it is a result of a continuous recording from a single camera within the same location and time, the shot usually has the same definition in almost all applications and domains. Hence, the shot is a high candidate starting point for extracting the structure and components of the video data. In order to correctly extract shots, we will need to detect their boundaries, lengths, and types. To do so, we need to be aware of how shots are usually joined together in the first place. This is discussed in details in section III and the various techniques of shot detection are reviewed in section IV.
B . Video Representation In this sub-section we discuss the video representation for both compressed and uncompressed data. We first explore the additional dimensionality of video data and frame-rates with their associated redundancy in uncompressed data. Then, we discuss the compressed data representation and the techniques of reducing the various types of redundancies, and how that can be utilized for shot-detection.
1) Uncompressed Video Data The video sequence contains groups of successive frames. They are designed so that when they are played back, the human eye perceives continuous motion of objects within the video and no flickers are recognized due to the change from one frame to another. The film industry uses a frame-rate of 24 frames/sec for films. But the most two common TV standard formats are PAL and NTSC. The framerate in those two standards is either 25 frames/sec, for PAL TV standard, or 30 frames/sec for the NTSC TV standard. In case of the videos that are converted from films, some care need to be taken, especially due to the different frame-rates involved in the different standards. A machine, called telecine, is usually used in that conversion that involves the 2:2 pulldown or 3:2 pulldown process for PAL or NTSC respectively. Like the still image, each pixel within the frame has a value or more, such as intensity or colors. But video data has an extra dimension, in addition to the spatial dimensions of the still images, which is the temporal dimension. The changes between various frames in the video can be exhibited in any of the above attributes, pixel values, spatial and/or temporal. And the segmentation of video, as discussed in later sections, is based on detecting the changes in one or more of the above attributes, or their statistical properties and/or evolution. The video data also carry motion information. Motion information are among the useful information that can be used for segmenting video, as it give an indication of the level of activities and its dynamics within the video. Activity levels can change between the different parts of the video and in fact can characterize its parts. Unlike the individual image properties and pixel values, the motion information are embedded within the video data. Techniques that are based on motion information, as discussed in section V, have to extract them first. This is usually a computationally expensive task. From the discussion above, it can be noticed that in many cases, the video data will usually contain redundancy. For example, some scenes can be almost stationary where there are almost no movements or changes happening. In such cases, the 25 frames produced every second, assuming using the PAL standard, will be very similar, which is a redundancy. This example represents a redundancy in the temporal information, which is between the frames. Similarly, if large regions of the image have the same attributes, this represents a redundancy in the spatial information, which is within the same image. Both types of redundancy can be dealt with or reduced as described in the next subsections, with the compressed video data.
2) Compressed Video Data Video compression aims to reduce the redundancy exist in video data, with minimum visual effect on the video. This is useful in multimedia storage and transmission, among others. The compression can be applied on one or more of the video dimensions; spatial and/or temporal. Each of them is described, with focus on the MPEG standards, as follows: Spatial-Coding (Intra-Coding) The compression in the spatial dimension deals with reducing the redundancy within the same image or frame. There is no need to refer to any other frame. Hence, it is also called intra-coding, and the term "intra" here means "within the same image or frame". The Discrete Cosine Transform (DCT)
is widely used for intra-coding in the most common standards such as JPEG, MPEG-1, MPEG-2, and MPEG-4, although the wavelet transform is also incorporated in MPEG-4. One benefit of the intra-coding is that, as each individual frame is compressed independently, any editing of individual frames or re-ordering of the frames on the time axis can be done without affecting any other frames in the compressed video sequence. Temporal-Coding (Inter-Coding) The compression in the temporal dimension aims at reducing the redundancy between successive frames. This type of compression is also known as the inter-coding, and the term "inter" here means "across or between frames". Hence, the coding of each frame is related to its neighboring ones, on the time axis. In fact, it is mainly based on the differences from neighboring frames. This makes editing, inserting, or deleting individual frames not a straight-forward task, as the neighboring frames will be affected, if not also affecting the process. As it deals with changes between frames, motion plays a dominant factor. A simple translation of an object between two frames, without any other changes such as deformation, would result in unnecessarily large differences. This is because of the noticeable differences in pixel values in the corresponding positions. These differences will require more bits to be coded, which increases the bit-rate. To achieve better compression rates, one of the factors is to reduce the bit-rate. This is done through motion compensation techniques. The motion compensation estimates the motion of an object, between successive frames, and takes that into account when calculating the differences. So, the differences are mainly representing the changes in the objects properties such as shape, color, and deformation, but not usually the motion. This reduces the differences and hence the bit-rate. The motion compensation process results in important parts of the motion information. That are the motion vectors, which indicate the motion of various parts of the image between the successive frames, and the residual, which are the errors resulting from the motion compensation process. In fact, one big advantage of the temporal-coding is the extraction of the motion information that are highly important in video segmentation, and shot detection, as discussed in section IV. As the MPEG standards became one of the most commonly used in industry, a brief explanation of the MPEG structure and information, mainly those that are related to the techniques discussed in section IV, is below. MPEG MPEG stands for Moving Pictures Experts Group. In the MPEG standards, the frame is usually divided into blocks, 8x8 pixels each. The spatial-coding, or intra-coding, is achieved by using the Discrete Cosine Transform (DCT) on each block. The DCT is also used in encoding the differences between frames and the residual errors from the motion compensation. On the other hand, the temporal-coding, or inter-coding, is achieved by block-based motion compensation techniques. In MPEG, the unit area that is used for motion compensation is usually a block of 16x16 pixels, called macro-blocks (MB). However, in MPEG-4, moving objects can be coded through arbitrary shapes rather than the fixed-size macro-blocks. There are various types of frames in the MPEG standards. The I-frames, which are the intra-coded frames, the P-frames and B-frames, which are the frames that are predicted through motion compensation and the calculated differences between original frames. Each two successive I-frames will usually have
Figure 3. The sequence of the MPEG's I-, P-, and B-frames Bi-directional Prediction
Bi-directional Prediction Forward
P B B B B B
Forward-Predicted
Backwar d
I B B B B B
a number of P-frames and B-frames in between them, as depicted in fig. 3. The sequence of one I-frame followed by a number of P- and B-frames, till the next I-frame, is called a group of pictures (GOP) The MPEG provides various useful information about the changes between frames, including motion vectors, residual errors, macro-blocks types, and group of pictures. These information are usually utilized by shot-detection techniques that works on the MPEG compressed video data, as discussed in section IV.
III. Vidion In order to be able to extract the shots, it is important that we understand how they were joined together when scenes were created in the first place. Shots can be joined together using various types of transition effects. The simplest of these transition effects is the straight-forward cut and paste of individual shots adjacent to each other, without any kind of overlapping. However, with digital video and the advance in editing tools and software, more complex transition types are commonly used nowadays. Shot transitions can be classified in many ways, based on their behavior, the properties or features that they modify, the length and amount of overlapping, to name just a few. In this section, the most common shot transition types are introduced and defined. We start by presenting various classification approaches and their criteria. Then, a classification approach is chosen and its commonly used transition types are identified and explained. The chosen classification and its identified transition types will be referred to in the rest of the chapter. The first classification approach is based on the applied spatial and/or color modifications to achieve the transition. In this approach, shot transitions can be classified to one of the following four types: •
Identity transition: In an analogy with the identity matrix, this transition type does not involve any modification, neither in the color nor in the spatial information. That is none of the two shots that are involved in this transition will be subject to any modification. They will just be joined together as they are.
Spatial transition: As the name suggests, the main modification in this type is in the spatial details. The involved shots are subject to spatial transformations only. Chromatic transition: Again, as the name implies, the chromatic details will be the main target for modification. So, the involved shots are subject to modifications in their chromatic details, such as color and/or intensity. Spatio-Chromatic transition: This is the most general type of transition. This is because it can involve combinations of both chromatic and spatial modifications.
The second classification approach is based on the length of the transition period and the amount of overlapping between the two involved shots. This seems to be more common and its terminology can be identified in most video editing tools and software. In this approach, shot transitions can be classified to one of the following two main types: •
Hard-Cut transition: This type of transition happens suddenly between two consecutive frames. The last frame of the first shot is directly followed by the first frame of the second shot, with no gaps or modifications in between. Hence, there is no overlapping at all. This is illustrated in fig. 4.a. Gradual transition: In the gradual transitions, the switch from the first shot to the second shot happens over a period of time, not suddenly as in the hard-cut. The transition occupies multiple frames, where the two involved shots are usually overlapping. The length of the transition period varies widely, and in most cases it is one of the parameters that can be set within the editing tool, by the ordinary user. There are almost unlimited combinations of the length of transition period and the modifications that can be done on the overlapping frames from both shots. This results in enormous forms of the gradual transitions. However, the most commonly known ones are: ° Fade-out: This is a transition from the last frame of the first shot, to a monochrome frame, gradually over a period of time, as illustrated in the left half of fig. 4.c. ° Fade-in: This is almost the opposite of the fade-out. The fade-in is a transition from a monochrome frame to the first frame of the following shot, gradually over a period of time, as illustrated in the right half of fig. 4.c.
Figures 4. Illustrations of the various types of shot-transition a) hard-cut b) gradual transition c) fade-in and fade-out Shot 1
Shot 2 time
(a) Shot 1
Gradual Transition
Shot 1
Fade-out
Fade-in (c)
Dissolve: The dissolve involves the end and start frames of the first and second shots, respectively. The pixels' values of the transitional frames, between the original two shots, are determined by the linear combination of the spatially corresponding pixel values from those end and start frames. This is illustrated in fig. 4.b. Wipe: The wipe transition is achieved by gradually replacing the final frames of the first shot, by the initial frames of the second shot, on spatial basis. This could be horizontally, vertically, or diagonally.
With the above overview of the most common shot-transition types, and their definitions, we are almost ready to explore the shot-boundary detection techniques, which is the focus of the next section.
IV. SBounda After the above introduction to the most common shot-transitions, in this section we introduce the key approaches for the shot-transition detection. As you may imagine, the work in this area became numerous, and is still growing, and an exhaustive survey of the field is beyond the scope of just one chapter, if not beyond a full book. However, the aim in this chapter is to introduce the researchers to the main categories and key work in this field. Shot-boundary detection techniques can be categorized in a number of ways, depending on various factors, such as the type of the input video data, change measures, processing complexity and domain, as well as the type of shots tackled. The following is a brief overview of the various possible categorizations, based on each of those factors: 1.
The type of the input video data: Some techniques work directly on the compressed data and others work on the original uncompressed data. As we will see, most compression formats are designed in a way that provide some useful information and allow the detection techniques to work directly without need for decompression. This may improve the detection performance. The change measures are another important factor. As discussed before, segmenting the video into shots is based on detecting the changes between the different shots, which are significantly higher than changes within the same shot. The question is "what changes are we looking for?, and equally important, how to measure them?" There are various elements of the image that can exhibit changes and various techniques may consider various elements and/or combination of elements to look for changes in order to detect the shot boundaries. The image, and subsequently its features, is affected by the changes of its contents. Hence, measuring the changes in the image features will help in detecting the changes and deciding on the shot-boundary. Changes can be within the local features of each frame, such as the intensity or color values of each pixel and existing edges within the image. Changes can also be detected in a more gross fashion by comparing the global features, such as the histogram of intensity levels or color values. As we will see, the latter (i.e. global features) does not account for the spatial details, as it is usually calculated over the entire image, although some special cases of local histograms were developed. More complex features could be also be utilized, such as corners, moments, and phase correlation (Gao, Li, & Shi, 2006), (Camara-Chavez et al., 2007).
Other sources of changes within the video include motion of the objects and/or camera movements and operations, such as pan, tilt, and zoom. However, for the purpose of shot detection, the changes due to camera movements and operations need to be ignored, or more precisely compensated for, as long as the recording is still continuous with the same camera as in the definition of the shot in section II. This is another issue, referred to as the camera ego motion, where the camera movements and operations need to be detected from the given video and those motions are compensated for, to reduce false detections. Also, on a higher level, the objects included within the images as well as the simulation of the human attention models (Ma, Lu, Zhang, & Li, 2002) can be utilized to detect the changes and identify the various shots. Processing time and complexity: Various applications have different requirements in terms of the processing time and computational complexity. The majority, so far, can accommodate the off-line processing of the video data. So, video can be captured and recorded, then later be processed and analyzed, for example for indexing and summarization. Other applications can not afford for that and need immediate and real-time fast and efficient processing of the video stream. An example of that is the processing of the incoming TV/video broadcast. In such a case, fast processing is needed to adjust the playback frame rate to match the modern advanced displaying devices that have high refresh frame rates. This is to provide smoother motions and better viewing options. Processing domain: This is a classical classification in image processing. Some techniques apply the time-domain analysis, while others apply the frequency-domain analysis. Also, various transforms can be utilized, such as Discrete Cosine Transform (DCT) and wavelets transforms. This is especially when compressed data are involved. Type of shot-transition: As we discussed in section II, there are various types of shot-transitions, each has its own properties. Although the ultimate aim is to have techniques that can detect almost all transitions, this is always have a drawback somehow, such as the complexity and/or accuracy. Hence, for more accurate and computationally efficient performance, some techniques are more specialized in one or more related type of transitions, with the main overview categorizations is between the Hard-cut and Gradual transitions. Even the gradual transitions are usually broken down to various sub-types, due to the high variations and possibilities provided by the current digital video editing and effects software. In fact, some efforts are trying to model specific effects and use these models in detecting those transitions, whether from scratch or from the result of a more general shot-boundary detection techniques (Nam & Tewfik, 2005).
It can be noticed that many shot-boundary detection techniques can fall into more than one category of the above classifications. And depending on the specific domain and application, or even the interest of the reader, one or more classification could be more useful than others. But, if we try to present the detection techniques following each and every classification of the above, we will end up repeating a large portion of the discussion. For this reason, we prefer to pick the most common classifications that serve the wide majority of the domains and applications. We prefer to present the key detection techniques in a hierarchical approach. If you are thinking of dealing with video data for whatever application, and need to do shot-boundary detection, you would usually try to select techniques that are suitable to the type of your input data with the application constraints in mind. Hence, the type of the input video data would be one of the initial factors in such a selection. This will help you to focus on those techniques that are applicable to either compressed or
uncompressed data, depending on the type of your input data. Then, the focus is on detecting changes, quantitatively measuring those changes, and making the decision of detecting a boundary or not. To that extent, the classification that is based on the change measures would be useful. We can also distinguish between the main two types of transitions; hard-cut and gradual transitions. The summary of this hierarchical classification approach is depicted in fig. 5. In the following sub-sections, we present the key shot-boundary detection techniques according to the classification depicted in fig. 5. They are organized into three main sub-sections; (a) shot-cut detection from uncompressed data, (b) gradual-transitions from uncompressed data, (c) shot-boundary detection from compressed data. In addition, the performance evaluation is also discussed in sub-section "D" and some common issues are discussed in sub-section "E". But first, let us agree on the following notations that will be used later: Fi: represents the ith frame in the video sequence.
I ( x, y): represents the intensity I of a pixel that is located at the (x,y) coordinates, in the ith frame. Fi
CFk ( x, y): represents the kth color component of a pixel that is located at the (x,y) coordinates, in the ith i
frame.
H (v): represents the intensity histogram value for the vth bin of the ith frame. Fi k H FC i
(v): represents the color histogram value for the vth bin within the histogram of the kth color component, of the ith frame.
Figure 5. Hierarchical classi.cation of shot-boundary detection techniques Compressed Data
Uncompressed Data
Hard-Cut
Motio n
Hybri d
Histogra mPixel Bloc Motio k nPixel Obje ct
Edge Motio n Other s
A. Shot-Cut Detection from "Uncompressed Data" The uncompressed video data is the original video data, which are mainly the frames that are represented by their pixel contents. So, detection techniques need to utilize features from the original data contents, rather than from coding cues as in the compressed data. In the rest of this sub-section, the key shot-cut detection techniques that deal with the uncompressed data, are presented. They are grouped according to the main change measures as follows:
1) Local Features The local feature is calculated for specific location, or a region, within the image. It takes the spatial information into account. Examples of local features include the individual pixel's intensity and color as well as edges, to name just a few. The following are the key shot-cut detection techniques that rely on the most common local features: a) Pixel-Based Differences This category is considered as the very basic approach for detecting changes between frames. It is based simply on comparing the spatially corresponding pixel values, intensity or color. For each consecutive pair of frames, the difference is calculated between each pair of pixels that are in the same spatial location within their corresponding frames. The sum of those differences is calculated and a shot is detected if the sum is greater than a pre-defined threshold (Nagasaka & Tanaka, 1991) as follows:  X Y  ∑∑  x =1 y =1
I ( x, y) - I
Fi -1
( x, y)
  > T 
And for the color images, this is extended as follows:  X Y 3  k k  ∑∑∑ C F ( x, y) - C F ( x, y)  > T i i -1  x =1 y =1 k =1 
The above technique is known as the pair-wise pixel comparison, as it based on comparing the spatially corresponding pair of pixels in consecutive frames. A slight variation of this technique is introduced in (H. J. Zhang, Kankanhalli, & Smoliar, 1993). Instead of calculating and accumulating the differences in the pixel values, it only counts the number of pixels that are considered to be changed between consecutive frames and compare that with a threshold, as follows:  X Y  ∑∑ D , ( , F i F i -1 x y   x =1 y =1
 )  > T  
DF i, F i -1 ( x , y
  if 1 )=  0 
I ( x, y) - I Fi
( x, y) > T
      
Then the detection decision is based on comparing the number of changed pixels against a pre-defined threshold. In its strict form, when TD =0 in (3), it could be more computationally efficient, which is an important factor for the real-time applications. But it is clear that it will be very sensitive to any variations in the pixel values between frames. A common drawback in this category of techniques is that they are very sensitive to object and camera motions and operations. With even the simplest object movement, its position in the image will change, and hence its associated pixels. Hence, the spatially corresponding pixel-pairs, from consecutive frames, may not be corresponding to the same object anymore. This will simply indicate a change, according to the above formulas, that can exceed the defined threshold. Once the change is above the defined threshold, a decision will be made that a shot boundary has been detected, which is a false detection in such a case. As the measure is usually calculated between pairs of consecutive frames, this category is usually more suitable for detecting hard-cut transitions, as it happens abruptly. However, it is worth mentioning that some research has introduced the use of evolution of pixels' values, over multiple frames, to detect gradual transitions (Taniguchi, Akutsu, & Tonomura, 1997), (Lawrence, Ziou, Auclair-Fortier, & Wang, 2001). b) Edge-Based Differences Edges are important image feature that are commonly used, especially in image segmentation. Edges are detected from the discontinuity in the pixel intensity and/or color. This is with the assumption that pixels belonging to the same object are expected to exhibit continuity in their intensity or color values. So, edges could indicate a silhouette of a person or object, or separation between different objects as well as the background. This gives an idea of the contents of the image, which is considered a relatively higher level cue than the individual pixels. Hence, the change in those edges is expected to indicate the changes in the image contents, and when significant enough, indicates a shot change. The straight forward test, especially for detecting hard-cut shot transitions, is to compare the number of edge pixels between consecutive frames. If the difference is above a certain threshold, then there have been enough changes to consider a shot change. However, as discussed in the following sub-section (global features), this simple approach does not take the spatial information into account. As a result, we may have more missed shot boundaries. This happens when frames from different shots contain similar number of edge pixels, but in significantly different spatial locations. For the above reason, motion compensation techniques are utilized to compensate for motion between consecutive frames first (Zabih, Miller, & Mai, 1999). Then, the edges are compared, with indication of their spatial information. The number of edge pixels from the previous frame, which are within a certain distance from edges in the current frame, is called the exiting edge pixels, Pex. Similarly, the number of edge pixels from the current frame, which are within a certain distance from edges in the previous frame, is called the entering edge pixels Pen. Those are usually normalized by the total number of pixels in the frame. The two quantities are calculated for every pair of consecutive frames and the maximum value among them is selected to represent the difference measure. The decision of detecting a shot-boundary is based on locating the maxima points of the curve representing the above difference measure. Sharp peaks, on the curve of the difference measure, are usually an indication of hard-cut transitions. But low and wide peaks, which occupy longer time periods than the hard-cuts, are usually an indication of gradual transitions. In fact, for detecting the gradual transitions, the process needs to be extended to cover multiple frames (Smeaton et al., 1999), instead of only two consecutive frames. More details may be identified for detect14
ing specific types of gradual transitions. For example, for detecting dissolve transitions, edges can be classified into weak and strong edges, using extra two thresholds, as introduced in (Lienhart, 1999). As we can see, the edge-based differences approach can be used for both hard-cut and gradual transitions detection. However, its main limitation is its relatively high computational costs that makes it slow.
2) Global Features Unlike the local features, the global feature is calculated over the entire image. It is expected to give an indication of an aspect of the image contents or its statistics. For example, the mean and/or variance of pixel intensities or colors, over the entire image, can be used as a global representation of the frame's contents. This measure can be compared between frames and a boundary can be detected if the difference is above a pre-determined threshold. Other statistical measures, such as the likelihoodratio, can also be utilized. However, the histogram is the most popular and widely used global feature as discussed below. a) Histogram-Based Differences For the intensity level histogram, the intensity range is divided into quantized levels, called bins, and each pixel is classified into the nearest bin. Then the histogram represents the number of pixels associated with each bin. The same applies for the color histograms. The histogram of the intensity levels is usually used for grey-level images, and the color histograms are utilized for color images. Once the histogram is constructed for each frame, the change measure is calculated by comparing the histograms of the pair of consecutive frames. The change measure can be as simple as the differences between the histograms. Then the sum of those differences is again compared with a pre-defined threshold to decide about the detection. For intensity histogram, this can be as follows:  V  ∑  v =0
H (v) - H Fi
But for the color histogram, the following is an extended form that takes the various color components into account:  K V  ∑∑  k =1 v = 0 
H FC (v) - H FC (v) i
i -1
  > T  
The color components could be the RGB components, with K=3 in (5), or any other components of other color spaces. Although normally only two components are used, i.e. K=2. In fact, one of the issues related with the color histogram is the selection of the appropriate color space (Gargi, Kasturi, & Strayer, 2000), as in the discussion sub-section below. As can be seen above, the histogram is relatively easy to calculate. More importantly, it is less sensitive to object and camera motion than the pixel comparison techniques. This is because it is a global feature that does not involve spatial details within the frame. But, for the same reason, the technique may miss shot-cuts when different shots have similar range of total intensity or color values. A simple example is 15
two frames, one contains a checker board, and the other contains a rectangle with half black and half white. Using the histogram, with the same number of bins, the histogram values will be the same. On the other hand, false detections can also be encountered, due to intensity and/or color changes within the frames contents, although within the same shot. An improved color histogram-based technique was introduced in (Ionescu, Lambert, Coquin, & Buzuloiu, 2007), for animated movies, where the frame is divided into quadrants to localize the measures.
3) Intermediate Level As we saw in the above sub-sections, local and global features based techniques, each has its own advantages and limitations. The ultimate aim is to achieve more accurate shot-boundary detection, but also with less computation complexities. The block-based techniques are trying to address this balance between the advantages and limitations of using the local- and global feature-based techniques, to achieve those aims. a) Block-Based Differences In this approach, each frame is divided into equal-size areas, called blocks (Katsuri & Fain, 1991). These blocks are not spatially overlapping. The more blocks, the more the spatial details will be involved, with the extreme is having number of blocks equal to the number of pixels. Once the frame is divided into blocks, the rest of the work deals with the block as the smallest unit within the frame, instead of pixels. Although each block consists of a number of pixels, an individual block can be dealt with either as a single value, as we dealt with pixels, or as a sub-image. In the case of dealing with the block as a sub-image, histogram-based techniques can be applied (AhmedM, 1999), (Bertini, Del Bimbo, & Pala, 2001). In such a case, a histogram is calculated for each individual block. Then, the histograms of the spatially corresponding blocks, from consecutive frames, are compared and the difference measure is tested against a pre-defined threshold. The same difference measures of histogram-based techniques can be utilized here, as discussed in the above sub-section (Histogram-based differences). This approach is also known as the local histograms, as it involves spatial details to some extent. The block can also be dealt with as a single value, in an analogy to pixels. This can be seen as a reduced dimension of the original image, as every group of pixels will be replaced by the single value of the individual block that contains them. But first, we need to select the criteria for determining the block value that appropriately represents its contained pixels. Variety of measures can be used to determine the block value. For example, the mean and/or variance of pixel intensities or colors can be used (M. S. Lee, Yang, & Lee, 2001). Other statistical measures, such as the likelihood-ratio, can also be utilized (Ren, Sharma, & Singh, 2001). The above two ways of dealing with the block, as a single value or as a sub-image, can also be combined together (Dugad, Ratakonda, & Ahuja, 1998). Given the relatively cheaper computations of the histogram, it is used as a initial pass to detect hard-cut transitions and indicate potential gradual transitions. Then, the candidates are further checked using mean, variance, or likelihood-ratio tests between spatially corresponding blocks as usual. Involving multiple frames, and especially for off-line processing, the evolution over time of the above difference measures can also be tracked, processed, and compared with a threshold to detect shot-boundaries (Demarty & Beucher, 1999), (Lefevre, Holler, & Vincent, 2000). In (Lefevre et al.,
2000), the evolution of the difference values were used to indicate the potential start of gradual transitions, while the derivative of the difference measures, over time, has been utilized to detect the hard-cut transitions. As we mentioned above, the block-based techniques are considered as intermediate between local and global feature-based techniques. Hence, they are relatively less sensitive to the object and camera motions, than the pixel-based techniques, as the spatial resolution is reduced by using blocks instead of individual pixels. On the other hand, it can be relatively more computationally expensive than the global histogram-based techniques.
4) Motion-Based In the previously discussed techniques, the main principle was finding the changes in some features of the contents of consecutive frames. This is, at least implicitly, based on the concept that the video can abstractly be considered as a sequence of individual frames or still images. However, the video sequence has more than just playing back a group of still images, otherwise, it can be considered as a uniform slide-presentation. The video sequence carries other extra information, than a slide-show of still images. One of the extra important information in the video data is the motion information. The motion is conveyed by the temporal changes of object and/or camera positions and orientations over consecutive frames. We should recall that the video frame rates, especially the playback frame rates, are designed to convey smooth motion to the human eye. That is the rate of change of the displayed images is more than what the human eye can recognize, hence, no flickers can be noticed. Given the above, and recalling the shot definition from section II, it is expected that the motion encountered among the frames of the same shot will have continuity. However, the motion among frames from different shots will exhibit discontinuity. So, this continuity criterion can be utilized in detection of the shot-boundary. We should also differentiate between the motion originated from the movements of the contained objects and the motion originated from the camera movements and operations, such as pan, tilt, or zoom. The camera movements will usually result in a similar motion of the contents, assuming no moving objects. That is all the frame's pixels or blocks will shift, translate, or rotate consistently, almost in the same direction. This is except for zooming, although it can be seen in a similar fashion as motion will be radially consistent from/to the centre of the image. The motion originated from the camera movements or operations is known as the global motion, as it is usually exhibited over the entire image. The objects motion can be a bit more complicated. First of all, we would usually have more than one object in the video, each may be moving in different direction. With occlusion, things become even more complicated. This is all with the assumption that objects are simply rigid. In fact, deformable objects, like our skin, could also add to this complexity. Within the normal video sequence, many combinations of the above motions can be found. These are all challenges faced by motion estimation and compensation techniques. Based on the above discussion, various techniques were developed that utilize one or more of the above motion properties. When utilizing the global motion, it is assumed that it will be coherent within the same shot. If the coherence of the global motion is broken, a shot-boundary is flagged (Cherfaoui & Bertin, 1995). A template could also be designed where each pixel is represented by a coefficient that indicates how coherent it is with the estimated dominant motion. The evolution of the number of
pixels that are coherent with the dominant motion, with a pre-determined threshold, are used to decide on shot-boundary detection (Bouthemy, Gelgon, Ganansia, & IRISA, 1999). In utilizing the objects motion, techniques from the motion estimation are utilized to calculate motion vectors and correlation coefficients, of corresponding blocks, between frames. The similarity measure is calculated from those correlation coefficients, and critical points on its evolution curve correspond to shot-boundaries (Akutsu, Tonomura, Hashimoto, & Ohba, 1992), (Shahraray, 1995). Other techniques were developed that are based on the optical flow (Fatemi, Zhang, & Panchanathan, 1996) or correlation in the frequency domain (Porter, Mirmehdi, & Thomas, 2000). The computational cost of the motion estimation techniques are relatively high (Lefèvre, Holler, & Vincent, 2003), which can affect the performance of the techniques discussed above, in terms of the processing time. However, motion estimation are already incorporated in the current compression standards such as MPEG (see "video compression" in section II). Hence, more work has been done in this category but for the compressed data, as discussed in their corresponding section later in this chapter.
5) Object-Based Instead of applying the difference measures on individual pixels or fixed size blocks changes, in this category, efforts are made to obtain differences based on object-level changes. Based on color, size, and position of recognized objects, differences between frames can be computed (Vadivel, Mohan, Sural, & Majumdar, 2005). Objects are constructed by pixel grouping, through k-means clustering, followed by post-processing that includes connected component analysis to merge tiny size regions. Although it was presented on I-frames, an MPEG frame type, the techniques seems to be applicable for uncompressed data. Also, a semantic objects tracking approach was introduced in (Cheng & Wu, 2006) for the detection of both shot-cut and gradual transitions. Foreground objects are recognized and tracked based on a combined color and motion segmentation process. The numbers of entering and exiting objects or regions help in detecting shot changes, while the motion vectors help in detecting the camera motion. Although it is easy for humans to recognize and identify objects from images and videos, achieving this automatically is still an ongoing research challenge. Hence, the object-based approaches will always benefit from the advances in the image analysis and understanding field.
B . Gradual-Transition Detection from "Uncompressed Data" As explained in section III, the gradual transitions exhibit much less changes between consecutive frames, than in the case of hard-cut transitions. Consequently, the pre-determined threshold that is used to detect hard-cuts will always miss the gradual transitions. On the other hand, using a lower threshold will increase the false detection, due to motion and camera operations that introduce changes in a relative order to the changes from the gradual transitions. As the gradual transition occurs over longer period of time, the difference measure between two consecutive frames, which is used in most of the techniques described earlier, is not always useful to accurately detect such transitions. We need to track the changes over longer periods, i.e. multiple frames. One way, especially for off-line detection, is to track the evolution of the difference measure over the time of the given sequence. In a pixel-based approach introduced in (Taniguchi et al., 1997), the variation of pixel intensities is tracked and pixels are labeled according to the behavior of their variation
over time; sudden, gradual, random, or constant. The gradual transition can be detected by analyzing the percentage of the gradually changing pixels. In the rest of this sub-section, the most common techniques, for detecting gradual transitions, are discussed.
1) Twin-Threshold Approach This approach, introduced in (H. J. Zhang et al., 1993), is based on two observations. Firstly, although the differences between consecutive frames are not high, during the gradual transition, the difference between the two frames, just before and after the transition, is significant. It is usually in the order of the changes that occur in hard-cut, as those two frames are from different shots. Secondly, the changes during the gradual transition are slightly higher than the usual changes within the same shot. They also occupy longer period of time, as depicted in fig. 6. The twin-threshold algorithm, as the name suggests, has two thresholds, instead of one. The first threshold, TH, is similar to thresholds discussed before, and is usually high enough to identify the hard-cut transitions. The second threshold, TL , is the new introduced threshold. This threshold is adjusted so that the start of a potential gradual transition period can be detected. In other words, this threshold is set to identify the changes due to the gradual transition, from changes between frames within the same shot. Once the difference between two consecutive frames exceeds the TL , but still less than TH, this will mark the start of a potential gradual transition. From this frame onwards, an extra accumulated difference will be calculated, in addition to the difference between consecutive frames. This will continue until the difference between consecutive frames fall again below the threshold TL. This is potentially the end of gradual transition, if any. To confirm, whether a gradual transition was detected or not, the accumulated difference is compared to the threshold TH. If the accumulated difference exceeds the threshold TH, a gradual transition is considered. Otherwise, the marked potential start, of gradual transition, will be ignored and the procedure starts again. The procedure, from one of our experiments, is illustrated in fig. 7, with the addition that the value of TL is computed adaptively based on the average of the difference
Figure 6. Example of the difference measure for hard-cut and gradual transition
measure from previous frames, to adapt to the video contents. Choosing the threshold value is one of the important issues and discussed in a later sub-section with other common issues. One limitation of this algorithm is that it could produce false detection if significant object and/or camera motion and operations occurred. This is because those significant motions are expected to produce frame differences of almost the same order as the differences exhibited during the gradual transition. One suggested solution to reduce this false detection is to analyze detected transitions to distinguish real gradual transitions from global motion of camera and camera operations (H. J. Zhang et al., 1993).
2) Edge-Based Detection of Gradual Transitions The edge-based comparison has already been discussed earlier. In this subsection, we emphasize on its use for detecting gradual transitions. As edges are related to the objects within the images, its changes and strength can also help identifying the various types of gradual transitions. Based on the properties of each type of the gradual transition, as discussed in section III, the number and ratios of the entering and exiting edge pixels can indicate a potential shot change. In fade-out, the transition is usually end up with a constant color frame, which means that it has almost no entering edges. Hence, the exiting edges will be expected to be much higher. In the fade-in, the opposite will happen. In fade-in, the transition usually starts from a constant color frame that has almost no edges. Hence, the entering edges, from the new shot, will be expected to be much higher. So, by analyzing the ratios of the entering and exiting edges, fade-in and fade-out can potentially be detected. Dissolve transitions can also be detected as they are usually a combination of the fade-in and fade-out. Wipe transition needs a bit more attention due to the special changes in the spatial distribution. Based on the hypothesis that the gradual curve can mostly be characterized by the variance distribution of edge information, a localized edge blocks techniques was presented in (Yoo, Ryoo, & Jang, 2006). However, as mentioned earlier, the edge-based comparison is computationally expensive and relatively slower than others.
3) Motion-Based Detection of Gradual Transitions As discussed earlier, for the uncompressed data, most of the motion-based techniques use the evolution of the motion measure, whether global motion, motion vectors, correlation, or others, to decide on the shot-boundaries. A histogram-based approach is utilized in (H. J. Zhang et al., 1993) for detecting two directions of wipe transitions, specifically horizontal and vertical wipes. As in (Bouthemy et al., 1999), pixels that are not coherent with the estimated motion are flagged first. Then, vertical and horizontal histograms, of those flagged non-coherent pixels, are constructed for each frame. The histogram differences, of corresponding histograms of consecutive frame, are calculated and thresholds are used to detect a horizontal or a vertical wipe transition. In (Hu, Han, Wang, & Lin, 2007), motion vectors were filtered first to obtain the reliable motion vectors. Then, those reliable motion vectors are used to support a color-based technique and enhance the detection accuracy, for soccer video analysis. Also, the analysis of discontinuity in the optical flow between frames can be utilized in shot detection (Bruno & Pellerin, 2002).
Other Combined Techniques for Detection of Gradual Transitions Some other techniques, that may not fit exactly within the above categories, are discussed here. The Hidden Markov Models (HMM) were employed in (W. Zhang, Lin, Chen, Huang, & Liu, 2006) for detection of various types of shot transitions. An HHM is constructed and trained for each type of the shot transitions, which model its temporal characteristics. One of the advantages is that the issue of selecting threshold values is avoided. More advanced features can be employed such as corners, moments, and phase correlation. In (Gao et al., 2006), corners are extracted at the initial frame and then tracked, using Kalman filter, through the rest of the sequence. The detection is based on the characteristics of the changing measure. In another system, a two-pass hierarchical supervised approach was introduced (Camara-Chavez et al., 2007) that is based on a kernel-based Support Vector Machine (SVM) classifier. A feature vector, combining color histograms and few moment measures as well as the phase correlation, is extracted for each frame. In the first pass, shot-cuts are detected and used as guides for the second pass, where the gradual transitions are to be detected in-between shot-cuts.
Figure 7. The difference measure (top) and the accumulated difference measure (bottom) of the twinthreshold technique, for detection gradual transitions
Finally, some efforts are made to model the various editing effects and detect the transitions by fitting the corresponding edit model as in (Nam & Tewfik, 2005). While it could achieve promising results, the main concern is the almost unlimited, and ever increasing, number and variations of editing effects that are even customizable by video editing software users.
C . Sot-Boundary Detection from "Compressed Data" Compressed video data result from processing the original video data by coding the data, inter- or intra-coding as explained in section II, in such a way to reduce redundancy. This coding also provides useful information that helps the detection techniques. Examples include the motion vectors, and various types of frames and macro-blocks in the MPEG compression. The coding usually facilitates the shotboundary detection without decompression. Hence, it could also improve the detection performance, in terms of the computational costs. In MPEG compression, the frame is divided into 8x8 blocks. Blocks within the I-frames are intracoded using the Discrete Cosine Transform (DCT). For P-frames and B-frames, motion vectors, correlations and residual errors are calculated. Hence, different types of frames contain some different types of information. The rest of this subsection presents the detection techniques that deal with the compressed data. They are based on the most common features that can be extracted, usually directly without need for decompression, from the compressed data.
1) DCT-Based The I-frames of the MPEG compressed data, see section II, which are intra-coded frames, contains all the DCT coefficients. On the other hand, the P-frames and B-frames do not usually include the DCT coefficients as most of their blocks are inter-coded rather than intra-coded. Most blocks within those P- and B-frames include the DCT coefficients of the residual error resulting from the motion compensation. The DCT coefficients are representatives of the frame contents. In fact, the DC coefficients alone represent a low-resolution, or iconic, version of the image and are considered as an iconic version of the image. Hence, most of the pixel-based and histogram-based techniques could be applied for shot-boundary detection. Also, the difference in feature vectors, extracted from the DC components, is used to detect shot-cuts (Taskiran et al., 2004). The feature vector contains standard deviation and color histogram intersection of the YUV color space components, as well as other information about the various types of macro-blocks and the type of the frame. For gradual transitions, and for off-line processing, both the absolute difference and the variance of the DC coefficients are utilized to detect gradual transitions (Yeo & Liu, 1995), (Meng, Juan, & Chang, 1995). Gradual transition is represented by a plateau in the curve depicting the absolute difference between two frames separated by n frames, where n is longer than the known length of gradual transitions (Yeo & Liu, 1995). On the other hand, a dissolve will produce a downward parabolic curve when the variance of the DC coefficients is plotted (Meng et al., 1995). As mentioned above, the DCT coefficients, of frame contents, are mainly available within the Iframes. However, I-frames are only a percentage of the total frames in MPEG, as there are several P-frames and B-frames in-between I-frames. This introduces some temporal gaps, if only I-frames are used with DCT-based techniques, which increases the false detection. The straight forward solution is
to use the P-frames and B-frames as well, by obtaining the best possible estimation of their DCT coefficients. Alternatively, the technique can be combined with other techniques as discussed in the hybrid techniques section below.
2) Motion-Based As discussed in the motion-based detection techniques, from uncompressed data, the motion is an important property in video. And based on the shot definition, it is expected that the motion measure will have continuity over time, within the same shot. But will exhibit discontinuity across different shots. The computational cost of obtaining the motion vectors are relatively high (Lefèvre et al., 2003). But fortunately, motion information can be extracted from compressed data that have been coded through the currently available compression standards such as MPEG (see "video compression" in section II). This dramatically reduces the computational cost of the motion-based shot-boundary detection. From the MPEG coding, the blocks contained in the I-frames are all intra-coded. But the blocks in the P-frames and B-frames are of various types. P-frames contain both intra-coded as well as forwardpredicted blocks. B-frames are more complicated as they can contain any combination of forward-predicted, backward-predicted, bi-directional predicted, intra-coded, and/or skipped blocks. The number and the ratios between the various types of blocks can give some indication of changes and motion continuity (Yi, Rajan, & Chia, 2006), hence, helps in detecting shot-boundaries. For example, the more continuous and coherent the motion is, the less intra-coded blocks to be found. In other words, blocks that have high residual errors as a result of motion compensation, indicates a discontinuity of motion. Those blocks are intra-coded by the MPEG encoder. So, when the ratio of the intra-coded blocks is higher than a pre-defined threshold, a potential shot-boundary is flagged (Meng et al., 1995). The bit-rate for encoding the blocks, in MPEG compression, also exhibit variations, especially with the changes associated with shot-cuts. By monitoring and analysing the variations in the number of bits required to encode each block, shot-cuts can be detected when the differences in bit-rate exceed certain threshold (FENG, Kwok-Tung, & MEHRPOUR, 1996). Exhaustive survey of the compressed video data, and more features that can be utilized in shot detection, can be found in (Wang, Divakaran, Vetro, Chang, & Sun, 2003), (Vadivel et al., 2005).
3) Hybrid Techniques One or more techniques can be combined together to achieve better detection performance, whether the accuracy and/or the processing speed. A hierarchical multi-pass approach was introduced in (H. Zhang, Low, & Smoliar, 1995). This combines the DCT coefficients technique and the motion-based techniques. The DCT coefficients, which are already available in the I-frames, are utilized to locate potential shot-transitions. Then, motion information, from the in-between P-frames and B-frames, are utilized to confirm and refine the identified shot-boundary. As we can see, the motion information is only utilized from the frames lying in-between the two I-frames around the identified potential transition. This improves the computational costs. Also, by using the appropriate thresholds, it was reported that both shot-cuts and gradual-transitions can be detected. A similar two-pass hybrid approach was presented in (Koprinska & Carrato, 2002), although only use motion vectors and macro-blocks information. Potential shot-boundaries are identified by a rule-based module, then a refined pass using a neural structure detect and classify the various types, especially
gradual transitions. In (Zhao & Cai, 2006), a technique utilizing Adaboost and fuzzy theory was introduced. It utilizes various features and information, such as DCT coefficients, macro-block types, and color histograms. The important reported advantage is the robustness of the technique's performance against camera motion and large objects fast movements. The bit-rate variation technique from (FENG, Kwok-Tung, & MEHRPOUR, 1996) can also be combined with other techniques, such as those that use the DCT coefficients, to improve the detection (Divakaran, Ito, Sun, & Poon, 2003), (Boccignone, De Santo, & Percannella, 2000).
D. Performance Evaluation There are two main performance aspects that need to be assessed when evaluating or comparing shotdetection techniques, as well as many other techniques. These two aspects are the accuracy and the computational complexity. As we can expect, these aspects could be seen as being on opposite sides of a coin, or as two ends of one string. Usually improving one aspect would be on the cost of the other one. So, usually to improve the accuracy, more complicated techniques are developed, which will usually require more computational requirements. On the other hand, if we are after a real-time system, we might have to compromise and accept some less accuracy. Also, for the evaluation to be truly representative and trustable for comparing various techniques, it needs to be done in similar conditions and with very similar, if not the same, datasets. Hence, there is a need for benchmarks to be used. TRECVID (Over, Ianeva, Kraaij, & Smeaton, 2005) is one of the initiatives within the video retrieval task. Other efforts also facilitate reproducing the research results so that people can test it themselves. Some even allow the user to upload their own data and apply the techniques on it (Vadivel et al., 2005) , which provide a fair comparison. Otherwise, researchers will have to re-implement previous techniques to compare with, which waste time and effort, and can still be not accurate due to various interpretations and implementation decisions. In this section, we discuss the above two aspects of performance evaluation, the accuracy and the computational complexity, and the common metrics of measuring them for the shot-boundary detection.
1) Accuracy Ideally, we would hope that the detection technique can detect all the correct shot-boundaries, without missing any of them. However, the reality is that for some reasons, some shots can be missed, and also some non-existing boundaries may seem to be detected, which is called the false detection or false positive. One way of quantitatively evaluating the accuracy is by calculating the detection-rate and the error-rate (Tian & Zhang, 1999). The detection-rate is the percentage of the correctly detected shots, relative to the total number of shots. The total number of shots should be identified, somehow but usually manually, as a ground-truth. Similarly, the error-rate is the percentage of the falsely detected shots, relative to the total number of shots. The most commonly measures that are used to evaluate the accuracy of the detection techniques are the recall and the precision. The recall measure is similar to the detection rate, described above. The precision measure gives an indication on how precise the detection is, correct or false. We can denote the number of correctly detected shots by Nc, the number of falsely detected shots by Nf, and the
missed shots by Nm. Hence, the Nc+Nm represents the total number of shots, while the Nc+Nf represents the total number of detections by the detection technique. And the recall and precision can be obtained as follows: Re call =
NC+Nm
Pr ecision =
N C
+N
From the above definitions, the good detection techniques are expected to have high recall and high precision at the same time. It can be noticed that the above measures are mainly evaluating the performance in terms of the number of shots, whether correct, missed, or false. However, the accuracy in determining the length of the transition periods also needs to be evaluated. Additional measures, brought from the field of image segmentation, have been employed, namely over-segmentation and under-segmentation (Y. J. Zhang & Beijing, 2006). The lengths of the detected shots are compared with the lengths of the corresponding shots in the ground-truth. It is preferred to have low over- and under-segmentation values. So, overall, the good detection technique is expected to have high recall, high precision, low oversegmentation, and low under-segmentation values.
2) Computational Complexity Most of the evaluation is usually focused on the accuracy aspect of the performance. That could be acceptable for off-line applications. But once the processing time becomes a factor in the application, such as real-time applications, the computation costs have to be incorporated in the performance evaluation. In fact, most research of the literature provides quantitative figures of the accuracy of the presented detection techniques. But it is rare to find quantitative figures for the computational complexity. However, an exhaustive survey, where the evaluation is focused on the computational complexity, was presented in (Lefèvre et al., 2003). This was crucial as the survey was reviewing the real-time shot detection techniques, from uncompressed data. Their computational complexity measure considers the arithmetic and logical operations only. It represents the number of operations per frame where each arithmetic or logical operation, including the absolute operation, increments the computational complexity measure by one. Other operations, such as loops and conditional branching were not considered in that complexity measure. For the pixel-based techniques, the complexity measure will be a function of the number of pixels in each frame, which is denoted by P. So, for the intensity-based computations, the complexity is O(3P), while for the color version, the complexity is O(9P). Another estimated complexity, for edge-based techniques, is O(26P). However, for the block-based techniques, the number of blocks per frame is also incorporated, which is denoted by B. The complexity measure is estimated to be between O(2P+10B) and O(3P+15B), depending on the specific technique used. Similarly, for the histogram-based techniques, the measure will be a function of the number of bins, denoted by L. It was reported that, for intensity histogram difference, the complexity is O(3L). The complexity of the color version is O(6L), based on using only two color components, for the reasons mentioned in the discussion section next.
E . Cmmon Issues From the above review, and the preceded discussion about the various classifications, it is clear that the topic is large and interrelated in various ways. There are also some issues that are considered to be common to various groups of techniques. To avoid repeating them in various places, which is a redundancy that we tried to avoid, we preferred to discuss them in this sub-section. It can be noticed from the above review, that in almost all the techniques, the detection decision involves a pre-defined thresholds. Determining the values of those thresholds is one of the common issues. It is expected and understood that the value of many of those thresholds will depend on the domain and application as well as the particular video data. Fixed threshold values are usually chosen based on heuristics or through statistics of the existing data. Fixed pre-defined values would be acceptable for specific domains, given that they have been carefully chosen. However, adaptive and automatically chosen threshold values (Volkmer, Tahaghoghi, & Williams, 2004), (Bescos, Cisneros, Martinez, Menendez, & Cabrera, 2005), (Ionescu et al., 2007) would be much desirable to reduce the need for human intervention and to adapt to the varying data ranges. In fact, avoiding the thresholds would be even better as in (Yu, Tian, & Tang, 2007), where the Self-Organizing Map (SOM) network was used, which avoided the need for thresholds. Another issue, especially with the color histogram techniques, is selecting the appropriate color space. Usually, the images are converted from the RGB color space to other color spaces such as YUV, HSV, or YCbCr (Gargi et al., 2000). This is because those color spaces are closer to the color perception model of humans. Also, the intensity value component is usually not included in the color histogram to reduce false detections. The YCbCr color space is used in the MPEG video. For the gradual transitions in particular, large and sudden object motion and fast camera operations in dynamic scenes increases the false detection. This is because they produce relatively large differences that are in the order of the changes of the gradual transitions. From the detected and classified shots, higher logical components in the logical video structure can be constructed, such as video scenes and segments. Those higher level components are usually more dependent on the application domain and more semantic information. However, it is worth mentioning that some recent efforts of scene detection have been reported in(M. H. Lee, Yoo, & Jang, 2006), (Rasheed & Shah, 2005), (Adjeroh & Lee, 2004).
V. SUMMaand Video data are now available, and being easily generated, in large volumes. This puts a high demand for efficient access, search, and retrieval of these data, which have not been progressed in the same rate. To facilitate that, the data need to be annotated and indexed, by text, features, or semantic descriptions. This can be done manually, but given the huge volumes of data, it became impractical to do so. Hence, automatic or even semi-automatic video content analysis techniques need to be able to efficiently extract the video structure and components. Most of these techniques require processing and segmentation of the given video data. In this chapter, we presented the principles of the video representation and processing in preparation for video data mining, with focus on the shot-boundary detection as an important initial step. Firstly, the video structure and representation were explained, followed by definition and classification of the
common shot transitions. Then, the stat-of-the-art, of the key techniques for the shot-boundary detection, was presented. We have only focused on the visual modality of the video. Hence, audio and text modalities were not covered. Research in multi-modal detection can be found in the literature, but it was beyond the scope of this chapter. As can be seen from the review, a lot of research has been going on for the shot boundary detection, both on the compressed and uncompressed data. In uncompressed data, most techniques trying to detect changes based on comparing differences in pixel values, histograms, edges, and motion information. In compressed data, techniques are relying on the representation of the compressed data and the information provided in it. Compressed-domain techniques are based on DCT coefficients, motion information, or a hybrid. Other higher-level techniques, such as object-based techniques have also been presented. It was reported, through comparisons between various methods, that the histogram-based techniques are usually more reliable and efficient. But it has its limitations, in terms of sensitivity to changing illumination conditions within the shot and missing shot boundaries especially in the global histograms. Localized histograms address these problems to some extent. The edge-based techniques, although useful in detecting both shot-cut and gradual transitions, are computationally expensive. Motion-based techniques are useful in utilizing the temporal information, but expensive to compute on the uncompressed data. On the other hand, compressed-based techniques utilize the spatial and temporal coded information, mainly the motion and block types, to detect changes. As these information are already available within the compression coding, motion-based techniques are more efficient on the compressed data. However, may not be on the same level of accuracy. It is obvious that there is no individual technique that solves the problem completely. Hence, hybrid methods that combine two or more techniques were employed to improve the performance. The performance evaluation of the shot detection techniques includes both the accuracy and computational complexity. Accuracy evaluation is always the dominant and has been the focus of most papers in the literature. However, the computational complexity is also important, especially with real-time applications, and the rapid increase of the data volumes. Accuracy is evaluated through the precision and recall measures. They are calculated based on the number of correctly and falsely detected shots, as well as the missed one. Another important issue with the evaluation is that it needs to be done using a common data sets, and standards to be truly representative. Some initiatives have been made and more improvements are still needed. Common issues in almost all the detection algorithms include the selection of the threshold values and the appropriate color space to work with. Adaptive threshold were suggested, based on some statistical properties of the video data. For the color space, most techniques convert the images from the RGB color space to other color spaces such as YUV, HSV, or YCbCr, as they are closer to the color perception model of humans. It has also been noticed that, especially with the gradual transitions, large and sudden object motion and fast camera operations in dynamic scenes increases the false detection. This is because they produce relatively large differences that are in the order of the changes of the gradual transitions. Some specific models are designed to fit specific gradual transitions, which improves the accuracy and reduce the false detection. However, the increasing number, variability, and options of the possible digital effects these days makes it difficult to include models for all possible effects. Finally, the detected shots are expected to be classified and grouped to construct the higher components in the logical video structure; that are the scenes and segments. This can be done based on temporal, contents similarity, and/or semantic inputs. But they are usually more dependent on the specific application domain among other factors. Hence, most probably they involve some level of user
interaction, especially for the semantics. We also mentioned some efforts in scene detection that have been reported in the last couple of years.
ACKNno I would like to thank my lovely wife (Mrs G. Mohamed) very much for her kind help, support, and encouragement all the way through. Thanks also to my lovely children (Noran, Yusof, and AbdulRahman) for their patient and understanding during the time I have been working on this chapter. I also would like to thank my parents (Mr Adel Hassan, and Mrs Afaf Zaky) for their continuous support and encouragement.
REFERENCES Adjeroh, D., & Lee, M. (2004). Scene-adaptive transform domain video partitioning. IEEE Transactions on Multimedia, 6(1), 58-69. Ahmed M, K. A. (1999). Video segmentation using an opportunistic approach. Multimedia Modeling, 389–405. Akutsu, A., Tonomura, Y., Hashimoto, H., & Ohba, Y. (1992). Video indexing using motion vectors. SPIE, (1992) 1522-1530. Bertini, M., Del Bimbo, A., & Pala, P. (2001). Content-based indexing and retrieval of TV news. Pattern Recognition Letters, 22(5), 503-516. Bescos, J., Cisneros, G., Martinez, J., Menendez, J., & Cabrera, J. (2005). A unified model for techniques on video-shot transition detection. IEEE Transactions On Multimedia, 7(2), 293-307. Boccignone, G., De Santo, M., & Percannella, G. (2000). An algorithm for video cut detection in MPEG sequences. SPIE Conference on Storage and Retrieval of Media Databases, 523–530. Bouthemy, P., Gelgon, M., Ganansia, F., & IRISA, R. (1999). A unified approach to shot change detection and camera motion characterization. IEEE Transactions On Circuits And Systems For Video Technology, 9(7), 1030-1044. Bruno, E., & Pellerin, D. (2002). Video shot detection based on linear prediction of motion. IEEE International Conference on Multimedia and Expo, Camara-Chavez, G., Precioso, F., Cord, M., Philipp-Foliguet, S., De, A., & Araujo, A. (2007). Shot boundary detection by a hierarchical supervised approach. 14th International Workshop on Systems, Signals and Image Processing, 2007 and 6th EURASIP Conference Focused on Speech and Image Processing, Multimedia Communications and Services. 197-200. Cheng, S. C., & Wu, T. L. (2006). Scene-adaptive video partitioning by semantic object tracking. Journal of Visual Communication and Image Representation, 17(1), 72-97. Cherfaoui, M., & Bertin, C. (1995). Temporal segmentation of videos: A new approach. SPIE, 38.
Demarty, C. H., & Beucher, S. (1999). Morphological tools for indexing video documents. IEEE International Conference on Multimedia Computing and Systems, , 2 991-1002. Divakaran, A., Ito, H., Sun, H., & Poon, T. (2003). Scene change detection and feature extraction for MPEG-4 sequences. SPIE, 545. Dugad, R., Ratakonda, K., & Ahuja, N. (1998). Robust video shot change detection. IEEE Second Workshop on Multimedia Signal Processing, 376-381. Fatemi, O., Zhang, S., & Panchanathan, S. (1996). Optical flow based model for scene cut detection. Canadian Conference on Electrical and Computer Engineering, 470-473. FENG, J., Kwok-Tung, L., & MEHRPOUR, H. (1996). Scene change detection algorithm for MPEG video sequence. IEEE International Conference on Image Processing, 821-824. Gao, X., Li, J., & Shi, Y. (2006). A video shot boundary detection algorithm based on feature tracking. Lecture Notes In Computer Science, 4062, 651. Gargi, U., Kasturi, R., & Strayer, S. (2000). Performance characterization of video-shot-change detection methods. IEEE Transactions On Circuits And Systems For Video Technology, 10(1), 1-13. Hu, Y., Han, B., Wang, G., & Lin, X. (2007). Enhanced shot change detection using motion features for soccer video analysis. 2007 IEEE International Conference on Multimedia and Expo, 1555-1558. Ionescu, B., Lambert, P., Coquin, D., & Buzuloiu, V. (2007). The cut detection issue in the animation movie domain. Journal Of Multimedia, 2(4) Katsuri, R., & Fain, R. (1991). Dynamic vision. In R. Katsuri, & R. Fain (Eds.), Computer vision: Advances and applications (pp. 469-480) IEEE Computer Society Press, Los Alamitos, California. Koprinska, I., & Carrato, S. (2002). Hybrid rule-Based/Neural approach for segmentation of MPEG compressed video. Multimedia Tools and Applications, 18(3), 187-212. Lawrence, S., Ziou, D., Auclair-Fortier, M. F., & Wang, S. (2001). Motion insensitive detection of cuts and gradual transitions in digital videos. International Conference on Multimedia Modeling, , 266 Lee, M. H., Yoo, H. W., & Jang, D. S. (2006). Video scene change detection using neural network: Improved ART2. Expert Systems with Applications, 31(1), 13-25. Lee, M. S., Yang, Y. M., & Lee, S. W. (2001). Automatic video parsing using shot boundary detection and camera operation analysis. Pattern Recognition, 34(3), 711-719. Lefevre, S., Holler, J., & Vincent, N. (2000). Real time temporal segmentation of compressed and uncompressed dynamic colour image sequences. International Workshop on Real Time Image Sequence Analysis, 56-62. Lefèvre, S., Holler, J., & Vincent, N. (2003). A review of real-time segmentation of uncompressed video sequences for content-based search and retrieval. Real-Time Imaging, 9(1), 73-98. Lienhart, R. (1999). Comparison of automatic shot boundary detection algorithms. SPIE, 290–301.
Ma, Y. F., Lu, L., Zhang, H. J., & Li, M. (2002). A user attention model for video summarization. Proceedings of the Tenth ACM International Conference on Multimedia, 533-542. Meng, J., Juan, Y., & Chang, S. F. (1995). Scene change detection in an MPEG compressed video sequence. IS&T/SPIE Symposium, Nagasaka, A., & Tanaka, Y. (1991). Automatic video indexing and full-video search for object appearance. Second Working Conference on Visual Database Systems, 113-127. Nam, J., & Tewfik, A. (2005). Detection of gradual transitions in video sequences using B-spline interpolation. IEEE Transactions On Multimedia, 7(4), 667-679. Over, P., Ianeva, T., Kraaij, W., & Smeaton, A. F. (2005). TRECVID 2005-an overview. TRECVID, , 2005 Porter, S., Mirmehdi, M., & Thomas, B. (2000). Video cut detection using frequency domain correlation. 15th International Conference on Pattern Recognition, 413-416. Rasheed, Z., & Shah, M. (2005). Detection and representation of scenes in videos. IEEE Transactions On Multimedia, 7(6), 1097-1105. Ren, W., Sharma, M., & Singh, S. (2001). Automated video segmentation. International Conference on Information, Communication, and Signal Processing, Shahraray, B. (1995). Scene change detection and content-based sampling of video sequences. SPIE, 2-13. Smeaton, A., Gilvarry, J., Gormley, G., Tobin, B., Marlow, S., & Murphy, M. (1999). An evaluation of alternative techniques for automatic detection of shot boundaries in digital video. Irish Machine Vision and Image Processing Conference (IMVIP'99), 45-60. Taniguchi, Y., Akutsu, A., & Tonomura, Y. (1997). PanoramaExcerpts: Extracting and packing panoramas for video browsing. Fifth ACM International Conference on Multimedia, 427-436. Taskiran, C., Chen, J. Y., Albiol, A., Torres, L., Bouman, C., & Delp, E. (2004). IEEE Transactions On Multimedia, 6(1), 103-118. Tian, Q., & Zhang, H. J. (1999). Video shot detection and analysis: Content-based approaches. In C. Chen, & Y. Zhang (Eds.), Visual information representation, communication, and image processing () Marcel Dekker, Inc. Vadivel, A., Mohan, M., Sural, S., & Majumdar, A. (2005). Object level frame comparison for video shot detection. IEEE Workshop on Motion and Video Computing, Volkmer, T., Tahaghoghi, S., & Williams, H. (2004). Gradual transition detection using average frame similarity. Computer Vision and Pattern Recognition Workshop, 139. Wang, H., Divakaran, A., Vetro, A., Chang, S. F., & Sun, H. (2003). Survey of compressed-domain features used in audio-visual indexing and analysis. Journal of Visual Communication and Image Representation, 14(2), 150-183.
Yeo, B. L., & Liu, B. (1995). Unified approach to temporal segmentation of motion JPEG and MPEG video. International Conference on Multimedia Computing and Systems, 2-13. Yi, H., Rajan, D., & Chia, L. T. (2006). A motion-based scene tree for browsing and retrieval of compressed videos. Information Systems, 31(7), 638-658. Yoo, H. W., Ryoo, H. J., & Jang, D. S. (2006). Gradual shot boundary detection using localized edge blocks. Multimedia Tools and Applications, 28(3), 283-300. Yu, J., Tian, B., & Tang, Y. (2007). Video segmentation based on shot boundary coefficient. 2nd International Conference on Pervasive Computing and Applications, 630-635. Zabih, R., Miller, J., & Mai, K. (1999). A feature-based algorithm for detecting and classifying production effects. Multimedia Systems, 7(2), 119-128. Zhang, H., Low, C. Y., & Smoliar, S. W. (1995). Video parsing and browsing using compressed data. Multimedia Tools and Applications, 1(1), 89-111. Zhang, H. J., Kankanhalli, A., & Smoliar, S. (1993). Automatic partitioning of full-motion video. Multimedia Systems, 1, 10-28. Zhang, W., Lin, J., Chen, X., Huang, Q., & Liu, Y. (2006). Video shot detection using hidden markov models with complementary features. 593-596. Zhang, Y. J., & Beijing, C. (2006). In Zhang Y. J., Beijing C. (Eds.), Advances in image and video segmentation IRM Press. Zhao, Z., & Cai, A. (2006). Shot boundary detection algorithm in compressed domain based on adaboost and fuzzy theory. Lecture Notes In Computer Science, 4222, 617.
Image Features from Morphological Scale-Spaces Sébastien Lefèvre University of Strasbourg — CNRS, France
ABSTRACT Multimedia data mining is a critical problem due to the huge amount of data available. Efficient and reliable data mining solutions require both appropriate features to be extracted from the data and relevant techniques to cluster and index the data. In this chapter, we deal with the first problem which is feature extraction for image representation. A wide range of features have been introduced in the literature, and some attempts have been made to build standards (e.g. MPEG-7). These features are extracted using image processing techniques, and we focus here on a particular image processing toolbox, namely the mathematical morphology, which stays rather unknown from the multimedia mining community, even if it offers some very interesting feature extraction methods. We review here these morphological features; from the basic ones (granulometry or pattern spectrum, differential morphological profile) to more complex ones which manage to gather complementary information.
Inodu With the growth of multimedia data available on personal storage or on the Internet, the need for robust and reliable data mining techniques becomes more necessary than ever. In order these techniques to be really useful with multimedia data, the features used for data representation should be chosen attentively and accurately depending on the data considered: images, video sequences, audio files, 3-D models, web pages, etc. As features are of primary importance in the process of multimedia mining, a wide range of features have been introduced in particular since the last decade. Some attempts have been made to gather the most relevant and robust features into commonly adopted standards, such as MPEG-7 (Manjunath, Copyright © 2009, IGI Global, distributing in print or electronic forms without written permission of IGI Global is prohibited.
Image Features from Morphological Scale-Spaces
Salembier, & Sikora, 2002). For the description of still images, MPEG-7 contains an heterogeneous but complementary set of descriptors which are related to various properties (e.g. colour, texture, 2-D shape, etc). In addition to well-known standards such as MPEG-7, local or global descriptions of digital images can be achieved through the use of various toolboxes from the image analysis and processing field. Among these toolboxes, Mathematical Morphology offers a robust theoretical framework and a set of efficient tools to describe and analyse images. We believe it can be a very relevant solution for image representation in the context of multimedia mining. Indeed, its nonlinear behaviour comes with several attractive properties, such as translation invariance (both in spatial and intensity domains) and other properties (e.g. idempotence, extensivity or anti-extensivity, increasingness, connectedness, duality and complementariness, etc), depending on the morphological operator under consideration. Moreover, it allows very easily the construction of image scale-spaces from which can be extracted some robust features. The goal of this chapter is not to present once again a well-known standard such as MPEG-7 but rather to focus on a specific theory, namely the Mathematical Morphology, and to review how the tools it offers can be used to generate global or local features for image representation. This chapter is organized as follows. First we recall the foundations of Mathematical Morphology and give the necessary definitions and notations. Then we present the morphological one-dimensional features which can be computed from the images either at a local or a global scale but always from a scale-space analysis of the images. In a third section we review several extensions which have been proposed to gather more information than these standard features through multidimensional morphological features. Next we focus on the implementation aspects, and give indications on the available methods for efficient processing, which is needed as soon as these features are used with multimedia indexing. We underline the potential of these features in a following section by giving a brief survey of their use in various application fields. Finally we give some concluding remarks and suggest further readings related to the topic addressed in this chapter.
Ba Mathematical Morphology is a theory introduced about 50 years ago by Georges Matheron and Jean Serra. Since then, it has been a growing and very active field of research, with its regular International Symposium on Mathematical Morphology (ISMM) taking place every two years and a half, and several recent special issues of journals (Ronse, 2005; Ronse, Najman, & Decencière, 2007).
Theoretical Foundations Basically, Mathematical Morphology relies on the spatial analysis of images through a pattern called structuring element (SE) and consists in a set of nonlinear operators which are applied on the images considering this SE. Thus it can be seen as a relevant alternative to other image processing techniques such as purely statistical approaches or linear approaches. First works in Mathematical Morphology were related to binary image processing. The theoretical framework involved initially was very logically the set theory. Within this framework, the morphological operators were defined by means of set operators such as inclusion, union, intersection, difference, etc. However, despite initial efforts leading to stack
approaches, this theory has been shown insufficient as soon as more complex images such as greyscale images were considered. So another theoretical framework, namely the (complete) lattice theory, is now widely considered as appropriate to define morphological operators (Ronse, 1990). In order to define the main morphological operators from the lattice theory viewpoint, let us note f : E → T a digital image, where E is the discrete coordinate grid (usually 2 for a 2-D image, or 3 for a 3-D image or a 2-D+t image sequence) and T is the set of possible image values. In the case of a binary image, T = {0, 1} where the objects and the background are respectively represented by values equal to 1 and 0. In the case of a greyscale image, T can be defined on , but it is often defined rather on a subset of Z, most commonly [0, 255]. In case of multidimensional images such as colour images, multispectral or multimodal images, T is defined on n or Zn, with n the number of image channels. A complete lattice is defined from three elements: a partially ordered set (T, ≥), which could be the set inclusion order for binary images, the natural order of scalars for greyscale images, etc, an infimum or greatest lower bound ∧, which is most often computed as the minimum operator (this choice will also be made here for the sake of simplicity), a supremum or least upper bound ∨, which is similarly most often computed as the maximum operator.
Once a complete lattice structure has been imposed on the image data, it is possible to apply morphological operators using a structuring pattern. It is called structuring function (SF) or functional structuring element and noted g when defined as a function on a subset of T, and called structuring element (SE) and noted b when defined as a set on E. In this chapter and for the sake of simplicity, we will assume the latter case unless otherwise mentioned, and use the so-called flat structuring elements. Let us notice however that the features reviewed in this chapter can be easily computed with structuring functions without important modification (if any).
Erosion and Dilation From these theoretical requirements, one can define the two basic morphological operators. The first one called erosion is defined as: b
∧ q∈b
where p is the pixel coordinates, e.g. p = (x, y) in 2-D images or p = (x, y, z) in 3-D images. The coordinates within the SE are denoted by q and most commonly defined in the same space as p. In binary images, erosion will reduce white areas (or enlarge black areas). In greyscale or more complex images, it will spread the lowest pixel values (i.e. the darkest pixels in case of greyscale images) while removing the highest ones (i.e. the brightest pixels in case of greyscale images). In other words, the erosion results in an image where each pixel p is associated with the local minimum of f computed in the neighbourhood defined by the SE b. The main other morphological operator is called dilation and is defined in a dual way as: b
∨ q∈b
Here the result is an image where each pixel p is associated to the local maximum of f in the neighbourhood defined by the SE b. Thus it will enlarge areas with highest values (i.e. brightest pixels) while reducing areas with lowest values (i.e. darkest pixels). Another main difference is related to the SE:  contrary to the erosion where b is considered, here the dilation is applied using the reflected SE b= { - q | q ∈ b}. In other words, the dilation can be defined as: b
∨ q∈b
Mathematical morphology is of particular interest due to the numerous properties verified by its operators. Indeed, morphological operators such as erosion and dilation (but also the more complex ones) are invariant to (spatial and greyscale) translations, and are commutative, associative, increasing, distributive, dual with respect to the complementation, and can most often be broken down into simple operators. Erosion and dilation, as many other morphological operators, require the definition of a structuring element b. This parameter has a strong impact on the results returned by an operator. Main SE shapes are diamond ♦ , square  , cross , disc , and line — or |. A pixel and its 4- or 8-neighbourhood correspond respectively to a 3×3 pixel diamond- or square-shaped SE, also called elementary isotropic (or symmetric) SE. The shape of the SE can also be defined from a basic shape and an homothetic parameter (or SE size), so we will use the notation bλ = λb to represent a SE of shape b and size λ. For most of the SE shapes, bλ can be generated from λ – 1 successive dilations, i.e. bλ = δ(λ – 1)(b). This is obviously not
true with disc-shaped SE, where •λ = { p : d ( p, o) ≤ λ } with o the origin or centre of the disc, and d the exact or approximated Euclidean distance. Moreover, we can also consider a growing factor κ between successive λ sizes, i.e. bλ = κλb. For the sake of simplicity, the b parameter may be omitted in formulas, e.g. ελ = εb and δλ = δb . For elementary structuring elements (e.g. 1 or ♦1 ), we may also omit the λ = λ λ 1 1 parameter, i.e. ε = ε1 and δ = δ1, thus resulting in elementary erosion and dilation. We also state that ε 0( f ) = δ0( f ) = f. Figure 1 illustrates the basic structuring elements used in mathematical morphology. Since morphological operators are often applied several times successively, we will use the notation (n) ε ( f ) and δ(n)( f ) to denote respectively the n successive applications of ε and δ on f. In other words, ε(n) ( f ) = ε(1) (ε(n–1) ( f )) and δ(n) ( f ) = δ(1) (δ(n–1) ( f )), with ε(1) = ε and δ(1) = δ.
Figure 1. Illustrative examples of basic SE with increasing size λ
Figure 2. Binary erosion and dilation with square-shaped SE  of increasing size λ
Figure 3. Greyscale erosion and dilation with square-shaped SE  of increasing size λ
Even if most of the features presented in this chapter will be defined with flat SE b (i.e. sets), they can be easily defined also with structuring functions (SF) g. In this case, the basic operations are defined as: g
( f )( p ) =
f ( p + q ) - g (q ),
p∈E
f ( p - q ) + g (q ),
q ∈ supp( g )
and g
with supp(g) representing the support of g, i.e. the points for which the SF is defined.
Figure 2 and 3 illustrate the effects of morphological erosions and dilations applied respectively on binary and greyscale images with 8-connected elementary SE  of increasing size λ.
Opening and Cosing Erosion and dilation are used to build most of the other morphological operators. Among these operators, we can mention the well-known opening and closing filters where erosion and dilation are applied successively to filter the input image, starting with erosion for the opening and with dilation for the closing. Opening is defined by b
(f)=
 b
( b ( f ))
while closing is defined by b
These two operators respectively result in a removal of local maxima or minima and return filtered images which are respectively lower and higher than the input image. This is called the anti-extensivity property of the opening with γ( f ) ≤ f and the extensivity property of the closing with f ≤ φ( f ) (with the ≤ relation being replaced by the ⊆ relation if set theory is considered). Moreover, both opening and closing share some very nice properties (in addition to those of erosion and dilation). First they have the idempotence property since γb(γb( f )) = γb( f ) and φb(φb( f )) = φb( f ). Second they also ensure the increasingness property, i.e. if f ≤ g, γb( f ) ≤ γb(g) and φb( f ) ≤ φb(g). Since they verify these two properties, they are called morphological filters. Figure 4 and 5 illustrate the effects of morphological openings and closings applied respectively on binary and greyscale images with 8-connected elementary SE  of increasing size λ. The main concern with these two morphological filters is their very strong sensitivity to the SE shape, which will have a straight influence on the shapes visible in the filtered image. In order to avoid this problem, it is possible to involve the so-called algebraic filters which are a generalization of the morphological opening and closing defined above. For the sake of conciseness, we will use in this chapter the operator ψ to represent any morphological filter (e.g. γ or φ).
Algebraic F ilters The term algebraic opening (respectively closing) is related to any transformation which is increasing, anti-extensive (respectively extensive) and idempotent. Thus morphological (also called structural) opening and closing are a particular case of algebraic filters. The two main ways of creating algebraic opening and closing are recalled here. The first option relies on opening and closing by reconstruction, which are useful to preserve original object edges. More precisely, let us note εg(1)( f ) the geodesic erosion of size 1 of the marker image f with respect to the mask image g: εg(1)( f )(p) = ε(1)( f )(p) ∨ g(p) where the elementary erosion is limited (through a lower bound) within the mask, i.e. εg ≥ ε. 37
Figure 4. Binary opening and closing with square-shaped SE  of increasing size λ
Figure 5. Greyscale opening and closing with square-shaped SE  of increasing size λ
Similarly, the geodesic dilation of size 1 is defined by:
δg(1)( f )(p) = δ(1)( f )(p) ∧ g(p)
where the elementary dilation is limited (through an upper bound) within the mask, i.e. δg ≤ δ. These two operators are usually applied several times iteratively, thus we will use the following notations: εg(n)( f ) = εg(1)(εg(n–1) (f )) and δg(n)( f ) = δg(1)(δg(n–1) (f )) 38
From these two geodesic operators, it is possible to build reconstruction filters ρ which consist in successive applications of these operators until convergence. More precisely, the morphological reconstruction by erosion and by dilation are respectively defined by: ρgε( f ) = εg(j) (f ) with j such as εg(j)( f ) = εg(j–1) (f ) and ρgδ( f ) = δg(j) (f ) with j such as δg(j)( f ) = δg(j–1) (f ) Based on these reconstruction filters, new morphological filters which preserve object edges can be defined. Indeed, the opening by reconstruction γbρ( f ) of the image f using the SE b is defined as: γbρ( f ) = ρfδ(εb( f )) while the closing by reconstruction φbρ( f ) is defined by: φbρ( f ) = ρfε(δb( f )) In other words, for the opening (resp. closing) by reconstruction, the image f is used both as input for the first erosion (resp. dilation) and as mask for the following iterative geodesic dilations (resp. erosions). Contrary to their standard counterparts, these morphological filters by reconstruction remove details without modifying the structure of remaining objects. The second option consists in computing various openings (respectively closings) and select their supremum (respectively infimum). Here each opening is related to a different condition or SE. Let us consider a set B = (b)i of SE, we can then define respectively the algebraic openings and closings by: B
( f ) = ∨ b ( f ) b∈B
and B
( f ) = ∧ b( f ) b∈B
= B and = B with λB = (λb) . and we will use the shortcuts i Among the main algebraic filters, we can mention the area-based operators, which have the very interesting property to be invariant to the shape of the SE b under consideration. To do so, they consider the whole set of all SE of a given size λ, thus resulting in the following operators: a
( f ) = ∨{ b ( f ) | b is connected and card(b) = } b
( f ) = ∧{ b ( f ) | b is connected and card (b) = } b
Area filters ψa are a special case of more general attribute filters ψχ, with the attribute or criterion χ to be satisfied being related to the area, i.e. the Boolean function χ(b, λ) = {card(b) = λ}. Other attribute filters can be elaborated, in particular shape-related ones, involving for instance the perimeter χ(b, λ) = {card(b – ε(b)) = λ} or the moment of inertia
(b, ) = {∑ d (q, o) = } q∈b
(with d the Euclidean distance and o the origin of the SE b). More generally, attribute filters can be defined as:
Figure 6. Comparison between binary standard (structural) filters, filters by reconstruction, and area filters with increasing λ parameter
Figure 7. Comparison between greyscale standard (structural) filters, filters by reconstruction, and area filters with increasing λ parameter
( f ) = ∨{ b ( f ) | b is connected and ( b, )} b
( f ) = ∧{ b ( f ) | b is connected and (b, )} b
In Figure 6 and 7 are given some visual comparisons between structural filters, filters by reconstruction, and area filters, respectively on binary and greyscale images. We can observe the interest of filters by reconstruction and area filters to limit the sensitivity to the SE shape.
Apart from these basic operators, mathematical morphology offers a wide range of operators or methods to process images. We can cite the morphological gradient, the hit-or-miss transform to perform template matching or object skeletonisation, the watershed or levelling approaches for segmentation, the alternating sequential filters (ASF) for image simplification, etc. In this chapter, we will focus on morphological features extracted from the previously presented operators and we will not deal with some other morphological operators. The interested reader will find in the book from Soille (2003) a good overview of the morphological toolbox for image processing and analysis.
STAanda Image features are most often dedicated to a single type of information (e.g. colour, texture, spatial distribution, shape, etc). The most famous example is undoubtedly the image histogram which measures the probability density function of the intensity values in the image and which can be analysed through various measures (e.g. moments, entropy, uniformity, etc). However it is limited to intensity distribution and does not take into account the spatial relationships between pixels. On the opposite, approaches known under the terms of pattern spectra, granulometries, or morphological profiles are built from series of morphological filtering operations and thus involve a spatial information. We review here these different (either global or local) features in an unified presentation.
Multiscale Representation Using Morphological Filters We have introduced in section 1 the main morphological filters (i.e. opening and closing filters) which aim at removing details in the image, either bright details (with the opening) or dark details (with the closing), by preserving or not object edges. Thus they can be used to build multiscale representations of digital images by means of mathematical morphology. Most of these multiscale representations can be seen as nonlinear scale-spaces, if some of the original constraints are relaxed. The concept of scale space introduced by Witkin (1983) is defined as a family of filtered images {ϒt ( f )}t ≥0, with ϒ 0 ( f ) = f and various axioms (Duits, Florack, Graaf, & Haar Romeny, 2004), the multiscale representation being computed most often by means of a convolution by a Gaussian kernel:
ϒt ( f )( x, y ) = f ( x, y ) * g ( x, y, t ) = ∫
+∞
-∞
1 f (u, v) e 2 t2
( x -u )2 + ( y - v )2 2t 2
du dv
The main properties of a scale-space are relatively compatible with some of the morphological operators as pointed out by the work of Jackway (1992): • • •
causality, i.e. no additional structures are created in the image when t increases (indeed both height and position of extrema are preserved); recursivity
ϒt ( ϒ s ( f )) = ϒ s ( ϒt ( f )) = ϒt + s ( f ), ∀t , s ≥ 0 increasingness
f ≤ g , ϒt ( f ) < ϒt ( g ), ∀t > 0
• either extensivity ϒt ( f ) ≥ f , ∀t ≥ 0 or anti-extensivity ϒt ( f ) ≤ f , ∀t ≥ 0 which leads respectively to t1 ≤ t2, ¡t ( f ) ≤ ¡t ( f ), ∀t >0 1 2 t1 ≤ t2, ¡t ( f ) ≥ ¡t ( f ), ∀t >0 1
Thus some scale-spaces can be built straightforward from successive applications of morphological operators (such as erosion and dilations (Jackway & Deriche, 1996), or ASF (Matsopoulos & Marshall, 1992; K. Park & Lee, 1996; Bangham, Ling, & Harvey, 1996)), or using advanced morphological representations such as max-tree (Salembier, Oliveras, & Garrido, 1998). Here we will use the term (morphological) scale-space even for scale-spaces where the recursivity property is replaced by the absorption law defined by Matheron (1975) which is relevant for morphological filters:
∀t , s ≥ 0, ϒt ( ϒ s ( f )) = ϒ s ( ϒt ( f )) = ϒ max(t ,s ) ( f ) In addition to this property, the idempotence property also holds:
ϒt ( ϒt ( f )) = ϒt ( f ) A wide range of morphological operators can lead to scale spaces, such as openings and closings (Chen & Yan, 1989). Figure 8 illustrates the difference between a scale-space built with Gaussian and morphological (bottom) closing filters. One can clearly see the interest of morphological scale-spaces to retain object edges even with basic (i.e. structural) morphological filters. So morphological scale-spaces can be built by applying some morphological operators ϒt on the input image f, with increasing parameter t. In the morphological framework, t is directly related to the size λ of the structuring element b, and we will use the notation ϒ λ = ϒbλ . Indeed, for a given morphological filter ψλ, λ denotes the size of the SE, i.e. the size of neighbourhood used to compute minima or maxima (or the filter window). Then when λ increases, the morphological filter ψλ will remove more and more details from the input image (as shown by Figure 9), thus respecting the main property of scale-spaces (i.e. causality). Moreover, to satisfy the absorption property, the SE b under consideration has to be a compact convex set (Matheron, 1975). Let us note Π ( f ) = {Π ( f )} ≥0 the morphological scale-space, i.e. the series of successive filtered images using ψ with growing SE size λ:
Π (f)= Π (f) |Π (f)=
0≤ ≤ n
where ψ0( f ) = f and n + 1 is the length of the series (including the original image). This Π series is a nonlinear scale-space, with less and less details as λ is increasing from 0 to n. Instead of using a single SE, a SE set or generator B = (b)i can be used (Matheron, 1975), thus resulting in series made from algebraic filters:
Figure 8. Comparison between Gaussian (top) and morphological (bottom) scale-spaces
= B. The initial definition leading to Euclidean granulometries was considering the same with growing factor κ for all SE bi in B, i.e. (bi)λ = λκibi. It is also possible to make the κ factor depends on b, thus either B = (b, κ)i (where (bi)λ = λκibi) for homogeneous multivariate series (Batman & Dougherty, 1997) or B = (b, t, k)i (with ti being a strictly increasing function of κi, thus (bi)λ = λti(κi)bi for heterogeneous multivariate series (Batman, Dougherty, & Sand, 2000). Moreover, these series Πψ can be made using any ψ filter (see (Soille, 2003) for a deeper review of morphological filters). Thus, Π is also anti-extensive for any opening filter and extensive for any closing filter, resulting respectively in lower and lower images or higher and higher images as λ increases. Indeed, if we have λ1 ≤ λ2, then Π ( f ) ≤ Π ( f ) and Π 2 ( f ) ≥ Π 1 ( f ). 2 1 In order to avoid their highly asymmetric behaviour (Πψ is either anti-extensive or extensive), it is possible to gather opening and closing series to generate a single Π series of length 2n + 1:
Figure 9. Details removal by means of successive openings (top) and closings (bottom) by reconstruction
 Π - ( f ),   Π ( f ) = Π ( f ) | Π ( f ) = Π ( f ),  f,  
0 =0
    - n ≤
≤n
We can also build these symmetric series using any of the pair of opening/closing filters, and we will denote by Π ρ, Πα, Πa, Πχ the series created respectively with ψρ, ψα, ψa, ψχ. An illustration of this kind of dual series is given in Figure 10. Let us note that the merging of openings and closings can also be made by means of alternate sequential filters (ASF), which consists in successive openings and closing with SE of increasing size λ. Using ASF to compute the Π series results in the following definition:
Figure 10. From top left to bottom right, Π series of length 2n + 1 with n = 7 using structural filters γ and φ and 4-connected elementary SE ♦
   /2 (Π ASF if -1 ( f ))      Π ASF ( f ) = Π ASF ( f ) | Π ASF ( f ) =  ( -1)/2 (Π ASF if -1 ( f ))   if  f , 
is odd is even =0
    0≤
Of course the morphological filters γ and φ can again be replaced by any of their variants, e.g. their ρ reconstruction-based counterparts γρ and φρ, thus resulting in the ΠASF series. ASF have been proven to be a specific case of more general concepts, M- and N- sieves (Bangham et al., 1996). Another related feature is the lomo filter (Bosworth & Acton, 2003), which consists in the mean of two ASF applied until convergence, one starting with an opening (i.e. being defined as φnγn...φ1γ1) and the other starting with a closing operation (i.e. being defined as γnφn...γ1φ1). It is also possible to associate to each scale λ the difference between the union and the intersection of the ASF with successive λ size (Xiaoqi & ASF ∧ Π ASF Baozong, 1995), i.e. (Π ASF ∨ Π ASF -1 ) - (Π -1 ). Features can be extracted directly from the series Πψ (or from their symmetrical version Π), but most often it is more relevant to compute the differential version Δψ of this series where removed details are emphasized for each λ size. For anti-extensive filters such as openings, we have:
D (f)= D (f) |D (f)=Π
( f )-Π ( f )
while for extensive filters such as closings, we have:
D ( f ) = D ( f ) | D ( f ) = Π ( f )-Π
thus resulting in a single definition
with the assumption Δψ0 = 0. In this series, a pixel p will appear (i.e. have a non null value) in Δψλ ( f ) if it is removed by ψλ, the morphological filter ψ of size λ (or in other words, if it was present in ψλ–1( f ) but not anymore in ψλ( f )). Similarly for the Πψ series, it is possible to compute a symmetric version of Δψ by taking into account both opening and closing filters:
 D - ( f ),   D( f ) = D ( f ) | D ( f ) = D ( f ),  0,  
As an illustration, Figure 11 is the Δ counterpart of the Π series presented in Figure 10 using structural filters with diamond shaped-SE, while Figure 12 is the differential series built from the one presented in Figure 9 using filters by reconstruction. We also have to notice that even basic operators (not necessarily filters) can be used to build morphological series. Indeed, one can apply successive erosions ε or dilations δ to build a Π or Δ series:
Π (f ) = Π (f )|Π (f ) =
Figure 11. From top left to bottom right, Δ series of length 2n + 1 with n = 7 using structural filters γ and φ and 4-connected elementary SE Grey levels have been inverted and normalised for the sake of readability.
where ν denotes the basic morphological operator under consideration (e.g. ε or δ). The properties of these series will be however weaker than the previous series built from morphological filters. Depending on the desired properties of the series, one can even relax the constraint on the shape (compactness and convexity) of the SE in use. Among these operators which do not belong to morphological filters, we can even use difference operators. For instance, by considering the morphological gradient Gλ( f ) = δλ( f ) – ελ( f ) with increasing scale λ, we can build some morphological fractal measures (Soille, 2003). Another example is related to top-hats and the so called hat scale-spaces (Jalba, Wilkinson, & Roerdink, 2004). More precisely, the top-hat by opening is defined by τγ( f ) = f – γ( f ) while the top-hat by closing (or bottom-hat) is defined by τφ( f ) = φ( f ) – f. From these hat scale-spaces can be extracted the so called fingerprints introduced by Jackway and Deriche (1996) as local maxima or minima in the images filtered at different scales. Fingerprints can also be obtained by the filters by reconstruction (Rivas-Araiza, Mendiola-Santibanez, & Herrera-Ruiz, 2008). Various robust features can also be extracted from the analysis of scale-spaces made with top- and bottom-hat by reconstruction (W. Li, Haese-Coat, & Ronsin, 1997). Of course we Figure 12. Details emphasized by means of differences between successive openings and closing by reconstruction. Grey levels have been inverted and normalised for the sake of readability.
can also build other versions of these operators, e.g. using filters by reconstruction γρ and φρ. Any of the ψ resulting series Πτ can then be analyzed in a similar manner as the standard Π series. These different series are the basis of the standard morphological features widely used in image analysis and visual pattern recognition. These features are computed either at a local scale or a global scale.
Local Morphological Features The simplest way to extract morphological features from a digital image f using one of the Πψ or Δψ series consists in associating to each pixel p the vector Π ( f )( p ) = (Π ( f )( p ))0≤ ≤ n of size n + 1. In the remote sensing field, this principle led to the so called differential morphological profile (DMP) proposed by Pesaresi and Benediktsson in (Pesaresi & Benediktsson, 2001; Benediktsson, Pesaresi, & Arnason, 2003) which is computed using the reconstruction-based differential series: DMP( f )(p) = Δρ( f )(p) Figure 13 illustrates the behaviour of the DMP feature for pixels belonging to different areas of an image. This feature is a kind of structural feature, and an interesting alternative to spectral or textural features. Its size (initially equal to 2n + 1) can be strongly reduced by considering only the few most important maxima. It has been shown in (Benediktsson et al., 2003) that using only the first and second maximum values of ∆ρ( f )(p) for each pixel p ensures satisfactory recognition rates in supervised classification of remotely sensed images. Moreover, an attempt has been made in (Chanussot, Benediktsson,&Pesaresi, 2003) to use reconρ struction-based alternate sequential filters through the Δ ASF series as an alternative to the original DMP, ρ thus defining DMPASF( f )(p) = ΔASF ( f )(p). Alternatively, ASF-based scale-space representations can also be computed from the area-filters (Acton & Mukherjee, 2000b). Figure 13. Input image with 4 sample points (left) and corresponding DMP values (right) using 20 openings (negative indices) and 20 closings (positive indices).
In case of binary images, pixelwise feature extraction from morphological scale-space may consist in assigning to each pixel p of f the scale, or size λ, for which the filter (e.g. opening) manages to remove the pixel, thus resulting in the so called opening transform when considering successive applications of the opening filter (similarly, the closing transform is related to the scale-space obtained from successive closings). The definition is given by: Ξψ( f )(p) = max{λ ≥ 0 | ψλ( f )(p) > 0} with the convention Ξψ( f )(p) = 0 if f (p) = 0.
Figure 14. Illustration of various opening and closing transforms obtained with different filters and SE shapes
In other words, it can be easily computed by analysing the series Π and looking at each pixel p for the first image (with filter size λ + 1) such as Π +1 ( f )( p ) = 0. Figure 14 illustrates the opening and closing transforms considering various morphological filters and SE. The extension of this principle to greyscale images can lead to the definition of opening trees (Vincent, 2000). However, if one wants to keep a single value for each pixel, it is recommended to select the λ value resulting in the biggest drop in greyscale intensity when applying ψλ( f )(p), thus following the recommendation of Pesaresi and Benediktsson with DMP.
Global Morphological Features Besides the use of morphological series directly on a per-pixel basis, it is possible to involve them in global image features. In this case, pattern spectra and granulometries are certainly the most famous morphological features in the image analysis community (Soille, 2003). Granulometries and antigranulometries (also called size and anti-size distributions) are built by gathering the values of the series over all pixels p of the filtered image ψ( f ) through a Lebesgue measure, for instance a volume or sum operation. In the particular case of binary images, the image volume can either be computed as the sum of pixel values or as the amount of white pixels (or 1-pixels).The granulometry uses openings:
  Ω ( f ) = Ω ( f ) | Ω ( f ) = ∑ Π ( f )( p)  p∈E  0 ≤
while the antigranulometry relies on closings:
≤ n
From the properties of morphological filters, we can observe that Ωγ is monotonically decreasing while Ωφ is monotonically increasing. In order these measures to be invariant to image size and to represent cumulative distribution functions, they are worth being normalized, thus resulting in the new definition:
 Ω ( f ) Γ ( f ) = Γ ( f ) | Γ ( f ) = 1  Ω 0 ( f ) 0 ≤ 
with ψ denoting either γ or ψ. In Figure 15 are given the granulometric curves Γ for both opening and closing filters, considering the main standard SE. Another very interesting morphological global feature is the pattern spectrum Φ introduced by Maragos (1989), also called pecstrum (Anastassopoulos & Venetsanopoulos, 1991). It can be seen as the morphological counterpart of the well-known image histogram. Instead of measuring the distribution of intensities within an image, it aims at measuring the distribution of sizes (and to a lesser extent, of shapes). To do so, it gathers values of the differential series ∆ over all pixels:
  Φ ( f ) = Φ ( f ) | Φ ( f ) = ∑ D ( f )( p)  p∈E  - n ≤
Figure 15. Input image (left) and corresponding granulometric curve Γ (right) using 20 openings (negative indices) and 20 closings (positive indices).
Figure 16. Input image (left) and corresponding pattern spectrum Δ (right) using structural filters, filters by reconstruction and area filters
and the normalization ensures measures independent of the image size:
 Φ ( f ) Λ( f ) = Λ ( f ) | Λ ( f ) =  Ω 0 ( f ) - n ≤ 
Moreover, let us notice that the pattern spectrum can be easily computed as the histogram of the opening (or/and closing) transform. In Figure 16 are given the pattern spectra Λ for both opening and closing filters, using respectively structural filters, filters by reconstruction and area filters. Moreover, Figure 17 illustrates the relevance of the pattern spectrum in case of image with similar greylevel distribution. 52
When dealing with greyscale images, it is also possible to involve greyscale (or volume) SE, thus dealing with spatial but also to a lesser extent with intensity information. Moreover, some scalar attributes can be extracted from the previous 1-D morphological series. As representative examples, we can cite the average size and roughness (Maragos, 1989) computed respectively as the mean and the entropy of the signal, or the statistical moments computed on the granulometric 1-D curve and called granulometric moments (Dougherty, Newell, & Pelz, 1992; Sand & Dougherty, 1999). Besides the use of morphological filters to build global morphological filters, it is also possible to involve any operator ν and to exploit the morphological series (e.g. Πν) of images processed with this operator. In this case, the obtained series Ων are called pseudo-granulometries since they do not respect the fundamental requirements of granulometries (Soille, 2003). As another representative example of using Πν series, we can cite the covariance feature K, a morphological counterpart of the autocorrelation operator. To compute this feature, the SE b under consideration consists in a set of two points p1 and p2

and is defined by both a size 2λ =‖p1 p2 ‖     K v ( f ) = K v ( f ) | K v ( f ) = ∑ Π p∈E 
 ,v
and an orientation:
 ( f )( p)  0 ≤
   v = p1 p2 /‖p1 p2 ‖
Figure 17. Two input images (left) with similar histograms (top right) but different pattern spectra Λ (bottom right) Grayscale Histogram
Target squares Small squares
Gray level
Pattern Spectrum
0 SE size
where  ,v
  ( f )( p ) = f ( p - v ) ∧ f ( p + v )
This feature is illustrated by Figure 18. Another definition of the covariance has been given by Serra (1982) where the autocorrelation function is used, thus resulting in the operator ε' defined by
  ′ ,v ( f )( p ) = f ( p - v )· f ( p + v )
where the intersection ∧ is replaced by a product ˙ operation. We have not dealt yet with the case of semi-local features, i.e. features computed on an intermediate scale. The processing units at this scale are neither the single pixels nor the whole image, but rather parts of it, e.g. blocks or image regions. In this case, the global features can be computed similarly but within a limited area of the image, thus resulting in one morphological feature per block or region. An illustrative example of this approach is the work from Dougherty where each pixel is characterized by the granulometry computed from its neighbouring window (Dougherty, Pelz, Sand, & Lent, 1992). Even if these features (either local or global) appear as particularly relevant alternatives to usual image features such as image histograms, wavelets, or other textural features (just to mention a few), they still are limited to a single evolution curve and so cannot consider simultaneously several dimensions. More precisely, they deal only with the structural information extracted from morphological filters applied with growing SE sizes.
Muidi ion Despite their broad interest in image representation, the well-known morphological features reviewed so far are limited by their one-dimensional nature (i.e. they are computed as single evolution curves and thus cannot consider simultaneously several dimensions). 
Figure 18. Input image (left) and corresponding covariance curves K v (right) using 25 vector sizes λ  and 4 different orientations q for v .
We review here some recent multidimensional extensions which allow to build n-D (mostly 2-D) series of morphological measures. These extensions help to gather complementary information (e.g. spatial, intensity, spectral, shape, etc) in a single local or global morphological representation.
Size-Shape In the morphological series defined in the previous section, a unique parameter λ was considered for measuring the size evolution, through the SE bλ. We have indicated various ways to build the series of SE bλ based on the increasing λ parameter. Here we consider the SE κ as a growing factor of the initial shape b, i.e. b = ( -1) (b) with various shapes for κ (e.g. one of the basic shapes introduced in section 1.2). Let us notice that the size of κ has to be rather small to build measurements at a precise scale (or conversely large for coarse measurements) since it represents the growing factor of the SE series. Moreover, one has also to set the initial condition b, i.e. the initial SE, which can be of arbitrary shape, even equal to κ (thus resulting in the definition given in section 1.2). This definition assuming a single size varying parameter λ prevents us from performing accurate measurements. Indeed, it is not adequate to elliptical or rectangular shapes for instance, where the two independent axes should be taken into account. So several attempts have been made to build bivariate morphological series, thus allowing to obtain size-shape measurements. Figure 19. Three input images (top) and their respective 2-D Δ feature (middle). As a comparison, standard pattern spectra using square SE (bottom left), horizontal line SE (bottom centre) and vertical line SE (bottom right) are also given.
-D pattern spectrum
0
SE width
  0 0
SE height
Pattern spectrum with square SE
Squares Horizontal rectangles Vertical rectangles
 SE size
Pattern spectrum with vertical line SE
Pattern spectrum with horizontal line SE
Lefèvre, Weber, & Sheeren (2007) consider structuring elements with two different size parameters α and β that vary independently. More precisely, a way to define the 2-D series of SE bα,β is given by b , = (1 -1) ( (2 -1) (b)) = (2 -1) ( (1 -1) (b)) with κ1 and κ2 denoting the structuring elements used as growing factors in the two dimensions, and b the initial SE. In the case of rectangular SE series, a relevant choice for κ1 and κ 2 consists in 1-D SE such as horizontal and vertical lines respectively (with a length proportional to the degree of coarseness desired) and an initial rectangular SE b. The new Π series built using the 2-D set of SE bα,β is then computed as:
Π (f)= Π
(f )|Π
0≤ ≤ m 0≤ ≤ n
where the application of ψ on f with a SE bα,β is noted ψα,β( f ) and with the convention ψ0,0( f ) = f. Similarly, the Δ series measures the differential in both size dimensions:
 D ( f ) = D 
(f )|D
1 2Π 2
-1, -1
( f )-Π
-Π
, -1
 (f) 00≤≤
≤m ≤n
where D ,0 = D D 0, = D D 0,0 = 0. Figure 19 illustrates the potential interest of such 2-D features for sample images where standard granulometries are irrelevant. A similar approach has been proposed by Ghosh and Chanda (1998) who introduce conditional parametric morphological operators, and who build a 2D set of SE with increasing size, both on the horizontal and vertical dimensions. From this set of SE they finally compute the bivariate pattern spectrum for binary images. Bagdanov and Worring (2002) introduce the same feature under the term rectangular granulometry, while a slightly different definition has been given by Barnich, Jodogne, & Droogenbroeck (2006) to limit the SE to the largest non-redundant rectangles within the analysed object (in binary images). Moreover, a more general expression of m-parametric SE has been used in (Gadre & Patney, 1992) to define multiparametric granulometries. Batman et al. in (Batman & Dougherty, 1997; Batman et al., 2000) propose an alternative definition of α this series using Euclidean series Πψ ( f ) with the set of SE B = {–1,|1} where – and | denote respectively elementary horizontal and vertical SE. Moreover, they also introduce a univariate series by combining through the sum operations two series of SE bα and cβ built from initial SE b and c:
(f )+
Urbach, Roerdink, & Wilkinson (2007) also propose to combine size and shape information in a single 2-D granulometry. They rely on attribute filters (Breen & Jones, 1996) ψχ and use a max-tree representation (Salembier et al., 1998) of the image for computational reasons. Their 2-D series can be defined as:
(f)= Π
(f )∧
where the two criteria χ1 and χ2 are respectively related to the area (i.e. defining size) and the ratio of the moment of inertia to the square of the area (i.e. defining shape). While the first dimension (indexed by α and related to the criterion χ1) is related to size and respect the axioms of morphological scale-spaces, 56
the second dimension (indexed by β and related to the criterion χ2) is related to shape and should be scale-invariant, thus the increasingness property is replaced by the scale-invariance property, i.e. Sλ(Yt( f )) = Yt(Sλ( f )), ∀t > 0 with the transform Sλ( f ) being the scaling of the image f by a scalar factor λ.
S Besides the size of the SE, one can also vary its orientation (Werman & Peleg, 1985). Naturally this is relevant only with anistropic structuring elements and not for disc-shaped SE, nor with area-based filters. Let us note bλ,θ a SE of size λ and orientation θ. This SE is built from a rotation of the initial SE bλ with an angle θ, i.e. ∠(bλ, bλ,θ) = θ with ∠(b1, b2) the measured angle between orientations of b1 and b2. Based on this principle, the morphological series is then defined as:
0≤ ≤ n 0≤ ≤
where {θ0,...,θm} represents the set (of cardinality |θ|) of orientations considered, and ψλ,θ is a shortcut for ψb . Figure 20 illustrates the interest of such size-orientation features when the standard granulometry λ,θ is useless. Apart from the most simple angles (i.e. θ = kp/4), one has to tackle very carefully the problem of discretisation for rotated SE. Accurate approximations can be obtained by periodic lines (see the work from Jones and Soille (1996)) and require the use of several SE to get an accurate discrete representation of a continuous segment (Soille & Talbot, 2001). It is also possible to retain for each pixel at a given size, only the maximum or minimum value from the results returned by the morphological filter with the various orientations (Maragos, 1989). In this case however, the result is a 1-D series similar to the one which could be obtained by means of radial filters (Soille, 2003). Finally, from these size-orientation measures, other features can be extracted such as orientation maps proposed by Soille & Talbot (2001).
Sze-Spectral or Size-Colour Since digital images contain very often spectral or colour information, it is worth involving the spectral signature or colour of each pixel in the computation of the morphological representation. To do so, it is possible to first compute a morphological signature for each of the k spectral components (or bands) and then to combine these k signatures into a single one. With this two-step approach, the morphological series Π can be expressed as:
Π (f ) = Π
(f ) | Π
(f ) =
(f )
1≤ ≤ k 0≤ ≤ n
where fΩ is a greyscale image representing the Ωth spectral component of the multispectral or colour image f = {f w}1≤Ω≤k. In this definition, morphological filters are applied independently on each image band, thus the marginal strategy is used and the correlation among the different spectral channels is completely ignored. Moreover it can result in new spectral signatures or colours in the filtered images. To avoid these limitations, it is possible to rather consider a vectorial ordering when applying the morphological operators on the multispectral input image f (Aptoula & Lefèvre, 2007a). The purpose of a vectorial ordering is to give a way to order vectors and thus to compute vectorial extrema by means
of the two operators supv and infv. Assuming a given vectorial ordering, the fundamental dilation and erosion operators are written: v b
(f )( p ) = inf v f ( p + q ),
(f )( p) = sup v f ( p - q),
q∈b
and from these operators it is possible to write all vectorial versions of the morphological operators described previously in this chapter. The new size-spectral morphological series is finally computed as:
where v (f ) = ( f ) in the specific case of a marginal ordering. A comparison of marginal and vectorial strategies is given in Figure 21, considering a similar size distribution but a different spatial distribution in each colour band. For a comprehensive review of vectorial orderings and multivariate mathematical morphology, the reader can refer to the survey from Aptoula & Lefèvre (2007a). An example of colour pattern spectrum can be found in (Ledda & Philips, 2005) while a comparison between several vectorial orderings has a also been proposed recently by Gimenez and Evans(2008) using the series ΠASF ( f ). Nes and d'Ornellas (1999) consider colour pattern spectra with linear SE of variable directions (at each scale λ, the maximum pattern spectrum among the various orientations is selected). Rivest (2006) deals with radar signals and propose adequate granulometry and power spectrum by introducing a vector ordering dedicated to complex data.
Size-Intensity In greyscale images, the pixel intensity values are used either directly (at local scale) or gathered with the sum operator (at global scale). So the distribution of intensity values in the image is not taken into account with standard morphological features, which can be a real issue since intensity distribution (usually measured by an histogram) is a key feature to represent image content. Computing the histogram on morphological scale-spaces has been proposed by Lefèvre (2007) to take into account both size and intensity distributions. To do so, let us use the Kronecker delta function: i, j
if i = j if i ≠ j
1 = 0
and the histogram function h f : T → :
hf ( ) = ∑ p∈E
, f ( p)
which measures the number of occurrences of each greylevel η in the image f. Alternatively, we can also use the normalised histogram function h′f : T → [0,1] where
h′f ( ) =
hf ( ) | supp( f ) |
with |supp( f )| the cardinality of the support of f, i.e. the number of pixels in f. The formulation of the 2-D size-intensity morphological feature is then given by the following Π series:
(f)=h
0≤ ≤ 0≤ ≤ n
where { 0 , …, m } represents the different greylevels or bins in the histogram. Figure 22 shows the relevance of size-intensity morphological features when both granulometry and histogram are irrelevant. For the sake of clarity, greylevel 0 (i.e. black pixels) has been omitted in the plots. Its derivative counterpart can be given by the following Δ series:
D (f)= D
( f )-
-1 (
This feature can be seen as a morphological alternative to the very effective multiresolution histograms computed from Gaussian linear scale-spaces (Hadjidemetriou, Grossberg, & Nayar, 2004). Spatial and intensity information can also be gathered by the use of structuring functions (SF) as proposed by Lotufo and Trettel (1996). More precisely, let us define the SF gλ, η as a non-planar cylinder of radius λ and amplitude η. A size-intensity feature is then built using various λ and η values:
where ψλ, η is here a shortcut for ψg . It has been noticed in (Lotufo & Trettel, 1996) that both the classic λ,η histogram and the pattern spectrum can be derived from this measure by considering respectively λ = 0 (i.e. a single pixel) and η = 0 (i.e. a flat disc-shaped SE). A similar feature called granold has been proposed by D. Jones and Jackway (2000) by first decomposing the greyscale image into a stack of binary images and then computing the granulometry for each binary image (i.e. at each greyscale threshold), thus resulting in the following series:
(T ( f ))
where Tη denotes the thresholding function:
1 T ( f )( p ) =  0
if ( f p) ≥ if ( f p)
< mwhen there are cross junction segments that are not contained within any single junction). In O ', we construct a path from x 'to y 'corresponding to Γ( x, y; O) as follows (e.g. Fig. 6 (b)): C ( x ', y '; O ') = [Γ( p '0 , p '1 ; R '1 ), Γ( p '1 , p '2 ; R '2 ),..., Γ( p 'l -1 , p 'l ; R 'l )]br / br / Note that C ( x ', y '; O ') is not necessarily the shortest path in O '. Denote d ( x ', y '; O ') as the length of C ( x ', y '; O '), it has the following property due to Fact 1 and Fact 2: | d ( x, y; O) - d ( x ', y '; O ') |≤ m1 ≤ m br / br / (3)br / br / Figure 6. (a) Decomposition of Γ( x, y; O) (the dashed line) with x = p0 , p3 = y . Note that a segment ~ can go through a junction more than once (e.g. p1 p2 ). (b) Construction of C ( x' , y '; O' ) in o' (the dashed ~ line). Note that C ( x' , y '; O' ) is not the shortest path. Reprinted with permission from "Shape Classification Using the Inner-Distance", H. Ling and D.W. Jacobs, IEEE Trans on Pattern Anal. and Mach. Intell. (PAMI), 29(2):286-299, (2007). © 2007 IEEE.br / br / Figure 7. With the same sample points, the distributions of Euclidean distances between all pair of points are indistinguishable for the four shapes, while the distributions of the inner-distances are quite different. Reprinted with permission from "Shape Classification Using the Inner-Distance", H. Ling and D.W. Jacobs, IEEE Trans on Pattern Anal. and Mach. Intell. (PAMI), 29(2):286-299, (2007). © 2007 IEEE.br / br / 112br / br / Shape Matching for Foliage Database Retrievalbr / br / On the other hand, since O can be articulated from O 'through f-1, we can construct C ( x, y; O) from Γ( x ', y '; O ') in the same way that we construct C ( x ', y '; O ') from Γ( x, y; O). Then, similar to (3), we have | d ( x ', y '; O ') - d ( x, y; O) |≤ m ' br / br / (4)br / br / Combining (3) and (4), d ( x, y; O) - m ' ≤ d ( x, y; O) - m ' ≤ d ( x ', y '; O ') ≤ d ( x ', y '; O ') ≤ d ( x, y; O ) + mbr / br / 1: • • br / br / This implies Theorem 1. # Regarding changes of inner-distances under articulation, two remarks can be made from Theorembr / br / The inner-distance is strictly invariant for ideal articulated objects. This is obvious since ε=0 for ideal articulations. Since ε is very small by definition, for most pairs of x, y, the relative change of inner-distance is very small. This means the inner-distance is insensitive to articulations.br / br / We further clarify several issues. First, the proof depends on the size limitation of junctions. The intuition is that a junction should have a relatively smaller size compared to parts, otherwise it is more like a part itself. A more precise part-junction definition may provide a tighter upper bound but sacrifice some generality. The definition also captures our intuition about what distinguishes articulation from deformation. Second, the part-junction model is not actually used at all when applying the innerdistance. In fact, one advantage of using the inner-distance is that it implicitly captures part structure, whose definition is still not clear in general.br / br / Inner-Distances and Part Structures In addition to articulation insensitivity, we believe that the inner-distance captures part structures better than the Euclidean distance. This is hard to prove because the definition of part structure remains unclear. For example, Basri et al. (1998) gave a shape of shoe that has no clear part decomposition, although it feels like it has more than one part. Instead of giving a rigorous proof, we show how the inner-distance captures part structure with examples and experiments. Figures 2, 7 and 12 show examples where the inner-distance distinguishes shapes with parts while the Euclidean distance runs into trouble because the sample points on the shape have the same spatial distributions. For example, the original shape context [6] may fail on these shapes. One may argue that the Euclidean distance will also work on these examples with an increased number of landmark points. This argument has several practical problems. First, the computational cost will be increased, usually in a quadratic order or higher. Second, no matter how many points are used, there can always be finer structures. Third, as shown in Fig. 8, for some shapes this strategy will not work. During retrieval experiments using several shape databases, the inner-distance based descriptors all achieve excellent performance. Through observation we have found that some databases (e.g., MPEG7) are difficult for retrieval mainly due to the complex part structures in their shapes, though they have little articulation. These experiments show that the inner-distance is effective at capturing part structures (see Fig. 12 and (Ling & Jacobs 2007) for details). 113br / br / Shape Matching for Foliage Database Retrievalbr / br / Aside from part structures, examples in Fig. 8 show cases where the inner-distance can better capture some shapes without parts. We expect further studies on the relationship between inner-distances and shape in the future.br / br / Articulation Invariant Signatures To build shape descriptors with the inner-distance is straightforward. Theoretically, it can be used to replace other distance measures (e.g. the Euclidean distance) in any existing shape descriptors. In this section, the inner-distance is used to build articulation invariant signatures for 2D shapes using multidimensional scaling (MDS) similar to (Elad & Kimmel 2003). In the next section, we will show how to use the inner-distance to extend the shape context for shape matching. n n Given sample points P ≡ { pi }i =1 on a shape O and the inner-distances {dij }i , j =1 between them, MDS n n finds the transformed points Q ≡ {qi }i =1 such that the Euclidean distances{eij (Q) =|| qi - q j ||}i , j =1 minimize the stress S(Q)defined as:br / br / ∑ w (d - e ) S (Q) = ∑d i< jbr / br / ijbr / br / i< jbr / br / ijbr / br / ijbr / br / 2 ijbr / br / 2br / br / br / br / (5)br / br / where wij are weights. In our experiment, we use the least squares MDS with wij =1. The stress can be minimized using the SAMCOF (Scaling by Maximizing a Convex Function) algorithm (Borg & Groenen 1997). SAMCOF is an iterative algorithm that keeps decreasing the objective function, i.e., the stress (5). The details can be found in (Elad & Kimmel 2003). Fig. 9 shows two examples of the articulation invariant signatures computed by the above approach. It can be seen that although the global shape of the two original objects are quite different due to articulation, their signatures are very similar to each other. More examples of articulation invariant signatures can be seen in Fig. 16. It is attractive to use the articulation invariant signature for classifying articulated shapes. In our experiments we combine it with the shape context. The method contains three steps: (1) use the innerdistance and MDS to get the articulation invariant signatures; (2) build the shape context on the signa-br / br / Figure 8. With about the same number of sample points, the four shapes are virtually indistinguishable using distribution of Euclidean distances, as in Fig. 7. However, their distributions of the inner-distances are quite different except for the first two shapes. Note: (1) None of the shapes has (explicit) parts. (2) More sample points will not affect the above statement. Reprinted with permission from "Shape Classification Using the Inner-Distance", H. Ling and D.W. Jacobs, IEEE Trans on Pattern Anal. and Mach. Intell. (PAMI), 29(2):286-299, (2007). © 2007 IEEE.br / br / 114br / br / Shape Matching for Foliage Database Retrievalbr / br / Figure 9. (b) shows the articulation invariant signature of (a). (d) shows the articulation invariant signature of (c). Reprinted with permission from "Shape Classification Using the Inner-Distance", H. Ling and D.W. Jacobs, IEEE Trans on Pattern Anal. and Mach. Intell. (PAMI), 29(2):286-299, (2007). © 2007 IEEE.br / br / tures; (3) use dynamic programming for shape context matching. The third step is described in detail in the next section. We call this approach MDS+SC+DP. The experimental results show significant improvement compared to the shape context on the original shapes.br / br / Inner-Distance Sape Context: Matching and Retrieval Inner-Distance Shape Context (IDSC) To extend the shape context defined in (1), we redefine the bins with the inner-distance. The Euclidean distance is directly replaced by the inner-distance. The relative orientation between two points can be defined as the tangential direction at the starting point of the shortest path connecting them. However, this tangential direction is sensitive to articulation. Fortunately, for a boundary point p and its shortest path Γ( p, q; O) to another point q, the angle between the contour tangent at p and the direction of Γ( p, q; O) at p is insensitive to articulation (invariant to ideal articulation). We call this angle the inner-angle (e.g., see Fig. 11) and denote it as ( p, q; O). The inner-angle is used for the orientation bins. A similar idea is used in [6], which uses the local coordinate system to achieve rotation invariance. In practice, the shape boundary may be distorted by noise that reduces the stability of the inner-angle. To deal with this problem, we smooth the contour using a small neighborhood before computing the inner-angle. Note that the inner-angle may not be unique. However, the ambiguity only happens in very few situations so it does not significantly affect the shape descriptor we are going to build. The inner-angle is just a byproduct of the shortest path algorithms and does not affect the complexity. Once the inner-distances and orientations between all pair of points are ready, it takes O(n2) time to compute the histogram (1). This extension can also be viewed as of first relocating the sample points according to the inner-distance, then counting to get the histogram. This is illustrated in Fig. 10 (d-f). Compared to (b), six points in (d) are moved to different bins, causing a change in of the numbers of points in some bins. Fig. 12 shows examples of the shape context computed by the two different methods. It is clear that SC is similar for all three shapes, while IDSC is only similar for the beetles. From this figure we can see that the inner-distance is better at capturing parts than SC.br / br / Shape Matching Through Dynamic Programming The contour matching problem is formulated as follows: Given two shapes A and B, describe them by point sequences on their contour, say, p1,p2...pnfor A with n points, and q1q2...qmfor B with m points. 115br / br / Shape Matching for Foliage Database Retrievalbr / br / Without loss of generality, assume n ≥ m. The matching ρ from A to B is a mapping from 1, 2, ... , n to 0, 1, 2, ... , m, where pi is matched to q Ii ) if (i ) ≠ 0 and otherwise left unmatched. The matching ρ should minimize the match cost H(ρ) defined as nbr / br / H ( ) = ∑ c(i, (i )) br / br / (6)br / br / i =1br / br / where c(i,0) = is the penalty for leaving pi unmatched, and for 1 ≤ j ≤ m, c(i, j ) is the cost of matching Pi to q j. This is measured using the χ2 statistic as in [6]br / br / c(i, j ) =br / br / 2 1 K (hA,i (k ) - hB , j (k )) ∑ 2 k =1 hA,i (k ) + hB , j (k )br / br / (7)br / br / where jA,iand hB,jare the shape context histograms of pi and q jrespectively, and K is the number of histogram bins. Figure 10. Construction of the shape context (SC) and the inner-distance shape context (IDSC). (a) Sampled points along a shape. The SC and IDSC at point p will be built. Four log-distance bins and four orientation bins are used. (b) The bins and the number of points inside each bins. (c) SC at p. (d) When building IDSC, the inner-distance is used. It works as if moving some points to different bins according to the inner-distance and the inner-angle (q in the graph). q is moved to q'. (e) The number of points inside each bin according to the inner-distance. (f) IDSC at p.br / br / 116br / br / (a)br / br / (b)br / br / (c)br / br / (d)br / br / (e)br / br / (f)br / br / Shape Matching for Foliage Database Retrievalbr / br / Since the contours provide orderings for the point sequences p1p2...pn and q1q2...qm, it is natural to restrict the matching ρ with this order. To this end, we use dynamic programming (DP) to solve the matching problem. DP is widely used for contour matching. Detailed examples can be found in (Thayananthan et al. 2003, Petrakis et al. 2002, Basri et al. 1998). We use the standard DP method (Cormen et al. 2001) with the cost functions defined by (6) and (7). In other words, we want to minimize (6) subject to the sequence ordering and starting points p1, q1. The key formula for the matching is  H (i - 1, j - 1) + c(i, j )  H (i, j ) = min  c(i - 1, j ) +  c(i, j - 1) + br / br / where H(i, j) is the minimum matching cost of matching subsequence p1p2...pi and q1q2...q j. And the cost of the whole matching is H(ρ) = H(n, m).br / br / Figure 11. The inner-angle q(p, q; O) between two boundary points. Reprinted with permission from "Shape Classification Using the Inner-Distance", H. Ling and D.W. Jacobs, IEEE Trans on Pattern Anal. and Mach. Intell. (PAMI), 29(2):286-299, (2007). © 2007 IEEE.br / br / Figure 12. Shape context (SC) and inner-distance shape context (IDSC). The top row shows three objects from the MPEG7 shape database, with two marked points p, q on each shape. The next rows show (from top to bottom), the SC at p, the IDSC at p, the SC at q, the IDSC at q. Both the SC and the IDSC use local relative frames (i.e. aligned to the tangent). In the histograms, the x axis denotes the orientation bins and the y axis denotes log distance bins. Reprinted with permission from "Shape Classification Using the Inner-Distance", H. Ling and D.W. Jacobs, IEEE Trans on Pattern Anal. and Mach. Intell. (PAMI), 29(2):286-299, (2007). © 2007 IEEE.br / br / 117br / br / Shape Matching for Foliage Database Retrievalbr / br / By default, the above method assumes the two contours are already aligned at their start and end points. Without this assumption, one simple solution is to try different alignments at all points on the first contour and choose the best one. The problem with this solution is that it raises the matching complexity from O(n2) to O(n3). Fortunately, for the comparison problem, it is often sufficient to try aligning a fixed number of points, say, k points. Usually k is much smaller than m and n, this is because shapes can be first rotated according to their moments. According to our experience, for n, m=100, k=4 or 8 is good enough and larger k does not demonstrate significant improvement. Therefore, the complexity remains O(kn2)= O(n2). Bipartite graph matching is used in (Belongie et al. 2002) to find the point correspondence ρ. Bipartite matching is more general since it minimizes the matching cost (6) without additional constraints. For example, it works when there is no ordering constraint on the sample points (while DP is not applicable). For sequenced points along silhouettes, however, DP is more efficient and accurate since it uses the ordering information provided by shape contours.br / br / Shape Distances Once the matching is found, we use the matching cost H(ρ) as in (6) to measure the similarity between shapes. One thing to mention is that dynamic programming is also suitable for shape context. In the following, we use IDSC+DP to denote the method of using dynamic programming matching with the IDSC, and use SC+DP for the similar method with the SC. In addition to the excellent performance demonstrated in the experiments, the IDSC+DP framework is simpler than the SC+TPS framework (2) (Belongie et al. 2002). First, besides the size of shape context bins, IDSC+DP has only two parameters to tune: (1) The penalty t for a point with no matching, usually set to 0.3, and (2) The number of start points k for different alignments during the DP matching, usually set to 4 or 8. Second, IDSC+DP is easy to implement, since it does not require the appearance and transformation model as well as the iteration and outlier control. Furthermore, the DP matching is faster than bipartite matching, which is important for retrieval in large shape databases. Figure 13. Shapes of three leaves ((a), (b) and (c)) are not enough to distinguish them. Their texture ((d), (e) and (f) respectively) apparently helps. Reprinted with permission from "Shape Classification Using the Inner-Distance", H. Ling and D.W. Jacobs , IEEE Trans on Pattern Anal. and Mach. Intell. (PAMI), 29(2):286-299, (2007). © 2007 IEEE.br / br / 118br / br / Shape Matching for Foliage Database Retrievalbr / br / The time complexity of the IDSC+DP consists of three parts. First, the computation of inner-distances can be achieved in O(n3) with Johnson or Floyd-Warshall's shortest path algorithms, where n is the number of sample points. Second, the construction of the IDSC histogram takes O(n2). Third, the DP matching costs O(n2), and only this part is required for all pairs of shapes, which is very important for retrieval tasks with large image databases. In our experiment using partly optimized Matlab code on a regular Pentium IV 2.8G PC, a single comparison of two shapes with n=100 takes about 0.31 second.br / br / Hand Shortest Path Texture Context In real applications, shape information is often not enough for object recognition tasks. On the one hand, shapes from different classes sometimes are more similar than those from the same class (e.g., Fig. 13). On the other hand, shapes are often damaged due to occlusion and self-overlapping (some examples can be found in Fig. 1). A combination of texture and shape information is desirable for this problem. In (Belongie et al. 2002) appearance information is included in the SC+TPS framework by considering appearance around landmark points. In this section, we will introduce a new descriptor that considers the texture information inside the whole shape. In previous sections, the inner-distance is shown to be articulation insensitive due to the fact that the shortest paths within shape boundaries are robust to articulation. Therefore, the texture information along these paths provides a natural articulation insensitive texture description. Note that this is true only when the paths are robust. In this section, we use local intensity gradient orientations to capture texture information because of their robustness and efficiency. To gain articulation invariance, the angles between intensity gradient directions and shortest path directions are used. In the following we call these angles relative orientations. Given shape O and two points p, v on it, we use ( p, v; O) to denote the relative orientation with respect to the shortest path Γ( p, v; O). An example is shown in Fig. 14.br / br / Figure 14. (a) Relative orientation α(p, v; O) at point v. The arrow points to local intensity gradient direction. (b) The SPTC at a landmark point is a three-dimensional histogram. (a) is reprinted with permission from "Shape Classification Using the Inner-Distance", H. Ling and D.W. Jacobs, IEEE Trans on Pattern Anal. and Mach. Intell. (PAMI), 29(2):286-299, (2007). © 2007 IEEE.br / br / br / br / (a) (b)br / br / 119br / br / Shape Matching for Foliage Database Retrievalbr / br / Based on the above idea, we propose the shortest path texture context (SPTC) as a combined shape and texture descriptor. SPTC is an extension of the IDSC in that it measures the distributions of (weighted) relative orientations along shortest paths instead of the joint distributions of inner-distance and innerangle distributions of landmark points. In our application, the relative orientations are weighted by gradient magnitudes when building into SPTC. For texture undergoing large non-uniform illumination change, it might be better to use non-weighted relative orientations. Given n landmark points x1, x2, …, xn sampled from the boundary of shape O, the SPTC for each xi is a three-dimensional histogram hi (we abuse notation to use hi again for the histograms). Similar to IDSC, SPTC uses the inner-distance and the inner-angle as the first two dimensions. The third dimension of SPTC is the (weighted) relative orientation that takes into account the texture information along shortest paths. To build hi, for each xj, j ≠ i, a normalized histogram of relative orientation along the shortest path Γ( xi , x j ; O) is added into the relative orientation bin located at the inner-distance and inner-angle bin determined by xj. The algorithm is described in Algorithm 2. Note that when the number of relative orientation bins nr =1, SPTC reduces to IDSC. A similar idea of using ``relative orientation'' is used by Lazebnik et al. (2005) for rotation invariant texture description. Shape context had also been extended for texture description by including intensity gradient orientation (Mikolajczyk & Schmid 2005). SPTC is different from these methods in three ways. First, SPTC combines texture information and global shape information while the above methods work for local image patches. Second, the above methods sample the orientations at a large number of pixels inside a patch, which is too expensive for our task without utilizing shortest paths. Third, none of the previous methods is articulation invariant. Another related work by Zhao and Davis (2005) used the color information along the shortest path for background subtraction. Instead of color information, we use gradient orientation, which is more robust to lighting change (Chen et al. 2000), which is very important for classification tasks. In the next section, SPTC is tested with two leaf image databases and excellent performance is observed.br / br / Fo In this section we briefly describe a prototype electronic field guide system. The prototype is the product of a collaboration between researchers from Columbia University, the Smithsonian Institution, and the University of Maryland [3]. The image retrieval system allows for online visual searching. In the field, a botanist can photograph an unknown leaf to the system and get the most visually similar leaves in the database. Another important target of an EFG system is to provide an easy-to-use browsing interface that helps not only botanists, but all users. For example, it would be very helpful to embed an EFG system in a web server, which further provides online access and browsing services. Techniques from several research areas are necessary to fulfill this task: 1. br / br / 120br / br / An image retrieval algorithm enables image-based browsing as well as query. This is the main topic of this chapter. The inner-distance based approaches give a reliable solution to the problem. The shape matching algorithms not only play a key role in the visual search task, they also form the basis of visual clustering that has been shown to be very helpful in image browsing tasks (Agarwal 2005).br / br / Shape Matching for Foliage Database Retrievalbr / br / Figure 15. Modules of the electronic field guide prototypebr / br / 2. br / br / 3. 4. br / br / Computing hardware. A portable computing device is a must. This is much less of a problem nowadays than before, thanks to numerous popular mobile devices such as laptops, but more usable portable devices are still needed. A flexible interface. The interface should provide an easy browsing environment as well as an easy interface to database systems that contain textual and image information. Interface hardware. A traditional retrieval system is usually based on textual input. While this is easy and requires only a keyboard (or similar input device), it sacrifices convenience. In addition, it is often not easy to summarize an unknown species in a way that is appropriate for a text-based retrieval system. A natural extension is use image based searching. For this reason, an input device (usually a camera) is needed to provide a query image to the image-based retrieval system. Communication between a camera and a computing device can be easily and automatically done through wireless communication systems, such as Bluetooth, while systems with built in cameras will be even more convenient.br / br / Figure 15 is a summary of the software part of our prototype system. On the top there are the two interfaces to users, image browsing and retrieval. In the middle are four modules that support the two interfaces, which are built on top of three modules, including Photomesa (Bederson 2001), which is a zoomable and reusable user interface toolkit, the inner-distance based shape matching (IDSC) techniques and an image processing module. Details of the system can be found in (Agarwal et al. 2006). The system is tested in several real field test trips on the Plummers Island on the Potomac River near Washington DC. During these tests, the botanists picked leaves and took pictures of them. The pictures were then be automatically uploaded to our system and the searching results were provided by our system in real time. Both the reliability and usability of our system are confirmed in these real tests.br / br / EXPERIMENTS Aand Articulated Database To show the articulation insensitivity of the inner-distance, we apply the proposed articulation invariant signature and the IDSC+DP approach to an articulated shape data set we collected. The dataset containsbr / br / 121br / br / Shape Matching for Foliage Database Retrievalbr / br / Algorithm 2: Shortest path texture contextbr / br / hi at landmark point xibr / br / hi  3-D matrix with zero entries everywhere for j=1 to n, j ≠ ibr / br / Γ( xi , x j ; O)  shortest path from xi to x jbr / br / hˆ  1-D weighted histogram of the relative orientations along Γ( xi , x j ; O) / * Normalize hˆ */ hˆ  hˆ / | hˆ | d id  the inner-distance bin index computed from d ( xi , x j ; O) idbr / br / forbr / br / ( xi , x j ; O )br / br /  the inner-angle bin index computed from idbr / br / nrbr / br / =1 tobr / br / hi (d id , endfor endfor hi  hi / | hibr / br / idbr / br / |br / br / ,br / br / idbr / br / /br / br / )  hi (d id , /br / br / * idbr / br / ,br / br / idbr / br / nr is the number of relative orientation bins */ ) + hˆ( ) idbr / br / * Normalize hi */br / br / 40 images from 8 different objects. Each object has 5 images articulated to different degrees (see Fig. 16). The dataset is very challenging because of the similarity between different objects (especially the scissors). The holes of the scissors make the problem even more difficult. The parameters in the experiment are: n=200, nd =5, nq =12. Since all the objects are at the same orientation, we align the contours by forcing them to start from the bottom-left points and then set k=1 for DP matching. The articulation invariant signatures of the shapes are computed and shown in Fig. 16 (b). To evaluate the recognition result, for each image, the four most similar matches are chosen from other images in the dataset. The retrieval result is summarized as the number of 1st, 2nd, 3rd and 4th most similar matches that come from the correct object. Table I shows the retrieval results. It demonstrates that both the articulation invariant signature and the IDSC help to improve recognition a lot. This verifies our claim that the inner-distance is very effective for objects with articulated parts. The experiment also shows that IDSC works better than MDS for the articulated shapes. One reason is that the MDS may cause loss of information since it uses the Euclidean distance to approximate the inner-distance. Aside from the articulated dataset, the inner-distance based approaches have been applied to several other public datasets, including the MPEG7 shape dataset, Kimia datasets, ETH-80 datasets. Excellent performance is observed in all the experiments. Details of these experiments can be found in (Ling & Jacobs 2007).br / br / Swedish Leaf Database The Swedish leaf dataset comes from a leaf classification project at Linkoping University and the Swedish Museum of Natural History (Soderkvist 2001). The dataset contains isolated leaves from 15 different Swedish tree species, with 75 leaves per species. Fig. 17 shows some representative silhouette examples. Some preliminary classification work has been done in (Soderkvist 2001) by combining simple features like moments, area and curvature etc. We tested with Fourier descriptors, SC+DP, MDS+SC+DP,br / br / 122br / br / Shape Matching for Foliage Database Retrievalbr / br / Figure 16. (a) Articulated shape database. This dataset contains 40 images from 8 objects with articulation. Each column contains five images from the same object. (b) MDS of the articulated shape database using the inner-distances. Reprinted with permission from "Shape Classification Using the Inner-Distance", H. Ling and D.W. Jacobs, IEEE Trans on Pattern Anal. and Mach. Intell. (PAMI), 29(2):286-299, (2007). © 2007 IEEE.br / br / br / br / (a) (b)br / br / Table 1. Retrieval result on the articulate dataset Distance Typebr / br / Top 1br / br / Top 2br / br / Top 3br / br / Top 4br / br / L2br / br / 25/40br / br / 15/40br / br / 12/40br / br / 10/40br / br / SC+DPbr / br / 20/40br / br / 10/40br / br / 11/40br / br / 5/40br / br / Art. Inv. Signaturebr / br / 36/40br / br / 26/40br / br / 17/40br / br / 15/40br / br / IDSC+DPbr / br / 40/40br / br / 34/40br / br / 35/40br / br / 27/40br / br / Figure 17. Typical images from Swedish leaf data base, one image per species. Note that some species are quite similar, e.g. the 1st, 3rd and 9th species. Reprinted with permission from "Shape Classification Using the Inner-Distance", H. Ling and D.W. Jacobs, IEEE Trans on Pattern Anal. and Mach. Intell. (PAMI), 29(2):286-299, (2007). © 2007 IEEE.br / br / 123br / br / Shape Matching for Foliage Database Retrievalbr / br / IDSC+DP and SPTC+DP with parameters n=128, nd =8, nq =12, nr =8 and k=1. Each species contains 25 training samples and 50 testing samples per species. The recognition results with 1-nearest-neighbor are summarized in Tab. II. Notice that unlike other experiments, the articulation invariant signature works a little better than IDSC on the leaf images. One possible explanation is that, as a real image dataset, the inner-angle for leaves are less robust due to boundary noise. Also notice that SPTC improves IDSC as we had expected.br / br / Smithsonian Isolated Leaf Database This data set comes from the Smithsonian project (Electronic Field Guide, 2008). We designed an Electronic Field Guide image retrieval system that allows online visual searching. The task is very challenging because it requires querying a database containing more than one hundred species and real time performance requires an efficient algorithm. In addition, the pictures taken in the field are vulnerable to lighting changes and the leaves may not be flattened well. We evaluated the proposed approaches on a representative subset of the leaf image database in the system (http://www.ist.temple.edu/~hbling/data/SI-93.zip). The subset contains 343 leaves from 93 species (the number of leaves from different species varies). In the experiment, 187 of them are used as the training set and 156 as the testing set. Note that there are only two instances per class in the train-br / br / Table 2. Recognition rates on the Swedish leaf dataset. Note that MDS+SC+DP and SPTC achieved the same rates Methodbr / br / [Soderkvist 2001]br / br / Fourierbr / br / SC+DPbr / br / MDS+SC+DPbr / br / IDSC+DPbr / br / SPTC+DPbr / br / Recognition Ratebr / br / 82%br / br / 89.6%br / br / 8812.%br / br / 95.33%br / br / 94.13%br / br / 95.33%br / br / Note that MDS+SC+DP and SPTC got same rates.br / br / Figure 18. The Smithsonian dataset. This dataset contains 343 leaf images from 93 species. Typical images from each species are shown. Reprinted with permission from "Shape Classification Using the Inner-Distance", H. Ling and D.W. Jacobs, IEEE Trans on Pattern Anal. and Mach. Intell. (PAMI), 29(2):286-299, (2007). © 2007 IEEE.br / br / 124br / br / Shape Matching for Foliage Database Retrievalbr / br / ing set on average. The retrieval performance is evaluated using performance curves which show the recognition rate among the top N leaves, where N varies from 1 to 16. For the efficiency reasons mentioned above, only 64 contour points are used (i.e. n=64). The similarity between leaves is measured by the shape context distance Dsc (see Sec. II or (Belongie et al. 2002). This distance is based on a greedy matching and should not be confused with the bipartite matching Figure 19. Recognition result on the Smithsonian leaf dataset. The ROC curves shows the recognition rate among the top N matched leaves. Reprinted with permission from "Shape Classification Using the Inner-Distance", H. Ling and D.W. Jacobs, IEEE Trans on Pattern Anal. and Mach. Intell. (PAMI), 29(2):286-299, (2007). © 2007 IEEE.br / br / Figure 20. Three retrieval examples for IDSC and SPTC. The left column shows the query images. For each query image, the top four retrieving results are shown to its right, using IDSC and SPTC respectively. The circled images come from the same species as the query image. Reprinted with permission from "Shape Classification Using the Inner-Distance", H. Ling and D.W. Jacobs, IEEE Trans on Pattern Anal. and Mach. Intell. (PAMI), 29(2):286-299, (2007). © 2007 IEEE.br / br / 125br / br / Shape Matching for Foliage Database Retrievalbr / br / based approach. Therefore it is faster than DP-based matching. Other parameters used in the experiment are nd =5, nq =12, and nr =8. Note that k is not needed because DP is not used here. The performance is plotted in Fig. 19. It shows that SPTC works significantly better than other methods. Fig. 20 gives some detailed query results of SPTC and IDSC, from which we can see how SPTC improves retrieval result by also considering texture information.br / br / Con In this chapter, we present an inner-distance based shape matching algorithm for the application of foliage image retrieval. With experiments on two leaf databases involving thousands of images, our method demonstrates excellent performance in comparison to several state-of-the-art approaches. The approach is adopted in an electronic field guide system that has been tested in several real field test trips. There are two main issues that are worth future study. First, the algorithm requires segmentation for preprocessing. In our current system images are taken by putting leaves against a white paper background. This is apparently not an ideal solution especially for use in the field. Second, more efficient schemes are needed for fast retrieval and browsing, especially for a large database or an online system. One possibility is through smart indexing techniques instead of the currently used nearest neighbor algorithm. We look forward to future development in these directions.br / br / ACKNOWLEDGMENT We would like to thank J. W. Kress, R. Russell, N. Bourg, G. Agarwal, P. Belhumeur and N. Dixit for help with the Smithsonian leaf database, and O. Soderkvist for the Swedish leaf data. This work is supported in part by NSF (ITR-03258670325867) and by the US-Israel Binational Science Foundation grant number 2002/254.br / br / REFERNECES Agarwal, S., Awan, A., & Roth, D. (2004). Learning to Detect Objects in Images via a Sparse, Part-Based Representation. IEEE Trans. on Pattern Analysis and Machine Intelligence, 26(11), 1475-1490. Agarwal, G. (2005). Presenting Visual Information to the User: Combining Computer Vision and Interface Design. Master Thesis, Univseristy of Maryland, 2005. Agarwal, G., Belhumeur, P., Feiner, S., Jacobs, D., Kress, J.W., Ramamoorthi, R., Bourg, N., Dixit, N., Ling, H., Mahajan, D., Russell, R., Shirdhonkar, S., Sunkavalli, K., & White, S. (2006). First Steps Toward an Electronic Field Guide for Plants. Taxon, 55(3), 597-610. Basri, R., Costa, L., Geiger, D., & Jacobs, D. (1998). Determining the Similarity of Deformable Shapes. Vision Research, 38, 2365-2385. Bederson, B.B. (2001). PhotoMesa: A Zoomable Image Browser Using Quantum Treemaps and Bubblemaps. ACM Symposium on User Interface Software and Technology, CHI Letters, 3(2), 71-80.br / br / 126br / br / Shape Matching for Foliage Database Retrievalbr / br / Belongie, S., Malik, J., & Puzicha, J. (2002). Shape Matching and Object Recognition Using Shape Context. IEEE Trans. on Pattern Analysis and Machine Intelligence, 24(24), 509-522. Biederman, I. (1987). Recognition--by--components: A theory of human image understanding. Psychological Review, 94(2), 115-147. Blum, H. (1973). Biological Shape and Visual Science. J. Theor. Biol., 38, 205-287. Bookstein, F. (1989). Principal Warps: Thin-Plate-Splines and Decomposition of Deformations. IEEE Trans. on Pattern Analysis and Machine Intelligence, 11(6), 567-585. Borg, I., & Groenen, P. (1997). Modern Multidimensional Scaling: Theory and Applications. Springer. Chen, H., Belhumeur, P., & Jacobs, D. (2000). In search of Illumination Invariants. IEEE Conf. on Computer Vision and Pattern Recognition, I, 254-261. Cormen, T., Leiserson, T, Rivest, R., & Stein, C. (2001). Introduction to Algorithms, 2nd edition. The MIT Press. Elad, A., & Kimmel, R. (2003). On Bending Invariant Signatures for Surfaces. IEEE Trans. on Pattern Analysis and Machine Intelligence, 25(10), 1285-1295. An Electronic Field Guide: Plant Exploration and Discovery in the 21st Century. (2008). Retrieved December 12, 2008, fromhttp://herbarium.cs.columbia.edu/. Feldman, J., & Singh, M. (2005). Information along contours and object boundaries. Psychological Review, 112(1), 243-252. Felzenszwalb, P. (2005). Representation and Detection of Deformable Shapes. IEEE Trans. on Pattern Analysis and Machine Intelligence, 27(2), 208-220. Felzenszwalb, P., & Huttenlocher, D. (2005). Pictorial Structures for Object Recognition. International Journal of Computer Vision, 61(1), 55-79. Felzenszwalb, P., & Schwartz, J. (2007). Hierarchical Matching of Deformable Shapes. IEEE Conference on Computer Vision and Pattern Recognition. Fergus, R., Perona, P., & Zisserman, A. (2003). Object Class Recognition by Unsupervised Scale-Invariant Learning, IEEE Conference on Computer Vision and Pattern Recognition, II, 264-271. Gandhi, A. (2002). Content-based image retrieval: Plant species identification. MS thesis. Oregon State University. Gorelick, L., Galun, M., Sharon, A., Basri, R., & Brandt, A. (2004). Shape Representation and Classification Using the Poisson Equation. IEEE Conference on Computer Vision and Pattern Recognition, (pp. 61-67). Grenander, U., Srivastava, A., & Saini, S. (2007). A Pattern-Theoretic Characterization of Biological Growth. IEEE Transactions on Medical Imaging, 26(5), 648-659.br / br / 127br / br / Shape Matching for Foliage Database Retrievalbr / br / Grimson, W.E.L. (1990). Object Recognition by Computer: The Role of Geometric Constraints. Cambridge, MA: MIT Press. Hamza, A. B., & Krim, H. (2003). Geodesic Object Representation and Recognition. In I. Nystrom et al. (Eds.), Discrete Geometry for Computer Imagery, LNCS, 2886, 378-387. Hoffman, D. D., & Richards, W. A. (1985). Parts of recognition. Cognition, 18, 65-96. Im, C., Nishida, H., & Kunii, T. L. (1998). Recognizing plant species by leaf shapes-a case study of the Acer family. International Conference on Pattern Recognition, 2, 1171-1173. Keogh, E., Wei, L., Xi, X., Lee, S-H., & Vlachos, M. (2006). LB_Keogh Supports Exact Indexing of Shapes under Rotation Invariance with Arbitrary Representations and Distance Measures. VLDB. Kimia, B. B., Tannenbaum, A. R., & Zucker, S. W. (1995). Shapes, shocks, and deformations, I: The components of shape and the reaction-diffusion space. International Journal of Computer Vision, 15(3), 189-224. Lazebnik, L., Schmid, C., & Ponce, J. (2005). A sparse texture representation using affine-invariant regions. IEEE Trans. Pattern Anal. Mach. Intell., 27(8), 1265-1278. Leibe, B., & Schiele, B. (2003). Analyzing Appearance and Contour Based Methods for Object Categorization. IEEE Conference on Computer Vision and Pattern Recognition, II, 409-415. Ling, H., & Jacobs, D. W. (2005). Using the Inner-Distance for Classification of Articulated Shapes. IEEE Conference on Computer Vision and Pattern Recognition, II, 719-726. Ling, H., & Jacobs, D. W. (2005). Deformation Invariant Image Matching. IEEE International Conference on Computer Vision, II, 1466-1473. Ling, H., & Jacobs, D. W. (2007). Shape Classification Using the Inner-Distance. IEEE Trans on Pattern Anal. and Mach. Intell., 29(2), 286-299. Liu, T., & Geiger, D. (1997). Visual Deconstruction: Recognizing Articulated Objects. Energy Minimization Methods in Computer Vision and Pattern Recognition, (pp. 295-309). Mikolajczyk, K., & Schmid, C. (2005). A Performance Evaluation of Local Descriptors. IEEE Trans. Pattern Anal. Mach. Intell., 27(10), 1615-1630. Mokhtarian, F., & Abbasi, S. (2004). Matching shapes with self-intersections: application to leaf classification. IEEE Trans. on Image Processing, 13(5), 653-661. Mori, G., & Malik, J. (2003). Recognizing Objects in Adversarial Clutter: Breaking a Visual CAPTCHA. IEEE Conference on Computer Vision and Pattern Recognition, I, 1063-6919. Nilsback, M., & Zisserman, A. (2006). A Visual Vocabulary for Flower Classification. IEEE Conf. on Computer Vision and Pattern Recognition, 2, 1447-1454. Osada, R., Funkhouser, T., Chazelle, B., & Dobkin, D. (2002). Shape Distributions. ACM Trans. on Graphics, 21(4), 807-832.br / br / 128br / br / Shape Matching for Foliage Database Retrievalbr / br / Petrakis, E.G.M., Diplaros, A., & Milios, E. (2002). Matching and Retrieval of Distorted and Occluded Shapes Using Dynamic Programming. IEEE Trans. on Pattern Analysis and Machine Intelligence, 24(11), 1501-1516. Saitoh, T., & Kaneko, T. (2000). Automatic Recognition of Wild Flowers. International Conference on Pattern Recognition, 2, 2507-2510. Sclaroff, S., & Liu, L. (2001). Deformable shape detection and description via model-based region grouping, IEEE Trans. on Pattern Analysis and Machine Intelligence, 23(5), 475-489. Schneiderman, H., & Kanade, T. (2004). Object Detection Using the Statistics of Parts. International Journal of Computer Vision, 56(3), 151-177. Sebastian, T.B., Klein, P.N., & Kimia, B.B. (2004). Recognition of Shapes by Editing Their Shock Graphs. IEEE Trans. on Pattern Analysis and Machine Intelligence, 26(5), 550-571. Siddiqi, K, Shokoufandeh, A., Dickinson, S.J., & Zucker, S.W. (1999). Shock Graphs and Shape Matching. International Journal of Computer Vision, 35(1), 13-32. Soderkvist, O. (2001). Computer Vision Classification of Leaves from Swedish Trees. Master Thesis. Linkoping University. Thayananthan, A., Stenger, B., Torr, P.H.S., & Cipolla, R. (2003). Shape Context and Chamfer Matching in Cluttered Scenes. IEEE Conference on Computer Vision and Pattern Recognition, I, 127-133. Thompson, D.W. (1992) On Growth and Form. (republished), Dover Publication. Tu, Z., & Yuille, A.L. (2004). Shape Matching and Recognition-Using Generative Models and Informative Features. European Conference on Computer Vision, 3, 195-209. Veltkamp, R.C., & Hagedoorn, M. (1999). State of the Art in Shape Matching, Technical Report UUCS-1999-27, Utrecht. Wang, Z., Chi, Z., & Feng, D. (2003). Shape based leaf image retrieval. IEEI proc. Vision, Image and Signal Processing, 150(1), 34-43. Weiss, I., & Ray, M. (2005). Recognizing Articulated Objects Using a Region-Based Invariant Transform. IEEE Trans. on Pattern Analysis and Machine Intelligence, 27(10), 1660- 1665. Yahiaoui, I., Herve, N., & Boujemaa, N. (2005). Shape-based image retrieval in botanical collections. Zhao, L., & Davis, L. S. (2005). Segmentation and Appearance Model Building from an Image Sequence. IEEE International Conference on Image Processing, 1, 321-324.br / br / 129br / br / 130br / br / Chapter Vbr / br / Similarity Learning for Motion Estimation Shaohua Kevin Zhou Siemens Corporate Research Inc., USA Jie Shao Google Inc., USA Bogdan Georgescu Siemens Corporate Research Inc., USA Dorin Comaniciu Siemens Corporate Research Inc., USAbr / br / ABSTRACT Motion estimation necessitates an appropriate choice of similarity function. Because generic similarity functions derived from simple assumptions are insufficient to model complex yet structured appearance variations in motion estimation, the authors propose to learn a discriminative similarity function to match images under varying appearances by casting image matching into a binary classification problem. They use the LogitBoost algorithm to learn the classifier based on an annotated database that exemplifies the structured appearance variations: An image pair in correspondence is positive and an image pair out of correspondence is negative. To leverage the additional distance structure of negatives, they present a location-sensitive cascade training procedure that bootstraps negatives for later stages of the cascade from the regions closer to the positives, which enables viewing a large number of negatives and steering the training process to yield lower training and test errors. The authors apply the learned similarity function to estimating the motion for the endocardial wall of left ventricle in echocardiography and to performing visual tracking. They obtain improved performances when comparing the learned similarity function with conventional ones. Copyright © 2009, IGI Global, distributing in print or electronic forms without written permission of IGI Global is prohibited.br / br / Similarity Learning for Motion Estimationbr / br / INTRDUCTION Image Matching and Similarity Function Image matching is fundamental to various computer vision tasks. In motion estimation, image matching happens along the temporal dimension, e.g., comparing consecutive frames to establish correspondences over time or tracking points of interest. In image registration, image matching happens along the spatial dimension, e.g., comparing two heterogeneous images for establishing spatial correspondences. Image matching is also vital to content-based retrieval, face recognition, and application of the same kind, where comparing testing and training image is needed. Underlying an image matching process lays an indispensable component of similarity function. A similarity function is a two-input function s(I, I') that measures how closely the test patch I' is visually similar to the template patch I. A typical use of similarity function in, say, motion estimation and image registration algorithms is as follows: given two images I and I' and a target point (u,v) whose motion vector or spatial correspondence to be estimated, one finds the shift that has the (local) maximum similarity. If the minimum is sought, one can simply negate the similarity function.br / br / ( uˆ , vˆ) = arg max s ( I (u , v), I '(u + u , v + v)), ( u , v )∈Wbr / br / (1)br / br / where I(u,v) is a local patch extracted from the image I, centered at (u,v), and W is the searching window. In motion estimation, the two images I and I' are successive frames, e.g., I=It-1 and I'=It; in image registration, the two images I and I' are the image pair to be registered. In retrieval and recognition applications, the use of similarity function is as follows:br / br / nˆ = arg max s ( I n , I '), n =1,.., Nbr / br / (2)br / br / where {In; n=1,2,…,N} are gallery images stored in the database, and I' is a query image that is used to sort the database. The principal difference between (1) and (2) lies in the search space where the maximum is found: The search space for the first type of application (e.g., motion estimation and image registration) is a spatial window, and that for the second type of applications (e.g., retrieval and recognition) is on the index of the images in the gallery database. In this chapter, we concentrate on the specific application of motion estimation. It is obvious that applications like image registration, retrieval and recognition, etc. can be also tackled with a minor modification.br / br / Smilarity Function Under Complex Appearance Variations Matching two images under complex appearance variations is challenging due to lacking an appropriate similarity function. Similarity functions proposed in the literature are mostly generic and inadequate for handling complex appearance variations. A motivating example is illustrated in Fig. 1. Consider a stress echocardiographic video (stress echo) (Otto, 2004), a series of 2D ultrasound images of the human heart captured after the patient undergoes exercise or takes special medicine. We focus on wall motion analysis to characterize the functionality of the heart. To be specific, we measure the motion of the endocardium (or the inner border) of the left ventricle (LV). As shown in Fig. 1 thebr / br / 131br / br / Similarity Learning for Motion Estimationbr / br / Figure 1. A stress echo sequence with six consecutive frames and according annotations of the LV endocardium. Due to the rapid heart rate (177 beats per second), the LV appearance varies significantly within a short period of six consecutive frames. (From (Zhou et al. 2006) at IEEE 2006)br / br / LV endocardium presents severe appearance changes over a cardiac cycle due to nonrigid deformation, imaging artifacts like speckle noise and signal dropout, movement of papillary muscle (which is attached to the LV endocardium but not a part of the wall), respiratory interferences, unnecessary probe movement, insufficient sampling for stress echo, etc. For the above example of motion estimation, we will illustrate in later sections that the performances achieved by generic similarity functions are not satisfactory. To bridge the performance gap, we propose to learn a discriminative similarity function using the boosting framework (Freund & Schapire, 1997) (Friedman, Hastie, & Tibshirani, 2000). Our proposal is grounded on the following arguments: • br / br / • br / br / br / br / 132br / br / Albeit complex, the appearance variations manifest certain structures in an application-dependent fashion. In the stress echo example, the deformation of the LV endocardium is governed by the mechanical movement of the heart muscle, which, for example, is cyclic in nature. Therefore, it is possible to collect enough data to cover the structured appearance variations, thereby leading us to use machine learning tools to learn a similarity function. A good similarity function for image matching should possess the following two properties: its response map is both smooth and unimodal (ideally peaked at the correct location). In the chapter, we explore the second property. In the extreme, the similarity function operates like a discriminative function: positive if the centers of two paired image patches are in correspondence and negative if out of correspondence.br / br /  1, s ( I , I ') =  0,br / br / if (I,I' ) is a corresponding pair; otherwise.br / br / (3)br / br / Similarity Learning for Motion Estimationbr / br / br / br / • br / br / This fits the concept of discriminative learning. If we treat the image pairs in correspondence as positives and the rest as negatives (as shown in Fig. 2(a,d,e)), the similarity function becomes a discriminative function that separates two classes. Given an annotated video database, we can learn such a discriminative function using examples extracted from the database. In general, generic similarity functions, which will be briefly reviewed in Section II, work for certain imaging scenarios but break down for others, depending on the way in which the appearance changes. For example, the sum of squared distance (SSD) works best for isotropic Gaussian noise, the sum of absolute distance (SAD) for the Laplacian noise, and the CD2 (Cohen & Dinstein, 2002) for fully developed speckle noise. On one hand, there is no such a global and generic similarity function universally good for all scenarios; on the other hand, it is very likely that different local regions of the image are best suited for different similarity functions. This strongly motivates us to take a boosting approach (Freund & Schapire, 1997) (Friedman, Hastie, & Tibshirani, 2000) that combines locally good similarity function (still weak though) into a globally strong similarity function that works best for the scenario exemplified by an annotated database.br / br / Section III presents the issues related to learning such a discriminative similarity function. Among a variety of binary classification methods, we invoke the LogitBoost algorithm (Friedman, Hastie, & Tibshirani, 2000) to selectively combine weak learners into one strong similarity function. We associate a weak learner with a Haar-like local rectangle feature (Papageorgiou, Oren, & Poggio, 1998) (Viola & Jones, 2001) to accommodate fast computation. The weak learner takes an image pair as input and uses the two feature responses collected from both images. We construct the weak learner as a nonparametric 2D piecewise constant function of the two feature responses in order to strengthen its modeling power, thereby bringing savings in both training speed and storage requirement. Selecting negatives is crucial to the training accuracy and consequently influences the final performance. The negatives implicitly possess a location parameter measuring their closeness to the positives. To leverage the additional distance structure of the negatives, we present a location-sensitive cascade training procedure that bootstraps negatives for later stage of the classifier cascade from the regions closer to the positives. This allows not only viewing a large number of negatives as in the regular cascade training (Viola & Jones, 2001) but also steering the training process with respect to the desired final accuracy. Further, we empirically show that the location-sensitive cascade yields lower training and test errors than the regular one. In Section IV, we present the so-called BoostMotion approach that experiments the boosted similarity function for motion estimation. We compare the discriminative similarity function with conventional similarity functions using the stress echo sequences and obtain improved performance when estimating the motion of the LV endocardium. We also contrast the BoostMotion approach, which takes a pair of images as input, with a learning-based detection algorithm, which takes a single image patch as input, and demonstrate the importance of temporal information in motion estimation. Then we insert the BoostMotion module into a naive tracker that estimates the motion vector frame by frame and hence is prone to drift. In the experiment of tracking regular echo sequences, we show that using the discriminative similarity function reduces drifting. We finally test the discriminative similarity function in general visual tracking. Section V summarizes the chapter.br / br / 133br / br / Similarity Learning for Motion Estimationbr / br / GENERIC SIMILARITY FUNCTION In this section, we briefly review similarity functions proposed in the literature. Because we focus on the applications like motion estimation, we refer the readers to (Santini & Jain, 1999) for a review of similarity functions used elsewhere. We first present a rough categorization of generic similarity function and then address the difficulty in applying them into real applications of motion estimation and image registration.br / br / Categorization of Generic Similarity Function The similarity functions for motion estimation proposed in the literature are generic in nature and mostly arise from the following two assumptions: • br / br / The assumption of constant brightness. In a simplistic form, it assumes a probabilistic model as follows.br / br / br / br / I t (u + u , v + v) = I t -1 (u , v) + et (u , v), br / br / • br / br / where et(u,v) is the observation noise. The assumption of parametric noise model, that is, the noise variable et(u,v) in (4) is of certain parametric form. Often, the parametric noise model yields a parametric similarity function.br / br / (4)br / br / Depending on the noise model, similarity function can be roughly categorized as intensity-based, histogram-based, and modality-specific. Other similarity functions for motion estimation as well as image registration are available too.br / br / Intensity-Based Similarity Function Examples of intensity-based similarity function include sum of square distance (SSD) (Brox, Bruhn, Papenberg, & Wiecker, 2004) (Lucas & Kanade, 1981) (Shi & Tomasi, 1994), sum of absolute distance (SAD), etc. The SSD similarity function takes the following form: SSD: s (I , I ') = I - I ' 2, br / br / (5)br / br / where the L2 norm is used. The SSD is also equivalent to assuming an isotropic Gaussian noise model. The minimum value of the SSD is reached only when I=I'. The SSD can be generalized in two ways. First, it can be based on images derived from the original ones. For example, if the gradient image (denoted by Iˆ = ∂I / ∂ , { ∈ x, y}) is used, this corresponds to the "gradient constancy" assumption (Brox, Bruhn, Papenberg, & Wiecker, 2004).br / br / s ( I , I ') = Iˆ - Iˆ ' 2br / br / 134br / br / (6)br / br / Similarity Learning for Motion Estimationbr / br / Learning an affinity function using gradient information is also proposed in (Fowlkes, Martin, & Malik, 2003) (Lunqvist, Bengtsson, & Thurfjell, 2003). Second, the L2 norm can be replaced by an Lp norm. If p=1, the similarity function becomes the SAD function, which is equivalent to assuming a Laplacian noise model. SAD: s (I , I ') = I - I ' 1 br / br / (7)br / br / Histogram-Based Similarity Function Assuming that the histogram of I is given by h(I), the Bhattacharyya distance (Comaniciu, Ramesh, & Meer, 2000) is defined asbr / br / BHA: s (I , I ') = ∫ h( I )h( I ')d . br / br / (8)br / br / The Bhattacharyya distance s(I, I') is always less than one becausebr / br / ∫br / br / h( I )h( I ')d ≤br / br / 1 (h( I ) + h( I '))d = 1, 2∫br / br / (9)br / br / and achieves one only when h(I)=h(I'). The condition I=I' is sufficient for h(I)=h(I'). Other histogram-based similarity functions include the chi-square distance, earth moving distance, KL divergence, Jensen-Shannon distance, etc.br / br / Modality-Speci.c Similarity Function In the chapter, we focus on ultrasound images in the motion estimation experiment. The ultrasoundspecific CD2 similarity function proposed by Cohen and Dinstein (Cohen & Dinstein, 2002) is specially designed for handling a fully-developed speckle noise in an ultrasound image and shown to be effective by Boukerroui, Alison, & Brady (2003) for tracking ultrasound sequences. The CD2 function is defined asbr / br / i i' CD 2 : s (I , I ') = -∑ log( + ), i' ibr / br / (10)br / br / where the pixels i and i' belong to I and I', respectively. The maximum of CD2 is obtained only when i i' i=i' for all i (or equivalently, I=I') because + ≥ 2. i' ibr / br / Other Similarity Function We here also mention two similarity functions widely used in the literature: normalized cross correlation (NCC) and mutual information (MI). The NCC is used to compensate an affine photometric variation and is defined as NCC:br / br / s( I, I ') = -br / br / ( I - ( I )) • ( I '- ( I ')) / N , ( I ) ( I ')br / br / (11)br / br / where • denotes the dot product, N is the number of pixel in the image I, and µ(.) and σ(.) take the sample mean and standard deviation, respectively. The maximum is achieved only when I=aI'+b. Further to 135br / br / Similarity Learning for Motion Estimationbr / br / compensate the non-uniformity in the above linear model, the NCC is generalized to local normalized correlation coefficient (LNC). To calculate LNC, the image patch is first divided into several sub-regions whose NCC is computed. The mean of all NCC's is defined as LNC. One popular similarity function used for medical image registration is the so-called mutual information (MI) (Viola & Welles, 1997) defined as: MI: s (I , I ') = ∫ h( I , I ')br / br / h( I , I ') d , h( I )h( I ')br / br / (12)br / br / where h(I,I') is the joint histogram. It is easy to prove that MI≥0; only when I and I' are independent, i.e., h(I,I')=h(I)h(I'), the MI reaches the minimum 0. Image registration is to find the maximum value of MI by deforming I'. Note that mutual information is not modality-specific, which makes it common for cross-modality registration.br / br / Difficulty from Generic Smilarity Function One reason why the generic similarity functions are not working for real applications because the two assumptions are largely violated, that is, the brightness is not constant and/or the noise model is not parametric. For example, Fig. 2(a) shows a pair of consecutive frames in a stress echo sequence and Fig. 2(b) the cropped LV images from the two frames. When we take the difference of the LV images as in Fig. 2(b), we observe a lot of structures (mainly due to nonrigid deformation, disappearing papillary muscle) and speckles in Fig. 2(c). By learning the similarity function, we avoid any possibly unrealistic assumption.br / br / DISCRIMIina In this section, we address how to learn the discriminative similarity function (Zhou, Shao, Georgescu, & Comaniciu, 2006). We first review the LogitBoost algorithm and then proceed to the practical choice of weak functions used in boosting. Finally, we discuss how to deal with the numerous negative examples by leveraging their additional distance structure in terms of the closeness to the positives.br / br / Boosting We invoke the framework of boosting to learn the similarity function. Boosting iteratively selects weak learners to form a strong learner using an additive form: F ( x) =br / br / ∑br / br / f m ( x )∈Ωbr / br / f m ( x), br / br / (13)br / br / where F(x) is the strong learner, fm(x)'s are the weak learners, and Ω is the structural space where the weak learners reside. There exist many classifiers in the literature, such as multi-layer perceptron, support vector machine, Gaussian process, boosting, etc. Among them, support vector machine and boosting are probably the most influential ones due to their known large margin property or good generalization capability. We choose boosting over support vector machine mainly due to computational reasons. Using support vec-br / br / 136br / br / Similarity Learning for Motion Estimationbr / br / Figure 2. (a) A pair of consecutive frames in a stress echo sequence. (b) The cropped LV images from the two frames. (c) The difference of the two LV images. (d) The positive pair for the left annulus point. (e) Four negative pairs for the left annulus point, i.e., the left end point on the contour.br / br / (a)br / br / (b) (c) (d)br / br / (e)br / br / tor machine needs to store a lot of support vectors, which means a huge memory space, and to compare the query example with support vectors using some kernel function, which means a slow evaluation. In contrast, boosting combines weak learners that are based on local rectangle features into a strong learner and thus operates as a feature selector. As these features are calculated online, there is no need to store the training dataset in the memory and it is rapid to evaluate the boosted classifier. Boosting has three key components: (1) structural space; (2) target/cost function; and (3) selection algorithm. Different variants of boosting are proposed in the literature depending on different choices of the key components. We decide to use the LogitBoost algorithm (Friedman, Hastie, & Tibshirani, 2000) summarized in Fig. 3. The LogitBoost algorithm differs from the commonly used AdaBoost algorithm approach (Freund & Schapire, 1997) in the following two aspects. First, they optimize different cost functions. The AdaBoost algorithm minimizes an upper bound of the target misclassification error; the LogitBoost algorithm directly minimizes a negative binomial log-likelihood, which is a natural choice for a binary classification problem. Second, the weak learner in the AdaBoost is a hard classifier while that in the LogitBoost is not: experimental evidence seems to favor the latter. 137br / br / Similarity Learning for Motion Estimationbr / br / Figure 3. The two-class LogitBoost algorithm (Friedman, Hastie, & Tibshirani, 2000). (From (Zhou et al. 2006) at IEEE 2006) Two-class Logit Boost Algorithm (positive y=1 and negative y=0) 1. Input: (i) Training data {xi;.i=1,2,…,N} and their corresponding class labels {yi;.i=1,2,…,N}. (ii) The structural space Ω. 2. Start with weights wi=1/N, i=1,2,…,N, F(x)=0, and probability estimates p(xi)=1/2. 3. Repeat for m=1,2,...,M: a. Compute working responses and weights:br / br / zi = b.br / br / yi - p ( xi ) ; wi = p ( xi )(1 - p ( xi )) . p ( xi )(1 - p ( xi ))br / br / Fit the function fm(x) by a weighted least-squares (LS) regression of zi to xi with weights wi. Nbr / br / f m ( x) = arg min ( f ) = ∑ wi ( zi - f ( xi )) .2 f ∈Ωbr / br / c.br / br / (14)br / br / Update F(x) and p(x) viabr / br / 1 f m ( x), 2br / br / (16)br / br / exp( F ( x)) . exp( F ( x)) + exp(- F ( x))br / br / (17)br / br / F ( x) ← F ( x) +br / br / p( x) =br / br / (15)br / br / i =1br / br / Output the classifier sign[F(x)].br / br / In our context, a data point x is an image pair x=(I,I'). One obvious choice for the boosted similarity function s(I,I') is the probability of the class label y(I,I') being 1, that is s(I,I')=p(I,I'). s ( I , I ') = p ( I , I ') =br / br / exp( F ( I , I ')) , exp( F ( I , I ')) + exp(- F ( I , I '))br / br / (18)br / br / Using boosting to learn a similarity function has recently attracted attention. In (Jones & Viola, 2003), Jones and Viola learned a similarity function for face recognition. Even though the similarity function takes two image patches as input, it is the difference of the two images that are actually used, which is in the same spirit of constructing intrapersonal and interpersonal spaces (Moghaddam, Jebara, & Pentland, 2001), because the boosted local features are computed on top of the difference image. In (In, 2006 Yu, Amores, Sebe, & Tian independently proposed to boost a strong function by combining various generic similarity functions. The similarity function learned in such a way frees the assumption of parametric noise model, but is still affected by the constant brightness assumption.br / br / Sructural Space, Weak Learner, and Feature Selection The structural space is the key to the performance of the learned classifier. In general, the space should be rich enough such that a simple additive function in (13) can characterize a complex decision boundary. In addition, the weak learners in the structural space should be robust to appearance variation and fast to evaluate. As mentioned earlier, combining the generic similarity functions such as those listed in Section II into a strong committee (Yu, et al., 2006) is still limited. Since different similarity functions might be effective for different local regions, we construct the weak learners f(I,I') based on Haar-like localbr / br / 138br / br / Similarity Learning for Motion Estimationbr / br / rectangle features (Papageorgiou, Oren, & Poggio, 1998) (Viola & Jones, 2001), whose rapid evaluation is enabled via the use of integral image. As shown in (Viola & Jones, 2001), (1) it is easy to construct numerous local rectangle features, which renders a rich structural space, and (2) the local rectangle feature, whose response is normalized by the standard deviation of the image patch, is relatively robust to appearance variation. A weak similarity function, based on a local rectangle feature, compares two local rectangle regions belonging to the two images I and I', respectively. As illustrated in Fig. 4(a), we parameterize the rectangle feature g by (r,c,dr,dc,t) where (r,c) is the starting point of the rectangle, (dr, dc) is the height and width, and t is the feature type. There are six feature types as shown in Fig. 4(a); other feature types can be used too. Given a rectangle feature g with a configuration (r,c,dr,dc,t) and an image pair (I,I'), we compute two feature responses g(I|r,c,dr,dc,t) and g(I'|r,c,dr,dc,t) from the two integral images associated with I and I', respectively. In principle, we can allow that two local rectangles have different parameters; however we refrain from doing this because empirically this shows no clear advantage but significantly increases training complexity. We focus on the 2D feature space of the two feature responses g(I) and g(I') and model the weak learner f(I,I') as a 2D piecewise constant function of g(I) and g(I'), which has the following form: f ( I , I ' | r , c, dr , rc, t ,{br / br / Jbr / br / Kbr / br / jk },{T j },{Tk '}) = ∑∑ j =1 k =1br / br / jkbr / br / [ g ( I ) ∈ T j ] ∧ [ g ( I ') ∈ Tk '], br / br / (19)br / br / where [π] is an indicator function of the predicate π and αjk is the constant associated with the region Rjk. In the above, we use a tessellation of the 2D feature space into non-overlapping regions { Rjk= Tj ^ Tk'; j=1,2,…,J, k=1,2,…,K}, where {Tj, j=1,2,…,J} and {Tk', k=1,2,…,K} are J and K non-overlapping intervals for the feature response g(I) and g(I'), respectively. The crucial step in the LogitBoost algorithm in Fig. 3 is step 3(b): fitting a weighted least square regression of zi to xi with weights wi. It operates as a feature selection oracle: picking up from the structural space Ω the weak learner f(I,I'|r,c,dr,dc,t,{αjk},{Tj},{Tk'}) that minimizes the weighted least square cost ε(f) defined in (15). Mathematically, we have the following optimization task: f m ( I , I ') = arg min r ,c , dr , rc ,t ,{br / br / Nbr / br / jk },{T j },{Tk '}br / br / Jbr / br / Kbr / br / ∑ w ( z - ∑∑ i =1br / br / ibr / br / ibr / br / j =1 k =1br / br / jkbr / br / [ g ( I i ) ∈ T j ] ∧ [ g ( I i ') ∈ Tk ']) 2, br / br / (20)br / br / where (Ii,Ii') is the ith training image pair. This is a huge parameter space to search. The following three steps are used for efficiency. 1. br / br / 2. br / br / br / br / First we empirically fix the Tj , Tk' for a given configuration (r,c,dr,dc,t) as follows: Calculate the feature responses from all training data points, determine the minimum and maximum response values, and uniformly divide them into the J (or K) intervals for image I (or image I'). We then find the optimal weights αjk, given a weak learner f associated with a feature (r,c,dr,dc,t), by minimizing the weighted least square cost ε(f) in (15). It is easy to show that the optimal weight αjk is the weighted response z of all data points falling into the region Rjk.br / br / ∑ = ∑br / br / Nbr / br / jkbr / br / i =1 Nbr / br / wi zi [ g ( I i ) ∈ T j ] ∧ [ g ( I i ') ∈ Tk ']br / br / i =1br / br / wi [ g ( I i ) ∈ T j ] ∧ [ g ( I i ') ∈ Tk ']br / br / . br / br / (21)br / br / 139br / br / Similarity Learning for Motion Estimationbr / br / Figure 4. (a) A weak similarity function compares two local rectangle regions belonging to images I and I', respectively. Panels from (b) to (d) illustrate the process of fitting a 2D piecewise constant function: (b) the field of w*z for all positives; (c) the field of -w*z for all negatives; and (d) the fitted 2D piecewise constant function. (From (Zhou et al. 2006) at IEEE 2006)br / br / (a) (b)br / br / (c) (d)br / br / 3. br / br / Finally, we exhaust all possible configurations of (r,c,dr,dc,t) to find the smallest weighted least square cost ε(f) for the step 3(b) in Fig. 3, which implements the feature selection. In other words, the weak function f based on the feature (r,c,dr,dc,t) with its optimal weights that yields the smallest weighted least square cost ε(f) is selected.br / br / Fig. 4 illustrates the process of finding the optimal weights in the piecewise constant function for a given feature. Fig. 4(b) visualizes the field of wi*zi=yi-p(xi)=1-p(xi) for all positives used in the first experiment of motion estimation in Section IV, where the color intensity corresponds to the value of w*z: the redder the plus sign is, the less likely the data point x is positive. The diagonal structure in Fig. 4(b) shows that the two feature responses of the positives are roughly same, which matches the intuition of motion estimation. Fig. 4(c) visualizes the field of -wi*zi=p(xi) for all negatives: the greener the circle sign is, the less likely the data point x is negative. As shown in Fig. 4(c), the negatives are characterized by a widely-dispersed nature. Fig. 4(d) shows the fitted 2D piecewise constant function: the constant coefficients αjk along the diagonal lines are high, while off-diagonal ones are low. Using nonparametric 2D piecewise constant functions as weak learners brings two extra benefits in terms of computation. • br / br / 140br / br / Saving in training time. Take the 1D case for example; 1D simple regression stumps (Viola & Jones, 2001) that `binarize' the feature response are often used as weak learners in the literature. It is easybr / br / Similarity Learning for Motion Estimationbr / br / • br / br / to verify that any 1D piecewise constant function can be constructed by combining multiple 1D simple regression stumps. The similar holds for the 2D case. Such a combination strengthens the modeling power of weak learners and consequently accelerates the training process. Our empirical evidence shows that the learning time is almost inversely proportional to the number of thresholds used in the weak learner. One may argue that it brings the risk of overfitting. But boosting has the capability of combating the overfitting (in terms of classification though) even when the weak learner overfits; refer to the discussion part of (Friedman, Hastir, & Tibshirani, 2000). Further, in practice we smoothed the fields of w*z and w before taking the division in (20) to ameliorate the overfitting of the weak learner itself. Saving in memory requirement. Boosting training requires calculating a matrix, whose row corresponds to the local rectangle feature and whose column to the training image pair. It is desired to store such a matrix in the memory in order to speedup the training process. Typically, the number of rectangle features is huge, e.g., more than 150K for a 24*24 image by an exhaustive construction (Viola & Jones, 2001). In one of our experiments, we kept about 40K rectangle features and about 10K training image pairs. To store the above matrix in a float precision, it consumes about 40K*10K*4* 2=3.2GB memory space, which exceeds the 2GB limit, the maximum contiguous block of memory of a 32-bit operating system. However, to learn the piecewise constant functions in our setting, we only need to store the interval index in the memory; refer to (20). In practice, we used 16 intervals, implying that an unsigned char precision is enough to store two indices, leading to a moderate memory requirement of about 400MB.br / br / Location-Sensitive Cascade Training Generating positives and negatives, which are pairs of images, from annotated videos is illustrated in Fig. 2(d,e) and Fig. 5(a). Given a pair of two successive frames (the left and right images in Fig. 5(a)), it contributes one positive by cropping two image patches centered at the target pixel (denoted by the red color) from the left and right frames, respectively. To generate negatives, we maintain the same image patch cropped from the left frame, i.e., centered at the target pixel, but force the center of the image patch cropped from the right frame away from the target pixel. The negatives possess an additional location parameter that measures their distances to the target. Theoretically, the number of negatives is infinite if a non-integer pixel grid is used. To cover as many negatives as possible, we follow (Viola & Jones, 2001) to train a cascade of strong classifiers, which is a degenerate decision tree. After learning the strong classifier for the current stage using AdaBoost, the strong classifier threshold is tuned to pass all positives (or the majority of positives depending on the pre-specified detection probability). To train the strong classifier at a later stage, we maintain the same set of positives but bootstrap a new set of negatives that pass all previous strong classifiers (i.e., false positives). During scanning all test subwindows, the cascade structure is able to eliminate the negatives quickly. The motion estimation or image registration accuracy is directly related to the selection of the negatives. On one hand, if the negatives are far away from the positives, it is easy to learn a perfect classifier but the accuracy is not guaranteed. On the other hand, if the negatives are too close to the positives, the accuracy is improved but it is hard to train a flawless classifier and might step into the zone of overfitting because the training positives and negatives are too confusing. Often in medical applications, different experts disagree with each other about the ground truth; thus, the performance only needs to be addressedbr / br / 141br / br / Similarity Learning for Motion Estimationbr / br / Figure 5. (a) Two successive frames and their corresponding positives and negatives. (b) Location-sensitive cascade training. (c) Performance comparison among location-sensitive cascade, regular cascade, and detection. (From (Zhou et al. 2006) at IEEE 2006)br / br / (a) (b) br / br / (c)br / br / in a pre-specified precision, say the inter-user variability. To this end, we design a location-sensitive cascade training procedure that takes into account the location factor of the negatives. As illustrated in Fig. 5(a), we divide all pixels into several negative regions according to their distances to the target pixel, where the target pixel is denoted by the red color and the negative regions are color-banded and with different strips. While preserving the features of the regular cascade training, we impose an additional constraint for the location-sensitive cascade training procedure: the negatives for several consecutive stages of the cascade are restricted to be from the same negative region. Further, the later stages use negatives closer to the positives; however, bootstrapping is still applied even across the boundary of the stages that are attached to different negatives regions. This procedure is graphically illustrated in Fig. 5(b). Refer to section IV (the first experiment of motion estimation) for a comparison between the location-sensitive cascade training and regular one, where we show that the location-sensitive cascade empirically yields lower training and test errors (also see Fig. 5(c)). Apart from the above performance improvement, location-sensitive cascade training enables steering the performance: If we find that the desired of accuracy is only 8 pixels instead of 5 pixels, the target achieved by the initial location-sensitive training, then we simply skip, during testing, the later stages of the cascade that is trained using the negatives less than 8 pixels away. But, the regular cascade training procedure does not enjoy such flexibility: The best guarantee is to retrain. Fig. 6 summarizes the proposed locationsensitive cascade training algorithm. In practice, we need to convert the cascade output into a similarity function that measures the confidence of being positive. Suppose that the cascade consists of L stages and stage l has a strong classifier Fl(I,I') that can be converted to posterior probability sl(I,I') using (18). Given the degenerate nature of the cascade, we approximate the final similarity function as: s ( I , I ') ≈ ∏lL=1 sl ( I , I ') = ∏lL=1br / br / exp(2 Fl ( I , I ')) . exp(2 Fl ( I , I ')) + 1br / br / (22)br / br / For the negatives rejected at an early stage L'L, we stop evaluating them at later stages and simply set a dummy probability sl(I,I')=e if lL', where e0.5 is a small amount (we set e=0.1 in our experiments).br / br / 142br / br / Similarity Learning for Motion Estimationbr / br / Boo In this section, we investigate the use of the boosted discriminate similarity function in motion estimation (which is referred to BoostMotion). We first review the related literature on learning-based visual tracking and highlight our contribution. We then present our extensive experiments using echocardiographic and general tracking sequences.br / br / Review of Learning-Based Visual Tracking It should be emphasized that visual tracking is different from motion estimation. The latter concerns only two successive frames; while the former concerns the motion estimation of a whole video sequence. While a naive tracker estimates motion recursively frame by frame, tracking is more than motion estimation as the above naive tracker is prone to drift away. To overcome drifting, the tracking algorithm does the following: (1) updates the appearance model strategically and/or (2) performs temporal smoothing/fusion. Underlying learning-based tracking algorithms lays a data-driven procedure. In most cases except (Lepetit, Pilet, & Fua, 2004), a binary classifier that discriminates the object of interest from the background is learned either offline or online. If the classifier is learned offline, this solves tracking as a detection problem. In (Mikolajczyk, Choudhury, & Schmid, 2001), temporal smoothing is enforced by casting the detector output as the observation likelihood in a particle filter setting. Avidan (2001) proposed a support vector tracking algorithm that learns a support vector machine (SVM) from the training data using the polynomial kernel. The SVM score, after the Taylor expansion, is analytically maximized for every frame. In 2003 Williams, Blake, & Cipolla built on the relevance vector machine to perform tracking, where temporal fusion is applied. In 2004 Lepetit, Pilet, & Fua artificially generated exemplars (using affine transform or 3D model) for each feature point that is treated as a class and used the 1-NN neighbor searching to determine the class label. If the classifier is learned online, the appearance model updating is embedded into the classifier. In the work of Collins and Liu (2003) the appearance model is represented by a set of features that are selected online based on the variance ratio of the log likelihood function, which is empirically estimated. Ensemble tracking developed by Avidan (2005) invokes the AdaBoost to learn the classifier.br / br / Figure 6. The location-sensitive cascade training algorithm The location-sensitive cascade training algorithm 1. Input: an annotated dataset of pairs of frames, K = the number of cascade stages, Nk.= the number of weak classifiers for the kth.stage, PDk = the detection probability for the kth.stage, NREGk = the negative region for the kth.stage. 2. For k.=.1:K /*loop over the number of stages*/ • Data preparation: Always keep the same set of positives. If (k==1) then random sample the negatives inside the NREG1, otherwise bootstrap the negatives (e.g., the false positives from the previous stages up to k-1) from the range NREGk. When sampling or bootstrapping, the number of negatives is kept almost the same to that of positives. • Invoke the LogitBoost algorithm (e.g., Fig. 3) to learn a strong classifier. Stop training when the number of weak learners reaches.Nk. • Tune the strong classifier threshold to match the desired PDk. Output: a cascade of strong classifiers.br / br / 143br / br / Similarity Learning for Motion Estimationbr / br / After tracking, the classifier is updated by adding the recently tracked results. Since the AdaBoost is a feature selection process, the ensemble tracker also represents the appearance model by features that are updated over time. The approaches reviewed above commonly learn a classifier to differentiate the foreground object from the background. However, such a classifier gives no account of temporal information essential to motion estimation: No pairwise comparison along the temporal dimension is performed. In other words, the input to the classifier is always a single image patch not a pair of images. The proposed BoostMotion approach explores the possibility of using image pairs as inputs and embeds the temporal statistics into a learned similarity function. In (Viola, Jones, & Snow, 2003) the temporal difference images are used in boosting a pedestrian detector (again not a similarity function). Learning the spatial statistics of optical flow is addressed in (Roth & Black, 2005).br / br / Experiments on Echocardiogram The echocardiographic sequences are characterized by severe imaging artifacts and rapid appearance changes, which render traditional tracking algorithms that use conventional similarity functions ineffective. Here we address tracking the LV endocardial wall, which is parameterized by 17 landmark points along the contour as shown in Fig. 7(a) and then interpolated using a cubic spline.br / br / Figure 7. (a) Two successive original frames and their ground truth landmarks and contours. (b) From left to right: the response maps of the similarity functions of NCC, CD2 and BoostMotion. (From (Zhou et al. 2006) at IEEE 2006)br / br / (a)br / br / (b)br / br / 144br / br / Similarity Learning for Motion Estimationbr / br / Motion Estimation In the first experiment, we used the apical four chambers (A4C) view of stress echo sequences. Since an echocardiographic image is a 2D slice of 3D heart, different orientations of the ultrasound probe give different views of the heart. In clinical practices, standard cardiac views are captured, among which the A4C view is one such view that images all four heart chambers, namely left and right ventricles and left and right atria. We had 339 A4C sequences, providing 3162 frame pairs. The ground truth contour annotations for all 339 sequences were provided by one experienced sonographer. We randomly divided 339 sequences into two sets: the training set contains 270 sequences with 2543 pairs of frames and the test set 69 sequences with 619 pairs of frames. To reduce appearance variation, we aligned each video frame with respect to a mean shape using a rigid similarity transform and conducted experiments on the aligned domain. We reported the results of estimating the motion vector of the left annulus point, i.e., the left end point of the LV endocardium, which is characterized by drastic appearance changes mainly due to the valve movement. Given the correct left image patch, we exhaustively searched within a searching neighborhood the best right image patch that maximizes the similarity function. We estimated the motion vector for all test image pairs and measured the estimation error in terms of absolute displacement. For a pair of image patches, we set the patch size as 35 x 35 and the searching window W as [-21, 21] x [-21, 21]. We used the location-sensitive cascade training to learn a cascade of eight stages. We divided the whole search neighborhood into eight negative regions depending on their closeness to the center: specifically, they are R1={21,20,19}, R2={18,17,16}, R3={15,14,13}, R4={12,11}, R5={10,9}, R6={8,7}, R7={6,5}, R8={4,3} pixels away from the center. To train the lth stage of the cascade, we bootstrapped negatives from the region Rl. For comparison, we also trained a regular cascade by randomly selecting out negatives at least three pixels away from the ground truth. Fig. 5(c) plots the curves of the training and test errors against the number of cascade stages. The location-sensitive cascade training consistently reduces the test error till overfitting is reached; while the regular cascade training saturates the performance even at the second stage. We contrasted the BoostMotion approach, which uses pairs of images as inputs, with a detection algorithm, which uses single image patches as inputs. For the detection algorithm, we also trained a cascade of eight stages. From Fig. 5(c), even the training error of the detector is higher than that of the BoostMotion, which implies that the positives and negatives, which are single image patches, are very confusing even when they are far apart. Using the paired inputs significantly reduces the training and test errors. This is expected because motion estimation compares two images and thus temporal information is essential: Motion estimation based on one image is insufficient. We compared the learned similarity function with the four conventional similarity functions reviewed in Section II. Table 1(a) reports the median and standard deviation of the estimation error. For the BoostMotion and detection approaches, we reported the minimum test error by varying the number of cascade stages. The BoostMotion approach yields the lowest training and test errors. In terms of test error, on average, it is only 3.47 pixels away from the ground truth, while the best among the others is the CD2 similarity function whose estimation error is 4.38 pixels. The poor performance of the Bhattacharyya similarity function is probably due to the highly noisy nature of the ultrasound image and that only gray image is used. Fig. 7(b) displays the response maps of different similarity functions for the sample pair of frames in Fig. 7(a). The response map of the BoostMotion is peaked around the ground truth with a compact support region. Most of the off-center pixels are black because they arebr / br / 145br / br / Similarity Learning for Motion Estimationbr / br / Table 1. (a) Motion estimation training and test errors and (b) tracking performance based on the contour distances obtained using different similarity functions and a detector approach. Approach SSD NCC BHAbr / br / (a) Motion estimation training error 4.62 ± 2.48 4.59 ± 2.38 11.15 ± 6.66br / br / (a) Motion estimation test error 4.54 ± 2.57 4.49 ± 2.45 11.40 ± 6.75br / br / (b) Tracking error point-to-point 10.56 ± 2.37 11.14 ± 2.45 14.31 ± 3.19br / br / (b) Tracking error segmental Hausdorff 5.63 ± 1.25 5.84 ± 1.32 7.22 ± 1.62br / br / CD2 Detection BoostMotionbr / br / 4.39 ± 2.28 8.34 ± 4.99 2.32 ± 1.72br / br / 4.38 ± 2.17 8.68 ± 5.07 3.47 ± 2.14br / br / 7.32 ± 2.45 NA 4.28 ± 1.24br / br / 3.70 ± 1.30 NA 2.16 ± 0.74br / br / rejected by early stages of the cascade. In terms of computational time, the BoostMotion is slower than the generic similarity functions, because we used 654 weak learners.br / br / Echocardiography Tracking In the second experiment, we invoked the naive tracking algorithm that estimates motion vector frame by frame to perform echocardiography tracking (Cohen & Dinstein, 2002) (Boukerroui, Alison, & Brady, 2003) (Jacob, Noble, & Blake, 1998) (Mikic, Krucinski, & Thomas, 1998) (Zhou, Comaniciu, & Gupta, 2005). We used 445 regular echocardiographic sequences of apical two chambers (A2C) view where appearance changes are less pronounced than the stress echo. The A2C view is another canonical echo view used in clinical practices, where only left ventricle and left atrium are present. The ground truth annotations were provided by three experts, each annotating a non-overlapping subset of the whole dataset. We randomly divided the 445 sequences into a training set of 356 sequences and a test set of 89 sequences. The alignment is conducted in a recursive fashion. To calibrate the contour tracking accuracy, we need to measure the proximity between two contours. We used two measures. The first is a simple point-to-point measure: the average distance of the landmark displacement defined asbr / br / ∑br / br / 17 i =1br / br / | pi - g i |2 /17,br / br / where pi and gi are the ith landmark point on the probe and ground truth contours, respectively. The second is the so-called segmental Hausdorff distance (segHD) that allows certain degree of non-rigidity deformation during contour matching. The segHD between two corresponding landmark points x and x' on the two curves C and C', respectively, is defined as the Hausdorff distance between two segments ω(x) and ω(x'), where ω(x) defines a segment around x on the curve C. We then take the mean of the segHD of all landmarks as the distance between C and C', denoted by dsegHD(C, C').br / br / shd ( x, x ') = HD( ( x), ( x ')); ( d segHD C , C ') = ∫ shd ( x, x ')dC / ∫ dC . xbr / br / xbr / br / (23)br / br / The tracking performances in terms of the above distances are listed in Table 1(b). Since each test sequence yields a contour distance, Table 1(b) documents its median and standard deviation. The segmental Hausdorff distance is about half smaller than the point-to-point distance as the former toleratesbr / br / 146br / br / Similarity Learning for Motion Estimationbr / br / Figure 8. Sample frames of two echocardiographic sequences with the ground truth (green) and tracking result (yellow) overlaid. (a,b) Frames 0, 5, 10, and 15 of the first sequence with results by (a) BoostMotion and (b) CD2. (c,d) Frames 0, 5, 10, and 15 of the second sequence with results by (c) BoostMotion and (d) CD2.br / br / (a)br / br / (b)br / br / (c)br / br / (d)br / br / 147br / br / Similarity Learning for Motion Estimationbr / br / non-rigid deformation during contour matching and seems more compatible to our perception of contour difference. Using the discriminative similarity function substantially reduces the tracking error, with its corresponding point-to-point distance and segmental Hausdorff distance being 4.28 and 2.16 pixels, respectively; while the second best similarity function (CD2) yields a tracking errors of 7.32 and 3.70 pixels, respectively. Therefore, utilizing the boosted similarity function greatly reduces drifting and produces a temporally smooth contour. Some tracking examples are presented in Fig. 8, which compares BoostMotion with CD2, the second best similarity function.br / br / Visual Tracking with Temporal Adaptation In the previous two experiments, we assumed the existence of a large database that covers the appearance variances typically observed in the application of interest. While the above medical application provides such a database and hence is suitable for demonstrating the effectiveness of BoostMotion, it is not difficult to adapt the discriminative similarity function on the fly, making it appropriate for a general visual tracking problem. Here we present our experimental results along this line. We experimented with our previous appearance-adaptive tracking approach reported in (Zhou, Chellappa, & Moghaddam, 2004), which is based on a state space model for an affine parameterization and a particle filter method. In the observation process of the state space model, the online appearance model, which is a generative mixture of a stable component and a wondering component (Jepson, Fleet, and El-Maraghi, 2003), is updated to absorb changing appearance; in the state transition process of the state space model, an adaptive velocity is predicted based on the incoming appearance and hence guides a better particle generation; the number of particles is made adaptive to the prediction error too. Because the main contribution of the chapter is on the similarity function itself, we simply replaced the generative online appearance model in the observation likelihood with the discriminative similarity function. To initiate learning the similarity function, we manually annotated two starting frames with a bounding box. To generate positives, we perturbed the ground truth boxes in both frames by a small amount and cropped 20 images pairs. To generate negatives, we kept the image patches in the first frame within the positive perturbation region while cropping image patches in the second frames outside the positive perturbation region but within the particle sampling region. In total, we cropped 200 negative image pairs. We then learned a classifier of 25 weak learners. To adapt the similarity function to incoming appearance, we proceeded as follows. Once the tracking is done on the current frame t, we augmented the training dataset by cropping positives and negatives based on tracking results of the current and previous frames and removed the training data generated from t0=5 frames ago. Therefore, we maintained a training dataset of 100 positives and 1000 negatives except for the first 5 frames. During training, we controlled the number of local rectangle features to be around 1000. Overall, the tracker can handle a few frames per second on a PC with a 3GHz CPU and 3GB memory. Fig. 9 displays the tracking results. Even though we displayed color image, we used the gray values (the average of RGB channels) as input. In the first example, we tracked a moving head in a surveillance video of 150 frames captured by PETS2001, which is available from http://www.cvg.rdg.ac.uk/PETS2001/. In the second example, we tracked a car driving in fog captured by researchers in Karlsruhe University. The sequence is from http://i21www.ira.uka.de/image/_sequences. In both examples, the challenges lie in small target size, dynamic foreground appearance, cluttered and changing background, etc. Despite all these challenges, our algorithm successfully tracks the target. We observe that the detector-based approach, which leverages no temporal information, yields more jittery results. 148br / br / Similarity Learning for Motion Estimationbr / br / Figure 9. The top two rows are about tracking a moving head: frames 1, 30, 60, 90, 120, and 150. The bottom two rows are about tracking a driving car in fog: frames 1, 30, 60, 90, 120, and 150.br / br / Su We presented an approach to learning a similarity function for matching image under complex appearance changes, which are exemplified by an annotated video database. This type of similarity function is applicable to motion estimation. We used the LogitBoost algorithm to selectively combine weak learners into a strong similarity function. The weak learners were constructed as nonparametric 2D piecewise constant functions of the feature responses, which enhanced modeling power and brought savings in training time and storage requirement. Because the motion estimation accuracy is tied with the selection of negatives, which have an additional location parameter measuring their distances to the positives, we proposed to train a cascade structure in a steerable manner using a location-sensitive negative bootstrapping. Compared with the regular cascade, the location-sensitive cascade achieved lower training and test error. Finally, we tested the learned similarity function on motion estimation forbr / br / 149br / br / Similarity Learning for Motion Estimationbr / br / echocardiographic sequences and general visual tracking: the learned similarity function outperformed conventional similarity functions.br / br / REFERENCES Otto, C. (2004). Textbook of Clinical Echocardiography. Saunders. Freund, Y., & Schapire, R. (1997). A decision-theoretic generalization of online leaning and an application to boosting. J. Computer and System Sciences, 55(1), 119-139. Friedman, J., Hastie, T., & Tibshirani, R. (2000). Additive logistic regression: a statistical view of boosting. Ann. Statist., 28(2), 337–407. Cohen, B., & Dinstein, I. (2002). New maximum likelihood motion estimation schemes for noisy ultrasound images. Pattern Recognition, 25, 455-463. Papageorgiou, C., Oren, M., & Poggio, T. (1998). A general framework for object detection. In Proc. of ICCV. Viola, P., & Jones, M., (2001). Rapid object detection using a boosted cascade of simple features. In Proc. of CVPR. Santini, S., & Jain, R. (1999). Similarity measures. IEEE Trans. PAMI, 21(9), 871–883. Brox, T., Bruhn, A., Papenberg, N., & Wiecker, J., (2004). High accuracy optical flow estimation based on a theory of warping. European Conf. Computer Vision, (pp. 25–36). Lucas,B., & Kanade, T. (1981). An iterative image registration technique with an application to stereo vision. In Proc. DARPA IU Workshop, (pp. 121–130). Shi, J., & Tomasi, C. (1994). Good features to track. In Proc. of CVPR, (pp. 593–600). Fowlkes, C., Martin, D., & Malik, J. (2003). Learning affinity functions for image segmentation: Combining patch-based and gradient-based approaches. In Proc. of CVPR. Lundqvist, R., Bengtsson, E., & Thurfjell, L. (2003). A combined intensity and gradient-based similarity criterion for interindividual SPECT brain scan registration. EURASIP Journal on Applied Signal Processing, 1, 461 – 469. Comaniciu, D., Ramesh, V., & Meer, P. (2000). Real-time tracking of non-rigid objects using mean shift. Proc. of CVPR, 2, 142–149. Boukerroui, D., Alison, J.N., & Brady, M. (2003). Velocity estimation in ultrasound images: A block matching approach. IPMI, (pp. 586–598). Viola, P., & Welles, W.M. (1997). Alignment by maximization of mutual information. Int. J. Comp. Vision, 24(2), 137–154. Zhou, S., Shao, J., Georgescu, & Comaniciu, D. (2006). Boostmotion: Boosting a discriminative similarity function for motion estimation. Proc. of CVPR, 2, 1761–1768.br / br / 150br / br / Similarity Learning for Motion Estimationbr / br / Jones, M., & Viola, P. (2003). Face recognition using boosted local features. In Proc. of ICCV. Moghaddam, B., Jebara, T., & Pentland, A. (2001). Bayesian face recognition. Pattern Recognition, 33, 1771-1782. Yu, J., Amores, J., Sebe, N., & Tian, Q. (2006). Toward robust distance metric analysis for similarity estimation. In Proc. of CVPR. Lepetit, V., Pilet, J., & Fua, P., (2004). Point matching as a classification problem for fast and robust object pose estimation. In Proc. of CVPR. Mikolajczyk, K., Choudhury, R., & Schmid, C. (2001). Face detection in a video sequence - a temporal approach. In Proc. of CVPR. Avidan, S. (2001). Support vector tracking. Proc. of CVPR, 1, 184–191. Williams, O., Blake, A., & Cipolla, R., (2003). A sparse probabilistic learning algorithm for real-time tracking. In Proc. of ICCV, 2003, 353–360. Collins, R., & Liu, Y. (2003). On-line selection of discriminative tracking features. In Proc. of ICCV. Avidan, S. (2005) Ensemble tracking. In Proc. of CVPR, 2, 494–501. Viola, P., Jones, M., & Snow, D. (2003). Detecting pedestrians using patterns of motion and appearance. In Proc. of ICCV, (pp. 734– 741). Roth, S., & Black, M.J. (2005). On the spatial statistics of optical flow. In Proc. of ICCV. Jacob, G., Noble, A., & Blake, A. (1998). Robust contour tracking in echocardiographic sequence. In Proc. Intl. Conf. on Computer Vision, (pp. 408–413). Mikic, I., Krucinski, S., & Thomas, J.D. (1998). Segmentation and tracking in echocardiographic sequences: Active contours guided by optical flow estimates. IEEE Trans. Medical Imaging, 17, 274–284. Zhou, X.S., Comaniciu, D., & Gupta, A. (2005). An information fusion framework for robust shape tracking. IEEE Trans. PAMI, 27(1), 115–129. Zhou, S., Chellappa, R., & Moghaddam, B. (2004). Visual tracking and recognition using appearanceadaptive models in particle filters. IEEE Trans. Image Processing, 11, 1434-1456. Jepson, A.D., Fleet, D.J., & El-Maraghi, T.F. (2003). Robust online appearance models for visual tracking. IEEE Trans. PAMI, 25, 1296–1311.br / br / 151br / br / 152br / br / Chapter VIbr / br / Active Learning for Relevance Feedback in Image Retrieval Jian Cheng National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, China Kongqiao Wang Nokia Research Center, Beijing, China Hanqing Lu National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, China br / br / ABSTRACT Relevance feedback is an effective approach to boost the performance of image retrieval. Labeling data is indispensable for relevance feedback, but it is also very tedious and time-consuming. How to alleviate users' burden of labeling has been a crucial problem in relevance feedback. In recent years, active learning approaches have attracted more and more attention, such as query learning, selective sampling, multi-view learning, and so forth. The well-known examples include Co-training, Co-testing, SVMactive, etc. In this literature, the authors will introduce some representative active learning methods in relevance feedback. Especially, they will present a new active learning algorithm based on multi-view learning, named Co-SVM. In Co-SVM algorithm, color and texture are naturally considered as sufficient and uncorrelated views of an image. SVM classifier is learned in color and texture feature subspaces, respectively. Then the two classifiers are used to classify the unlabeled data. These unlabeled samples that disagree in the two classifiers are chose to label. The extensive experiments show that the proposed algorithm is beneficial to image retrieval.br / br / Copyright © 2009, IGI Global, distributing in print or electronic forms without written permission of IGI Global is prohibited.br / br / Active Learning for Relevance Feedback in Image Retrievalbr / br / InTRODUCTION Content-Based Image Retrieval (CBIR) has been one of the most active research topics in computer vision and pattern recognition fields since 1990's (Rui, Huang, & Chang, 1999; Smeulders, Worring, Santini, & Gupta, 2000; Datta, Li, & Wang, 2005). Most of existing CBIR systems adopted low-level features (color, texture, shape, etc) to represent images. However, it is inadequate to describe the semantic concepts with the low-level features of images, which is named semantic gap. The concept of semantic gap has been extensively used in the CBIR research community to express the discrepancy between the description of the low-level features extracted from the images and the semantic understanding of human. To narrow the semantic gap, a straightforward way is to take human in the loop. As one of important ways of human-in-the-loop, Relevance Feedback (RF) is a query modification technique, which was initially developed in document retrieval and then introduced into CBIR during mid 1990's (Picard, Minka, & Szummer, 1996; Rui, Huang, Ortega, & Mehrotra, 1998). Relevance feedback attempts to capture the user's preference through iterative feedback and query refinement. In each round, the user is requested to provide feedbacks regarding the relevance or irrelevance of the current retrieval results. Then the classifier will be refined based on the feedback results. The pool of unlabeled images is classified as relevance or irrelevance by the learned classifier, and the relevant images are ranked and returned to user for next round labeling. With the interactive labeling and learning procedures, systems can learn user's preferences and improve the performance of image retrieval. Many relevance feedback algorithms have been proposed for image retrieval in past years (Huang, & Zhou, 2001; Zhou, & Huang, 2003). The early work was mainly inspired by term-weighting and relevance feedback in document retrieval (Rocchio, 1971). Rui and Huang (1998) introduced the query refinement algorithm based on term-frequency and inverse-document-frequency in text retrieval into CBIR. Picard, et al. (1996) grouped the images or regions into hierarchical trees whose nodes were constructed through single-link clustering, and then weighted on grouping. These methods fall into heuristic-based formulation with empirical parameter adjustment. Later, there are more works focusing on learning-based strategy and many classic machine learning techniques are applied. Tieu and Viola (2000) assumed that an image was generated by a sparse set of visual cause and that images which were visually similar share causes. They proposed a mechanism for computing a very large number of highly selective features which captured some aspects of this causal structure, then used Boosting to learn a classification function which only relied on 20 features. In (Vasconcelos, & Lippman, 2000), Gaussian mixture model on DCT coefficient was used as image representation, then Bayesian inference was applied for image region matching and learning. Hong, et al. (2000) treated relevance feedback as a binary classification problem and incorporated Support Vector Machines (SVM) into the classification process. However, an inevitable issue in performing relevance feedback is the small sample size. The fact makes many learning methods inefficient, such as Bayesian, boosting, even SVM. In general, more labeled data will result in higher accuracy. While the labeled data is limited, the unlabeled data is often abundant and easy to obtain. Therefore, the key issue addressing the small sample size is how to efficiently utilize the unlabeled data. Recently, the idea of semi-supervised learning (SSL), i.e. learning from both labeled and unlabeled data, has attracted much attention in information retrieval (Zhu, 2006). One recent approach is to apply Transductive SVM (TSVM) to exploit the unlabeled data (Joachims, 1999; Wang, Chan, & Zhang, 2003). However, some study challenged that TSVM might not be so helpful from unlabeled data in theory and in practice (Zhang, & Oles, 2000).br / br / 153br / br / Active Learning for Relevance Feedback in Image Retrievalbr / br / On the other hand, labeling is tedious and time-consuming and users do not have patience to provide enough feedbacks. So it is extremely necessary to significantly alleviate users' burden of labeling while achieve comparable performance. To address this issue, active learning, in contrast to passive learning, is introduced to image retrieval. The main difference between active learning methods and the regular passive methods lies in how to choose the unlabeled samples from a pool of unlabeled data. Active learning is to actively select samples in order to achieve the maximal information gain, or the minimized uncertainty in decision making. The earlier approaches of active learning take a constructive approach to query generation in the sense that each query is constructed by setting the values of the attributes so that the query is as informative as possible (Angluin, 1988). Selective sampling is an alternative important approach of active learning (Freund, Seung, Shamir, & Tishby, 1997; Tong, & Koller, 2000). Selective sampling is typically applied to classification tasks in which the learner has access to a large number of unlabeled examples. There are two main schemes for sampling algorithms: stream-based and pool-based. The stream-based scheme assumes that unlabeled data are a stream and are presented in sequence to learner, while learner can go through the entire pool and select sample to label in pool-based condition. One of the most famous works in image retrieval is SVM Active proposed by Tong and Chang (2001). SVM Active is a pool-based algorithm. It employs SVM as learner and think that the samples lying beside the boundary can reduce the version space as fast as possible, i.e. eliminating the hypotheses. Therefore, in each round of relevance feedback, the images that are closest to the support vector boundary are returned to users for labeling. A much recent approach of active learning is multi-view based learning. In multi-view method, the feature set is partitioned to several subsets (views) which are sufficient and compatible for learning the target concept. The seminal work was presented by Blum and Mitchell (1998), Co-training algorithm. After that, many multi-view algorithms appeared. The representative works are Co-EM (Nigam, & Ghani, 2000), Co-Boost (Collins, & Singer, 1999), Co-Testing (Muslea, Minton, & Knoblock, 2000). These multi-view learning algorithms exploit unlabeled examples to boost the performance of classifiers learned in each view by bootstrapping the views from each other, so they are semi-supervised algorithms in nature. In this chapter, we will introduce some representative active learning methods in relevance feedback. Especially we will present a new active learning algorithm based on multi-view, named Co-SVM. In Co-SVM algorithm, color and texture are naturally considered as sufficient and uncorrelated views of an image. Two SVM classifiers are trained in color feature space and texture feature space, respectively. Then, the contention samples of two SVM classifiers in different feature representations are considered as the most informative samples and taken out to ask users to label. The extensive experiments show that the proposed algorithm can improve both efficiency and effectiveness of relevance feedback in content-based image retrieval.br / br / ACTIVE LEARNING FOR RELEVANCE FEEDBACK In relevance feedback, labeling is a troublesome task for users. Many learning methods fail due to most of users without patience to label sufficient sample. Active learning is a learning strategy designed to efficiently exploit the unlabeled data and reduce the labor cost of labeling. The basic idea of active learning is to select the most informative samples for labeling in each round feedback. Among the variousbr / br / 154br / br / Active Learning for Relevance Feedback in Image Retrievalbr / br / active learning techniques, SVM based method and Multi-view based method are the most promising method. In the section, we will give a brief introduction to the two techniques.br / br / Spport Vector Machines Based Active Learning Support Vector Machines (SVM) is built on structural risk minimization theory (Vapnik, 1998). With the labeled samples, SVM learns a hyper-plane (i.e., classification boundary) separating one class samples from the other class samples with maximum margin. Being an efficient binary classifier, SVM is particularly suitable for the classification task in relevance feedback of image retrieval (Joachims, 1999; Hong, Tian, & Huang, 2000). The task of learning user's preference can be regarded as one of learning a SVM binary classifier. An SVM captures the query concept by separating the relevant images from the irrelevant images with a hyper-plane in feature space. The samples on a side of hyper-plane are looked as relevance, and on the other side are looked as irrelevance. When the data set is not linear separable, the data usually need to be mapped into a high-dimensional feature space using an implicit function. Given a set of labeled samples (x1, y1),...,(xn, yn), xi is the feature representation of sample in input space xi ∈ R N, yi ∈ {–1, +1} is the class label (-1 denotes negative and +1 denotes positive). The basic goal of SVM is to find a separating hyper-plane that separates the training data with a maximal margin. The primal form of SVM in linear separable case can be expressed as: 1 min ....... .w T w 2 subject.to..yi ( wT xi + b) ≥ 1br / br / Training SVM classifier leads to the following quadratic optimization problem: nbr / br / min W ( ) = min{-∑ i =1br / br / ibr / br / +br / br / 1 n n ∑∑ yi y j 2 i =1 j =1br / br / ibr / br / jbr / br / k ( xi , x j )}br / br / Subject to: nbr / br / ∑y i =1br / br / ibr / br / ibr / br / ∀i,0 ≤br / br / =0 ibr / br / ≤Cbr / br / Where C is a constant, k is kernel function which is an identical function in case of linear separable. The hyper-plane (classification boundary) is (w · x) + b = 0. Wherebr / br / 155br / br / Active Learning for Relevance Feedback in Image Retrievalbr / br / nbr / br / w=∑ i =1br / br / x yibr / br / i ibr / br / andbr / br / 1 b = - w ⋅ [ xr + xs ] , 2 here xr and xs are any support vectors satisfied ar, as ≥ 0, yr = 1, ys = –1. The classification function can be written as: f ( x) = sign(∑ ibr / br / ibr / br / yi ⋅ k ( xi ⋅ x) + b)br / br / The earlier work generally applied SVM in relevance feedback using a randomly selected training set. Schohn and Cohn (2000) observed that they could achieve better performance from a small subset of the data than that achieved using all available data. They noticed that only unlabeled samples within the margin would have effect on classifier if the data was linearly separable, and stopped labeling data once the margin had been exhausted. Based on the assumption, they thought that the margin would exhaust when the sample closest to hyper-plane was support vector. At the same time, Tong and Koller (2000) also found this phenomenon in text classification. Taking advantage of the duality between parameter space and feature space, they arrived at three algorithms that attempted to reduce version space as much as possible at each query. These algorithms provide a guide to select better unlabeled samples, and lead to the same results as that Schohn and Cohn (2000) obtained. Tong and Chang further applied the idea to relevance feedback of image retrieval, i.e. SVMactive algorithm (Tong, & Chang, 2001). Before introducing the SVMactive algorithm, the concept of version space must be addressed, as it is critical to SVM based active learning. If there exist a set of hyper-planes that can linearly separate the training data in the input space, this set of hyper-planes or hypotheses is called the version space (Mitchell, 1982). This can be defined asbr / br / = { f ∈ H | yi f ( xi ) 0, i = 1,, n} or = {w ∈W | w = 1, yi ( w ⋅ ( xi )) > 0, i = 1,, n}
where H is the set of possible hypothesis and W is the parameter space for the unit vectors w. Note that the space H and W are dual, that is to say, points in the feature space correspond to hyper-plane in the parameter space and vice versa. The basic motivation of SVMactive is that the query sample should split the current version space into two equal parts as much as possible. According to the duality, the query sample should be close to hyper-plane. The relevance feedback process using SVMactive is comprised of the two steps: 1.
SVMactive learns the classifier quickly via active learning. The active part of SVMactive selects the most informative samples, i.e. the samples closest to hyper-plane, to train the SVM classier.
Active Learning for Relevance Feedback in Image Retrieval
Once the classifier is trained, SVMactive returns the top-k most relevant samples. These are the k samples farthest from the hyper-plane on the query concept side.
Multi-View Based Active Learning The traditional machine learning algorithms usually access to the entire set of domain features, which is called single-view learner. By contrast, in the multi-view setting one can partition the domain's features into subsets (views) that are sufficient for learning the target concept. Existing multi-view learning algorithms belong to semi-supervised algorithms. These algorithms exploit unlabeled data to boost the accuracy of the classifiers learned in each view by bootstrapping the views from each other. Blum and Mitchell (1998) presented the seminal work of multi-view based active learning. They provided the first formalization of learning in the multi-view framework, Co-training algorithm (Blum, & Mitchell, 1998). Co-training algorithm trains two classifiers separately on two sufficient and redundant views (feature subsets), and uses the predictions of each classifier on unlabeled samples to augment the training set of the other. Blum and Mitchell proved that two independent, compatible views could be used to PAC-learn a concept based on few labeled and many unlabeled samples. After Co-training algorithm appeared, many types of extensions were proposed. Collins and Singer (1999) presented a boosting-like framework based on Co-training that was biased towards learning hypotheses that predicted the same label on most of the unlabeled samples. Collins and Singer's work give an explicit objective function that measures the compatibility of the learned hypotheses and use a boosting algorithm to optimize it. Dasgupta, et al. (2001) reported the theoretical justification for some aspects of Co-training algorithm, and gave a new PAC-style bound on generalization error. Intuitively, they showed that the ratio of contention points to unlabeled samples was an upper-bound on the error rate of the classifiers learned in the two views. The idea of Blum and Mitchell (1998) also has been extensively incorporated into other machine learning algorithms. Nigam and Ghani (2000) applied multi-view learning for EM algorithm, named Co-EM algorithm (Nigam, & Ghani, 2000). In Co-EM algorithm, two classifiers are trained in two views respectively, and then recommend prediction to each other by turns during the training process of EM. Zhou and Li (2005) extended the co-training to semisupervised regression algorithm, i.e. COREG algorithm (Zhou, & Li, 2005). This algorithm employs two k-nearest neighbor regressors using different distance metrics. Like Co-training style, each regressor labels the unlabeled sample which can be most confidently labeled for the other regressor. The above mentioned Co-training algorithm and its extensions are based on the idea of bootstrapping the views from each other, which utilize the compatibility of learned classifiers with different views. In contrast, as another important multi-view based active learning algorithms, Co-testing algorithm uses the multiple views to detect the contention points (Muslea, Minton, & Knoblock, 2000). Co-testing algorithm relies on the following observation: if the hypotheses learned in each view predict a different label for an unlabeled sample, at least one of predictions is wrong. By asking the user to label such contention points, Co-testing algorithm is guaranteed to provide useful information for the view that made mistake. Co-testing is an iterative algorithm that requires a few labeled and many unlabeled samples. First, Co-testing uses the few labeled samples to learn a hypothesis in each view. Then it applies the learned hypotheses to all unlabeled samples and detects the content points that are predicted different label by hypotheses. Finally, it ask user to label these content points and add the newly labeled samples to the training set. The original Co-testing algorithm also has been extended to a family of Cotesting algorithms (Muslea, Minton, & Knoblock, 2006). The major difference is the query selection
strategies. There are three typical selection strategies: naïve, aggressive, and conservative. The naïve strategy chooses the contention points at random way; the aggressive strategy chooses the contention points that are the least confident of hypotheses makes the most confident prediction; the conservative strategy chooses the contention points on which the confidence of the predictions made by hypotheses is as close as possible.
Co For large-scale image database retrieval problem, labeled images are always rare compared with unlabeled images. It has become a hot topic how to utilize the large amounts of unlabeled images to improve the performance of the learning algorithms when only a small set of labeled images is available. Tong and Chang (2001) proposed an active learning paradigm for image retrieval, named SVM Active algorithm (Tong, & Chang, 2001). SVM Active considers the representation of an image as a single mixture view so that the least confident points are those which are the closest to the boundary. Usually, the feature representation of an image is a combination of diverse features, such as color, texture, shape, etc. For a specific image, the contribution of different features is significantly different. On the other hand, the importance of the same feature is also different for different images. For example, color is often more prominent than shape for a landscape image. However, the retrieval results are the averaging effort of all features, which ignores the distinct properties of individual feature. The works mentioned above have suggested that multi-view learning could do much better than the single-view learning in eliminating the hypotheses consistent with the training set (Blum, & Mitchell, Figure 1. The flowchart of Co-SVM algorithm
1998; Muslea, Minton, & Knoblock, 2000). In this chapter, from multi-view viewpoint, we propose a novel active learning algorithm by integrating the spirit of SVM Active and Co-testing, which named CoSVM algorithm (Cheng, & Wang, 2007).
Two-View Scheme It is natural and reasonable to assume that color features and texture features are two sufficient and uncorrelated views of an image. In each round, the contention samples which disagree in the two views are returned to user to label. Assume that x = {c1,...,ci, t1,...,tj} is the feature representation of an image, where {c1,...,ci} and {c1,...,cj} are color attributes and texture attributes, respectively. For simplicity, we define the feature representation V = VC × VT, and {c1,...,ci} ∈ VC, {t1,...,tj} ∈ VT. In order to find relevant images as much as possible, like the general relevance feedback methods, SVM is used to learn a classifier h on these labeled samples with the combined view V at the first stage. The unlabeled set is classified into positive and negative. Then m positive images are returned to user to label. At the second stage, SVM is used to learn two classifiers hC and hT on labeled samples with color view VC and texture view VT, respectively. A set of contention samples that disagree in the two views is recommended to user to label. That is, the contention samples are classified as positive by hC (CP) while are classified as negative by hT (TN), or are classified as negative by hC (CN) while are classified as positive by hT (TP). In each classifier, the distance between sample and the hype plane (support vector boundary) can be looked as the confidence degree. The larger the distance, the higher the confidence degree is. In order to makes sure that user can label the most informative samples, these samples which are close to hype plane in both views are recommended to user to label. In summary, the flowchart of Co-SVM algorithm is described in figure1. In Co-SVM algorithm, the relevance feedback procedure is divided into two stages. The aim is to improve not only the efficiency but also effectiveness of feedback. At the first stage, in order to insure users finding relevant images as much as possible, we rank the images by their similarity to the query on the combined view V like most of common relevance feedback algorithms. That is for effectiveness. The second stage let users to get the most informative images to label so that maximally reduce user's burden. This is for efficiency. The Figure 2 depicts the simple steps of Co-SVM algorithm applied to relevance feedback.
Multi-View Scheme The proposed algorithm in two-view case is easily extended to multi-view scheme. Assume the feature representation of a color image is defined as V = V1 × V2 × ... × V k, each Vi, i = 1,...,k correspond to different view of the color image. Then k SVM classifiers hi can be individually learned on each view. All unlabelled data are classified as positive (+1) or negative (-1) by k SVM classifiers, respectively. Define the confident score: S ( x) =
∑ sign(h ( x)) i =1
The confident score can reflect the consistency of all classifiers on a specified example. Higher the confident score, more consistent the classification is. Inversely, the lower score indicates that the classification is uncertain. The labeling on these uncertain samples will results in maximum improvement of performance. So the unlabelled samples whose confident scores are the lowest are considered as the contention samples.
Eperiments To validate the effectiveness of the proposed algorithm on improvement of performance, we compare it with Tong&Chang's SVM Active and the traditional relevance feedback algorithm using SVM (Hong, Tian, & Huang, 2000). Experiments are performed on a subset selected from the Corel image CDs. There are 50 categories in our subset. Each category contains 100 images, 5000 images in all. The categories have different semantic meanings, such as animal, building, landscape, etc. In our experiments, the main purpose is to verify if the learning mechanisms of Co-SVM are useful, so we only employed simple color and texture features to represent images. The color features include Figure 2. The pseudo-code of Co-SVM algorithm Active learning with Co-SVM: Input: a query image. Step 1.
Ranking the database in descending order of similarity to the query.
If user is unsatisfied; repeat the follows Step 2. Training a SVM classifier h with the entire view V and classifying Set{unlabeled}. Labeling the top m images as Set*{relevant} or Set*{irrelevant} Set {relevant} = Set {relevant} ∪ Set*{relevant} Set {irrelevant} = Set {irrelevant} ∪ Set*{irrelevant} Set {unlabeled} = Set {unlabeled} – {Set*{relevant} ∪Set*{irrelevant}} Step 3.
Training a SVM classifier hC with color view VC and classifying Set {unlabeled} Set {unlabeled} = {CP ∪ CN} Training a SVM classifier hT with texture view VT and classifying Set {unlabeled} Set {unlabeled} = {TP ∪ TN}
Finding contention samples Set {contention} = {CP ∩ TN} ∪ {CN ∩ TP}
Step 5. Ranking Set {contention} by their similarity to the query calculated in step 1, and labeling the least confident n samples as Set*{relevant} or Set*{irrelevant}. Set {relevant} = Set {relevant} ∪ Set*{relevant} Set {irrelevant} = Set {irrelevant} ∪ Set*{irrelevant} Set {unlabeled} = Set {unlabeled} - {Set*{relevant} ∪Set*{irrelevant}} End of repeat Output: the relevant images.
125-dimensional color histogram vector and 6-dimensional color moment vector in RGB. The color mean and color variance in Red, Green and Blue are calculated to form the 6-dimensional color moment vector. The texture features are extracted using Discrete Wavelet Transformation (DWT). Each wavelet decomposition results in 4 subimages: down-sampled image and wavelets in three orientations. In total, we perform 3-level wavelet decomposition and obtain 10 wavelet subbands. The mean and variance averaging on each of 10 subbands are arranged to a 20-dimensional texture feature vector. RBF kernel is adopted in SVM classifiers. The kernel width is learnt by cross-validation approach. The first 10 images of each category, 500 images in total, are selected as query images to probe the retrieval performance. In each round, only the top 10 images are labeled and 10 most confident images selected from contention set are labeled. All accuracy in the following text is the averaging accuracy of all test images. Figure 3 is the accuracy vs. scope curve of the three algorithms after the third and fifth rounds of relevance feedback, respectively. From the comparison results we can see that the proposed algorithm (Co-SVM) is better than SVM Active (Active SVM) and the traditional relevance feedback method (SVM). Furthermore, we investigate the accuracy of the three algorithms within top 10 to top 100, and with 5 rounds feedback. The results of top-30 and top-50 are depicted in figure 4. The detailed results are summarized in table 1. The results reported in table 1 show that Co-SVM achieves higher performance than that of SVM Active and the traditional relevance feedback algorithm. Figure 3. The accuracy of image retrieval with 3 feedbacks (left) and 5 feedbacks (right)
Figure 4. The accuracy of image retrieval averaging on top-30 (left) and top-50 (right)
Active SVM
CoSVM
Table 1. Comparison among the proposed algorithm (Co-SVM), the traditional feedback algorithm (SVM) and Tong and Chang's SVMActive(Active SVM)
DiSCUSSIONS Active learning has been well-studied in data mining. The representative algorithms include Co-training and Co-Testing both created from multi-view viewpoint. Co-training algorithm adopts cooperative learning strategy. Two classifiers are separately learned on two different views of the labeled data. Then the training set is enlarged for each classifier by taking in the most confident unlabelled samples from another classifier. Co-training algorithm requires that the multiple views of data are compatible and redundant. In fact, we have attempted to augment the performance of both color and texture classifiers by adding the most confident unlabelled samples to complement the labeled set, but the results were worse. Considering the condition of co-training, it is not surprise to find that color attribute and texture attribute are not compatible but uncorrelated for a color image. In contrast, co-testing requires that the views should be sufficient and uncorrelated which makes the classifiers more independent for classification. The deficiency of co-testing is that it does not increase in itself any information except what given by users through labeling. Like the previous works, Co-SVM algorithm selects the unlabeled samples in batch way. Intuitively, the uncertainty should be propagated to neighbors once an unlabeled sample is labeled. Therefore, uncertainty of each unlabeled sample should be dynamically updated during the feedback process. However, the update will brings mass increase in computational cost. How to balance the uncertainty update and computational cost is what we must consider. The further investigations are conducting and partial of recent results have been reported in (Zhang, Cheng, Lu, & Ma, 2007, 2008).
Con In this chapter, we proposed a novel active learning algorithm for selective sampling in relevance feedback, Co-SVM algorithm. In order to obtain the maximal improvement of performance, the relevance feedback is divided into two stages. At the first stage, we rank the unlabeled images according to their similarity to the query and let user to label the top images like the common relevance feedback algorithms. In order to reduce the labeling requirement, a set of additional informative samples are selected by Co-SVM to label at the second stage. The experimental results show that the Co-SVM achieves obvious improvement compared with SVM Active and the traditional relevance feedback algorithm without active learning.
REFERENCES Angluin, D. (1988). Queries and concept learning. Machine Learning, 2, (4), 319-342. Blum, A., & Mitchell, T. (1998). Combining labeled and unlabeled data with co-training. In Proceedings of the 11th Annual Conference on Computational Learning Theory, (pp. 92-100). Cheng, J., & Wang, K.Q. (2007). Active learning for image retrieval with Co-SVM. In Pattern Recognition, 40(1), 330-334.
Collins, M., & Singer, Y. (1999). Unsupervised models for named entity classification. In Proceedings of the Empirical NLP and Very Large Corpora Conference (pp. 100-110). Datta, R., Li, J., & Wang, J.Z. (2005). Content-based image retrieval – approaches and trends of the new age. In Proceedings of the 7th ACM SIGMM workshop on Multimedia Information Retrieval (pp. 253-262). Dasgupta, S., Littman, M., & McAllester, D. (2001). PAC generalization bounds for co-training. In Proceedings of 14th Neural Information Processing Systems (pp. 375-382). Freund, Y., Seung, H.S., Shamir, E., & Tishby, N. (1997). Selective sampling using the query by committee algorithm. Machine Learning, 28(2-3), 133-168. Hong, P., Tian, Q., & Huang, T.S. (2000). Incorporate support vector machines to content-based image retrieval with relevance feedback. In Proceedings of IEEE International Conference on Image Processing (pp. 750-753). Huang, T.S., & Zhou, X.S. (2001). Image retrieval with relevance feedback: from heuristic weight adjustment to optimal learning methods. In Proceedings of International Conference on Image Processing (pp. 2-5). Joachims, T. (1999). Transductive inference for text classification using support vector machines. In Proceedings of the 16th International Conference on Machine Learning (pp. 200-209). Mitchell, T. (1982). Generalization as search. Artificial Intelligence, 28, 203-226. Muslea, I., Minton, S., & Knoblock, C.A. (2000). Selective sampling with redundant views. In Proceedings of the 17th National Conference on Artificial Intelligence (pp. 621-626). Muslea, I., Minton, S., & Knoblock, C.A. (2006). Active learning with multiple views. Journal of Artificial Intelligence Research, 27, 203-233. Nigam, K., & Ghani, R. (2000). Analysis the effectiveness and applicability of co-training. In Proceedings of Information and Knowledge Management, (pp. 86-93). Picard, R.W., Minka, T.P., & Szummer, M. (1996). Modeling user subjectivity in image libraries. In Proceedings of International Conference on Image Processing, (pp. 777-780). Rocchio Jr., J.J. (1971). Relevance feedback in information retrieval. In G. Salton (Eds.), The SMART Retrieval System: Experiments in Automatic Document Processing, (pp. 313-323). New Jersey: Prentice-Hall. Rui, Y., Huang, T. S., & Chang, S. F. (1999). Image retrieval: current techniques, promising directions and open issues. Journal of Visual Communication and Image Representation, 10(1), 39-62. Rui, Y., Huang, T.S., Ortega, M., & Mehrotra, S. (1998). Relevance feedback: a power tool in interactive content-based image retrieval. IEEE Transactions on Circuits and Systems for Video Technology, 8(5), 664-655. Schohn, G., & Cohn, D. (2000). Less is more: active learning with support vector machines. Proceedings of the 7th International Conference on Machine Learning (pp. 839-846).
Smeulders, A.W.M., Worring, M., Santini, S., Gupta, A., & Jain, R. (2000). Content-based image retrieval at the end of the early years. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(12), 1349-1380. Tieu, K., & Viola, P. (2000). Boosting image retrieval. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (pp. 228-235). Tong, S., & Chang, E. (2001). Support vector machine active learning for image retrieval. In Proceedings of the 9th ACM International Conference on Multimedia, (pp. 107-118). Tong, S., & Koller, D. (2000). Support vector machine active learning with applications to text classification. In Proceedings of the 7th International Conference on Machine Learning, (pp. 999-1006). Vapnik, V. (1998). Statistical learning theory. New York: Wiley. Vasconcelos, N., & Lippman, A. (2000). Bayesian relevance feedback for content-based access of image and video libraries. In Proceedings of IEEE Workshop on Content-based Access of Image and Video Libraries (pp. 63-67). Wang, L., Chan, K.L., & Zhang, Z. (2003). Bootstrapping SVM active learning by incorporating unlabelled images for image retrieval. In Proceedings of IEEE Computer Vision and Pattern Recognition, (pp. 629-634). Zhang, T., & Oles, F. (2000). A probability analysis on the value of unlabeled data for classification problems. In Proceedings of the 17th International Conference on Machine Learning, (pp. 1191-1198). Zhang, X.Y., Cheng, J., Lu, H.Q., & Ma, S.D. (2007). Weighted Co-SVM for image retrieval with MVB strategy. In Proceedings of IEEE International Conference on Image Processing, (pp. 517-520). Zhang, X.Y., Cheng, J., Lu, H.Q., & Ma, S.D. (2008). Selective sampling based on dynamic certainty propagation for image retrieval. In Proceedings of the 14th International Multimedia Modeling Conference, (pp. 425-435). Zhou, X.S., & Huang, T.S. (2003). Relevance feedback in image retrieval: A comprehensive review. Multimedia Systems, 8(6), 536-544. Zhou, Z.H., & Li, M. (2005). Semi-supervised regression with Co-training. In Proceedings of the International Joint Conference on Artificial Intelligence (pp. 908-913). Zhu, X. (2006). Semi-supervised learning literature survey. Wisconsin University, http://pages.cs.wisc. edu/~jerryzhu/pub.
Visual Data Mining Based on Partial Similarity Concepts Juliusz L. Kulikowski Polish Academy of Sciences, Poland
ABSTRACT Visual data mining is a procedure aimed at a selection from a document's repository subsets of documents presenting certain classes of objects; the last may be characterized as classes of objects' similarity or, more generally, as classes of objects satisfying certain relationships. In this chapter attention will be focused on selection of visual documents representing objects belonging to similarity classes.
InTRODUCTION In numerous scientific research, medical examination, technological applications, education, artistic activity, and other areas visual data play a substantial, sometimes irreplaceable role. Intensive progress in image acquisition, recording, enhancement and storage technology in the last decades caused a dramatic increase in total volume of visual documents stored in archives and repositories of various types over the world. However, because of legal or organizational difficulties and low effectiveness of available remote data access and visual information retrieval methods only a small part of useful information from visual documents can be mined and effectively used in new investigations, decision making and/or in application tasks solution. This is why the interest to new methods of data mining from visual documents is still actual. A visual document is here considered as an electronic digital file consisting of a textual and a graphical part, the last being, in fact, a digital representation of a graph or an image. The textual part of the document may contain some formal characteristics of the document as a whole (numerical identifier, emission date, source, accessibility conditions, etc.) as well as of its graphical part: image contents char-
Visual Data Mining Based on Partial Similarity Concepts
acteristics (string of key words, a semantic classification code, etc.), code of image modality, technical parameters, quality characteristics, and so forth. The graphical part of a visual document is assumed to present graphs or images in one of admissible standard forms that makes it possible to be displayed and visualized in order to be manually or automatically processed. Image processing means here image selection, enhancement, compression/decompression, transmission, visualization as well as any type of image analysis: functional transformation, parameters extraction, recognition, contents' interpretation, application value assessment, and so forth. Visual data mining is a procedure aimed at a selection from a documents' repository subsets of documents presenting certain classes of objects; the last may be characterized as classes of objects' similarity or, more generally, as classes of objects satisfying certain relationships. In this chapter attention will be focused on selection of visual documents representing objects belonging to similarity classes.
GENERAL CONCEPT OF similari Observing the surrounding world as an extremely sophisticated variety of objects, relationships between them and processes we try to put order on it for making it more comprehensible. One of ways to reach this is establishment of similarities between objects or processes which make it possible to apply some statements about individual objets or processes to the members of their similarity classes. This way of action is not only a human discovery. A wild animal first-time testing an apple and finding it good fixes in its mind a pattern of similar size, form, color and scent objects. In its later experiences the pattern may be corrected by exclusion the small size and dark-green color objects which finally leads to a fixation in the mind a pattern (a concept) of "good for being eaten" objects. Establishment of similarity concepts and detection of similarities in individual cases in many living beings is thus a capability of fundamental importance. Non less important seems explaining what, in general, does the similarity mean. Strong similarity. In the forthcoming, the notion object will be assigned to any formal representative (geometrical point, vector, string of symbols, etc.) of a real or abstract being, characterizing its features and identifying it among other beings. Similarity is a property of a class C of objects that in the simplest case can be described by a formal similarity relation [Rasiowa H., Sikorski R. ,1968]: 10 reciprocal (each object a is similar to itself), 20 symmetrical (if object a is similar to object b then b is similar to a), and 30 transitive (if a is similar to b and b is similar to c then a is similar to c). For the reasons that will be explained below, the above-defined similarity can be called a strong similarity concept. Similarity of triangles is a well known example of similarity in the above-given sense: two triangles on an Euclidean plane are similar if angular measures of the pairs of respective angles are equal. This property can easily be extended on any set of triangles. Similarity of polygons on a plane can also be based on the concept of pair-wise equality of angular measures of the pairs of respective angles. Strong similarity can be based on various similarity criteria assigning different sense of similarity to the objects under consideration. For example, it is possible to assume that in the set C of triangles "similar" are the triangles of the same surface measure, of the same sum of lengths of edges, etc.
Figure 1. Similarity classes of a set of printed characters
Figure 2. Example of non-transitiveness of similarity
It can be shown that if the above-given similarity relation is described in a class C of objects then C can be divided into a family FC of mutually disjoint subclasses C(1), C(2),…, etc., called similarity classes, such that any two objects belonging to the same similarity class are similar in the sense of the relation and no two objects belonging to different similarity classes do so. In the above-mentioned case of similarity of triangles each similarity class consists of the triangles described by the same triple of fixed angular measures (admitting their permutations).
E Let us consider a set of printed characters shown in Fig. 1. The characters form three subsets constituting similarity classes established by assigned to them phonetic values, independent of their typeface. Weak similarity. The above-presented concept of similarity, as it has been shown above, suits well to establishment of similarity between well defined, standardized objects. However, in the case of visual objects analysis such situation occurs rather exceptionally. A difficulty lies in the similarity transitiveness condition which in practice is not always satisfied. For example, if D (a daughter) is in a certain sense similar to F (her father) then also, in the same sense, F is similar to D. Moreover, D may be also similar to M (her mother). However, it does not imply that F is similar to M.
Example 2. A sequence of characters shown in Fig. 2 has been ordered so that each character is approximately similar to its closest neighbors in the row. However, it is evident that similarity does not hold between the extreme characters, A and B. As a consequence, the notion of similarity classes has lost an exact sense in this case. For image processing and pattern recognition purposes one should thus distinguish between the above-defined, strong-sense similarity, and a weak sense similarity being a sort of a neighborhood, reciprocal and symmetrical but not transitive relation. More exactly speaking, in the weak similarity
case transitiveness holds in a limited degree that can be characterized by a notion of similarity measure [Kulikowski J.L., 2002]. Similarity measure. Let C be a class of objects and let a, b, c be some of its members. Then it can be defined a function: σ: C × C → [0,…,1]
assigning to any pair of objects belonging to C a number from the (continuous or discrete) closed interval [0,…,1] so that the conditions:
I. σ(a,a) ≡ 1,. II. σ(a,b) ≡ σ(b,a III.
σ(a,b) ⋅ σ(b,c) ≤ σ(a,c)..
are satisfied. The I and II conditions correspond, respectively, to the reciprocity and symmetry of similarity. The III condition reminds the well known "triangle inequality" of a distance measure. If C is a metric space (a space in which a notion of distance measure d(a,b) between its elements has been introduced) then a distance-type similarity measure σ(a,b) between the objects on the basis of d(a,b) can be defined as: σ(a,b) = exp[ − h⋅ d(a,b)]
where h is a real positive scaling coefficient. Then, taking into account that according to the "triangle inequality": d(a,c) ≤ d(a,b) + d(b,c) for any a, b, c ∈ C, the property III from (2) follows directly. It thus establishes lower bounds for similarity measures of objects linked by chains of other objects whose similarity measures pair-wise have been assessed. Therefore, the III condition plays a role of a weakened transitivity condition. It can easily be proven that in the above-given case σ(a,a) ≡ 1 and σ(a,b) = 0 when d(a,b) = ∞. The distance-type similarity measure is widely used in the k-nearest-neighbors (kNN) pattern recognition algorithms [Kulikowski J.L., 2002]. The conditions I, II and III hold also in other types of similarity measures, two of them being of particular interest in image processing. Let a, b, … be the elements of a multi-dimensional linear vector space. An angular measure of the angle between the vectors a and b will be denoted by ang(a,b). Then, an angular-type similarity measure σ(a,b) can be defined as σ(a,b) = 1 − sin[ang(a,b)].
Let us remark that in certain applications (pattern recognition, signal detection, etc.) the function cos[ang(a,b)] is often used as a correlation measure between a and b., However, this function, satisfying the (2).I and II, does not satisfy the (2).III condition. The right-hand side of (4) can be calculated if an inner product (a,b) of the vectors is known. If the vectors have the form: a = [a1, a2,…, ai,…], b = [b1, b2,…, bi,…], a1,…, ai,…,b1,…, bi… being some their real components, then their inner product is given by the formula [3]:
(a,b) =Σ ai⋅bi (i)
(the sum being taken over all components of the vectors). In particular, the norm |a| of the given vector being used as its length's measure is given by the formula: | a |= (a, a )
The angular similarity measure can be thus calculated as: ( a, b) = 1 -
[(a, b)]2 | a |2 ⋅ | b |2
The formula is based on the assumption that vectors a, b have finite non-zero norms. It can be proven that for the angular similarity concept σ(a,b) = 1 when a = γ⋅b, where γ is a real coefficient, while σ(a,b) = 0 if a is perpendicular to b. Independence of the length of vectors is another property of the angular similarity measure, the property being important in certain types of applications.
Example 3 Let us denote by C a 3-dimensional vector space of colors coded in RGB system. Using an angular-type similarity measure means in this case that σ(a,b) = 1 corresponds to a pair of spots coded, respectively, by a and b, being of the same color despite a difference between their brightness levels. Let us suppose that three spots, whose colors can be roughly described as light green, vivid green and sea-green have been coded, respectively, in the RGB system by the vectors: a = [145, 247, 172], b = [14, 120, 63], c = [40, 120, 118]. We shall calculate the angular-type similarity measure of the pairs of colors. For this purpose it is necessary to calculate the norms and inner products of the vectors, using the formulae (6) and (5). For example: |a|2 = 1452 + 2472 + 1722 = 111 618, |a| = 334.0928, |b|2 = 142 + 1202 + 632 = 18 565, |b| = 136.2534, (a,b) = 145⋅14 + 247⋅120 + 172⋅63 = 42 506. In similar way we obtain: |c|2 = 29 924, |c| = 172.9855, (a,c) = 55 736, (b,c) = 22 394. Finally, using formula (7) we obtain:
( a, b) = 1 - 1 -
425062 = 0.6421 111618 ⋅ 18565
In similar way it can be obtained: σ(a,c) = 0.7345, σ(b,c) = 0.6881. Let us remark that the similarity measures will be different if another color coding system is used. Similarity measure based on qualitative features of objects (called below a logical-type similarity) is another case of the weak similarity concept. Let for the given set C of objects it has been defined a set of K logical tests given in a functional form: τk: C × C → [0, 1] for k = 1, 2,…, K,
i.e. such that to any pair of objects a symbolic value 0 ( false) or 1(true) is assigned. Assigning 1 means that the given pair of objects satisfies a condition of coincidence of a qualitative property represented by the corresponding tests. For example, τk (a, b) = 1 may mean that two mammograms, a and b, have been acquired by the same type of X-ray devices, two fingerprints belong to the same person, two aerial photos have been taken in the same season, etc. For a given set of K logical tests to each pair of objects a, b, it can be thus assigned a binary vector v(a,b) consisting of components representing the coincidence of selected qualitative properties of the objects. So defined vectors can be taken as a basis of a logical similarity measure. Formally, the measure will be given by the formula: ψ : [0, 1]K →[0,…,1]
where ψ is a function assigning to any vector v a real number from the interval [0,…,1], and such that: 1. 2.
ψ(0) = 0, ψ(1) = 1, 0 and 1 denoting, correspondingly, a K-component vector consisting of 0s or 1s only; ψ is a monotonically non-decreasing function, i.e. for any K-component binary vectors p, q, such that q ≠ 0, it is ψ(p ⊕ q) ≥ ψ(p).
The operation ⊕ is interpreted above as maximum, i.e. 0 ⊕ 0 = 0, 1⊕1 = 1⊕ 0 = 0 ⊕ 1 = 1. Condition 1 seems evident; condition 2 means that the logical similarity measure value should not be decreased by any additional positive results of logical tests. A function ψ satisfying the above-given conditions can be chosen in many ways. Two particular situations arise when: a/ ψ(v) = 1 if v = 1 and ψ(v) = 0 otherwise, and b/ ψ(v) = 0 if v = 0 and ψ(v) = 1 otherwise. Situation a/ means that two objects are determined similar when all logical tests are satisfied by their features. On the contrary, situation b/ means that one positive value of tests is sufficient the objects to be recognized similar. Among the abovementioned extremities a variety of other solutions is possible. One of them can be based on the concept of weighed sum of tests. Let for the given set of tests it will be arbitrarily chosen a K-component vector w of normalized non-negative weight coefficients:
∑w =1
= 1,
where wκ denotes a relative weight assigned to the importance of the tests τκ , κ = 1,2,…,K, in total similarity assessment. Then one can put:
[ (a, b)] =
1 K ∑ w ⋅ (a, b) K +1
The following example should illustrate the above-described concept of logical-type similarity measure.
Eample 4. Let us assume that it is given a set Ω of microscopic images representing liver biopsy histologic specimens. To each image additional description is attached containing the following data: 1/ image identifying code, 2/ donor's personal identifying code, 3/ donor's age, 4/ date of image acquisition, 5/ laboratory's identifying code, 6/ type of microscope, 7/ type of specimen staining. For certain investigations it is necessary to select a subset of images satisfying the following conditions: a/ donors of the corresponding specimens are of an age kept within a fixed time-interval, b/ date of image acquisition is limited to several fixed years, c/ images have been acquired in a given group of laboratories and using several types of microscopes, d/ specimens have been prepared using a fixed type of staining. In addition, it is desired that conditions a/ and b/ are strongly satisfied, while c/ and d/ should be taken into account with fixed weights corresponding to their relative importance level. For this purpose logical test-functions for the features 3/ - 8/ should be established, feature 1/ being not important for comparing the images and feature 2/ - not substantial for image selection. The tests may be then chosen as follows: t1 =
t2 =
t3=
1 if both donors' age belong to the given time-interval, 0, otherwise; 1.if the dates of acquisition of both images belong to the given time-interval, 0,.otherwise; 1.if images have been examined in the laboratories of a given group, 0,.otherwise; 1.if image examinations have been performed using microscopes of a given group, 0,.otherwise; 1.if images have been examined after the given type of specimen staining, 0,.otherwise;
Finally, the logical similarity measure will be defined as follows: (a, b) = 1(a, b) ⋅ 2(a, b) ⋅
1 5 ∑ ( a, b) 3 =3
So-defined similarity measure means that images a ,b are recognized similar only if age of specimens' donors and image acquisition time are kept in strongly fixed intervals while type of laboratory, type of microscope and type of specimen staining are taken into consideration with weights equal 1/3. Multi-aspect similarity measures. In the above-given example the following important property of similarity measures has been used:
Property 1. If σ1, σ2, …, σn are similarity measures described on a given set C of objects, then their product: σ = σ1 ⋅ σ2 ⋅ …⋅ σn
also satisfies the general similarity measure conditions. The proof of this property is very simple and will be omitted. The Property 1 makes it possible to construct a large variety of multi-aspect similarity measures as products of the single-aspect ones. In particular, combinations of distance-, angular-, and logical-based similarity measures are possible. This possibility can be useful in visual data mining, where logical similarity measures suit well to formal characterization of visual documents while other similarity aspects, directly concerning the graphical documents' part, can be evaluated and compared using any types of similarity measures. The next property extends this possibility.
Property 2. If σ is a similarity measure satisfying the conditions (2).I, II, III, then σγ, where γ, γ > 0, is also a similarity measure such that for any pair of objects a, b it is σγ(a,b) ≤ σ (a,b) if γ ≥ 1 and σγ(a,b) < σ (a,b) if 0 < γ < 1. This property follows from the fact that: 1/ xγ is a non-negative function such that for any γ > 0 it is 0γ = 0, 1γ = 1, 2/ for 0 ≤ x ≤ 1 it is xγ ≤ x if γ ≥ 1 and xγ ≥ x if γ ≤ 1 and 3/ from (2).III it follows that for any objects a, b it is log σ(a,b) + log σ(b,c) ≤ log σ(a,c); therefore: γ⋅log σ(a,b) + γ⋅log σ(b,c) ≤ γ⋅log σ(a,c) what leads to the conclusion σγ (a,b)⋅σγ (b,c) ≤ σγ (a,c). Property 2 makes it possible to relatively support some terms in the expression (12) by using them in power γ < 1 or to suppress them by using in power γ > 1. Therefore, a more general expression for multi-aspect similarity measure is: σ = (σ1) γ1⋅ (σ2) γ2⋅ …⋅ (σn) γn
where γ1, γ2,..,γn are some strengthening coefficients. Weak similarity classes. Similarity measure makes possible to extend the concept of objects' similarity classes. However, unlike the strong similarity, in the weak similarity case similarity classes,
as it will be shown below, cannot be determined in an unique way. Let us assume that C is a countable set of objects on which a similarity measure σ has been defined. Let us take into consideration the set C×C (a Cartesian product) of all possible pairs (a,b) of objects of C and a set S of the values of similarity measures σ(a,b) of the pairs. Let σ1, σ1 ≤ 1, be the maximum value in S. One can select the pairs of objects whose similarity measure is σ1 . Let us denote by C(1), where C(1) ⊆ C, a subclass of objects belonging to such pairs; C(1) will be called the 1st similarity level of the objects. There may exist in C(1) subsets of objects similar each to each other one with the maximum similarity measure σ1. The largest (i.e. not extendable) subsets having this property will be called 1st-level similarity classes. So, the 1st similarity level will be decomposed into a certain number of 1st-level similarity classes; however, it should be remarked that some of the 1st-level similarity classes may consist of single objects. Then, for the rest of objects forming a subset C \ C(1) the above-described procedure can be repeated: if σ2 is the maximum similarity measure among its members then a 2nd similarity level C(2) ⊆ C \ C(1) can be selected and 2nd-level similarity classes can be defined in the above-described manner. The procedure should be continued up to the moment when all objects of C are included into some similarity classes. Each so-defined similarity class consists of objects whose similarity to other members of the same similarity class is strongly fixed, but not obviously equal 1. Therefore, similarity measure can also be interpreted as inner similarity compactness measure of the classes of various levels. The above-described procedure of selection of similarity levels was based on the assumption that a maximum number in the set S of values of similarity measures can be found. However, this assumption can be not true if the class C is countable and the similarity measure is continuous. In such case the scale of similarity measure values should be discretized (divided into a finite number of sub-intervals) and then the corresponding number of similarity levels can be delimited. The partition of the similarity levels C(1), C(2), etc. into weak similarity classes is not unique, as it will be shown by the following example.
Example 5. Let us assume that at i-th level of the above-described procedure of partition of the objects into similarity classes a subclass C(i), C(i) ⊆ C, of objects corresponding to the similarity measure σi has been selected. In order to find similarity classes on this level C(i) will be represented by a graph whose nodes have been assigned to the objects and edges connect pairs of nodes corresponding to the pairs of objects whose similarity measure is exactly equal to σi. An example of a graph of this type is shown in Fig. 3 Then we are looking for maximum subsets of nodes in the graph such that each pair of nodes is connected within the subset by an edge. This problem is well known in the graph theory as a problem of finding cliques in the graph; several algorithms of solution of the problem have been described in the literature [5,6]. Fig. 3 A, B, C show that in general there may be several different partitions of a graph into cliques (the cliques are bordered by plotted lines); however, the number of cliques and their size (number of nodes) are strongly fixed. In our case each clique represents a similarity class of objects, the inner similarity compactness level in all similarity classes generated by the given graph being the same. It thus can be concluded that there are many possibilities to define a similarity measure satisfying the general conditions and adjust it to a given application problem. Weak similarity can be considered as a reciprocal and symmetrical fuzzy relation imposed on a set of objects, the fuzziness being determined by a similarity measure satisfying the conditions (2). I, II and III. It leads to a partition (in general, in
Figure 3. Different partitions of a subclass of objects into weak similarity subclasses
Figure 4. Partition of a class of objects into weak similarity subclasses
several possible ways) of the given class C of objects into weak similarity subclasses which can be arranged in a hierarchical structure as shown in Fig. 4. Subclasses belonging to a fixed level of the hierarchy are characterized by a fixed inner similarity compactness. The higher is the level index (the lower is the level position on the scheme), the lower is the inner similarity compactness of the subclasses on this level. Similarity of any two objects belonging to similarity subclasses on two consecutive levels is lower than the inner similarity compactness on the higher level. Such partition of the class of objects into subclasses according to their descending similarity to other objects plays an important role in image retrieval and selection, as it will be shown next.
Detection ooai iag It is well known that a natural language has a composite, multi-level structure. On the lowest level of a speech language organization phonemes can be distinguished. Morphemes, words, phrase structures, sentences, statements, etc. are the elements of higher levels of the speech language structure. Similar, multi-level composition holds also in images, including those stored in the form of visual documents in archives, data bases, etc. On the lowest level separate pixels or their simple combinations can be distinguished; basic geometrical details (primitives) or morphological forms on the higher level can be detected; more sophisticated structures as combinations of some simpler ones on the next levels can be observed, etc. On the top level a total scene as a composition of lower-level objects can be recognized. Hierarchical structure of visual data makes the problem of visual data management a very difficult one. The above-described concepts of strong or weak similarity suit well to image analysis if they are related to image features on a strongly defined composition level. This can be illustrated by the graphical structures shown in Fig. 5. There are no differences between the structures on the 1st (pixel) and 2nd (geometrical primitives in the form of short line segments) composition levels. On the 3rd level two types of geometrical details in the form of upright standing and reversed small triangles can be recognized. Similarly, on the 4th level there are two upright standing and one reversed triangle. The patterns a/ and b/ are similar on the 3rd composition level and dissimilar on the 4th level. On the other hand, the patterns a/ and c/ are similar on the 4th level and dissimilar on the 3rd one. Patterns b/ and c/ represent an interesting case of a cross-similarity.
Figure 5. Examples of similarity on various image composition levels
Figure 6. Example of partial similarity of two images': the centers of similarity in image a/ and b/ are marked by rectangles
Another example is shown in Fig. 6a,b representing two landscape photos. On the top image composition level no similarity between the images exists. However, on the lower level a certain sort of similarity between some architectural and natural objects can be observed. On the basic levels a lot of similar local details also can be detected. The examples show that it may happen, in general, that two or more images totally dissimilar on a higher composition level still contain similar details on some lower levels. Similarity on the basic (1st or 2nd) levels is rather frequently observed; however, usually it is of no practical interest. Different situation may arise on the middle image composition levels where detection of some details may be of great practical importance for the user. This, for example, may concern detection of similarity of specific details hidden in large collections of astronomic, biological, medical, criminological etc. images. For this purpose a concept of partial similarity of images may be useful. Two (or more) images will be called partially similar if on a given composition level in both (or more) images there can be detected objects satisfying a given criterion of similarity. Such objects will be called centers of similarity. Two examples of such centers have been marked in Fig. 6 a/ and b/ by rectangles. A subset Ξ(j) ⊆ Ξ consisting of all partially similar images taken from a set Ξ and such that their partial similarity is based on a fixed set of centers of similarity, will be called a class of partial similarity of the images. It is evident that the class of partial similarity is also strongly connected with a given criterion of similarity. Finding the classes of partial similarity in a given set Ξ will be called, generally, a partial similarity-based retrieval problem. Two kinds of such problems can be distinguished: 1. 2.
When one or several strongly fixed reference objects are indicated and the problem consists in finding the classes of partial similarity based on the given set of centers of similarity; When, instead of several reference objects, only a strong similarity class of objects is indicated as the one whose members should be taken into consideration as possible reference objects; the task consists in finding in the set of images all classes of partial similarity based on the centers of similarity belonging to the given similarity class.
The first case corresponds, e.g., to a situation when in a set of fingerprints the ones containing a strongly fixed configuration of fingerprint minutes are searched. For this purpose the set is looked through in order to select the regions of interest (ROI) where the fingerprint minutes can be found, then the minutes are recognized, their similarity to those of the reference fingerprints is assessed and their configuration is compared with this of the reference fingerprint. The class of partial similarity of images in this case is assumed to correspond, on a certain probability level, to the same fingerprint donor. The second case arises when, e.g., in a set of photos presenting groups of persons the subsets of photos on which the same (however, individually not indicated) person(s) can be recognized. The class of partial similarity of images consists in this case of photos on which at least one, primarily not indicated, person is visible on at least two photos. Detection of images in the second case is more difficult than in the first one. First photo for the analysis can be randomly taken from the given set. Then, the ROIs corresponding to the persons in the photo should be selected. Each well visible person in the ROIs is taken into consideration as a reference object for finding similar to it objects in the ROIs selected in other photos. When finding the members of the classes of similarity based on the reference objects taken from the first photo is finished, additional reference objects from the second, third, etc. photos can be taken in order to continue the above-described procedure. It is evident that a solution of the
second-type of partial similarity-based retrieval problem is much more time-consuming than the first type one. However, effective solution of the first type retrieval problems can be taken as a basis of a solution of the second-type ones. For a given set of visual documents a partial similarity measure s(Vi, Vj) of any pair Vi, Vj of images belonging to some documents can be defined. For this purpose, first, a class C of objects (e.g. fingerprint minutes, human faces, architectural details, tumors in radiological images, etc.), which should be taken into consideration as a basis of partial similarity of images assessment, should be specified. Let Gi = [g1i,…,gmi], Gj = [g1j,…,gnj] be local visual objects of the given class C primarily selected for analysis in Vi and Vj, respectively. Then all pairs of the type [gµi, gνj] for µ = 1,2,…,m, ν = 1,2,…,n, should be taken into account. Each pair contributes to the partial similarity measure of s(Vi,Vj) if the similarity σ(gµi, gνj) of the objects satisfies the following logical condition (objects' concordance test): t(g i, g j) =
1 if 0 ≤ .0 ≤ .(g i, g j), 0 otherwise,
where σ 0 is a fixed threshold level, 0 ≤ σ 0 ≤ 1. A partial similarity function then can be defined as:
∑ ∑ s (V ,V ) = m
n =1
t ( gi , g j )
Using this function it is possible, for example, to evaluate partial similarity of mammograms according to the number of pairs of calcifications of similar size and form detected in the analyzed images. However, assessment of partial similarity of images is a time-consuming operation depending on the number of different pairs of the analyzed objects. This dependence becomes linear in the individual reference objects based similarity task solution case, when the set of reference objects is strongly limited. The time consumption of partial similarity of images assessment can be reduced if a significant part of non-perspective objects within a procedure based on simplified selection criteria is rejected. For this purpose it can be established a sequence of objects' multi-aspect similarity measures: σ(1)(a, b), σ(2)(a, b), … , σ(k)(a, b). Let Σ(κ), κ = 1,2,…,k, denote the subset of single-aspect similarity measures on which σ(κ)(a, b) has been based. If the Σ(κ) s are chosen so that the inclusions Σ(1) ⊂ Σ(2) ⊂ … ⊂ Σ(k) hold then the abovegiven sequence represents step-by step strengthened similarity criteria. Each next similarity formula is given there as a product of the preceding formula and some additional partial similarity measures. For a given reference object a = g* and a threshold level σ0(1) using σ(1)(g*,b) and based on them concordance test a subset C(1), C(1) ⊆ C, of objects can be selected. The elements of C(1) are the first-step candidates for a class of objets similar to g*. Then, taking into account the similarity measure σ(2)(g*, b) and a threshold level σ0(2) and applying them to the elements of C(1) a subset C(2), C(2) ⊆ C(1), can be selected, etc. Repetition of such procedure leads to a descending sequence of subsets C ⊇ C(1) ⊇ C(2) ⊇ … ⊇ C(k). Finally, C(k) consists of the most similar to g* objects. Images containing the elements of C(k) are the searched ones, forming a class of partial similarity defined by g*. The calculation-time reduction is achieved due to the fact that the initial selection steps are based on simplified concordance test, while the next selections are applied to reduced subsets of visual objects.
Relative Partial Sii In the most general partial similarity detection cases non-numerical objects' similarity assessment is admissible. Instead of the above-described concordance tests t(gµi, gνj) whose logical values are expressed in a binary scale (0 = "false", 1 = "true") the following statements: h(a, b; σ ): "a is similar to b in the sense of σ ", for any a ,b ,… ∈ C then can be taken into consideration and their logical values in a more general sense can be evaluated. This leads to the concept of partial similarity assessment based on relative (topological) logic [Hempel C.G. 1937, Kulikowski J.L. 1986, Vessel Ch.A. 1970]. In such case, instead of using any a priori fixed logical scale it is assumed that for any two statements, e.g. h(a,b) and h(c,e), (the argument σ being here neglected for the sake of simplicity) the following logical relationships are admitted: 1. 2. 3. 4.
h(a,b)  h(c,e), i.e. "h(a,b) is not more logically valuable than h(c,e)"; h(a,b)  h(c,e), i.e. "h(a,b) is not less logically valuable than h(c,e)"; h(a,b)  h(c,e) AND h(a,b)  h(c,e), i.e. "h(a,b) AND h(c,e) are equally logically valuable", which also can be denoted as h(a,b) ≈ h(c,e); NEITHER h(a,b)  h(c,e), NOR h(a,b)  h(c,e), i.e. "h(a,b) AND h(c,e) are logically incomparable", which also can be denoted by h(a,b) ? h(c,e).
In particular, it can be assumed that: 1st h(a ,b; σ)  h(c ,e; σ) if for a given similarity measure σ(a, b) it is σ(a ,b) ≤ σ(c, e), 2nd h(a ,b; σ')  h(a ,b; σ") if σ'(a, b) = σ"(a ,b) and σ' and σ" are such that Σ ' ⊂ Σ ". If a = g* is a reference object for which similar objects in a set of images are searched then for any other objects gµ , gν a situation h(g*, gµ) ≈ h(g*, gν ) means that the statements about the similarity of gµ to g* as well as of gν to g* are logically equivalent; therefore, gµ and gν belong to the same class of similarity to g*. However, the above-mentioned relationships  ,  , ≈ , ? among the statements can be established, as well, in any less-formalized way (e.g. suggested by human experts). For a given reference object g* and a set C of visual objects a relative similarity class (RSC) H' can be defined as a subset H'⊆ C of all objects b, c, d ,… for which the relationships h(g*, b) ≈ h(g*, c) ≈ h(g*, d) ≈ … etc. hold and no such relationship occurs between two statements, say, h(g*, e) , h(g*, f ) if e and f belong to different RSCs. The last condition implies that if H' and H" are any two disjoint subsets of visual objects selected, at the first step of analysis, as candidates to form two different RSCs, then for any a, b ∈ H' and any c, d ∈ H" simultaneous relationships h(g*, a)  h(g*, c) and h(g*, b)  h(g*, d) for the sake of logical consistency should be detected and excluded. Let H' and H" be, at the
Figure 7. Example of a Hasse diagram
first step, two different subsets of visual objects that by the experts have been indicated as similar to a reference object g* from the point of view of two different similarity criteria. Then they can be taken as the first approximation of the candidates to form two RSCs. For this purpose all pairs of statements [h(g*, am), h(g*, cn)] such that h(g*, am) ∈ H', and h(g*, cn) ∈ H" should be taken into consideration and one of the relationships  ,  or ≈ to each of them by the experts should be assigned. However, if for certain pairs it is impossible to be done, the relationship ? to them should be assigned. Then, a possible logical inconsistency between H' and H" can be corrected according to the following rules: 1st if between the elements of H' and those of H" any of the relationships  ,  or ? prevails then it should be extended on all remaining pairs of the elements of H' and those of H" ; 2nd if between the elements of H' and those of H" the relationship ≈ prevails then it should be extended on all remaining pairs of the elements of H' and of H"; in such situation H' and H" should be merged in a common RSC. According to the established dominating type of logical relationship between the statements in different RSCs H' and H" a similar relationship:  ,  , ≈ or ? between H' and H" can be established. In similar way logical consistency between all initially selected candidates for RSCs can be established. The correction procedure leads thus to semi-ordering of the family F of RSCs: between any two RSCs H' and H" one of the relationships (i.e. H"  H') or H' ? H" may occur. This semi-ordering relation can be represented by a directed contourless graph, called a Hasse diagram G. The nodes of G are assigned to the RSCs and an arc (ordered edge) between any ordered pair [H', H"] of them exists if H'  H" and no other RSC H satisfies both H'  H and H  H". Example of a Hasse diagram illustrating a semiordering of a family of RSCs is shown below. Two RSCs: are separated by a dotted line from the rest of the graph. They satisfy Hλ ? Hη , however, they are such that no other RSC H exists such that even Hλ  H or Hη  H; such RSCs form a subset of maximally logically valuable statements. As a result, the corresponding subset of visual objects Cλ ∪ Cη should be taken into consideration as this of relatively the most similar ones to g*. Visual data mining based on relative partial similarity concepts may be, in general, less effective than this, based on strongly defined similarity criteria. However, in certain application areas partial similarity measures cannot be precisely evaluated. In such case relative similarity is the only possible way to use information given in a suggestion like: "From the facial expression point of view B seems to be more similar to A than C" into a computer-aided decision process.
Con Visual data mining is a tool of extraction useful visual data dispersed in large repositories of visual documents and hidden among other visual data of lower value for the users. This goal can be reached in two ways: by direct indication of a reference pattern (visual object) that should be found in a set of visual documents or by rough indication of a class of objects expected to occur (maybe, not in identical form) in two or more documents. In both cases a general notion of similarity and/or similarity measure and based on them partial similarity can be used to solve the problem. These notions admit, in particular, using various image features to define multi-aspect similarity criteria. An extension of the visual data mining methods on the case of non-numerical similarity criteria is also possible. In particular, when similarity of selected visual objects to the reference objects without clearly defined formal backgrounds are suggested by the experts, selection of the subsets of the visual objects (and, consequently, of containing them partially similar visual documents), satisfying the requirement of similarity to the
given reference object, can be reached due to the relative similarity concepts. The last approach puts a bridge between formalized and traditional (manual) visual data similarity assessment and based on them visual documents retrieval methods.
REFERENCES Hempel, C.G. (1937). A Purely Topological Form of Non-Aristotelian Logic. Journ. Symb. Logic, 2(3). Kulikowski, J.L. (1986). Decision Making in a Modified Version of Topological Logic. Proceedings of the Seminar on "Non-conventional Problems of Optimization". Prace IBS PAN, Part I, 134, Warsaw: IBS PAN, 24-44. Kulikowski, J.L. (2002). From Pattern Recognition to Image Interpretation. Biocybernetics and Biomedical Engineering, 22(2-3), 177-197. Rasiowa, H., & Sikorski, R. (1968). The Mathematics of Metamathematics. Warsaw: PWN. Vessel, Ch.A. (1970). On the Topological Logic (in Russian). Non-Classical Logic. Moscow: Nauka.
Semantic Analysis
Image/Video Semantic Analysis by Semi-Supervised Learning Jinhui Tang National University of Singapore, Singapore Xian-Sheng Hua Microsoft Research Asia, China Meng Wang Microsoft Research Asia, China
ABSTRACT The insufficiency of labeled training samples is a major obstacle in automatic semantic analysis of large scale image/video database. Semi-supervised learning, which attempts to learn from both labeled and unlabeled data, is a promising approach to tackle this problem. As a major family of semi-supervised learning, graph-based methods have attracted more and more recent research. In this chapter, a brief introduction is given on popular semi-supervised learning methods, especially the graph-based methods, as well as their applications in the area of image annotation, video annotation, and image retrieval. It is well known that the pair-wise similarity is an essential factor in graph propagation based semisupervised learning methods. A novel graph-based semi-supervised learning method, named StructureSensitive Anisotropic Manifold Ranking (SSAniMR), is derived from a PDE based anisotropic diffusion framework. Instead of using Euclidean distance only, SSAniMR further takes local structural difference into account to more accurately measure pair-wise similarity. Finally some future directions of using semi-supervised learning to analyze the multimedia content are discussed.
Image/Video Semantic Analysis by Semi-Supervised Learning
INTR Digital image and video collections are growing rapidly in recent years, accompanied with the decreased cost of storage devices, high transmission rates and improved compression techniques. The demand for solutions to manage image/video database is increasing tremendously. It is a common theme to develop automatic analysis techniques for deriving metadata to describe the visual content at semantic level (Smith & Schirling, 2006). Thus, automatically annotating image and video at the semantic concept level has emerged as an important topic in the multimedia research community as it is an elementary step for obtaining these metadata. The concepts of interest include a wide range of categories such as scenes (e.g., urban, sky, mountain.), objects (e.g., airplane, car, face, etc.), events (e.g., explosion-fire, peoplemarching, etc.) and certain named entities (e.g. person, place, etc.) (Naphade, et al., 2006; Snoek, et al., 2006). As manually annotating large image or video archive is labor-intensive and time-consuming, efficient automatic annotation methods are highly desired. To this end, generally statistical models are built from manually pre-labeled samples, and then the labels are automatically assigned to the unlabeled samples using these models. However, this process has a major obstacle: frequently the labeled data is limited so that the distribution of the labeled data typically cannot well represent the distribution of the entire data set (including labeled and unlabeled), which usually leads to inaccurate annotation results. Semi-supervised learning, which attempts to learn from both labeled and unlabeled data, is a promising approach to deal with the above issue. As a major family of semi-supervised learning, graph based method is becoming one of the most active research area in semi-supervised learning community in recent years. Many works on this topic are reported in the literature of machine learning community (Carreira-Perpinan & Zemel, 2005; Seeger, 2001) and some of them have been applied to multimedia semantic analysis. In Section II, we briefly introduce several semi-supervised learning techniques, including self-training, co-training and transductive SVM, and then we focus on graph-based methods (Zhu, 2005a). The graph-based methods define a graph with each vertex corresponding to each sample in the dataset, and the weighted edges reflect the similarity between neighboring samples. The objective of most graph-based methods is estimating a prediction function on the graph. Zhu, Ghahramani, & Lafferty (2003) have made two assumptions for this function: (1) the predicted scores on the unlabeled data should be close to the given labels of the unlabeled data; (2) the function should be smooth on the whole graph. These two assumptions directly lead to the Gaussian random field method, in which the graph combinatorial Laplacian is used as a regularizer. Besides these two assumptions, Zhou, Bousquet, Lal, Weston, & Scholkopf (2003) further made the structural assumption: points on the same structure (typically referred to as a cluster or a manifold) are likely to have the same label. This leads to the local and global consistency method, which uses the normalized Laplacian as the regularizer. Some more sophisticated regularization frameworks are also briefly introduced in this section, e.g., local learning regularization (Wu & Scholkopf, 2007), Tikhonov regularization (Belkin, Matveeva, & Niyogi, 2004) and manifold regularization (Belkin, Niyogi, & Sindhwani, 2006). In Section III, we present some applications of semi-supervised learning in video/image semantic analysis. Obviously a key-point in graph based semi-supervised learning is the pair-wise similarity as it is the basis of the label propagation. However, one basic assumption, structural assumption, which is an essential of graph-based methods, is not embedded into the pair-wise similarity measure. Meanwhile, it is shown that the same results of normal graph-based methods can be deduced from the partial differential equation (PDE) based isotropic diffusion (Sapiro, 2001). Accordingly, a novel graph-based method,
named Structure-Sensitive Anisotropic Manifold Ranking (SSAniMR) (Tang, Hua, Qi, Wang, Mei, & Wu, 2007), is derived from a PDE based anisotropic diffusion framework (Perona & Malik, 1990), by embedding the structural assumption into the similarity measure. Finally, we will discuss some future directions for graph-based semi-supervised learning in the multimedia applications, including multi-label semi-supervised learning (Wu & Jing, 2006), multi-instance semi-supervised learning (Rahmani & Goldman, 2006) and multi-graph learning (Tong, et al., 2005).
Somi-Supervivi In this section, we briefly introduce semi-supervised learning methods, including self-training (Yarowsky, 1995), co-training (Blum & Mitchell 1998), transductive SVM (Joachims, 1999) and graph-based methods (Zhu, 2005a).
Self-Training and Co-Training Self-training is a basic semi-supervised learning method. In self-training, a classifier is initially trained with a small number of labeled samples, and this classifier is used to predict the labels of unlabeled samples. Then, the most confident unlabeled samples (include both positive and negative ones) are added into the training set to retrain the classifier. The procedure repeats until some conditions are satisfied, and this is the so-called self-training method (as the classifier tries to teach itself with its own predictions, this method is also called self-teaching or bootstrapping). In co-training, features are split into two sets which are conditionally independent given the class. Each sub-feature set is assumed to be sufficient to train a good classifier. Co-training uses the predicted samples to retrain the classifiers. This is analogous to self-training, and the difference is that the classifier in self-training teaches itself while co-training uses the predictions of one classier to teach the other classifier. Two classifiers are trained initially with the labeled samples on the two sub-feature sets respectively. Then each classifier predicts the labels of unlabeled samples. The most confident unlabeled samples from each classifier are added into the training set for the other classifier. Afterwards, the two classifiers are retrained and the procedure repeats. There are several noteworthy issues in co-training (Zhu, 2005a). The sub-features must be good enough so that the labels predicted by each learner on the unlabeled samples can help improve the performance of the other classifier. And the sub-features sets should be conditionally independent so that one classifier's high confident predicted samples are iid samples for the other classifier.
Transductive Support Vector Machines (TSVM) Transductive support vector machine (TSVM) is an extension of standard support vector machine (SVM). Regular SVM tries to induce a general decision function for a learning task. It only uses labeled data to find a maximum margin linear boundary in the Reproducing Kernel Hilbert Space (RKHS). TSVM takes into account a particular test set and tries to minimize misclassifications of just those particular examples (Joachims, 1999). So, it tries to find a linear boundary which has the maximum margin on both the labeled samples and the unlabeled samples. The decision boundary has the smallest generalization error bound on unlabeled data (Vapnik, 1998). In many practical tasks, TSVM can achieve
Figure 1. The difference between SVM and TSVM
TSVM
a better performance than normal SVM, especially when only a small training set is available. SVM works on p(y|x) directly. This brings up the danger of leaving p(x) outside of the parameter estimation loop, if p(x) and p(y|x) do not share parameters (Zhu, 2005a). It is worth noticing that since we can only get information about p(x) from unlabeled data, semi-supervised learning cannot help if p(x) and p(y|x) do not share parameters (Seeger, 2001). To build the connection between p(x) and the discriminative decision boundary, TSVMs do not allow putting the boundary in high density regions. The differences between SVM and TSVM are shown in Fig. 1. The maximum margin boundary for SVM is shown in the left figure (plotted with dashed lines) while the maximum margin boundary is shown in the right figure (plotted with solid lines). In the cases like the one shown in the figure, TSVM will get better results than SVM. However, finding the optimal TSVM solution is NP-hard. Several algorithms (Joachims, 1999; Bennett & Demiriz, 1999; Chapelle & Zien 2005; Fung & Mangasarian 1999) have been proposed for approximation.
Graph-Bsed Methods As a major family of semi-supervised learning, graph-based methods have attracted more and more recent research. Many works on this topic are reported in the literature of machine learning community (Chapelle, Zien, & Scholkopf, 2006; Zhu, 2005a). Graph-based methods define a graph with the nodes representing the samples (including labeled and unlabeled ones) in the dataset while the edges reflecting the pair-wise similarity between samples. Graph based methods are nonparametric, discriminative, and transductive in nature. Some basic notations for the graph-based semi-supervised learning are defined as follows: let X={x1; … ; xl; xl+1; … ; xn} be a set of n samples (i.e., video shots for our application) in Rm (feature space with m dimensions). The first l points are labeled as yL= [y1; y2; … ; yl]T with yi ∈{0, 1} (1 ≤ i ≤ l) and the remained points xu (l + 1 ≤ u ≤ n) are unlabeled. The vector of the predicted labels of all samples is represented as f, which can be split into two blocks after the l-th row:
f L  f =  fU 
Consider a connected undirected graph G = (V, E) with the vertex set V corresponding to the n data points. Here V = L∪U, where the vertex set L = {x1; …; xl} contains labeled points and the vertices in set U = {xl+1; …; xl+u} are unlabeled ones. The edges E are weighted by the n×n pair-wise similarity matrix. Most graph-based methods can be viewed as an estimation of a decision function f on the graph. The function f is required to satisfy two conditions: (1) it should be close to the given labels yL on the labeled nodes, and (2) it should be smooth on the whole graph. This can be expressed in a regularization framework where the first term is a loss function (also called fit term), and the second term is a regularizer.
Min-Cut Blum & Chawla (2001) formulate semi-supervised learning as a graph min-cut problem. Let G be the undirected edge-weighted graph with nonnegative weights. A cut C of G is any nontrivial subset of V, and the weight of the cut is the sum of weights of edges crossing the cut. A mincut is then defined as a cut of G of minimum weight. The problem is polynomial time solvable as a series of network flow problems or using the algorithm of Stoer & Wagner (1994). It is a binary case, that is to say, f i ∈ {0,1}. Equivalently mincut is the mode of a Markov random field with binary labels (Boltzmann machine). The loss function can be viewed as a quadratic loss with infinity weight, i.e.,
∞ ∑ i∈L ( f i - yi∈L ) , 2
and the regularizer is 1 ∑ wij ( fi - f j )2. 2 i, j From the fit term, we can see that the labels of the training samples will not change in the entire process. Putting the fit term and regularizer terms together, mincut can be seen as an optimization problem by minimizing:
1 2 w ( fi - f j ) 2 + ∞ ∑ i∈L ( fi - yi∈L ) ∑ i , j ij 2
subject to the constraint f i∈V∈{0, 1}.
Gaussian Random Fields and Harmonic Functions Method Zhu, Ghahramani, & Lafferty (2003) propose a Gaussian random fields and harmonic function method with a regularizer of graph combinatorial Laplacian. It will be abbreviated to GRF method in this chapter. It is a continuous relaxation to the discrete Markov random fields. The optimal prediction function f is estimated by minimizing the energy: 1 2 ∑ wij ( fi - f j )2 + ∞∑ i∈L ( fi - yi ) 2 i, j , 2 = f T + ∞ ∑ i∈L ( f i - yi )
where ∆ = D – W is the combinatorial Laplacian, W is the n × n similarity matrix and D is a diagonal matrix with Dii = ∑ j wij. The main difference between GRF method and min-cut is that f i∈R in GRF method and f i∈V∈{0, 1} in min-cut.
Local and Global Consistency Method The local and global consistency method proposed by Zhou, Bousquet, Lal, Weston, & Scholkopf (2003) is another widely-applied method. It adopts a regularizer of the normalized graph Laplacian D–1/2DD–1/2 = I – D–1/2WD–1/2. We abbreviated this method to consistency method in this chapter. The optimal prediction function f * is defined to minimize a cost function Q( f ): 1 n f = arg min Q( f ) = arg min ( ∑ wij 2 i , j =1 f f *
fi -
∑ i =1
f i - f j ),
The first term of Q( f ) is a smoothness constraint, which indicates that a good predicting function should not change too much between neighboring points. The second term is a fit constraint, which means that a good classifying function should not change too much from the initial label assignment. The trade-off between these two competing constraints is captured by a positive parameter µ. Note that the regularization term contains both the labeled and unlabeled data.
Linear Neighborhood Propagation (LNP) Instead of considering pair-wise similarity relationships in traditional graph-based methods, LNP (Wang & Zhang, 2006) attempts to use the neighborhood information of each point to construct the graph G. The reconstruction matrix W is constructed by minimizing
= xi - ∑ j:x ∈N ( x ) wij x j j
∑ j:x ∈N ( x ) wij ( xi - x j ) j
= ∑ j , k :x
j , xk ∈N ( xi )
wij wik ( xi - x j )T ( xi - xk ) wij G ijk wik
subject to Σjwij = 1 and wij ≥ 0.
Once the W is constructed, the optimal prediction function f is obtained by minimizing
Q( f ) =
1 n ∑ ∑ wij ( fi - f j )2 + 2 i =1 j:x j ∈N ( xi )
∑( f i =1
- yi ) 2.
Local Learning Regularization (LL-Reg) In (Wu & Scholkopf, 2007), the authors formulate the above graph based methods as a quadratic optimization problem, minn f T Rf + (f - y ) C (f - y ), T
f ∈
where R ∈ Rn×n is the regularization matrix, f = [ f1,...,fn] ∈ Rn is the vector of real valued solution, y = [y1,...,yj, 0, ...,0] ∈ Rn and C ∈ Rn×n is a diagonal matrix with the entry Cii = Cl > 0 for 1 ≤ i ≤ l, and Cii = Cu ≥ 0 for l + 1 ≤ i ≤ n, where Cl and Cu are two parameters. We can see that R = D in GRF method and R = D–1/2DD–1/2 in consistency method. LL-Reg (Wu & Scholkopf, 2007) replaces the regularizer fRf with
n i =1
( fi - oi ( xi )) 2,
where oi(⋅) denotes the output function of a model, trained locally with some supervised learning algorithms using the labeled data. Then the optimization problem becomes minn f - o + (f - y ) C (f - y ) f ∈ 2
where o = [o1 ( x1 ),..., on ( xn )]T.
A solution for oi (xi) is given as: oi (xi) = afi,
where fi ∈  ni is the vector[ f i] for xj ∈ Ni (Ni denotes the set of neighboring points of xi, not including xi, and ni denotes |Ni |), and
T i
1T - 1T XiT Xi ( I + XiT Xi ) -1 ni - 1T XiT Xi ( I + XiT Xi ) -11,
where 1 is the vector of all 1's, Xi ∈  d ×ni denotes the matrix [xj-xi] for xj ∈ Ni. Here Ni is independent of fi and it is different for different xi. Note that fi is a sub-vector of f, so equation (13) can be written in a matrix form as: 189
o = Af
where o is the same as in (8), while the matrix A = [aij ] ∈  n×n is constructed as follows: ∀xi and xj 1 ≤ i, j ≤ n, if xj ∈ Ni, then aij equals the corresponding element of αi in (10), otherwise aij equals 0. Similar as αi, the matrix A is also independent of f. Substituting (11) into (8), we can see that the regularizer in LL-Reg is R = (I - A) (I - A)
LNP can also be represented in this local learning regularization framework with oi ( xi ) = ∑ x ∈N wij f j. j
Other Regularizations Belkin, Matveeva, & Niyogi (2004) propose a Tikhonov regularization algorithm by minimizing
1 / k ∑ i =1 ( f i - yi ) 2 + f T Sf , n
where S = D or Dp with some certain integer p. The manifold regularization framework (Belkin, Niyogi, & Sindhwani, 2006) minimizes the cost function: 1 l ∑V ( xi , yi , f ) + l i =1
f I ,
wherein two regularization terms are employed. Here V is an arbitrary loss function; K is a 'base kernel', e.g. a linear or RBF kernel, and I is a regularization term induced by the labeled and unlabeled data.
Graph Construction The construction of the graphs is "at the heart" of graph based semi-supervised learning methods (Zhu, 2005a). However, this issue has not been studied extensively. Balcan et al. (2005) build graphs for video surveillance using strong domain knowledge. The graph consists of time edges, color edges and face edges. Robust graphs are built in (Carreira-Perpinan & Zemel, 2005) from multiple minimum spanning trees by perturbation and edge removal. The graph construction may be domain specific as it needs prior knowledge embedded. Detailed discussions on this problem can be found in Chapter 3 and Chapter 7 of Zhu's PhD thesis (Zhu, 2005b).
Learning with Directed Graphs Most of the graph-based semi-supervised learning methods are based on undirected graphs. There are few methods based on directed graphs. Zhou, Scholkopf, & Hofmann (2005) investigate how to exploit the link structure of the graph and proposed a regularization framework for classifications on the directed graphs. The fundamental assumption of this method is the category similarity of co-linked nodes in a directed graph. It assumes that nodes with highly overlapping parent sets are likely to have same category. This assumption is shown to be effective for Web classification. Zhou, Huang, & Scholkopf
(2005) further generalize their work to a general framework semi-supervised learning on directed graphs. It takes a transition matrix as input and gives a closed form solution on unlabeled data. We have briefly introduced some common semi-supervised learning methods. All these methods can be categorized as manifold learning, which have a close relation to spectral graph theory (Chung, 1997). There are still many other related methods. A more comprehensive survey can be found in (Zhu, 2005a).
imag videmananaemi-Supervivi Learning Generally the labeled samples in multimedia semantic analysis come from the users during an interactive session. Manual annotating is labor-intensive and time-consuming. To reduce the labor of users, lots of efforts have been made to obtain good results using a very small amount of labeled data. As unlabeled multimedia data are much easier to be obtained, many semi-supervised learning approaches have been applied for image/video annotation and content based image/video retrieval, by leveraging a large amount of unlabeled samples with certain assumptions. This section provides a brief introduction of these works. As there are lots of works on the applications of semi-supervised learning in the multimedia area, we do not give a complete survey here, only part of these works are introduced. Co-training is often applied in the multimedia analysis. Song, Hua, Dai, & Wang (2005) adopt the co-training method to video annotation, and in (Song, et al., 2006) they further apply a semi-supervised ensemble method for video annotation. Yan & Naphade (2005) analyze the drawbacks of co-training in video annotation, and propose an improved co-training style algorithm named semi-supervised crossfeature learning. In (He, Zhang, Zhao, & Tong, 2005), a co-ranking framework is proposed to re-rank the retrieved images to move the irrelevant ones to the tail of the ranked list returned by a web image search engine. As web-based images possess both intrinsic visual contents and text annotations, Feng, Shi, & Chua (2004) propose to bootstrap the learning process for web image annotation by adopting a co-training approach involving classifiers based on two orthogonal set of features – visual and text. In (Wang, Hua, Song, Lai, Dai, & Wang, 2007), co-training is applied to annotate the video shot size. Zhou, Chen, & Dai (2006) enhance the relevance feedback for image retrieval by integrating the merits of co-training and active learning into the relevance feedback process. Graph-based methods are the most popularly used semi-supervised learning techniques in multimedia area recently. He et al. (2004; 2006) propose manifold-ranking approaches to propagate the label information of image samples from positively-labeled ones to unlabeled samples. In (Yuan, et al., 2006), a manifold ranking method based on feature selection is proposed for video annotation. Wang, Jing, Zhang, & Zhang (2006) propose a method based on random walk with restarts to refine the results of image annotation. A method based on kernel density estimation is proposed in (Wang, Hua, Song, Yuan, Li, & Zhang, 2006) for video semantic detection, in which the authors show that this method has close relationship with the graph-based semi-supervised learning. In (Tang, Hua, Qi, Song, & Wu, 2008), the authors analyze the limitation of linear neighborhood propagation (Wang & Zhang, 2006) for video annotation and propose a kernel-mapped linear neighborhood propagation method. In (Tong, et al., 2005), Tong et al. propose an image retrieval method to deal with two modalities in graph-based semi-supervised learning scheme. Motivated by this, Wang, Hua, Yuan, Song, & Dai (2007) propose a method OMG-SSL to integrate multiple graphs into a regularization and optimization framework to
sufficiently explore their complementary nature for video annotation. A PageRank-like multi-graph fusion method is proposed in (Liu, Lai, Hua, Huang, & Li, 2007) to simultaneously leverage textual relevancy, semantic concept relevancy, and low-level-feature-based visual similarity for video search re-ranking. In (Rui, Yuan, & Yu, 2006), a model based on semi-supervised clustering with semantic soft constraints is proposed for image annotation, in which both visual features and semantic meanings are utilized. Some graph-based methods try to analyze the image/video data with existed domain specific knowledge or context information. Tang, Hua, Mei, Qi, & Wu (2007) embed the temporal consistency of video data into the graph-based semi-supervised learning and propose a temporally consistent Gaussian random field method for video annotation. Liu, Li, Ma, Liu, & Lu (2006) propose a novel automatic image annotation method based on manifold ranking, in which the visual and textual information are well integrated, and the word-to-word correlations obtained from WordNet and the pair-wise co-occurrence are taken into consideration to expand the annotations and prune irrelevant annotations for each image. A context reranking approach (Hsu, Kennedy, & Chang, 2007) is proposed to leverage the recurrent patterns to improve the initial text search results for content based video search. This approach is formulated as a random walk problem along the context graph, where video stories are nodes and the edges between them are weighted by multimodal contextual similarities. Graph-based methods are also combined with other learning method to handle the image/video analysis problem. A semi-supervised framework for image retrieval is proposed in (Hoi & Lyu, 2005) through a fusion of graph-based semi-supervised learning and support vector machines. In (Tang, Hua, Qi, & Wu, 2007), a two-stage semi-supervised multiple-instance learning based typicality ranking scheme is proposed for natural scene annotation. Besides co-training and graph-based methods, there are few other semi-supervised learning techniques applied to multimedia area. In (Li & Sun, 2006), a semi-supervised conditional random field method is proposed to image annotation. This method can effectively capture the spatial dependency between the neighboring labels while exploit the unlabeled data to improve the joint classification performance. Although TSVM has shown superior performance in many applications, it is hardly used in multimedia analysis due to the large computational cost. Table 1 gives a summarization of these works. From this table, we can see the popularity of graphbased semi-supervised learning, which is the topic this chapter focuses on.
Struun Anisotropic Manifold Ranking We have introduced several semi-supervised learning methods and their applications in multimedia semantic analysis. As an example, in this section we will detail a novel graph-based method SSAniMR (Tang, Hua, Qi, Wang, Mei, & Wu, 2007), which is applied to annotate the video semantics.
Motivation The basic assumption used in graph-based semi-supervised learning is: nearby points are likely to have the same label. It is referred to as neighborhood assumption in this chapter. Meanwhile, pair-wise similarity measure is a key point for graph-based methods, as it is the basis of label propagation, which
Table 1. A summarization of several works on the applications of semi-supervised learning in multimedia area Approach "Semi-supervised cross feature learning for semantic concept detection in videos" (Yan & Naphade, 2005) "Semi-automatic video annotation based on active learning with multiple complementary predictors" (Song, Hua, Dai, & Wang, 2005) "Ef.cient semantic annotation method for indexing large personal video database" (Song, et al., 2006) "Boosting web image search by co-ranking" (He, Zhang, Zhao, & Tong, 2005) "A bootstrapping framework for annotating and retrieving WWW images" (Feng, Shi, & Chua, 2004) "An efficient automatic video shot size annotation scheme" (Wang, Hua, Song, Lai, Dai, & Wang, 2007) "Enhancing relevance feedback in image retrieval using unlabeled data" (Zhou, Chen, & Dai, 2006) "Semi-supervised learning for image annotation based on conditional random fields" (Li & Sun, 2006) "Manifold-ranking based image retrieval" (He, et al., 2004) "Generalized manifold-ranking based image retrieval" (He, et al., 2006) "Manifold-ranking based video concept detection on large database and feature pool" (Yuan, et al., 2006) "Image annotation refinement using random walk with restarts" (Wang, Jing, Zhang, & Zhang, 2006) "Video annotation based on temporally consistent Gaussian random field" (Tang, Hua, Mei, Qi, & Wu, 2007) "Automatic video annotation by semi-supervised learning with kernel density estimation" (Wang, Hua, Song, Yuan, Li, & Zhang, 2006) "Video Annotation Based on Kernel Linear Neighborhood Propagation" (Tang, Hua, Qi, Song, & Wu, 2008) "Typicality ranking via semi-supervised multiple-instance learning" (Tang, Hua, Qi, & Wu, 2007) "A semi-supervised active learning framework for image retrieval" (Hoi & Lyu, 2005) "An adaptive graph model for automatic image annotation" (Liu, Li, Ma, Liu, & Lu, 2006) "Image annotations based on semi-supervised clustering with semantic soft constraints" (Rui, Yuan, & Yu, 2006) "Video search reranking through random walk over document-Level context graph" (Hsu, Kennedy, & Chang, 2007) "Graph-based multi-modality learning" (Tong, et al., 2005) "Optimizing multi-graph learning: towards a unified video annotation scheme" (Wang, Hua, Yuan, Song, & Dai, 2007) "Video search reranking via multi-graph propagation" (Liu, Lai, Hua, Huang, & Li, 2007)
Learning method Co-training
Application Video annotation
Co-training
Video annotation
Image retrieval
Image annotation
Semi-supervised CRF
Graph-based Graph-based Graph-based
Image retrieval Image retrieval Video annotation
Graph-based
Graph-based Graph-based
Image retrieval Video annotation
is an essential technique in most of the graph-based methods. Neighborhood assumption can be easily enforced into the pair-wise similarity as
exp(- || xi - x j ||2 /2 wij =  0
i≠ j i= j
where xi is the low-level feature vector of sample i. This assumption is very useful and widely applicable. However, it sometimes is too weak, since most real-world dataset (especially the image or video dataset) has more complex structures than this assumption can capture (Bousquet, Chapelle, & Hein, 2003), such as the samples clustered in a certain region may have the same labels. This can be formulated as structural assumption or global consistency assumption (Zhou, Bousquet, Lal, Weston, & Scholkopf, 2003): points on the same structure (typically referred to as a cluster or a manifold) are likely to have the same label. Unfortunately, like the one defined in (14), typical similarity definitions do not take the structural assumption into account. Instead, this assumption is generally combined into most methods through an iterative label propagation process. Though weighted by distance, the label propagation is still isotropic, which will be shown later. This means the direct labeling contribution (i.e. the propagation coefficient which will be discussed later) from one sample to another, which is proportional to the pair-wise similarity, neglects the influence of the structural difference. It is believed that embedding the structural assumption into the pair-wise similarity will furthermore improve the performance of the general graph-based methods (Tang, Hua, Qi, Mei, & Wu, 2007). An exemplary case is shown in Fig. 2, in which each point represents a sample in the feature space. The distance between A and C is equal to the distance between A and B, that is, wAC equals to wAB according to the normal similarity measure. However, it is more reasonable if the similarity wAB is larger than wAC, as A and B are on the same structure while A and C are on different structures. So we need to enhance the propagation between the samples in the same structure while weakening the counterpart between the samples in the different structures, according to the structural assumption. Accordingly, Structure-Sensitive Anisotropic Manifold Ranking (SSAniMR) (Tang, Hua, Qi, Wang, Mei, & Wu, 2007) is derived from a Partial Differential Equation (PDE) based anisotropic diffusion framework (Perona & Malik, 1990; Sapiro, 2001), by embedding the structural assumption into the similarity measure. From the view of PDE-based diffusion, we can see that the label propagation in SSAniMR is anisotropic, which is intrinsically different from the isotropic label propagation process in general graph-based methods. That is why this method is named using "anisotropic".
Structure-Sensitive Anisotropic Manifold Ranking In this section, we first introduce related graph-based methods, then analyze the influences of the point probability density on general graph-based approaches, and finally detail the SSAniMR. Figure 2. Example for the influence of structure assumption A
Related Graph-Based Semi-Supervised Learning Methods As aforementioned, graph-based semi-supervised learning methods have attracted lots of interests in machine learning community recently. Many methods have been proposed, such as the well-used Gaussian random fields and harmonic function based method (GRF method) (Zhu, Ghahramani, & Lafferty, 2003), and the local and global consistency method (consistency method) (Zhou, Bousquet, Lal, Weston, & Scholkopf, 2003). Most graph-based semi-supervised learning methods have a label propagation procedure; a key point in which is the propagation coefficients (also can be seen as transition probability or pseudo-transition probability). Pair-wise similarity measure is the basis of propagation coefficient. In (Zhu, Ghahramani, & Lafferty, 2003), the similarity between samples i and j is defined as:
wij = exp( -∑ d =1 m
( xid - x jd ) 2 2 d
if i ≠ j, and wii = 0. Here xik is the k-th component of sample i (represented as a vector xi ∈ Rm), and s1, ..., sm are scale hyper-parameters for each dimension. This definition has a large number of parameters, which are not optimally learned in terms of difficulty. Therefore, the pair-wise similarity in (14), which is defined on the normalized features, is commonly applied in practice. Based on the similarity measure (14), the propagation coefficient from vertex j to vertex i is defined in GRF method (Zhu, Ghahramani, & Lafferty, 2003) and consistency method (Zhou, Bousquet, Lal, Weston, & Scholkopf, 2003) respectively as: sij = wij / di
sˆij = wij / di d j
where di = ∑ j wij.
It is obvious that sij ≥ 0, ∑ j =1 sij = 1 and sˆij ≥ 0, ∑ sˆij ≠ 1. j =1 n
Influences of Probability Density The knowledge of the joint probability density p(x; y) is enough to achieve the optimal classification in supervised learning (taking argmaxyp(x; y) as decision function), while in semi-supervised learning, even if one knows the density distribution p(x) of the samples, there is no unique or optimal way of using p(x) (Bousquet, Chapelle, & Hein, 2003). Here the influences of point probability density on the normal graph-based semi-supervised learning methods are discussed. A most popular density estimation method is the Parzen window density estimation (Duda, Stork, & Hart, 2000), by which the probability density of a sample i is estimated by
pi =
1 Ni
∑ k(x
j ~i
- x j )
where Ni is the number of samples connect to sample i, and k(x) is a kernel function that satisfies k(x) > 0 and ∫ k ( x)dx = 1. If we use Gaussian kernel, that is,
k ( xi - x j ) =
exp(-
|| xi - x j ||2
di,
we can see that pi =
since 1 2 N is a constant, so di can be seen as a measure of density for sample i. This demonstrates that both sij and sˆij have intrinsically taken the influence of density into account. The propagation coefficient in (16) can be seen as the transition probability from j to i. However, this coefficient just considers the influence of the individual density but neglects the influence of density difference between samples. Rewrite the propagation coefficient in (17) as: sˆij = wij / di d j =
wij d j -1/ 2 dj ( ) = sij ( ) -1/ 2 di di di
The term dj/di can be regarded as a measure of density difference between samples j and i. We can extend (17) to a general case: sˆij = sij ⋅ g (
) di
where g(x) is a strictly decreasing function of x. That is to say, for a fixed vertex i, the propagation coefficient from its nearby vertex j to itself strictly decreases accompanied with the increment of the density difference dj/di. Therefore, the propagation coefficient in (17) takes the influence of density difference into account. However, there are two drawbacks of this coefficient definition. First, we can see that
sˆ ≠ 1,
j ij
which means it cannot be seen as a transition probability. We call this propagation coefficient pseudotransition probability. Second, a fixed g(x), that is, g(x) = (x)-1/2, may not be the optimal degradation
function, since frequently the optimization objectives vary with different real-world applications. Recall the example in Fig. 2, the density dC is smaller than the density dB. According to (17), the propagation coefficient from B to A will be smaller than the counterpart from C to A, which obviously violates the structural assumption.
From PDE-Based Diffusion to Structure-Sensitive Anisotropic Manifold Ranking In this subsection, we first discuss the connection between normal graph-based semi-supervised learning and isotropic heat diffusion. And then the SSAniMR is deduced from an anisotropic diffusion framework. The discrete version of the heat diffusion equation (Sapiro, 2001),
∂f = Df , ∂t where ∆ is the combinatorial Laplacian, can be written as:
f i ( t ) - f i ( t -1) =
∑ j ≠i wij
∑w ( f j ≠i
( t -1) j
- f i ( t -1) ).
Notice that wii ( f i ( t -1) - f i ( t -1) ) ≡ 0,
that is to say, the value of wii does not affect the result, so we can set wii = 0. Let
di = ∑ j =1 wij, n
equation (23) can be rewritten as: f i ( t ) - f i ( t -1) =
∑w ( f j =1
∑w j =1
- f i ( t -1) )
f j( t -1) - f i ( t -1)
Represent it with matrix form, we have f ( t +1) - f ( t ) = (D-1W - I )f ( t ),
and therefore
f ( t +1) = D-1Wf ( t ) = Sf ( t ),
where f(t) is the label vector at iteration point t. W = [ wij ]n×n is the similarity matrix with wii = 0, S is the normalization of W with the entries 197
sij =
wij di
and D = diag(di) is the diagonal matrix with entries di = ∑ j =1 wij. n
Split the matrix S after the l-th row and l-th column, we have
S LL S= SUL
S LU  , SUU 
and split the vector f(t) like in (1), then (26) can be rewritten as: f L( t ) = S LL f L( t ) + S LU fU( t )  (t ) (t ) ( t ) fU = SUL f L + SUU fU
Enforce the constraint that the labels of the labeled samples will not change in the diffusion procedure, that is f L( t ) ≡ y L, into the second equation in (28), we get:
fU( t ) = SUL f L( t -1) + SUU fU( t -1) = SUL f L( t -1) + SUU (SUL f L( t - 2) + SUU fU( t - 2) ) = ... t -1
i t fU(0) )SUL y L + SUU = (∑ SUU i =0
It is obvious that sij ≥ 0, sii = 0 and
s = 1,
j =1 ij
S and SUU are both non-negative matrices. When we connect each vertex to all other vertices (the case with a sparse representation of the matrix will be discussed in the section of experiments), every vertex j ∈ L will satisfy sij > 0 when j ≠ i, which results in that ∑ sij > 0. j∈L Therefore
sij =∑ j =1 sij - ∑ j∈L sij ≤ 1 - ∑ j∈L sij < 1. n
j∈U
According to the spectral diameter bound for the non-negative matrix in the matrix theory (Chung, 1997): n
( A) ≤ max ∑ aij. 1≤ i ≤ n
j =1
We have: (SUU ) ≤ max
l +1≤ i ≤ n
∑s
< 1.
Therefore i t lim ∑ i = 0 SUU = lim(I - SUU ) -1 (I - SUU ) = (I - SUU ) -1 t -1
t →∞
and t lim SUU = 0, t →∞
where 0 is a (n - l)×(n - l) matrix with each entry equals to 0. Through iterations until convergence, the optimal result will be obtained: fU* = lim fU( t ) t →∞
 t -1 i  t )SUL y L + SUU fU(0)  = lim (∑ SUU t →∞  i =0  -1 . = (I - SUU ) SUL y L
This result is the same as the result of a normal graph-based method: Gaussian Random Field (GRF) (Zhu, Ghahramani, & Lafferty, 2003). Since the heat diffusion is isotropic, the label propagation procedure in this method is isotropic though weighted by distance, which does not accord with the structural assumption of graph-based methods. As aforementioned, we need to enhance the label propagation within a cluster while weakening the counterpart through different clusters according to structural assumption. In general, the density variation within a cluster will be smaller than the density variation between different clusters. From this, structural assumption encourages propagating label information within a region with uniform or close density in preference to propagating across the density boundaries. In the example shown in Fig. 3, we want to enhance the direct propagating strength from B to A while suppressing the counterpart from C to A. Accordingly we consider an anisotropic diffusion equation:
∂f (34) = div( g (| ∇p |)∇f ) ∂t This is a higher dimensional generalization of the anisotropic diffusion equation in (Perona & Malik, 1990), in which g(x) is a nonnegative decreasing function and p is the density distribution. Discretize the equation like the Perona-Malik Discrete Formulation (Sapiro, 2001; Perona & Malik, 1990), we have f i ( t ) - f i ( t -1) =
∑w g j ≠i
( f j( t -1) - f i ( t -1) )
where wij is the distance weighted similarity and gij is the density similarity.
Figure 3. Left figure shows the direct propagating strength from B to A equals to the counterpart from C to A. In the right figure we enhance the direct propagating strength from B to A while suppressing the counterpart from C to A.
= ∑ j =1 wij / ∑ j =1 wij gij, n
Notice that wii = 0 and set
∑w g
 -1 ( W ⋅ G ) - I )f ( t -1) f ( t ) - f ( t -1) = (D
Let di
∑ wg = ∑ w g and represent (36) using matrix form: n
ij j =1
therefore  ( t -1)  -1 ( W ⋅ G )f ( t -1) = Sf f (t ) = D
where S represents the matrix whose entry in i-th row and j-th column is ( gij wij ) / ∑ j =1 ( gij wij ) and sii = 0, n
which takes both the pair-wise distance and density difference into account. Then the predicted label vector of unlabeled data will be obtained: fU( t ) = S UL f L( t -1) + S UU fU( t -1).
It is worth noticing that, although (39) has similar form with (29), they have different intrinsic meanings: the label propagation in (29) is distance weighted isotropic while the propagation in (39) is structure-sensitive anisotropic. Enforce the constraint f L( t ) = y L, similar to the iterative process above we can obtain the result:
fU* = lim fU( t ) = ( I - S UU ) -1 S UL y L. (40) t ->∞ This iterative method deduced from anisotropic diffusion is named as Structure-Sensitive Anisotropic Manifold Ranking (SSAniMR) (Tang, Hua, Qi, Wang, Mei, & Wu, 2007).
The Semi-Supervised Learning View of SSAniMR In this subsection, we can see that SSAniMR is actually a graph-based semi-supervised learning method based on a structure- sensitive similarity measure. As aforementioned, the similarity measure will be more accurate if the structural assumption is embedded into it and the density variation within a cluster generally will be smaller than the density variation between different clusters. We assume that the similarity between two samples not only decreases with respect to the increment of their distance in the feature space, but also decreases with the increment of their density difference. This pair-wise similarity is more consistent with the structural assumption. Define a density similarity between xi and xj as:
gij = exp( -
( pi - p j ) 2 2
where pi is the probability density of sample xi. The new structure-related pair-wise similarity is defined as:
w ij = wij ⋅ gij = exp( -
|| xi - x j ||2 2
) ⋅ exp(-
while i ≠ j and w ii = 0. The first term in the right side of (42) shows that the similarity between two samples decreases with respect to the increment of their distance in the feature space. The second one demonstrates that the similarity decreases with the increment of the density difference. That is to say, this similarity definition not only considers the neighborhood assumption, but also takes the structural assumption into account. Motivated by the two basic assumptions, we can infer the real-valued labels for the unlabeled samples by minimizing the energy function Q( f ) =
1 n ( f i - yi ) 2 ∑ wij gij ( fi - f j )2 + ∞∑ 2 i , j =1 i∈L
Then the prediction function is f * = argminf Q( f ). Intuitively, the first term of Q( f ) describes the total variation of the data labels with respect to the local structures, also called smoothness term. This term means that a good prediction function should not change too much between nearby density-similar points. The second term enforces the invariant constraint for the labels of labeled data f i * = yi (i ∈ L), also called fit term. Differentiate Q( f ) with respect to f and represent it with matrix form:
∂Q  * + ∞(f * - y ) = 0 = f * - Sf L L ∂f f =f *
which can be transformed into
 * f * = Sf s.t. f L* = y L
 -1 ( W ⋅ G ) and · represents the component-wise Hadamard product (Duda, Stork, & Hart, where S = D 2000). W is the normal similarity matrix with entries wij defined in (1) and G is the density similarity  = diag (d ) is the diagonal matrix with entries matrix with entries gij defined in (12). D i n di = ∑ j =1 wij gij.
That is to say, the propagation coefficient in SSAniMR is w ij wij gij = n sij =  di ∑ wij gij j =1 It is obvious that sij ≥ 0, sii = 0 and ∑
s = 1.
So it can be seen as a transition probability. And the gij incorporate the influence of density difference while the parameter sp controls the significance of the influence. Split the matrix S after the l-th row and l-th column, like in (28), then (45) can be rewritten as:
fU* = S UL f L* + S UU fU* s.t. f L* = y L
Solve this linear equation, consequently the optimal solution for fU is obtained: fU* = ( I - S UU ) -1 S UL y L.
Consequently, the same results of SSAniMR are deduced from two different viewpoints. The deep research in PDE-based diffusion (Sapiro, 2001) will give us new insights for graph-based semi-supervised learning.
Eperiments SSAniMR is evaluated on a widely used benchmark video dataset by comparing it with the Columbia374 SVM detectors (Yanagawa, Chang, Kennedy & Hsu, 2007) and two popularly used graph-based methods.
Experimental Settings Experiments are conducted on the benchmark video corpus of the TRECVID 2005, which is consisted of about 170 hours of TV news videos from 13 different programs in English, Arabic and Mandarin (TRECVID, http://www-nlpir.nist.gov/projects/trecvid). The development (DEV) set of TRECVID05 is used in our experiments. After automatic shot boundary detection, the DEV set contains 137 broadcast
Figure 4. Comparisons of results using color moments
news videos with 43,907 shots. Some shots are further segmented into sub-shots, and there are 61,901 sub-shots for the DEV set. For each sub-shot, 39 semantic concepts are labeled according to LSCOMLite annotations (Naphade, et al., 2006). These annotated concepts consist of a wide range of genres, including program category, setting/scene/site, people, object, activity, event, and graphics. The DEV set is separated into four partitions with 90 videos as training subset, 16 videos as validation subset, 16 videos as fusion subset, and 15 videos as test subset. Here training subset is used to train models, validation subset is used to validate the optimal parameters, fusion subset is used to choose the linear fusion coefficients and test subset is used for testing. More details about the dataset separation can be found in (Yanagawa, Chang, Kennedy & Hsu, 2007). For each concept, systems are required to return ranked-lists of up to 2000 sub-shots, and system performance is measured by the official performance metric Average Precision (AP) (Trec-10 proceedings appendix on common evaluation measures. http://trec.nist.gov/pubs/trec10/appendices/measures. pdf ) in the TRECVID tasks. The AP corresponds to the area under a non-interpolated recall/precision curve and it favors highly ranked relevant sub-shots. The APs over all the 39 concepts are averaged to create the Mean Average Precision (MAP), which is the overall evaluation result. Two low-level feature sets are used in the experiments: the first one is 225-D block-wise color moments in LAB color space, which are extracted over 5×5 fixed grid partitions, each block is described by a 9-D feature; and the second one is 75-D edge distribution layout. Experiments are conducted on the two feature sets respectively and then the two results are combined through linear fusion.
Implementation Issues When implementing the SSAniMR algorithm, we need to calculate the inversion or the multiplication of the large-scale matrices in (40) or in the iterations of (39), which are difficult to be implemented subject to the limitation of both the computing ability and the memory quantity. For example, the video data set typically is very large (e.g., the DEV set in TRECVID05 has 61,901 sub-shots); it is difficult to storage the similarity matrix and compute its inversion. To deal with this issue, the graph is simplified by only
Figure 5. Comparisons of results using edge distribution layout
 and S are sparse. Moreover, we can also calculate connecting neighboring points, and thus matrices W it offline. In this way, the quantity of memory and the processing time requested are greatly reduced. There are two methods to find appropriate set of neighboring points for calculating the sparse representation of the matrices W and S (Duda, Stork, & Hart, 2000): (a) k-NN: find the k nearest neighbors for each point; and (b) ε-NN: find nearest neighbors in the super-sphere centered at current point with radius of ε. Another issue is to guarantee the convergence of the iterations. As mentioned before, for each sample xi,
sij < 1
is a necessary condition to ensure the convergence of the iteration process (39). But when using k-NN or ε-NN to choose the connecting neighbors, this requirement may not be satisfied since the chosen neighbors may all belong to the unlabeled set U, thus
sij
will equal to 1. That is to say, the convergence of the iterative process cannot be ensured in this case. To tackle this difficulty, a degradation factor λ (λ = 1 – δ, where δ is a small arithmetic number) is introduced into (38) as
 ( t ). f ( t +1) = Sf Subject the invariant constraint f L( t ) ≡ y L, we obtain
Figure 6. Comparisons of results after linear fusion
fU( t ) = S UL f L( t -1) + S UU fU( t -1) t -1
= ∑ ( S UU )i ( S UL )y L + ( S UU )t fU i =0
It has been shown that ∑ j∈U sij ≤ 1 and 0 < l < 1, which easily result in ( S UU ) < 1. This leads to lim( S UU )t = 0 and t →∞
t -1
lim ∑ ( S UU )i = ( I - S UU ) -1. t →∞
i =0
fU* = lim fU( t ) = ( I - S UU ) -1 ( S UL )f L. t →∞
In applications with large-scale data, we can use (50) to replace (39) to implement SSAniMR.
Experimental Results In (Tang, Hua, Qi, Wang, Mei, & Wu, 2007), k-NN is adopted to find the neighboring points (k is set to 30 empirically). This process is relatively time-consuming. It needs about 15 hours (with Intel P4 3.0G and 2G memory). However, this process is also required in other graph-based methods and can be calculated offline. The density is estimated by the distance-weighted Parzen window (Duda, Stork, & Hart, 2000). Once the nearest neighbors are obtained, the density estimation is very fast. The parameters in these methods are all tuned to be nearly optimal through validations while l is empirically set to 0.99. Time costs of GRF method and consistency method are both about one minute, and SSAniMR costs about two minutes, all after the nearest neighbors are obtained. Although SSAniMR is a little slower than the other two methods, it is computationally effective as the processing time is always a challenging problem in TRECVID tasks.
For performance evaluation, SSAniMR is compared with the Columbia374 SVM detectors (Yanagawa, Chang, Kennedy & Hsu, 2007) and two popularly used graph-based methods: GRF method (Zhu, Ghahramani, & Lafferty, 2003) and consistency method (Zhou, Bousquet, Lal, Weston, & Scholkopf, 2003). The experimental results are shown in Fig. 4 (using color moments), Fig. 5 (using edge distribution layout), and Fig. 6 (for fused results) respectively. Comparing these results, we can see that SSAniMR performs the best on 21 of the all 39 concepts using color moments and performs the best on 25 of the all 39 concepts using edge distribution layout. After linear fusion, SSAniMR performs the best on 22 of the all 39 concepts. So SSAniMR is better than other methods for detecting most of these concepts. At the same time, we can see that SSAniMR's performance on some of these concepts is worse than the other three algorithms. This may be caused by the following reasons: (1) the structural assumption (i.e. manifold assumption) may not be well satisfied for these concepts in the low-level feature space SSAniMR adopted; (2) the regularization framework of SSAniMR (which is similar to GRF) is not as good as that of consistency method for detecting some concepts; (3) the parameter validation process of SSAniMR in this paper is not as subtle as other three algorithms. The MAP of SSAniMR is 0.427801 using color moments, which has an improvement of 6.26%, 2.01% and 25.57% over SVM, GRF method and consistency method, respectively. It has a MAP of 0.377531 using edge distribution layout, which respectively has an improvement of 17.09%, 2.08% and 46.72% over SVM, GRF method and consistency method. Through linear fusion, SSAniMR obtains a MAP of 0.450665, which has an improvement of 6.80% over SVM, 2.36% over GRF method and 22.25% over consistency method. All these comparisons demonstrate that SSAniMR is more appropriate than other semi-supervised learning methods and is effective for video semantic annotation.
DiSCUSSION Aand In this chapter we have introduced the popular semi-supervised learning methods and their applications in image/video content analysis. Besides, we have shown that the normal graph-based semi-supervised learning method can be deduced from PDE based diffusion. Accordingly, a novel semi-supervised learning algorithm named Structure-Sensitive Anisotropic Manifold Ranking is derived from PDE based anisotropic diffusion. Experiments conducted on the TRECVID dataset have demonstrated its effectiveness. It is worth noting that although many semi-supervised methods have been applied to image/video semantic analysis and have shown encouraging performance, their effectiveness on larger-scale dataset still need further validation. For example, in TRECVID high-level feature extraction, the best results are still obtained by purely supervised methods (TRECVID, http://www-nlpir.nist.gov/projects/trecvid). This may be attributed to several reasons, such as the high complexity of visual data and semantic concepts. Thus, semi-supervised learning based semantic analysis is still a challenging issue. There is also huge room for improvement of the existing methods. Here we briefly highlight several possible future works on similarity metric and graph-based learning: 1.
Region-level features can be incorporated into the pair-wise similarity metric, instead of only using frame-level features. Most of existing approaches are using frame-level features to model concepts. However, it is observed that actually concepts often more closely related to regions. Semi-Supervised Multiple Instance Learning (SSMIL) is a promising solution to this issue, where
a pioneer work has been reported in (Rahmani & Goldman, 2006). However, how to make it work for large-scale high-diverse video concepts is still not studied. Pair-wise similarity metrics are the same for all concepts in existing approaches. However, lowlevel features and local structures may have different influences for different concepts. To learn a specific similarity measure optimized for each concept, for example, parameterized by the choice of features, distance metrics, and structure differences, is promising to improve the performance. In other words, we may construct a concept-sensitive similarity measure, which take concepts into consideration in the similarity measure. Concepts are correlated rather than isolated. Leveraging label correlations in graph-based semisupervised learning to simultaneously model multiple concepts may have better performance than modeling them separately.
R TRECVID: TREC Video Retrieval Evaluation. http://www-nlpir.nist.gov/projects/trecvid. Trec-10 proceedings appendix on common evaluation measures. http://trec.nist.gov/pubs/trec10/appendices/measures.pdf Balcan, M.-F., Blum, A., Choi, P.P., Lafferty, J., Pantano, B., Rwebangira, M.R., & Zhu, X. (2005). Person identification in webcam images: An application of semi-supervised learning. In Proc. ICML Workshop on Learning with Partially Classified Training Data. Belkin, M., Matveeva, I., & Niyogi, P. (2004). Regularization and semi-supervised learning on large graphs. In Proc. Workshop on Computational Learning Theory. Belkin, M., Niyogi, P., & Sindhwani, V. (2006). Manifold regularization: A geometric framework for learning from labeled and unlabeled examples. Journal of Machine Learning Research, 7, 2399-2434. Bennett, K., & Demiriz, A. (1999). Semi-supervised support vector machines. Advances in Neural Information Processing Systems, 11. MIT Press. Blum, A., & Chawla, S. (2001). Learning from labeled and unlabeled data using graph mincuts. In Proc.18th International Conference on Machine Learning. Blum, A., & Mitchell, T. (1998). Combining labeled and unlabeled data with co-training. In Proc. Workshop on Computational Learning Theory. Bousquet, O., Chapelle, O., & Hein, M. (2003). Measure based regularization. Advances in Neural Information Processing Systems, 15. MIT Press. Carreira-Perpinan, M.A., & Zemel, R.S. (2005). Proximity graphs for clustering and manifold learning. Advances in neural information processing systems, 17. MIT Press. Chapelle, O., Zien, A., & Scholkopf, B. (Eds.). (2006). Semi-supervised learning. MIT Press. Chapelle, O., & Zien, A. (2005). Semi-supervised classification by low density separation. In Proc.Tenth International Workshop on Artificial Intelligence and Statistics.
Chung, F. (1997). Spectral graph theory. American Mathematical Society. Duda, R., Stork, D., & Hart, P. (2000). Pattern classification. JOHN WILEY, 2nd edition. Feng, H., Shi, R., & Chua, T.-S. (2004). A bootstrapping framework for annotating and retrieving WWW images. In Proc. ACM Multimedia. Fung, G., & Mangasarian, O. (1999). Semi-supervised support vector machines for unlabeled data classification. Technical Report 99-05. Data Mining Institute, University of Wisconsin Madison. He, J., Li, M., Zhang, H.-J., Tong, H., & Zhang, C. (2004). Manifold-ranking based image retrieval. Proc. ACM Multimedia. He, J., Li, M., Zhang, H.-J., Tong, H., & Zhang, C. (2006). Generalized manifold-ranking based image retrieval. IEEE Transaction on Image Processing, 15(10). He, J., Zhang, C., Zhao, N., & Tong, H. (2005). Boosting web image search by co-ranking. In Proc. International Conference on Acoustics, Speech, and Signal Processing. Hoi, S.C.H., & Lyu, M.R. (2005). A semi-supervised active learning framework for image retrieval. In Proc. IEEE Conference on Computer Vision and Pattern Recognition. Hsu, W.H., Kennedy, L. S., & Chang, S.-F. (2007). Video search reranking through random walk over document-level context graph. In Proc. ACM Multimedia. Joachims, T. (1999). Transductive inference for text classification using support vector machines. In Proc. International Conference on Machine Learning Li, W., & Sun, M. (2006). Semi-supervised learning for image annotation based on conditional random fields. In Proc. International Conference on Image and Video Retrieval. Liu, J., Li, M., Ma, W.-Y., Liu, Q., & Lu, H. (2006). An adaptive graph model for automatic image annotation. In Proc. ACM International Workshop on Multimedia Information Retrieval. Liu, J., Lai, W., Hua, X.-S., Huang, Y., & Li, S. (2007). Video search reranking via multi-graph propagation. In Proc. ACM Multimedia. Naphade, M., Smith, J.R., Tesic, J., Chang, S.-F., Hsu, W., Kennedy, L., Hauptmann, A. G., & Curtis, J. (2006). Large-scale concept ontology for multimedia. IEEE Multimedia, 16(3). Perona, P., & Malik, J. (1990). Scale-space and edge detection using anisotropic diffusion. IEEE Transaction on Pattern Analysis and Machine Intelligence, 12(7). Rahmani, R., & Goldman, S. (2006). Multiple-instance semi-supervised learning. In Proc. International Conference on Machine Learning. Rui, X., Yuan, P., & Yu, N. (2006). Image annotations based on semi-supervised clustering with semantic soft constraints. In Proc. Pacific-Rim Conference on Multimedia. Sapiro, G. (2001). Geometric partial differential equation and image analysis. Cambridge University Press.
Seeger, M. (2001). Learning with labeled and unlabeled data. Technical Report. University of Edinburgh. Smith, J.R., & Schirling, P. (2006). Metadata standards roundup. IEEE Multimedia, 13(2). Snoek, C.G.M., Worring, M., Gemert, J.C.V., Geusebroek, J.-M., & Smeulders, A.W.M. (2006). The challenge problem for automated detection of 101 semantic concepts in multimedia. In Proc. ACM Multimedia. Song, Y., Hua, X.-S., Dai, L., & Wang, M. (2005). Semi-automatic video annotation based on active learning with multiple complementary predictors. In Proc. ACM International Workshop on Multimedia Information Retrieval. Song, Y., Hua, X.-S., Qi, G.-J., Dai, L., Wang, M., & Zhang, H.-J. (2006). Efficient semantic annotation method for indexing large personal video database. In Proc. ACM International Workshop on Multimedia Information Retrieval. Stoer, M., & Wagner, F. (1994). A simple min cut algorithm. Algorithms--ESA '94, LNCS 855, 141147. Tang, J., Hua, X.-S., Qi, G.-J., Mei, T., & Wu, X. (2007). Anisotropic manifold ranking for video annotation. In Proc. IEEE International Conference on Multimedia & Expo. Tang, J., Hua, X.-S., Mei, T., Qi, G.-J., & Wu, X. (2007). Video annotation based on temporally consistent Gaussian random field. Electronics Letters, 43(8). Tang, J., Hua, X.-S., Qi, G.-J., Wang, M., Mei, T., & Wu, X. (2007). Structure-sensitive manifold ranking for video concept detection. In Proc. ACM Multimedia. Tang, J., Hua, X.-S., Qi, G.-J., & Wu, X. (2007). Typicality ranking via semi-supervised multiple-instance learning. In Proc. ACM Multimedia. Tang, J., Hua, X.-S., Qi, G.-J., Song, Y., & Wu, X. (2008). Video annotation based on kernel linear neighborhood propagation. IEEE Transactions on Multimedia, 10(4). Tong, H., He, J., Li, M., Zhang, C., & Ma, W.-Y. (2005). Graph based multi-modality learning. In Proc. ACM Multimedia. Vapnik, V. (1998). Statistical learning theory. Springer. Wang, C., Jing, F., Zhang, L., & Zhang, H.-J. (2006). Image annotation refinement using random walk with restarts. In Proc. ACM Multimedia. Wang, F., Zhang, C., Shen, H. C., & Wang, J. (2006). Semi-supervised classification using linear neighborhood propagation. In Proc. IEEE Conference on Computer Vision and Pattern Recognition. Wang, F., & Zhang, C. (2006). Label propagation through linear neighborhoods. In Proc. 23th International Conference on Machine Learning. Wang, M., Hua, X.-S., Song, Y., Yuan, X., Li, S., & Zhang, H.-J. (2006). Automatic video annotation by semi-supervised learning with kernel density estimation. In Proc. ACM Multimedia.
Wang, M., Hua, X.-S., Song, Y., Lai, W., Dai, L., & Wang, R. (2007). An efficient automatic video shot size annotation scheme. In Proc. International Multimedia Modeling Conference. Wang, M., Hua, X.-S., Yuan, X., Song, Y., & Dai, L. (2007). Optimizing multi-graph learning: towards a unified video annotation acheme. In Proc. ACM Multimedia. Wu, M., & Jing, R. (2006). A graph-based framework for relation propagation and its application to multilabel learning. In Proc. ACM Conference on Research & Development on Information Retrieval. Wu, M., & Scholkopf, B. (2007). Transductive classification via local learning regularization. In Proc. 11th International Conference on Artificial Intelligence and Statistics, (pp. 624-631). Yan, R., & Naphade, M. (2005). Semi-supervised cross feature learning for semantic concept detection in videos. In Proc. IEEE Conference on Computer Vision and Pattern Recognition. Yanagawa, A., Chang, S.-F., Kennedy, L., & Hsu, W. (2007). Columbia University's baseline detectors for 374 LSCOM semantic visual concepts. Columbia University ADVENT Technical Report #222-20068. Yarowsky, D. (1995). Unsupervised word sense disambiguation rivaling supervised methods. In Proceedings of the 33rd Annual Meeting of the Association for Computational Linguistics. Yuan, X., Hua, X.-S., Wang, M., & Wu, X. (2006). Manifold-ranking based video concept detection on large database and feature pool. In Proc. ACM Multimedia. Zhou, Z.-H., Chen, K.-J., & Dai, H.-B. (2006). Enhancing relevance feedback in image retrieval using unlabeled data. ACM Transactions on Information Systems, 24(2), 219–244. Zhou, D., Bousquet, O., Lal, T.N., Weston, J., & Scholkopf, B. (2003). Learning with local and global consistency. Advances in Neural Information Processing Systems, 15. MIT Press. Zhou, D., Scholkopf, B., & Hofmann, T. (2005). Semi-supervised learning on directed graphs. Advances in Neural Information Processing Systems, 17. MIT Press. Zhou, D., Huang, J., & Scholkopf, B. (2005). Learning from labeled and unlabeled data on a directed graph. In Proc. 22nd International Conference on Machine Learning. Zhu, X., Ghahramani, Z., & Lafferty, J. (2003). Semi-supervised learning using Gaussian fields and harmonic function. Proc. International Conference on Machine Learning. Zhu, X. (2005a). Semi-supervised learning literature survey. Technical Report 1530, Department of Computer Sciences, University of Wisconsin, Madison. Zhu, X. (2005b). Semi-supervised learning with graphs. PhD Thesis, CMULTI-05-192.
Content-Based Video Semantic Analysis Shuqiang Jiang Chinese Academy of Sciences, China Yonghong Tian Peking University, China Qingming Huang Graduate University of Chinese Academy of Sciences, China Tiejun Huang Peking University, China Wen Gao Peking University, China
ABSTRACT With the explosive growth in the amount of video data and rapid advance in computing power, extensive research efforts have been devoted to content-based video analysis. In this chapter, the authors will give a broad discussion on this research area by covering different topics such as video structure analysis, object detection and tracking, event detection, visual attention analysis, and so forth. In the meantime, different video representation and indexing models are also presented.
INTRODUCTION In recent years, digital equipments such as digital camera and digital video camera have become very popular, which makes the digital video data growing in an exponential speed. Information in the Internet has also changed from one-fold text information to multimedia information such as graphs, image, music Copyright © 2009, IGI Global, distributing in print or electronic forms without written permission of IGI Global is prohibited.
Content-Based Video Semantic Analysis
and particularly video. Broadly speaking, the Internet can be regarded as a database which contains a big size of video data. Digital video is playing more and more important role in human's life in terms of working, education, entertainment, and so on. The increasing amount of digital video brings up a lot of problems which are never met before. For example, it is time-consuming and labor-intensive to manually manage the large scale, rapidly increasing digital video collection. As a result, it is more difficult for users to find the desired video contents from the large scale video database. Therefore it is crucial in nowadays to automatically understand digital video data and make them convenient for people to access. During the recent two decades, video content analysis and understanding has been an active research area, and a whole range of models and methods have been proposed. One of the most famous achievements is the Content Based Video Retrieval (CBVR). CBVR firstly extracts the low-level visual and/or acoustical features of videos, and then finds videos that have low-level features similar to those of the query example. As this technology needs little human intervention and does not take human's perception into consideration, it is not satisfactory in most real applications. In fact, human often judge the similarity of videos according to the objects they describe, the events they contain, and the feeling they express, rather than the video features such as color, shot, motion intensity. This difference between the low-level features and high level understanding is referred to as semantic gap, which in fact has become an important barrier of natural communication between computer and human. In order to bridge the semantic gap, lots of methods have been proposed, including relevance feedback, object recognition, event detection, and so on. In this chapter, we will outline the structure for content-based video semantic analysis and discuss some related technologies from low-level content analysis such as feature extraction, structure analysis, object detection and tracking, to high-level semantic analysis such as event detection, attention analysis and video mining.
Lotraction and Representation Generally speaking, low-level feature representation includes visual feature extraction, description, dimension reduction and indexing. After the video is segmented and key frames are chosen, low-level image features can be extracted from these key frames. Low-level visual features such as color, texture, edge and shapes can be extracted from the key frame set in video and represented as feature descriptors. After post-processing on the feature descriptors such as dimension reduction, they can be stored in the database using indexing models for future queries. There are two categories of visual features: global features that are extracted from a whole image, and local or regional features that describe the chosen patches of a given image.
Global Features In the sense of human perception, an image can be characterized from various aspects, consequently leading to different image feature extraction methods. Typically, three categories of these methods are commonly used: color-based, texture-based and shape-based (especially edge-based) methods. Color: Color is the most widely used feature for its close relationship with the objects and the scene in the image and its little dependence with the image scale, view direction and view point. Due to these
properties, color-based methods are often able to generate robust representations of an image. In general, three main problems should be taken carefully into account in color-based methods: (a) Which is the proper color space according to the application; (b) how to quantitatively quantize the color feature into a vector for future indexing and similarity comparison; (c) how to properly measure the vector distance in the color feature vector space. Typical color feature representations include color histogram, color moments (Stricker & Orengo, 1995), color set (Smith & Chang, 1995), color correlogram (Vetterli, 1995), and color coherence vector (Pass & Zabih, 1996). Among them, color histogram is the oft-used one which describes the relative quantity of each color in the image while overlooking the spatial relationships between different colors. Texture: Texture is common vision phenomenon, which is, as stated by David A. Forsyth, "easy to recognize and hard to define" (Forsyth & Ponce, 2003). Smith and Chang (2003) defined texture as a color and illumination independent visual feature which reflects the homogeneity of an image. Some others say that texture is a densely and evenly arranged pattern of visible elements and a texture element is a repeated uniform intensity region of simple shape. Clearly, texture is an important part of an image and many methods are developed to extract texture features (Smith & Chang, 2003). Shape: Shape is another important visual feature in images. However, since shape features are extracted by using image segmentation which is still a hard problem with little progress in achieving accurate and robust results, shaped-based methods are usually applied to salient objects in images (Aslandogan &Yu, 1999). There are many shape feature representations, such as Fourier shape descriptors, moment invariants, finite element method, and turning function and wavelet descriptor.
Local Features The local feature extraction process can be roughly divided in two phases: region detection and region description. The feature detector is firstly used to detect a stable and characteristic region, and then the feature descriptor is used to describe the detected region in a precise, concise and robust way. In the two phases, one primary tackling issue is how to carry out the corresponding mission in a scale, rotation, illumination and noise invariant way. Detectors: In order to obtain a robust detection result, the feature detectors should take into consideration the factors such as the affine transformation, photometric transformation and scale differences. Harris proposed a corner detector which is well known as Harris Detector (Harris &Stephens, 1988). Harris corner is not scale and affine invariant, but it contributes as the bases for many detectors. For example, Mikolajczyk and Schmid (2004) extended Harris detector to a scale and affine invariant detector. Lowe (2004) proposed a SIFT (Scale Invariant Feature Transform) detector which detects extremum in the DoG (Difference of Gaussian) space. Descriptors: The raw pixel information of the detected region is sensitive to the illumination change, rotation and noise, thus is not a robust feature description. SIFT (Lowe, 2004) descriptor rotates the region according to the main intensity direction of the region, divides the region into several equal area sub-regions and uses the orientation histograms of all the sub-regions as the feature description. Ke and Sukthankar (2004) applied principal component analysis (PCA) on the intensity gradient map of a 41*41 patch on the detected region to get a lower dimensional descriptor. Other techniques such 213
as Gabor filters and wavelets, steerable filters and generalized moment invariants are also used in the regional feature description.
VidEO STRUCTURE ANALYSIS To efficient access of video data, parsing video structures at different granularities is the first step. Generally speaking, videos can be hierarchically represented by five levels: key frame, shot, scene, story unit and video, as illustrated in Fig.1. The following are some of the definitions, and detailed discuss them thereafter: •
Video frame: The basic components of video stream. Video data is made up of continuous frames. Every frame is an independent image. Different video standard may have different frame rate. In PAL standard, the frame rate is 25 per second, while in NTSC standard, it is 30 per second. Key frame: The most representative frames in the video stream. Video shot: An unbroken sequence of frames recorded from a single camera, which is the minimal physical unit of video structure analysis. A video shot normally represents a continuous action in both space and time. Video scene: A collection of shots whose content are similar. It is a median representation of video content and contains some semantic information. With the help of video scene, we can analyze higher-level video content better. Story units: Story unit generally represents a whole event at a higher level. It is made up of several continuous shots which may belong to different scenes.
Key Frame Extraction In order to understand the content of a video, there need to first decompose the video into a set of scenes, and then extract some key frames that most represent the content of each scene. Compared with the Figure 1. A hierarchical video representation
original video data, key frames can represent the scene more concise and are more adequate for video indexing, browsing and retrieval. There are many works on the area of key frame extraction. We can briefly categorize the related work into 4 classes: sampling-based, shot-based, segment-based, and others (Li, Zhang, & Tretter, 2001). The following are detailed introductions. Since shot is a video segment with a continuous period on certain scenario, shot based key frame extraction methods are natural. And it is the dominating way among the current solutions. A very simple method is to extract the first frame of every shot as the key frame. However, more sophisticated and powerful methods have been developed. In work (Zhang, Wu, Zhong, &Smoliar, 1997), the first frame is always chosen as the key frame. Then the color histogram distance between the subsequent frames and the latest key frame is computed. If the distance exceeds a threshold, a new key frame will be chosen. In (Wolf, 1996), the frames at the local minima of motion features are selected as key frames. Mosaic based approach can be employed to generate a synthesized panoramic image that can represent the entire content in an intuitive manner. This method is especially useful for the camera tilting/panning sequences. A video segment is a higher-level video unit, which can be a scene, an event, or even the entire sequence. In (Uchihashi, Foote, Girgensohn, & Boreczky, 1999), a video is segmented into some segments. The segments may be clusters clustered using some distance metrics. Then the centers of clusters are extracted as key frames. Other works (Campisi, Longari, & Neri, 1999), (Kim & Hwang, 2000) used techniques, such as wavelet transformation, face detection, etc. to fulfill the task. The first work (Campisi, Longari, & Neri, 1999) proposed a progressive multi-resolution method based on wavelet transformation to extract key frames. The draw back of this method is that the computation complexity is too high. In the second work, Kim & Hwang (2000) made the frames containing people and faces with a high likeliness to be key frames In conclusion, key frame extraction is an important step to structure the whole video. Many people take great effort to find effective method for key frame extraction. There are many different approaches. But a general metric to evaluate the approaches is not proposed yet, because key frame extraction is a relatively subjective action. Different persons may regard different frames as key frames, and different applications tasks may require different kinds of key frames.
Shot Boundary Detection To better facilitate the user's access requirement such as browsing and retrieval, video content must be structuralized by some schemes. Video shot is the fundamental element above the frame level. Video shot is defined as an unbroken sequence of frames recorded from a single camera (Rui, Huang, & Mehrotra, 1999). It is a physical entity and is delimited by shot boundaries. Therefore, to build up the semantic table-of-content structure for the video, the first step is to locate every shot by finding its beginning and end, what we call the task shot boundary detection (SBD for short) (TRECVID website, 2008). The video creators may use a variety of editing types to transfer from one shot to another. 99% of them will fall into one of the following three types: (1) hard cut, (2) fade, and (3) dissolve. Related works on this topic will mainly focus on these three categories. The simplest way to find the hard cut is measuring the pair-wise difference of consecutive frames is to compute the mean absolute change of intensity. A modification of this technique is only counting
the pixels that change considerably from one frame to another (Otsuji, Tonomura, & Ohba, 1991). The problem of this kind of approaches is the sensitivity to the camera motion. To reduce this influence, a 3 × 3 low pass filter is applied to each pixel before the computing of pair-wise difference (Zhang, Kankanhalli, & Smoliar, 1993). Block matching based approach further alleviates the influence of motion. The algorithm in (Shahraray, 1995) computes the discontinuity between one block on frame i with the block in frame i+Δ that matches most with the former within a certain adjacent area, where Δ is the step size. Then the difference between frames can be computed. A popular alternative to the pixel-wise and block-wise approaches is the color histogram difference (Boreczky & Rowe, 1996). Experimental results show that the color histogram difference is very robust in detecting the hard cut, where the only problem lies in the manual setting of the threshold. Zhang, Kankanhalli and Smoliar (1993) proposed a twin threshold color histogram method to detect fades and dissolves. Since histograms ignore spatial changes within a frame, this feature is considerably more insensitive to object motion than pixel-wise comparisons. However, a histogram difference remains sensitive to camera motion, such as panning, tilting, or zooming (Zhang, Kankanhalli, & Smoliar, 1993), which bring false alarms in shot detection for certain video files. To detect more types of shot boundary, the edge change ratio (ECR) is used as a feature in (Zabih, Miller, & Mai, 1995). This is because when shot boundary occurs, there will be new edge appearing at the position far away from the former position, and the former will disappear gradually. According to the method in (Zabih, Miller, & Mai, 1995), hard cuts, fades, dissolves and wipes exhibit a characteristic pattern in the ECR time series. The standard deviation of pixel intensity proposed by (Lienhart, Kuhmünch, & Effelsberg, 1997) detects fade-in and fade-out respectively. The fades are modeled as a linearly increasing or decreasing pattern along the time axis. The algorithms check the slope of pixel intensities by computing a line of regression repeatedly, when the absolute slope is larger than the predefined threshold, a fade-in or fade-out is detected. Arman, Hsu and Chiu (1993) extracted the feature of compressed domain to detect the shot boundary. The differences of DCT coefficients of two frames are computed as the distance measure. Camera zoom/pan is likely effect the detection performance. To reduce the influence, Ueda, Miyatake and Yoshizawa (1991) computed the feature of motion vector by block matching, and detect the camera pan/zoom to precisely find the shot boundary. In this part, we describe the problem of shot boundary detection; discuss some difficulties that may influence the detection performance. The best detection rate of hard cut could reach 90% or so, and the detection rate of gradual transition is not satisfactory, and the false alarm is high compared with hard cut solutions. After all, the SBD problem still need to be investigated since the gradual transition of shot remains a hard problem that has not been solved well until recently.
Scene Analysis and Story Unit Segmentation Scene analysis and story unit segmentation play an important role in video analysis. Usually scene and story unit contains intact events that happen in the video. Users are mostly interest in the important event in the video and are more intend to see a complete story than uncompleted one. And scene and story unit have more exact event boundaries and complete event descriptions than shot. So the scene analysis and story unit are the important for semantic analysis of video content. Many researchers have worked in this area. Lin, et al. (2001) and Huang, et al. (1998) try to develop a method for all types of videos. A technique for shot content representation and similarity measure using 216
sub-shot extraction and representation is presented in (Lin & Zhang, 2001), that paper employs dominant color histograms and spatial structure histograms to measure content variation and to represent sub shots. Huang, Liu and Wang (1998) worked on joint audio-visual segmentation, and employs decision curve formulation mechanism to detect the story boundary. In fact, developing a general method for scene analysis and story unit segment is a very difficult task. In many times, it cannot become true due to the differences in different domains. The more we know about the domain knowledge of the video data, the better techniques we can develop about it. So researchers are mainly focused on specific domain, such as movies (Hanjalic, Lagendijk, & Biemond, 1999) (Sundaram & Chang, 2000), news (Zhu, et al., 2001) (Iurgel, et al., 2001) (Hsu & Chang, 2003), and sports (Kijak, Gravier, & Gros, 2003) (Kijak, Oisel, & Gros, 2003) (Nitta, Babaguchi, & Kitahash, 2002) (Wang, Liu, & Yang, 2005). Hanjalic, et al. (1999) used the visual features to analyze the movie video; it divides the image into blocks and subsequently uses block-matching to compute the similarity between shots. Then it uses the overlapping links method to extracting candidate boundaries. Sundaram, et al. (2000) combined the features of visual and audio, including color histogram, spectral flux, and multi-channel cochlear decomposition and spectral vectors. It uses temporal distance and shot duration measures to compute the similarity between shots. Then memory model (Sundaram & Chang, 2000) is employed to segment scenes. In news video analysis, Zhu, et al. (2001) employed visual and text features for scene analysis. Then it uses rules regarding anchorperson shots and captions as the main mechanism for identifying news topics. Iurgel, et al. (2001) used audio and semantic features to segment the story units; the semantic feature used here is the anchorperson shot. When segmenting the story unit, it employs not only the rules similar to (Zhu, et al., 2001), but also HMM model for boundary detection. The video, text, audio and semantic features is employed in (Hsu & Chang, 2003); the text feature here is the key-phrases in the introduction and conclusion of a news topic. These features are combined together for better segmenting story units. It uses the maximum entropy model (ME) to systematically fuse separate candidate story boundary sets, which are produced by mid-level features of various types. Many researchers have devoted to sports video structure analysis. Statistical models such as the Hidden Markov Models have been used to analysis the structure of the video (Kijak, Gravier, & Gros, 2003) (Kijak, Oisel, & Gros, 2003). Especially in (Kijak, Gravier, & Gros, 2003), E. Kijak, et al. used both the visual and audio features. Nitta, et al. (2002) focused on grasping the story from the speech transcript by using a probabilistic framework based on Bayesian networks. Recently, Wang, et al. (2005) used templates based method to detect repetitive patterns in sport video. In summary, story unit detection is an important and meaningful work in video analysis. However because this task is a high level concept, compared with shot boundary detection, it is a more challenging work. But specific domain knowledge of the video content can be used to develop some specific method on extracting story units.
Object Based Video Understanding and Representation Object Detection Objects detection is the first step in many video analysis and understanding applications, including object tracking, event detection, semantic annotation and retrieval of videos. Recently, video objects detection have received an increasingly attention in the literature. However, this is not an easy task. 217
The main problem in object detection is the amount of variation in visual appearance such as shape, pose, size, color and surrounding environment. An object detector must accommodate all these variations and distinguish the objects from other patterns that may occur in the visual world. Computing efficiency is also an important problem. Fast algorithm should be developed in order for detection in real-time.
Related Works Many approaches have been proposed for object detection in video streams. Generally speaking, these algorithms can be classified as two categories: global appearance-based approaches and componentbased approaches. Global appearance-based approaches take an object as a unit and perform classification on features generated from the entire object. Many statistical learning mechanisms are explored to identify object patterns. For example, Neural networks are used to detect general objects (Garcia & Delakis, 2004); Support Vector Machines (SVM) (Osuna, Freund & Girosi, 1997) and Naive Bayes classifiers (Schneiderman & Kanade, 2000) are used to locate human faces and cars; boosting algorithms (Viola & Jones, 2001) are widely applied to detect objects, such as faces and text. Component-based methods treat an object as a collection of parts. These methods first extract some object components, and then detect objects by using geometric information among object parts. Mohan, et al. (2001) proposed an object detection approach by components. In their approach, a person is represented by components such as head, arms, and legs, and then SVM classifiers are used to detect these components and to decide whether a person is present or not. Naquest and Ullman (2003) used fragments as features and perform object recognition with informative features and linear classification. Agarwal, et al. (2004) extracted a part vocabulary of side-view cars using an interest operator and learned a Sparse Network of Winnows classifier to detect side-view cars. Fergus, et al. (2003) and Leibe, et al. (2004) also used interest operators to extract objects' parts and perform detection by probabilistic representation and recognition on many object classes, such as motorbikes, human faces, airplanes, and cars. In another point of view, object detection problem is typically solved by two stages: candidate generation and candidate verification (Ye, et al., 2005). Candidate generation generates regions of an image that may contain an object, and candidate verification further determines whether the candidate region is an object or not. To generate candidate, "Exhaustive search" and "Selective search" are two commonly used methods. The first one considers all the windows in an image as candidates while "selective" methods consider only the specific regions with specific features. In the verification procedure, supervised learning are normally used method to detect objects, classifiers may include Support Vector Machines (SVMs), Bayesian networks, Neural Networks (NNs), and boosting models. This method may require a large training data set to achieve a reasonably good performance.
A Technical Solution We will introduce a solution of text detection for example. Text detection is a classical problem in the field of object detection. Text in images and video frames carries important information which is a very compact and accurate clue for video summarization and retrieval. Based on the features selected, text detection algorithm can be classified into three kinds: (1) color and connected component based algorithm, (2) video temporal information based algorithm, and (3) edge/texture based algorithm. The first class is based on the analysis of homogeneous color and grayscale
components that belong to characters (Jain &Yu, 1998). The second class is used to detect video captions by considering that the appearance of a caption will bring difference in successive frames (Tang, et al., 2002). Precise shot boundary detection is required before identifying the appearance of a text line. Furthermore, if a text line appears on a shot boundary, temporal information will fail to detect it. Edge and texture features (Wu, et al., 1999) (Kim, et al. 2003) are the most popular features and work well in many cases. It is based on the idea that text region may contain a lot of edges and textures. Most text detection algorithms follow a coarse-to-fine framework. In the coarse-detection section, candidate text regions are detected. In order to locate candidate regions, candidate pixels are first detected. In this step, some filters are usually used, such as Sobel edge filter, Canny edge filter, Harr wavelet filter, etc. Then candidate pixels are selected according to these highlighted pixels. Candidate regions can be located using moving window algorithm. A fixed size window is defined (for example: 10*5) and moves all over the image. Some statistical information is calculated in the box and is used to identify if it is a candidate region. Candidate pixels density information and local dynamic threshold are the most simple and popular way. The size of the window is very important. Different sizes will generate different results. Another way called density-based region growing algorithm, is also useful to locate candidate text regions. This algorithm calculates the density information of every pixel and achieves better result. But it is time consuming. Once the text region is located, it is separated into text lines according to projection information; thus the position of the space can be found. In the fine-detection section, candidate regions are identified by classifiers. In this procedure, two things are the most important: features and classifiers. Based on the review of previous methods, we know that edge and texture features are most frequently used. Text can be regarded as a kind of texture pattern, but the texture properties such as regularity and directionality are weak. It just contains some character strokes that form a text line in a special orientation. Then, only one kind of texture feature is insufficient to model text's texture pattern., A lot of texture feature are usually extracted, including: moments features, histogram features (energy histogram and direction histogram), co-occurrence features, scan line features, etc. Sometime some approaches (such as PCA, forward search algorithm) are used to decrease the feature dimensions. Compared with other classifiers such as neural network and decision tree, SVM is easier to train, needs fewer training samples and has better generalization ability. So it is the most popular classifier in text detection. Adaboost has also been used and be proved to be effective. Training data is also important. The number of the samples is depended on the dimensions of the feature. Whatever, false samples should be more than true samples to get a better interface in the feature space.
Short Summary Object detection is a difficult problem. Although a lot of work has been done, we still have a long way to go. Maybe better features and better machine learning tools can achieve better results in the future. We are also looking forward to new framework which can make a revolutionary progress.
Object Tracking Moving object tracking is a major issue in many video analysis systems. Tracking means to locate the object from frame to frame, in other words, to find out the trajectory of the moving object. It enables several important applications such as: (1) security and surveillance to recognize people, to provide
better sense of security using visual information; (2) medical therapy – to improve the quality of life for physical therapy patients and disabled people; (3) retail space instrumentation – to analyze shopping behavior of customers, to enhance building and environment design; (4) video abstraction – to obtain automatic annotation of videos, to generate object-based summaries; (5) traffic management – to analyze flow, to detect accidents; (6) sports strategy analysis- to analyze the trajectory of the players; (7) human-computer interaction – to recognize gesture, track eye gaze for data input to computers. In its simplest form, tracking can be defined as the problem of estimating the trajectory of an object in the image plane as it moves around a scene. In other words, a tracker assigns consistent labels to the tracked objects in different frames of a video. Additionally, depending on the tracking domain, a tracker can also provide object-centric information, such as orientation, area, or shape of an object.
Related Works Object tracking methods can be divided into three categories: kernel tracking, silhouette tracking and point tracking (Yilmaz, et al., 2006). Next, a concise introduction and some representative work for each method will be provided. Kernel tracking. Kernel can be regarded as the shape and appearance of the object. So kernel can be a template of a geometry shape or an ellipse with feature histograms. Apparently, for object's feature, every pixel's contribution is not equal. So we assign different weights to pixels according a function, which is named as "kernel". For example, an isotropic kernel
k ( x) =
1- | x |, x ≤ 1   0, | x |> 1
assigns larger weight to pixels nearer to the center and the object's feature histogram can be computed as 2
q (u ) = C
∑ k( x
) [b ( xi ) - u ], u = 1, .., m ,
where n is the number of pixels of the object, b(xi) is function that associates to the pixel at location xi the index b(xi) of its bin, m is the number of bins and C is the normalization constant, which is derived by imposing constraint m
∑ q (u ) = 1 . u =1
Next, in the current frame, we want to find the region that most similar to the object model or evaluate the object's motion. The popular methods are mean-shift, KLT and eigentracking. Point Tracking. Objects can be represented by a number of points. In this case, tracking can be performed by finding the correspondence points. So point detection algorithms are needed. Then we should match these detected points with points that used to represent the object in the previous frame. Point correspondence methods can be divided into two classes: deterministic (e.g. Hungarian algorithm, Greedy
algorithm) and statistical methods (e.g. Kalman filter, Particle filter (Isard &Blake, 1998)). The statistical methods explicitly take the object measurement and consider uncertainties in correspondence while deterministic methods use qualitative motion heuristics to constrain the correspondence problem. Silhouette tracking. Some objects such as hand, body and shoulder have complex shapes and silhouette representations provide an explicit shape descriptor for those objects. Silhouettes have many different representations and the most popular representation is binary (0/1) mask, which mask the object region with 1 and non-object region with 0. Silhouette tracker is designed to find the object region in current frame according to the model generated in the previous frame. It uses the information (e.g. appearance density, shape model) of the object region and is performed through either shape matching or contour evolution. The goal of shape matching is to find out the object silhouette in the current frame. On the other hand, contour tracking want to get object's new position by evolving an initial contour. The main evolving methods are state space (e.g. HMM) and direct minimization of some energy function (e.g. gradient descent). Apparently, silhouette tracking can deal with various complex shapes.
A Technical Solution In this part, we will introduce a well know tracking method: Particle filtering (Isard &Blake, 1998). Given an object, we want to use some points to represent it. Then how to select points or samples? Usually, different sub-region of the object region plays different roles. That is, some sub-regions are very important while some are not. For those important sub-regions, more samples (particle) should be selected. Those particles are not equal. Some are more important while some are less. How to get their weights or posterior probabilities? i In order to develop the details of the algorithm, let xt represents the state (e.g. position) of the i'th i i N i particle at time t, and let wt denotes the weight of the particle at time t. So { x0: k , w0: k }i =1 denote a random measure that characterizes the posterior pdf p ( x0: k | z1: k ), where x0:k is the total states of all the particles up to time k. The weights are normalized such that
∑w i
= 1.
Then, the posterior density at k can be approximated as: Ns
p ( x0: k | z1: k ) ≈
∑w
( x0: k - x0: k )
How to get the weights? If the samples are drawn from an importance density q ( x0: k | z1: k ), then i
wk ∝
p ( x0: k | z1: k )
q ( x0: k | z1: k ) i
p ( x0:k | z1:k ) =
p ( x0:k , z1:k ) p ( z1:k )
p ( z k | x0: k | z1: k -1 ) p ( x0: k | z1: k -1 ) p ( z1: k -1 ) p ( z k | z1: k -1 ) p ( z1: k -1 )
p ( z k | x0: k | z1: k -1 ) p ( x0: k | z1: k -1 ) p ( z k | z1: k -1 )
p ( z k | x0: k | z1: k -1 ) p ( xk | x0: k -1 | z1: k -1 )
p ( z k | z1: k -1 )
× p ( x0: k -1 | z1: k -1 )
p ( z k | xk ) p ( xk | xk -1 )
p ( x0:k -1 | z1:k -1 ) ∝ p ( z k | xk ) p ( xk | xk -1 ) p ( x0:k -1 | z1:k -1 )
p ( z k | z1:k -1 )
So i
p ( z k | x k ) p ( x k | x k -1 ) i
q ( xk | x0: k -1 , z1: k )
wk -1 .
If importance density satisfies the equation:
q ( xk | x0: k -1 , z1: k ) = q ( xk | xk -1 , z k ), only a filtered estimate of p ( xk | z1: k ) is required at each time step. After a few iterations, it is possible that few particles have too large weights and others have negligible weights. This is the degeneracy phenomenon which is an undesirable effect in particle. One method is selecting a very large number of particles, but this is impractical. Good choice of importance density and resampling are two effective solutions. Particle filtering approximates the density directly as a finite number of samples. A number of different types of particle filter exist, and some have been shown to outperform others when used for particular applications. However, when designing a particle filter for a particular application, it is the choice of importance density that is critical.
Short Summary In this part, we first give a simple introduction to object tracking. Then we briefly introduce some main tracking methods. Last, a popular tracking method, particle filter, is introduced as a technical solution. There are many other solutions such as mean shift, condensation, eigentracking and so on.
Object Rpresentation and Indexing Object representation and indexing is the way to describe objects, by which objects can be easily retrieved, classified and detected in image or video dataset. So this method should be distinctive and efficiently computed. Imagine that you are standing beside a car (Fig.2) and you want to describe it to an auto-navigation robot, so that it can recognize cars in real-time. Thus, how to describe or represent cars? Should we only use color, shape or other features? Generally, objects can be represented by their shapes and appearances. The object shape representations mainly include points, primitive geometric shapes, silhouette and contour, articulated shape and skeletal models. Common appearance representations in the context of object can be divided into: (1) probability densities of object appearance, (2) templates, (3) active appearance models, (3) multi-view appearance models, and (4) integral image, etc.
Indexing is something that serves to guide, point out, or otherwise facilitate reference, especially in video dataset. According to the type of data being stored, video indexes can be classified into the following categories that have been discussed before: key frame, video shots, video story and video summary. Object representations are the basis for high level stage process, such as detection, classification and tracking. So the right representation plays a critical role in multi-media computing. Some representative method will be described in the following according to examples illustrated in Fig.3: •
Points-based method: The object is represented by the centroid (Fig.3 (a), in tracking) or by a set of points-based patch, such as SIFT (Lowe, 2004), MSEA (Matas, et al. 2002) (Fig. 3(b), in classification context). In general, the point representation is suitable for representing and tracking objects that occupy small regions. Primitive geometric shapes: Object shape is represented by a rectangle, ellipse (Fig.3 (c), (d)), etc. Object motion for such representations is usually modeled by translation, affine, or projective transformation. Though primitive geometric shapes are more suitable for representing simple rigid objects, they are also used for non-rigid ones.
Figure 2. Different cars in different view points
Figure 3. Object representations. (a)point or centroid, (b) multi points, (c) rectangular patch, (d) elliptical patch, (e) part-based multiple patches, (f) object skeleton, (g) complete object contour, (h) control points on object contour, (i) object silhouette.
Object silhouette and contour: Contour representation defines the boundary of an object (Fig. 3 (g), (h)). The region inside the contour is called the silhouette of the object (Fig.3 (i)). Silhouette and contour representations are suitable for indexing complex non-rigid shapes, such as human motion and hand motion analysis. The silhouette or contour can be described as shape context (Thayananthan, et al., 2003), or histogram of gradient (Xu & Li, 2002). Articulated shape and models: Articulated objects are composed of body parts that are held together with joints. For example, the human body is an articulated object with torso, legs, hands, head, and feet connected by joints. The relationship between the parts is governed by kinematics motion model, ex. joint angle, etc. In order to represent an articulated object, one can model the constituent parts using cylinders or ellipse as show in Fig.3 (e). Skeletal models: Object skeleton can be extracted by applying medial axis transformation to the object silhouette. This model is commonly used as a shape representation for objects recognition. Skeleton representation can be used to model both articulated and rigid objects (Fig.3 (f)). Chamfer matching (Thayananthan, et al., 2003) is also used for comparing the skeletal model. Probability densities of object appearance: The probability density estimates of the object appearance can either be parametric, such as Gaussian and mixture of Gaussians, or nonparametric, such as Parzen windows and histograms. The probability densities of object appearance features (color, texture) can be computed from the image region. Templates: Templates are formed using simple geometric shapes or silhouettes. An advantage of a template is that it carries both spatial and appearance information. Templates, however, only encode the object appearance generated from a single view. Thus, they are only suitable for view invariant objects whose poses do not vary considerably during the course (Comaniciu & Meer, 2002). Integral image: An integral image I is an intermediate representation for the image and contains the sum of gray scale pixel values of image N with height y and width x, i.e., x
I ( x, y ) = ∑∑ N ( x′, y ′). x′ = 0 y ′ = 0
The integral image is computed recursively, by the formulas:
I ( x, y ) = I ( x, y - 1) + I ( x - 1, y ) + N ( x, y ) - I ( x - 1, y - 1) with
I ( -1, y ) = I ( x, -1) = I ( -1, -1) = 0, therefore it requires only one scan over the input data. This intermediate representation I(x,y) allows the computation of a rectangle feature value at (x,y) with height and width (h,w) using four references
F ( x, y , h, w) = I ( x, y ) + I ( x + w, y + h) - I ( x, y + h) - I ( x + w, y ). This representation is widely used in object detection for fast computing object statistic features in (Viola & Jones, 2000).
Generally speaking, there is strong relationship between the object representations and the application algorithms, such as detection, tracking et al, and object representations are usually chosen according to the application domain. Video indexing is a category representation for fast retrieval of the video information.
Hig LEVEL VIDEO SEMANTIC ANAlYSIS Aandanding In order to understand and analysis the visual data, some high level semantic information should be extracted such event information, perceptual attention information and mining information. In the following, we will have a detailed introduction on these areas.
Event Detection Video events are normally defined as the interesting or important events which capture user attentions. For example, a soccer goal event is defined as the ball passing over the goal line without touching the goal posts and the crossbar. With the advances of storage capabilities, computing power and multimedia technology, the research on semantic event detection become more and more active in recent years, such as video surveillance, sports highlight detection, TV/Movie abstraction and home video retrieval etc. Through event detection, consumers can retrieve specific video segments quickly from the long videos and save much time in browsing. Moreover, event detection makes it possible to deliver the video clips into mobile devices or webs. However, semantic event detection is still a challenging problem due to the large semantic gap and the difficulty of modeling temporal and multimodality characteristics of video streams. In visual surveillance for example, after successfully tracking the moving objects from one frame to another in an image sequence, the problem of understanding object behaviors from image sequences appears naturally. Behavior understanding involves the analysis and recognition of motion patterns, and the production of high-level description of actions and interactions. In fact, event detection is based on the correct recognition of human behavior and object movement. Events scratched from the scene under surveillance could be divided into two large categories: normal events and abnormal events. Naturally, abnormal behavior or event detection after behavior understanding and description of behaviors would be more important, for it's wildly utilization.
Related Works In the past few years, significant research has been devoted to the video event detection. Haering, et al. (2000) proposed a three-level video event detection method and apply it to animal-hunt detection in wildlife documentaries. Three levels are feature level; neural network based classifier level; and shot descriptor level respectively. A three-level video event detection method is also applied in (Wang, et al. 2006). The authors employed Conditional Random Fields (CRFs) to fuse temporal multi-modality cues for event detection. Ekin, et al. (2003) combined object-based features and cinematic features to develop an event detection algorithm and a summarization system for soccer videos. Shot are classified into four view (shot) types according to the dominant color features. Then high-level events such as goal, referee and penalty box are detected based on object-based features from the specific views. Bai,
et al. (2007) also presented an event detection model for sports video; this model is based on perception concepts and finite state machines (PC-FSMs). PC-FSMs are described formally in terms of state graphs, then a graph matching method is used to discover semantic event automatically. Xie, Shyu, and Chen (2007) proposed a video event detection method which utilizes both the distance-based and rule-based data mining techniques; they try to provide a domain independent solution for rare event detection in video streams. An important application in event detection is visual surveillance, where abnormal detection and behavior prediction are significant in practice. In applications of visual surveillance, not only should visual surveillance systems detect abnomalies such as traffic accidents and car theft etc, according to requirements of different functions, but also predicts what will happen according to the current situation and raise an alarm for a predicted abnormal behavior. Implementations are usually based on one or other of the following two methods. 1.
Probability reasoning and prior rules combined methods. A behavior with small probability or against the prior rules would be regarded as an anomaly (Owens & Hunter, 2000) (Lou, Liu, Tan, & Hu, 2002). Behavior-pattern-based methods. We can predict an object behavior by matching the observed sub-behavior of the object with the learned patterns (Johnson &Hogg, 1996) (Sumpter & Bulpitt, 2000). Generally, patterns of behaviors in a scene can be constructed by supervised or unsupervised learning of each object's velocities and trajectories, etc.
A Technical Solution In this part, we introduce a method of exciting event detections in broadcast soccer video as an example (Ye, et al., 2005). Framework of the proposed method is described in Fig.4. First, for each of the video frame, the playfield area is classified based on low-level features, and frames are hierarchically classified into defined views. Finally, view label, camera motion and shot boundary descriptions together with temporal relationship are fed into an SVM classifier to identify events. In our implementation, "shoot on goal", "placed kick", "break by offence" are selected for experiments. To segment playfield, training models are automatically built by Gaussian Mixture Models (GMMs) (Jiang, et al., 2004). The model is updated online by an incremental procedure. Then three kinds of midlevel descriptions are extracted to represent the soccer video content. For each of the soccer frame, we can semantically assign a view label to them by a hierarchical classification procedure. The views are:
Figure 4. The framework of exciting event detection in soccer video Incremental learning Soccer Video Playfield Detection
View Classification View descriptor& Camera motion& Shot descriptors
Exciting Event Detection
"goal mouth view", "corner view", "middle field view", "player close-up view" and "out-field view". For each of the level we extract low-level features of playfield ratio, projection profile of non playfield, and shape for classification. SVM classifiers are employed to perform the classification task on extracted features. An incremental scheme is adopted to improve the extensibility.
Short Summary We have reviewed the state-of-the-art of event detection in video content analysis and visual surveillance. Although a large amount of work has been done in this scope, many issues are still open and deserve further research.
Visual Attention in Video Analysis Visual attention is one of the most important features of Human Visual System (HVS). It can filter the signals received by the HVS and select the important ones to be processed. Visual attention has two mechanisms, bottom-up and top-down, correspond to stimulus-driven and object-driven respectively. The bottom-up mechanism is closely related to human sensory system, which is sensible to the contrast including both global effect and contextual difference. The top-down mechanism is closely related to human brain, which is a much complex mechanism. Computational attention model enables computers to understand images and videos in the manner of HVS. It has been a hot research point for years and has been successfully applied in many fields including image compression, video summarization, image and video browsing on small displays and so on. Limited by the capacity of human information processing system, only a small part of the information received by sensory system should have the chance to be processed by brain. The ability of human visual system to concentrate mental effort on sensory or mental events is referred as attention. Before being processed by brain the information should be filtered twice. The first one happens before pattern recognition, which is a stimulus driven process, referred as bottom-up attention. The second one happens after pattern recognition, which is a task driven process, referred as top-down mechanism. Computational visual attention can also be classified into the above two types similarly. The task of attention analysis is to detect the Region of Interest of images and the frames of interest of videos (Solso, et al. 2003). Bottom-up attention analysis: Saliency map based attention model is the earliest and the most popular approach for bottom-up attention analysis. Till now all the proposed methods are saliency map based. The method is different in the following four aspects: the choice of the perceptive unit, the mechanism to compute saliency, the feature combination method, and extraction of the region of interest through saliency map. Saliency map is the representation of the saliency of each perceptive unit. Here perceptive unit is defined as a set of pixels perceived as a union in attention analysis. A perceptive unit can be a pixel, a block, or a region. A region can be of any size, any scale, and contain much more perceptive information compared with a pixel or a block (Liu, et al., 2007). For the center-surround structure of receptive field of human eye, a perceptive unit with large contrast will attract more attention. So contrast is widely used in attention analysis. Itti, et al. (1998) proposed a "center-surround" method, in which the "center" is the blocks at low level of Gaussian pyramid, and "surround" is the corresponding ones at high levels. The distance between different scales is used to generate saliency maps. Ma and Zhang (2003) used the contrast between an image block and its neigh-
borhood to represent its saliency. Global contrast and statistical information (Zhai & Shah, 2006) are also used to obtain the saliency map. In attention analysis, all of the features of the image can be used, including color, texture, intensity, orientation and so on. Each of the features will generate a saliency map and the fusion mechanism may affect the result heavily. Feature fusion can be both linear and nonlinear. Linear fusion is much easier by calculating the weighted average of the saliency maps. The weights of each feature can be predefined empirically or by machine learning (Itti & Koch, 1999). While nonlinear fusion is more complex but is more suitable from the point of view of perception (Hua & Zhang, 2004). The final work in attention analysis is to extract the region of interest from the saliency map. Itti, et al. (1998) used a winner-take-all mechanism; Ma and Zhang (2003) used fuzzy growing method to extract the attentive region. All the above is about attention analysis of images. For videos, another important feature should be considered: motion. This is because people always draw attention to the objects with distinct motion. In addition, there is an important property of human perception that it is affected strongly by knowledge context. According to this property, some object may catch more attention such as human face, text and so on. So face detection and text detection are often used in visual attention analysis (Ma, et al., 2005). Another kind of region which would draw people's great attention in an image is human face. So if an attention unit belongs to a part of human face, most work will give a high saliency value to it. Some work proposed a model based on camera motion. It considers the motion of camera will lead the audience to pay attention to the certain area. For example, if the camera pans left, people will always pay attention to the left part of the video frame; if the camera zooms in, people will always pay attention to the center part of the video frame. Top-down attention analysis: The bottom-up method uses only low level features and does not need any prior knowledge. So it is simple and could satisfy most general applications. But the perception of human is always based on some certain prior knowledge. For example people always pay some attention to the objects and events which are meaningful. Top-down model is adequate for evaluating the attention caused by people's prior knowledge. Some work learns to pair the low-level signatures from a series of video clips with the corresponding eye positions (Peters & Itti, 2007). Machine learning methods are also used in top-down attention model to detect specific objects and events (Peters & Itti, 2007) (Navalpakkam & Itti, 2006). In summary, visual attention model is an effective way to extract the essential content from the redundant video data. Now people are developing many new methods to extract the attentive parts in pictures and videos.
Video Mining Data mining is the process of finding useful patterns or extracting previously unknown knowledge from a massive set of data (Oh, Lee, & Hwang, 2005). Different from textual information which has been studied for a long time, video contents have special characteristics: they are continuous sequences with temporal relations among them, and each video segment normally contains abundant information itself. Mining useful statistical information from the whole video content to help users have a better understanding of it is a relatively difficult task and has not been fully explored yet. Compared with object-, structure- or event-based video semantic information as discussed before, statistical mining information is hard to be acquired by human labeling.
Due to the semantic gap between human perception and computer-centered low-level features, many video understanding techniques such as shot boundary detection and event extraction remain open problems. Thus investigation on video mining is at its early stage as mining solutions normally need the extracted semantic information. In fact, the current status of video mining is still at the pre-processing stage, such as video clustering (Tavanapong & Zhou, 2004). The main motivation of video mining is to find undiscovered knowledge from the stream based on visual and audio cues. The knowledge may typically include structure information within a video clip or association information among various clips, as well as trend information based on the analysis for a massive size of video set. Xie, et al. (2003) provided a solution to find temporal structures of sports video by an unsupervised multi-scale statistical Hidden Markov method without employing domain knowledge. To find topic associations among news videos, Pan and Faloutsos (2002) proposed a tool called "GeoPlot" to provide the information of correlation of news events occurrence. Mining undiscovered video information is usually based on apparent semantic concept extraction techniques in video clips, such as object concept, scene concept and event concept. In the authors' mind, video mining need a long way to go to explore the full potential of this topic as most of the video understanding problems are still unsolved yet.
Su Psychological and biological experiments show that more than 80% percent of the information we receive is chaptered by the visual system. As an important information carrier, Visual media which contains both images and videos, has lots of attractive features such as visualization, data carrying capacity for large amount of data, easier for understanding, and so on. It is common to us that video can easily and naturally express complicated information, which can be hardly matched by the traditional text based media. From the view point of the psychological study, the main reason why the video is more informative is because the content we get from the video can be easily mapped as semantic concepts. In the multimedia analysis field, we are trying to let computers imitate human's mapping from low level data to high level video content understanding, which can be seen as bridging the "Semantic Gap". Criterions of human's judgments for video media are different from the ones of computer vision system. For example, the objects in the real world are 3-dimensional, while the objects captured by the cameras are mapped into 2-dimensional space, and the real object contains many concepts for human while the digital images are just gathering of pixels for computers. This difference is always referred as "Sensory Gap". Consequently, one of the most urgent tasks of video analysis is conquering "Semantic Gap" and extracting high level information from the video. Video content analysis and understanding is proposed aiming at solving this problem. Although more and more researchers have engaged in this research and more and more methods have been proposed, the work on video semantic analysis and understanding is still not good enough and lots of problems need to be further investigated.
ACKNOWLEDGMENT The authors are supported by grants from Chinese NSF under contract No. 60702035 and 60773136, National Hi-Tech R&D Program (863) of China under contract No. 2006AA01Z117 and 2006AA010105, and National Key Technology R&D Program under contract No. 2006BAH02A10. We would like to
sincerely appreciate the help and valuable input provided by Shiliang Zhang, Xuekan Qiu, Guorong Li, Shuhui Wang, Xiaojun Li, Lei Hu, Lingfang Li, Shugao Ma, Shaojie Cai, Yu Gong and Chunxi Liu, Huiying Liu, etc.
R Agarwal, S., Awan, A., & Roth, D. (2004). Learning to detect objects in images via a sparse, part-based representation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 26(11), 1475–1490. doi: 10.1109/TPAMI.2004.108. Arman, F., Hsu, A., & Chiu, M.Y. (1993). Image processing on compressed data for large video database, ACM international conference on Multimedia, (pp. 267-272). doi: 10.1145/166266.166297. Aslandogan, Y.A., & Yu, C.T. (1999). Techniques and systems for image and video retrieval. IEEE Transactions on Knowledge and Data Engineering, 11(1), 56-63. doi: 10.1109/69.755615. Bai, L., Lao, S.Y., Jones, G., & Smeaton, A.F. (2007). A semantic content analysis model for sports video based on perception concepts and finite state machines. IEEE International Conference on Multimedia and Expo, (pp. 1407-1410). doi: 10.1109/ICME.2007.4284923. Boreczky, J.S., & Rowe, L.A. (1996). Comparison of video shot boundary detection techniques. In Proc. IS&T/SPIE Storage and Retrieval for Still Image and Video Databases IV, 2664, 170-179. Campisi, P., Longari A., & Neri, A. (1999). Automatic key frame selection using a wavelet based approach. In Proc. of SPIE, 3813, 861-872. Comaniciu, D., & Meer, P. (2002). Mean shift: a robust approach toward feature space analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 24(5), 603-619. doi: 10.1109/34.1000236. Ekin, A., Tekalp, A. M., & Mehrotra, R. (2003). Automatic soccer video analysis and summarization. IEEE Trans. on Image Processing, 12(7), 796-807. doi: 10.1109/TIP.2003.812758. Fergus, R., Perona, P., & Zisserman, A. (2003). Object class recognition by unsupervised scale-invariant learning. In Proceedings of the Ninth International Conference on Computer Vision and Pattern Recognition, 2, 264–271. Forsyth, D.A., & Ponce, J. (2003). Computer vision: A modern approach. Prentice Hall. Garcia, C., & Delakis, M. (2004). Convolutional Face Finder: A Neural Architecture for Fast and Robust Face Detection. IEEE Translations on Pattern Analysis And Machine Intelligence, 26(11), 1408–1423. doi: 10.1109/TPAMI.2004.97. Haering, N., Qian, R.J., & Sezan, M.I. (2000). A semantic event detection approach and its application to detecting hunts in wildlife video. IEEE Trans. on Circuits and Systems for Video Technology, 10(6), 857–868. doi: 10.1109/76.867923. Hanjalic, A., Lagendijk, R., & Biemond, J. (1999). Automated high-level movie segmentation for advanced video retrieval systems. IEEE Transactions on Circuits and Systems for Video Technology, 9(4), 580–588. doi: 10.1109/76.767124.
Harris, C., & Stephens, M. A. (1988). A combined corner and edge detector. In Alvey Vision Conference, (pp. 147–151). Hsu, W., & Chang, S.F. (2003). A statistical framework for fusing mid-level perceptual features in news story segmentation. IEEE International Conference on Multimedia and Expo, (pp. 413-416). Hua, X.S., & Zhang H.J. (2004). An attention-based decision fusion scheme for multimedia information retrieval. Proceedings of Pacific-Rim conference on Multimedia. Huang, J., Liu, Z., & Wang, Y. (1998). Integration of audio and visual information for content-based video segmentation. IEEE International Conference on Image processing, (pp. 526-530). doi: 10.1109/ ICIP.1998.727252. Isard, M., & Blake, A. (1998). CONDENSATION—Conditional density propagation for visual tracking. International Journal of Computer Vision, 29(1), 5–28. Itti, L., Koch, C., & Niebur, E. (1998). A model of saliency-based visual attention for rapid scene analysis. IEEE Trans on Pattern Analysis and Machine Intelligence, 20(11), 1254-1259. doi: 10.1109/34.730558. Itti, L., & Koch, C. (1999). A Comparison of feature combination strategies for saliency-based visual attention systems. SPIE Human Vision and Electronic Imaging IV (HVEI'99), 3644, 73-82. Iurgel, U., Meermeier, R., Eickeler, S., & Rigoll, G. (2001). New approaches to audio-visual segmentation of TV news for automatic topic retrieval. IEEE International Conference on Acoustics, Speech, and Signal Processing. (pp. 1397-1400). doi: 10.1109/ICASSP.2001.941190. Jain, A.K., & Yu, B. (1998). Automatic text location in images and video frames. Pattern Recognition, 31, 2055–2076. Jiang, S.Q., Ye, Q.X., Gao, W., & Huang, T.J. (2004). A new method to segment playfield and its applications in match analysis in sports video, ACM International Conference on Multimedia, (pp. 292-295). doi: 10.1145/1027527.1027594. Johnson, N., & Hogg, D.C. (1996). Learning the distribution of object trajectories for event recognition. Image and Vision Computing, 14, 609-615. Ke, Y., & Sukthankar, R. (2004). PCA-SIFT: A more distinctive representation for local image descriptors. In Proc. Conf. Computer Vision and Pattern Recognition, 2, 511-517. doi: 10.1109/CVPR.2004.1315206. Kijak, E., Gravier, G., Gros, P., Oisel, L., & Bimbot, F. (2003), HMM based structure of tennis videos using visual and audio cues. IEEE International Conference on Multimedia and Expo. doi: 10.1109/ ICME.2003.1221310. Kijak, E., Oisel , L., & Gros, P. (2003). Hierarchical structure analysis of sport video using HMMS. IEEE International Conference on Image Processing, 2, II-1025-8. Kim, C., & Hwang, J.N. (2000). An integrated scheme for object-based video abstraction. ACM international conference on Multimedia, (pp. 303 – 311). Kim, K.I., Jung, K., & Kim, H. (2003). Texture-based approach for text detection in images using support vector machines and continuously adaptive mean shift algorithm. IEEE Transactions on Pattern Analysis and Machine Intelligence, 25, 1631–1639. doi: 10.1109/TPAMI.2003.1251157. 231
Leibe, B., Leonardis, A., & Schiele, B. (2004). Combined object categorization and segmentation with an implicit shape model. ECCV2004 Workshop on Statistical Learning in Computer Vision, (pp. 17-32). Li, Y., Zhang, T., & Tretter, D. (2001). An Overview of Video Abstraction Techniques. HP Labs Technical Report, HPL-2001-191, 20010809. Retrieved from http://www.hpl.hp.com/techreports/images/pdf. Lienhart, R., Kuhmünch, C., & Effelsberg, W. (1997). On the detection and recognition of television commercials. In Proceedings of the International Conference on Multimedia Computing and Systems, Ottawa, Ontario, Canada, (pp. 509-516). Lienhart, R. (1999). Comparison of automatic shot boundary detection algorithms. In Proc. IS&T/SPIE Storage and Retrieval for Image and Video Databases VII, 3656, 290–30. Lin, T., & Zhang, H. (2001). Video content representation for shot retrieval and scene extraction. International Journal of Image Graphics, 3(1), 507–526. Liu, H., Jiang, S., Huang, Q., Xu, C., & Gao, W. (2007). Region-based visual attention analysis with its application in image browsing on small displays. ACM International Conference on Multimedia, (pp. 305-308). 10.1145/1291233.1291298. Lowe, D. (2004). Distinctive image features from scale-invariant key points. Int'l J. Computer Vision, 2(60), 91-110. doi: 10.1023/ B:VISI.0000029664.99615.94. Lou, J., Liu, Q., Tan, T., & Hu, W. (2002). Semantic interpretation of object activities in surveillance algorithms in the area of computer vision and machine system. Pattern Recognition, 3, 777-780. Ma, Y.F., & Zhang, H.J. (2003). Contrast-based image attention analysis by using fuzzy growing. ACM international conference on Multimedia, (pp. 374-381). doi: 10.1145/957013.957094. Ma, Y.F., Hua, X.S., Lu, L., & Zhang, H.J. (2005). A generic framework of user attention model and its application in video summarization. IEEE Trans on Multimedia, 7(5), 907- 919. doi: 10.1109/ TMM.2005.854410. Matas, J., Chum, O., Urban, M. & Pajdla, T. (2002). Robust wide baseline stereo from maximally stable extremal regions. British Machine Vision Conference, (pp. 384-393). Mikolajczyk, K., & Schmid, C. (2004). Scale & affine invariant interest point detectors. Int. J. Computer. Vision, 60(1), 63-86. doi: 10.1023/ B:VISI.0000027790.02288.f2. Mohan, A., Papageorgiou, C., & Poggio, T. (2001). Example-based object detection in images by components. IEEE Transactions on Pattern Analysis and Machine Intelligence, 23(4), 349–361. doi: 10.1109/34.917571. Naquest, M.V., & Ullman, S. (2003). Object recognition with informative features and linear classification. Ninth International Conference on Computer Vision, 1, 281–288. doi: 10.1109/ICCV.2003.1238356. Navalpakkam, V., & Itti, L. (2006). An integrated model of top-down and bottom-up attention for optimizing detecting speed. IEEE Conference on Computer Vision and Pattern Recognition, 2, 2049-2056. doi: 10.1109/CVPR.2006.54.
Nitta, N., Babaguchi, N., & Kitahashi, T. (2002). Story based representation for broadcasted sports video and automatic story segmentation. IEEE International Conference on Multimedia and Expo, 1, pp. 813-816. doi: 10.1109/ICME.2002.1035906. Oh, J.H., Lee, J. K., & Hwang, S. (2005) Video data mining: current status and challenges. In J. Wang (Eds.) Encyclopedia of Data Warehousing and Mining, (pp. 104-107). Idea Group Inc. and IRM Press. doi: 10.1002/sam.10003. Osuna, E., Freund, R. & Girosi, F. (1997). Training Support Vector Machines: an Application to Face Detection. International Conference on Computer Vision and Pattern Recognition, (pp. 130–136). doi: 10.1109/CVPR.1997.609310. Otsuji, K. Y., Tonomura, & Ohba, Y. (1991). Video browsing using brightness data. in Proc. SPIE/IS&T VCIP'91, 1606, 980–989. Owens, J., & Hunter, A. (2000). Application of the self-organizing map to trajectory classification. In Proc. Third IEEE Visual Surveillance Workshop, (pp. 77-83). doi: 10.1109/VS.2000.856860. Pan, J., & Faloutsos, C. (2002). GeoPlot: Spatial data mining on video libraries. Int'l Conf. Information and Knowledge Management, (pp. 405-412). Pass, G., & Zabih, R. (1996). Histogram refinement for content-based image retrieval. IEEE Workshop on Applications of Computer Vision, (pp. 96-102). doi: 10.1109/ACV.1996.572008. Peters, RJ., & Itti, L. (2007). Beyond bottom-up: Incorporating task-dependent influences into a computational model of spatial attention. IEEE Conference on Computer Vision and Pattern Recognition, (pp. 1-6). doi: 10.1109/CVPR.2007.383337. Rui, Y., Huang, T.S., & Mehrotra, S. (1999) Constructing Table-of-Content for videos, ACM Multimedia System, 7(5), 359–368. doi: 10.1007/s005300050138. Schneiderman, H., & Kanade, T. (2000). A Statistical Method for 3D Object Detection Applied to Faces and Cars. IEEE Conference on Computer Vision and Pattern Recognition, 1, 746-751. doi: 10.1109/ CVPR.2000.855895. Shahraray, B. (1995). Scene change detection and content-based sampling of video sequences. In Proc. IS&T/SPIE, 2419, 2–13. Smith, J.R., & Chang, S. F. (1995). Tools and techniques for color image retrieval. In Proc. SPIE: Storage and Retrieval for Image and Video Database, 2670, 426-437. Smith, J.R., & Chang, S.F. (1996). Automated binary texture feature sets for image retrieval. In Proc IEEE Int Conf Acoust, Speech and Signal Proc, May, (pp. 2239-2242). doi: 10.1109/ICASSP.1996.545867. Solso, R.L., Maclin, M.K. & Maclin, O.H. (2003). Cognitive psychology. Perception and Attention, (pp. 70-103). Pearson Education. Stricker, M.A., & Orengo, M. (1995). Similarity of color images. In Proc. SPIE: Storage and Retrieval for Image and Video Database, 2420, 381-392. Sumpter, N., & Bulpitt, A.J. (2000). Learning spatio-temporal patterns for predicting object behavior. Image and Vision Computing, 18(9), 697-704. 233
Sundaram, H., & Chang, S.F. (2000). Video scene segmentation using video and audio features. IEEE International Conference on Multimedia and Expo, 2, (pp. 1145-1148). doi: 10.1109/ICME.2000.871563. Tang, X., Gao, X.B., Liu, J., & Zhang, H. (2002). Spatial-temporal approach for video caption detection and recognition. IEEE Transactions on Neural Networks, 13, 961–971.doi: 10.1109/TNN.2002.1021896. Tavanapong, W., & Zhou, J. (2004). Shot clustering techniques for story browsing, IEEE Transaction on Multimedia, 6(4), 517-527. doi: 10.1109/TMM.2004.830810. Thayananthan, A., Stenger, B., Torr, P., & Cipolla, R. (2003) Shape context and chamfer matching in cluttered scenes. International Conference on Computer Vision and Pattern Recognition, 1, 127-133. doi: 10.1109/CVPR.2003.1211346. The TRECVID conference website. (2008). Retrieved June 6, 2008, from http://www-nlpir.nist.gov/ projects/trecvid/. Uchihashi, S., Foote, J., Girgensohn, A., & Boreczky, J. (1999). Video manga: Generating semantically meaningful video summaries. ACM international conference on Multimedia, (pp. 383–392). Udea, H., Miyatake, T., & Yoshizawa, S. (1991). IMPACT—An integrated natural-motion-picture dedicated multimedia authoring system. New York. ACM Press, (pp. 343-250). Vetterli, J.K.M. (1995). Wavelets and subband coding. Prentice Hall. Viola, P., Jones, M. (2001). Robust Real Time Object Detection. IEEE ICCV Workshop on Statistical and Computational Theories of Vision, 2, II- 465-8. doi: 10.1109/ICME.2003.1221654. Viola, P., & Jones, M. (2001). Rapid object detection using a boosted cascade of simple features. IEEE Conference on Computer Vision and Pattern Recognition, 1, 511-518. doi: 10.1109/CVPR.2001.990517. Wang, T., Li, J., Diao, Q., Hu, W., Zhang, Y., & Duolong, C. (2006). Semantic event detection using conditional random fields. Computer Vision and Pattern Recognition Workshops, (pp. 109-109). doi: 10.1109/CVPRW.2006.190. Wang, P., Liu, Z., & Yang, S. (2005). A probabilistic template-based approach to discovering repetitive patterns in broadcast videos. ACM international conference on Multimedia, (pp. 407-410). doi: 10.1145/1101149.1101238. Wolf, W. (1996). Key Frame Selection by motion analysis. IEEE International Conference on Acoustics, Speech, and Signal Processing, (ICASSP'96), 2, 1228-1231. doi: 10.1109/ICASSP.1996.543588. Wu, V., Manmatha, R., & Riseman, E.M. (1999). Textfinder: an automatic system to detect and recognize text in images, IEEE Transactions on Pattern Analysis and Machine Intelligence, 20, pp. 1224–1229. doi: 10.1109/34.809116. Xie, L., Chang, S.F., Divakaran, A., & Sun, H.(2003). Unsupervised mining of statistical temporal structures in video. In A. Rosenfeld, D. Doremann, & D. Dementhon (Eds.), Video Mining. Kluwer Academic Publishers. Xie, Z. X., Shyu, M.L., & Chen, S.C. (2007). Video event detection with combined distance-based and rule-based data mining techniques. IEEE International Conference on Multimedia and Expo, (pp. 2026-2029). doi:10.1109/ICME.2007.4285078. 234
Xu, X., Li. B. (2002). Head tracking using particle filter with intensity gradient and color histogram. Journal of Computer Science and Technology, 17(6), 859-864. Ye, Q.X., Huang, Q.M., Gao, W., & Jiang, S.Q.(2005). Exciting event detection in broadcast soccer video with mid-level description and incremental learning. ACM International Conference on Multimedia, (pp. 455-458). doi: 10.1145/1101149.1101250. Ye, Q.X., Huang, Q.M., Gao, W., & Zhao D.B. (2005). Fast and robust text detection in images and video frames. Image and Vision Computing, 23, (6), 565-576. doi: 10.1016/j.imavis.2005.01.004. Yilmaz, A. Javed, O., & Shah, M. (2006). Object Tracking: A Survey. ACM Computing Surveys, 38(4), doi:10.1145/1177352.1177355. Zabih, R., Miller, J., & Mai, K. (1995). A feature-based algorithm for detecting and classifying scene breaks. ACM international conference on Multimedia, (pp. 189-200). doi: 10.1145/217279.215266. Zhang, H. J., Wu, J., Zhong, D., & Smoliar, S. W. (1997). An integrated system for content based video retrieval and browsing. Pattern Recognition, 30(4), 643-658. Zhang, H., Kankanhalli, A., & Smoliar, S. W. (1993). Automatic partitioning of full-motion video. Multimedia System, 1 10–28. doi: 10.1007 /BF01210504. Zhai, Y., & Shah, M. (2006). Visual attention detection in video sequences using spatio-temporal cues. ACM international conference on Multimedia, (pp. 815-824). doi: 10.1145/1180639.1180824 Zhu, X., Wu, L., Xue, X., Lu, X., & Fan, J. (2001). Automatic scene detection in news program by integrating visual feature and rules. The second IEEE Pacific-Rim conference on multimedia, 2195, 837–842.
Applications of Semantic Mining on Biological Process Engineering Hossam A. Gabbar University of Ontario Institute of Technology, Canada Naila Mahmut Heart Center - Cardiovascular Research Hospital for Sick Children, Canada
ABSTRACT Semantic mining is an essential part in knowledgebase and decision support systems where it enables the extraction of useful knowledge form available databases with the ultimate goal of supporting the decision making process. In process systems engineering, decisions are made throughout plant / process / product life cycles. The provision of smart semantic mining techniques will improve the decision making process for all life cycle activities. In particular, safety and environmental related decisions are highly dependent on process internal and external conditions and dynamics with respect to equipment geometry and plant layout. This chapter discusses practical methods for semantic mining using systematic knowledge representation as integrated with process modeling and domain knowledge. POOM or plant/process object oriented modeling methodology is explained and used as a basis to implement semantic mining as applied on process systems engineering. Case studies are illustrated for biological process engineering, in particular MoFlo systems focusing on process safety and operation design support.
INTR The recent challenges of industrial, medical, and social systems are focused on how to provide intelligent learning systems that can deal with increasing complexity and multidimensional knowledge structures. The problem became worse when dealing with different scales of domain knowledge in multimedia Copyright © 2009, IGI Global, distributing in print or electronic forms without written permission of IGI Global is prohibited.
Applications of Semantic Mining on Biological Process Engineering
formats. Researchers and professionals from industry are seeking practical approaches to extract useful knowledge to support decisions throughout process / system life cycle. Recent advances in semantic mining motivated researchers and industry in different system disciplines to adopt and apply semantic mining techniques for traditional and multimedia databases and knowledgebase systems. This chapter will discuss problems and solutions related to application of advanced semantic mining using formal representation of domain knowledge for process systems. The proposed approach can be divided into the following major stages: • • • • •
Model construction of process systems Knowledge structuring and representation of process models Construction of formal meta-language and language to represent domain knowledge and activities of process models Knowledge acquisition and validation (qualitative and quantitative) Semantic mining for decision support of process life cycle activities
The management of these stages requires robust modeling methodology, which facilitated the integration between nano, micro, and macro process levels. Each process level is abstracted using building blocks in three basic views: static, dynamic, and operation (Gabbar et al., 2001; 2003; 2004). In view of the proposed modeling and simulation methodology, control layer is used to support the design and operation of the underlying system as integrated with simulation environment. The proposed system engineering approach supports change management, HSE (health, safety, and environment), recycling, energy, and sustainability. Planning and scheduling are analyzed and managed for the different hierarchical levels i.e. nano, micro, and macro. For example, process of reconstruction of human cell functions includes MoFlo cell sorter systems.
Figure 1. Process engineering framework
ovERVIEW OF SEMANTIC MINING FOR PROCESS ENngining Almost all efforts made in semantic mining was directed towards discovering new knowledge or search for important knowledge within large number of data elements, databases, or knowledgebases in any form, e.g. html pages, multimedia, text, etc. Semantic mining is always associated with process where primitive data is collected, updated, and manipulated. Such process involves organizational and behavioral changes. For example, semantic mining for biological processes is associated with basic biological data, e.g., samples, and phenomena linked with such biological data such as cell separation, transformation, or demolishing. In order to effectively apply and benefit from knowledge obtained from semantic mining, it is essential to define clearly how the knowledge is created, maintained, and managed with respect to process model in terms of structure, behavior, and operation. The concept of process based semantic mining is the ultimate approach for successful semantic mining. In other words, the construction of database / knowledgebase is considered as part of the semantic mining where meta-data are constructed about domain knowledge that supports semantic mining process. There are several approaches that have been done in the area of semantic mining for process engineering. Among these approaches are those that have been directed towards operating procedures representation, synthesis, and verification.
Semantic Mining for Pocess Operation Design To meet the ever-increasing market requirements, production and manufacturing plants are becoming more complex. The design of efficient operating procedures for complex production plants faces different challenges and tends to be impractical without providing clear, structured, and complete operating procedures for normal and abnormal operation. Providing structured and modular way to describe operating procedures will facilitate the operation design and management, will reduce operator errors, and will make it easier for operator to understand and follow operating procedures with less time and efforts. Different attempts are made to structure operating procedures and provide automated solutions to design and synthesize operating procedures. ANSI/ISA-S88 is one the developed standards by ANSI, which proposed a structured way to define operating procedures for batch plants in hierarchical way (ANSI/ISA). It proposed mapping between plant structure hierarchy and operating procedures of batch plants. Many researchers have based their automated solutions of operating procedures synthesis on the framework described by ANSI/ISA-S88. However, the way operating procedures are represented varies from one approach to another. Viswanathan (1998), Johnsson (1998), and Arzen (1994) used Graphchart to represent operating procedures in graphical way. Such approach provides operator with friendly user interface to visualize operating procedures hierarchically. However it requires quite long preparation stage to define the different classes, objects, and definitions of operating procedures, so that they can be mapped to graph symbols. To overcome the limitations expressed by the graphical representations methods, formal methods are introduced. Gabbar (2003) proposed recipe formal definition language, or RFDL, to represent operating procedures in formal way as mapped to domain knowledge. In his solution, editor and parser are used to validate the syntax of operating procedures using set of predefined RFDL statements. The proposed method is used to design automated solution, called AOPS to synthesize master and control recipe of batch plants. However, there was major limitation in dealing with the semantics of the underlying plant operation due to the lack of robust integration with domain knowledge. In addition, the proposed approach requires further methods to validate the generated operating procedures.
To overcome these limitations and in order to provide general purpose operating procedure synthesis solutions for batch, continuous, as well as discrete production plants, this research work proposes an engineering formal language (called EFL), which is used to synthesize standard operating procedures (or SOP) in accurate and effective manner. To simplify complex operations, the concept of generic or meta-operation is introduced where operation libraries are defined as building blocks to define and represent SOP for complex plants, with less errors and time. Operation engineering approach is proposed based on robust modeling methodology of the underlying plant process. Operation is structured and mapped to plant structure and generic definition of plant operation is proposed along with intelligent topology analysis methodology. Engineering formal language is proposed and used to represent standard operating procedures of different case studies from batch and oil & gas plants. Integrated automated solution for operation engineering is described to automate the synthesis of SOP and meta-operation, and to manage operation execution. Plant operation is sets of instructions that are executed by operator and/or system to produce target products. Standard operating procedures are commonly used to describe plant operation with the required details to ensure quality, safety, steady, optimum and other life cycle factors while operating the plant. Operation engineering is the process to manage SOP throughout its life cycle, which typically includes SOP design, validation, execution, and control. SOP design is a complex process that is usually done starting from the process design stage. In order to overcome such complexity, it is essential to structure operation according to the associated structure area. This requires hierarchical partitioning of the underlying plant structure (i.e. topology) and the definition of operation for each structure area in different hierarchical levels.
Operation: Structure Hierarchy ANSI/ISA-S88 is one useful standard for chemical batch plants that proposed hierarchical definitions of plant structure and the associated operation hierarchy, as shown in figure 2.
Figure 2. ANSI/ISA-S88 structure (Modified by Gabbar 2004)
In figure 2, structure hierarchy is defined as cell, unit, OIA (or operation isolation area), and EIA (or equipment isolation area). OIA is defined as the topological area that is surrounded by flow control devices such as valves. EIA is defined as the topological area that includes the underlying process equipment (i.e. reactor), which is surrounded by flow control devices. Previously, the terms OIA and EIA were used in case of batch plants (Gabbar et al., 2004). However, in order to provide general solution that suits batch, continuous, and discrete manufacturing processes, the concepts of OIA and EIA are proposed.
MoFo The recent advancement in biological and medical research mobilized biotechnology and biomedical manufacturers to provide advanced systems to automate different critical and functions in the biology and biomedical fields. One critical process is the separation of cells or biological elements, which is essential for many research experiments and biotechnology production. There are a variety of cell separation technologies, which are used by biological, clinical, pharmaceutical, and medical research labs. Among the latest of the cell sorting technologies is MoFlo, which is high-performance cell sorter used for precision analysis and sorting of cells, bacteria, chromosomes, macromolecular particles, and other similar sized particles. Most of the recent research is based primarily on the separation process, which provides the required biological elements to analyze, measure, reconstruct human body elements i.e. pancreas, liver, etc. The separation process is developed primarily using cell sorter technology. Currently, and for the coming few years, MoFlo machine, which is one type of Fluorescence Activated Cell Sorter, or FACS, is one of the latest and reliable cell sorter and biological particles separation system that is used widely in biological, medical, pharmaceutical research labs as well as in biotechnology industry. The operation of such system is relatively complex and costly, where it requires doing many experiments till achieving the required accurate cell sorting results. Each experiment is quite expensive where it might cost around US$100 for one experiment (i.e. one day). It takes quite long time, up to 6 months, till it can produce the required cell colony, which is the maximum grouping of the desired cells. In addition, the operation of MoFlo is associated with high risk where it contains some dangerous / hazardous operation scenarios. MoFlo manufacturers and makers are investigating system and bioengineering approaches
Figure 3. Modeling of typical MoFlo-based biological process
to improve the operation of such critical equipment and to assure cost and risk reduction of R&D and production systems of the underlying bio-process. To support MoFlo makers as well as users, this research proposes knowledge-based system engineering approach, which is used to analyze MoFlo operation, and propose systematic way to design efficient operation with reduced operational cost and risks. In the following section, proposed system engineering approach will be presented. In section three, MoFlo system model will be presented. In section four, the proposed operation model and engineering environment will be presented along with some hazardous operation scenarios and its analytical view.
Biological Research Process Hierarchy MoFlo as part of the integrated biological research system includes different levels of interests: (a) nano level, which includes marker materials used to mark and separate cells; (b) micro level, which includes cells (around 50 micro meter); and (c) macro level, which includes components of the MoFlo system. The link between MoFlo machine and other machines and devices within the lab / production process is shown in the higher level of interest, which requires process engineering as well. Figure 3 shows the different levels of processing and control for MoFlo systems.
MoFo Process Model Formalization Lifecycle engineering is based on robust modeling methodology, which enables the construction of process design model in different hierarchical levels and link them together while designing the operation of MoFlo system with the considerations of lifecycle assessment of different factors such as health, safety, energy, cost, maintenance, etc. Model building blocks are used to construct model of complex processes / systems, where object-oriented modeling concepts i.e. inheritance, polymorphism, encapsulation are used to systematically construct process models. The proposed modeling methodology (called POOM or plant/process object-oriented modeling) enabled the design of the MoFlo operation and the whole process where operation model elements are defined in each building block, while operation knowledge are associated and accumulated during the design stage. Control layer includes control elements that ensure safe, steady, and optimized operation. This can be modeled as control rules and logic associated with the different building blocks of process model of the underlying MoFlo process. Figure 4 shows the proposed iterative modeling mechanism, which is used to construct the underlying MoFlo process model. It starts with design of model elements: static, i.e. structure; dynamic, i.e. behavior; and operation, i.e. methods / recipe. Using these design model elements, and in view of the target functions, operation is designed in terms of methods and recipe (i.e. operation steps) for the different operation scenarios. Each operation scenario is simulated and evaluated in view of the required performance indicators such as cost, accuracy, time, etc. In view of the simulated results, tuning can be made to design model elements, operation model elements, and/or operation scenarios. Figure 5 shows the links between the three different views of equipment (i.e. MoFlo), product (i.e. sorted cells), and process (i.e. human reconstruction) models. The integration among these models with business and system models is essential for lifecycle assessment and high performance MoFlo operation.
Figure 4. Based process modeling
Figure 5. System engineering approach
MoFlo Cytomation System MoFlo is high precision, high performance, and modular Fluorescence Activated Cell Sorter, FACS, system used for research and clinical purposes to sort or separate cells, bacteria, and/or other macromolecular particles. Using MoFlo Cell Sorter, multiple parameters can be analyzed for heterogeneous samples, which can be sorted in four separate collecting tubes at speed approaching 70,000 events per second. It can analyze up to 16 parameters for each particle, while with the newest technology of MoFlo DSP it can analyze up to 32 parameters.
MoFlo System Components Figure 6 shows the system components of MoFlo system, which has around 27 components. The MoFlo system is composed of three major components: Summit Workstation, Electronic Console, and Optical Bench. Figure 6 shows the basic elements of each component in hierarchical manner, reflecting the function decomposition.
Figure 6. MoFlo system architecture
Figure 7. MoFlo function modeling
MoFlo System Mdeling The proposed modeling methodology is used to model MoFlo in three views: static, dynamic, and operation. SU or structure unit is used to model each structural element such as component, subassembly, or subsystem. The connection among the different SU's is defined as a port. Ports are information port, control port, sensor port, material port, or energy port. Each port could be input, output, or input/output. The different behaviors are defined for each SU so that control layer can select the most advantageous behavior to achieve the target operation. This can be determined in view of state variables, which include goal variables, control variables, disturbance variables, and other state variables that represent the behavior of each component. Operation recipe and instructions are synthesized / defined within each SU where they are defined with the use of simulation to ensure the limits of the values of each process variable. Details about the proposed POOM modeling methodology are explained by Gabbar (2001; 2002; 2004). Computer-aided modeling environment (called CAPE-ModE) on the basis of POOM has been proposed and developed by Gabbar (2004). Figure 8 shows example structure model of MoFlo where three SU's are defined for one main SU "Optical Bench". Ports are defined as a connection between these SU's. Ports transfer energy such as electricity, process variable information such as sensor port, etc. Each of the three SU components, defined in figure 8, has set of state variables i.e. process variables and set of behaviors as well as operating procedures. In addition, control rules / constraints and logic are defined within each SU and its detailed structural elements so that they can execute the defined operating procedures. The proposed control layer will be enable the resolution of conflicts among control rules and logic in the same hierarchical level and coordinate with upper and lower hierarchies.
Kno One of the major difficulties that face lifecycle engineering is the definition of flexible structure to support the acquisition and management of lifecycle data and knowledge. This problem is more difficult when addressing engineering systems where business and engineering data are required to be linked and maintained in systematic way to support lifecycle activities. It is widely known that the key success factor to engineering enterprises is the robust data and knowledge modeling and management, which satisfy life cycle activities requirements. Standards, such as STEP, provided unified structure to manage engineering data, however, the realization of such approach requires more efforts to be unified and practically implemented. ANSI/ISA S95 provided better model and practice which can effectively be
Figure 8. POOM-based hierarchical structure unit model
used to manage life cycle data with respect to product / process design. The proposed data and knowledge modeling approach is based on defining knowledge structure hierarchy based on domain ontology, which is used to define data elements within each building block class. This enabled linking domain knowledge with the different model elements in all hierarchical levels. For example, cell class will be associated with attributes such as type, category, status, etc. While marker class will be associated with attributes such as function, maker, limitation, working condition, etc. These classes will be linked to domain knowledge where cell types and sorting criteria will be defined. In higher levels, MoFlo system can be linked to functions related to human cell reconstruction process where separated cells are linked to the requirements initiated from regenerative medicine or reconstruction of human cells. This requires integrated computer-aided modeling environment that facilitates the integration between domain knowledge and model elements in different hierarchies. Based on the proposed modeling framework for MoFlo system, operation engineering approach is presented on the basis of formal methods. In the following section, formal methods are explained.
Engin EERIingLanguag (EFL) Formal Methods Overview The proposed system approach is based on proposed formal method to represent process design model and define operation in systematic and formal manner that enable the automatic design and synthesis of the operation of such complex and critical equipment.
Formal Methods What is formal method? Fischer (1998) stated that formal methods are used to specify processes and systems, such as formal description techniques (FDT's) such as Estelle and SDL. Astesiano (2002) stated that formal method could be used to support activities in the development process, such as: support & validate engineering requirements and process design specifications; validate intermediate specifications against initial and final specification; provide smooth code generation with the validation against design/intermediate specifications; check the quality of some specification/code; reuse (replay) a part of the development process (single activity) by changing something in the inputs (scenario evaluation); and produce a new version of the already developed end-product (maintenance). In addition, formal methods can be used to design and validate the operation of the underlying system, especially for complex systems. This has been shown in different applications such as the use of formal methods to synthesis operating procedures of chemical plants (Gabbar, 2004). Wide and narrow definitions of formal methods can be found in the literature. For example, Nancy Leveson states: A broad view of formal methods includes all applications of (primarily) discrete mathematics to software engineering problems. This application usually involves modeling and analysis where the models and analysis procedures are derived from or defined by an underlying mathematically- precise foundation. The gaps between users' intentions and formal specifications and between physical implementations and abstract proofs create inherent limitations to formal methods, no matter how much they may be developed in the future. . 245
Benefits of Using Formal Methods to Engineering Design & Operation These are some reported benefits of using formal methods in engineering design & operation: • • • • •
Formal methods are directly applicable during the requirements, design, construction, and operation of engineering systems. Formal methods provide means for validation, verification, and the application of design rationales. Formal methods enable the management of change throughout the different stages of engineering and system lifecycle. Formal methods can be used as the base to develop robust software engineering solutions for engineering applications. Formal methods are entwined with lifecycle models that may provide an alternative to the Waterfall model, namely rapid prototyping, the Cleanroom variant on the spiral model, and "transformational" paradigms. Formal methods support precise and rigorous specifications of those aspects of a computer system capable of being expressed in the language. Since defining what a system should do, and understanding the implications of these decisions, are the most troublesome problems in software engineering, this use of formal methods has major benefits. In fact, practitioners of formal methods frequently use formal methods solely for recording precise specifications, not for formal verifications (Hall, 2001).
Formal Methods Examples Some of the most well known formal methods consist of or include specification languages for recording a system's functionality. These methods include: • • • • •
Z (pronounced "Zed") Communicating Sequential Processes (CSP) Vienna Development Method (VDM) Larch Formal Development Methodology (FDM)
Formal methods can include graphical languages. Data Flow Diagrams (DFD's) are the most well known graphical technique for specifying the function of a system. DFD's can be considered a semiformal method, and researchers have explored techniques for treating DFD's in a completely formal manner. Petri nets provide another well-known graphical technique, often used in distributed systems (Peterson, 1977). Petri nets are a fully formal technique. Finally, finite state machines are often presented in tabular form. This does not decrease the formalism in the use of finite state machines. So the definition of formal methods provided earlier is quite encompassing.
Limitations of Formal Methods Given the applicability of formal methods throughout the lifecycle, and their pervasive possibilities for almost all areas of software and systems engineering, why are they not more widely visible? Part of the 246
problem is educational. Revolutions are not made by conversion, but by the old guard passing away. More recent university graduates tend to be more willing to experiment with formal methods. On the other hand, the only barrier to the widespread transition of this technology is not lack of knowledge on the part of practitioners. Formal methods do suffer from certain limitations. Some of these limitations are inherent and will never be overcome. Other restrictions, with research and practice, will be removed as formal methods are transitioned into wider use. Empirical evidence does suggest, however, that formal methods can make a contribution to the problem of adequately capturing requirements. The discipline of producing a formal specification can result in fewer specification errors. Furthermore, implementers without an exceptional designer's knowledge of the application area commit fewer errors when implementing a formal specification than when relying on hazy knowledge of the application (Goel, 1991). These benefits may exist even when the final specification is expressed in English, not a formal language (Meyer, 1985). A specification acts as a "contract" between a user and a developer. The specification describes the system to be delivered. Using specifications written in a formal language to complement natural language descriptions can make this contract more precise. Finally, developers of automated programming environments, which use formal methods, have developed tools to interactively capture a user's informal understanding and thereby develop a formal specification (Zeroual, 1991). Still, formal methods can never replace deep application knowledge on the part of the requirements engineer, whether at the system or the software level. The application knowledge of the exceptional designer is not limited to one discipline. For example, an avionics application might require knowledge of flight control, navigation, signal processing, and electronic countermeasures. Whether those drawing on interdisciplinary knowledge in developing specifications come to regard formal methods as just another discipline making their life more complicated, or an approach that allows them to simply, concisely, and accurately record their findings, will only be known with experience and experimentation. From the above detailed explanations about formal methods, it has been proved that formal methodsbased approach is the best choice to design and represent operation of complex systems such as MoFlo. The following section will highlight the proposed human-centered operation engineering approach and how formal methods can be used effectively to design the operation of MoFlo.
Egineering Formal Language Formal representation provides a systematic framework to construct and validate the syntax of the underling system towards building standard representation approaches. Formal languages are used in most of the engineering applications to reduce the time and efforts required to communicate and manage the underling system or process. Formal language implies absolutely accurate and precise definitions of the underling system, which can be used to validate the system / process and can be used as a base for computer-aided solutions. One can say that the use of formal languages will help overcoming major engineering problems. Operating procedures, as one element in engineering applications, is used to describe plant operation, e.g. operation of chemical batch plants. For efficient operating procedures synthesis, a structured way is required to represent the syntax and semantics of operating procedures as mapped to domain knowledge. The construction of domain knowledge can be systemized if a robust model formalization methodology is used. Many researchers have investigated the construction of process and plant models, and proposed
modeling methodologies to build the corresponding domain knowledge. Most of these modeling methodologies showed that plant model includes three views: structure, behavior, and operation (Gabbar et al., 2000; Lu et al., 1995, 1997). Utilization of formal representation of operating procedures facilitates the integration between operation, structure and behavior views, which will have a positive impact on the synthesis process of operating procedures. From the other side, formal representation will provide means to build and organize domain knowledge more efficiently. Many researchers are motivated to use computer languages such as HTML, XML, or even build their own languages on the basis of these computer languages to describe their systems or processes. For example, world batch forum proposed the use of XML to represent batch recipe (WBF). However, it is essential to define engineering language in a higher layer, which can deal with the syntax and semantics of operating procedures as linked to plant model (structure and behavior), before addressing the implementation issues using computer languages such as XML, HTML, etc. It will be practical if two separate layers are defined to represent operating procedures. The higher layer will enable designers and engineers to define operating procedures in standard and formal way as mapped to the underling system, regardless of the implementation way. This will enable them to think freely without the limitations of any computer language. Based on the higher layer, the lower layer can be designed or selected for implementation. In general, the process of designing operating procedures requires flexible way to define and maintain the syntax and semantics of operating procedures as it involves different participants with different views. It is relatively easy to maintain the syntax of the defined higher layer language, as it doesn't involve modifications to any computer program. This can be transferred (i.e. translated) into computer language, which is relatively easy and can be implemented for any destination (i.e. host) computer language in the lower layer (e.g. HTML, XML, etc.). As described by Carre (1989) and reported by Toyn (1998), formal language should be logically sound and unambiguous and the semantics must not be too complex, otherwise formal reasoning will become impractical. The proposed recipe formal language is based on the definition of simple and clear Englishlike statements, which are composed of keywords that are mapped to domain knowledge. The structure of the statements defines the syntax while the mapping to domain knowledge defines the semantics. The proposed recipe formal language fulfills the basic requirements of any formal language where hierarchical definitions are used to simplify the representation of complex procedures. For example, one can say: "material-movement is move material from area to area". This can be further explained as: "material is gas or liquid or solid fluid", while area could be "area is equipment or cell or unit" As a conclusion, the use of formal representation will facilitate the definition of complex operation in structured way as mapped to domain knowledge. In addition, using formal language to represent operating procedures will facilitate the development of automated solutions to synthesis operating procedures.
Operation Egineering of MoFlo As mentioned earlier, the operation of MoFlo system is quite complex and requires few months before being able to produce reliable results. This is due to complex operation and limited understanding of operators/users of MoFlo who are mainly biological or medical researchers / technicians, or industrial workers. In addition, the written operating procedures are relatively abstract and not well organized. Moreover, it requires many steps to be done manually, and many parameters to be defined and tuned, which requires good and enough training for MoFlo operators. In addition, it requires providing auto-
mation for the different manual operation tasks. This section describes the proposed human-centered operation design framework, which is used to cover the above mentioned limitation in current MoFlo operation.
MoFlo Operation Synthesis Operation synthesis has mainly two sides, designer side, which goes from operation designer till it reaches computer format operating procedures, and the operation side, which links operation instructions from MoFlo to the user. Operation can be viewed as set of microinstructions that are stored within MoFlo, which will be triggered by button or condition. These microinstructions are stored within each subsystem individually as they can work independently. Operation design is usually made during the design stage where the condition of the MoFlo is simulated and the different state variables are calculated at each time step in view of the selected operation. The resulted sets of microinstructions are stored within each subsystem. This process is usually done through computer where microinstructions are synthesized and stored within computer, till final test, then transferred into machine either on chips or any sort of memory devices. During operation, these operation instructions are executed and resulted data are displayed to operator / user as a visual language, i.e. voice, light, number, test, image, etc. Figure 9 shows proposed operation synthesis cycle, starting from operation designer, where operation is converted from natural language into formal language. Such formal language is translated into computer language, which is used to represent microinstructions. XML is proposed as a neutral language to represent operation instructions. The proposed operation synthesis framework provides an answer to the following major issues: • • •
Easy and understandable configuration of microinstructions with intelligent human interface User friendly design of visual language for easy human interactions Synthesis of microinstructions so that it can be mapped easily to understandable visual language
Figure 9. Operation design framework
Smooth mapping between operating procedures with microinstructions Considerations of malfunction and recovery operations and their conversion into microinstructions and visual language for safe operation
Proposed Approach Summary The following are summary of the proposed operation synthesis approach: • • • •
Synthesize microinstructions and map to visual language for intelligent human interface. Construct domain knowledge of biomedical equipment design & operation Design formal language (meta-language) to be used to synthesize microinstructions Develop editor and parser to capture the meta-language and to synthesize (generate) microinstructions for define operation as per the design spec of the medical equipment and functions (operation) Map microinstructions to visual language and propose intelligent human interface Define set of possible malfunctions and define recovery procedure, and synthesize the corresponding microinstructions and visualization.
Figure 10 shows the process flow for operation design and execution where designer creates and maintains design knowledge, which will be reflected into formal language in the computer system, then embedded instructions within MoFlo system. Operator will be maintaining operation knowledge, which is accumulated from the design stage. Gabbar (2003) proposed useful formal representation called RFDL or recipe formal definition language, which was used for batch plants. Such solution enabled the systematic definition of batch recipe
Figure 10. Operation design execution system architecture
using keywords and predefined statements. However, RFDL was limited to batch plants and did not cover issues such as concurrent operation and logic handling for behavior generation and operation control decisions. In this research, and in order to overcome these limitations, engineering formal language, or EFL, is proposed to express engineering knowledge including vocabulary from domain knowledge and associated rules for control and process constraints. The proposed formal language is in the form of English-like statements. Each EFL statement is composed of keywords called tokens. The standard form of EFL statement is: LHS-token :: token_1 token_2 …. token_n
[S1]
Where LHS-token is the left hand side token that will be assigned to the sequence of tokens in the right hand side of "IS". Tokens are classified as shown in table 1. Among these token classifications is the "input" token: which is used to inform operation management system to acquire input from user, such as operating parameters. Another token type is "variable" token: which means that such token will be further defined using another EFL statement (i.e. in the left hand side). Another token type called "operator" is used to represent all operators for EFL statements such as "and". These operators are used by the EFL editor to understand and check the syntax of EFL statements, such as nested statements. Another token type called "lookup" is used to retrieve list of corresponding values from lookup table (i.e. in the database). Another token class is "constant", which is used as it is within EFL statements. EFL statements are defined hierarchically. In other words, EFL statements are used as a meta-language to define SOP. The complete set of EFL statements should have at least one EFL statement for each variable token (i.e. variable token to appear in the left hand side of EFL statement). For example, operation actions are classified as: Operator-Action, System-Action, Procedure_Action, Material_Action, Topology_Action, External_Action, Control_Action, and Time_Action. Such operation action can be expressed as the following EFL statement [S2]: Operation-Action :: Operator-Action | System-Action | Procedure_Action | Material_Action | Topology_Action | External_Action | Control_Action | Time_Action [S2]
Figure 11. Operation ontology structure
The above SOPFL statement can be further explored using other SOPFL statements. For example, procedure action can be expressed as per EFL statement S3 where "START", "TERMINATE", and "RESUME" constant tokens are used to express the primitive procedure action statement, while "procedure_id" is a lookup token, which is used to display all possible values of procedures defined within the system. Procedure_Action :: ( START | TERMINATE | SUSPEND | RESUME ) procedure_id [S3] EFL statements (meta-language) and SOP statements are composed of vocabulary and rules. The vocabulary is described using ontology where concepts of the domain knowledge are classified. Figure 11 shows sample of the concepts classified using ontology. The developed ontology model is used as a base to construct EFL statements and to validate the semantics of operating procedures. Currently, base ontology model has been developed using ontology editor and converted into database repository, which is used by the proposed automated solution to design plant operation. Domain operation ontology can be used to construct EFL statements. For example, S2 can be constructed using parts of the ontology described in figure 3, where operation action can be classified as transportation action, control action, etc. Similarly, master and control recipe statements can be defined on the basis of EFL where keywords are linked to domain knowledge such as plant design model, material, functions, products, recipe formula, etc. For example, master recipe statement (MOVE material1 FROM t1 TO t2) is derived from EFL statements S4 & S5. Transportation_Action :: MOVE material FROM Topology_Area TO Topology_Area [S4] Topology_Area :: cell_id | unit_id | oid_id | ([class_function] [class_material] equip_class) | equipment_id [S5] Where material1 is selected from a lookup list of all materials defined within the domain knowledge of the used plant model. "FROM" and "TO" are constant keywords, while t1 and t2 are equipment id's, which are selected from lookup list of all equipments defined in the plant model. The proposed EFL statements can be used within EFL-based editor to synthesize operating procedures with complete syntax and semantic validations. However, during the definition of SOP of complex plant, there are varieties of repeated operations that are used in different topological areas and for different functions. These repeated operations are defined independently, although they are almost the same. In addition, complex operation is usually composed of set of simpler operations, which are also repeated in different operations. The following section describes the concept of meta-operation, which is proposed to overcome such situation.
Mapping Operation Structure to MoFlo Structure Model In order to design and synthesize MoFlo system operation, it is essential to define hierarchical structure model and to identify corresponding control levels, as shown in figure 2. Similar to ANSI/S88, which defined mapping between control levels (i.e. cell, unit, equipment) and operation levels (procedure, unit procedure, operation, and phase). In MoFlo, structure levels are defined as subsystem, unit, equipment, and component. The proposed operation hierarchy can be shown in figure 12. 252
Figure 12. Operation – Structure hierarchical mapping
EFLfor MoFlo Keywords are extracted from domain knowledge and classified as per the meta-classes of the design knowledge. For example, set of keywords is extracted from the sample subsystem i.e. "Tilt Block", "Agitation", etc. Editor and parser are used to define the operation of the underlying MoFlo as follows:
Startup and Alignment The following is the Ignite Laser startup operation: 1. 2. 3. 4. 5. 6.
Open cooling water valves (if applicable) Turn key on laser power supply Switch ON remote console to ignite laser Allow laser to warm up and stabilize Set laser to appropriate power and aperture Record information in MoFlo System Log The above operating procedures can be defined using formal language statement ML1.
OPEN [ML1] In this example, is a list of all possible control devices in the MoFlo system such as valves of (cooling-water, PMT Controller, Pulse Monitor Oscilloscope, etc.) Similarly, control setting can be defined using ML2. Set <device> to <setting-value> [ML2]
In this statement, <device> is lookup token that is linked to a list of devices such as Laser.
Example Laser Alignment Operation Single Laser Alignment operation can be viewed as: 1. 2. 3. 4.
Check switch setting on ADC and Dot Plot boards Ensure stream is vertical using pinhole image and camera Check laser beam for max diffraction & just below nozzle tip Using appropriate calibration particles, establish an appropriate event rate. Coarsely align the instrument by placing the bead-flash over the image of the top pinhole 5. Visualize acquired data by selecting channels 1 vs 2 and 3 vs 4 on the dot plot oscilloscopes 6. Optimize fluorescence intensity, then optimize %CV 7. Optimize forward light scatter intensity, then %CV 8. Toggle the drop drive on and off while turning to verify the absence of any optical noise while drop drive is on 9. Repeat steps: 3 through 6, if necessary 10. If necessary, align dichroic mirrors in front of PMTs Set of meta-language of SOPFL can be defined for all operating procedures, which will be reflected into the computer-aided engineering environment to synthesize and validate the operating procedures of the underlying MoFlo.
Figure 13. MoFlo Operation. (a) Fluorochrome-tagged cells or particles leave the CytoNozzle traveling single file in a charged stream. (b) A laser beam excites the fluorochromes, which fluoresce and emit photons at specific wavelengths. (c) Charged droplets, each containing one cell or particle, break away from the stream and pass between two charged plates. (d) The electric field created by the charged plate directs each cell or particle to a designated receptacle, e.g. – or +. The receptacle at center is waste i.e. unrecognized.
Operation Reliability Algorithm is proposed to measure the operation reliability in terms of purity, recovery, and yield. This will be the base to evaluate different operation scenarios to select the operation scenario with highest reliability factor. Sorting definitions are shown in Appendix 1.
Hazardous Operation Scenarios of MoFlo Cell Sorter To avoid the scattering of the biological particles, which might be dangerous to operators and humans, a very high voltage of 3000 Volts, is used in the cylinder around the sample probe. This high voltage is controlled by switch and lamp. If, for any reason, the operator forgot the lamp and tries to touch the sample, it might cause death cases. This requires some design changes in the current system, to either switch off the high voltage once human hands or body is close, or by providing surrounding isolator to protect operator or humans. Another hazardous situation might occur when the laser beam, which is used to excite the fluorochromes, hits the body or eyes of human. It causes harmful effect on human body. To avoid such cases, design changes might be recommended and/or operation regulations must be specified to restrict operators to avoid the direction of the laser beams while operation, as shown in Figure 13. Operation engineering environment is proposed to design and ensure failsafe operation using the concept of formal methods, which links system design with microinstructions of the underlying MoFlo system. Another example of safety requirements is in the illumination chamber. Illumination table houses the illumination chamber, which contains risk on operator and environment. System design and operation design are required.
Figure 14. Modeling framework of multimedia objects
Muing and Viuaion Figure 14 shows the mappings between activities within the operational systems and the associated data elements. Data elements include operational data, maintenance data, etc. Operational data elements include SOP (or standard operating procedures), sensor, alarm, simulation data, etc. These data elements can be visualized in different formats based on predefined preferences such as textual format, 2D, 3D, etc. For example, SIS or safety instrumented system is linked with data elements such as sensor data which can be visualized in 2D or 3D.
Prooo Computer-Aided Operation Engineering Environment Computer-aided operation engineering environment CAPE-ModE (Gabbar et al., 2002) is proposed to capture MoFlo system model and associate data and knowledge elements related to: structure, function, behavior, and operation model elements. Figure 15 shows the system topology and function modeling editor within the proposed computeraided engineering environment. User menus are developed to enable the designer, engineer, and operator to analyze the different lifecycle views for the underlying MoFlo.
Figure 15. MoFlo process design within computer-aided process engineering environment
Operator Interface Sstem Operator interface can be first modeled using task-activity-data-system relationship, as shown in figure 16. In such approach, different operator activities are defined and modeled using basic activity model symbols. For each activity, set of tasks are defined in more details along with the associated data elements and linked systems. There are several systems that usually used by operator. For example, ERP or enterprise resource planning is used for resource planning and management such as human resources, tools, etc. CAD is computer-aided design which is usually used by engineers. MOC or management of changes is a system used to trace and manage the required authorization for different changes during plant operation. CAPE-SAFE is an integrated enterprise safety management system (Gabbar, 2001), which is proposed to manage safety activities throughout plant life cycle. The proposed modeling approach will enable understanding different activities and map to the different automated systems that appear within operator interface. For example, operator activities can be defined for cases where fault or abnormal situation is detected. In such cases, operator validates sensor data, simulation data, operation history, and use incident / accident data to confirm possible root causes and consequences. FDS or fault diagnostic system was proposed by Gabbar (2006) to do such task and provide list of possible fault propagation scenarios. For each activity, list of tasks are identified and related information elements are listed along with the reference systems that owns / maintains such information elements. For example, sensor data can be detected from DCS system. Providing such list will enable the reengineering of operator interface systems. Also it will enable the engineering of virtual plant environment with suitable visualization and interaction mechanisms, such as 2D / 3D diagrams, voice, animation, textual, video, images, etc.
Figure 16. Proposed interface system modeling
Modeling Human Factors One critical factor to the design of virtual plant is the analysis of human factors that are related to the different activities of plant operation (Dyke, 1988; Nachreiner, 2006). The proper analysis of human factors will ensure practical design of virtual plant environment. The proposed approach is based on considering list of human factors that affect each operation activity, as shown in the activity-task-information model in figure 3. Such approach will enable building complete repository of all related human factors that affect the different activities of plant operation. For example, operation tasks that require visualization and interaction mechanisms should be supported with suitable virtual plant elements that reduce human errors related to failure to recognize, failure to percept, or failure to interact. List of human factors are gathered from literature, industrial reports, and previous research work and classified and linked with operation tasks and activities, as shown in table 1 (Nishiyama, 2006; Naval Safety Center, n.d.; Sasou, n.d.). Such classification can be evaluated qualitatively and quantitatively, which will support the selection of different design features of the target virtual plant. Detailed quantitative human performance analysis is described in several references (Khan, 2006). Based on human factor analysis, essential design features of the target virtual plant are identified so that it can reduce risks and ensure process safety. The different human errors are mapped to operator activities, data elements, and visualization format, as shown in figure 17. Human factors are used to define and select data visualization, which is essential to design any user interface (Hall, 2001). In addition to
Table 1. Human factor classifications Human Errors with Alarms / Equipment / Simulator / Interface Systems:
A: Failure to read / cognize data / messages / measures B: Failure to take right decision / link information from different windows C: Wrong response to emergency D: Improper procedure executed E: Physical stress of using mouse / keyboard / eye movement, i.e. switch among different windows F: Loss of situation awareness G: Failure to backup data H: Failure to conduct an adequate brief I: Inadequate supervision errors, inappropriate or absent supervision J: Failure to track performance measures K: Inadequate documentation from different systems / windows
Figure 17. Knowledge structure of human-based interface systems
the visualization of operator activities, operator awareness and intention can also be considered (Feibush, 2000; Overbye, 1997). For example, to overcome the human error of "failure to read" during recovery operation a suitable visualization format can be selected such as sound/image. Similarly, when failure to read is associated with upper pressure limit, sound or bell should be used to ensure safe operation. Based on the described virtual plant modeling approach, the following section describes the proposed integrated operator interface, which includes human factors and process requirements, with the ultimate goal of enabling operator to comprehend plant and process conditions in real time basis.
Con Semantic mining is the current and future approach to achieve intelligent systems. During the last era several techniques are proposed to structure domain knowledge using ontological engineering methods and practices. The logical next step is to seek advanced techniques to learn from such constructed knowledge via data mining. One more advanced step is to extract meta-knowledge via semantic mining where it is essential to construct intermediate level of meta-knowledge that is used to facilitate the learning practices. On the other hand, process engineering has been tightly linked with semantic mining in two aspects: (1) the analysis, design, and implementation of semantic mining support systems are best designed by considering the analysis of semantic mining process and constructing corresponding process models; (2) semantic mining is used to support process engineering for different applications, such as chemical, petrochemical, biological, social, and service processes. This chapter explained techniques and methods to deal with both tracks with examples from biological process using MoFlo cell sorter advanced equipment. The selection of cell sorter as a case study is critical which was motivated by the fact that biological data are in large numbers and lack of adequate meanings to basic data elements. This requires construction of meta-data and meta-rules about basic data elements which facilitate the semantic mining process. The proposed POOM or process object-oriented modeling methodology offers comprehensive yet simple way to construct detailed process models while structuring domain knowledge in systematic way that facilitates decision making related to life cycle activities. In order to realize practical computer-aided process engineering environment it is essential to formally define meta-language and language (called engineering formal language or EFL) that adequately describe process models and activities. In addition, meta-rules and rules are constructed to completely describe the underlying process models. Parser and editors are developed in view of EFL and rules / meta-rules to facilitate the semantic mining of the underlying process. The proposed semantic mining framework can be applied to processes from different disciplines to manage and support life cycle activities.
ACKNOWLEDGMENT This work is implemented with the support of University of Toronto (Canada) and Okayama University (Japan).
R ANSI/ISA-TR84.00.02, Part 1 Safety Instrumented Functions (SIF). ANSI/ISA-S88.01, 1995. Batch Control. Part 1. Models and terminology. Arzen, K.E. (1994). Grafcet for intelligent supervisory control applications. Automatica, 30(10), 15131525. Astesiano, E. & Reggio, G. (2000). Formalism and method. Theoretical Computer Science, 236, 3-34. Chadha, H.S., Baugh Jr, J.W. & Wing, J.M. (1999). Formal specifications of concurrent systems. Advances in Engineering Software, 30, 211-224. Dyke, F.H. (1988). The human dynamics of operator interfaces. IEEE Conference Record of 1988 Fortieth Annual Conference of Electrical Engineering Problems in the Rubber and Plastics Industries (pp. 36-39). Eason, G., Noble, B., & Sneddon, I. N. (1955). On certain integrals of Lipschitz-Hankel type involving products of Bessel functions. Phil. Trans. Roy. Soc., A247, 529-551. Falla, M. (n.d.) Results and Achievements from the DTI/EPSRC R&D Programme in Safety Critical Systems. Feibush, E., Gagvani, N., & Williams, D. (2000). Visualization for situational awareness. IEEE Computer Graphics and Applications, 20(5), 38- 45. Gabbar, H.A. (2005). FDS: Fault Diagnostic System, Technical Notes, Okayama University. Retrieved from, http://syslab2.mech.okayama-u.ac.jp/staff/gabbar/fds.html. Gabbar, H.A. (2006). Design of Integrated Fault Diagnosis System (FDS). Paper presented at International Conference of IEEE SMC, Taiwan. Gabbar, H.A., Aoyama, A., & Naka, Y. (2004a). Model-Based Computer-Aided Design Environment for Operational Design. Journal of Computers & Industrial Engineering, 46(3), 413-430. Gabbar, H.A., Aoyama, A., & Naka, Y. (2004b). Recipe Formal Definition Language For Operating Procedures Synthesis. Journal of Computers & Chemical Engineering, 28(9), 1809-1822. Gabbar, H.A., Chung, P.W.H., Suzuki, K., & Shimada, Y. (2000). Utilization of unified modeling language (UML) to represent the artifacts of the plant design model. In Proceedings of "PSE Asia 2000" International Symposium on Design, Operation and Control of Next Generation Chemical Plants, PS5 (pp. 387-392). Kyoto-Japan. Gabbar, H.A. & Naka, Y. (2003, August). Computer-Aided Operation Design Environment for Chemical Production Plants. Paper presented at ICCTA'2003 – IEEE, 12th International Conference on Computer Theory and Applications, Alexandria, Egypt. Gabbar, H.A., Shimada, Y. & Suzuki, K. (2002). Computer-Aided Plant Enterprise Modeling Environment (CAPE-ModE) – Design Initiatives. Computers in Industry, 47(1), 25-37.
Gabbar, H.A., Yamashita, H., & Suzuki, K. (2006). Integrated Plant Maintenance Management Using Enhanced RCM Mechanism. International Journal of Performability Engineering, 2(4), 369-381. Hall, S.K., Cockerham, K.J., & Rhodes, D.J. (2001). Applying human factors in graphical operator interfaces. In 2001 Conference Record of Pulp and Paper Industry Technical Conference (pp. 241-246). Huang, H.M.. (1996). Operator interface and situation perception in hierarchical intelligent control: a case study. In Proceedings of the 1996 IEEE International Symposium on Intelligent Control (pp. 68-73). Human Factor Analysis and Classification System Presentation Preview. Naval Safety Center. Retrieved from, http://www.safetycenter.navy.mil/PRESENTATIONS/aviation/hfacs.htm IEC 61508. Proposed framework for addressing human factors in IEC 61508. Jamieson, G.A. & K.J. Vicente (2005). Designing effective human-automation-plant interfaces: A control-theoretic perspective. Human Factors, 47, 12-34. Johnsson, C. & Arzen, K.E. (1998). Grafchart for recipe based batch control. Computers & Chemical Engineering, 22, 1811-1228 Khan, F.I., Amyotte, P.R., & DiMattia. D.G. HEPI: A new tool for human error probability calculation for offshore operation. Safety Science, 44(4), 313-334. Kirkwood, R.L., Locke, M.H., & Douglas, J.M. (1988). A prototype expert system for synthesizing chemical process flowsheets. Computers & Chemical Engineering, 12(4), 329-343. Lakshmanan, R. & Stephanopoulos, G. (1990). Synthesis of operating procedures for complete chemical plants – I Hierarchical, structured modelling for nonlinear. Computers & Chemical Engineering, 14(3), 301-317. Meliopoulos, A.P.S., Cokkinides, G., Beker, B., & Dougal, R. (2000). A new tool for visualization and animation of power component and system operation System Sciences. In Proceedings of the 33rd Annual Hawaii International Conference on System Sciences (pp. 4022). Nachreiner, F. Nickel, P., & Meyer, I. (2006). Human factors in process control systems: The design of human–machine interfaces. Safety Science, 44(1), 5-26. Nishitani, H., Kawamura, T., & Suzuki, G. (2000). University-industry cooperative study on plant operations. Computers and Chemical Engineering, 24, 557-567. Nishiyama, K. (2005). Development of human factor analysis system for human errors prevention. Unpublished Master Thesis, Okayama University, Japan. Overbye, T.J., Gross, G., Laufenberg, M.J., & Sauer, P.W. Visualizing power system operations in an open market. Computer Applications in Power, 10(1), 53-58. Rao, M., Sun, X, & Feng, J. (2000). Intelligent system architecture for process operation support. Expert Systems with Applications, 19, 279-288. Sasou, K. Human factor analysis on criticality accident, Human Factors Research Center, CRIEPI, Japan.
APPENDIXndi (1) These are the detailed equations of calculating performance parameters of MoFlo systems.
Purity (%) =
Number - of - particles - of - int erest - in - sorted - sample Total - number - of - particles - in - sorted - sample
Recovery (%) =
Number - of - particles - of - int erest - in - sorted - sample Number - of - particles - of - int erest - sorted - as - indicated - on - sort - counter Yield (%) =
Number - of - particles - of - int erest - in - sorted - sample Number - of - particles - of - int erest - in - original - sample
Intuitive Image Database Navigation by Hue-Sphere Browsing Gerald Schaefer Aston University, UK Simon Ruszala Teleca, UK
aBSTRACT Efficient and effective techniques for managing and browsing large image databases are increasingly sought after. This chapter presents a simple yet efficient and effective approach to navigating image datasets. Based on the concept of a globe as visualisation and navigation medium, thumbnails are projected onto the surface of a sphere based on their colour. Navigation is performed by rotating and tilting the globe as well as zooming into an area of interest. Experiments based on a medium size image database demonstrate the usefulness of the presented approach.
INTRODUCTION Efficient and effective ways for managing and browsing large image databases are increasingly sought after. This is due to the sheer explosion of availability of digital images in the last few years. Nowadays, the sizes of home user's image collections are already typically in the 1,000s while professional image providers overlook databases in excess of 1,000,000 images. Common tools display images in a 1-dimensional linear format where only a limited number of thumbnail images are visible on screen at any one time, thus requiring the user to search back and forth through thumbnail pages to view all images and locate the relevant ones. Obviously, this constitutes a time consuming, impractical and exhaustive Copyright © 2009, IGI Global, distributing in print or electronic forms without written permission of IGI Global is prohibited.
Intuitive Image Database Navigation by Hue-Sphere Browsing
way of searching for images, especially in larger catalogues. Furthermore, the order in which the pictures are displayed is based on attributes like file names and does not reflect the actual image contents and hence cannot be used to speed up the search. Recently, several approaches have been introduced which provide a more intuitive interface to browsing and navigating through image collections (Ruszala & Schaefer, 2004). The basic idea behind most of these is to place images which are visually similar, as established through the calculation of image similarity metrics based on features derived from image content, also close to each other on the visualisation screen, a principle that has been shown to decrease the time it takes to localise images of interest (Rodden et al., 1999). One of the first approaches was the application of multidimensional scaling (MDS) (Kruskal & Wish, 1978) used to project images being represented by high dimensional feature vectors to a 2-dimensional visualisation plane (Rubner, Guibas, & Tomasi, 1997). In the PicSOM system (Laaksonen et al., 2000) tree-structured self organising maps are employed to provide both image browsing and retrieval capabilities. In (Krishnamachari & Abdel-Mottaleb, 1999) a hierarchical tree is employed to cluster images of similar concepts while the application of virtual reality ideas and equipment to provide the user with an interactive browsing experience was introduced in (Nakazato & Huang, 2001). In this chapter we present a simple and fast approach to image database navigation. All images are projected onto a spherical globe; navigation through the image collection is performed by rotation of the sphere and zooming in and out (Schaefer & Ruszala, 2005). The use of a spherical object is not a coincidence, rather it stems directly from the type of features that are used for navigation. We utilise the median hue and median brightness (in HSV colour space) to calculate a pair of co-ordinates for each image in the database. As hue describes a circular quantity (0°=360°) whereas brightness is not, a sphere is a natural choice of geometrical body to encapsulate the combination of these two. The proposed method hence provides an effective, intuitive and efficient interface for image database navigation as is demonstrated on a medium sized image collection (Schaefer & Stich, 2004).
RELATED WORK Several approaches which provide a more intuitive interface for image database navigation, compared to the traditional linear display of thumbnails, have been recently introduced in the literature. Rubner et al. were among the first (Rubner, Guibas, & Tomasi, 1997) and suggested the application of multidimensional scaling (MDS) (Kruskal & Wish, 1978) to calculate the locations of images and displaying them in a global 2-dimensional view on a single screen. Using this method, all images in a database are (initially) shown simultaneously; their locations are dependent on their visual similarity (based on features such as colour, texture or shape descriptors) compared to all other images features in the database. If two images are very similar in content they will also be located close to each other on the screen and vice versa. The user can browse the database easily from a top-down hierarchical point of view in an intuitive way. The main disadvantage of the MDS approach is its computational complexity. First a full distance matrix for the complete database, i.e. all pairwise distances between any two images in the collection, needs to be calculated. MDS itself is then an iterative process which successively re-arranges the locations of each image minimising the distances between images on screen and their actual (feature-based)
database distances. Interactive visualisation of a large number of images is hence difficult if not impossible to achieve. Furthermore, adding images to the database requires re-computation of (part of) the distance matrix and re-running MDS. A computationally more attractive approach was presented by Krishnamachari and Abdel-Mottaleb (1999). They employed a hierarchical structure derived through a clustering process to access the images in a database. The user can navigate the images by traversing the tree structure. Unfortunately the user experience is limited by the fact that it is not possible to gain an overview of the whole database as for example MDS provides. In addition, at any particular level of the tree the child nodes are just displayed in a linear fashion without taking into account their relative similarities. In (Laaksonen et al., 2000) the authors employ self organising maps (SOMs) for establishing image similarity. A hierarchical version of SOM, namely a tree structured self organising map is used both for retrieval and for visualisation purposes where the map itself can be navigated. 3D MARS (Nakazato & Huang, 2001) represents a virtual reality approach to image database navigation. Images are displayed on four walls in an immersive CAVE environment where browsing the database is done using a special wand to select images and make queries. While being an interesting novel idea it requires specialist equipment not available to the average user wanting to explore their personal collections. Although there exists also a desktop version called ImageGrouper which can be used with ordinary CRT displays, using this program is fairly difficult as there is no option for global browsing; only searching for an image first and then browsing from that location in the database is possible.
IMaguaon HUE-SPHERE Our aim is to provide a simple yet effective interface for image database visualisation and navigation. It should allow fast and intuitive browsing of large image collections and not rely on any specialised equipment not available to the average user. We separate our approach into two tasks: selection of suitable features and visualisation of the image collection. Among typical features used for image retrieval those describing the colour content are certainly the most popular ones (Smeulders et al., 2000). We follow this and describe each image by its median colour. However, rather than employing the standard RGB colour space we use the HSV space (Sangwine & Horne, 1998). Of this we take only the hue and value attributes as the saturation descriptor is deemed less important for describing image content. Value, which describes the brightness of a colour is defined in HSV as (Sangwine & Horne, 1998)
V =
R+G + B 3
where R, G, and B are red, green and blue pixel values. V ranges between 0 and 1 where 0 corresponds to pure black and 1 to pure white. Hue "is the attribute of a visual sensation according to which an area appears to be similar to one of the perceived colours, red, yellow, green and blue, or a combination of two of them" (CIE, 1989) and is the attribute that is usually associated as 'colour'. Hue in HSV is defined as (Sangwine & Horne, 1998)
H = cos -1
0.5[(R - G )+ (R - B )] (R - G )(R - G )+ (R - B )(G - B )
It is apparent that hue constitutes an angular attribute; H goes from red to yellow to green to blue back to red and is also often referred to as hue circle. We now want to find a geometrical body that can be used to derive a co-ordinate system for placing thumbnails of the images contained in a database as well as to serve as the actual surface onto which those thumbnails are projected. Looking at the two attributes we have selected, H and V, we almost naturally end up with the body of a sphere, or a spherical globe. The hue circle describes one dimension of the sphere. As all colours with high V values are similar i.e. close to white and the same holds true for those colours with low V which become similarly close to black, and as black and white by definition don't have a hue quality, the two points V=0 and V=1 describe the poles of the globe. The use of a globe not only comes naturally with the choice of features, it also has other clear advantages. The concept of a globe will almost certainly be familiar to the average user as it is a direct analogy of the earth globe. It therefore provides a very intuitive interface to the user who will have experience on how to navigate and find something on its surface. Furthermore it allows us to employ a familiar co-ordinate system based on latitude and longitude. Longitude describes the circumference of the globe (i.e. the east-west co-ordinate) and lies in the interval [0°;360°]. Latitude describes the north-south direction from pole to pole and ranges from -90° at the south pole to +90° at the north pole. Clearly each point on the surface of the globe can be uniquely described by a pair of longitude/latitude co-ordinates. Our approach to visualising image databases is simple and straightforward. It also provides a very fast method for accessing the image collection. Images are transformed to an HSV representation and the median hue and value attributes are calculated. We make use of the median rather than the mean in order to provide some robustness with regards to the image background. Also, we actually calculate the median R, G, and B values and transform those to HSV which is computationally more efficient and results only in a slight deviation in terms of accuracy. From these the co-ordinates on the globe are then determined (see also Figure 1), where H directly translates to longitude and V is rescaled to match the latitude range (V=0.5 corresponds to 0° latitude, i.e. a position on the equator). A thumbnail of the
Figure 1. Hue-value co-ordinate system used
image is then projected onto the surface of sphere at the calculated co-ordinates (since those thumbnails are distorted more towards the poles we actually remap all images to be located between the great circles with ±80° latitude). Once the locations for all images have been calculated and their thumbnails projected, the system shows an initial view of the globe and is ready for navigation. Figure 2 shows the initial view of our image database globe based on the UCID (Schaefer & Stich, 2004) image database which contains some 1300 images. Since the co-ordinates are extracted directly from the images, our approach is very fast and hence much more efficient than methods such as MDS which rely on a comparably slow and computationally expensive iterative convergence procedure. Also does the addition of images to the database require only the computation of its median hue and brightness with no further calculations necessary as is the case for MDS and similar approaches. In addition the features that we employ are intrinsically suitable for query-based image retrieval (Stricker & Orengo, 1995). Furthermore, the axes of the co-ordinate system are well defined and meaningful which is in contrast to those obtained by MDS or other similar techniques (such as principal components analysis) where axes are not associated with attributes. Finally, as the database globe is displayed on an ordinary CRT monitor no specialist equipment is necessary.
IMAGE DATABASE NAaviga As mentioned above the interface starts with an initial view such as the one shown in Figure 2 (which has a central point of 30° longitude, i.e. an average reddish/yellowish hue, and 0° latitude, i.e. medium brightness). From Figure 2 we can also see the controls the user has at his/her disposal for navigation purposes. As these are kept fairly simple and again due to the average user's familiarity of localising places on an earth globe, navigation is straightforward and intuitive.
Figure 2. Initial globe view of the UCID dataset
The controls allow rotation of the globe around both the vertical and the horizontal axis and zooming in and out of a region of interest. The user can perform these operations using the keyboard where the cursor keys are translated to rotation of the sphere and the + and − keys are available for zooming. Similarly the mouse can be used for navigation: moving the mouse prompts a rotation whereas the scroll wheel (or the left and right mouse buttons) are responsible for zoom operations. Especially attractive is the application of a trackball: here the ball itself corresponds directly to the image globe, hence rotating the track ball rotates the sphere on screen whereas the two buttons are again used for zooming in and out. From the previous section it becomes clear that a rotation around the vertical axis will focus on a different average hue of the displayed images. In Figure 3 we show the resulting view after a clock-wise rotation by 90° which causes the average hue of the shown images to shift from red/yellow to greenish. Further rotation reveals images of the remaining hues as shown in Figure 4 which displays the results of two further 90° rotations exhibiting images of bluish and magenta hues respectively. Rotation around the horizontal axis will either shift the display to images that are darker (rotation towards south pole) or brighter pictures (rotation towards north pole). While the global view of the sphere allows for easy selection of a general hue/brightness area, it is clear that in this view single images are hard to make out. Therefore a zoom function is provides which allows to user to restrict their attention to a smaller, more localised area of interest. In Figure 5 we show an example of a zoomed-in area where the user was interested in images with a beach/ocean view. It is clear that further zoom operation(s) can be applied to localise single images of interest. We note, that for none of the operations used for browsing the images, i.e. neither for rotation nor for changing the zoom factor, any additional calculations need to be performed (in contrast to e.g. MDS which will re-calculate all co-ordinates for a zoomed-in area (Rubner, Guibas, & Tomasi, 1997)) as the co-ordinates at which the pictures are placed do not change. We are therefore able to provide an image browsing environment that can operate in real time. Furthermore, while we provide a dedicated appli-
Figure 3. Database view after a rotation around the hue axis by 90°
Figure 4. Globe view with central hue of 210° (left), and 300° (right)
Figure 5. Zoomed-in area displaying beach/ocean images
cation for image database navigation we can also export our image globe to a VRML model (VRML Consortium, 1997) which can then be used and browsed with a suitable viewer.
Con CLUSION We have introduced an efficient and effective approach to visualise and browse large image collections. Thumbnails of images are projected onto a spherical globe which acts as the medium the user interacts with. Navigation is simple, intuitive, and fast as has been demonstrated on a medium sized image dataset. While we have used colour features to determine the position of the thumbnails on the sphere's surface, the method is generic and other types of features can be equally employed.
REFERENCES CIE. (1989). International Lighting Vocabulary. CIE Publications 17.4, Commission International de L'Eclairage, 4th edition, 1989. Krishnamachari, S., & Abdel-Mottaleb, M. (1999). Image browsing using hierarchical clustering. In 4th IEEE Symposium on Computers and Communications. Kruskal, J.B., & Wish, M. (1978). Multidimensional scaling. Sage Publications. Laaksonen, J., Koskela, M., Laakkso, P., & Oja, E. (2000). PicSOM – content-based image retrieval with self organising maps. Pattern Recognition Letters, 21:1197-1207. Nakazato, M., & Huang, T.S. (2001). 3D MARS: Immersive virtual reality for content-based image retrieval. In IEEE Int. Conference on Multimedia and Expo. Rodden, K., Basalaj, D., Sinclair, D., & Wood, K. (1999). Evaluating a visualization of image similarity as a tool for image browsing. In IEEE Symposium on Information Visualization, pages 36-43. Rubner, J., Guibas, L., & Tomasi, C. (1997). The earth mover's distance, multi-dimensional scaling, and color-based image retrieval. In Image Understanding Workshop, pages 661–668. Ruszala, S.D., & Schaefer, G. (2004). Visualisation models for image databases: A comparison of six approaches. In Irish Machine Vision and Image Processing Conference, pages 186–191. Sangwine, J., & Horne, R.E.N. (1998). The Colour Image Processing Handbook. Chapman & Hall. Schaefer, G., & Ruszala, S. (2005). Image database navigation: A globe-al approach. In Int. Symposium on Visual Computing, volume 3804 of Springer Lecture Notes on Computer Science, pages 279-286. Schaefer, G., & Stich, M. (2004). UCID - An Uncompressed Colour Image Database. In Storage and Retrieval Methods and Applications for Multimedia 2004, volume 5307 of Proceedings of SPIE, pages 472–480. Smeulders, A.W.M., Worring, M., Santini, S., Gupta, A., & Jain, R. (2000). Content-based image retrieval at the end of the early years. IEEE Trans. Pattern Analysis and Machine Intelligence, 22(12):1249–1380. Stricker, M., & Orengo, M. (1995). Similarity of color images. In Conf. on Storage and Retrieval for Image and Video Databases III, volume 2420 of Proceedings of SPIE, pages 381–392. VRML Consortium. (1997). The Virtual Reality Modeling Language. ISO/IEC IS 14772-1.
Multimedia Resource Annotation
Formal Models and Hybrid Approaches for Efficient Manual Image Annotation and Retrieval Rong Yan IBM T.J. Watson Research Center, USA Apostol Natsev IBM T.J. Watson Research Center, USA Murray Campbell IBM T.J. Watson Research Center, USA
ABSTRACT Although important in practice, manual image annotation and retrieval has rarely been studied by means of formal modeling methods. In this chapter, the authors propose a set of formal models to characterize the annotation times for two commonly-used manual annotation approaches, that is, tagging and browsing. Based on the complementary properties of these models, the authors design new hybrid approaches, called frequency-based annotation and learning-based annotation, to improve the efficiency of manual image annotation as well as retrieval. Both our simulation and experimental results show that the proposed algorithms can achieve up to a 50% reduction in annotation time over baseline methods for manual image annotation, and produce significantly better annotation and retrieval results in the same amount of time.
INTRODUCTION Recent increases in the adoption of devices for capturing digital media along with the ever-greater capacity of mass storage systems have led to an explosive amount of images and videos stored in personal Copyright © 2009, IGI Global, distributing in print or electronic forms without written permission of IGI Global is prohibited.
Formal Models and Hybrid Approaches for Ef.cient Manual Image Annotation and Retrieval
collections or shared online. To effectively manage, access and retrieve these data, a widely adopted solution is to associate the image content with semantically meaningful labels, a.k.a. image annotation (Kustanowitz & Shneiderman, 2004). Two types of image annotation approaches are available: automatic and manual. Automatic image annotation, which aims to automatically detect the visual keywords from image content, have attracted a lot of attention from researchers in the last decade (Barnard et al., 2002; Jeon et al., 2003; Li & Wang, 2006; Griffin et al., 2006; Kennedy et al., 2006). For instance, Barnard et al. (2002) treated image annotation as a machine translation problem. Jeon et al. (2003) proposed an annotation model called cross-media relevance model (CMRM), which directly computed the probability of annotations given an image. The ALIPR system (Li & Wang, 2006) used advanced statistical learning techniques to provide fully automatic and real-time annotation for digital pictures. Kennedy et al. (2006) considered using image search results to improve the annotation quality. These automatic annotation approaches have achieved notable success, especially when the keywords have frequent occurrence and strong visual similarity. However, it remains a challenge to accurately annotate other more specific and less visually similar keywords. For example, the best algorithm for the CalTech-256 benchmark (Griffin et al., 2006) reported a mean accuracy of 0.35 for 256 categories with 30 examples per category. Similarly, the best automatic annotation systems in TRECVID 2006 (Over et al., 2006) produced a mean average precision of only 0.17 on 39 concepts. Along another direction, recent years have seen a proliferation of manual image annotation systems for managing online/personal multimedia content. Examples include PhotoStuff (Halaschek-Wiener et al., 2005) and Aria (Lieberman et al., 2001) for personal archives, Flickr.com and ESP Game (von Ahn & Dabbish, 2004) for online content. This rise of manual annotation partially stems from its high annotation quality for self-organization/retrieval purpose, and its social bookmarking functionality in online communities. Manual image annotation approaches can be categorized into two types as shown in Figure 1 (details in Section III). The most common approach is tagging, which allows users to annotate images with a chosen set of keywords ("tags") from a vocabulary. Another approach is browsing, which requires users to sequentially browse a group of images and judge their relevance to a pre-defined keyword. Both approaches have strengths and weaknesses, and in many ways they are complementary to each other. But their successes in various scenarios have demonstrated the possibility to annotate a massive number of images by leveraging human power. However, manual image annotation can be tedious and labor-intensive. Therefore, it is of great importance to consider using automatic techniques to speed up manual image annotation. In this work, we assume users will drive the annotation process and manually examine each image label in order to guarantee the annotation accuracy, but in addition we use automatic learning algorithms to improve the annotation efficiency by suggesting the right images, keywords and annotation interfaces to users. This is different from automatic image annotation, which aims to construct accurate visual models based on low-level visual features. But from another perspective, efficient manual annotation can bring benefits to automatic annotation algorithms, because they are typically built on manually annotated examples. To quantitatively analyze and optimize manual annotation, it is necessary to start with some annotation time/efficiency models that formulate the annotation process. In the absence of existing studies, we propose two annotation time models to describe the process for two popular manual annotation approaches, i.e., tagging and browsing. To the best of our knowledge, this is the first attempt to quantify the manual image annotation process, and it serves as a theoretical foundation to analyze large-scale manual annotation without time-consuming user studies. Based on the time models, we further propose two hybrid annotation algorithms – frequency-based annotation and learning-based annotation – by
Formal Models and Hybrid Approaches for Efficient Manual Image Annotation and Retrieval
Figure 1. Examples of manual image annotation systems. Left: tagging (Flickr and ESP Game), Right: browsing (EVA and XVR).
merging tagging and browsing into a unified framework. They can automatically learn from low-level features and adaptively chooses browsing/tagging interfaces for the right set of keywords/images. Both our simulation and empirical results on the TRECVID'05 (Over et al., 2006) and Corel (Barnard et al., 2002) collections confirm the validity of the time models, as well as demonstrate that the proposed algorithms can achieve an up to 50% reduction in annotation times and can consistently outperform the baseline methods in the entire annotation process. By extending similar mechanism to the interactive retrieval task on TRECVID'07 collection, we also observed a considerable improvement on the retrieval effectiveness under the same amount of retrieval time.
REL Applying automatic processing methods to assist manual image annotation is related to a large body of previous work. For example, existing systems have leveraged speech recognition, external semantic networks or time/location information to facilitate the manual annotation process (Kustanowitz & Shneiderman, 2004). However, since these systems require support from additional sources, their applicability is more restricted than the automatic methods that only use information from image content. From the learning perspective, relevance feedback also explores the idea of using learning algorithms to refine the ranking results. MARS (Rui et al., 1997) is one of the first image retrieval systems using relevance feedback. Recent relevance feedback approaches have applied more advanced machine learning tools such as SVMs and kernel biased discriminant analysis (Zhou & Huang, 2001). MiAlbum (Wenyin et al., 2001) uses such a feedback mechanism to iteratively improve annotation quality for personal im-
ages. But in contrast to relevance feedback, which needs to work on one query at a time, learning-based annotation should be able to simultaneously operate across multiple keywords and dynamically switch to any keywords or interfaces in the learning process. The idea of active image/keyword/interface selection is also related to the methods of active learning (Tong & Chang, 2001), which iteratively asks a user to review and provide the correct labels for some selected unlabeled data. Typically, the unlabeled examples can be selected by means of either minimization of the learner's expected error or maximization of information gain (Tong & Chang, 2001). The effectiveness of active learning for reducing annotation cost in automatic image annotation has been demonstrated by previous work (Yan & Hauptmann, 2004). Despite the connections on active image selection, learning-based annotation has some fundamental differences with active learning, because it mainly aims to collect a maximal number of annotations based on the time models, instead of attempting to build accurate visual models on low-level features. This leads to several implementation differences, such as image selection strategies, base learning models and so on. Moreover, learning-based annotation needs to investigate how the learning algorithm can be applied in a hybrid annotation environment, but this is rarely discussed in active learning.
MANUAL IMAGE ANNOTATION METHODS AND TIME MODELS In this section, we introduce and discuss two types of manual image annotation approaches, i.e., tagging and browsing. We also propose two models to measure their annotation efficiency, which offers the foundation for the rest of our discussions. Formally, let us suppose we have to annotate a set of images I={Il}l=1..L with a set of keywords W={W k}k=1..K.1 Without loss of generality, we sort {W k} in a descending order of their frequency. So W1 is the most frequent word and WK is the least frequent one. Lk is the number of relevant images for W k, and Kl is the number of keywords associated with Il. The goal of manual annotation is to identify the relevance between each pair of image Il and keyword W k, or equivalently, annotate image Il with keyword W k. Once the relevance between Il and all the keywords have been identified, we say Il is annotated, otherwise Il is unannotated.
Tagging Tagging allows the users to annotate images with a chosen set of keywords("tags") from a controlled or uncontrolled vocabulary. This type of approaches is the basis for most of the current image annotation/ tagging systems, although it can be implemented in a variety of ways with respect to interface designs and user incentives. For example, Flickr encourages users to create free-text tags for each uploaded image. It views tags as the central component for sharing, retrieval and discovery of the user-generated content. ESP Game (von Ahn & Dabbish, 2004) motivates users to annotate photos with freely chosen keywords in a gaming environment. PhotoStuff (Halaschek-Wiener et al., 2005) provides a domainindependent tool that facilitates creating and publishing annotations for image content. One advantage for tagging is that annotators can use any keywords in the vocabulary or freely choose arbitrary words to annotate the target images. However, this flexibility might result in a "vocabulary problem" (Furnas et al., 1987), which means multiple users or a single user in a long period can come up with different words to describe the same concept. Moreover, it can be more time-consuming for users to type new keywords, as compared with simply browsing and judging the relevance between images and pre-defined keywords. 275
In order to quantitatively analyze the efficiency of tagging approaches, we must design a formal model to represent its expected annotation time for each image. To begin, we can assume that the more keywords users annotate, the larger the annotation time is. Our user study described in Section VII-A confirmed that this assumption is reasonable, but it also shows that the annotation time is not exactly proportional to the number of keywords. This is because, for each image, users always need additional time up-front to understand the image content in order to make their decisions. The above observations suggest modeling the tagging time Tl for the lth image as a function of four major factors, i.e., the number of image keywords Kl, the average time for designing/typing one word tf, the initial setup time for annotation ts and a noise term ε, which follows a zero-mean probability distribution. Based on our user study, we find it is sufficient to adopt a linear time model to represent the annotation time for each image, i.e., Tl =Kltf +ts+ε. Its mean can be derived as tl =Kltf +ts. For a total of L images, the overall expected annotation time is L
t = ∑ K l t f + Lts l =1
or t = ∑ Lk t f + Lt s
These two formulations are equivalent because L
∑ Kl= ∑ Lk ,
but they compute the total number of keywords from two orthogonal dimensions. We mainly consider the second formulation in the following analysis. Note that we assume users can find all relevant keywords – or equivalently, annotate every keyword – in one tagging step. It might be more reasonable to consider the possibility of missing keywords in the tagging process. This issue could be mitigated, however, by using keyword auto-completion interfaces and specific-to-general label propagation techniques. Moreover, due to a lack of statistics to support the modeling of missing keywords, we will pursue this direction in the future. Another point worth mentioning is that this time model does not require the parameters te and tf to be constant in all the annotation scenarios. Instead, they can be affected by a number of factors, such as interface design, input device, personal preference and so on. For example, annotation on cell phones (Wilhelm et al., 2004) can have a much larger tf than annotation on desktop computers. Therefore, rather than attempting to estimate fully accurate parameters for any specific settings, this paper mainly focus on examining the correctness of the proposed time models, and use them as a foundation to develop better manual annotation algorithms. We expect the proposed time models and the following analysis will generalize over a wide range of settings. Our experiments also evaluate the sensitivities of annotation performance over these parameters. Meanwhile, it is useful to re-estimate these coefficients for either a new user or a new environment.
Browsing Another type of annotation approach, browsing, requires users to browse a group of images, so as to judge the relevance of each image to a given keyword. The number of images per group can vary from 1 to a large number such as 20. Because browsing annotation needs to start with a controlled vocabulary defined by domain experts or a seeded keyword manually initialized by users, it is not as flexible and as widely applied as tagging. However, browsing has advantages on several aspects. For instance, it allows users to provide more complete annotation outputs than tagging (Volkmer et al., 2005), because
users only focus on one specific keyword at a time and they do not need to remember all possible keywords over a long period. Moreover, the time to annotate one keyword by browsing is usually much shorter than that in tagging, since users have a relatively simple binary judgment interface and a stable annotation context in the entire process. Therefore, recent years have seen more and more browsing annotation systems being developed. One such example is the Efficient Video Annotation (EVA) system (Volkmer et al., 2005), which allows multiple users to collaboratively annotate the same image collection by browsing and identifying keywords from a controlled vocabulary. Extreme video retrieval (XVR) (Hauptmann et al., 2006), although motivated by video search, follows a similar idea by asking users to quickly browse the automatic search results to judge their relevance. Similar to tagging, we design a formal model to quantify the efficiency of browsing. First, the overall annotation time should be related to the number of images and the number of unique keywords. According to Section VII-A, we also find that the time for annotating a relevant image is significantly larger than the time for skipping an irrelevant image, because users tend to spend more time in examining the correctness on relevant images. Thus we model the browsing annotation time Tk for the kth keyword using four major factors, i.e., the number of relevant images Lk, the average time to annotate a relevant image tp, the average time to annotate an irrelevant image tn and a zero-mean noise term ε. The number of irrelevant images is simply L– k =L – Lk and hence a reasonable linear time model is Tk =Lktp+(L – Lk)tn+ε. For a total of K keywords, the overall expected annotation time is K
t = ∑  Lk t p + ( L - Lk )tn .
To summarize, these two annotation approaches are essentially complementary from many perspectives. For example, tagging has fewer limitations on the choice of words and users only need to consider relevant keywords for each image. But the annotated words must be re-calibrated due to the vocabulary problem. It also requires more time to determine and input the given keyword. On the contrary, browsing must work with one pre-defined keyword at a time and requires users to judge all possible pairs of images/keywords. But the effort to determine image relevance by browsing is usually much less than that by tagging, i.e., tp, tn is typically much smaller than tf, ts. Therefore, tagging is more suitable for infrequent keywords such as specific person/location names, and browsing works better for frequent keywords such as "person"/"face".
FREQcy-Based AnnotaT The complementary properties of tagging and browsing provide an opportunity to develop more efficient algorithms for manual image annotation by merging their strengths. Because our analysis suggests that tagging/browsing is suitable for infrequent/frequent keywords respectively, we develop a hybrid annotation algorithm called frequency-based annotation, which first partitions the keyword space into two sets based on word frequency and then applies the best annotation strategy to label each set of words. To illustrate the basic idea, we can view the image annotation task as a problem of filling binary relevance information in an empty matrix of size K×L. In this case, tagging is equivalent to annotating the matrix row by row, as shown in Figure 2(a), and browsing is equivalent to annotating column by
column, as shown in Figure 2(b). However, since neither of these approaches are always ideal for the entire keyword space, it is useful to split the keywords into two disjoint sets shown in Figure 2(c). In order to reduce the annotation time, we can apply browsing for the frequent (or general) keywords in the left set, and apply tagging for the infrequent (or specific) keywords on the right set. By doing so, users can quickly browse through the frequent words in a short time and leave the rest of keywords to be picked up by tagging. To estimate the best boundary to split the keyword space, we can minimize the total expected annotation time as a function of the boundary B. The annotation time consists of two parts including the browsing annotation cost for the words more frequent than wB, and the tagging annotation cost for the words less frequent. Based on the time models presented in Eqn(1) and Eqn(2), we have the total expected time as follows, B
k = B +1
t ( B) = ∑  Lk t p + ( L - Lk )tn  + ∑ Lk t f + Lt s = LBtn + Lt s + ∑ Lk (t p - tn ) + ∑ Lk t f .
Figure 2. Illustration of four image annotation approaches based on a matrix representation, where each row stands for one image, and each column stands for one unique keyword.
(a) Tagging
(b) Browsing
(c) Frequency-based
(d) Learning-based
To obtain the boundary that achieves the minimal annotation time, we can compute the largest B that satisfy t(B)≤t(B–1), or equivalently, the largest B satisfying
tn LB . > L t f + tn - t p
Algorithm 1 summarizes the frequency-based annotation algorithm, where all the frequency information Lk is assumed to be known and the keywords W k are sorted in the descending order of word frequency. The first step of this algorithm is to estimate the optimal boundary B based on either Eqn(4). The choice of the equations depends on whether the detailed frequency information is available, because Eqn(4) can provide a more accurate estimate, but it needs the exact frequency Lk as inputs. After B is determined, the algorithm requires users to annotate all the images via browsing for the top-B frequent keywords, followed by tagging all the images with the rest of the keywords. Finally, it outputs the annotations for all the images.
LE-Based AnnotaT The frequency-based annotation provides a simple hybrid framework to merge the advantages of tagging and browsing, but it requires all the frequency statistics to be available in order to determine the most frequent keywords. Unfortunately, accurate frequency statistics are not always available. Moreover, frequency-based annotation limits itself to use browsing annotation only for frequent keywords, which ignores the potential chance for improving efficiency on the other infrequent keywords. Therefore, we propose another approach called learning-based annotation which can automatically learn from visual features and adaptively choose annotation interface for the right set of keywords/images. Figure 2(d) illustrates the basic idea of the learning-based annotation using the same annotation matrix as before. The algorithm starts by tagging some initial selected images. With more annotations collected, it attempts to dynamically find a batch of unannotated images for potential keywords and ask users to annotate them in a browsing interface. The algorithm will iterate until all the images are
shown in the tagging interface so as to guarantee none of the keywords is missing. This is in contrast to frequency-based annotation, which determines a fixed set of keywords for browsing before the annotation process. The advantages for the learning-based methods are two-fold: 1) it does not need to re-order the keyword space by frequency; 2) even for infrequent keywords, it can potentially discover a subset of images that are mostly relevant for them, and improve the annotation efficiency by switching to a browsing interface.
Analysis To design the learning-based annotation algorithm, it is instructive to study when it can have a smaller annotation time than simple tagging/browsing. Let us first break down its total annotation time by keywords, or equivalently, column by column in the annotation matrix. Suppose the proposed algorithm has obtained Lk relevant labels for keyword W k. Then we can assume βkLk relevant labels come from browsing and the other (1–βk)Lk labels come from tagging, where βk is called browsing recall. However, βk is not enough to describe the total annotation time, because, based on the proposed time models, browsing irrelevant images will also introduce additional cost. Therefore we need to introduce browsing precision γk to represent the proportion of relevant images in all the browsed images. In this case, the number of irrelevant browsed images is βkLk (1–γk)/γk and the total annotation time is, K  t = ∑ k =1 
 1- k  Lk  t p + tn  + (1 k  
 ) Lk t f  + Lts . 
Our analysis in the Appendix shows that if an annotation algorithm is more efficient than tagging/ browsing, its browsing recall and precision must satisfy
 1 - abk ≥ max  0,  1- a
 , 
 a k ≥ max  a,  abk - (1 -
  k)
where a=tn /(tf +tn–tp), bk =L/Lk. These inequalities offer us more insights to develop efficient annotation algorithms. Eqn(7) suggests that for keyword W k, we should use browsing to annotate at least (1–abk)Lk/(1–a) relevant images. Otherwise, simple browsing/tagging annotation can be a better choice in this case. Furthermore, it also indicates that the browsing precision should be larger than a lower bound related to a, bk and βk. To summarize, for a given keyword, an efficient annotation algorithm should be able to select a sufficient number of images for users to browse, and a sufficient proportion of these browsed images should be relevant to the predicted keyword. Based on above analysis, we should switch to browsing interface if we expect the browsing precision can at least be higher than the lower bound (1–abk) /(1–a). Therefore, the problem boils down to adaptively finding a set of unannotated images that are likely to be relevant for a given keyword based on the existing annotation. To this end, we translate manual annotation into an online learning problem, i.e., for keyword W k, learn the visual model from available relevant images I'k ={I'j}j=1..m', and use it to predict additional relevant images from the unannotated image pool U'k ={Il|Il.I'k}. To enable this learning process, we assume each image I is associated with a number of low-level features x. Based on the
annotation provided by users, we can learn their visual patterns by using kernel logistic regression, which aims to optimize the following empirical risk function for keyword W k, m′
R ( f ) = ∑ log(1 + e − yi f ( xi ) ) + λ ‖f 2‖ , i =1
where xj is the feature for I'j, yk∈{–1,1} is binary relevance label, and H denotes a reproducing kernel Hilbert space(RKHS) generated by an arbitrary positive definite kernel K. According to the representer theorem, the relevance of unannotated images can be estimated from the minimizer f(x) with weights αi for each training example, m′
f (·) = ∑ i K (xi ,·)
When users annotate an additional label (ym,xm),m=m'+1, the optimization function must be updated accordingly. To reduce the computational demand, only the weight for the new example is updated based on the Newton-Raphson method. Since the optimization function is convex, the Newton method can guarantee to find the global optimum. To be more specific, the gradient and Hessian of the risk function with respect to αm can be written as,
∂R( ) ∂ 2 R( ) = K mT p + K mT , = K mT WK m + K mm 2 ∂ m ∂ m
where K is the kernel Gram matrix, Km is the vector of {K(xm,.)}, Kmm is the element of K(xm,xm), p denote the logistic model 1/(1+exp(–Kα)), and W denote the matrix diag(pi(1–pi)). The Newton updates can be straightforwardly derived from the gradient and Hessian function. These updates are iterated until the risk function converges or the iteration number is larger than a threshold. In our implementation, we select the RBF kernel to model non-linear decision boundary between positive/negative examples. The kernel matrix is pre-computed before the annotation process in order to reduce online computational resources. Finally, after the optimal weight α m is found, we can simply update the prediction function by f(•)←f(•)+α mK(xm,•).
Algorithm Details The learning-based hybrid annotation is summarized in Algorithm 2. This algorithm starts with a number of unannotated images and a vocabulary of keywords. It first selects an image from the unannotated pool for tagging. After this image is completely tagged by the user, all the related variables are updated for its corresponding keywords W k, i.e., the set of relevant images I'k, the kernel weight α as well as the prediction function f k (•) based on Eqn(8). By thresholding the prediction function, we generate the set of estimated relevant images Rk that represent the most potentially relevant images identified by the system. If the number of these images is larger than a pre-defined batch size S, the browsing interface is activated to annotate all the images in Rk. To avoid switching interfaces too frequently and disturbing the user experience, we typically set the batch size to be a large number and only invoke the browsing interface when there are a large number of relevant images available.
Since the set of estimated images are not guaranteed to be relevant, we use the new image annotations to predict the browsing precision and update the learning parameters accordingly. As shown in Eqn(7), browsing precision needs to be sufficiently large for the proposed algorithm to improve efficiency. Therefore, in the next step, if we find the browsing precision is larger than a lower bound, we will reduce the adaptive threshold θk and thus the browsing interface can be used to annotate as many images as possible until the browsing precision is too low. The bound, i.e., max(a,m'k/mk), is derived from Eqn(7) with βk =1 and m'k/mk to approximate the true frequency ratio Lk/L, where m'k is the number of relevant images and mk is the total annotated images for W k so far. Above steps will be iterated until all the images are tagged.
APPLcaTteractive RRETREl Although the aforementioned approaches are discussed in the context of manual image annotation, its idea and analysis can be simply extended to the interactive retrieval systems (Hauptmann & Witbrock, 1996; Snoek et al., 2004; Foley et al., 2004), which provide a means to help users to look up image/video using associated text and visual content in an interactive manner. Interactive retrieval can be categorized into three categories, • •
A general class of users aims at browsing over a large number of videos from diversified sources, where users have no specific target except for finding interesting things. Another class of users wants to do arbitrary search by retrieving an arbitrary video satisfying his information need that can be presented by text keywords or visual examples.
The third class of users, complete search/annotation, aims to discover every relevant video that belong to a specific information need. To support these users, the retrieval systems must possess more automatic processing power to reduce the huge manual annotation efforts.
Since the last paradigm is often time consuming and labor extensive, there are a number of interactive retrieval approaches that have been developed to reduce the overall retrieval effort in this case, such as Informedia (Hauptmann & Witbrock, 1996), MediaMill (Snoek et al., 2004), Fischlar (Foley et al., 2004) and so on. However, all of these interactive retrieval systems consider dealing with one query topic at a time. But it is not uncommon that users want to search for multiple related query topics simultaneously. For example, a security officer might want to search the surveillance video collection for a list of suspicious persons or a list of abnormal activities. Since multiple queries does not exist in isolation, it is possible to exploit the redundancy between queries so as to reduce the total retrieval time, or in other words, improve the retrieval performance within the same amount of time. In this sense, we can view multi-query retrieval as a manual image/video annotation task, which aims to annotate the entire collection with the given keywords/queries. Therefore, by treating the query topics as keywords, studies on manual image annotation approaches can bring us deeper insights on improving the performance of multi-topic interactive retrieval. In the following discussions, we describe the interface design of the proposed multi-topic image/video retrieval system using hybrid learning-based annotation. The main components of the retrieval system consist of a single-image tagging interface and a single-query browsing interface. Figure 3 shows a screenshot of the tagging interface that aims to label one image simultaneously with 25 queries provided by TRECVID'07. The image browser in the middle displays the target image for users to label, where the target images are automatically selected based on the learning-based annotation algorithm in Algorithm 2. Users can use the mouse or the arrow keys on the keyboard to navigate the choice of images from the collection. On the right side, the query list shows all the query topics to retrieve. In this list, all the queries are denoted by a small number of keywords manually extracted from the query topic. For instance, "Dog_person_walk" means the query of "finding the video shots of person walking with
Figure 3. The tagging interface of the multi-topic retrieval system
Figure 4. The browsing interface of the multi-topic retrieval system
Figure 5. The shot information dialog of the multi-topic retrieval system
dogs". Also, all queries are categorized into multiple query groups, such as the group related to "people", "street" and "vehicle". Such a grouping is able to facilitate the manual judgment effort for the general users, and thus reduce the time for redundant decision on the same image. For example, if users are examining an image with no persons available, they can skip all the person-related queries at the same time. To classify the displayed image as relevant to one of the queries, user can input the related query keywords using the editor control on top of the query list, or double click the corresponding keyword to indicate its relatedness to the displayed image. On the left side, the action panel automatically generates the interface switching actions suggested by the learning-based retrieval algorithm. Rather than fully controlling the interface selection by the system, we leaves the option of switching interface to users, who can choose to either keep using the current interface, or take the next action in the panel to switch to a new interface on the suggested queries or images. If the interface is switched, the system will then load the corresponding queries and images to the image browser. Figure 4 shows a screenshot of the browsing interface that aims to retrieve a grid of images for a TRECVID'07 query "finding video shots of three or more people sitting on a table". This interface has a similar layout with the tagging interface, but with several key differences. For example, the selected
images are organized in a 3x3 image grid in the image browser which allows users to quickly browse through. The selected queries suggested by the learning-based annotation algorithm is highlighted in the query list. Users can click a mouse button or press the space key on a specified image to toggle the relevance of the query to the image. The images that are judged relevant to the given queries are overlaid with a red border and the irrelevant images are overlaid with a yellow border. Similar to the tagging interface, the action panel lists all the interface switching actions automatically generated by the learning-based annotation algorithm. Users can decided to stay with the current query, change to a new query or switch back to the tagging interface. In the case of video retrieval, users can invoke a separate shot information browser (Figure 5 so as to examine all the I-Frames in the selected video shot. The ability to browse individual video frames is important especially when the query topics are related to any temporal events/activities such as "people walking" or "people going up stairs".
EXPERMETS Our experiments are carried out on three large-scale image/video collections. The first and the second collection is generally referred to as the TRECVID collection (Over et al., 2006), which is the largest video collection with manual annotations available to the research community. In the first collection, We use the TRECVID 2005 development set (TRECVID in short) which includes a total of 74,523 keyframes. This collection consists of broadcast news videos captured from 11 different broadcasting organizations across several countries. Each image is officially annotated with 449 semantic labels (Naphade et al., 2006). For each image, we generate a 150-dimensional color histogram as the visual features. The second collection, i.e., the TRECVID 2007 collection (TRECVID'07 in short), includes a total of 22,084 keyframes from Netherlands Institute for Sound and Vision. They are fully labeled on 25 official queries. For each image, we generate a 166-dimensional color histogram and 108-dimensional co-occurrence texture as the visual features. In the following, we use both collections to simulate and examine the performance of the learning-based hybrid retrieval algorithm. We also report its real-world retrieval performance for searching both image collections.
Figure 6. Left: tagging interface. Right: tagging time statistics. The dash line is fitted based on the tagging model
The third collection is compiled from the Corel image dataset (Corel in short) on 155 keywords and currently shared online by Barnard et al. (2002). The collection is organized into 10 different samples of roughly 16,000 images, where each image is associated with a vocabulary of 155 keywords such as "aircraft", "sky", "water" and so on. We use the first sample of the images which contains a total of 15,766 images. The low-level visual features are generated on image segments provided by N-Cuts. Each segment is associated with 46 features including size, position, color, oriented energy (12 filters), and a few simple shape features. In the following discussions, we first present two user studies to confirm the validity of the tagging and browsing time models proposed in Section III-B. We use them to simulate and examine the efficiency of the learning-based hybrid annotation algorithm. Finally, we report the real-world annotation performance of hybrid annotation by asking a user to annotate the image collection in one hour.
Annotation Time Models To examine whether the proposed tagging/browsing time models are reasonable in practice, we conducted user studies on two different types of annotation systems. To validate the tagging time model, we developed a keyword-based annotation system. Figure 6 shows the screenshot of the tagging system, together with the distribution of the average tagging time against the number of keywords. The TRECVID collection with its controlled vocabulary is used in this experiment. A total of 100 randomly selected images have been annotated by a user. The number of annotated keywords per image ranges from 1 to 7. We also generate a dashed line by fitting the time statistics with the tagging model. It can be observed that the average annotation time for an image has a clear linear correlation to the number of keywords. Another interesting observation is that the dashed line does not go across the origin after an extrapolation to zero keywords. This means it takes additional time for a user to start annotating a new image due to a switch of the annotated images. Based on the linear fitting, we can estimate the tagging model parameters for this user to be tf =6.8 seconds and ts =5.6 seconds. To validate the browsing time model, we asked three different users to browse 25 keywords (extracted from 25 TRECVID'05 queries) in the TRECVID collection. The system screenshot is shown in Figure 7. In this study, we slightly rewrite the browsing model to be, tp+tnLk/Lk =t/Lk, where Lk/Lk is the Figure 7. Left: browsing interface. Right: browsing time statistics. Dash lines are fitted based on the browsing model
ratio between the number of irrelevant images and relevant images, or called negative-positive ratio, t/Lk is the average time for collecting one positive image. If the proposed browsing model is satisfied, we should be able to identify a linear relation between Lk/Lk and t/Lk. Figure 7 plots the distributions of these two factors for the three users. As we can see, the dashed lines estimated by linear regression fit the true distribution quite well, which confirmed the browsing time model is reasonable for the practical annotation environment. We estimate the model parameters by averaging all three users, i.e., tp =1.4 seconds, and tn =0.2 seconds.
Simulated Annotation Results In this section, we evaluate the performance of the annotation algorithms in large-scale image collections. Similar to the typical experimental settings for active learning, we use the ground-truth image labels to simulate real user annotations in order to avoid spending prohibitive human resources for evaluation. The annotation time is obtained based on the time model parameters tf,ts,tp and tn estimated in the previous section. Note that these parameters only serve as a reasonable estimation for the annotation parameters in a desktop environment. It is possible for these parameters to change considerably for other interfaces, devices and annotators. But given the flexibilities of the proposed time models, we expect our analysis is generally applicable without being limited to specific settings. We also carry out additional experiments to study the sensitivities on the choice of time model parameters. To illustrate how the learning-based annotation algorithm can automatically switch the annotation methods and select a batch of images to annotate by browsing, Figure 8 shows the initial 14 images which learning-based annotation asks for tagging in the TRECVID collection, and then the first batch of 30 images selected for browsing. The top of this figure lists all the initial tagged images and their associated keywords. After all the initial images are tagged, the algorithm found 5 of these images are annotated as "politicians" and predicted a large number of images can be potentially related to the key-
Figure 8. The images selected by learning-based annotation for the TRECVID collection. Top: the initial 14 images for tagging (with their keywords shown). Bottom: the first group of 30 images for browsing.
word "politicians" based on low-level visual features. Thus afterwards, it switched to present a batch of 30 images and ask the user to annotate keyword "politicians" by browsing. Because a significant faction of the browsed images are related to the keyword "politician", the user can save a lot of time without re-typing the same keyword again. This learning-based algorithm also helps to calibrate the annotated keywords, provide more complete annotations on each concept and make browsing annotation applicable even for an uncontrolled vocabulary. Next, we present the annotation performance of frequency-based annotation (FBA) and learning-based annotation (LBA) together with two baseline algorithms, i.e., tagging (Tag) and browsing (Browse). All the algorithms are evaluated on the TRECVID collection and the Corel collection. For frequency-based annotation, we select the optimal boundary B using Eqn(4) by assuming the frequency of each keyword is known. For learning-based annotation, we set the RBF kernel parameter ρ to 1 and the batch size S in Algorithm 2 to 50 unless stated otherwise. Because we assume all the user annotations are correct, the widely-used accuracy-based performance measure is no longer applicable in the scenario of manual image annotation. Therefore, we propose three new measures – macro-recall, micro-recall and hybridrecall – to evaluate the annotation quality. In this work, recall is defined as the ratio of the number of relevant images collected at a certain point of the labeling process, to the total number of relevant images. Similar to their counterpart in text classification (Yang & Pedersen, 1997), macro-recall ra is the average of the recalls for each keyword, and micro-recall ri is the recall on the entire image-keyword space. In some sense, macro-recall measures the annotation diversity and micro-recall measures the annotation completeness, where both of them are important to describe the annotation quality. Hybrid-recall is the harmonic mean of the macro-recall and micro-recall, rh =2rari/(ra+ri), designed using the same principle as the F1 measure (Yang & Pedersen, 1997). In addition, we also report the total annotation time when the annotation process is completed, or equivalently, when all three recalls become 1. Figure 9 provides a detailed comparison between four annotation algorithms on the TRECVID collection and the Corel collection. All three recall measures are reported with a growing annotation time until the annotation process ends. It is obvious to see that both frequency-based annotation and learning-based annotation are superior to the baseline tagging/browsing methods in terms of the total annotation time. For example, in the TRECVID dataset, the time for frequency-based annotation to complete the labeling process is only 60% of the tagging annotation time and 36% of the browsing annotation time. Learning-based annotation further reduces the annotation time to be 50% of tagging time and 30% of browsing time. This is achieved without using any frequency information in contrast to frequency-based annotation. Their improvement in the Corel collection is also considerable, although it is relatively smaller than that in the TRECVID collection because of a lower number of keywords per image on average. A comparison between the curves on micro-recall and macro-recall shows that tagging is good at improving the macro-recall, while browsing does well in improving micro-recall. This is because tagging random images can bring a wide coverage of various keywords at the beginning, but browsing methods only focus on annotating the most frequent keywords at the early stage. This shows an important trade-off between micro-recall and macro-recall. Thus it is more impressive to observe that learning-based annotation is able to outperform tagging/browsing in terms of both macro-recall and micro-recall, as well as hybrid-recall. Our last observation is that learning-based annotation has a significantly higher hybrid-recall at the end of its labeling process. This indicates within the same amount of time, learning-based annotation allows us to collect much more image keywords with higher annotation diversity. In order to gain deeper insight into the improvement of learning-based annotation, we plot the true browsing recall and browsing precision in Figure 10 in the TRECVID collection. The results on the 100 288
most frequent keywords are shown. For the purpose of comparison, we also show the minimal browsing recall required for learning-based annotation to outperform tagging/browsing, and the minimal precision given its true recall βk. In the browsing recall curve, we can find that βk for most keywords is higher than the lower bound except for the first 9 keywords. It is generally decreasing with the keyword frequency going down, which means, if the keyword is less frequent, a smaller proportion of the images will be covered by browsing. In the precision curve, the number of keywords that violate the minimal requirement increases to 20. It suggests that if the system knows the 20 most frequent keywords, it is better to use browsing alone to annotate them. For the other keywords, a large number of them produce a significantly higher precision over the minimal requirement, which finally leads to the overall
TRECVID
Figure 9. Annotation performance as a function of annotation time on the TRECVID and Corel collections
Figure 10. The distribution of browsing recall and browsing precision for the 100 most frequent keywords. Dashed lines show the minimum recall/precision to outperform tagging and browsing.
improvement for learning-based annotation. Some considerably improved keywords are 28-"Anchor", 29-"Interview", 30-"Overlayed Text", 86-"Soccer", 87-"George Bush", which clearly benefits from the common visual appearance in their relevant images. To evaluate the sensitivity of learning-based annotation with respect to its parameters, we designed a series of experiments and report the learning curves in terms of total annotation time for the TRECVID collection.2 Figure 11 compare the annotation time of three algorithms, i.e., Browse, Tag and LBA, with the model parameters ta, tb, te and tf multiplied by a scaling factor from 0.5 to 1.5. It degrades to our previous results when the scaling factor is 1. As can be seen, the proposed algorithm can still obtain significant improvement over browsing/tagging with various time model parameters, especially when ta, tb is lower or tf is higher than the default values. This confirms the insensitivity of our analysis. Figure 12(a) compares the annotation time with the browsing batch size S in Algorithm 2 varied from 10 to 200. With a larger batch size, the total annotation time always becomes higher. But since a smaller batch size might increase the cost for users by switching the annotation contexts more frequently, a user study is needed to determine the best batch size. Finally, because the observation in Figure 10 suggests using only browsing to annotate the most frequent concepts, we carry out the following experiments: before applying the learning-based annotation, we used browsing approaches to annotate a number of
Figure 11. Annotation time of LBA with scaled time model parameters on TRECVID collection
(a) Scaling both a, b
(b) Scaling e
(c) Scaling f
Figure 12. Annotation time of LBA with scaled learning parameters on TRECVID collection
the most frequent keywords (assuming they are known) for all images. Figure 12(b) shows the learning curves with different number of browsed keywords. As we can see, if we can identify the 15 - 20 most frequent keywords, the total annotation time can be reduced by 10%. But identifying these keywords is still a challenge.
Empirical Annotation Results on TRECVID'05 To verify the simulation results, we implemented a hybrid annotation system and asked a user to manually annotate a subset of the TRECVID collection, which includes the keyframes of 10 randomly selected videos. We recorded the statistics for three annotation approaches, i.e., tagging, browsing and learning-based annotation, around every five seconds. Each annotation process lasted for one hour. The annotation parameters are re-estimated using linear regression on all 2142 annotation statistics. For this user, these parameters are set to tp =1.16s,tn =0.17s,tf =4.01s,ts =2.02s, which are slightly less than the value used in the simulation. Table I shows the number of tagged/browsed images at the end of the annotation process, and compares the estimated annotation time (based on time models) with the true annotation time recorded. It can be found that the estimated time closely approximates the true annotation time and the error is less than 4% in all three cases. This again confirms the correctness of the proposed time models in a large-scale annotation environment. Figure 13 shows the curve of hybrid recall as a function of annotation time.3 The hybrid recall curves of tagging and browsing are similar to each other because
Figure 13. Comparing hybrid recall of three manual annotation approaches against the annotation time on the TRECVID collection
Table 1. Comparing estimated annotation time (Test.) with true annotation time (Ttrue). The number of annotated images are also shown (Ni: tagged images, Nt: tagged words, Np: relevant browsed images, Nn: irrelevant browsed images) Method Tag Browse
Ni 405 0
Nt 706 0
Np 0 1393
Nn 0 11693
Test. 3649s 3603s
Ttrue 3600s 3608s
of their complementary properties on the macro-recall and micro-recall. The learning-based annotation, on the other hand, achieves a 50% improvement over tagging and browsing in terms of hybrid recall. This observation is in line with the simulation results presented before.
Retrieval Results on TRECVID'07 The last experiment describes the results and statistics of our TRECVID'07 interactive retrieval submission on 25 official queries using the proposed multi-topic hybrid retrieval system. The initial retrieval results are generated by an automatic retrieval system, of which the overall architecture was a combination of speech-based retrieval with automatic query refinement, model-based query retrieval and re-ranking based on automatic mapping of textual queries to semantic concept detectors, semantic
Figure 14. Comparing the mean average precision of the proposed method with the other TRECVID'07 submissions
Figure 15. The number of relevant shots identified by tagging and browsing on the TRECVID'07 queries
retrieval based on automatic mapping of visual query examples to concept detectors, and visual retrieval based on light-weight learning and smart pseudo-negative sampling. All processing was done at the sub-shot level based on the master shot boundary reference, where each sub-shot was represented by a single keyframe and a corresponding speech transcript segment. All ranking results were generated at the sub-shot level first and then aggregated at the shot level by taking the maximum confidence score across all sub-shots for each master shot. More details of the automatic retrieval system can be found in (Campbell et al., 2007). The entire retrieval process lasted for 6 hours for 24 topics, which is translated into 15 minutes per query. We first examined the top 200 shots for each queries, iterate within each query until none of the unseen images are further suggested by the system. Then we switched back and forth on the tagging/browsing interfaces to retrieve multiple queries simultaneously. Our interactive system achieves a mean average precision of 0.36 on TRECVID'07 collection. With a simple temporal expansion at the end of the search result list, i.e., add discounted scores to the shots that are sufficiently close to the retrieved shots (within 3 shots) (Yan & Hauptmann, 2007), can boost the MAP to 0.43. Figure 14 compared the performance of the proposed system with other official interactive retrieval submissions in the TRECVID'07 evaluation. It can be seen that our system outperforms the other retrieval systems by a noticeable margin, which again confirms the effectiveness of the multi-topic retrieval approach. It is worthwhile to point out that since the other systems restrict the query time to be no more than 15 minutes, their performance might not directly comparable to ours. But from another perspective, this observation clearly demonstrates the advantages of multi-topic retrieval if we need to search for multiple query topics at the same time. The statistics of our retrieval system shows that around 60% of the image-topic pairs in the entire video collection have been either browsed or tagged. Our system strikes a good balance on browsing and tagging the retrieved shots, where tagging produced 1726 retrieved shots and browsing produced 884. In contrast, according to our statistics, simple browsing interface can only label around 10% of the collection using the same amount of time. Simple tagging interface can label more image-topic pairs than browsing. However, it will miss most of the labeled shots provided by browsing. Figure 15 plots the breakdown of the number of browsed and tagged shots for each query. It shows that a larger proportion of the shots are retrieved by the tagging approach, but the browsing approach also contributes a comparable number of shots for each query. For two queries including "finding a women in the interview scene" and "finding market scene", the browsing approach offers a larger number of truth shots due to the visual similarity between the truth shots in these two queries.
CONCLUSION In this paper we have presented two hybrid approaches for improving the efficiency of manual image annotation. This approach was inspired by our quantitative study of two widely used annotation approaches, i.e., tagging and browsing. We have proposed models to describe the processing time for tagging and browsing, and the validity of these models has been confirmed by our user studies. The quantitative analysis makes clear the complementary nature of tagging and browsing, and led us to propose a learning-based hybrid annotation algorithm which adaptively learns the most efficient annotation interface for selected keywords and images. Our simulation and empirical results on the TRECVID and Corel collections have shown that both proposed algorithms can achieve an up to 50% annotation time reduction
compared with the baseline methods. In particular, learning-based annotation considerably outperforms the other approaches in terms of macro-recall, micro-recall and hybrid-recall in the entire annotation process. In the interactive retrieval experiments for TRECVID'07, the proposed method also enjoys a considerable improvement on the retrieval effectiveness under the same amount of retrieval time. We expect this work to open up new research directions in modeling manual image annotation. For instance, the current annotation time models are by no means perfect. They can be refined to incorporate more user factors such as keyword missing rate, context switching cost, or vocabulary size. We can also consider other learning algorithms to support fast visual model updates. Finally, the interface design and user incentive for hybrid annotation algorithms can be further discussed.
ACK This material is based upon work funded in part by the U. S. Government. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the U.S. Government.
REFERE Barnard, K., Duygulu, P., Forsyth, D., de Freitas, N., Blei, D., & Jordan, M. (2002). Matching words and pictures. Journal of Machine Learning Research, 3. Campbell, M., Haubold, A., Liu, M., Natsev, A. P., Smith, J.R., Tesic, J., Xie, L., Yan, R., & Yang, J. (2007). IBM research TRECVID-2007 video retrieval system. In Proceedings of NIST TRECVID-2007. Foley, C., Gurrin, C., Jones, G., Lee, H., McGivney, S., O'Connor, N. E., Sav, S., Smeaton, A. F., & Wilkins, P. (2004). Trecvid 2005 experiments in dublin city university. In Proceedings of NIST-TRECVID. Gaitherberg, DC. Furnas, G.W., Landauer, T.K., Gomez, L.M., & Dumais, S. T. (1987). The vocabulary problem in human-system communication. Communication of the ACM, 30, 964–971. Griffin, G., Holub, A., & Perona, P. (2006). The caltech-256 (Technical Report). Caltech. Halaschek-Wiener, C., Golbeck, J., Schain, A., Grove, M., Parsia, B., & Hendler, J. (2005). Photostuff - an image annotation tool for the semantic web. In Proceedings of the 4th international semantic web conference. Hauptmann, A., & Witbrock, M. (1996). Informedia news on demand: Multimedia information acquisition and retrieval. In Intelligent multimedia information retrieval. Menlo Park, CA. Hauptmann, A.G., Lin, W.-H., Yan, R., Yang, J., & Chen, M.-Y. (2006). Extreme video retrieval: joint maximization of human and computer performance. In Proceedings of the 14th ACM international conference on Multimedia (pp. 385–394). New York, NY, USA. Jeon, J., Lavrenko, V., & Manmatha, R. (2003). Automatic image annotation and retrieval using crossmedia relevance models. In Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval (pp. 119–126). Toronto, Canada. 294
Kennedy, L.S., Chang, S.-F., & Kozintsev, I.V. (2006). To search or to label? predicting the performance of search-based automatic image classifiers. In Proceedings of the 8th ACM international workshop on Multimedia information retrieval (pp. 249–258). New York, NY, USA. Kustanowitz, J., & Shneiderman, B. (2004). Motivating annotation for personal digital photo libraries: Lowering barriers while raising incentives (Technical Report). HCIL, Univ. of Maryland. Li, J., & Wang, J. Z. (2006). Real-time computerized annotation of pictures. In Proceedings of the 14th ACM international conference. on Multimedia (pp. 911–920). Santa Barbara, CA, USA. Lieberman, H., Rozenweig, E., & Singh, P. (2001). Aria: An agent for annotating and retrieving images. Computer, 34, 57–62. Lin, W.-H., & Hauptmann, A. G. (2006). Which thousand words are worth a picture? experiments on video retrieval using a thousand concepts. In Proceedings of IEEE International Conference On Multimedia and Expo (ICME). Naphade, M., Smith, J. R., Tesic, J., Chang, S.-F., Hsu, W., Kennedy, L., Hauptmann, A., & Curtis, J. (2006). Large-scale concept ontology for multimedia. IEEE MultiMedia, 13, 86–91. Over, P., Ianeva, T., Kraaij, W., & Smeaton, A.F. (2006). Trecvid 2006 overview. In Proceedings of NIST-TRECVID. Rui, Y., Huang, T., & Mehrotra, S. (1997). Content-Based image retrieval with relevance feedback in MARS. In Proceedings of IEEE International Conference on Image Processing (pp. 815–818). Snoek, C., Worring, M., Geusebroek, J., Koelma, D., & Seinstra, F. (2004). The MediaMill TRECVID 2004 semantic viedo search engine. In Proceedings of NIST-TRECVID. Tong, S., & Chang, E. (2001). Support vector machine active learning for image retrieval. In Proceedings of the 9th ACM Intl. Conf. on Multimedia (pp. 107–118). Volkmer, T., Smith, J.R., & Natsev, A. (2005). A web-based system for collaborative annotation of large image and video collections: an evaluation and user study. In Proceedings of the 13th ACM international conference on Multimedia. Hilton, Singapore. von Ahn, L., & Dabbish, L. (2004). Labeling images with a computer game. In Proceedings of the SIGCHI conference on Human Factors in computing systems. Vienna, Austria. Wenyin, L., Dumais, S., Sun, Y., Zhang, H., Czerwinski, M., & Field, B. (2001). Semi-automatic image annotation. In Proceedings of Interact: Conference on HCI. Wilhelm, A., Takhteyev, Y., Sarvas, R., House, N.V., & Davis, M. (2004). Photo annotation on a camera phone. CHI '04 extended abstracts on Human factors in computing systems (pp. 1403–1406). Vienna, Austria. Yan, R., & Hauptmann, A.G. (2004). Multi-class active learning for video semantic feature extraction. In Proceedings of IEEE International Conference on Multimedia and Expo (ICME) (pp. 69–72). Yan, R., & Hauptmann, A.G. (2007). A review of text and image retrieval approaches for broadcast news video. Information Retrieval, 10, 445–484.
Yang, Y., & Pedersen, J.O. (1997). A comparative study on feature selection in text categorization. In Proceedings of the 14th International Conference on Machine Learning (pp. 412–420). Zhou, X.S., & Huang, T.S. (2001). Comparing discriminating transformations and SVM for learning during multimedia retrieval. In Proceedings of the 9th ACM international conference on Multimedia (pp. 137–146). Ottawa, Canada.
APPE .DERIVATION FOR EQN(7) We analyze the annotation time for each keyword separately. For a given keyword Tk, the time of learning-based annotation should be lower than simply tagging or browsing Tk, otherwise either tagging or browsing should be used. When the learning-based annotation time is smaller than the tagging time for Tk, we can have
⇔ tp +
tn ≤ t f ⇔
) Lk t f ≤ Lk t f
tn t f + tn - t p
Similarly, when the learning-based annotation time is smaller than the browsing time for Tk, we can have k
⇔ (1 -
)(t f + tn - t p ) +
) Lk t f ≤ Lk t p + ( L - Lk )tn
tn -
L tn ≤ 0. Lk
By defining a=tn /(tf +tn–tp) and bk =L/Lk, we can simplify this inequality to be,
(1 -
 ) + a 
 1 ab + -b ≤ 0 ⇔ ≤ a k 
The above inequality holds if and only if the following two conditions hold (since γk≤1), k
- 1 + ab ≥ 1, a k
a ab +
Eqn(7) can be obtained by merging inequalities (10) and (11).
Active Video Annotation: To Minimize Human Effort Meng Wang Microsoft Research Asia, China Xian-Sheng Hua Microsoft Research Asia, China Jinhui Tang National University of Singapore, Singapore Guo-Jun Qi University of Science and Technology of China, China
ABSTRACT This chapter introduces the application of active learning in video annotation. The insufficiency of training data is a major obstacle in learning-based video annotation. Active learning is a promising approach to dealing with this difficulty. It iteratively annotates a selected set of most informative samples, such that the obtained training set is more effective than that gathered randomly. The authors present a brief review of the typical active learning approaches. They categorize the sample selection strategies in these methods into five criteria, that is, risk reduction, uncertainty, positivity, density, and diversity. In particular, they introduce the Support Vector Machine (SVM)-based active learning scheme which has been widely applied. Afterwards, they analyze the deficiency of the existing active learning methods for video annotation, that is, in most of these methods the to-be-annotated concepts are treated equally without preference and only one modality is applied. To address these two issues, the authors introduce a multi-concept multi-modality active learning scheme. This scheme is able to better explore human labeling effort by considering both the learnabilities of different concepts and the potential of different modalities. Copyright © 2009, IGI Global, distributing in print or electronic forms without written permission of IGI Global is prohibited.
Active Video Annotation
INTRDUCTION With rapid advances in storage devices, networks and compression techniques, large-scale video data is becoming available to more and more average users. The management of these data becomes a challenging task. To deal with this issue, it has been a common theme to develop techniques for deriving metadata from videos to describe their content at syntactic and semantic levels. With the help of these metadata, the manipulations of video data can be easily accomplished, such as delivery, summarization, and retrieval. Video annotation is an elementary step to obtain these metadata. Ideally, video annotation is formulated as a classification task and it can be accomplished by machine learning methods. However, due to the large gap between low-level features and to-be-annotated semantic concepts, typically learning methods need a large labeled training set to guarantee reasonable annotation accuracy. But because of the high labor costs of manual annotation (experiments prove that typically annotating 1 hour of video with 100 concepts can take anywhere between 8 to 15 hours (Lin, Tseng, & Smith, 2003)), this requirement is usually difficult to meet. Active learning has proved effective in dealing with this issue (Ayache & Quénot, 2007; Chen & Hauptmann, 2005; Cohen, Ghahramani, & Jordan, 1996; Panda, Goh, & Chang, 2006). It works in an iterative way. In each round, the most informative samples are selected and then annotated manually, such that the obtained training set is more effective than that gathered randomly. In this chapter, we discuss the application of active learning in video annotation, which mainly consists of two parts. In the first part, we provide a survey of the existing active learning approaches, especially their applications in image/video annotation and retrieval. We analyze the sample selection strategies in these methods and categorize them into five criteria: risk reduction, uncertainty, positivity, density, and diversity. Afterwards, we detail a widely-applied Support Vector Machine (SVM)-based active learning approach. In the second part, we analyze the deficiency of the existing active learning methods in video annotation, including: (1) the to-be-annotated concepts are treated equally without preference and (2) only a single modality is applied. To address these two issues, we introduce a multi-concept multi-modality active learning scheme, in which multiple concepts and multiple modalities can be simultaneously taken into consideration such that human effort can be more sufficiently exploited The organization of the rest of this chapter is as follows. In Section II, we briefly review the related works, including video annotation and active learning. In Section III, we provide a survey of the sample selection strategies for active learning. In Section IV, we introduce the SVM-based active learning scheme. In Section V, we discuss the deficiency of the existing active learning approaches for video annotation, and then we present the multi-concept multi-modality active learning scheme in Section VI. Finally, we conclude the paper in Section VII.
RELATED WORKS Video Annotation Video annotation (also referred to as "video concept detection", "video semantic annotation" (Naphade & Smith, 2004a), or "high-level feature extraction" (TRECVID, http://www-nlpir.nist.gov/projects/trecvid)) is regarded as a promising approach to facilitating higher level manipulations of video data by bridg-
ing the semantic gap between low-level features and high-level users' needs (Hauptmann, 2005). As noted by Hauptmann (2005), this splits the semantic gap between low level features and user information needs into two, hopefully smaller gaps: (a) mapping the low-level features into the intermediate semantic concepts and (b) mapping these concepts into user needs. Annotation is exactly the step to accomplish the first mapping, as illustrated in Fig. 1. Typically video annotation can be accomplished by machine learning methods. First, videos are segmented into short units (such as shot and sub-shot). Then, low-level features are extracted from each unit (e.g., key-frames of each unit) to describe its content. Video annotation is then formalized to learn a set of predefined high-level concepts for each unit based on these low-level features. Figure 2 illustrates a typical learning-based video annotation scheme. Note that the concepts may not be mutually exclusive (such as "outdoor" and "building"), thus a general scheme is to conduct a binary classification procedure for each concept. Given a concept,
Figure 1. Video annotation bridges the semantic gap by mapping users' needs to semantic concepts
Figure 2. A typical learning-based video annotation scheme. Based on the models built upon training samples, a new sample is easily annotated with the concepts (or high-level features) of "lake", "mountain" and "sky".
each unit is then labeled "positive" or "negative" according to whether it is associated with this concept. As shown in Fig. 2, a new video clip is annotated with "lake", "mountain" and "sky" since the corresponding concept classifiers/detectors output the "positive" results. Naphade and Smith (2004b) have given a survey on TRECVID high-level feature extraction benchmark, where a great deal of different algorithms applied in this task can be found.
Active Learning Active learning, usually known as pool-based active learning, is an interactive learning technique designed to reduce human annotation effort. As previously mentioned, manual annotation is a laborintensive and time-consuming process. Active learning aims to tackle this difficult by telling users to annotate only the most useful samples. A typical active learning system is composed of two parts, i.e., a learning engine and a selection engine. It works in an iterative way, as illustrated in Fig. 3. In each round, the learning engine trains a model based on the current training set. Then the selection engine selects a batch of the most informative unlabeled samples for manual annotation, and adds these samples to training set. In this way, the obtained training set is more informative than that gathered by random sampling. It is obvious that the sample selection strategy plays a crucial role in an active learning scheme. Cohen, Ghahramani, & Jordan (1996) suggest that the optimized active learning approach should select the samples that minimize the expected risk. In (Cohen, et al., 1996; Roy & McCallum, 2001), the reduced risk is estimated with respect to each unlabeled sample, and then the most effective samples are selected. However, for most learning methods, it is infeasible to estimate the risk. Thus, practically most active learning methods adopt an uncertainty criterion, i.e., to select the samples closest to the classification boundary (Tong & Chang, 2001; Tong & Koller, 2000). Zhang et al. and Wu et al. further proposed to incorporate the density distribution of samples into the sample selection process (Zhang & Chen, 2003;
Figure 3. An illustration of active learning. It selects the most informative samples for manual labeling, such that the obtained training set is more effective than that gathered by random sampling. Training Data
Unlabeled Data
Informative samples
Wu, Kozintsev, Bouguet, & Dulong, 2006). Brinker et al. (Brinker, 2003) pointed out that the selected samples should be diverse. Several works also demonstrate that positive samples should be enhanced when applying active learning to image/video annotation or retrieval (Ayache & Quénot, 2007; Vendrig, Hartog, Leeuwen, Patras, Raaijmakers, Rest, Snoek, & Worring, 2002). In the next section, we will provide a more detailed survey of the existing sample selection methods for active learning. Active learning techniques have been widely applied in image/video annotation and retrieval. In (Qi, Song, Hua, Zhang, & Dai, 2006), Qi et al. propose a video annotation framework with SVM-based active learning. In (Goh, Chang, & Lai, 2004), Goh et al. propose a concept-dependent active learning method for image annotation, so that different sample selection strategies are adopted for different concepts. Active learning is also frequently combined with Semi-Supervised Learning (SSL) methods1 (Chapelle, Zien, & Schölkopf, 2006; Zhu, 2007). It is analyzed in (Muslea, Minton, & Knoblock, 2002) that combining active learning and semi-supervised learning is a stable approach to utilizing unlabeled data. In (Song, Hua, Dai, & Wang, 2005), Song et al. propose a video annotation scheme integrating active learning and a multi-view SSL method named co-training. In (Hoi & Lyu, 2005), Hoi et al. integrate active learning and a graph-based SSL method for image retrieval.
Sa mple S election Strategies in Active Leaa Obviously the sample selection strategy is crucial for an active learning method. In this section we provide a review of typical sample selection strategies.
Risk Reduction Intuitively, the optimal sample selection strategy is consistent with the aim of the learner, i.e., minimizing the expected risk which can be denoted by
E[ L( yˆ ( x ), y ( x )) | x ]p ( x )dx
where y(x) is the true label of sample x, yˆ ( x ) is the classifier output, p(x) is the probability density function of sample distribution, and L is a loss function that measures the difference between y(x) and yˆ ( x ). If the reduction of the expected risk of labeling each unlabeled sample can be estimated, then the optimal sample selection can be achieved. Cohen et al. (1996) have adopted such an approach for two different learners. But unfortunately, estimating the expected risk is not an easy task for most learning methods. Roy and McCallum (2001) have proposed to empirically estimate the reduced expected risk based on Monte Carlo sampling. But this approach is computationally intensive. Thus, many other heuristic sample selection strategies have been proposed. Here we categorize these strategies into the following four criteria, including uncertainty, positivity, diversity and density, as illustrated in Fig. 4.
Uncertainty Applying Uncertainty criterion means that the most uncertain samples should be selected. This heuristic stems from the fact that in many learning algorithms the essential classification boundary can be preserved based solely on the nearby samples, and the sample that are far from the boundary can be
regarded as redundant. For example, in a SVM model the hyperplane can be mainly constructed based on only the support vectors (i.e., the samples that lie closest to the hyperplane). Consider a binary classification problem. Denote by Pˆ (C0 | x ) the estimated posterior class probability of class k given x (k = 0 or 1). Then the uncertainty measure of a sample x can be simply denoted by
uncertainty ( x ) = Pˆ (C0 | x ) - 0.5
Note that there are also several other sample selection strategies that can be categorized into this criterion. For example, when multiple learners exist, the samples that have the maximum disagreement amongst them can be selected. This is the so-called query-by-committee method (Freund, Seung, Shamir, & Tishby, 1997; Seung, Opper, & Sompolinsky, 1992). Here the disagreement of multiple learners also can be regarded as an uncertainty measure, and thus this strategy is categorized into the uncertainty criterion as well.
Positivity Positivity strategy is usually applied in image/video annotation and retrieval. In these tasks, samples are classified to be positive or negative according to whether they are relevant to the given concept or query. Of course the above uncertainty measure can also be applied in these tasks. But in many cases it is found that the using positivity criterion, i.e., directly selecting the samples that have the highest probabilities to be positive, is more effective (Ayache & Quénot, 2007; Gosselin & Cord, 2004; Vendrig, et al., 2002). This is due to the fact that positive samples are usually less than negative ones in these tasks, and the distribution of negative samples is usually in very broad domain. Thus, positive samples should contribute more than negative samples2. The positivity criterion can be simply denoted by
positivity ( x ) = Pˆ (C0 | x )
Density Several works indicate that the samples within the regions of high density should be selected. From Eq. (1) we can also see the impact of the prior density distribution p(x). Wu et al. define a "representativeness" measure for each sample according to its distance to nearby samples (Wu, et al., 2006). Zhang and Chen (2003) estimate data distribution pˆ ( x ) by Kernel Density Estimation (KDE) (Parzen, 1962), and then utilize it in sample selection. Besides that, there are also several other clustering-based methods which first group the samples, and then only select the cluster centers in active learning (Nguyen & Smeulders, 2004; Qi, et al., 2006; Song, et al., 2005). Note that the cluster centers usually lie in the regions with high density. Thus, these works can also be regarded as applying a "density" strategy, i.e., trying to select samples in dense regions. Here we introduce the approach proposed in (Zhang & Chen, 2003), by which the probability density function pˆ ( x ) is estimated by pˆ ( x ) =
∑ K ( x - x ) j
where n is the number of all samples (including both labeled and unlabeled samples) and K(x) is a kernel function that satisfies K(x) > 0 and ∫K(x)dx = 1. Consequently, the density measure can be defined by normalizing pˆ ( x )to [0, 1] as follows n
density( x ) =
∑ K(x - x ) j
max ∑ K ( x - x j ) i
Diversity Previous studies demonstrate that the selected samples should be diverse (Brinker, 2003; Wu, et al., 2006). This strategy can keep the variety of the selected samples, such that the selected samples will not be constrained in a more and more restricted area. Here we consider the method proposed in (Wu, et al., 2006). Given a kernel K, the angle between two samples xi and xj is defined as
cos(< xi , x j >) =
K ( xi , x j )
K ( xi , x j ) K ( xi , x j )
Then the diversity measure can be estimated as
diversity ( xi ) = 1 - max x j ∈
K ( xi , x j ) K ( xi , x j ) K ( xi , x j )
where  is the current labeled training set. There are also several variations for this criterion. In (Hoi, Jin, Zhu, & Lyu, 2006), Fisher information matrix is adopted for sample selection to keep the diversity of the selected samples, and in (Dagli, et al., 2006) an information-theoretic diversity measure is proposed based Shannon's entropy. Up to now, we have discussed five typical sample selection criteria. But it is worth noting that these criteria are also frequently combined for sample selection. For example, in (Naphade and Smith, 2004b) the uncertainty, density and diversity criteria have been combined to form a sample selection strategy. In (Yuan, Zhou, Zhang, Wang, Zhang, Wang, & Shi, 2007), Yuan, et al. adopt the uncertainty criterion, and they also propose to shift the boundary such that more positive samples can be selected. So, this strategy also can be viewed as a combination of the uncertainty and the positivity criteria. Table 1 illustrates several works that adopt active learning in image/video annotation or retrieval. We have illustrated the description of the applications, the adopted learning methods and sample selection criteria.
SVM-Based Active Learning From Table 1 we can see that the SVM-based active learning scheme, in particular associated with the uncertainty sample selection criterion, is the most widely-applied approach. Thus, in this section we introduce this scheme in detail. From this scheme we can also see the main principle of active learning.
Figure 4. An illustration of the four sample selection criteria for active learning
Table 1. Several existing works applying active learning in image/video annotation or retrieval Works
Adopted Learning Methods
Adopted Sample Selection Criteria
Ayache and Quénot (2007)
Uncertainty; Positivity;
Chen, et al. (2005)
Dagli, et al. (2006)
Uncertainty; Diversity
Goh, et al. (2004)
Gosselin, et al. (2004)
Bayes Classifier; k-NN; SVM
He, et al. (2004b)
Hoi and Lyu (2005)
Graph-Based SSL (Zhu, Ghahramani, & Lafferty, 2003)
Naphade and Smith (2004a)
Panda, et al. (2006)
Qi, et al. (2006)
Uncertainty; Density
Song, et al. (2005)
Gaussian Mixture Model (Redner & Walker, 1984)
Sychay, Chang, & Goh (2002)
Tang, et al. (2007)
Video Typicality Ranking
Graph-Based SSL (Zhou, et al., 2004)
Tong and Chang (2001)
Vendrig, et al. (2002)
Maximum Entropy Classifier (Berger, Pietra, & Pietra, 1996)
Wu, et al. (2006)
Personal Photo Retrieval
Uncertainty; Density; Diversity
Yuan, et al. (2006)
Uncertainty; Positivity; Diversity
Zhang and Chen (2003)
Object Categorization
Kernel Regression (Zhang and Chen, 2004)
Support Vector Machine SVMs can be simply regarded as hyperplanes that separate the training samples with a maximal margin (Vapnik, 1995), as illustrated in Fig. 5. The training samples that lie closest to the hyperplane are called support vectors. Here we present a sketch introduction to the soft-margin SVM. For a binary classification problem, given a training data set which contains l labeled samples (x1, y1), (x2, y2), …, (xl, yl), xi∈R d, yi∈{0, 1}, the classification hyperplane is defined as
< w, Φ ( x) > +b = 0
where Φ(.) is a mapping from R d to a Hilbert Space , and denotes the dot product in . Thus, the decision function f(x) is
f ( x) = sign(< w, Φ ( x) > +b)
SVM aims to find the hyperplane with the maximum margin between the two classes, i.e., the optimal hyperplane. This can be obtained by solving the following quadratic optimization problem
min w ,b ,
l 1 2 w + C ∑ i 2 i =1
yi (< w, Φ ( xi ) > +b) ≥ 1 i
≥ 0, ∀i = 1, 2,..., l
In the optimization process, the mapping Φ(.) will only appear in the dot product . Therefore, the mapping can be implicitly accomplished by defining a kernel K(., .), i.e., K(xi, xj) = . The matrix K with components K(xi, xj) is usually named Gram Matrix.
Figure 5. A simple example of SVM. The training samples that lie closest to the hyperplane are called support vectors.
Active Learning with SVM Here we only consider the uncertainty sample selection criterion. Previously we have mentioned that generally the uncertainty criterion is a heuristic strategy for sample selection, but in fact for SVM this strategy can also be derived from an optimization scheme, i.e., the minimization of the expected size of version space. Version space is defined to be the set of consistent hypotheses (Mitchell, 1982). It means that a hypothesis f is in version space if it satisfies f(xi) > 0 if yi = 1 and f(xi) < 0 if yi = -1 for every training sample xi. For simplicity, we only consider an unbiased SVM model, i.e., the separating hyperplane that passes through origin. Then the version space can be denoted by  = {w
w = 1, yi 〈 w , Φ ( xi )〉 > 0, i = 1, 2, , n}
In (Tong & Chang, 2001; Tong & Koller, 2000), it has been proved that selecting the samples closest to the current hyperplane in each round is an approximate approach to reducing version space in a fastest way. Detailed analysis and proof can be found in (Tong & Chang, 2001; Tong & Koller, 2000). Note that the samples closest to the hyperplane are exactly the ones with highest uncertainty measure. Thus, the uncertainty criterion is particularly suitable for SVM-based active learning. This partially explains why this combination has shown encouraging performance in many different learning tasks. The detailed implementation of SVM-based active learning scheme is illustrated in Fig. 6.
A Diictive Leaaningedd Video Annotation According to the previous introduction, we can see that many different active learning methods have already been applied in video annotation and encouraging results have also been reported. However, these methods are all studied in a relatively simple context. More specifically, in these methods the toFigure 6. The pseudo-code of SVM-based active learning approach with the uncertainty sample selection criterion Input: ; /* labeled training set for a certain concept */ ; /* unlabeled set for a certain concept */ AT; /* the iteration time in active learning */ h; /* the batch size for sample selection */ Output: f /* annotation results */ Functions: SVMTrain(); /* SVM training algorithm */ Begin: for t.= 1, 2, …, AT. f. SVMTrain(); Select h samples according to arg min xi ∈ fi Manually label these samples, and move them from  to ; end
be-annotated concepts are usually sequentially annotated with fixed effort and only a single modality is applied. Consequently, these methods can be improved in two directions, i.e., considering the context of multi-concept annotation and multi-modality learning.
Multi-Cncept Annotation First, we consider the manual annotation for multiple concepts. Intuitively, there are two schemes, i.e., sequential annotation and simultaneous annotation. In the first scheme, each concept is exhaustively annotated for all samples before proceeding to the next, whereas in the second scheme each sample is simultaneously annotated with all concepts. In (Naphade & Smith, 2004a), a study once showed that simultaneously annotating multiple concepts may be more efficient. But recently a more detailed empirical evaluation has demonstrated that sequential annotation is superior to simultaneous annotation, in terms of both annotation accuracy and efficiency (Volkmer, Smith, & Natsev, 2005). Thus, here we only consider the sequential annotation approach. When applying active learning to multi-concept annotation, the assignment of users' annotation effort becomes a problem. A most straightforward way is to equally assign the effort to all concepts. For example, suppose annotators may provide an effort of labeling s samples in all, and then s/c samples should be labeled for each concept, where c is the number of concepts. But this strategy may induce a waste of the annotation effort. For example, several concepts may be difficult to learn with existing features and some other concepts may already have accurate models, then labeling more samples for these concepts can hardly improve their performance. Thus, it is more rational to dedicate the annotation effort to other concepts.
Multi-Modality Learning To differentiate or describe a variety of concepts, a large number of low-level features usually have to be applied. But a high-dimensional feature space frequently leads to the "curse of dimensionality" problem (Beyer, Goldstein, Goldstein & Shaft, 1999; Wu, Chang, Chang, & Smith) which may induce performance degradation. To deal with this issue, a natural method is to replace the high-dimensional learning task by multiple low-dimensional dimensional learning tasks, i.e., separately apply different modalities to learning algorithms and then fuse the results (Snoek, Worring & Smeulders, 2005). Here a modality is viewed as a description to video data, such as color, edge, texture, audio, and text. This method is usually called "multi-modality learning" or "multimodal fusion". Sometimes it is also named "late fusion" (Snoek, et al., 2005), whereas the method of using concatenated high-dimensional global feature vector is named "early fusion", as illustrated in Fig. 7. As demonstrated by a great deal of experiments, late fusion is able to improve video annotation performance compared with early fusion. But in most of the existing active learning methods only a single modality is applied. In (Chen & Hauptmann, 2005; Chen, et al., 2005), Chen et al. proposed a simple multi-modality active learning method with SVM, which selects a certain number of samples closest to the hyperplane of each sub-model (i.e., the model trained based on each individual modality). Although this method has shown better performance than single-modality-based active learning, it does not take the "discriminative abilities" of different modalities into account. For example, some features may not be discriminative enough for the concept to be annotated and some other sub-models may already have accurate models, and consequently the active learning process can only achieve very limited improvements for the these sub-models (this is analogous to the previously discussed deficiency of sequential multi-concept annotation). 308
Figure 7. A comparison of the early and late fusion schemes
Multiiept Multiiliie Leaa In the above section, we have discussed the deficiency of the existing active learning-based video annotation approaches, i.e., the neglect of the context of multi-concept annotation and multi-modality learning. Now we introduce a multi-concept multi-modality active learning scheme which can simultaneously address the above two issues. The proposed method iteratively selects a concept and a batch of samples for manual annotation, as illustrated in Fig. 8. The concept is chosen to be the one that is expected to get the highest performance gain. Then a batch of samples is selected for annotation with
Figure 8. An illustration of multi-concept multi-modality active learning scheme
this concept. The numbers of the selected samples for different modalities are adapted according to the performance gains of their corresponding sub-models. In this way, the proposed approach can take both the learnabilities of multiple concepts and the discriminative abilities of multiple modalities into consideration. For a specific modality, four of the previously introduced criteria, including uncertainty, positivity, density and diversity, are combined to select samples. For the selected concept, a graph-based SSL algorithm named manifold-ranking (He, Li, Zhang, Tong, & Zhang, 2004a; Zhou, et al., 2004) is conducted on each modality with labeled samples for its effectiveness and efficiency. As previously analyzed, the existing sequential multi-concept annotation does not take into account the impacts of the effort assigned to different concepts. To deal with this issue, in the proposed scheme an optimal concept is selected according to certain criterion for annotation in each round, such that the labeling effort can be most effectively utilized. The process is illustrated in Fig. 9. It can be viewed as a novel exchangeable multi-concept annotation approach, as opposed to the traditional sequential and simultaneous multi-concept annotation methods. We illustrate these three multi-concept annotation processes in Fig. 9 for comparison. In the proposed scheme, a concept and a batch of samples are selected for manual annotation in each round, and then manifold-ranking is employed to learn new sub-models of multiple modalities for the selected concept (although the manifold-ranking algorithm is adopted, we highlight that it can also be replaced by any other learning method, such as SVM and k-NN, when applying the multi-concept multi-modality active learning scheme). After several iterations, for each concept the results obtained from multiple sub-models are linearly fused with the weights decided by cross-validation based on labeled set. The detailed implementation is illustrated in Fig. 10.
Figure 9. A comparison of the three video annotation schemes (a) Simultaneous multi-concept annotation
(b) Sequential multi-concept annotation concept arrangment
(c) Exchangeable multi-concept annotation select a concept
concept 2 concept 3
... annotate all concepts
concept k 1
Labeling a batch of samples
... concept 1 concept 2
concept 1 concept 2 concept 3
Labeling a batch of samples Labeling a batch of samples
annotate all concepts
Figure 10. The pseudo-code of the proposed multi-concept multi-modality active learning approach Input: i = φ; /* labeled training set for i-th concept, 1≤i≤c*/ i = {x1, x2, …, xn}; /* unlabeled set for i-th concept, 1≤i≤c */ AT; /* the iteration time in active learning */ h; /* the batch size for sample selection */ ; /* concept set */ Output: fi /* annotation results for i-th concept, 1≤i≤c */ Functions: ConceptSelection(); /* select a concept from , see Section VI .A*/ Manifold-Ranking(, ); /* manifold-ranking, see Section VI. B */ SampleSelection(, , h); /* select a batch of samples for the selected concept, see Section VI. C */ Begin: for t = 1, 2, …, AT k = ConceptSelection(); /* select a concept */  = SampleSelection(k, k, h); /* select a set of samples for this concept */ Manually label samples in S, and move set  from k to k; fk = Manifold-Ranking(k, k); /* obtain the annotation results for this concept */ end
Concept Selection To establish the concept selection strategy, firstly we have to define the evaluation of the performance of multi-concept annotation. Here a straightforward method is adopted, i.e., using the average of the performance of all concepts. It can be denoted by perf ave =
1 c ∑ perfi, c i =1
where perf i is the performance of the i-th concept3. The Mean Average Precision (MAP) measure, which is widely applied in TRECVID (TREC-10 Proceedings appendix on common evaluation measures. From http://trec.nist.gov/pubs/trec10/appendices/measures.pdf), can be viewed as an example of this approach. Based on this performance measurement, a greedy strategy can be adopted by keeping selecting the concept that is expected to get the highest performance gain. The expected performance gain for each concept is approximated by the performance variation between the latest two learning iterations, which can be estimated from the newly labeled sample batches. It is noteworthy that more sophisticated performance measurements can also be applied by assigning different weights to the concepts, i.e., c
perf ave = ∑ i =1
perfi
This measurement can guarantee the annotation accuracies of those concepts with high weights. In this case, we only have to weight the expected performance gains by λi as well, and then implement the concept selection method accordingly. Obviously the proposed method needs an initial stage such that the performance gains of all concepts can be initialized. In the implementation each concept can be annotated for two iterations in this stage, and then the performance gains of all concepts are initialized. After these iterations, a concept is selected in each round and only the performance gain for this selected concept has to be updated.
Manifold-Ranking Manifold-ranking has already been widely applied in the field of image and video content analysis (He, et al., 2004a; Yuan, et al., 2006). Here it is adopted to learn a sub-model for each modality with the selected concept (note that the discussion about manifold-ranking and sample selection in these two sub-sections is based on a certain concept, i.e., the concept selected by the proposed concept selection method). This method is chosen due to the following two reasons: 1. 2.
It is a graph-based SSL method which can leverage a large amount of unlabeled samples, and it is able to achieve promising performance even with very limited training samples. It has low computational cost as it can be implemented by an efficient iterative process.
Let  = {x1, x2, …, xl} and  = {xl+1, xl+2, …, xn} be the labeled set and unlabeled set for the given concept, respectively. Considering there are M modalities, each sample xi is represented by {xi1, xi2, …, xiM} where xim is the feature representation for xi in the m-th modality. Define a vector y+ = {y1+, y2+, …, yn+}, where yi+ = 1 if xi is a labeled positive sample, and yi+ = 0 otherwise. Conversely, define y-={y1-, y2-, …, yn-}, where yi- = -1 if xi is a labeled negative sample, and yi- = 0 otherwise. Then the manifoldranking process can be implemented for the m-th modality as follows: 1. 2. 3. 4. 5.
Define a graph over all samples in m-th modality as follows: xim and xjm are connected if xim belongs to the N-nearest neighborhood of xjm, and vice versa. Define an affine matrix Wm by letting Wijm = exp(-||xim - xjm||/2σ) if xi and xj are connected and i ≠ j, and otherwise Wijm = 0. Construct a matrix Sm = D-1/2WmD-1/2 in which D is a diagonal matrix with its (i, i)-element equals to the sum of the i-th row of Wm. Initialize [ fm+, fm-] (the final results are independent of the initial values). Here f im+ and f im- can be regarded as the positive score and negative score of xim, respectively. Then iterate [ fm+, fm-] = αS×[ fm+, fm-]+(1-α) [y+, y-] for T times, where α is a parameter in (0, 1). Combine fm+ and fm- as fm = βfm++ fm- for output, where β is a positive weight.
The above manifold-ranking process is similar to the method adopted in (He, et al., 2004a). There are several issues that need to be addressed in the above process. The first one is that a sparse graph is adopted. This strategy is frequently applied in graph-based learning methods as it can significantly reduce computational and storage costs while retaining close performance (He, et al., 2004a). Another issue is that the L1 distance metric is used in the construction of the affine matrix W. This is because empirical experiments have demonstrated that L1 distance better approximates the perceptual difference for many visual features (He, et al., 2004a; Stricker & Orengo, 1995). In step 5 the results are generated
as fm = βfm++ fm-. This is due to the fact that positive samples are believed to contribute more in concept learning (He, et al., 2004a). Accordingly, a weight β is used to modulate the effect of positive samples, and typically we set β > 1. The final results are obtained by linearly combining the results from M modalities, where the weights are decided by cross-validation on the newly labeled samples.
Sample Selection First, a sample selection strategy for individual modality is established by integrating the four criteria introduced in Section III, including uncertainty, positivity, density, and diversity. According to the principles introduced in Section III, in this scheme the uncertainty and positivity measures are defined as uncertanty ( xim ) = 1 - f i m
positivity ( xim ) = f i m +
Between the uncertainty and positivity criteria, we have analyzed that the uncertainty criterion is better when positive and negative samples are balanced, and contrarily the positivity criterion may be superior if positive samples are much less than negative ones. Thus, the uncertainty and positivity measures are first integrated to form an informativeness measure as follows informativeness ( xim ) = 2 × frequency × uncertanty ( xim ) + (1 - 2 × frequency ) × positivity ( xim )
where frequency measures the percentage of positive samples in a labeled training set, i.e.,
∑y frequency =
+ i
If frequency is near 0, i.e., the positive samples are very scarce, and then they contribute much more than negative samples in the learning of the concept. In this case, f im+ has a weight near 1, and the samples more likely to be positive are selected. If frequency is high (say, near 0.5), then the positive and negative samples are balanced, and thus 1-|f im| has higher weight and the samples closer to classification boundary are selected. For computing density and diversity measures, the Exponential kernel is adopted, i.e., K(x)=exp(||x||/2σ). Then, from Eq. (5) and Eq. (7) we can obtain n
density (xim ) =
m ij
diversity (x ) = 1 - max Sijm
max ∑ W i
m i
x j ∈L
Now these criteria are combined as
effectiveness ( xim ) = × density ( xim ) × informativeness ( xim ) + (1 - ) × diversity ( xim )
where γ is a weight in [0, 1].
Up to now we have addressed the sample selection strategy for a single modality. A remained problem is how to select samples for multiple modalities. Chen et al. have analyzed that samples should be selected for each individual modality to keep the specificity of multiple feature sets (Chen & Hauptmann, 2005; Chen, et al., 2005). Thus they select equal number of samples for each modality. But this method does not take the potentials of different modalities into account. For several modalities that are not discriminative enough for the given concept, it will achieve a "saturation" state after several active learning iterations, and then more samples should be selected for other modalities. Thus, here the sample selection strategy is constructed based on the performance gains of these modalities. Analogous to that defined for concepts, the performance gain for a modality is defined as the performance variation of the modality between the latest two learning iterations. Denote by perf m the performance gain of m-th modality. Then the numbers of selected samples are set to be proportional to the performance gains of multiple modalities, i.e.,
hm =
perf m M
× h
m =1
where h is the batch size. The above strategy is based on an assumption that the performance gain of a modality is higher if the modality is further from saturation. If a modality has a high performance gain, then more samples are selected for its sub-model in the next iteration; otherwise, if a modality achieves saturation state, then few samples are selected for its sub-model. Experiments will demonstrate that this adaptive approach is better than selecting a fixed number of samples for each modality.
Discussion In this scheme, there is an assumption that the effort of labeling a sample with a concept is fixed. But in fact the effort may vary across different concepts and samples. The study in (Volkmer, et al., 2005) shows that different concepts may lead to different average annotation times per sample. Annotating different samples also may cost different effort even with the same concept. For example, the time cost for annotating a frame may depend upon the typicality (Tang, et al., 2007) and recognizability of the objects in it (see (Volkmer, et al., 2005)). But if the costs for different samples and concepts can be obtained, the sample selection and concept selection methods in our proposed scheme can be easily adapted as well by taking these costs into consideration.
Experiments To evaluate the performance of the multi-concept multi-modality scheme, experiments have been conducted following the guideline of TRECVID 2005 high-level feature extraction task. TRECVID 2005 dataset consists of 273 broadcast news videos and is about 160 hours in duration. The dataset is split into a development set and a test set. The development videos are segmented into 49532 shots and 61901 sub-shots, and the test videos are segmented into 45766 shots and 64256 sub-shots. A keyframe is selected for each sub-shot, and from the key-frame we extract the following six feature sets: (1) block-wise color moment based on 5-by-5 division of the image (225D); (2) HSV correlogram (144D); (3) HSV histogram (64D); (4) wavelet texture (128D); (5) co-occurrence texture (16D); and (6) lay-out edge distribution histogram (75D). Each feature set is regarded as a modality, and thus we obtain six modalities. 314
Ten predefined benchmark concepts are annotated, including "Walking_Running", "Explosion_Fire", "Maps", "Flag-US", "Building", "Waterscape_Waterfront", "Mountain", "Prisoner", "Sports", and "Car". Other experimental settings, including the values of the parameters in the algorithm and the computation of MAP, can be found in (Wang, Hua, Tang, Song, & Dai, 2007).
Experiments for Sample Selection First, experiments are conducted to demonstrate the effectiveness of the proposed sample selection strategy. The following three schemes are compared: Scheme 1: Integrate a global effectiveness measure as effectiveness(xi) = Σ{Δperfm×effectiveness(xim)}, and then select h samples according to this measure. Scheme 2: Select h/M samples for each modality, i.e., the method proposed in (Chen & Hauptmann, 2005). Scheme 3: Randomly select samples. Denote by s the number of samples to be annotated in all. It is equally divided and assigned to all concepts. The batch size h is set to 500. The results are illustrated in Fig. 11. From the figure we can see that the proposed approach remarkably outperforms the other three schemes. It can achieve about twice MAP compared with random sample selection when s = 1×105.
Experiments for Concept Selection To evaluate the performance of the proposed concept selection method, it is compared with the following three schemes: Scheme 1: Sequential annotation, i.e., equally assign the annotation effort to all concepts as described in the above sub-section. Scheme 2: Assign annotation effort according to the annotation performance of these concepts. More specifically, first we estimate the annotation performance of each concept regarding the full development set as training set (the detailed results can be found in (Wang, et al., 2007)), and then we let the numbers of the annotated samples to be proportional to the performance of these concepts. It implies that the concepts that are easier to learn will be assigned more effort in each round. Scheme 3: Random concept selection method, i.e., in each round a concept is randomly selected.
Figure 11. Performance comparison for different sample selection methods (h = 500)
Figure 12. Performance comparison for different concept selection methods (h = 500)
The experimental results are illustrated in Fig. 12. From the figure we can see that the performance of the proposed concept selection method increases much faster than those of Scheme 2 and Scheme 3 as the annotated samples grow. Now we compare the proposed approach with Scheme 1. It is worth noting that these two methods both have taken the learnabilities of the concepts into consideration. Scheme 1 statically assigns annotation effort according to the performance of different concepts, whereas the proposed approach dynamically selects the optimal concept in each round. From Fig. 12 we can clearly see the superiority of our proposed method.
Computational Cost According to the previous description of the manifold-ranking method, it is easy to derive that the computational cost in each active learning iteration scales as O(M×T×N×n), where M is the number of
Table 2. The practical values of the notations notation n M N T
description number of samples number of modalities neighborhood size iteration time in manifold-ranking
practical value 126157 6 25 20
modalities, T is the iteration time in the manifold-ranking process, N is the neighborhood size, and n is the number of samples. We illustrate the definitions of all these notations and their detailed values in the experiments in Table 2 for clarity. In practical experiments, the response time for interactive annotation is about 15 seconds (Pentium 1.8G Hz, 512M RAM). Thus, the proposed framework is quite efficient, and it is able to deal with large-scale datasets.
Con CLUSION and In this chapter, we have discussed the active learning-based video annotation approach. We provide a brief survey of the existing active learning methods and categorize the sample selection strategies in these methods into the following five criteria: risk reduction, uncertainty, positivity, density, and diversity. We then analyze the deficiency of the existing active learning methods for video annotation, i.e., concepts are sequentially annotated without preference and only one modality is applied. We introduce a novel multi-concept multi-modality active learning scheme which can address these issues. By taken the learnabilities of the concepts and the potentials of different modalities into consideration, the proposed active learning scheme can better explore human effort. There are also several challenging open issues in active learning-based video annotation. Since active learning involves the integration of human and computer, we categorize these issues into two problems: (1) the improvement of the efficiency of manual annotation and (2) better learning schemes.
Making Manual Annotation More Efficient In active learning-based video annotation approach, users have to annotate the samples returned by the sample selection engine. The efficiency of manual annotation is thus an important issue, though it receives relatively less attention in research community. Obviously, by speeding up the manual annotation process, more training samples can be obtained in a fixed time. Several studies have been conducted on this issue (Lin, et al., 2003; Volkmer, et al., 2005; Yan, Natsev, & Campbell, 2007), and some conclusions have been obtained, such as the sequential annotation approach is better than the simultaneous annotation approach, and the annotation accuracy of a concept is correlated with its frequency (Volkmer, et al., 2005). But many issues are still worth further studying, for example, displaying and annotating only a few video clips per page without scrolling vs. scrolling and annotating many clips at a time, using mouse vs. using keyboard for navigation and annotation, and laying out the clips randomly vs. by grouping visually similar ones into clusters. A suitable tool will significantly facilitate users and improve the efficiency of manual annotation.
Applying Better Learning Schemes The learning schemes can also be improved in several directions. In the existing active learning-based video annotation approaches, including the proposed multi-concept multi-modality active learning scheme, multiple concepts are learned separately. But concepts are correlated rather than isolated (Naphade, et al., 2006), and existing studies have shown that jointly learning multiple concepts can achieve better performance (Qi, Hua, Rui, Tang, Mei, & Zhang, 2007). Thus, the integration of correlative multi-concept learning and active learning methods is worth studying. Another interesting issue is incorporating active learning into multi-instance setting, since multi-instance learning4 has already shown encouraging performance in video annotation, especially for annotating object concepts (Cohen, et al., 1996; Gu, Mei, Tang, Wu, & Hua, 2008).
R TRECVID: TREC Video Retrieval Evaluation. http://www-nlpir.nist.gov/projects/trecvid. In TREC-10 Proceedings appendix on common evaluation measures. From http://trec.nist.gov/pubs/trec10/appendices/measures.pdf. Ayache, S., & Quénot, G. (2007). Evaluation of active learning strategies for video indexing. Signal Processing: Image Communication, 22, 692-704. Berger, A., Pietra, S.D., & Pietra, V.D. (1996). A maximum entropy approach to natural language processing. Computational Linguistics, 22(1), 39-71. Beyer, K., Goldstein, J., Ramakrishnan, R., & Shaft, U. (1999). When is "nearest neighbor" meaningful? In Proceedings of International Conference on Database Theory, (pp. 217-235). Brinker, K. (2003). Incorporating diversity in active learning with support vector machines. In Proceedings of International Conference on Machine Learning, (pp. 59-66). Chapelle, O., Zien, A., & Schölkopf, B. (Eds.). (2006). Semi-supervised learning. MIT Press. Chen, M., & Hauptmann, A. (2005). Active learning in multiple modalities for semantic feature extraction from video. In Proceedings of AAAI workshop on learning in computer vision. Chen, M., Christel, M., Hauptmann, A., & Wactlar, H. (2005). Putting active learning into multimedia applications: dynamic definition and refinement of concept classifiers. In Proceedings of ACM Multimedia, (pp. 902-911). Chen, Y., & Wang, J.Z. (2005). Image categorization by learning and reasoning with regions. Journal of Machine Learning Research, 5, 913-939. Cohen, D.A., Ghahramani, Z., & Jordan, M.I. (1996). Active learning with statistical models. Journal of Artificial Intelligence, 4, 129-145. Dagli, C.K., Rajaram, S., & Huang, T.S. (2006). Leveraging active learning for relevance feedback using an information-theoretic diversity measure. In Proceedings of International Conference on Image and Video Retrieval, (pp. 123-132).
Dietterich, T.G., Lathrop, R.H., & Lozano-Pérez, T. (1997). Solving the multiple-instance problem with axis-parallel rectangles. Artificial Intelligence, 89, 31-71. Freund, M., Seung, H.S., Shamir, E., & Tishby, N. (1997). Selective sampling using the query by committee algorithm. Machine Learning, 28, 133-168. Goh, K. –S., Chang, E. Y., & Lai, W. –C. (2004). Multimodal concept-dependent active learning for image retrieval. In Proceedings of International Conference on Machine Learning, (pp. 564-571). Gosselin, P.H., & Cord, M. (2004). A comparison of active classification methods for content-based image retrieval. In Proceedings of International Workshop on Computer Vision Meets Databases, (pp. 51-58). Gu, Z., Mei, T., Tang, J., Wu, X., & Hua, X.–S. (2008). MILC2: a multi-layer multi-instance learning approach for video concept detection. In Proceedings of International Multimedia Modeling Conference, (pp. 24-34). Hauptmann, A.G. (2005). Lessons for the Future from a Decade of Informedia Video Analysis Research. In Proceedings of ACM International Conference on Image and Video Retrieval, (pp. 1-10). He, J. R., Li, M., Zhang, H.–J., Tong, H., & Zhang, C. (2004a). Manifold-ranking based image retrieval. In Proceedings of ACM Multimedia, (pp. 9-16). He, J. R., Li, M., Zhang, H –J., Tong, H., & Zhang, C. (2004b). Mean version space: a new active learning method for content-based image retrieval. In Proceedings of ACM Workshop on Multimedia Information Retrieval, (pp. 15-22). Hoi, S. C.H., Jin, R., Zhu, J., & Lyu, M.R. (2006). Batch mode active learning and its application to medical image classification. In Proceedings of International Conference on Machine Learning, (pp. 417-424). Hoi, S.C.H., & Lyu, M.R. (2005). A semi-supervised active learning framework for image retrieval. In Proceedings of International Conference on Computer Vision and Pattern Recognition, (pp. 302309). Lin, C., Tseng, B., & Smith, J. (2003). VideoAnnEx: IBM MPEG-7 annotation tool for multimedia indexing and concept learning. In Proceedings of International Conference on Multimedia & Expo. Lin, C., Tseng, B., & Smith, J. (2003). Video collaborative Annotation forum: establishing ground-truth labels on large multimedia datasets. In Proceedings of TRECVID Workshop. Mitchell, T. (1982). Generalization as search. Artificial Intelligence, (pp. 203-226). Muslea, I., Minton, S., & Knoblock, C.A. (2002). Active + Semi-supervised learning = Robust multiview learning. In Proceedings of International Conference on Machine Learning, (pp. 435-442). Naphade, M., & Smith, J.R. (2004a). Active learning for simultaneous annotation of multiple binary semantic concepts. In Proceedings of International Conference on Image Processing, (pp. 77-80). Naphade, M., & Smith, J.R. (2004b). On the detection of semantic concepts at TRECVID. In Proceedings of ACM Multimedia, (pp. 660-667).
Naphade, M., Smith, J.R., Tesic, J., Chang, S.–F., Hsu, W., Kennedy, L., Hauptmann, A.G., & Curtis, J. (2006). Large-scale concept ontology for multimedia. IEEE Multimedia Magazine, 13(3), 86-91. Nguyen, H.T., & Smeulders, A. (2004). Active learning using pre-clustering. In Proceedings of International Conference on Machine Learning, (pp. 623-630). Panda, N., Goh, K., & Chang, E.Y. (2006). Active learning in very large image databases. Journal of Multimedia Tools and Applications Special issue on Computer Vision Meets Databases, 31(3), 249-267. Parzen, E. (1962). On the estimation of a probability density function and the mode. Annals of Mathematical Statistics, 33(3), 1065-1076. Qi, G., Song, Y., Hua, X.–S., Zhang, H.–J., & Dai, L.–R. (2006). Video annotation by active learning and cluster tuning. In Proceedings of Computer Vision and Pattern Recognition Workshop. Qi, G., Hua, X.–S., Rui, Y., Tang, J., Mei, T., & Zhang, H.–J. (2007). Correlative multi-label video annotation. In Proceedings of ACM Multimedia, (pp. 17-26). Redner, R., & Walker, H. (1984). Mixture densities, maximum likelihood and the EM algorithm. SIAM review, 26(2), 195-239. Roy, N., & McCallum, A. (2001). Toward optimal active learning through Monte Carlo estimation of error reduction. In Proceedings of International Conference on Machine Learning. Seung, H.S., Opper, M., & Sompolinsky, H. (1992). Query by committee. In Proceedings of Annual Workshop on Computational Theory, (pp. 287-294). Snoek, C.G.M., Worring, M., & Smeulders, A.W.M. (2005). Early versus late fusion in semantic video analysis. In Proceedings of ACM Multimedia, (pp. 399-402). Song, Y., Hua, X.-S., Dai, L.-R., & Wang, M. (2005). Semi-Automatic Video annotation based on active learning with multiple complementary predictors. In Proceedings of ACM Workshop on Multimedia Information Retrieval, (pp. 94-104). Stricker, M., & Orengo, M. (1995). Similarity of color images. In Proceedings of SPIE 2420, Storage and Retrieval for Image and Video Databases, (pp. 381-392). Sychay, G., Chang, E.Y., & Goh, K. (2002). Effective image annotation via active learning. In Proceedings of International Conference on Image Processing, (pp. 209-212). Tang, J., Hua, X.–S., Qi, G., Gu, Z., & Wu, X. (2007). Beyond accuracy: typicality ranking for video annotation. In Proceedings of International Conference on Multimedia & Expo, (pp. 647-650). Tong, S., & Chang, E. Y. (2001). Support vector machine active learning for image retrieval. In Proceedings of ACM Multimedia, (pp. 107-118). Tong, S., & Koller, D. (2000). Support Vector Machine Active Learning with Applications to Text Classification. In Proceedings of International Conference on Machine Learning, (pp. 999-1006). Vapnik, V.N. (1995). The nature of statistical learning theory. New York: Springer Verlag.
Vendrig, J., den Hartog, J., Leeuwen, D., Patras, I., Raaijmakers, S., Rest, J. van, Snoek, C., & Worring, M. (2002). TREC feature extraction by active learning. In Proceeding of Text Retrieval Conference (TREC). Volkmer, T., Smith, J.R., & Natsev, A. (2005). A web-based system for collaborative annotation of large image and video collections. In Proceedings of ACM Multimedia, (pp. 829-901). Wang, M., Hua, X.–S., Tang, J., Song, Y., & Dai, L.–R. (2007). Multi-concept multi-modality active learning for interactive video annotation. In Proceedings of International Conference on Semantic Computing, (pp. 321-328). Wu, Y., Chang, E.Y., Chang, K.C.-C., & Smith, J.R. (2004). Optimal multimodal fusion for multimedia data analysis. In Proceedings of ACM Multimedia, (pp. 572-579). Wu, Y., Kozintsev, I., Bouguet, J.-Y., & Dulong, C. (2006). Sampling strategies for active learning in personal photo retrieval. In Proceedings of International Conference on Multimedia & Expo, (pp. 529532). Yan, R., Natsev, A., & Campbell, M. (2007). An efficient manual image annotation approach based on tagging and browsing. In Proceedings of ACM Multimedia Workshop on The Many Faces of Multimedia Semantics, (pp. 13-20). Yuan, J., Zhou, X., Zhang, J., Wang, M., Zhang, Q., Wang, W., & Shi, B. (2007). Positive sample enhanced angle-diversity active learning for SVM based image retrieval. In Proceedings of International Conference on Multimedia & Expo, (pp. 2202-2205). Yuan, X., Hua, X.–S., Wang, M., & Wu, X. (2006). Manifold-ranking based video concept detection on large database and feature pool. In Proceedings of ACM Multimedia, (pp. 623-626). Zhang, C., & Chen, T. (2003). Annotating retrieval database with active learning. In Proceedings of International Conference on Image Processing, (pp. 595-598). Zhou, D., Bousquet, O., Lal, T.N., Weston, J., & Schölkopf, B. (2004). Learning with local and global consistency. In Proceedings of Advances in Neural Information Processing Systems, (pp. 321-328). Zhou, Z.–H. (2004). Multi-instance learning: a survey. Technical Report, Department of Computer Science & Technology, Nanjing University, China. Zhu, X. (2007). Semi-supervised learning literature survey. Technical Report (1530), University of Wisconsin-Madison, www.cs.wisc.edu/~jerryzhu/pub/ssl_survey.pdf. Zhu, X., Ghahramani, Z., & Lafferty, J. (2003). Semi-supervised learning using Gaussian fields and harmonic functions. In Proceedings of International Conference on Machine Learning, (pp. 912-919).
ndno
Semi-Supervised Learning (SSL) is another approach to dealing with the difficulty of training data insufficiency. It attempts to leverage a large amount of unlabeled samples based on certain
assumptions, such that the obtained models can be more accurate than those that achieved by purely supervised methods. More details about semi-supervised learning can be found in (Chapelle, et al., 2006; Zhu, 2007). There may be another important reason, i.e., ranking performance rather than classification accuracy is usually adopted for evaluation in these tasks. For example, in TRECVID benchmark the Average Precision (AP) measure is adopted (TREC-10 Proceedings appendix on common evaluation measures. From http://trec.nist.gov/pubs/trec10/appendices/measures.pdf). This implies that finding more positive samples with high confidence is more important than correctly classifying the samples. So, the positivity criterion may outperform the uncertainty criterion since it is able to find more positive samples. To make notations consistent, the indices for concept and modality are denoted by subscripts and superscripts, respectively. But the sample indices are denoted by subscripts as well. We have tried to avoid the simultaneous occurrence of these three indices for clarity. For examples, the notation Perf i and Perfm indicate the performance of the i-th concept and the m-th modality respectively, and xim indicates the feature representation for xi in the m-th modality. Multi-Instance Learning (MIL) aims to handle the problems in which the labels are not available for each individual instance but only for bags of instances. A bag is labeled positive if it contains a positive instance, and it is labeled negative if all the instances in it are negative. It is suitable for annotating object concepts in images or video samples, since the samples will be labeled positive or negative only according to the existence of the objects. More details about MIL can be found in (Chen & Wang, 2005; Dietterich, Lathrop, Lozano-Pérez, 1997; Zhou, 2004).
Annotating Images by Mining Image Search Xin-Jing Wang Microsoft Research Asia, China Lei Zhang Microsoft Research Asia, China Xirong Li Microsoft Research Asia, China Wei-Ying Ma Microsoft Research Asia, China
ABSTRACT Although it has been studied for years by computer vision and machine learning communities, image annotation is still far from practical. In this chapter, the authors propose a novel attempt of modeless image annotation, which investigates how effective a data-driven approach can be, and suggest annotating an uncaptioned image by mining its search results. The authors collected 2.4 million images with their surrounding texts from a few photo forum Web sites as our database to support this data-driven approach. The entire process contains three steps: (1) the search process to discover visually and semantically similar search results; (2) the mining process to discover salient terms from textual descriptions of the search results; and (3) the annotation rejection process to filter noisy terms yielded by step 2. To ensure real time annotation, two key techniques are leveraged – one is to map the high dimensional image visual features into hash codes, the other is to implement it as a distributed system, of which the search and mining processes are provided as Web services. As a typical result, the entire process finishes in less than 1 second. Since no training dataset is required, our proposed approach enables annotating with unlimited vocabulary, and is highly scalable and robust to outliers. Experimental results on real Web images show the effectiveness and efficiency of the proposed algorithm. Copyright © 2009, IGI Global, distributing in print or electronic forms without written permission of IGI Global is prohibited.
Annotating Images by Mining Image Search
INTRODUCTION The number of digital images has exploded with the advent of digital cameras which requires effective image search techniques. Although it is an intuitive way of image search since "a picture worths a thousand word", currently the Query-By-Example (QBE) scheme (i.e. using images as queries) is seldom adopted by commercial image search engines. The reasons are at least twofold: 1) the semantic gap problem. It is also the fundamental problem in Content-based Image Retrieval (CBIR) field. The techniques to extract low-level features, such as color, texture, and shape etc., are far from powerful to represent the semantics contained in an image, and hence given an image, its search results may be quite different conceptually to the query, although they possess same chromatic or textural appearances. 2) Computational expensiveness. It is well known that the inverted indexing technique ensures the practical and successful usage of current text search engines. This technique uses keywords as entries to index documents containing them, which also aligns seamlessly with the Query-by-Keyword (QBK) scheme adopted by text search engines. Thus given a number of query keywords, the search result can just be the intersection of documents indexed by these keywords separately (if no ranking functions applied). However, in the image search case, since images are 2D media, and the spatial relationship between pixels is critical in conveying the semantics of an image, how to define image "keyword" is still an open question. This prevents the inverted indexing technique from being directly used in image search, which creates a critical efficiency problem of using image visual features to search. Due to the above reasons, there is a surge of interests on image auto-annotation and object recognition in recent years. Researchers try to define ways to automatically assign keywords onto images or image regions. While all the previous work build learning models to annotation images, in this chapter, we attempt to investigate how effective a modeless and data-driven approach, which leverages the huge amount of commented image data in the Web, could be. This is reasonable because (1) to manually label images is a very expensive task (Naphade, Smith, Tesic, Chang, Hsu, Kennedy, Hauptmann, & Curtis, 2006), and in many cases, different people tend to have different explanations on a single image. Such inconsistency even makes image labeling a research topic that many questions should be addressed: What kind of images can be labeled consistently? What is the strategy of labeling images that can ensure the consistency, etc? All these obstacles lead to the lack of training data, which discourages researchers from learning a practically effective annotation model even if we have such a huge image data set in the Web. (2) Some researchers proposed very efficient and effective image encoding approaches (Fan, Xie, Li, Li, & Ma, 2005) which convert an image into an N-bit hash code so that image visual distance can be measured simply by the Hamming distance between the hash codes. This enables large-scale content-based image search in real-time and set a light on combining image visual appearance into the commercial search engines that are purely based on text search since both image indexing (e.g. matching the first n bits of two hash codes) and retrieval are of O(1) complexity. Imagine that an image set that is large enough is available, so that for each query image at least one duplication can be found in the image set, and then what we need to do is just to annotate the query by the comments or surrounding texts of the duplication. As analyzed above, this is applicable in real-time since each image is converted into a hash code so that large-scale image indexing is enabled. This motivated us to leverage Web data and crawl Web images for a data-driven image annotation approach since the Web not only deposits huge amount of images but also they generally have human assigned comments or descriptions as their surrounding texts. Apparently the ideal case above is still far from what our current technique can achieve; however, we can retrieve a group of very similar im324
ages to the query, or the near-duplicate, instead of the duplications. These search results, intuitively, will cover the major semantics inside the query image. On the other hand, the textual descriptions of these near duplications are most often noisy and diverse which are far from optimal as annotations, so that intuitively, data mining techniques can successively be applied onto the search results to learn the common keywords or phrases in a majority voting sense, which can thus be assigned to the query image as the learned annotations. This is the key idea of our proposed modeless and data-driven approach in this paper. It not only investigates how much data can help us in recognizing an image as the saying "data is the king", but also demonstrates a way to support practical image annotation which can handle large-scale image data in real-time. The paper is organized as follows. Section II gives a survey on the previous works on image annotation, which provides the reader a general picture on current art. Section III describes our insights of the image annotation problem, which motivate us the proposed modeless and data-driven approach as a novel and practical solution compared with the previous model-based ones. Section IV presents our modeless approach in detail and Section V provides experimental results. Several discussions are inspired in Section VI and the paper is concluded in Section VII with a few outlooks of possible improvements.
R In the early works on image annotation, many researchers sought to users' relevance feedback to assign labels to a given image. For example, Liu, Dumais, Sun, Zhang, Czerwinski, & Field (2001) asked the user to label a given image in the relevance feedback stage and then propagate these labels to all the positive images suggested by the retrieval system. Shevade and Sundaram (2006) made a further step. They attempted to propagate the image annotations by calculating the propagation likelihood based on WordNet (Fellbaum, 1998) synonym sets as well as image low-level features, and present only those images that are most ambiguous to the user for relevance feedback. Due to the explosion of digital images and the semantic gap problem, image auto-annotation has become a hot research topic in recent years. And most of the researchers work on two major research directions. One attempt is to learn the joint probabilities between images and words. Most of this type of research tries to learn a generative model leveraging a training dataset and then infer the labels for the new-coming images according to the learnt correlations. For example, Blei et al. extended the Latent Dirichlet Allocation (LDA) model to the mix of words and images and proposed a so-called Correlation LDA model (Blei and Jordan, 2003). This model assumes that there is a hidden layer of topics, which are a set of latent factors and obey the Dirichlet distribution. And words and regions are conditionally independent given the topics. This work used 7,000 Corel photos and a vocabulary of 168 words for annotation. Barnard, Duygulu, Freitas, Forsyth, Blei, & Jordan (2003) proposed a generative model and tested various statistical models to learn the joint probabilities of image regions and words. They used 16,000 Corel photos with 155 words and automatically annotated 10,000 test images. Li and Wang (2003) proposed a two-dimensional multi-resolution hidden Markov model to connect images and concepts. They used 60,000 Corel photos with 600 concepts. In one of their recent work, they improved this model and propose a real-time annotation approach named Alipr which attracts attention from both academic and industry. Lavrenko, Manmatha, & Jeon (2003) proposed a continuous relevance model, a generative model which directly associates continuous features with words and achieved significant improvement in performance. Jeon, Lavrenko, & Manmatha (2003) and Jeon and 325
Manmatha (2004) extended this model to learn the joint probabilities between a set of words and image features. The training set they used were about 56,000 Yahoo! news images with noisy annotations and a vocabulary of 4,073 words. This is the largest vocabulary used in the previous work, and they discussed noisy-annotation filtering and speeding-up schemes. The generative model proposed by Zhang, Zhang, Li, Ma, & Zhang (2005) added a hidden layer of semantic concepts to connect visual features and words. The concepts are discovered to explicitly exploit the synergy among the modalities. All the approaches mentioned above suggest generative models. As a different approach, Pan, Yang, Faloutsos, & Duygulu (2004a) formulated this problem as learning a graph structure. They constructed a two-layer graph on training images and their associated captions, and used the random-walk-withrestarts algorithm to estimate the correlations between new images and the existed captions. In another work of Pan, et al. (2004b), they further extended the model into a three-layered graph by taking into consideration the image regions. Also, Monay and Gatica-Perez (2003, 2004) tested the performance of latent space models on this task. Duygulu, Barnard, Freitas, & Forsyth (2002) and Barnard, Duygulu, Freitas, & Forsyth (2003) represented images by a group of blobs, and then used a statistical machine translation model to translate the blobs into a set of keywords. Another group of researchers attempted to learn classifiers based on the training data. Chang, Kingshy, Sychay, & Wu (2003) started their process by manually labeling one caption for each image in a small training set, then train an ensemble of binary classifiers, each classifier for a specific label. Li, Goh, & Chang (2003) proposed a Confidence-based Dynamic Ensemble model which was a two-stage classification method. Carneiro and Vasconcelos (2005) attempted to establish a one-to-one mapping between semantic classes and the groups of database images that share semantic labels, and the criterion is to minimize the probability of errors. Mori, Takahashi, & Oka (1999) uniformly divided each image into sub-images with key words, and then used vector quantization of the feature vectors of sub-images to estimate which words should be assigned to a new image. The work above generally requires a training stage to learn prediction models, thus the generalization capability should be a crucial assessment for practical effectiveness of the proposed approaches. Recently, some researchers began to leverage Web-scale data for image understanding (Duygulu, et al., 2002, Yeh, Tollmar, & Darrell, 2004). An interesting work was proposed by Yeh, Tollmar, & Darrell (2004) which identifies locations by searching the Internet. Given a picture of an unknown place, it firstly obtains a small number of visually relevant Web images using content-based search, then extracts a few keywords from the descriptions of these images. A text-based search is successively performed and the search results are further filtered by visual features. The disadvantages of (Yeh, et al. 2004) are that due to the efficiency problem, only a small number of relevant images can be retrieved as seeds which possibly degrades the performance, and the semantic gap problem will inevitably bias the final results. However, ignoring all these vulnerabilities, it is still an important work which pioneers a different point of view to investigate the image annotation problem.
Theooind – the Motivation This section provides our insights into the image auto-annotation problem, which directly inspired the idea of our modeless and data-driven image annotation approach.
Fundamentally, the aim of image auto-annotation is to find a group of keywords w* that maximizes the conditional distributions p(w | Iq), as shown the first row in Eq.1, where Iq is the uncaptioned query image and w are keywords or phrases in the vocabulary. Applying the Bayes rule, we obtain Eq.1(a), where Ii denotes the ith image in the database. This corresponds to the generative model shown in Fig. 1(a), in which annotations are generated directly from the images. If we assume that there is a hidden layer of "topics", so that images are distributed w.r.t. a group of topics, and it is from these topics that words are generated, then we obtain a topic model as shown in Fig. 1(b), which corresponds to the function shown in Eq.1(b), where tj represents the jth topic in the topic space. w* = arg max w p(w | I q ) = arg max w ∑ i p(w | I i ) ⋅ p( I i | I q )
 (a)
= arg max w ∑ i (∑ j p(w | t j ) ⋅ p(t j | I i ) )⋅ p( I i | I q ) (b)
Most of the previous generative approaches can be categorized into these two formulations. Moreover, since the model of Eq.1(b) investigates the relationships between images and words in a more exhaustive way, it is generally reported as more effective in many previous works (Barnard, et al., 2003, Blei and Jordan, 2003, Monay and Gatica-Perez, 2004, Zhang, eta al. 2005). However, in this paper we interpret Eq.1 in a different way. More specifically, we view Eq.1 from the angle of search and data mining. Recall that the goal of a typical search process is to retrieve a set of images I* given a query Iq such that for each Ik ∈ I*, we have I q = arg max p( I i | I q ) Ii ∈ I *
Moreover, contrary to traditional approaches which learn p(w | Ii) by defining either a generative model or discriminative model, we can just mine it out given a set of images, and such a mining process attracts interests recently, which is typically called label-based clustering techniques (Toda and Kataoka 2005, Zeng, He, Chen, & Ma, 2004). Figure 1. Generative models for image auto-annotation
M (a) the two-layer model. Words are directly generated from visual features
(b) the three-layer model. Words are generated from a hidden layer of "topics"
Different from the traditional document-based clustering approaches, e.g. k-means, which cluster documents based on the similarity of features and yield non-overlapped clusters, label-based approaches are typically applied to search results, which attempt to generate indicative and detailed descriptions to categorize the retrieved documents into clusters as well as to facilitate users' browsing, and generally yield overlapped clusters. The art is in learning the informative and classifiable words or phrases which summarize and categorize a subset of the documents. For example, Zeng, et al. (2004) defined five features to represent each n-gram and then learnt a linear regression model to rank the n-grams; the top-ranked n-grams were used as the corresponding clusters' names to differentiate the documents into clusters. Note that a pre-requisition of label-based clustering approaches is that such techniques should be applied to search results. Since all the documents are retrieved given a single query, or say they are relevant in some sense, it reduces the noise and diversity. Hence the most important feature of the "cluster names", or the learnt indicative words or phrases, is to provide further details of a cluster of documents, and hence such techniques are typically used for improving the display of a search engine (Vivisimo, 2007). This pre-requisition is automatically satisfied in our case, as stated in Section I. Instead of finding duplications for a query image (as is an excessive demand), we find each query image a group of nearduplications, or relevant images both visually and semantically (i.e. have similar textual descriptions), which are just the "search results" required by a label-based clustering technique, and hence we get to the solution of p(w | Ii). Based on the analysis above, we propose a novel solution of image annotation by first retrieving a subset of similar images {Ii} to the query (i.e. p(Ii | Iq)), and then mining annotations from this image subset {Ii} leveraging a label-based clustering technique (i.e. p(w | Ii)), which solves Eq.1(a).
The Modeless Annotation Approaoaa We have illustrated that the idea of annotating an image by those informative and distinctive phrases mined from its search results is reasonable. Here in this section we not only provide the detailed solution, but also show our ways of ensuring its effectiveness and efficiency simultaneously.
The Reformulation q
Before the illustration, let us reformulate Eq.1 in a more "image" way. Let Θ q = i I i denotes the set of q images relevant images I i =1,, n to query Iq, so that p(Θq | Iq) simulates the search process, and p(w | Θq) simulates the mining process. The label-based clustering approach is then to discover topics such that p ( w | Θ q ) = arg max t p ( w | t ) ⋅ p (t | Θ q )
where t is approximated by the "cluster names" given by the clustering approach.
Hence we have w* = arg max w p(w | I q ) = arg max w ∑ i p(w | Θq ) ⋅ p(Θq | I q )
= arg max w arg max t p(w | t ) ⋅ p(t | Θq )  ⋅ p(Θ q | I q ) (b)
From Eq.4, we can see that there are three critical factors that affect the effectiveness of the proposed approach, namely the retrieval process p(Θq | Iq ), the mining process p(t | Θq ), and the filtering process p(w | t). We do not want to propose a label-based clustering technique here in this paper, but adopt an existing and proved-effective approach instead called Search Result Clustering (SRC) proposed by Zeng, et al. (2004) which is a released product by Microsoft Research Asia (SRC, 2006). As for the filtering process, we propose a simple but effective relevance ranking and annotation rejection approach to select and rank the mined cluster names, and use the top ones as the output annotations. Intuitively, the preciseness of the identified annotations depends on the effectiveness of the SRC technique. However, since SRC is purely based on textual descriptions rather than taking image visual features into consideration as well, the effectiveness of SRC is thus degraded in our scenario. We will leave it as a future work and concentrate on the entire solution of a search and mining based image auto-annotation approach in this paper. Figure 2. The flowchart of the modeless image annotation process of mining annotations from image search results. It contains three steps: the search stage (labeled by "(1)" in the figure), the mining stage ("(2)") and the filtering stage ("(3)"). The uncaptioned image and an initial keyword "sunset" assigned serve as the queries. The highlighted yellow block shows the output annotations. Images on w orld w ide w eb
W eb Im age C raw ling, Visual & Text Indexing
q uery (1) Query-by-example
A n n o ta te
Visual Index System
Query-by-keyword
Re-Rank Text Index System
based on visual features
clouds lake tree ...
Annotation r ejection & r elevance r anking
(2) s earch r esult c lustering
The retrieval process is a bit tricky in our current implementation, but we believe that it helps move a step forward in image understanding. It is well known that the semantic gap problem dooms the Contentbased Image Retrieval (CBIR) approaches, which is thus called a fundamental problem in CBIR fields. It is caused by the gap between human semantics and the low-level visual features insufficiently effective given current feature extraction skills (Huang, Kumar, Mitra, Zhu, & Zabih, 1997). On the other hand, many researchers have proved that although individually the visual features and textual descriptions of images are ambiguous, e.g. both images labeled as "tiger lily" and "white tiger" are relevant to query keyword "tiger", they tend not to be when combined together (Barnard, Duygulu, Freitas, & Forsyth, 2001, Blei and Jordan, 2003, Li and Wang, 2003, Li and Wang, 2006). Hence we suggest to divide-and-conquer the semantic gap problem. Suppose two steps: (1) given a query image, find one accurate1 keyword (or phrase); and (2) given the image and the keyword, find more complementary keywords or phrases that detail the content of the image. The second step is easy to comprehend. By providing an initial keyword, we reduce the ambiguity of the image query and promise the search results to be relevant both visually and semantically at least to some extent. On the other hand, the requirement in the first step is not as lacking in subtlety as it may first seem. For example, for desktop images, users usually name the folders by locations or event names, and for Web images, generally there are plenty of textual descriptions from which the initial keyword can be chosen. We shrink from discussing the first step and leave it as another future work, and start our approach from the second step, thus the image annotation approach formulate our current proposal is given in Eq.5 below: w* = arg max w p(w | I q , wq ) = arg max w ∑ i p(w | Θq ) ⋅ p(Θq | I q , wq ) = arg max w arg max t p(w | t ) ⋅ p(t | Θq )  ⋅ p(Θ q | I q , wq )
Sketch of the Proposed Modeless Image Annotation Approach The flowchart of the modeless image annotation process of mining annotations from image search results is shown in Fig. 2. It contains three steps: the search stage (labeled by "(1)" in the figure), the mining stage ("(2)") and the filtering stage ("(3)"). The uncaptioned image and an initial keyword "sunset" serve as the queries. The highlighted yellow block shows the output annotations. As a pre-processing step, a large scale Web image dataset is firstly crawled and reserved as a prerequisition of a data-driven approach, which will be illustrated in Section IV.C, and then visual and textual indices of the images are built up to support the real-time process, on which we give the details in Section IV.D. Given the query image, we adopt the traditional Query-By-Example (QBE) method in CBIR to retrieve visually similar images. Intuitively, when the image database contains millions or billions of images, image retrieval based on pair-wise computation of the Euclidean distance is too time consuming and thus impractical, and we solve this problem by two means: Firstly, Query-By-Keyword (QBK) method is conducted before QBE retrieval, and thus we can leverage the inverted indexing technique2 to retrieve the "semantically" similar images in real time (the time complexity is O(1)). Normally, the number of candidate images that are possibly relevant to the query will be greatly reduced in this step, and hence improves a great deal of the efficiency of the QBE retrieval step. Secondly, instead of calculating the Euclidean distances of two images based on the original low-level feature extracted, we encode each
feature vector into a hash code which is binary bitwise, so that efficient distance measures such as the Hamming distance can be leveraged to greatly increase the search efficiency. We detail this step in Section IV.D. In this way, we obtain a small subset of images that are both visually and semantically similar to the query image, which concludes the search stage. The next step is to mine a number of terms (i.e. words or phrases) from the textual descriptions of the search results, which we adopt a previous work by Zeng et al. (Zeng, et al., 2004) named SRC (SRC, 2006), as described in Section III for this task. The cluster names output by SRC are assumed the topics in Eq.5. Then an annotation rejection step which we call "filtering" is applied onto t to approximate p(w | t) as annotation selection. The intention of this step is to further improve the precision of the output annotations for two reasons: (1) as aforementioned, the SRC technique (Zeng, et al., 2004) is based on textual features only and does not take visual features into consideration, hence the cluster names learnt may contain unsuitable terms for the query image. (2) Current SRC generates about 20 clusters3, which is too large a number to promise high precision in our image annotation scenario. We detail this step in Section IV.E. The user interface of our propose approach is shown in Fig. 3. It supports various query image suggestion schemes – the user can either upload a query image or select one from its QBE search results; s/he can type in a query term and select a query from its QBK results, or click the "Random" button to randomly select one. The gray block highlights the query image and when clicking on the word "annotate," the annotation process begins. The statistical number of time cost in search and mining are shown in the blue bar below the query image. As for the real example in Fig. 3, 500 images are retrieved from the 2.4 million Web image database and the time cost in the search and mining process are 15 and 563 milliseconds respectively, which is in real time. The terms in blue are the output annotations, which are cluster names yielded by SRC (Zeng, et al., 2004) and separated with the sign "|". In Fig. 3, three terms survived the annotation filtering step. The underneath tab form displays a subset of the 500 relevant images retrieved in the search stage whose textual descriptions took part in the SRC mining. In the following sections, we give the details of both technical and engineering designs which support effective data-driven annotation in real time.
Dataset S election: One Factor to Ensure an Effective Data-Driven Approach Since it is not only a very tedious and expensive task to manually label training data, but also very difficult to produce consistent labeling (Naphade, et al., 2006), the learning approaches always suffer from the small training data problem which degrades their power in handling outliers. For example, most previous works propose their researches based on Corel images, which contains only about 60 thousands high-quality images, and the images of the same category tend to be very similar visually. On the other hand, recall that to ensure the effectiveness our modeless and data-driven approach of annotating images by mining search results, a basic assumption is that we can find near-duplicate images, or images that share semantics with the uncaptioned query image, so collecting a large scale image dataset is necessary. Due to these reason, we resort to the Web. In fact, recently, some researchers have begun to investigate the usage of Web images and reported its effectiveness in many areas (Yeh, Tollmar, & Darrell, 2004, Weijer, Schmid, & Verbeek, 2006, Li, Wang, & Fei-Fei, 2007). However, since Web images are generally noisy and their surrounding text may
Figure 3. The interface of the proposed modeless and data-driven image annotation approach. The gray block highlights the query image. The blue bar below it gives the statistical number of the number of images retrieved from the 2.4 million Web images and the time cost in search and mining process respectively. The blue keywords show the learned annotations. They are actually three cluster names output by SRC (Zeng, et al., 2004) and "|" separates each name. The tab form displays the relevant images retrieved in the search stage which take part in the mining step.
The Summer Palace
be of very low quality, it is not necessary that any Web images will help and hence post-processing is generally applied to ensure the quality of collected Web images (Weijer, et al., 2006, Li, et al., 2007). Hence in our approach, we collect 2.4 million images from a few photo forums. The advantages are that (1) since these photos are taken by photographers, they generally have high resolution; (2) when uploading their works, the photographers will provide comments which more or less describe the photos; (3) most of the photos are nature images which have simpler semantics compared to artificial ones or portraits like the Corbis dataset; (4) there are plenty of images which are of the same semantic but with diverse appearance, which ensures a higher generalization ability of the proposed approach than using Corel images as the training data. Three examples from the 2.4 million photo forum images are shown in Fig. 4. We can see that although the descriptions are noisy, more or less they hit a few contents in the corresponding images (e.g. "tiger" in the first example) or suggest keywords (e.g. "forest") that statistically co-occur with the central objects. Table I provides statistics on the 2.4 million photos. The dictionary contains 362,669 words and each photo has 19.6 textual descriptions on average, which to our knowledge, is the largest database used in image annotation, without saying real-time annotation systems.
Table 1. Statistics on the 2.4 million database
The Promise of Real-Time The obstacles to real-time annotation in our case include the content-based retrieval process and the handling of the 2.4 million images4. We solve the first problem by mapping image visual feature vectors into hash codes and the second one by setting up a distributed system.
Accelerating the Search Process by Mapping Visual Features to Hash Codes Typically, the similarity of two images is measured in Euclidean space. However, since visual features are generally of high dimensional, measuring pair-wise Euclidean distance becomes a bottleneck in retrieval efficiency even if the search space is greatly reduced by text-based search given the query keyword. To accelerate the content-based search process, an effective way is to compress the images. Some previous works proposed vector quantization techniques, for example, Zhu, Zhang, Rao, & Srihari (2000) segments an image into regions, and quantizes the regions to obtain a few keyblocks. The widely used inverted index technique in text-based search field is thus naturally applied onto such a kind of keyblock-based discrete representation, and in this way, the content-based image retrieval problem is converted seamlessly into a text-based one which greatly improves the retrieval efficiency. In this paper, we adopt another technique of image hash code generation (Fan, et al., 2005, Wang, Li, Li, & Ma, 2006) since it is more scalable than vector quantization methods (Zhu, et al., 2000) and hence is better fit for large-scale databases. Related Works on Hash Encoding. The hash code generation algorithms (Fan, et al., 2005, Wang, et al., 2006) were originally proposed to detect duplicate images, i.e. images which are visually identical to each other. The basic idea is: suppose that visual features are mapped into bit streams, with higher bits representing more important contents of an image, and lower bits representing less important contents, thus duplicate images should have equal hash codes (i.e. the bit streams), and the equality is easily measured by the Hamming distance (i.e. the "AND" operation). This technique can also be applied to discover near-duplicates, by comparing only the higher bits of two hash codes. Intuitively, since the higher bits represent the more important contents, so that two images are more possibly alike if more highest bits match. The hash code generation algorithm proposed in (Fan, et al., 2005) is shown in Fig. 5. Firstly, it transforms an input color image to gray scale and divides it evenly into 8x8 blocks. Each block is represented by the average intensity so that the original image is converted into an 8x8 matrix Μ with each element Iij in Eq.6.
Figure 4. Three examples of the 2.4 million photos collected from a few photo forum websites. These photos are generally of high quality and have comments from the photographer.
Figure 5. The hash code generation process proposed in (Fan, et al., 2005), which is originally used for identical image search. This figure is adapted from (Fan, et al., 2005).
I ij
∑ ∑ = w -1
n -1
x=0
y =0
Int ( x , y )
w× h
where Int(x, y) is the intensity of the (i, j)-th matrix entry, and w, h denote the block width and height respectively. 2-D DCT transformation (Rao and Yip, 1990) is then applied to Μ. The DC coefficient, which is the average of M, is omitted to eliminate the effect of luminance. The remaining AC coefficients are zigzagged into a sequence, and the first 48 coefficients in lower frequencies are collected as the feature vector Xm of the original image, which is further mapped to a n-dimensional vector Yn by a PCA (Joliffe, 1986) model PT trained on 5,500 Web, as shown in Eq.7: T
Yn = P X m
The hash code is thus generated from Yn using the encoding function shown in Fig. 6.
Wang, et al. (2006) proposed a similar idea but is more efficient, as shown in Fig. 7. Images are hierarchically divided into blocks and average luminance is used as features. The projection model is still PCA trained from 11 million iFind (Chen, Liu, Hu, Li, & Zhang, 2001) images, and the 24 eigenvectors which correspond to the 24 largest eigenvalues are retained. The intuition behind using PCA for dimension reduction is that the PCA space is essentially a rotated version of the original feature space. If all the feature dimensions are kept, it draws the same conclusion on data points' distances as measured in the Euclidean space. Moreover, since PCA dimension reduction is achieved by cutting off the lower variance dimensions, the information loss is so small that can be ignored, while on the other hand, the hash code based matching can be significantly speeded up. Mapping Visual Features to Hash Codes. Though it is proposed in duplicate finding, the same technique can be applied to accelerate search process with low information loss. Recall that since the higher bits represent the more important contents in an image, if some hash codes have equal n highest bits, they are thus indexed by the same key. Henceforth we can significantly speed up the search process in visual space by creating an inverted index based on the hash codes. In our approach, we use 36-bin color Correlogram (Huang, et al., 1997) instead of intensity and luminance (Fan, et al., 2005, Wang, et al., 2006) as the original visual features. The reasons are twofold: (1) the goal of (Fan, et al., 2005, Wang, et al., 2006) is to find duplicate images which are identical both in layout and contents, hence luminance of gray-scale images is an effective feature which is also efficient. However, our goal is to find relevant images — images that may not have the same layout and contents but share similar concepts. Hence to keep the color, texture and shape, etc., features is important in our case. Obviously Correlogram features keep more information than the gray-image based ones; and (2) Correlogram is also a widely used feature in content-based image retrieval (Rui, Huang, & Chang, 1999, Carson, Thomas, Belongie, Hellerstein, & Malik, 2002, Cox, Miller, Minka, Papathomas, & Yianilos, 2000). Since 3-channel (e.g. RGB) features are used in our case, rather than the 1-channel one in (Fan, et al., 2005), 2-D DCT transformation is inapplicable here, and hence we adopt the hash code generation approach proposed by Wang, et al. (2006). However, we extract the 144-dimensional color Correlo-
Figure 6. The hash code generation algorithm proposed in (Fan, et al., 2005)
Figure 7. The hash code generation process proposed in (Wang, et al., 2006) R aw Im ages
Im age F eature P rojection m atrix
K-dim feature
H ash C ode
Duplicate Detection via Hash Code G rouping
B it allocation/ Q uantization
grams for each image and learn the 144x32 PCA projection model based on the same 11 million iFind images as in (Wang, et al., 2006). Then we use this model to offline map all the 2.4 million photos we collected into hash codes. Hash Code Based Image Retrieval. Each image is now represented by a 32-dimensional bit streams, with the higher bits represent the major information while the lower bits describe the less informative ones. We design and compared four distance measures in our current implementation to check the effectiveness and efficiency of hash code based image retrieval, the results are given in Section V. The four distance measures are listed below: •
Hash code filtering plus Euclidean distance. If the highest n bits of an image exactly match those of the query image, then the Euclidean distance of these two images are measured based on their Correlograms. This measure is a tradeoff between search effectiveness and efficiency since the time consuming Euclidean distance computing is required only for a small set of images. On the other hand, mapping Correlograms to hash codes brings information loss, thus calculating image similarities based on the original features may ensure a better effectiveness. In our experiments, n = 20. Hamming distance. It measures the number of different bits of two hash codes. Obviously this is the most efficient measurement. Its effectiveness depends on that of the hash code generation approach. Weighted Hamming distance. Hamming distance assume all bits have equal weights. But intuitively, since the higher bits are more important, differences in higher bits should be larger-weighted. We propose a weighting function as below: Firstly, we evenly group a 32-bit hash code into 8 bins. From left to right, each bin contains 4-bits. Then we weight the Hamming distance on the ith bin by 2(8–i), 1≤i≤8. Obviously the weighting function magnifies the difference on the higher bits. It is simple but effective as proved in our experiments. Euclidean distance. The Euclidean distance based on original Correlogram features is also given as a baseline to assess the performance of the above hash code-based measurements.
We rank the images in the ascending order of the distances respectively and return the top N as search results.
Efficiently Handling the Large-Scale Database with Distributed Computing In part 1), we demonstrate the method of supporting content-based search in real time and proposed a hash code generation approach as well as different distance measures. However this is not enough to support real time image annotation when the database contains millions or billions of images. In this subsection, we describe our design of a distributed system, which as shown in Fig. 3, finishes the search process in 15 milliseconds and the mining process in 563 milliseconds, based on 2.4 million images. The system architecture is shown in Fig. 8. The key is that the content-base search engine, text search engine and SRC clustering engine are provided as Web services using C# Remoting technique. Each service register a distinct TCP port and listening to it, when there is a service request, the service accepts inputs, does its own work, and sends back outputs. Moreover, by building up this distributed system, both visual and textual features can be kept in memories of servers, and no disk I/O is requested, so obviously it is quite efficient and easily scalable. 336
Te Control of SRC (Z eng, et al., 2004) Outputs As mentioned in Section IV.A, we use the search result clustering technique named SRC (Zeng, et al., 2004) to learn the topics based on the visually and semantically similar search results. A left problem is that SRC requires users to provide the number of clusters |nsrc| wanted, which is difficult to select without observing the data. We provide an experiential algorithm as Eq. 8 below to select |nsrc|: nsrc = max( Θ
200 , 4)
where |Θ*| is the number of retrieved images. 200 is an empirical value of cluster size suggested by SRC to ensure the saliency of learned cluster names. On the other hand, if |Θ*| is too small, SRC tends to group all images in one or two clusters and hence images in one cluster may be too diverse to produce meaningful cluster names. To avoid this, we force the algorithm to output at least 4 clusters, while 4 is another empirically selected parameter. Moreover, because SRC extracts all n-grams Θ*(n ≤ 3) as its candidate key phrases, if Θ* is too large, time cost of the SRC service may be very high. Hence, we set max Θ* = 200. The final problem is annotation rejection. As aforementioned (refer to Section IV.B), it is unsuitable to output all cluster names yield by SRC as the annotations, hence to define criteria for annotation rejection is necessary to ensure the performance. Two criteria are compared in our current implementations: •
Maximum cluster size criterion. This is a very simple but effective annotation rejection method, which uses the size of each cluster, i.e. the number of images in each cluster, as the score to rank each cluster name, and the top ranked ones are assigned onto the query image as the final annotations. This is equivalent to the Maximum a Posteriori estimation (MAP) which assumes that "the majority speaks the truth" and is statistically reasonable since the cluster names are learned from search results which are assumed visually and semantically relevant to the query image. Average member image score criterion. The term "member image" of a cluster A indicates an image that is assigned to A, and the cluster score is measured as the average similarity of its member images to the query image. It is reasonable since the more relevant member images a cluster contains, the more probably the concepts learned from this cluster represents the content of the query image.
We keep the top ranked clusters, merge their cluster names by removing the duplicate words and phrases, and output as the learned annotations, which closes the entire system process.
Evaua 2.4 million photos from several online photo forums are collected as the database, from which the relevant images are retrieved to annotate the query image. These photos are not only of high quality, but also have rich descriptions, as discussed in Section IV.C. In order to evaluate the effectiveness and efficiency of our proposed approach, we conducted a series of experiments based on two datasets. One is an open dataset, of which the images are collected
from Google image search engine (Google, 2006). 30 Google images from 15 categories (as shown in Table II) were randomly selected. However, to give a more objective evaluation of the effectiveness, we deliberately selected a few vague query keywords, e.g. using "Paris" as the query keyword to annotate a photo of "Sacre Coeur". Because no ground truth labels are available, we manually evaluated the retrieval performance on this dataset. The other testing dataset is a benchmark content-based image retrieval database provided by the University of Washington (UW)5. Images in this dataset have about 5 manually labeled ground truth annotations on average, and the UW folder names were used as the query keywords (see Table III). A problem with this dataset is that for many images, not all objects are annotated. For a fair evaluation, we manually revised the results to accept synonyms and correct annotations that do not appear in UW labels.
Experiments on Google Images Evaluation Criterion The evaluation of image auto-annotation approaches is still an open problem. Typically in many previous works (Blei, et al., 2003, Carneiro and Vasconcelos, 2005, Jeon and Manmatha, 2004), image retrieval technique is used for evaluation. Generally in these approaches, the entire dataset is separated into two parts, one for training and the other for testing such that images in these two datasets share almost the same vocabularies and data distributions, and ground truth annotations are available for the testing
Figure 8. System architecture of the proposed modeless and data-driven image annotation approach. The visual search engine, text search engine and SRC engine are implemented as services, which make up of a distributed system using C# Remoting technique.
7EB ) MAGE #R AWL ER
3EAR CH
$OCUMENT 1UEUE
7EB 5)
,AR GE 3CAL E #ONT ENT BAS ED ) MAGE 3EAR CH%NGI NE 'R OUP
0HOT O 2EPOS I T OR Y
(T ML 0AR S ER
Sni ppet s 4EXT -ET ADAT A 2EPOS I T OR Y  3QL 3ER VER
/F F L I NE DAT A F L OW /NL I NE DAT A F L OW
4EXT 3EAR CH%NGI NE
System Output Butterfly Nature Florals xx
4EXT ) NDEXI NG
4I T L E #OMMENT S #AT EGR OR Y
6I S UAL ) NDEXI NG
32# 3ER VER
dataset. Thus in their evaluation, a number of keywords can be selected from the vocabulary to query the annotated testing dataset and the precision and recall are calculated for the returned images. Let G be the ground truth relevant images and L be the retrieved images, retrieval precision and recall are defined as in Eq.9. Precision =
LG L
, Recall =
LG G
i.e. precision evaluates the proportion of relevant images in the retrieval results, while recall calculates that of the relevant images in all the relevant images contained in the database. However, Eq.9 suggests only the probability of identifying "correct" annotations. But for an annotation system, generally the output annotations belong to three categories: "perfect", "correct", and "wrong". The "perfect" annotations hit the specific objects inside an image, e.g. "tiger" for the first image in Fig. 4. The "correct" annotations are those that are not-wrong but also not perfect, e.g. "France" for an image about Eiffel tower. The "wrong" annotations are the incorrect outputs, which have nothing to do with the contents of the query image. We believe that a comprehensive evaluation should take all these three categories into consideration. Hence we propose Eq.10 to evaluate the performance. It modifies the normalized score measure (Barnard, et al., 2003, Li and Wang, 2006), which only categorizes the annotations into "right" or "wrong". E=
p + 0.5 × r - w m
where m denotes the number of annotations predicted. p, r, w are the number of "perfect", "correct", and "wrong" annotations respectively. Note that to emphasize the preference for "perfect" annotations, we punish the "correct" ones by a lower weight 0.5. Obviously when all the predictions are "perfect", E = 1, while if all are wrong, E = –1. Since there are no ground truth annotations available for this dataset, it is impossible for us to evaluate the recall performance.
System Effectiveness Fig. 9 shows how annotation precision varies as the similarity weight changes. This weight weights the average similarity of images retrieved in the content-based search stage, and the product serves as the threshold to filter out irrelevant images, i.e. images whose similarity to the query image is lower than Table 2. Queries from google Apple, Beach, Beijing, Bird, Butterfly, Clouds, Clownfish, Japan, Liberty, Lighthouse, Louvre, Paris, Sunset, Tiger, Tree
Table 3. Queries from U.washington Australia, Campus, Cannon beach, Cherries, Football, Geneva, Green lake, Indonesia, Iran, Italy, Japan, San juan, Spring flower, Swiss mountain, Yellowstone
Figure 9. Annotation precision measured by Eq.10. It shows respectively the precision curve of using the four distance measures described in Section IV.D vs. the similarity weight as a threshold to control the number of image search results yield at the search stage. Max Cluster Size Criterion E 0.7
Hamming Distance Weighted Hamming Eucli_Dist on Correlo Text only
0.6 0.5 0.4 0.3 0.2 0.1 0
Similarity Weight
(a) Precision w.r.t. maximum cluster size criterion Avg Member Image Score E
(b) Precision w.r.t. average member image score criterion
this threshold will not be delivered to the SRC mining stage. This parameter determines Θ* in Eq.8 and thus directly affects the learned clusters and the predicted annotations. The reason of proposing such a threshold strategy is that, since the similarity of images varies greatly; it is very difficult to select a hard threshold which can promise high system performance for any queries. On the contrary, our soft threshold is query-dependent, it leverages the information provided by content features of the image search results.
The green square curve in Fig. 9 represents the process without the content-based search step, i.e. it relies purely on text-based search to yield near duplicates, and SRC clustering is based on the images that are "semantically" similar but may not visually similar. It serves as the baseline method here. Also, since no visual features are available, annotations predicted by this method uses the maximum cluster size criterion (see Section IV.E). Fig. 9(a) shows annotation precision curves of the four content-based search distance measures (refer to Section IV.D) under maximum cluster size criterion. From this figure, we can see that the weighted Hamming distance measure performs the best. This is reasonable because it emphasizes the feature components that captures the important information in an image while at the same time de-emphasize the unimportant ones. It is interesting that Euclidean distance measure based on the original Correlogram features performs comparatively to the Hamming distance measure in effectiveness. This shows that the information-loss of mapping original features into hash codes (refer to Section IV.D) can be ignored on this dataset, or say, the model for hash code generation is effective. Another interesting result is that the hash code filtering plus Euclidean distance method performs badly. The reason may be that the higher 20 bits of hash codes learned are too coarse to distinguish relevant images from irrelevant ones. All the distance measures perform much better than the baseline method. This shows that requiring the visual similarity of clustering images is necessary and important. Fig. 9 (b) shows the performance with maximum average member image score criterion. It is generally worse than that with the maximum cluster size criterion. A possible reason is due to the semantic gap. Recall that SRC algorithm clusters images purely based on their surrounding texts and ignores image visual features. Thus images in one cluster may have very different visual features even if they belong to the same category. Obviously this will not affect the maximum cluster size criterion but only the average member image score criterion, because the latter one uses visual similarity to score the clusters. Another possible reason may be that the images which share similar descriptions vary greatly in their visual appearances. Note that the system's performance jumps when the threshold is too large such that Θ* is too small to ensure satisfying SRC clustering performance. Fig.10 shows a few examples of the annotation results. The boldfaced keywords indicate the queries. It can be seen that our approach is able to find correct and most of the time perfect complementary annotations.
System Efficiency We have give the readers a sense of the efficiency in Fig. 3, which shows that our annotation process is fulfilled in real time, although the database contains 2.4 million images, which is the largest till now to our knowledge. In the subsection, we conduct one more experiment to evaluate the efficiency statistically. For all the queries, we collect their search results, which are about 24,000 images on average, and test the system efficiency according to the four distance measures: Hamming Distance, Weighted Hamming Distance, Hash code filtering plus Euclidean distance, and the traditional Euclidean distance based on original Correlogram features. The hardware environment is a computer with one Dual Intel Pentium 4 Xeon hyper-threaded CPU and 2G memory. The time cost for computing the pair-wise distance between each
Figure 10. A few examples of the annotation results yield by the modeless and data-driven image annotation approach
Figure 11. Search efficiency vs. the four distance measures. Pair-wise distances are measured based on 24,000 images. It proves the efficiency of the proposed idea. Moreover, the Hamming distance is the most efficient, which is also reasonable. Time Costs (sec)
0.14 0.12 0.1 0.08 0.06 0.04 0.02 0 Hamming
Weighted Hamming
Hash filtering+Eucli
Eucli Dist Distance Measure
of the 24,000 images and the query image, as well as ranking the images accordingly, is shown in Fig. 11, which cost 0.034, 0.072, 0.051 and 0.122 seconds respectively. We can see that computing Euclidean distance is nearly 4 times slower of calculating Hamming distance.
The hash code filtering plus Euclidean distance measure is the second efficient. The reason is that the hash codes here serve as the inverted indices to identify the relevant images which is of O(1) computation complexity. Time cost for this measure is consumed by the Euclidean distance calculation afterwards. Note that the above evaluations were conducted with all features loaded into memory. If disk access is required, we can imagine that hash code-based measures will be even faster than the original visual feature-based ones.
Experiments on UW Dataset In this subsection, we evaluate the performance of our proposed approach based on the benchmark U.Washington database. Because ground truth is available and this dataset is comparably small, we use precision and recall as the evaluation criteria. Fig. 12 shows the maximum precision and recall of our approach w.r.t. the two annotation rejection criteria vs. the four distance measures. All images in the database are used for the evaluation. Again the weighted Hamming distance measure performs the best. An interesting point is that the average member image score criterion now works better. This is because few images in our 2.4 million photograph database are visually similar to the UW images, while
Figure 12. Precision and recall yielded corresponding to the four distance measure given the two annotation rejection criteria based on all the images in UW dataset. 0.3
Avg. Member Image Score Maximum Cluser Size
0.25 0.2 0.15 0.1 0.05 0 Hamming Distance
Hash Filter + Eucli_Dist
Eucli_Dist on Correlo
Distance Measure 0.25
Avg. Member Image Score Maximum Cluster Size
0.2 0.15 0.1 0.05 0 Hamming Distance
Eucli_Dist on Correlo Distance Measure
the UW images of the same category share similar visual appearance. Moreover, the average image score strategy helps to rank higher the clusters whose member images are more visually relevant to the query thus is less biased by the irrelevant member image descriptions. It is worth mentioning that the real performance of our system is actually much better than shown in Fig. 12. Since the evaluation shown in Fig.12 did not grant synonyms, e.g. "beach" and "coast", and semantically relevant keywords, e.g. "Geneva" and "Switzerland", as correct answers. Moreover, UW ground truth annotations may ignore some contents of an image, so that our current evaluation assumes them to be "wrong" even if the predicted annotations are correct, just because they do not appear in UW ground truth. To prove this, we manually examined the predicted annotations of 100 randomly selected queries. The corrected precision and recall are 38.14% and 22.95% respectively, nearly 12% precision improvement, with weighted Hamming distance measure, the average member image score criterion, and the similarity weight 1.2. Fig. 13 show four examples of our output, which hit no UW ground truth labels and gives zero precisions in the strict evaluation but are indeed correct answers. Moreover, since our method is a modeless one which has no supervised learning stage, and the UW images are "outliers" to our database from which few relevant images can be found, the task is thus much tougher for us than for the previous approaches. On the contrary, in most of the previous works, training data and testing data are selected from the same dataset and the training dataset is usually much larger than the testing dataset, e.g. (Barnard, et al., 2003, Blei and Jordan, 2003) use 4,500 Corel images for training and 500 images for testing, and the performance is around 20-30%. This shows that our system is more effective in predicting annotations, and is robust in handling outsiders.
DIi Compared to the previous works, or traditional computer vision and machine learning ones which built up generative or discriminative annotation models, in this paper, we propose a novel attempt of modeless image annotation, which investigates how effective to a data-driven approach only can be, and suggest mining annotations for an uncaptioned image by mining its search results. It has at least three advantages: (1) no training dataset and supervised learning are required and hence is free from the problem of lack of training data; (2) since no training stage is available, and hence it requires no predefined vocabularies while vocabulary creation is still an open research topic (Naphade, et al., 2006); (3) by leveraging the Web data, we can obtain diverse images for the same semantics which ensure a
Figure 13. Four examples of output annotations produced by our approach based on UW database, which hit no UW ground truth labels but are indeed correct answers. Campus college, campus life , center, tree Iran mashhad, kish island , esfahan
Football stadium, school football , football game , football player Cannon Beach haystack rock
high generalization capability or say the practical usage of the proposed annotation approach. This is a noticeable advantage to using the Corel Stock-style database in which images of the same concept have similar visual appearances with clean descriptions. Moreover, our approach is highly scalable and very robust to outsider queries.
Con CLUSION Compared to the previous annotation approaches which build up generative or discriminative annotation models, in this paper, we propose a novel attempt of modeless image annotation, which investigates how effective a data-driven approach can be, and suggest mining annotations for an uncaptioned image by mining its search results. We collected 2.4 million images with their surrounding texts from a few photo forum websites. The entire process contains three steps: (1) the search stage, which yields visually and semantically search results given the uncaptioned images as queries; (2) the mining stage, which applies a labelbased clustering technique to mining topic labels from the textual descriptions of the search results as candidate annotations; (3) the annotation rejection stage, which filtered the candidate annotations to ensure the preciseness of the final outputs. Compared with the previous works, our method saves labor both in training data collection and vocabulary creation, and is highly scalable and robust to outliers. And as to our knowledge, it is a realtime approach that handles the largest number of images. To ensure real time annotation, two key techniques are leveraged – one is to map the high dimensional image visual features into hash codes, the other is to implement it as a distributed system, of which the search and mining processes are provided as Web services. As a typical result, the entire process finishes in less than 1 second. Experiments conducted on both an open database (Google images) and a benchmark database (UW CBIR database) proved the effectiveness and efficiency of this proposed approach. There are much room to improve our proposed approach, e.g. as mentioned in Section IV.B, to propose label-based clustering technique which takes also image visual features into consideration is necessary and important for further performance improvement. And we would like to investigate how much performance gain we can obtain by embedding learning approaches into our current implementation.
ACKNOWLEDGMENT The authors would like to thank Feng Jing, Hua-Jun Zeng, Kefeng Deng, Bin Wang and Le Chen for their sincere helps on technical discussions, assistances, as well as system implementations. Many thanks should also be given to the volunteers that help in manually evaluating our experimental results.
REFERENCES Barnard, K., Duygulu, P., Freitas, N., Forsyth, D., Blei, D., & Jordan, M. (2003). Matching words and pictures. Journal of Machine Learning Research, (pp. 1107-1135).
Barnard, K., Duygulu, P., Freitas, N., & Forsyth, D. (2003). Recognition as translating images into yext. Internet Imaging IX, Electronic Imaging. Barnard, K., Duygulu, P., Freitas, N., & Forsyth, D. (2001). Clustering art. Computer Vision and Pattern Recognition, (pp. 434-439). Blei, D., & Jordan, M.I. (2003). Modeling annotated data. Annual International ACM SIGIR Conference, Toronto, Canada. Cai, D., He, X., Li, Z., Ma, W.-Y., & Wen, J.-R. (2004). Hierarchical clustering of WWW image search results using visual, textual and link information. ACM International Conference on Multimedia, (pp. 952-959). Carneiro, G., & Vasconcelos, N. (2005). A database centric view of semantic image annotation and retrieval. In Proceedings of ACM SIGIR Conf. on Research and Development in Information Retrieval. Carson, C., Tomas, M., Belongie, S., Hellerstein, J.M., & Malik, J. (2002). Blobworld: image segmentation using expectation-maximization and its application to image querying. IEEE Transactions on Pattern Analysis and Machine Intelligence, 24(8), 1026-1038. Chen, Z., Liu, W.Y., Hu, C., Li, M., & Zhang, H.J. (2001). iFind: a web image search engine. Proc. of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, (pp. 450-458). Chang, E., Kingshy, G., Sychay, G., & Wu, G. (2003). CBSA: content-based soft annotation for multimodal image retrieval using Bayes Point Machines. IEEE Trans. on CSVT, 13(1), 26-38. Cox, I.J., Miller, M.L., Minka, T.P., Papathomas, T.V., & Yianilos, P.N. (2000). The Bayesian image retrieval system, PicHunter: theory, implementation, and psychophysical. IEEE Transactions on Image Processing, 9(1), 20-37. Duygulu, P., Barnard, K., Freitas, N., & Forsyth, D. (2002). Object recognition as Machine Translation: Learning a Lexicon for a Fixed Image Vocabulary. In Seventh European Conference on Computer Vision, 2002, (pp. 97-112). Fan, X., Xie, X., Li, Z., Li, M., & Ma, W.-Y. (2005). Photo-to-search: using multimodal queries to search the Web from mobile devices. ACM SIGMM International Workshop on Multimedia Information Retrieval. Fan, J.P., Gao, Y.L., & Luo, H.Z. (2004). Multi-level annotation of natural scenes using dominant image components and semantic concepts. In Proc. ACM International Conference on Multimedia. Fan, J.P., Gao, Y.L., Luo, H.Z., & Xu, G.Y. (2004). Automatic image annotation by using concept-sensitive salient objects for image content represent. In 27th Annual International ACM SIGIR Conference. Fellbaum, C. (Ed.) (1998). WordNet: an electronic lexical database. MIT Press, Cambridge, Massachusetts. Ghoshal, A., Ircing, P., & Khudanpur, S. (2005). Hidden Markov models for automatic annotation and content-based retrieval of images and video. In 28th Annual International ACM SIGIR Conference. Google image. (2008). http://images.google.com. 346
Huang, J., Kumar, S.R., Mitra, M., Zhu, W.-J., & Zabih, R. (1997). Image indexing using color correlograms. IEEE Conf. on Computer Vision and Pattern Recognition, San Juan, Puerto Rico. Jeon, J., Lavrenko, V., & Manmatha, R. (2003). Automatic image annotation and retrieval using crossmedia relevance models. In Proc. of the Annual Int. ACM SIGIR Conf. on Research and Development in Information Retrieval. Jeon, J., & Manmatha, R. (2004). Automatic image annotation of news images with large vocabularies and low quality training data. ACM International Conference on Multimedia. Joliffe, I.T. (1986). Principal component analysis. New York: Springer-Verlag. Lavrenko, V., Manmatha, R., & Jeon, J. (2003). A model for learning the semantics of pictures. NIPS. Li, B.T., Goh, K., & Chang, E. (2003). Confidence-based dynamic ensemble for image annotation and semantics discovery. In Proc. ACM International Conference on Multimedia, (pp. 195-206). Li, J., Wang, G., & Fei-Fei, L. (2007). OPTIMOL: automatic Object Picture collecTion via Incremental MOdel Learning. IEEE Computer Vision and Pattern Recognition. Li, J., & Wang, J. (2003). Automatic linguistic indexing of pictures by a statistical modeling approach. IEEE Transactions on Pattern Analysis and Machine Intelligence, 25(9), 1075-1088. Li, J., & Wang, J. (2006). Real-time computerized annotation of pictures. ACM International Conference on Multimedia. Liu, W.Y., Dumais, S., Sun, Y., Zhang, H., Czerwinski, M., & Field, B. (2001). Semi-automatic image annotation. In Proc. Of Interact: Conference on HCI, (pp. 326-333). Li, X., Wang, X.-J., Wang, C., & Zhang, L. (2007). SBIA: search-based image annotation by leveraging Web-scale images. ACM International Conference on Multimedia. Monay, F., & Gatica-Perez, P. (2003). On image auto annotation with latent space models. In Proc. ACM Int. Conf. on Multimedia. Monay, F., & Gatica-Perez, P. (2004). PLSA-based image auto-annotation: constraining the latent space". In Proc. ACM InternationalConference on Multimedia. Mori, Y., Takahashi, H., & Oka, R. (1999). Image-to-word transformation based on dividing and vector quantizing images with words. In First International Workshop on Multimedia Intelligent Storage and Retrieval Management. Naphade, M., Smith, J. ., Tesic, J., Chang, S.-F., Hsu, W., Kennedy, L., Hauptmann, A., & Curtis, J. (2006). Large-scale concept ontology for multimedia. IEEE Multimedia Magazine, 13(3). Pan, J.-Y., Yang, H.-J., Faloutsos, C., & Duygulu, P. (2004). Automatic multimedia cross-modal correlation discovery. In Proceedings of the 10th ACM SIGKDD Conference. Pan, J.-Y., Yang, H.-J., Faloutsos, C., & Duygulu, P. (2004). GCap: graph-based automatic image captioning. International Workshop on Multimedia Data and Document Engineering. Rao, K.R. & Yip, P. (1990). Discrete cosine transform: algorithms, advantages, applications. New York: Academic. 347
Rui, Y., Huang, T.S., & Chang, S.-F. (1999). Image retrieval: current techniques, promising directions, and open issues. Jounal of Visual Communication and Image Representation, 10(1), 39-62. Shevade, B., & Sundaram, H. (2006). Incentive based image annotation. Technical Report, Arizona State University. AME-TR-2004-02. SRC (2006). Search result clustering toolbar in Microsoft Research Asia. http://rwsm.directtaps.net/. Toda, H. & Kataoka, R. (2005). A search result clustering method using informatively named entities. 7th ACM Workshops on Web Information and Data Management . Vivisimo (2007). http://www.vivisimo.com. Volkmer, T., Smith, J., Natsev, A., Campbell, M., & Naphade, M. (2005). A web-based system for collaborative annotation of large image and video collections. In Proc. ACM International Conference on Multimedia M. Wang, B., Li, Z.W., Li, M.J., & Ma, W.-Y. (2006). Large-scale duplicate detection for Web image search. IEEE International Conference on Multimedia & Expo. Wang, X.-J., Ma, W.-Y., Xue, G.-R., & Li, X. (2004). Multi-model similarity propagation and its application for Web image retrieval. ACM International Conference on Multimedia, (pp. 944-951). Weijer, J., Schmid, C., & Verbeek, J. (2006). Learning color names from real-world images. IEEE Computer Vision and Pattern Recognition. Yahoo News Search. (2006). http://news.search.yahoo.com/news. Yeh, T., Tollmar, K., Darrell, T. (2004). Searching the Web with mobile images for location recognition. IEEE Computer Society Conference on Computer Vision and Pattern Recognition, (2), 76-81. Zeng, H.J., He, Q.C., Chen, Z., & Ma, W.-Y. (2004). Learning to cluster Web search results. In Proc. of the 27th Annual International Conference on Research and Development in Information Retrieval, (pp. 210-217). Zhang, R.F., Zhang, Z.F., Li, M.J., Ma, W.-Y., & Zhang, H.J. (2005). A probabilistic semantic model for image annotation and multi-modal image retrieval. In Proc. of the IEEE International Conference on Computer Vision. Zhu, L., Zhang, A., Rao, A., & Srihari, R. (2000). Keyblock: an approach for content-based image retrieval. In Proceedings of ACM Multimedia.
ENndno
Note that we do not require it to be perfect, at least in our approach, e.g. given an image about the Eiffel Tower, the "perfect" keyword is "Eiffel" but an accurate one can be "France". This is a crucial technique for text search engines. It stores the keyword-document relationship into a so-called inverted index file, whose entries are keywords and values are the document ids
to represent which document is indexed by a given keyword, or say which document contains this keyword. Although SRC accepts user-assigned number to indicate how many categories are preferred, it is a suggested number that statistically produces the highest performance. We need to keep in mind that our system should be easily scale up, e.g. to support real-time annotation with billions of images. http://www.cs.washington.edu/research/imagedatabase/groundtruth/
Semantic Classification and Annotation of Images Yonghong Tian Peking University, China Shuqiang Jiang Chinese Academy of Sciences, China Tiejun Huang Peking University, China Wen Gao Peking University, China
ABSTRACT With the rapid growth of image collections, content-based image retrieval (CBIR) has been an active area of research with notable recent progress. However, automatic image retrieval by semantics still remains a challenging problem. In this chapter, the authors will describe two promising techniques towards semantic image retrieval—semantic image classification and automatic image annotation. For each technique, four aspects are presented: task definition, image representation, computational models, and evaluation. Finally, they will give a brief discussion of their application in image retrieval.
In With the advance of multimedia technology and growth of image collections, content-based image retrieval (CBIR) is therefore proposed, which finds images that have low-level visual features (e.g., color, texture, shape) similar to those of the query image. However, retrieving images via low-level features has proven unsatisfactory since low-level visual features cannot represent the high-level semantic content Copyright © 2009, IGI Global, distributing in print or electronic forms without written permission of IGI Global is prohibited.
Semantic Classification and Annotation of Images
Figure 1. Examples of images for semantic image categorization.
Inside City
Open Country
Figure 2. Examples of images for automatic image annotation.
Original manual sky jet Plane smoke annotations Automatic sky jet plane water tree annotations
bear close-up face polar
cat rocks tiger water
bear snow polar grass water
water cat tiger people swimmers
flowers porcupine rodent
sky sun tree water
leaf plants flowers sunset water garden horizon sun clouds
of images. To reduce the so-called semantic gap (Smeulders, Worring, Santini, Gupta, & Jain, 2000), a variety of techniques have been developed. In this chapter, we discuss promising techniques on two important aspects of CBIR  (a) semantic image classification, and (b) automatic image annotation. Each component plays an important role in the greater semantic understanding of images. General speaking, the semantics of images can be categorized into four levels from the lowest to the highest (Wang, Li, & Wiederhold, 2001): • • • •
semantic types (e.g., landscape photograph, clip art), object composition (e.g., a bike and a car parked on a beach, a sunset scene), abstract semantics (e.g., people fighting, happy person, objectionable photograph), and detailed semantics (e.g., a detailed description for a given picture).
In (Song, Wang, & Zhang, 2003), the image semantics can be further grouped into local semantic level and thematic level (or global semantic level). Currently, most works in image classification and annotation have been done in the first two levels. Good progress has been made at the first level, which corresponds closely to image's physical attributes such as indoor vs. outdoor. The second level of
semantics seems more difficult to extract than the first. In this chapter, we pay our primary interest to the second level to investigate the problems of image classification and annotation. Semantic image classification aims to classify images into (semantically) meaningful categories. Potentially, the categorization enhances image retrieval by permitting semantically adaptive searching methods and by narrowing down the searching range in a database. Fig. 1 shows several representative categories of images. Roughly speaking, two basic strategies can be found in the literature. The first uses low-level visual features, such as color, texture or shape. Then different classifiers are applied on these visual features (e.g., Szummer & Picard, 1998; Vailaya, Figueiredo, Jain, & Zhang, 2001; Wang, et al., 2001; Grauman & Darrell, 2005; Zhang, Berg, Maire, & Malik, 2006). The second strategy adds an intermediate representation between the low-level features and semantic concepts (e.g., Li & Perona, 2005; Wang, Zhang, & Li, 2006). In the context of Web images, additional features are also exploited by image classification algorithms, such as the text extracted from the page that contains an image (Cai, He, Li, Ma, & Wen, 2004) or the link information between images or pages (Tian, Huang, & Gao, 2005). Automatic image annotation is the process of automatically assigning metadata in the form of captions or keywords to an image. Different with image categorization, annotated words for each image may exhibit a large variability in general (Monay & Gatica-Perez, 2007). That is, several words can describe one or more regions or even the whole image (see Fig. 2), which makes automatic image annotation a distinct research problem. There have been many works done in automatic image annotation, differentiated by the auto-annotation models such as word co-occurrence model (Mori, Takahashi, & Oka, 1999), translation model (Duygulu, Barnard, Fretias, & Forsyth, 2002), cross-media relevance model (Jeon, Lavrenko, & Manmatha, 2003; Feng, Manmatha, & Lavrenko, 2004), hierarchical aspect model (Barnard & Forsyth, 2001; Barnard, Duygulu, Freitas, Forsyth, Blei, & Jordan, 2003), probabilistic latent semantic analysis (pLSA) model (Quelhas, Monay, Odobez, Gatica-Perez, Tuytelaars, & Gool, 2005; Sivic, Russell, Efros, Zisserman, & Freeman, 2005; Monay & Gatica-Perez, 2007), latent Dirichlet allocation (LDA) model (Barnard, et al., 2003; Blei & Jordan, 2003), 2D multi-resolution hidden Markov model (2D-MHMM) (Li & Wang, 2003; Wang & Li, 2002), supervised multi-class labeling (Carneiro, Chan, Moreno, & Vasconcelos, 2006), relevant low-level global filters (Oliva & Torralba, 2001), and so on. In this chapter we shall only be able to summarize some of the various methods. Up to this point in time, automatic image retrieval by semantics still remains a challenging problem due to the difficulty in object recognition and image understanding. However, automatic image categorization and annotation can be used to help generate meaningful retrieval. Thus in Section IV, we will give a brief discussion of image retrieval with the help of image annotation and categorization. Finally, Section V concludes this chapter.
Semantic Imagagation In this section, we begin by formulating the problem of semantic image classification, and then describe the representation models and classification algorithms.
Problem Formulation Semantic image classification is a technique for classifying images based on their semantics from a domain of some given predefined concepts, such as building, waterside, landscape, cityscape, and so
forth. While semantic image classification is a limited form of image understanding, its goal is not to understand images by the way human beings do, but merely to assign images to semantic categories (Wang, et al., 2001). Nevertheless, classifying images is still not an easy task owing to their variability, ambiguity, and the wide range of illumination and scale conditions that may apply (Bosch, Zisserman, & Muyyoz, 2008). Conventionally, semantic image classification is posed as a single-label classification problem where each image is classified into one mutually-exclusive category, such as indoor vs. outdoor (Szummer & Picard, 1998), or textured vs. non-textured (Wang, et al., 2001). Single-label image classification. Consider a database D = {d1,...,dN} of images di and a set of labels C = {c1,...,cL}. Let H be the set of classifiers for D → C. Thus the goal is to find a classifier h ∈ H maximizing the probability of h(d) = c, where c ∈ C is the ground truth label of d, i.e., c = arg max P (ci | d ). i
In practice, different statistical learning approaches, such as Bayesian probability models (Vailaya, et al., 2001; Aksoy, Koperski, Tusk, Marchisio, & Tilton, 2005; Li & Perona, 2005), linear discriminant analysis (Pappas, Chen, & Depalov, 2007) and support vector machines (SVM) (Serrano, Savakis, & Luo, 2004; Zhang, et al., 2006), have been successfully employed for single-label image categorization. In the image classification domain, however, it is likely that many images may belong to multiple semantic classes. Considering examples shown in Fig. 1, the three images of category coast may be also viewed as sky scenes, and some images of category inside city can be classified into category street or building. In this case, the base categories are non-mutually-exclusive and may overlap by definition. Correspondingly, semantic image categorization is formulated as a multi-label classification problem where each image can be described by multiple class labels (e.g., a coast scene with a sky in the background). As before, this problem can be formulated as follows (Boutell, Shen, Luo, & Brown, 2004): Multi-label image classification. Let H be the set of classifiers for D → B where B is a set of C-dimensional binary vectors, each of which indicates membership in the categories in C (+1=member, -1=non-member). Thus the goal is to find the classifier h ∈ H that minimized a distance (e.g., Hamming) Figure 3. Overview of semantic image classification approaches
between h(d) and bd for a new image d, or equivalently, to find one or more class labels in C and for a threshold δ such that P(c | d) > δ, ∀c ∈ C. The mathematical formulation and its physical meaning are distinctively different from those used in the single-label image classification. Moreover, the multi-label classification models can be directly used for automatic image annotation (Goh, Chang, & Li, 2005; Carneiro, et al., 2006). As noted in (Boutell, Brown, & Luo, 2002), the approaches for semantic image classification are divided into two main categories: exemplar-based and model-based. On one hand, most systems use an exemplar-based approach that employs statistical pattern recognition techniques on vectors of lowlevel features (such as color, texture, or edges) or semantic features (such as sky, faces or grass). On the other hand, model-based approaches use the expected configuration of a scene where the scene's configuration is the layout (relative location and sizes) of its objects created from expert knowledge of the scene. However, it is only possible to build a scene model for a constrained scene type and such a model is usually not generalizable to other scene types. From a viewpoint of scene modeling and representation in images, Bosch, Muňoz, & Marti (2007) divided the approaches for semantic image classification into two basic strategies. The first uses lowlevel features such as global color or texture histograms, the power spectrum, and is normally used to classify only a small number of scene categories (indoor vs. outdoor, city vs. landscape etc.) (Szummer & Picard, 1998; Vailaya, et al., 2001; Wang, et al., 2001; Grauman & Darrell, 2005; Zhang, et al., 2006). The second strategy uses an intermediate representation before classifying scenes (Li & Perona, 2005; Wang, et al., 2006), and has been applied to cases where there are a larger number of scene categories. The two approaches are referred to as low-level scene modeling and intermediate semantic modeling respectively. The intermediate semantic modeling provides a potentially larger amount of information that can be exploited to achieve a higher performance on image classification. Fig. 3 summarizes the semantic image classification approaches. Scene modeling and classifier are two main components in the procedure, which will be further discussed in the following subsections.
Scene Modeling Scene modeling is necessary for a number of vision tasks, including classification and segmentation. In this subsection, we simply discuss two scene modeling approaches for image classification. More details can be found in (Bosch, et al., 2007).
Low-Level Modeling The problem of low-level scene modeling has been studied in image and video retrieval for several years (e.g., Szummer & Picard, 1998; Vailaya, et al., 2001; Wang, et al., 2001; Grauman & Darrell, 2005; Zhang, et al., 2006). Color, texture and shape features have been widely used in combination with supervised learning methods to classify images into several semantic categories. The underlying assumption is that the type of scene can be directly described by the visual properties of the image. For instance, a forest scene presents highly textured regions (trees), or a sunset scene typically has saturated colors (mostly yellow and red). According to the way in which low-level features are extracted, it is possible to distinguish two trends of low-level modeling and representation approaches (Bosch, et al., 2007):
1. Global approach: Each scene is represented by low-level visual features from the whole image. In (Vailaya, et al., 2001), different global features are used at each level in the hierarchical classification of vacation images: indoor/outdoor (using spatial color moments); city/landscape (using edge direction histograms and coherence vectors); sunset/forest/mountain (using spatial moments, color histograms and coherence vectors), and so on. In (Zhang, et al., 2006), the image classification problem is addressed in the framework of measuring similarities based on global features such as shape and texture. Similarly, global features are used in (Chang, Goh, Sychay, & Wu, 2003) to produce a set of semantic labels by an ensemble of binary classifiers. Due to the complexity of visual content, good classification performance cannot be achieved by considering only a single type of features alone (Shen, Shepherd, & Ngu, 2005). 2. Block-based approach: The image is first partitioned into several blocks and then low-level visual features are extracted from each block. These blocks are then classified, and finally the scene is categorized from the individual classification of each block. This approach was originally used for indoor-outdoor classification problem in (Szummer & Picard, 1998). Similarly, an image is represented by a set of regions in the SIMPLIcity system (Wang, et al., 2001), each of which is characterized by color, texture, shape, and location. Then the system classifies images into semantic categories, such as textured-nontextured, graph-photograph. There are some other works (e.g., Paek & Chang, 2000; Serrano, et al., 2004) that use this approach for image classification, differentiated by the classifiers that are employed for blocks and the way how to infer the semantic category of the whole image from block classification results. For example, Paek and Chang (2000) developed a framework to combine multiple probabilistic classifiers in a belief network; while Serrano et al. (2004) used SVM for a reduction in feature dimensionality so that the block beliefs can be combined numerically. Even good performance obtained by above proposals, the low-level modeling approach suffers several limitations (Bosch, et al., 2007). Firstly, this approach is normally used to classify only a small number of scene categories (indoor vs. outdoor, textured vs. nontextured, or city vs. landscape, etc). Secondly, it is often difficult to generalize theses methods using low-level visual features to additional image data beyond the training set. Finally, this approach lacks of an intermediate semantic image description that can be extremely valuable in determining the scene type.
Intermediate Semantic Modeling The modeling of scenes by a semantic intermediate representation was next proposed in order to reduce the gap between low-level and high-level image processing, and therefore to match the scene model with the perception we humans have. Nevertheless, the semantic meaning of a scene is not unique, and different semantic representations of images have been proposed. Bosch et al. (2007) summarized three approaches of semantic modeling as follows: 1. Semantic objects: Objects of an image are detected and labeled by local classifiers as known semantic objects, and then the scene is represented by the occurrence of these semantic objects. This approach mainly relies on an initial segmentation of the image into meaningful regions that coarsely match objects or object parts. This assumption translates into several variations of image representation models. In (Aksoy, et al., 2005), scene representation is achieved by decomposing
the image into prototype regions and modeling the interactions between these regions in terms of their spatial relationships. Scene is also represented by regions in (Fredembach, Schröder, & Süsstrunk, 2004), but with the addition of eigenregions (the principal components of the intensity of the region). In (Mojsilovic, Gomes, & Rogowitz, 2002), the image is first segmented using color and texture information to find the semantic indicators (e.g. skin, sky, water), and then these objects are used to identify the semantic categories (i.e. people, outdoor, landscapes). Concept sensitive salient objects are detected in (Fan, Gao, Luo, & Xu, 2005) as the dominant image components for image classification. Moreover, some works (e.g. Luo, Savakis, & Singhal, 2005; Yang, Kim, & Ro, 2007) integrated the low-level and semantic features for classification by a general-purpose knowledge framework that employs a Bayesian Network (BN), or a two-layer SVM classifier. The limitation of this approach is that no automatic segmentation algorithm is currently capable of dividing an image into consistently meaningful objects or object parts. 2. Local semantic concepts: The semantic of an image is represented by intermediate properties extracted from local descriptors. In this case, a dictionary of visual words or local semantic concepts is first identified to define the semantics of images from local information (e.g., blue sky, mountain with snow). The visual words distribution can be learned for each scene category and then used for scene classification. This approach avoids object segmentation and detection by using a fixed-grid layout. Using this representation, three typical image descriptors can be found in the literatures. The first type is the so-called concept occurrence vector (COV), which measures the frequency of different semantic concepts in a particular image. In (Vogel, 2004) and (Vogel & Schiele, 2004, 2007), each regular region is classified into semantic concept classes such as water, rocks, or foliage, and an image is then represented by a COV to measure the occurrence frequency of these local concepts. The average COV over all members of a category defines the category prototype. Scene classification is carried out by using the prototypical representation itself or Multi-SVM approaches. Inspired by statistical text analysis, the second type of image descriptors is bag-of-words or bag-ofvisterms (Mori, et al., 1999), formed by using a visual analogue of a word and vector-quantizing local Figure 4. Illustration of four steps to compute the bag-of-words when working with image (quoted from Fig. 5 in (Bosch, et al., 2007))
visual features. Recent works have shown that bags-of-words representation is suitable for scene classification, demonstrating impressive levels of performance (Li & Perona, 2005; Quelhas, et al., 2005; Bosch, Zisserman, & Muňoz, 2006; Perronin, 2007; Bosch, et al., 2008). As shown in Fig. 4, constructing the bag-of-words from images involves the following steps (Bosch, et al., 2007): (1) Automatically detect regions/points of interest, (2) compute local descriptors over those regions/points, (3) quantize the descriptors into words to form the visual vocabulary, (4) find the occurrences in the image of each specific word in the vocabulary in order to build the bag-of-words (histogram of words). Exiting works can be differentiated by the way how the features are used. Scale-Invariant Feature Transform (SIFT) (Lowe, 1999) features are computed on a regular grid (Li & Perona, 2005; Bosch, et al., 2006) or around an interest point (Quelhas, et al., 2005). Different with the above approaches that characterize an image with a single histogram, Perronin (2007) represented an image as a set of histograms, one per class describing whether an image is more suitably modeled by the universal vocabulary or a class-specific visual vocabulary. Using this bag-of-words representation, an image containing instances of several objects is modeled as a mixture of topics or object categories. This topics distribution over the images can be used to classify an image as belonging to a certain scene. Some probabilistic latent space models, such as probabilistic latent semantic analysis (pLSA) model (Quelhas, et al., 2005; Sivic, et al., 2005; Bosch, et al., 2006, 2008) and latent Dirichlet allocation (LDA) model (Li & Perona, 2005; Sivic, et al., 2005), have been adapted to model scene categories. In the next subsection, we will further discuss how pLSA and LDA are used for image categorization. The third type is a variation of bag-of-words descriptor by using contextual knowledge like common spatial relationships between neighboring local objects or the absolute position of objects in certain scenes. The underlying assumption is that spatial structure could provide valuable information to achieve better results for scene classification. In (Lazebnik, Schmid, & Ponce, 2006), the spatial-pyramid is proposed as a simple computationally efficient extension of an orderless bag-of-words image representation by partitioning the image into increasingly finer sub-regions and computing histograms of local features found inside each sub-region. In (Fergus, Li, Perona, & Zisserman, 2005), pLSA is extended to include absolute position and spatial information in a translation and scale invariant manner respectively. 3. Semantic properties: The semantic of an image is described by a set of statistical properties/qualities of the image, including naturalness (vs. man-made), openness (presence of a horizon line), roughness (fractal complexity), expansion (perspective in man-made scenes), and ruggedness (deviation from the horizon in natural scenes) (Oliva & Torralba, 2001). Each feature corresponds to a dimension in the spatial envelope space, and all features together represent the dominant spatial structure of a scene. These properties may be reliably estimated using spectral and coarsely localized information. A significant advantage of this modeling approach is that the semantic is related to global configurations and scene structure, instead of local objects or regions, and consequently, neither segmentation nor the processing of local regions or objects is required (Bosch, et al., 2007). In conclusion, the intermediate semantic modeling provides a potentially larger amount of information that must be exploited to achieve a higher performance on scene classification. However, it also makes more difficult the problem we are tackling because it habitually involves a local/region processing like (a not necessarily accurate) object recognition (Bosch, et al., 2007).
Classification Algorithms Given the image representation, one primary tackling point is learning approach itself so that classifier realizes minimal bound of error in real tasks. In existing systems, many popular statistical learning approaches have been successfully used. In this subsection, we will describe several representative classification algorithms for images.
Hierarchical Image Classification with Low-level Scene Representation In 2001, Vailaya et al. show that within a constrained domain, low-level features can successfully discriminate between many scene types arranged hierarchically. In the hierarchical classification task, each image is represented by a feature vector extracted from the image and different visual features are used at each level depending on the classification problem. Then the classification problem is addressed by using Bayesian decision theory. Formally, consider N training samples, each belonging to one class in C = {c1,...,cL}. A vector quantizer is used to extract q codebook vectors, vj, from the N training samples. The class conditional density of a feature vector y = {y(1),..., y(m)} given the class c ∈ C, i.e., f (y | c), is then approximated by a mixture of Gaussians (with identity covariance matrices), each centered at a codebook vector, resulting in: q m  || y ( i ) - v (ji ) ||2 f (y | c) ∝ ∏ ∑ m(ji ) exp   2 2 i =1 j =1 
  
where mj(i) is the proportion of training samples assigned to vj. The Bayesian classifier is then defined using the maximum a posteriori (MAP) criterion as follows: cˆ = arg max{ p (c | y )} = arg max{ f (y | c) p (c)} c∈C
c∈C
where p(c) represents the a priori class probability. Their system achieved a good performance at each level of the hierarchy over a set of 6931 vacation photographs. Many other works also exploit low-level scene modeling for semantic image classification. For example, three kinds of classifiers, KNN, SVM and GMM, are used in (Shen, et al., 2005). SVM is also used in (Serrano,et al., 2004) for a reduction in feature dimensionality without compromising image classification accuracy. Due to the space limitation, the details are not included in this chapter.
Scene Classification Using Semantic Object Detection The salient objects are defined as the dominant image components that are semantic to human being and are also visually distinguishable (Fan, et al., 2005). For example, the salient object "sky" is defined as the connected image regions with large sizes (i.e., dominant image components) that are related to the human semantics "sky". By detecting the concept-sensitive salient objects, an image classification technique was developed in (Fan, et al., 2005) to achieve automatic image annotation at the concept level. Thus here we use this approach as an example to illustrate how to detect salient objects in images and then how to exploit them for semantic image classification.
To detect the concept-sensitive salient objects, a set of 32 detection functions are learned in oneagainst-all manner from the labeled image regions, each for a specific type of these salient objects. A detection function consists of three steps: (1) automatic image segmentation by using mean shift algorithm, (2) binary region classification by using SVMs with an automatic scheme for searching the optimal model parameters and (3) label-based aggregation of the connected similar regions for salient object generation. This process is automatic and the human interaction is only involved in the procedure to label the training samples (i.e., homogeneous regions) for learning the detection functions. The semantic image classification system takes the following steps: (1) Given a specific test image di (which may consist of multiple types of different salient objects), the underlying salient objects di = {o1,...,on} are detected automatically. (2) The class distribution of the salient objects di = {o1,...,on} is then modeled as a finite mixture model (FMM) P (y , c j | , c j , c j ). (3) The test image di is finally classified into the best matching semantic image concept cj with the maximum posterior probability. P (c j | y , d i , ) =
P(y , c j| ,
P(y, c j | , j =1
) ( j) cj
) ( j)
where = { , c j , c j | = 1, , } is the set of the mixture parameters (i.e., c j and the mixture component number k) and relative weights (i.e., wc ) for the classifiers, y is the n-dimensional visual features j for representing the relevant concept-sensitive salient objects, and P(cj) is the prior probability of the semantic concept cj. An adaptive Expectation-Maximization (EM) algorithm is proposed to determine the optimal model structure and parameters simultaneously. Similar scene classification algorithms can also be found in (Mojsilovic, et al., 2002; Fredembach, et al., 2004; Aksoy, et al., 2005; Luo, et al., 2005). The authors argue that with the more advances of automatic image segmentation, better classification could be achieved by using this approach.
Scene Classification with Latent Space Representation Recently, the bag-of-words model has been widely used in image classification and made great progress. However it may suffer from two issues (Quelhas, et al., 2005): polysemy  a single visual word (i.e., visterm) may represent different scene content, and synonymy  several visterms may characterize the same image content. In order to disambiguate the representation, probabilistic latent space models, such as pLSA (Hofmann, 2001) and LDA (Blei, Ng, & Jordan, 2003), have been applied to capture cooccurrence information between visterms in a collection of images. Here we simply describe how these models are used to latent topic discovery in semantic image classification. Suppose that we have an image set D = {d1,...,dN} with words from a visual vocabulary W = {w1,...,wV}. The data is a V × N co-occurrence table of counts Nij = n(wi, dj), where n(wi, dj) denotes how often the visterm wi occurred in an image dj. pLSA (Hofmann, 2001) assumes the existence of a latent variable z ∈ Z in the generative process of each word w ∈ W in an image d ∈ D. As shown in Fig. 5(a), the observed variables w and d are conditionally independent given the latent variable z, thus a joint probability P(w, z) over V × N is defined by the mixture: P ( w | d ) = ∑ z∈Z P( w | z ) P( z | d )
where P(w | z) are the topic-specific distributions, and each image is modeled as a mixture of topics, P(z | d). The equation expresses each image as a convex combination of |Z| topic vectors. This amounts to a matrix decomposition as shown in Fig. 5(b) with the constraint that both P(w | z) and P(z | d) are normalized to make them probability distributions. Given that both w and z are discrete variables, the conditional probability distributions P(z | d) and P(w | z) are multinomial. The parameters of these distributions are estimated by the EM algorithm (Hofmann, 2001). Given a new image dnew, P(z | dnew) can be inferred by using the folding-in method, which maximizes the likelihood of the image dnew with a partial version of the EM algorithm while keeping the learned P(w | z) fixed. More details about pLSA can be found in (Hofmann, 2001). pLSA is appropriate to disambiguate the bag-of-words representation because it provides a correct statistical model for clustering in the case of multiple object categories per image (Quelhas, et al., 2005). Consequently, pLSA can be used to generate a robust, low-dimensional scene representation. Given a new image dnew, the model determines the mixture coefficients P(z | dnew) for each image dnew where z ∈ Z. The image dnew is then classified as the semantic category z according to the maximum of P(z | dnew) over z ∈ Z. In (Quelhas, et al., 2005) and (Sivic, et al., 2005), pLSA is applied on sparse SIFT features for recognizing compact scene categories, where SIFT features are computed around an interest point. While in (Bosch, et al., 2006, 2008), pLSA is applied on dense SIFT features to obtain a better scene categorization, where SIFT features are computed on a regular grid and using concentric patches around each point to allow scale variance. Moreover, spatial information can also be exploited in the framework of pLSA to improve scene classification performance. In 2005, Fergus et al. develop two new models, ABS-pLSA and FSI-pLSA, by extending pLSA to include absolute position and spatial information in a translation and scale invariant manner respectively. For example, the ABS-pLSA model quantizes the location within the image into one of X bins. The joint density on the appearance and location of each region is then represented. Thus P(w | z) in pLSA becomes P(w,x | z), a discrete density of size (V × X) × N. Inspired by ABS-pLSA and spatial-pyramid matching (SPM) algorithm (Lazebnik, et al., 2006), Bosch, et al. (2008) proposed the SP-pLSA model by using the X bins at each resolution level l, weighting the bins for each level as in SPM. Thus P(w | z) in pLSA becomes P(w, x, l | z). The same pLSA update equations can be applied to these models in learning and recognition. In contrast to pLSA, LDA treats the multinomial weights P(z | d) over topics as latent random variables. The LDA model is extended by sampling those weights from a Dirichlet distribution, the conjugate prior to the multinomial distribution. This extension allows the LDA model to assign probabilities to data outside the training corpus and uses fewer parameters, thus reducing overfitting (Sivic, et al., 2005). Graphically, LDA is a three-level hierarchical Bayesian model, in which each item of a
Figure 5. (a) pSLA graphical model. Filled circles indicate observed random variables; unfilled are unobserved. (b) Graphical representation of the matrix decomposition form of pLSA. (c) LDA graphical model.
collection is modeled as a finite mixture over a set of latent aspects (Blei, et al., 2003). Each aspect is, in turn, modeled as an infinite mixture over an set of laten aspect probabilities. The LDA model is shown in Fig. 5(c). The goal is to maximize the following likelihood:   P ( w | , ) = ∫ P( | )  ∑ P( z | ) P( w | z , )  d  z∈Z 
where θ is the multinomial parameter over the topics, and P(θ | α) is Dirichlet distribution parameterized by the hyper-parameter α. As the figure makes clear, there are three levels to the LDA representation. The parameters α and β are collection-level parameters, assumed to be sampled once in the process of generating a collection. The variables θd are image-level variables, sampled once per image. Finally, the variables zd and xd are element-level variables and are sampled once for each element in each image. n n More details about LDA can be found in (Blei, et al., 2003). In the LDA case, the unseen mixed-category images can be classified based on the topic mixture weights θ, which can be computed using the samples drawn by the Gibbs sampler (Sivic, et al., 2005). Li and Perona (2005) proposed two variations of LDA to represent and learn image categorization models, in which local regions are first clustered into different intermediate themes (local concepts) and then into categories. Probability distributions of the local regions as well as the intermediate themes are both learnt automatically, bypassing any human annotation apart from labeling a single category to the training images.
Web Image Classification With the rapid growth of digital images on the Internet, Web image classification has become an important research topic. Web images have a lot of properties which are quite different from those images in small database such as Corel and family album (Cai, et al., 2004). For example, Web images are associated with text and link information. Many works (e.g., Hu & Bagga, 2004; Kalva, Enembreck, & Koerich, 2007) use the text content from the document that contains an image as the semantic features for classifying that image. In (Lempel & Soffer, 2002; Cai, et al., 2004; Wang, Ma, Xue, & Li, 2004), link information is also exploited to improve Web image retrieval. Here we will describe how to exploit visual, textual and link information for Web image classification by using our work (Tian, et al., 2005) as an illustrating example. Fig. 6(a) illustrates the relational representation of images in the context of Web. Given a collection D = {d1,...,dN} of Web images, each image di ∈ D can be represented by a vector of global or local visual features, fi(V). By extracting the surrounding text (and other related text such as the filename and URL of the image), each image can also be represented by a textual term vector fi(T), in which each term can be weighted by its term frequency (TF) or term frequency inverse document frequency (TFIDF). Then in the joint visual-textual feature space, each image can be represented as fi(C) = fi(V) , fi(T) . Moreover, with the assumption that images which are co-contained in pages are likely to be related to the same topic, and images which are contained in pages that are co-cited by a certain page are likely related to the same topic (Lempel & Soffer, 2002), we can derive the image adjacency matrix AI~I among the images in D. The matrix AI~I naturally defines an image graph GI, and can be used to derive semantically richer representation of linked images. The first representation is derived directly from the matrix AI~I, referred to as linkage relationship vector (LRV). Namely, the link features of each image di can be
represented by the i-th column fi(L) of AI~I. Alternatively, each image di can be represented by a classbased weighted vector  fi (L) =  wi ,1 , wi ,2 , , wi ,5 L ,
where wi,2(1 ≤ k ≤ 5L) are the weighted frequencies of the five important link relations (i.e., co-containedness, in-link, out-link, co-citation and co-reference) between image di and its neighboring images of different classes, L is the number of the class labels C = {c1,...,cL}. This representation is referred to as class-based LRV (CLRV). By extending SVM to the relational domain, a relational support vector classifier (RSVC) is proposed. If the vector ά and the scalar b are the parameters of the hyperplane learned from the training images, then the RSVC decision rule is defined as:  l  yi = sgn  ∑ j y j (1 - ) K (fi(C) , f j(C) ) + K (fi(L) , f j(L) )  + b   j =1  l   s= gn  ∑ j y j K (fi(L) , f j(L) ) + i  ,  j =1 
where yi is its class label indicator, K (fi(C) , f j(C) ) and K (fi(L) , f j(L) ) are the content kernels and linkage kernels respectively, β is the combination weight, and l
= (1 - )∑ j =1
y j K (fi(C) , f j(C) ) + b
is the decision function of a conventional SVM, and l denotes the number of support vectors (SVs). Given a new image dnew, the link feature of dnew is computed by adding dnew to the collection of the training and labeled test images D, and then RSVC is used to classified the image dnew. On a sports Web image collection crawled from Yahoo!, RSVC yields significant improvement in accuracy over SVM using visual and/or textual features.
Figure 6. (a) The representation of Web images. (b) An example link graph of Web images Cameraman
MakedBy Image
Visual Features
MakedAt
Textual Features
Accompanying (Surrounding)
LinkTo Page P
Discussion This section is meant to capture the landscape of semantic image classification by clearly formulating the problem, and simply discussing recent representative works from the viewpoints of scene modeling and classification algorithms. We now discuss some research issues connected to the ideas discussed here. First, latest trends are using the bag-of-words representations jointly with different latent topic discovery techniques (Bosch, et al., 2007). However, bag-of-words representation should be further improved by using contextual knowledge. It's only a good start point that the spatial information is taken into account in this representation. Second, as mentioned in (Boutell, et al., 2004), seldom works address multi-label image classification. Such a problem poses challenges to the classic pattern recognition paradigm and demands a different treatment. Finally, in spite of progress, no semantic classification algorithm is currently capable of classifying images with complex scene into a very large number of classes with relatively high accuracy. Indeed, classifying images with complex scene is still a challenging task owing to their variability, ambiguity, and the wide range of illumination and scale conditions that may apply. Such image classification methods are extended for image annotation with the goal of approaching greater semantic understanding of images.
Auotic Imagagtion In this section, we begin by presenting two typical tasks of image annotation. Then, we shall discuss two important aspects for automatic image annotation  image representation and auto-annotation models. Finally, we shall discuss how to evaluate the image annotation systems.
Task Definition Typically, automatic image annotation systems take advantage of annotated images to link the visual and textual modalities by using machine learning techniques. Accordingly, they can be posed as a task of either supervised or unsupervised learning (Carneiro, et al., 2006).
Supervised Annotation In the supervised annotation systems (Chang, et al., 2003; Li & Wang, 2003; Smith, Lin, Naphade, Natsev, & Tseng, 2003; Carneiro, et al., 2006), multi-label image classification algorithms are extended for image annotation: All training images are labeled different concepts, where each concept contains a group of semantically related keywords that describe different aspects of that concept. For example, the concept "wild animal" can be described as "animal, wild life, grass, snow, rock" (Li & Wang, 2003). Given these categorized images, one generative or discriminative classifier is trained for each concept and then applied to all test images which are annotated with respect to the presence or absence of the concept. One important problem is how to statistically model images for each concept. In (Carneiro, et al., 2006), images are represented as bags of localized feature vectors, and the mixtures associated with all images annotated with a common concept pooled into a density estimate for the corresponding class. Annotation is then performed with a minimum probability of error rule, based on these class densi-
ties. In (Li & Wang, 2003), a 2D multi-resolution hidden Markov model (2D-MHMM) is learned on a fixed-grid segmentation of all category examples. The likelihood of a new image given each category's 2D-MHMM allows assigning words for this image. Similarly, one SVM is trained for each concept in the model vector indexing approach (Smith, et al., 2003) while a binary classifier (Bayes Point Machine and SVM) is trained for each word (totally 116 words) in Content-Based Soft Annotation (CBSA) system (Chang, et al., 2003). The major advantage of supervised annotation is that models for different concepts can be independently trained and retrained. If images representing new concepts or new images in existing concepts are added into the training database, only the models for the involved concepts need to be trained or retrained. However, if the vocabulary size is very large, there are a relatively large number of models that need to be elaborately trained. Moreover, since a concept contains a group of keywords that have a related meaning, labeling an image by all keywords of the concept may produce incorrect annotations. For example, not all images of the concept "wild animal" have the visual content that corresponds to snow or rock. Automatic image annotation systems therefore tend to unsupervised learning approach in recent years.
Unsupervised Annotation Most automatic image annotation systems (e.g., Mori, et al., 1999; Barnard & Forsyth, 2001; Duygulu, et al., 2002; Barnard, et al., 2003; Blei & Jordan, 2003; Jeon, et al., 2003; Feng, et al., 2004; Monay & Gatica-Perez, 2007) tend to solve the problem in greater generality by resorting to unsupervised learning. The basic idea is to introduce one auto-annotation model to characterize the relation between words and visual features. During training, a set of words is assigned to each image, and unsupervised learning techniques are run over the entire database to train one auto-annotation model for the whole vocabulary. Given a new image, visual features are extracted, and inference on the learned model is then used to assign annotations to the image. A whole range of statistical assumptions about visual features and image annotations can lead to different auto-annotation models, from the simple empirical distribution estimation to complex generative or discriminative models. Typical unsupervised auto-annotation models include word co-occurrence model (Mori, et al., 1999), translation model (Duygulu, et al., 2002), cross-media relevance model (Jeon, et al., 2003; Feng, et al., 2004), multi-modal hierarchical aspect model (Barnard & Forsyth, 2001; Barnard, et al., 2003), pLSA model (Monay & Gatica-Perez, 2007), LDA model (Barnard, et al., 2003; Blei & Jordan, 2003), etc. The main advantage of unsupervised annotation is that only one model is learned for all words in the vocabulary. Consequently, this approach is better suited for large vocabularies than the supervised learning approach. As mentioned in (Barnard & Forsyth, 2001), however, one major limitation is that this approach relies on semantically meaningful segmentation while automatic segmentation is still an open problem in computer vision (Shi & Malik, 2000).
Image Rpresentation All image annotation approaches use either a fixed-grid layout or an image segmentation algorithm to divide images into a number of irregularly-shaped regions. This entails that the features should be extracted from local image regions for annotation. To extract such local features and build the auto-
annotation models, there are three kinds of image representations: blob-based, grid-based and keypointbased. The first relies on large-scale image regions; while the two others are based on a larger number of smaller scale regions, uniformly extracted from a fixed grid or identified by a point detector. The three representations encode different image properties and are therefore expected to produce different effects on image annotation. Fig. 7 shows several examples of the three image representations.
Blob-Based Representation Blobs were proposed by Duygulu et al. (2002) to capture the content of an image with a set of homogeneous regions. As shown in Fig. 7(b), each image can be represented as a certain number of blobs. These blobs may coarsely match objects or object parts. This naturally brings up an assumption that there is some implicit correspondence between regions and words in the annotated images. To build blobs, images are first segmented into regions using the normalized cut segmentation algorithm (Shi & Malik, 2000) which requires a maximum count of regions to avoid too small regions. Each region is represented by 36 features including color (18), texture (12), and shape/location (6). The K-means clustering algorithm is then applied to the region descriptors, quantizing them into a multi-dimensional blob representation. Let m denote the blob number in image d, then the image d can be represented by the corresponding histogram b(d) of size m (Monay & Gatica-Perez, 2007) b(d ) = {n(b1 ),, n(bi ),, n(bm )}
where n(bi) denotes the number of regions in image d that are quantized into the blob bi. Similar to the word-by-document matrix in text mining, the blob-by-image matrix representation is also proposed in (Pan, Yang, Duygulu, & Faloutsos, 2004) by weighting with the TFIDF scheme. Blob-based image representation has been widely used in automatic image annotation and retrieval, such as (Barnard & Forsyth, 2001; Duygulu, et al., 2002; Barnard, et al., 2003; Jeon, et al., 2003; Pan, et al., 2004). However, the effect of blobs depends on the automatic segmentation algorithms which haven't achieved human-like results. Moreover, as mentioned in (Feng, et al., 2004), the match between the segmented regions and objects in images is relatively poor in general.
Grid-Based Representation Instead of the use of a segmentation algorithm, some works (e.g. Mori, et al., 1999; Wang & Li, 2002; Li & Wang, 2003; Feng, et al., 2004) use the grid-based representation for image annotation with the help of a fixed-grid layout (as shown in Fig. 7(c)). Compared with blob-based representation, grid-based representation provides a number of advantages (Feng, et al., 2004), including significant reduction in the computational time, simplified parameter estimation, and the easier ability to incorporate context (e.g., relative position) into the auto-annotation model. One major limitation of this representation is that grid regions lack of significant semantic correspondence with objects or object parts. Using grid-based representation, local visual features can be extracted and quantized for every grid region. In (Wang & Li, 2002) and (Li & Wang, 2003), every image is characterized by a set of wavelet feature vectors extracted at multiple resolutions and spatially arranged on a pyramid grid. In (Mori, et al., 1999), the color and texture representations of all training image blocks are quantized into a finite set of visual terms (visterms). As mentioned in the previous section, this quantization form is called
Figure 7. Examples of image representations for annotation
(a) Original images
(b) Images with blobs
(c) Images with fixed-grids
(d) Images with keypoints
bag-of-visterms or directly bag-of-words. Bag-of-visterms ignores the spatial relationship between regions in an image. However, recent research results show that regions are spatially relevant and not independent of the global meanings of images. For example, the neighboring regions may have a strong recommendation of visual content (Ghoshal, Ircing, & Khudanpur, 2005). By incorporating spatial relationship, the auto-annotation models can obtain comparable annotation performance.
Keypoint-Based Representation Keypoints (also as salient points) refer to a set of local features detected in images such Harris corner points. Numerical feature vectors can be extracted from these keypoints such as Scale-Invariant Feature Transform (SIFT) (Lowe, 1999). SIFT transforms an image into a large collection of local feature vectors, each of which is invariant to image translation, scaling, and rotation, and partially invariant to illumination changes and affine or 3D projection. Recently, this representation has been successfully combined with probabilistic latent space models and has shown good performance in scene modeling (Li & Perona, 2005; Quelhas, et al., 2005) and image annotation (Monay & Gatica-Perez, 2007). The construction of the SIFT feature vector from an image involves the steps illustrated in Fig. 8. In this process, the image is first sampled with the difference-of-Gaussians (DoG) point detector at different scales and locations. SIFT descriptor is then computed on affine covariant region around the interest point. Each SIFT descriptor consists of a histogram of edge directions at different locations. All SIFT descriptors are then quantized by the K-means clustering algorithm to obtain a discrete set of local image-patch indexes. As a result, an image is represented by the histogram of its constituting local patches. The details of SIFT descriptor computation can be found in (Lowe, 1999).
Auto-Annotation Models Independent of what features are chosen, the question is how to model the relation between textual and visual features to achieve the best textual indexing. As the pioneering work, the co-occurrence model (Mori, et al., 1999) uses a fixed-grid image segmentation and a vector quantization step to transfer a set of words assigned to the whole image to each image part (i.e., visterm) and then model the co-occurrence of words with visterms. As pointed out by Feng, et al. (2004), however, the approach tends to annotate unknown images with high frequency words and requires large number of training samples to estimate the correct probabilities. Thereafter, a whole range of models and methods have been proposed in the literature, offering a large variety of approaches. In this subsection, we will describe four main models  translation model, cross-media relevance model, probabilistic latent space model and 2D-MHMM model.
Translation Model In 2002, Duygulu et al. proposed the translation model by treating image annotation as the task of translating from a vocabulary of blobs to a vocabulary of words. In this model, the word and blob modalities are considered as two languages. Given a set of training images, a classical machine translation model is used to annotate a test set of images: N
p ( w | b) = ∏∏ ∑ p (anj = i )t ( w = wnj | b = bni ),
n =1 j =1 i =1
where N is the number of images, Mn and Ln are the numbers of words and blobs in the n-th image respectively, an denotes the assignment an = {an1 ,, anM n } such that anj = i if blob bni translates to word wnj, p(anj = i) is the assignment probability, and t(w | b) is the probability of obtaining instance of word w given instance of blob b. An EM procedure is then proposed to estimate the probability distributions linking words and blobs. Once the model parameters are learned, words can be attached to a new image region. This region naming process is analogous with learning a lexicon from an aligned vocabulary. The translation model is a substantial improvement over the co-occurrence model. However, this model can capture the correlation between blobs and words but the dependencies between them are not captured. Moreover, the training stage with the EM algorithm requires a high computational effort and is very time-consuming when dealing with large datasets.
Cross-Media Relevance Model Also relied on a quantized blob image representation, the cross-media relevance model was proposed by Jeon, et al. (2003) to capture the complex dependency between blobs and words in images (not a Figure 8. SIFT feature extraction and quantization framework
simple one-to-one correspondence). Consider an example. In the image context, "tigers" are more often associated with "grass, water, tree or sky" and less often with man-made objects such as "car" or "computer". Therefore, it is assumed that for each image di = {wi1,, wiM , bi1,,biL } there exists some i i underlying probability distribution P(• | di) (called the relevance model of di). The relevance model can be thought of as an urn that contains all possible blobs that could appear in image di, as well as all words that could appear in the caption of di. To annotate a test image di, we need estimate the probability P(w | di) for every word w in the vocabulary,
P ( w | di ) ≈ P( w | bi1 ,, biLi ) =
P ( w, bi1 ,, biLi )
P ( w, bi1 , , biLi ) w
Then the joint distribution can be computed as an expectation over the training image set T,
P ( w, b1 , , bm ) =
∑ P(d
d k ∈T
) P ( w| d k )∏ P (bi | d k ) i =1
Here the conditional probability of a word w (or blob bi) given a training image dk, i.e., P(w | dk) (respectively, P(bi | dk)), is estimated by the count of word w (respectively, blob bi) in image dk smoothed by the average count of word w (respectively, blob bi) in the training set T. An extended version of this model uses the multiple Bernoulli distribution for modeling image annotation (Feng, et al., 2004). Compared to the translation model, the cross-media relevance model performs much better on the same dataset (with the same training and test images and the same features).
Probabilistic Latent Space Models More recently, probabilistic latent space models have attracted much attention in image annotation. A hidden variable (latent aspect) is assumed in the data generative process, which links the textual and visual modalities through conditional relationships. Under this assumption, an image is modeled as a mixture of latent aspects that generates the image regions (or features). A subset of these latent aspects is then selected to generate the text caption, which intuitively corresponds to the natural process of image annotation. In practice, this assumption translates into several model variations, including pLSA (Monay & Gatica-Perez, 2007) and LDA (Barnard, et al., 2003; Blei & Jordan, 2003). pLSA. Under the framework of PLSA, an image is modeled as a mixture of latent aspects that generates both image features and text words. To learn a pLSA model for the co-occurrence of visual and textual features in annotated images, the simplest method is a direct application of pLSA to the early integration of visual and textual modalities (Monay & Gatica-Perez, 2003). An asymmetric PLSA learning algorithm is proposed in (Monay & Gatica-Perez, 2007) to allows choosing between the textual and visual modalities to estimate the conditional distributions of latent aspects given the training images. Their experiments show that the best performance is achieved when the mixture of latent aspects is learned from the text words. LDA. Several variations of LDA have been proposed for image annotation, including correspondence LDA (Corr-LDA) (Blei & Jordan, 2003), Multi-modal LDA (MoM-LDA) (Barnard, et al., 2003). Corr-LDA is depicted in Fig. 9(a). This model can be viewed in terms of a generative process that first generates L region descriptions bi and subsequently generates the caption words. That is, for each of the M words, one of the regions is selected from the image and a corresponding word wj is drawn, conditioned on the
factor that generated the selected region. The distribution over words conditioned on a region an image is approximated by (Blei & Jordan, 2003) L
p ( w | b ) = ∑ ∑ q ( zi | i =1 zi ∈Z
) p( w | zi , ),
where q(zi | fi) is the approximate posterior distribution over the latent variables given a particular image/caption, fi are L K-dimensional multinomial parameters. Experimental results show that the additional constraint on word generation improves the overall annotation performance over less constrained LDA-based models. As shown in Fig. 9(b), MoM-LDA is another multi-modal extension to LDA. Given an image and a MOM-LDA, we can compute both an approximate posterior over mixture components and, for each mixture component, an approximate posterior Dirichlet over latent aspects. Using these parameters, we can perform image-based word-prediction by finding the corresponding distribution over words. Let j denote the approximate posterior over mixture components, and ga denote the corresponding approximate posterior Dirichlet. The distribution over words given an image (that is, a collection of blobs) is:
p ( w | b ) = ∑ p ( | ) ∑ p ( w | zi ) ∫ p ( zi | ) p ( |
)d .
zi ∈Z
One of the advantages of LDA-based mixture models is that they provide well-defined inference procedures for previously unseen images (Blei, et al., 2003). Thus they are expected to have more applications in the area of image annotation in the future.
2D Multi-Resolution Hidden Markov Model By adopting a supervised learning strategy, Li and Wang (2002; 2003) introduced a statistical modeling approach to the problem of automatic linguistic indexing of pictures. To model the stochastic process of associating an image with the textual description of a concept, a 2D-MHMM is learned on a fixed-grid segmentation of all category examples. In this model, every image is characterized by a set of wavelet feature vectors extracted at multiple resolutions and spatially arranged on a pyramid grid. Given each category's 2D-MHMM, the likelihood of a new image allows selecting words for this image. As mentioned before, the major advantages of this model are (1) models for different concepts can be independently trained, stored and retrained, and (2) spatial relation among image pixels within and across resolutions is taken into consideration. Figure 9. Graphical representation of different variations of LDA models for automatic image annotation
(a) CORR-LDA model
(b) MoM-LDA model
Evaluation Data Sets Corel image dataset is the de facto benchmark dataset used in the evaluation of image annotation. This set of 60,000 images consists of 600 CD-ROMs, each of which corresponds to a semantic concept or topic (e.g., beach, buildings, buses, dinosaurs) and typically contains about 100 images. Each image is described by several keywords from a controlled vocabulary. The Corel dataset was originally used in (Duygulu, et al., 2002) with a subset with 5,000 annotated images (denoted by Corel5K). Corel5K has been widely used as an evaluation standard by a number of research groups (e.g., Jeon, et al., 2003; Feng, et al., 2004; Carneiro, et al., 2006; Zhang, et al., 2006). Another oft-used Corel subset, denoted by Corel16K, consists of 16,000 annotated images (Barnard, et al., 2003; Monay & Gatica-Perez, 2007) and is an extension of Corel5K. By the adoption of such de facto standard datasets, significant progress has been accomplished in the recent past. However, the Corel image collection has been criticized because the annotations are nouns selected from a relatively small vocabulary and is relatively simple from an image retrieval point of view (Barnard, et al., 2003; Zhang, et al., 2006). Caltech101 is another widely used dataset for image annotation. Recently, the authors of Caltech101 have released a new version of their dataset, called Caltech256, including 256 picture categories. Some new image sets are also introduced in recent years for the evaluation of image annotation, by downloading real-world images from Internet (Chang, et al., 2003; Zhang, et al., 2006).
Performance Measures A whole ra

redpajama_set_name: RedPajamaCommonCrawl