Paper Review: Photo-based question answering

Interesting paper with some ideas on how to use images for searching and question answering.

Yeh, T., Lee, J. J., and Darrell, T. 2008. Photo-based question answering. In Proceeding of the 16th ACM international Conference on Multimedia (Vancouver, British Columbia, Canada, October 26 - 31, 2008). MM '08. ACM, New York, NY, 389-398.

This paper describes a three step method for answering photo-based questions. In traditional text-based query systems there are some problems when the questions performed by the user are centered on physical objects with distinct visual attributes. For example in the question “where can I buy this poster?” using text the user has to perform the question and accurately describe the object or image desired. Using photos the user would instead perform the question and submit an image of the desired poster reducing the amount of text required for the query. To answer the query the authors propose a three-layer system architecture. The first, is a template-based QA, where it takes the question and sees if there is a image database associated with it, then tries to find the image on that database. If it fails the information retrieval layer searches an internal repository to find similar questions already answered. If all fails the last layer answers the query using human-computation using experts or voluntary users feeding the result to the IR layer. In this work three prototypes are presented, one to do photo-based QA in Flikr other for Yahoo!Answers and a mobile application where the user takes pictures of physical objects and executes a question with the resulting image.