Content-based image retrieval + arts & humanities

Has computer imaging science managed to develop useful automated retrieval options for the formal attributes of (digitized) images, including works of art?

The label for this domain of computer science is content-based image retrieval (acronym: CBIR). It has been, and still is, a promising field of study, with large international conferences and research programs and great numbers of institutions and computer scientists seeking to develop retrieval tools. But is it useful for the arts & humanities? A quick-and-dirty inspection of literature shows that CBIR succeeds fairly well in building systems for the retrieval of images according to static image characteristics (viz. hue predominance, texture, some shape features and gross image structures), calculated for images as a whole – the region of interest is the entire image -, but that as a matter of fact the adjective “content-based” is misleading, since it has neither anything to do with Hjelmslev’s notion of content (data derived from (…) the domain objects being represented by (…) images, i.e. as opposed to what is on the plane of expression: data derived from images as such [SMOLIAR 1997]), nor with what leading art historians like Panofsky assume to be the “content” of artworks including the world of symbolical values [HARPRING 2002] . Eakins & Graham [EAKINS 1999] proposed to characterize image queries into three levels of abstraction:

  • (…) primitive features such as colour or shape,
  • logical features such as the identity of objects shown,
  • and abstract attributes such as the significance of the scenes depicted

Now CBIR is only partially applicable to the first item of this list which, by the way, is not completely equal to the often cited three partite scheme of Erwin Panofsky.

To get an impression of what CBIR can do try retrievr, an interactive application, based on CBIR technology. Retrievr has some interesting features – related to human computation – which I shall discuss elsewhere. The application is accessible via: It is really simple: just sketch with a simple tool any configuration of lines and colors, whereupon the retrieval engine will search for similar images in a collection of (mostly) photographs from flickr. Here is a screendump of one of my tryouts:

screendump retrievr

Apart from the “region of interest”-problem, an impeding factor to the successful application of the technology in humanities research is that CBIR systems should be implemented for specific collections of artifacts. They cannot be used off the shelf. Careful planning is necessary. CBIR systems process similarity data not “as you search”. To perform fast large image collections are processed (analyzed) in advance, and the results of the analysis are stored in a database.

An older example (since 1997) of a CBIR system accessing a digital museum collection is IBM’s QBIC (Query By Image Content) as it has been implemented for the Hermitage Museum in Leningrad. You can try it out here:

The State Hermitage Museum: Digital Collection — Powered by IBM

So, has computer imaging science managed to develop useful retrieval options for the formal attributes of (digitized) images, including works of art? I think the answer must be: not really ….as yet.

One thought on “Content-based image retrieval + arts & humanities

  1. Note: retrievr puts the picture “most similar” to your scribble in the upper left hand corner of the tabular display. Now try this:

    start making a random scribble
    inspect retrieval results
    try to arrange one of the results in the upper left hand corner, by slightly adapting your scribble

    Does that work? Why? (Or is it: why not?)

Leave a Reply