FACES (Faces, Art, and Computerized Evaluation Systems) is a project whose primary goal is to establish the initial parameters of the application of face recognition technology to works of portrait art.
In the application of this technology to actual (that is, photorealistic) human faces, a number of difficulties are inherent in a real or perceived alteration of appearance of the face through variations in facial expression, age, facial hair, angle of pose, lighting, and the very identity of the subject. With portraiture in sculpture, painting, and drawing, not only do all the problems that apply to photorealistic subjects pertain but these works of art also have their own additional challenges. Most notably, portrait art does not provide what might be called a photographic likeness but rather one that goes through a process of visual interpretation on the part of the artist.
To help address this challenge, in the first phase of FACES, initial subjects for study have been selected with as much control over variables as possible. At the same time, we employed a body of clear, purposely chosen test portraits from a cultural period that lends itself to these goals: Western Europe, fifteenth to early eighteenth century. In this, the conceptual logic of the images followed an ordered sequence of paradigms (that is, logically chosen bodies of related images directed toward a particular demonstrative end) whose purpose is to systematically establish the parameters just mentioned.
For example, we began by testing the death mask of a known individual against an identified sculptural portrait of the same individual: Lorenzo de'Medici.

Lorenzo de'Medici; death mask casting; by Orsino de'Benintendi; 1492 (Societá Colombaria, Florence).
Lorenzo de'Medici; probably after a model by Andrea del Verrocchio and Orsino de'Benintendi; probably 1478-1521 (National Gallery of Art, Washington).

That is, we tested an analogue--an unmediated image of the subject, not a work of art--against the image of a three-dimensional work of art that, in this case, physically approaches the subject in form and size but that nevertheless partakes of the subjectivity of artistic interpretation. In this, we obtained a match of 83.26% suggesting a probable match on the basis of the body of identified portraits used in this study for training the system.
Incrementally, we broadened our tests, introducing a similarly controlled but wide-ranging series of systematically chosen variations extending from more controlled paradigms to less controlled ones. Having already tested the same stage of an individual's life by the same artist, these new paradigms included the same stage of an individual's life by different artists, different stages of an individual's life by the same artist, different stages of an individual's life by different artists, and so on, all in three-dimensional imagery. Our paradigms then tested two-dimensional imagery (painting), first by simply comparing two two-dimensional images of the same subject by the same artist (the pair below have a match of 74.78%),

Gianlorenzo Bernini, self-portrait (age around 25); c. 1623 (Galleria Borghese, Rome).
Gianlorenzo Bernini, self-portrait (age around 27); c. 1625 (Ashmolean Museum, Oxford).

and then by mixing media by testing a number of sculpture vs. painting (three-dimensional vs. two-dimensional) paradigms employing a systematic series of distinctions similar to those already mentioned (the paradigm below has a match of 73.06%).

Urban VIII; by Gianlorenzo Bernini; c. 1632 (National Gallery of Canada, Ottawa).
Urban VIII; painting; by Gianlorenzo by Bernini; c. 1631-1632 (Palazzo Barberini, Rome).

Throughout all of this, we tested multiple paradigms by many different artists and works of art. The media of drawing and prints were also integrated into this testing.
Aside from these general parameters, we also tested for a number of related issues. Foremost among these is the problem of artistic style in the creation of a portrait: that is, the question of the degree of influence both of the style of the individual artist and of the period style.
For example, a given artist generally tends to render the same detail in the same way, even in an individualized portrait: the corner of an eye, for example, or the corner of a mouth. And so individual artistic style has been investigated through a close and systematic study of a large number of portraits of different sitters by the same artist to learn the individual style of the artist. We did this for a number of different artists, one example of which is Michiel Van Mierevelt, testing a relatively large number of his works.

Dudley Carleton, Viscount Dorchester; by Michiel Van Mierevelt; 1620 (National Portrait Gallery, London; NPG 3684).
Hugo de Groot; by Michiel Van Mierevelt; 1631 (Rijksmuseum, Amsterdam; SK-A-581).

The situation with period style is similar: some periods have certain expectations in regard to the rendering of the human face--a Roman nose, grave eyes, a strong jaw, an aristocratic expression, and so on--rather than a more strictly straightforward rendering of the individual features of the sitter. Accordingly, period style has been addressed through the study of an equally large number of portraits but now of the same sitter by different artists in order to model the period style--again, to teach the computer what aspects a portrait may owe to period expectations. One such paradigm from among the many we tested is concerned with a large number of portraits of Robert Dudley, 1st Earl of Leicester and the favorite of Queen Elizabeth I of England, all by different artists.

Robert Dudley, Earl of Leicester; attributed to Steven Van der Meulen; c. 1560-1565 (The Wallace Collection, London; P534).
Robert Dudley, Earl of Leicester; by Nicholas Hilliard; 1576 (National Portrait Gallery, London; NPG 4197).

The end goal of FACES is the restoration of lost identities to works of portrait art. At this stage, this typically involves images whose identities have been hypothesized but are not known with any certainty (the two images below have a match of 67.34%).

Mary Queen of Scots (?); c. 1570 (National Portrait Gallery, London; NPG 96).
Mary Queen of Scots; by François Clouet or Jacques Decourt; c. 1559 (BnF, Estampes, Paris; Rés. Na 22).

In this sometimes extremely tangled endeavor, we are insistent that this technology does not prove the identity of its subjects (some identities may never be known). Its results, however, match what is known with a given unknown, bringing a new scientific objectivity to a traditionally highly subjective area of art history while at the same time retaining the human eye as final arbiter.
Although FACES is strictly limited to the parameters described above, this technology certainly has other possible applications. These might include its use in adapted forms in which, rather than detect the similarities in facial constructs, it is applied to bodies of other variables in works of art that are unique to individual artists, such as carving techniques in sculpture and brushstrokes in painting, potentially opening up a whole new world in the identification of free standing sculpture, relief sculpture, small scale sculpture (such as ivories and medals), painting, stained glass, illuminated manuscripts, and so on. Also, once developed, such technology almost certainly could be applied to widely recognized but visually minor variations in architectural details (such as molding profiles), potentially revealing a wealth of information about building processes, building history, and architectural identities.
Finally, we hope that this method will eventually be widely taken up by museums and art conservation laboratories as a standard part of curatorial and preservation practice.
A very brief overview of the face recognition methodology: Portraits are subject to several complexities such as aesthetic sensibilities of the artist or social standing of the sitter. Moreover, the number of samples available to model these effects is often limited. For robust automated face recognition, it also becomes important to model the characteristics of the artist. From a set of portraiture where the identities of subjects is known, we derive appropriate features that are based on domain knowledge of artistic renderings and learn statistical models for the distributions of the match and non-match scores, which we refer to as the portrait feature space (PFS). The features considered include well-known facial recognition attributes like local features and anthropometric distances. Thereafter, we learn which of the chosen features were emphasized in various works involving (a) same artist depicting same sitter, (b) same sitter but by different artists, (c) same artist but depicting different sitters, and we show that the knowledge of these specific choices can provide valuable information regarding the sitter and/or artist. Further, we use the learned PFS on a number of cases that have been “open questions” to art historians. They are usually in the form of validating two portraits as belonging to the same person. Using statistical hypothesis tests on the PFS, we provide quantitative measures of similarity for each of these questions. It is, to the best of our knowledge, the first study that applies automated face recognition technologies to the analysis of portraits of multiple subjects in various forms - paintings, death masks, sculptures.

Note: Numerical results are missing or preliminary as the experiments are still ongoing. They will be updated as new results become available.

Sample Publications