The overall aim of the project is to study the place of illustrations in Franco-German art mediation between 1871 and 1960, using Computer Vision and Data Visualisation tools.

To do so, we rely on the curated database Deutsch-Französische Kunstvermittlung, and its user interface. We focus on the specific pages of the journals that were marked as interesting by previous researchers for the subject of Franco-German art mediation between 1871 and 1960, which span three different databases.

Using the IIIF representation of the data of interest, we automatically extract the illustrations using Computer Vision models for detection and segmentation. More specifically, we are training and using Detectron2, a model with Faster R-CNN FPN architecture developped by Facebook AI Research. 

Then, we enrich the illustrations metadata by several means. First, we predict the nature of the illustration (e.g. Is it a reproduction of a painting ? Is it a sculpture ? etc.) using the Cloud Vision API to get word descriptors of the illustration, and a Naïve Bayes Predictor on the labels to classify it into the appropriate category. Additionally, we automatically look for metadata for painting reproductions such as their title, artist or year of creation. Moreover, as it is interesting to look at the actual place of the image in the journals pages, we also classify the illustrations according to their size in the page.

Finally, the results are aggregated into a single easy-to-use visualisation, that allows anyone to research and look at illustrations ordered by their year of publication, filtering by keywords, and view the source document.

Github of the project

Autrice : Elisa Michelet