Art through the eyes of AI

Art through the eyes of AI: Advances that are helping us understand and reconstruct great works
In addition to the natural sciences, the field of art history has also undergone a revolution thanks to artificial intelligence. Machine learning and computer vision are shedding new light on the way we interpret our cultural heritage, a domain traditionally governed by conventional methods and analyses.

For decades, art scholars trained in classical methods were hesitant to embrace computational analysis, arguing that it was an overly simplistic approach to capturing the complexity and nuances of works of art. However, as David G. Stork’s recently published book “Pixels and Paintings” highlights, we are witnessing rapid advances in artificial intelligence algorithms, which are proving their worth by shedding new light on fine art paintings and drawings.

AI tools are unlocking how artists have used their understanding of the science of optics to convey light and perspective in their work, analyzing brushstrokes, colors, and styles, revealing aspects that often escape the human eye. They are also restoring the appearance of lost or hidden works of art and even computing the “meanings” of some paintings, identifying underlying symbols and themes.

The implementation of AI in art history faces its challenges, given the complexity of works of art, both in their composition and materials, and the fact that they carry many human meanings that often confuse algorithms. Most art historians continue to rely on their individual expertise, supported by research in laboratories, libraries, and in the field, to judge artists’ techniques.

In this scenario, collaborations between computer scientists and art experts are increasing, combining computational methods and traditional knowledge. The early successes of this “computer-aided understanding” approach fall into three categories: the automation of conventional ‘naked-eye’ analyses, the processing of subtleties in images that go beyond normal human perception, and the introduction of new approaches and classes of questions into art scholarship.

Artificial intelligence in art analysis

By analyzing vast data sets, artificial intelligence is providing a new understanding of works that transcend eras and styles, opening previously unexplored doors in the art world, revealing nuances and trends that until now had remained hidden even to the most trained eye.

Pose analysis, such a crucial element in portraiture, exemplifies the power of this new tool. Renaissance artists, as we know, portrayed important figures in profile, conveying an unmistakable solemnity and clarity. In contrast, primitivist artists such as Henri Rousseau and Henri Matisse preferred to paint ordinary people from the front, in a direct and unpretentious approach.

Artificial intelligence is enabling us to examine these styles on a monumental scale, with deep machine learning algorithms able to analyze tens of thousands of portraits in a matter of hours, a task that would take an art historian years to perform manually. More than simply identifying poses, these AI systems can infer the angles and orientation of the subjects in their portraits, opening a window into better understanding their intentions and contexts.

Recent research using deep neural networks has revealed fascinating insights, such as an analysis of more than 20,000 portraits that allowed us to group works by period and artistic movement, offering surprises such as how the tilt of faces and bodies in self-portraits varied, related to the artist’s posture, and even whether they were right- or left-handed.

These AI tools are also yielding new insights into landscape compositions, color schemes, brushstrokes, and perspectives across major artistic movements with markedly improved accuracy when combined with art historians’ knowledge of social norms, costumes, and artistic styles. Light in artworks

Traditional art analysis, carried out “by eye,” often reveals varying perceptions among different scholars, especially when it comes to aspects such as lighting in a work of art, which can mislead even the most attentive observers.

For example, the sharp contrast of light and shadow (chiaroscuro, from the Italian “light-dark”) of the 16th-century Italian painter Caravaggio contrasts sharply with the flat, graphic lighting present in the works of the 20th-century American artist Alex Katz. This is fertile ground for AI, which offers a more objective and accurate approach.
Experiments have shown that it is difficult for the human eye to accurately estimate the overall direction or inconsistencies of lighting across a scene, but computational techniques have overcome these human limitations.

Using the pattern of brightness along an outline, algorithms such as “shape from shading” and “occluded contour” can infer the direction of lighting, a technique that was already understood by Leonardo da Vinci in the 15th century.

A prime example of this application is the analysis of Johannes Vermeer’s 1665 painting “Meisje met de parel” (Girl with a Pearl Earring). Computational analysis of the highlights in the girl’s eyes, the reflection in the pearl, and the shadows on her face reveals a more complete understanding of the lighting in the scene, highlighting Vermeer’s extraordinary consistency and suggesting that the painting was executed from a live model.

These advanced methods are also being applied to works by artists such as Belgian surrealist René Magritte, where they can identify intentional inconsistencies in lighting. What’s more, these techniques have helped debunk bold theories about art, such as British artist David Hockney’s 2000 hypothesis that painters such as Jan van Eyck secretly used optical projections in their works. Computational analyses such as occluded contour analysis and optical ray tracing have disproved this theory more conclusively than traditional art historical methods.

Reconstructing masterpieces

Through advanced computational methods, we are witnessing the digital resurrection of cultural treasures that until recently were thought to be irretrievably lost.

A striking example of the revolutionary impact of AI in recovering lost artworks is the case of Vincent van Gogh’s “Two Fighters,” a pre-1886 work mentioned in his letters and rediscovered in 2012 through X-ray or infrared imaging beneath another painting. This is not an isolated case. AI is playing a crucial role in reconstructing works such as Gustav Klimt’s “Medicine,” lost in World War II, using analysis of preparatory sketches and photographs.

Perhaps one of the most impressive examples is the digital recovery of lost parts of Rembrandt’s “The Night Watch” (Dutch: “De Nachtwacht”), originally cropped to fit a specific space in Amsterdam’s City Hall. Using an oil on oak panel copy by Gerrit Lundens, AI identified how this copy deviated slightly from the original, and by “correcting” it was able to recreate the missing parts of the original.

These advances illustrate a fundamental truth: for AI to reach its full potential in art studies, access to vast data sets and computing power is essential, and museums around the world are contributing to this cause by making more and more images and information about art available online.

The future of AI in art history promises to be a fusion of technology and tradition, where every great work of art, along with countless lesser-known ones, will be available for analysis at high resolution and across expanded electromagnetic spectrums.

Just as generative AI tools like ChatGPT and Dall-E were trained on almost unimaginable data, future data sets used for art analysis will be even larger and more comprehensive.

The challenge of recovering lost works of art, destroyed by fire, flood, earthquake or war, is now seen as a problem of information recovery and integration, an approach that not only revives lost works of art but also enriches our global cultural heritage, which has been impoverished and depleted by these losses.

Art studies over the centuries have always been driven by the introduction of new tools, and now artificial intelligence is positioning itself as the next big step in this field. With the help of these emerging technologies, we are rediscovering and reconstructing pieces of our past in ways that were previously impossible, opening up new horizons for the appreciation and analysis of art.

Source