Art Algorithms

Art Algorithms Photo Credit: Architect of the Capitol.

Art conservation is traditionally a painstaking and time-consuming business. But the process of fixing up a damaged Da Vinci or a scarred Seurat may become a lot easier, thanks to the work of mathematicians. You'll find out why in this Science Update.


Restoring masterpieces with math. I'm Bob Hirshon and this is Science Update.

In the future, the work of Da Vinci and Botticelli may benefit from the work of Guillermo Sapiro. He's an engineering professor at the University of Minnesota.

Sapiro and his colleagues have developed mathematical algorithms to fill in holes, scratches, and creases in damaged masterpieces. He says they developed the math by watching conservators at the Minneapolis Institute of Arts.


Let's say that part of a bridge is missing—they will first try to fill in the lines of the bridge and then start to put in the color. And our algorithms work in the same way.

The big difference is time. Using computer modeling, these equations can map out a restoration in less than a minute that a human expert might take weeks to plan. Professionals can then use these models to guide their work.


So basically, they will get a digital version of their painting and they will say, "Oh, I should go and start here with blue," or things like that. So they can learn how our algorithm does it, to give them hints to restore the real art.

The work is funded by (believe it or not) the Office of Naval Research. Project leader Wen Masters says they're interested because the equations could also sharpen up surveillance photos and other electronic images. For the American Association for the Advancement of Science, I'm Bob Hirshon.

Making Sense of the Research

If you've ever used a computer program like Adobe Photoshop, you know that there are all kinds of ways to correct or alter images, especially digital pictures. Many film producers also rely on computers to enhance, alter, add, or delete elements from their pictures. For example, if you want to show someone gliding on the air, you might film her riding a skateboard and then digitally erase the skateboard from each frame of the film. But even with the help of a computer, figuring out exactly what to erase and what to paint back in can take a great deal of time and effort.

For art and photo conservators, there's a greater challenge. First of all, they have to correct real, tangible media, using paint instead of pixels. Secondly, they can't just fix up a damaged work to make it look nice. They have to restore the piece to something very close to its original condition—without having an original to compare it to.

Normally, this kind of educated guesswork can take days, weeks, or even months to map out. That's because a conservator needs to figure out not only what the finished product should look like but how to get there: for example, which color paint to layer in first. In many cases, there's no room for mistakes.

The mathematical algorithms that Sapiro's team developed speed up this process by propagating the intact image into the missing or damaged areas. Suppose, for example, that you had a Rembrandt portrait with a piece of the subject's nose missing. You'd scan a picture of the damaged portrait into a computer. Next, the computer would use the algorithms to analyze the image, especially the area surrounding the missing bit of nose. It would then use that information to figure out what probably belongs in the empty space, and map out exactly how to fill it in. (The smaller the empty space is compared to the whole picture, the more successful the program will be.) The human conservator would observe this computer model at work, make any necessary adjustments, and then go to work on the real thing.

Of course, the technique also works on digital images, like those mentioned in the first paragraph. The advantage here over traditional computer programs is that it works automatically. You don't need to painstakingly point and click your way through each alteration.

Another potential application of this technique is in image transmission, which is one of the Navy's areas of interest. If you've ever received pictures by email, you know that the bigger and more detailed they are, the longer they take to transmit. To send them faster, you can compress the images into smaller files that take up less data—but then you lose sharpness and detail.

If you had these algorithms pre-installed on your computer, however, you could use them to re-sharpen the compressed images after you receive them. In the future, algorithms like these could become standard features on computers and wireless devices, making it possible to receive smaller files and turn them into detailed images. In fact, the sender could intentionally withhold large parts of an image, allowing the algorithms on the receiving end to paint them back in. The algorithms can also be used to recover chunks of data that were lost because of noise and other errors in transmission.

Sapiro's colleagues on the project include Andrea Bertozzi of Duke University and Vincent Caselles and Marcelo Bertalmio of the University of Pompeu-Fabra, Barcelona, Spain.

Now try and answer these questions:

  1. What makes art conservation so painstaking?
  2. How do these algorithms help speed up the process?
  3. Name something these algorithms can't do—in other words, that the human conservator still has to contribute.
  4. Which of the following would this program be most able to help restore? Why?
    • A portrait with the face missing
    • A photograph of a mountain with a crease through the center
    • An abstract painting with a large corner ripped off
  5. Besides art conservation, name some other potential applications for this technology. For each application, are there any important drawbacks or limitations that would have to be taken into account? Explain.

For Educators

Image and Video Inpainting is the researchers' extensive website with examples of their work, papers presented at conferences, animated simulations, etc.

The Minneapolis Institute of Arts' site Restoring a Masterwork details the restoration of a 16th-century painting.

UCLA’s Image Processing Research Group does similar work.

Movie Mistakes is a site that keeps track of errors made in films, highlighting the need for digital image correction.

Related Resources

Self-Healing Robots
6-12 | Audio
Viking Longhouse
6-12 | Audio
Bouncing Babies
6-12 | Audio

Did you find this resource helpful?

Science Update Details

Grades Themes Project 2061 Benchmarks State Standards