AI movie restoration – Scarlett O’Hara HD

With convolutional neural networks and state-of-the-art image recognition techniques it is possible to make old movie classics shine again. Neural networks polish the image, reduce the noise and apply colors to the aged images. 

The first movies were created in the late nineteenth century with celluloid photographic film used in conjunction with motion picture cameras.

Skip ahead to 2018, when the global movie industry was worth $41.7 billion globally. Serving entertainment, cultural and social purposes, films are a hugely important heritage to protect. And that’s not always easy. Especially considering the fact that modern movies are produced and screened digitally, with the technology of celluloid tape fading into obsolescence.

Challenges in film preservation

The challenge and importance of preserving the cultural heritage of old movies has been underscored by numerous organizations including the European Commision, which noted that a lack of proper devices to play aging technology on could make it impossible to watch old films.

In deepsense.ai’s experience with restoring film, the first challenge is to remove distortions. Classics are usually recorded in low resolution while the original tapes are obviously aged and filled with noise and cracks. Also, the transition process from celluloid tape to digital format usually damages the material and results in the loss of quality.

By using AI-driven solutions, specifically supervised learning techniques, deepsense.ai’s team removed the cracks and black spots from the digitized version of a film. The model we produced uses deep neural networks trained on a movie with cracks and flaws added manually for training purposes. Having some films in original and broken form, the system learned to remove the flaws. An example of generated noise put on the classic Polish movie “Rejs” and the neural network’s output is displayed below.

The example clearly shows that our neural network can process and restore even a thoroughly damaged source material and make it shine again. The networks start to produce low-quality predictions when the images are so darkened and blurred that the human eye can barely recognize people in the film.

How to convert really old movies into HD

A similar training technique was applied to deliver a neural network used to improve the quality of an old movie. The goal was to deliver missing details and “pump up” the resolution from antiquated to HD quality.

The key challenge lay in reproducing the details, which was nearly impossible. Due to technological development, it is difficult for people to watch lower quality video than what they are used to.

The model was trained by downscaling an HD movie and then conducting a supervised training to deliver the missing details.

Move your mouse cursor over the image to see the difference.

The model performs well thanks to the wide availability of training data. The team could downscale the resolution of any movie, provide the model with the original version and let the neural network learn how to forge and inject the missing detail into the film.

A key misconception about delivering HD versions of old movies is that the neural network will discover the missing details from the original. In fact, there is no way to reclaim lost details because there were none on the originally registered material. The neural network produces them on the go on with the same techniques Thispersondoesnotexist and similar Generative Adversarial Networks use.

So, the source material is enriched with details that only resemble reality, but are in fact not real ones. This can be a challenge (or a problem) if the material is to be used for forensic purposes or detailed research. But when it comes to delivering the movies for entertainment or cultural ends, the technique is more than enough.

Coloring old movies

Another challenge comes with producing color versions of movie classics, technically reviving them for newer audiences. The process was long handled by artists applying color to every frame. The first film colored this way was the British silent movie “The Miracle” (1912).

Because there are countless color movies to draw on, providing a rich training set, a deep neural network can vastly reduce the time required to revive black and white classics. Yet the process is not fully automatic. In fact, putting color on the black and white movie is a titanic undertaking. Consider Disney’s “Tron,” which was shot in black and white and then colored by 200 inkers and painters from Taiwan-based Cuckoo’s Nest Studio.

When choosing colors, a neural network tends to play it safe. An example of how this can be problematic would be when the network misinterprets water as a field of grass. It would do that because it is likely more common for fields than for lakes to appear as a backdrop in a film. 

By manually applying colored pixels to single frames, an artist can suggest what colors the AI model should choose.

There is no way to determine the real color of a scarf or a shirt an actor or actress was wearing when a film rendered in black and white was shot. After all these years, does it even matter? In any case, neural networks employ the LAB color standard, leveraging lightness (L) to predict the two remaining channels (A and B respectively).

Transcription and face recognition

Last but not least, transcribing dialogue makes analysis and research much easier – be it for linguistic or cultural studies purposes. With facial recognition software, the solution can attribute all of the lines delivered to the proper characters.

The speech-to-text function processes the sound and transcribes the dialogue while the other network checks which of the people in the video moves his or her lips. When combined with image recognition, the model can both synchronize the subtitles and provide the name of a character or actor speaking.

While the content being produced needs to be supervised, it still vastly reduces the time required for transcription. In the traditional way, the transcription only takes at least the time of a recording and then needs to be validated. The machine transcribes an hour-long movie in a few seconds.

Summary

Using machine learning-based techniques to restore movies takes less time and effort than other methods. It also makes efforts to preserve the cultural heritage more successful and ensures films remain relevant. Machine learning in business gets huge recognition but ML-based techniques remain a novel way to serve the needs of culture and art.. deepsense.ai’s work has proven that AI in the art can serve multiple purposes, including promotion and education. Maybe using it in art and culture will be one of 2020’s AI trends.

Reviving and digitalizing classics improves the access to and availability of cultural goods and ensures that those works remain available, so future generations will, thanks to AI, enjoy Academy-awarded movies of the past as much as, if not more than, we do now.