In the last few years (2018-2021) I have begun to notice more and more colourised back and white photographs and videos. From the full works by big names like director Peter Jackson’s They Shall Not Grow Old where grainy First World War footage was upscaled to 4K resolution, colorised, ‘restored’ (using Computer Generated Imagery, or CGI) and edited into a narrative, through to enthusiasts making some of the earliest known photos and film look smooth, squeaky clean, and less old-looking. Some of these efforts have garnered millions of views on YouTube, showing that there is a great public appetite for this material.
In 2020, major genealogy website MyHeritage released a colourising tool for photographs called MyHeritage In Color which allows subscribers to upload their own photographs and have them magically tinted into believable colours. This tool is based upon an open source computer vision project called DeOldify, a machine learning (ML) application written by specialists Jason Antic and Dana Kelley. DeOldify is trained on a huge library of images using ‘deep learning’ techniques, and is excellent at identifying even small subjects in a photograph. If you have a technical mind, you can read all about the DeOldify project and how it works.
In October 2020, Adobe announced a new feature called “neural filters” in its industry-standard Photoshop software, allowing colourisation of photos in just a few clicks.
Colourisation of black and white photos is now mainstream. But how excited should we be?
Appeal
Why are colourised black and white photographs and videos so popular? I have many opinions on this, and I thought that I would use DeOldify to colourise some of my own family photos, and some of Tehmina’s, and see how we – and our families – thought and felt about seeing ‘colour’ versions of our ancestors that we have only ever seen in black and white.
The oldest photo that I have of my direct ancestors is a “tintype” from the late 1860s. It’s very fragile and not particularly clear, and the emulsion layer is coming away from the metal backing. In short, as you can see below, it looks pretty old.
What would this look like if colour photography had been used (had it been invented by then)? Using the public experimental DeOldify installation I came up with this colourised version:
The first thing that I noticed about this image wasn’t the rather vibrant skin tones (which I could tone down a little), but rather the background. I just hadn’t noticed that my several-greats grandfather and his sister are sitting in front of a painted backdrop with trees and what could well be a lake. The machine learning algorithm interpreted this, colouring in the ‘water’ in a vibrant blue. I hadn’t noticed it before. I experienced an emotional connection to my distant ancestors, looking out of the photo at me 160 or so years later. The colour made them seem more ‘real’. It’s hard to explain, but when you have a connection with a colourised subject, something just happens. There is certainly a strengthening of any connection that already exists.
I tried the same technique on some of Tehmina’s family photos and shared them with her family. The word ‘wow’ was used several times, and it was genuinely felt that they “bring them alive” and strengthen ancestral connections.
Accuracy and authenticity
When we can put aside those personal connections and the ‘wow’ factor, and think and look critically at the results, there is a lot that we should be cautious about. Even as the technology improves.
Firstly, the colour is not, and can never be, authentic and accurate. The machine learning (ML) algorithms can only identify objects and apply colour to them based upon how they have been ‘trained’. Are leaves and grass always green? Is the sky and sea always blue? Was that dress really orange? The results are nearly always believable, but not necessarily accurate.
DeOldify uses several approaches to add colour to black and white photographs. The main method uses a ‘generator’ to apply colours to objects that it recognises. A ‘discriminator’ statistically criticises those colour choices. The recommendations and criticisms are applied iteratively, using a technique called “Generative Adversarial Networks” (GAN) to provide a statistically plausible colour outcome.
There are other methods of colourising photos. I’m not sure which approach is used by Adobe Photoshop, but it is likely that they will work similarly, or at least undergo ‘training’ using banks of well-described images such as ImageNet. Here the neural networks are trained to recognise what an object looks like, and what range of colours they might be.
ImageNet, which is used in training DeOldify, is a scientific dataset. It consists of clear diagnostic photographs of thousands of objects, each carefully categorised and described.
Through scientific datasets like this, colourisation processes ‘learn’ what objects looks like, and what colour they should be. Colourisation methods will only be as good as the images that they were trained with.
Illusions of accuracy
But what about accuracy – which colours should be used and where? Training datasets are also likely to be culturally, racially or geographically biased. What colour was that Indian sari? What were the colours in the pattern along the hem? That Estonian national costume – what colour were the details in that design? Was that person’s skin colour really like that?
At the moment, as it stands, colourisation is a gimmick. A few clicks and an old photo gains a new look. The grass is green, the sky is blue, the clothes are colourful. Grey faces become flushed with colour. The time the photograph was taken seems less remote. But this is all an illusion.
Widespread editorial use
I am seeing colourised photos used for editorial purposes on technology websites. It has become a regular occurrence for bloggers to use colourised photos of old scenes to illustrate their posts. We must be careful that colourised photographs do not start to be seen as authentic. DeOldify encourages the use of a watermark to show that the image has been colourised, but this is not enforced.
Upscaled, smoothed and colourised videos garner millions of views on YouTube, where the original footage may have remained obscure. Technology blogs effusively cover extreme examples, fuelling interest in these false-colour representations of the past.
For all of its faults, we mustn’t underestimate the power of colourisation in encouraging new interest in heritage. However, we should encourage mainstream media to always state where AI-based colourisation has been used.
The future of computer vision
It is an interesting time in the world of computer vision. Machine Learning (ML), Artificial Intelligence (AI), and Neural Networks (NNs) are all now widespread terms that you will see in the technology press. These are fast-paced areas of computer science which are achieving wonderful things. In the cultural heritage sectors we must interact with computer scientists to help with nuance, bias, and accuracy. Can we use our knowledge, and our access to large sets of well-described historical imagery to help make this inevitably popular colourisation technology more accurate and more useful? I like to think that we can.
There is also huge potential for computer vision techniques to not just colour black and white images, but aid metadata creation. For example, the Photos app on my Mac can already recognise basic scenes based upon their content. A search for ‘beach’ does indeed, mainly, produce photos I’ve taken on or of beaches. Google Photos does the same. Flickr is also adding ‘machine tags’ that use image recognition to help people find more relevant photos.
Soon enough, with our sector’s involvement, perhaps we could be adding a new set of fields to our collections databases to receive data from Machine Learning algorithms analysing our digitised photographic collections. Archaeologists are already using AI to help identify pottery and other artefacts using large collections of detailed finds photographs. Watch this space.
Have a go at colourising your own photos and make up your own mind.
