More than meets the eye
New camera technologies are pushing the boundaries of imaging far beyond what the eye can perceive. Matthew Edgar, Miles Padgett, Daniele Faccio and Jonathan Leach explain how three of these novel cameras work and outline some future applications
Scientists have spent centuries studying the mechanism behind imaging. For the most part, their efforts have been aimed at replicating the functionality of the human eye – a complex organ that has, remarkably, evolved to detect light and some of its properties, such as colour and in some cases polarization (see “Unveiling your secret superpower”). As a model for cameras and other imaging systems, the eye has served us well. Like our eyes, modern cameras use lenses to form images, and they employ pupil-like apertures to limit the amount of light that falls on the photosensitive material inside, which functions like the retina. Indeed, camera technology has such close parallels with human biology that the existence of flash photography is almost surprising, given that we humans have not evolved to emit blindingly bright flashes of light from our foreheads when it is dark.
Jokes aside, there are several things that our eyes are unable to do. They cannot, for example, see around corners or sense light at wavelengths outside a narrow range of the electromagnetic spectrum. There are limits on how sensitive they are to low intensities and fast pulses of light. Conversely, some things that human vision does well, such as perceiving the world in 3D, have proved cumbersome and expensive to replicate with cameras.
Computer algorithms and modified hardware can enhance the operation of cameras, overcome their limitations and revolutionize their applications
If we take a step back for a moment, we can see that camera technology has gone through three significant shifts. Early cameras belonged to the analogue age, which saw the development of photosensitive chemicals used on plates (and later film) and, eventually, a shift from manually to electro-mechanically driven hardware. Next came the digital age, with the invention of charge-coupled devices (CCDs), sensors based on CMOS integrated-circuit technology and memory chips for image capture and storage. Now we are entering the computational age, when computer algorithms and modified hardware can enhance the operation of cameras, overcome their limitations and even revolutionize their applications. In particular, the advent of high-speed micro-electro-mechanical-systems (MEMS) devices, ultrafast timing electronics, single-photon-sensitive detectors and readily available high-performance computing have helped to pave the way for a new generation of camera technology.
As physicists working in experimental optics, we have spent the past few years developing camera technologies that exemplify this new age of imaging. One of these devices, a camera built at the University of Glasgow, can take infrared images in 2D or 3D without using a pixellated sensor – a key advantage because at non-visible wavelengths, such sensors can be extremely expensive. The second device, an imaging system developed at Heriot-Watt University in Edinburgh in collaboration with the University of Edinburgh, can produce video images of a speeding light pulse. The third device, also developed at Glasgow, makes it possible to take 3D images using a single, stationary camera – an approach that dismisses the conventions of stereo-imaging and allows more primitive hardware to be used instead. These so-called computational cameras are currently in the early stages of development, but they may soon find widespread applications in areas such as imaging hazardous gas leaks, seeing around corners and even creating 3D cameras using a 3D printer.
Hidden from view
Scientists have known that there is more to light than meets the eye since at least 1800, when the astronomer William Herschel discovered invisible “calorific rays” while investigating the relationship between heat and colour with a prism and a thermometer. Herschel’s discovery – now known as “infrared” light because its frequency lies just below the red end of the visible spectrum – encouraged another scientist, Johann Wilhelm Ritter, to conduct his own experiments, culminating in the discovery of ultraviolet light a year later.
Since then, physicists have developed a full understanding of the entire electromagnetic spectrum, from radio waves to gamma rays. We have also invested considerable effort in developing new sensors that enable us to see a universe that is invisible to the naked eye. However, new technology often comes with a prohibitively high price tag that limits its use. This is never truer than for camera sensors. In the visible part of the spectrum, high consumer demand has brought the cost of silicon-based detectors down to few pounds per megapixel, but the cost of an equivalent device for non-visible light can easily exceed hundreds of thousands of pounds.
In response to this challenge, researchers in the optics group at Glasgow have developed a camera that can acquire pictures at non-visible wavelengths without the need for a pixellated sensor at all (see image above). Instead, the camera uses a single photosensitive detector and an array of tiny mirrors called a digital micromirror device (DMD). So how does it work? Like any camera, it has a lens, and when light passes through the lens, an image forms at the focal plane. In a conventional camera, the sensor would be placed at the focal plane, but in this camera, that location is occupied by the DMD. Each of the hundreds of thousands of micromirrors in the DMD can be independently adjusted to an angle of either +12° or –12° with respect to the normal, in order to shine light onto a non-pixelated detector (a photodiode). To appreciate how this set-up works, imagine turning just one micromirror at a time, such that the light incident on the micromirror is reflected onto the photodiode. The photodiode will then record the intensity of light at that particular point in the image, and we could, in theory, build up the entire image by carrying out a line-by-line sweep (raster scanning) until the intensity for each “pixel” in the image has been measured.
Clearly, though, doing it this way would be extremely cumbersome. The efficiency (meaning the number of measurements required, and also how much light is “thrown away” for each measurement) scales inversely with the number of desired pixels in the image; moreover, in order for this procedure to work, the photodiode would need to be sensitive to very small amounts of light against a (typically noisy) detector background. So instead, we apply a series of 2D binary masks to simultaneously reflect different parts of the image to one of the two DMD outputs, and measure the associated intensities with photodiodes. Since we know which different binary masks we applied to the DMD, and we know the corresponding filtered intensities, we can use a computer algorithm to deduce the image.
It is possible to reconstruct an image using entirely random binary masks, but for quickest results, a perfect n-pixel image can be deduced by using a series of n unique (orthogonal) masks. Better still, in most images, the intensities of neighbouring pixels are very similar, which means that the information in them can be highly compressed (the familiar JPEG compression algorithm makes use of this fact). An image can therefore be deduced using far fewer masks than pixels in the image – a technique first demonstrated by researchers at Rice University in the US (IEEE Signal Processing Magazine March 2008, pp83–91). Coupled with high-display-rate DMDs (those we use can be adjusted 20,000 times per second). We have shown that this system can produce not only still pictures but also real-time video at wavelengths where pixels are cheap but camera sensors are expensive.
The shortwave infrared (SWIR) region of the electromagnetic spectrum (1000–3000 nm) provides some good examples of how such a camera might be used in practice. Because many industrially important hydrocarbons have absorption lines in this region, one possible application would be to use the camera to detect gas leaks. If we combined the camera with an illumination source tuned to generate light at a particular wavelength – such as the 1670 nm absorption line of methane, a highly flammable component of natural gas – then the leaking gas would show up as an ominous dark shadow in the camera’s images. A low-cost SWIR camera might also have applications in night vision by making use of “night-glow”, a phenomenon whereby the atmosphere re-emits a small amount of SWIR radiation at night due to various chemical reactions in the atmosphere.
In the meantime, though, the single-pixel infrared camera has some pretty good party tricks. At the 2014 Royal Society Summer Science Exhibition, attendees watched as, with a flick of a switch (which changed the detection medium from silicon to an indium–gallium–arsenide photodiode) an image from the camera revealed an object hidden behind a screen that was opaque to visible light (figure 1). And even fellows of the Royal Society gasped when this single-pixel camera revealed a “secret hidden message” underneath a coloured painting hanging in the background (figure 2).
Imaging light in flight
Photons travel so fast that normal cameras cannot even come close to capturing their motion. A top-of-the-range digital SLR camera, for instance, might have an exposure time of 1/8000th of a second, but in this time, a photon travels 37.5 km. In the laboratory, however, scientists have developed imaging technology with much shorter exposure times, on the order of picoseconds (10–12 s) and femtoseconds (10–15 s). These very short times let us capture images of light moving on very small length scales and even record video footage of light in motion. This quite remarkable feat is known as light-in-flight imaging.
The technique of light-in-flight imaging dates back to 1978, when Nils Abramson from the Royal Institute of Technology in Sweden created an interference pattern with a laser and used that pattern to track the movement of a thin sheet of light. By using optical rather than electronic devices, Abramson was able to produce extremely short exposure times; however, his method suffered from technical limitations that made it impractical for applications. In 2012 interest in light-in-flight imaging revived after researchers in Ramesh Raskar’s Camera Culture group at the Massachusetts Institute of Technology showed that a “streak camera” – a specialized device traditionally used for spectroscopy with ultrafast lasers – has an exposure time short enough to track light as it moves. They shone a femtosecond pulse at a drinks bottle filled with a scattering medium and tracked the light as it entered the bottle and scattered around inside it. More recently, a team of researchers at Washington University in St Louis demonstrated the first ultrafast photography of a single laser pulse (2014 Nature 516 74).
At Heriot-Watt we have extended this research using a new type of camera that is so sensitive and so fast that we can capture individual photons and take videos of pulses of photons as they travel through air (see image above). Developed by a team of researchers led by Robert Henderson at Edinburgh, the camera is made up of an array of single-photon-sensitive pixels. Being sensitive to single photons means that each pixel is around 10 times more sensitive than a human eye, but these pixels also have another special property: speed. Each pixel can be activated for just 50 ps – more than a billion times faster than you or I can blink.
These amazing properties allow us to perform light-in-flight imaging of a pulse of light as it travels through air (2015 Nature Communications 6 6021). The camera works in combination with a pulsed laser source. When the pulse of light leaves the laser, a signal is sent to the camera to start individual clocks associated with each of the pixels. Photons in the pulse of light then travel through air, and after scattering off air molecules and changing direction, a few of them head towards the camera. When a photon arrives at a pixel on the camera, the clock for that pixel is stopped, and the time of arrival is recorded. After many identical pulses from the laser, we build up a video of light travelling through air.
This camera lends itself to applications where precise timing information is needed. One such application is recording the scattered light from objects hidden from view – something that could, perhaps, enable us to look around corners. Researchers have already shown that the form of objects that are hidden from view can be retrieved (2012 Nature Comms 3 745), but the Heriot-Watt camera may allow us to take things further and track the movement of hidden objects in real time. This extra step will be useful in many applications where direct visual information is not available; for example, it could be used to detect the location of people or objects in dangerous and life-threatening situations.
3D made simple
The fundamentals of 3D stereo vision are well understood: having two different perspectives on a scene means that objects in the scene will move by different amounts relative to one another, depending on their distance from the two viewpoints. Even so, 3D movies have proved difficult for the film industry to produce and display due to the costly hardware and the processes involved. Perhaps as a result, 3D movies have had various epochs of success on the big screen, only to make a swift and quiet exit through the side doors a few years later.
In 2013 we at Glasgow published the first 3D images obtained from a collection of single-pixel detectors (like those described earlier) that surrounded a light projector (2013 Science 340 844). Each detector could be used to produce a separate image of the scene, but interestingly, we found that each image was spatially identical. In other words, even though the detectors were in different positions, they did not provide different perspectives on the scene being imaged. Instead, the location of the detectors determined the apparent illumination in each image, while the image’s perspective was set only by the position of the light projector. This means that in effect, the detectors are behaving like sources of light, casting shadows differently depending on the geometric features of the scene, while the projector acts like a camera (figure 3).
This is a nice trick, because in a system that produces multiple pictures of a scene, with different lighting conditions and from a single, fixed, perspective, 3D information about the scene can be retrieved using a well-established technique known as photometric stereo (1980 Opt. Eng. 19 191139). Using this technique, and assuming that the surfaces in the scene scatter light isotropically, it is possible to deduce the orientation (or “surface normal”) of each pixel in the image from the differences in measured pixel intensities for each lighting direction. As long as at least three intensity values are measured for any pixel, the problem is fully constrained and can be solved through linear equations. The surface normals can then be used to provide a gradient map, and subsequent integration yields a relative depth map for each pixel in the image.
The upshot of this is that it is possible to turn a 2D camera into a 3D camera using a few cheap LEDs. This is much easier than using a standard 3D technique such as stereo imaging, in which multiple cameras obtain 2D pictures simultaneously and then perform the computationally intensive task of pixel correspondence to determine depth. In 2014 we decided to show off this technique by building a low-cost adaptor for a standard SLR camera (see image at top of article) and taking 3D photos during the Royal Society Summer Science Exhibition. Although we see the world around us in 3D every day, we never actually see ourselves in 3D, only our 2D projection on a mirror or a photograph. So there is something a little odd about seeing your own face rotating in 3D on the display – or maybe it was just that our 3D reconstruction software wasn’t quite perfect yet, and no, your nose is not that pointy!
Computing cameras of the future
Imaging has become a fundamental part of our everyday lives. It is estimated that the world’s population took more than a trillion photos last year, and while cute pets and selfies no doubt made up a considerable fraction of those pictures, imaging devices remain vital tools for advanced scientific research and applications. The cameras described here are still under development, but as the pace of computing continues to quicken and the performance of detector technology improves, we can expect much more diversity in the cameras of our future.