Bridging the camera hardware gap with video enhancement software

Powerful processors have become more than clock frequency

In the earlier days of computers, up to the late-2000s, newer processors were faster because they packed more transistors and ran at a higher clock frequency. That is a measurement of how many calculations, one at a time, can be performed every second, measured in megahertz (MHz, millions of operations per second) or gigahertz (GHz, billions). The higher frequency was an easy way to market new computers, and to compare them.
Eventually, physics got in the way, as higher frequencies demand higher power consumption and produce more heat. There are two ways forward. CPUs nowadays perform faster by doing more complex operations less often. GPUs work by doing a very large amount of very small operations but at the same time (in parallel). This is perfect for graphics or machine learning, but that’s a topic for another time. Both approaches ensure that Moore’s law still holds.

It’s not just about the megapixels

The same type of “speed” arms race emerged with the advent of digital cameras, and later on again in smartphone cameras. While “photo quality” is a wide and ambiguous term, saying a camera has more megapixels quickly became a marketer’s dream – as with megahertz, the megapixel count is a simple number to describe how good a camera is. Obviously, it’s a lot more complicated than that, but since digital cameras were so new, a higher resolution generally meant a newer, improved camera. It was good enough. Until, of course, it wasn’t, and physics caught up yet again.

There are quite a few areas important to good quality still photography and video recording that become significantly impeded on smaller physical scales required to fit in small devices like smartphones. This impediment must be compensated for by clever video enhancement software. Think of using video stabilization software to improve a video as analogous to a hardware-based tripod, or using auto zoom instead of a scripted and professional TV studio.

Software can do what no small fortune worth of hardware can

There’s no doubt that manufacturing processes keep improving and camera modules get increasingly powerful. But you cannot change the laws of physics, captain. Even if you’re prepared to pay a small fortune, some qualities simply don’t manifest themselves on such small scales. Fortunately, with a lot of processing power available, video enhancement software can minimize or even eliminate some defects. We’ve already talked about hardware augmentations such as Optical Image Stabilization (OIS) in a previous post so we will focus on video enhancement software here.

First, there really is no “objective” way of looking at the world, even for electronic devices. How scenes are captured depends largely on well-known and similar-sounding features like auto focus, auto white balance and auto exposure time to try to capture a good photo. They do not necessarily replicate what you see with your own eyes. It is common to make these algorithms saturate the colors a bit “extra”, to make them “pop”.

These features are also a requirement for video recording, where the time dimension plays a critical role. While subsequent photos may retune these settings (for better or worse) between shots, they only need to smoothly and gently change between subsequent frames in a video. Various algorithms for computing these settings exist and are subject to ongoing research.

Case in point: the bokeh effect

Something popularized in smartphones is creating a more-or-less fake bokeh effect in portrait shots. This means the camera distinguishes between the foreground and background and blurs the background, creating an emphasis on the subject (usually a person). Large camera bodies can narrow the focus and create this effect naturally, but the small cameras in devices like smartphones and drones cannot.

Instead, machine learning can be used to understand to a certain degree what the foreground subject is, what its contours are and blur the rest of the image. A better, and increasingly common approach is to use a dual camera. The smartphone uses the offset of both lenses to calculate a depth map of the scene, just as our brain does with a pair of eyes.

The bokeh effect: The subject in the foreground is in focus while the background is blurry, which helps to emphasize the subject. Original photo by carlosluis on Flickr.

A synthetic process like this has some flaws that don’t occur naturally. Individual hairs are sometimes mistakenly blurred together with the background. Glasses in the foreground will be considered to be in the foreground, but the background seen through them will not be blurred, whereas it would be in a proper bokeh effect. So it’s not perfect but generally good enough.

Improving image quality with artificial brightness

Some devices also have an additional purpose for the multiple cameras. For example, the second sensor can be a high-resolution black-and-white sensor to supplement the primary sensor with additional brightness information, improving image quality. Specializing beyond a general depth map, a trained AI and dual cameras can construct a 3D map of a face and re-light it with fake studio lighting, highlighting points of the face like the nose, cheeks and chin that would have been emphasized by external studio light. This gives the image a dimensionality you could normally only achieve using external lighting solutions or a lot of post processing.

Taking zoom to the next level

Zoom functionality is harder to emulate. Although some optical zoom may be available, especially with multiple cameras, most of the smartphone zoom is digital. Digital zoom means just cropping the already available image. As there is no more information to use, quality is reduced. All hope is not lost, as AI may now be able to essentially recognize what is missing and fill in the details on the cropped image, as an artist would improvise on a canvas, based on prior experience of other photos. Laughing at “zoom and enhance” on TV crime shows may become a thing of the past, after all.

Even with parts of the photo removed, the AI was successful at reproducing the original with decent quality in some examples. Check out more examples.

Speaking of zooming, any distortions caused by movement are amplified by the amount of zoom. Video enhancement software can help by applying video stabilization and creating a smoother zoom experience with live auto zoom. Good quality is not just clever engineering but about helping the user accomplish tasks, like an accurate and smooth zoom. This, and more, is the subject of the next section.

Helping the user

There is a lot of computing power available in devices like smartphones and drones. All that power can be used for helping out with things that apply to the art of photography rather than the shooting itself. A lot of these features are available today while others are sure to become more common. It’s not enough to have the ability to help – true help should be automatic and quick.

Editing color, exposure, applying filters and other post-shot improvements are very popular. With modern AI methods, this can be done automatically to highlight important parts. Understanding what parts of an image are important is the subject of research on “image saliency”. This can be used to crop images by removing uninteresting parts all over, not just the edges. This is called seam carving, which made the rounds on the internet some years ago. Check out a video that explains the seam carving process really well.

Image recognition techniques can be applied to auto-shoot on smile, or even to auto-select the best photo from a burst of photos. This is helpful when taking photos of a child or pet, or a group of people. This is in fact also where software beats hardware. When filming yourself (vlogging, video messaging, video conferencing, etc.), a physical gimbal stabilizer would stabilize the surrounding world, but your head would swerve around. Image analysis can stabilize the image around your face instead. Advanced video enhancement software should be able to support this with a selfie mode along with automatically generating a time lapse from a long video.

Unfortunately, it doesn’t stop with just getting the right moment and composition. The final photo or video may also be perceived differently on different screens. It’s no different from how a great sound recording will be different when listened to on cheap headphones.

Make the most of new possibilities with powerful video enhancement software

Rapid progress in processors and cameras built into devices like smartphones, wearable cameras and drones along with powerful new video enhancement software features are driving a rapid increase in the importance, use and quality of video. This is not only filling YouTube with higher quality smartphone videos by everyone and their brother but also opening up very interesting applications in areas such as law enforcement, surveillance and emergency services.

For instance, improvements in video stabilization, auto zoom and overall quality are driving breakthroughs in the ability to identify and track criminals and find people in need of rescuing from a drone. The many improved capabilities for customizing lighting will also come in handy in use cases such as filming houses by drone for realtors and improving visibility in bodycam feeds from dark buildings and alleys.

Are you interested in learning more about how to make the most of the new possibilities of video enhancement software? Don’t hesitate to get in touch with us and discuss your challenges with our video quality experts. For inspiration, insights and best practices for the next generation of video enhancement, enter your email address and subscribe to our newsletter.