November 14, 2024
AI in driverless cars
It's likely that after a couple of decades, humans will be banned from driving cars altogether: because AI in smart cars will handle it better.
Why camera calibration is so important in computer vision
The cameras are used to capture the world around us. But frames are just representations of the world, and the actual relationship between the flat image and the real thing is not always obvious.
In order to extract meaningful insights about the real objects from the images, developers and users of computer vision systems need to understand camera characteristics — with the help of the calibration solutions
A camera is a device that uses a lens to focus light rays onto a sensor. If we create a super-simple mathematical model, the camera will look something like this. If such a simple camera is pointed, for example, at a tree, we will get an image of it. Every high school student knows that.
The simplest camera (camera obscura) doesn't even have a lens
— a hole in a wall or box can be used instead
Real cameras used in photography, videography, and computer vision are much more complex devices. For example, they use wide-angle lenses: which allow you to capture more of your surroundings.
There are even lenses that can capture almost everything around you in 360°.
Cameras, by design, also introduce a level of distortion. A prime example is the curvature of straight lines, which is usually more prominent near the edges of the frame. This effect is noticeable in any photo taken with a wide-angle lens.
In addition, it's like in a side-view mirror: "Objects in the mirror are closer than they appear" — the proportions of objects closer to the edges of the frame are distorted.
In such a photo, there is not a single straight line, and the child standing
in the center seems much closer than the children behind.
By the power of our abstract thinking, we easily "correct" such camera distortions. However, AI and computer vision are less advanced in that sense. Artificial intelligence perceives the picture as it is and cannot correct its flaws without an additional effort.
For example, if you put fisheye attachments on the cameras of self-driving cars and do not retrain the model, the car may get into an accident — the AI will not be able to accurately assess the distances and sizes of objects. To help AI, it's sometimes important to calibrate the camera.
People have been trying to correct lens distortion since long before computers and even digital cameras were invented. To do this, they manipulated the optical axis by changing the position and angles of the lens and photosensitive material.
The photographer raised or lowered the lens, and tilted it — watching
the reflection of the subject on the semi-transparent glass behind the camera.
This way it was possible to obtain photographs of objects of the correct shape — lines turned out to be straight and parallel, a circle became a circle, not an oval.
A guide to correcting distortion from camera manufacturer Linhof
But in the world of computer vision, things are much more complicated.
• Often we have to support various cameras from different manufacturers and with different specifications. But the algorithm should work for all of them. Or you may also want to reuse algorithms trained with data captured from a different camera.
• These cameras may have individual peculiarities, optical defects, and mounting inaccuracies.
• Camera parameters may change during operation (for example, due to temperature or vibration).
And above all, AI models in computer vision perceive the world through 2D representations. A computer vision system operates on data that is acquired from cameras — so if this data contains certain distortions during training, then the AI model expects the same during inference.
Good news! We can do it programmatically, and quite accurately and quickly!
To calculate parameters required for correcting optical distortion, we need a calibration template — an object whose size and shape we know exactly. Usually, the template is a rectangular image with known sizes and a contrasting pattern.
By moving such an object in the camera's field of view, we can estimate the parameters of the camera so later we can correct the perception of the image using a relatively simple mathematical algorithm.
Ideally, it is important for us to place the template in all areas of the frame. Often, camera calibration app is used for this purpose — to clearly see in which areas the template has not yet appeared. The algorithms assume the template to be perfectly flat, so after printing it on an ordinary A4 sheet, please attach it to a thick cardboard piece to keep the template from bending.
What we know about the pattern (for example this checkerboard one) is not only that all the lines in it are parallel or perpendicular. We also know the dimensions of each square, which allows us to calculate how the camera’s pixels correspond to known real-life objects.
The camera is mounted high. For example, we need to calibrate a camera that is mounted on a pole and sees a large parking lot. In this case, we can, for example, print a very large template, though one will have a hard time moving it may take a long time in the camera's field of view.
But there is another way. We can use surrounding objects for calibration! For example, we can assume that roads and markings on them are straight lines. And road signs or cars can be used as size samples.
The camera is not available at all. How do you calibrate a camera that is 56 million kilometers away from us? You have to give the camera its own pattern!
For example, this is how they calibrate the camera on the robotic arm of the Mars rover. The pattern is fixed right on the rover! Note that in addition to lens distortion, the color scheme is also calibrated. And the coin is used to calibrate the size of objects.
We don't know where the camera will be. For example, we ask players to use the cameras of their mobile devices to explore the size and proportions of a room — to work with the augmented reality helmet. We don't know anything about the rooms, and we can't ask users to use a pattern.
In this case, we can calibrate the camera using people!. For example, we can safely assume that the height of one person is a constant, and use that as a moving pattern. Or we can find objects in the video whose sizes and proportions we know —for example, paper sheets, money, phones, and other devices.
It is important to note that in all difficult cases, we sacrifice accuracy in favor of usability or problem solvability. Of course, if we calibrate the camera to a road marking or a dancing human figure, we will not be able to correct distortions perfectly. But such calibration may be sufficient for the tasks we need to accomplish.
We at OpenCV.ai are professionally engaged in computer vision and artificial intelligence technologies. We will be glad to discuss the task with you — and we will take into account the possibilities and needs of OpenCV.ai camera calibration!
This article has a sequel — read about how (and why) to calibrate multiple cameras at once.
.png)