November 14, 2024
AI in driverless cars
It's likely that after a couple of decades, humans will be banned from driving cars altogether: because AI in smart cars will handle it better.
Imagine being a business owner considering integrating artificial intelligence (AI) into your company's operations. You've understood the potential advantages and goals you want to achieve with AI, and you are ready to invest. However, you may need help with pricing parts of the project, both the initial development and continued support or licensing costs, to truly measure the return on your investment.
Like any automation project, building an AI-aimed solution demands an expert understanding of your business and guidance on engineering approaches. Turning your AI vision into production-ready reality requires a staged approach to experiments, productization and scaling. The team of OpenCV.ai experts regularly gets requests to guide and size such projects. This prompted us to write this article, with a distillation of our advice across a variety of industries and verticals.
In this series of articles, we will guide you through all the essentials, from hardware and software selection to the legal aspects of AI. Let’s start with Part 1 | Hardware.
A fundamental factor to consider while developing a CV solution is the hardware setup. In the following sections, we will delve deeply into the aspects of hardware selection, requirement specifications, limitations, and more. You will learn about:
• Choosing a camera that matches your CV solution
• Selecting the best computing platform based on limitations and requirements
• Understanding the impact of network connection and power supply for the CV solution development
In the realm of computer vision (CV), camera selection majorly impacts the effectiveness and efficiency of your product. Whether you're developing a CV solution for surveillance, object recognition, or any other application, understanding the nuances of camera selection is paramount. This section provides an in-depth exploration of factors to consider when choosing a camera type based on your task requirements, the necessity of high frames-per-second (FPS), and the relevance of depth estimation.
The resolution of your camera plays a critical role in determining the level of detail your AI model can operate with. When selecting a camera, it's important to consider the size of the objects you want to detect and monitor relative to the area you want to cover.
Imagine you have a security camera mounted on a building overlooking a parking lot. If your goal is simply to count the number of cars entering and exiting the parking lot, you may not need a very high camera resolution. You can identify cars as large objects without requiring fine details.
However, if you also want to perform license plate recognition from the same camera, the resolution becomes critical. License plates are relatively small and require more detail to read the characters accurately. In this case, you would need a much higher-resolution camera to capture clear images of the license plates, ensuring successful recognition and enhancing security.
The FPS capability of a camera determines its ability to capture fast-moving objects without introducing motion blur. When selecting a camera, consider the nature of your application and the speed of movement involved.
For instance, to track the movement of a puck in a hockey game using CV, a high FPS is necessary to capture the rapid movement of the puck without motion blur. For this, a camera with a much higher FPS is required than a regular surveillance camera that works well at 10-15 FPS for slower activities. With a higher FPS, the puck's trajectory can be precisely recorded, enabling the computer vision system to track its movement across the ice with accuracy.
In addition, the choice between a global shutter and a rolling shutter impacts the camera's ability to capture fast-moving objects effectively. When selecting a camera for scenarios involving fast movements, it's essential to choose one with a global shutter. Global shutter cameras capture the entire frame at once, eliminating the potential distortion and motion artifacts associated with rolling shutter cameras and ensuring accurate and clear image capture in high-speed situations.
Depth estimation is foundational to many CV solutions, enabling the models to perceive distances between objects. This can be critical for a wide array of applications such as 3D reconstruction and collision avoidance. When considering depth estimation, it's essential to recognize that various technologies and approaches are available.
One common technology for depth estimation is the use of stereo pairs, as seen in devices like the typical Intel RealSense camera. These systems rely on two cameras with slightly different perspectives to calculate depth information by triangulating the disparities between the images.
Another technology employed for depth sensing is LiDAR (Light Detection and Ranging). It comes in two primary forms: non-moving LiDAR, such as the RealSense L515, and moving LiDAR, frequently found in automotive applications (e.g., Velodyne LiDAR).
Yet another approach involves Infrared (IR) grid projectors, which project a grid pattern onto the scene and calculate depth based on how the grid deforms over objects.
Let's consider a drone navigation application. For example, by equipping drones with the Intel® RealSense™ camera, they can navigate through complex environments with a higher degree of autonomy. The camera's depth-sensing capability aids in detecting obstacles and terrain variations, enabling drones to avoid collisions and ensure safe flight paths.
However, acquiring and integrating such a camera incurs costs and adds complexity to the development process. Therefore, there are drone systems that do not rely on depth cameras, resulting in less autonomous navigation.
The applications of depth estimation are far-reaching and extend beyond just spatial understanding. For instance, consider Apple's Face ID, which utilizes depth-sensing technology to create accurate facial recognition systems. Furthermore, depth cameras enable the creation of 3D avatars, facilitate flat scanning, and open up possibilities in areas like augmented reality (AR) and virtual reality (VR).
Modern deep-learning models have made remarkable strides in predicting depths from RGB or improving sparse or low-resolution depth. Neural networks can estimate depth from a single image, a technique often referred to as monocular depth estimation. While this method may not yield depth in millimeters, it can be remarkably useful for applications like scene understanding, where a detailed depth map isn't strictly necessary. Similarly, deep learning models can reconstruct 3D scenes from a series of images with known camera positions, further expanding the possibilities for depth-related tasks.
If your project involves multiple cameras capturing the same scene, you may consider synchronizing their outputs. This will be useful for tasks that benefit from accurate spatial alignment or 3D reconstruction.
In a 3D scanning application, using camera synchronization ensures that the captured images can be accurately combined to create a detailed 3D model of the object or scene.
For computer-assisted surgery, precise 3D reconstruction is crucial, particularly for minimally invasive procedures. Surgeons need real-time 3D models of the patient's anatomy to guide their actions, especially for laparoscopic surgeries operating through small incisions while viewing a monitor.
To create an accurate 3D model, multiple cameras capture the surgical field from different angles. Synchronizing camera outputs ensures that images from different angles correspond to the exact same moment in time. This is an optimal solution, resulting in an accurate 3D model that the surgeon can rely on during the surgery.
Choosing the appropriate computing platform is a substantial decision that impacts the performance, scalability, and accessibility of your AI solution. Whether you're considering edge devices, mobile platforms, or cloud solutions, it's essential to understand the nuances of each option. This section dives into the factors to consider when selecting a computing platform to deploy and utilize deep learning models efficiently.
Let's discuss the advantages of using edge devices for on-the-spot inference:
• Low Latency: Edge devices can perform real-time inference without delay since data doesn't have to be sent to a remote server. This is vital for applications like robotics or monitoring that need immediate responses.
• Data Privacy: Edge devices provide the added benefit of on-premises storage of sensitive data to address privacy concerns. This ensures authorized access to the data.
• Offline Capability: Edge devices can operate independently, making them ideal for use in remote locations or areas with limited internet access where a consistent internet connection may not be available.
However, some factors need to be considered while using edge devices:
• Area covered by cameras: For example, if we have multiple cameras closely positioned on a store shelf, it's feasible to install only one edge device due to their proximity, which minimizes latency. This setup can provide excellent real-time response but might limit overall throughput. In contrast, when cameras are dispersed across a larger area, such as at intersections in a city, each camera may require its own dedicated edge device. While this approach increases hardware costs, it allows for higher throughput as each camera can be processed independently, trading off lower latency for improved overall system throughput.
• Computational Power: Edge devices have limited processing capabilities compared to larger systems. It is important to optimize complex deep learning models before deploying them on edge devices for efficient performance.
• Model Size: Deep learning models with many parameters may not work well on edge devices with limited memory. It is important to optimize models for available memory in these devices.
For example, a security camera with an edge device can quickly analyze on-site video footage and alert security staff of any suspicious activity without relying on remote servers.
You might consider using the NVIDIA Jetson Orin. It is a powerful edge computing platform with a GPU that accelerates complex deep learning models at the edge, reducing latency and enabling real-time decision-making. It's perfect for industrial automation, surveillance, and robotics. The Jetson Orin supports CUDA and TensorRT, which make deploying and optimizing deep learning models easier.
Mobile devices offer convenient access to AI solutions. They have sensors such as cameras and accelerometers that provide data for deep learning models. With these sensors, AI systems can collect information about the user's surroundings, location, and movements, providing a wealth of data for analysis.
A few things to note first:
• Resource Constraints: Mobile devices have limited processing power, memory, and battery life. One significant issue in this context is throttling, where the device's processing speed is intentionally reduced to prevent overheating and conserve battery life. To make the best use of these resources, it's essential to optimize models from an algorithmic perspective using techniques such as quantization, pruning, and model compression.
• Supported Models: When developing AI solutions for mobile devices, there are many manufacturers, each with their individual computing setup (CPU + GPU + NPU) and a custom inference engine that works best for their model or some subset of devices. Due to this variety, it is important to test compatibility and optimize the solutions to ensure a consistent user experience.
Consumer equipment, such as laptops and PCs, provide a wide range of user experiences, which are greatly impacted by the underlying hardware and architecture of these devices.
For instance, when developing applications for Apple's M1 and M2-powered devices, a key point is the ARM architecture that these devices utilize. ARM architecture differs from the more traditional x86 architecture used by Intel-powered devices. This architectural difference impacts how software is compiled, executed, and optimized for these devices.
Apple's M1/2 chips prioritize optimized code for their architecture, enabling speedy and effective performance. To make the most of Apple's M1/2 hardware, developers need to rework their code and take advantage of the platform’s specific capabilities. This approach optimizes both the performance and efficiency of applications on Apple chips.
On the other hand, if you are developing an application for x86 architecture, the OpenVINO™ Toolkit is an excellent option. Intel designed it to optimize and accelerate the deployment of deep learning models on Intel hardware architectures. This framework empowers developers to leverage the full potential of Intel processors, GPUs, VPUs, and FPGAs for accelerated inference tasks.
Additionally, you can consider the ONNX Runtime, which provides cross-platform support for efficient inference of ONNX format models, making it a versatile choice for optimizing and deploying models across various hardware environments.
Although the x86 architecture is commonly used in hardware, not all laptops have Intel processors. Processors from other brands, such as AMD, are also frequently used. As a result, one should consider potential variability in processor brands and ensure that your software remains compatible and optimized across this range of hardware options when developing applications for x86 architecture.
In summary, accommodating all scenarios is possible, but it requires making trade-offs. Developing separate solutions for each device can be expensive, while opting for a generic engine may result in less-than-optimal performance.
On-premise servers with powerful GPUs, particularly those made by NVIDIA, are essential in determining the cost of an AI solution. The selection of GPU has a significant effect on the efficiency of model inference. Although powerful GPUs can accelerate computation, they also come at a higher cost. This expense encompasses not just the initial purchase cost of the GPUs, but also ongoing costs, such as electricity consumption and potential maintenance.The choice of operating system affects deployment cost and ease. Linux is preferred for its stability, security, and compatibility with optimized libraries like CUDA, which can speed up GPU computations. Linux has a steeper learning curve but can be more cost-effective in the long run. Efficient AI model deployment requires proper server configuration and management. Windows may require more effort to optimize AI performance, which can increase deployment time and costs.
Cloud-based AI solutions offer scalability and high-performance computing capabilities, particularly for the inference and training of deep learning models. Cloud infrastructure often provides access to potent hardware resources such as GPUs or TPUs, significantly enhancing the speed and efficiency of deep learning model inference.
Well-known cloud providers such as Google Cloud and AWS (Amazon Web Services) are now widely recognized for their state-of-the-art cloud infrastructure and popular setups to support AI-driven workloads. The Nvidia A100 is a prominent example of a high-performance GPU architecture frequently utilized in Google Cloud setups for AI workloads.
The choice of virtual CPUs, RAM, and GPU depends on project requirements. The setup aims to optimize resource utilization and minimize costs. If quick inference is crucial, resources are configured accordingly, with or without a GPU. Typically, Nvidia T4 (16GB) works for most tasks, but Nvidia A100 (40GB) may be chosen for more memory and speed. Automatic scaling is used for varying task loads. Specific project needs, budget constraints, speed, and continuous resource usage dictate the setup, with occasional allocation of more powerful machines for throughput spikes.
Cloud solutions enable centralized management of AI models. Updates and improvements to models can be deployed to the cloud server, reducing the need to update models on individual devices. This streamlines maintenance and ensures consistency across the user base.
When picking a service, you should consider the following:
• Latency: Cloud-based inference may cause delays due to data transmission to and from remote servers, especially for real-time applications where immediate responses are critical.
• Data Privacy: Sending sensitive data to the cloud can raise privacy concerns and compliance with data privacy and residency laws should be ensured.
• Costs: it's important to weigh the long-term costs against owning your own hardware. Depending on the configuration, scalability, and hardware, there is a threshold beyond which cloud computing becomes more expensive than on-premise alternatives. Take, for instance, a large warehouse security system needing real-time processing for 100 cameras. Using a cloud-based solution, such as servers equipped with powerful GPUs like NVIDIA's A100s, offers scalability but incurs continuous expenses. These costs can become substantial over time, especially for 24/7 operations.
Cloud solutions offer unparalleled scalability, enabling you to handle fluctuating workloads and user demand. However, the complexity of cloud development increases with scalability requirements. Designing an architecture that dynamically allocates resources, auto-scales, and efficiently manages data flow becomes crucial for cost-effective and responsive cloud-based AI solutions.
In summary, if an application requires real-time responses, edge devices and on-premise servers can provide the necessary processing power. For reaching a broad user base, mobile devices and consumer devices offer the greatest accessibility.
It is important to optimize models to ensure optimal performance and battery life in resource-constrained environments, such as mobile devices.
Cloud solutions are a great option when scalability is crucial, but the complexity of development increases with scalability requirements.
Integrating AI into your business involves choosing the right hardware and platform, as well as addressing aspects such as network connection, power optimization, and electricity supply. These considerations impact the performance, reliability, and cost-effectiveness of your AI solution. This section explores network and power considerations and offers insights for informed decision-making.
Efficient data transmission between AI components is essential for accuracy and responsiveness. We evaluate the amount and frequency of data transmission between devices or components. Large data volumes require efficient compression and transmission techniques to minimize delays. Real-time applications may require higher data transmission frequency, which may require low-latency network setups.
Moreover, stable connections between components are required for efficient performance. A strong connection between cameras, processing units, and data networks allows seamless capturing, processing, and analyzing of visual data.
An unstable connection can cause issues, including cameras intermittently losing connection and disrupting data flow. It's crucial to prevent disruptions for real-time object tracking used in surveillance or autonomous vehicles in order for the entire CV solution to be effective.
If a stable network connection cannot be established, the devices will occasionally disconnect from the server. To effectively address this issue, developers need to set up a system to store the latest data on the device and continuously send it to the server. This will require more effort from engineers.
Power efficiency is important for AI solutions that operate on batteries or have energy restrictions. Optimizing power consumption can help extend device battery life and reduce operational costs:
• Choosing energy-efficient hardware components can provide a balance between performance and power consumption.
• Utilizing power-saving mechanisms like CPU and GPU idle states can effectively reduce power consumption when resources are not fully utilized.
• More computational resources are required for complex models, which can lead to higher power consumption. In scenarios where accuracy can be sacrificed for efficiency, consider using simplified or smaller models.
Also, the electricity should be stable. Interruptions in the power supply cause downtime and data loss since the devices have to be relaunched. To address this issue, systematic logging is required to keep track of critical system data, such as the state of algorithms and ongoing processes. These logs enable quick recovery, allowing systems to resume their tasks within minutes, which minimizes negative impacts on the efficiency of the CV solution.
Scalability refers to how well the AI solution can handle increased workloads without sacrificing performance. To plan for potential scalability needs, the network should be able to handle increased data traffic without becoming a bottleneck.
To avoid overloading a single channel, we distribute processing tasks across multiple devices or servers. This ensures that resources are used efficiently.
Overall, the quality of network connections and power supply has a significant impact on the effectiveness of various tasks within CV solutions. For reliable operations, it's important to have a stable power supply and network infrastructure. When both are consistently strong, developers don't need to worry about creating a mechanism for reporting problems or adapting further logic to data losses and data synchronization.
Sometimes, it is impossible to avoid certain situations. For example, when a device runs only on battery power or is in remote and inaccessible areas, such as a drone exploring remote terrains, we must anticipate the possibility of a loss of connection.
In such cases, investing in advanced software may be better than fixing issues related to network connectivity or power supply. This ensures that devices work effectively in environments where traditional infrastructure support is not practical or too expensive.
The development of AI solutions is a multifaceted process — each component plays a critical role in determining the overall performance, scalability, and cost-effectiveness of the solution.
When choosing the right camera, you need to consider factors such as resolution, FPS, and depth estimation. The computing platform you choose will affect real-time performance, accessibility, and scalability. However, there are trade-offs to consider when trying to accommodate all scenarios.
It's crucial to ensure that network transmission is efficient, and connections are stable since network instability can disrupt real-time applications. To optimize power on battery-operated devices, power-saving mechanisms are a must. Finally, you should ensure that the electricity supply is stable as well.
Designing an AI solution can be a challenging task, with many factors to reflect on. We are here to help you strike the right balance between quality and affordability. We understand that identifying requirements and selecting the appropriate components can be difficult, particularly when each component is interconnected.
We hope this article got you excited, or at least you learned about your choices when considering applying machine vision or AI to your business. If you’ve read this far, we are happy to provide a complimentary consultation. We want to hear from you, Email us your comments to digest@opencv.ai
.png)