These five computer vision technologies refresh your worldview（5）

5--Instance segmentation

In addition to semantic segmentation, instance segmentation segments instances of different classes, such as marking five cars in five different colors. In the classification, there is usually an image in which one target is the focus and the task is to say what the image is. But in order to split the instance, we need to perform more complex tasks. We see complex spots with multiple overlapping objects and different backgrounds. We not only classify these different objects, but also determine the boundaries, differences and relationships between them!

So far, we have seen how to use CNN features in many interesting ways to effectively locate different targets in images with bounding boxes. Can we extend these techniques to locate the exact pixels of each target, not just the bounding box? Explore the instance segmentation problem on Facebook AI using an architecture called Mask R-CNN.

Like Fast R-CNN and Faster R-CNN, the underlying principle of Mask R-CNN is simple. Given that Faster R-CNN works very well in target detection, can we extend it for pixel-level segmentation?

Mask R-CNN does this by adding a branch to the Faster R-CNN, which outputs a binary mask that indicates whether a given pixel is part of the target. This branch is a full convolutional network based on the CNN's feature map. Given the CNN feature map as input, the network outputs the matrix at all positions with 1s in the pixel belonging to the target, and outputs 0 (this is called binary mask) elsewhere.

In addition, when running on the original Faster R-CNN architecture without modification, the area of ​​the feature map selected by RoIPool (pool of interest area) is slightly out of alignment with the area of ​​the original image. Since image segmentation requires pixel-level specificity, unlike a bounding box, this naturally leads to inaccuracies. Mask R-CNN solves this problem by adjusting RoIPool to more precisely align by using a method called RoIAlign (region of interest alignment). In essence, RoIAlign uses bilinear interpolation to avoid rounding errors, resulting in inaccurate detection and segmentation.

Once these masks are generated, Mask R-CNN combines them with the classification and bounding boxes from Faster R-CNN to generate such an accurate segmentation:

   in conclusion

These five major computer vision technologies help computers extract, analyze, and understand useful information from one or a series of images. I haven't talked about many other advanced technologies, including style shifting, coloring, motion recognition, 3D objects, body pose estimation, and more. In fact, the cost of computer vision is too high to be explored in depth, and I encourage you to explore it further, whether through online courses, blog tutorials or official documentation. For beginners, I highly recommend the CS231n course because you will learn how to implement, train and debug your own neural network. As a bonus, you can get all the presentation slides and homework guides from my GitHub repository. I hope it will guide you to change your view of the world!

Follow Me

Link:Tenco

twitter:https://twitter.com/TencoTechnology

facebook:https://www.facebook.com/tencotechnology

linkedin:https://www.linkedin.com/in/nicole-song-64400b147/

These five computer vision technologies refresh your worldview（4）

4--semantic segmentation

At the heart of computer vision is the segmentation process, which divides the entire image into groups of pixels that can then be labeled and classified. In particular, semantic segmentation attempts to semantically understand the role of each pixel in an image (for example, is it a car, a motorcycle, or another type of class?). For example, in the above picture, in addition to identifying people, roads, cars, trees, etc., we must also depict the boundaries of each object. Therefore, unlike classification, we need to perform dense pixel-by-pixel prediction from the model.

Like other computer vision tasks, CNN has had great success in segmentation. One of the popular initial methods is to perform a patch classification through a sliding window in which each pixel is divided into classes using its surrounding images. However, this is computationally very inefficient because we do not reuse the shared features between overlapping patches.

Instead, the solution is the University of California, Berkeley's Full Convolutional Network (FCN), which promotes an end-to-end CNN architecture for intensive prediction without any fully connected layers. This allows split graphs to be generated for images of any size and is much faster than patch sorting methods. Almost all subsequent semantic segmentation methods use this paradigm.

However, there is still a problem: the convolution at the original image resolution will be very expensive. To solve this problem, the FCN uses downsampling and upsampling inside the network. The downsampling layer is called stripe convolution and the upsampling layer is called deconvolution.

Although the upsampling/downsampling layer is used, the FCN generates a coarse segmentation map due to information loss during pooling. SegNet is a more efficient memory architecture than FCNs that use the largest pooling and encoding-decoder framework. In SegNet, a fast/jump connection is introduced from a higher resolution feature map to improve the upsampling/downsampling roughness.

Recent semantic segmentation studies rely heavily on full convolutional networks such as expanded convolution, DeepLab and RefineNet.

Follow Me

Link:Tenco

twitter:https://twitter.com/TencoTechnology

facebook:https://www.facebook.com/tencotechnology

linkedin:https://www.linkedin.com/in/nicole-song-64400b147/

These five computer vision technologies refresh your worldview（3）

3--target tracking

Target tracking refers to tracking one or more targets of a particular interest in a given scenario. Traditionally, it has been used in video and real-world interactions, which are observed after initial target detection. It is now essential for autonomous driving systems, such as the self-driving vehicles of companies such as Uber and Tesla.

The target tracking method can be divided into two categories according to the observation model: the generation method and the discrimination method. The generation method uses a generation model to describe the apparent features and minimize reconstruction errors to search for targets, such as PCA. The discriminant method can be used to distinguish between the target and the background, and its performance is more robust, and it gradually becomes the main method of tracking. The discriminating method is also called detection tracking, and deep learning belongs to this category. In order to achieve tracking by detection, we detect candidate targets for all frames and use deep learning to identify the desired target from the candidates. Two basic network models are available: Stack Autoencoder (SAE) and Convolutional Neural Network (CNN).

The most popular deep network using SAE tracking tasks is the Deep Learning Tracker, which proposes offline pre-training and online fine-tuning networks. The process is like this:

Offline unsupervised pre-training A stacked denoising autoencoder using a large-scale natural image dataset to obtain a general target representation. By adding noise to the input image and reconstructing the original image, the stack denoising autoencoder can achieve more robust feature representation.

The coded portion of the pre-trained network is combined with the classifier to obtain a classification network, and then the network is fine-tuned using the positive and negative samples obtained from the initial frame, which can distinguish the current target from the background. DLT uses a particle filter as a motion model to generate candidate patches for the current frame. The classification network outputs the probability scores of these patches, indicating the confidence of their classification, and then selects the highest patch of these patches as the target.

In model updates, DLT uses a way to limit thresholds.

Due to its superiority in image classification and target detection, CNN has become the mainstream depth model for computer vision and visual tracking. In general, large-scale CNN can be trained both as a classifier and as a tracker. Two representative CNN-based tracking algorithms are Full Convolutional Network Tracker (FCNT) and Multi-Domain CNN (MD Net).

FCNT successfully analyzed and utilized the feature map of the VGG model, which was a pre-trained ImageNet and produced the following observations:

CNN feature maps can be used for positioning and tracking.

Many CNN feature maps are noisy or unrelated to the task of distinguishing a particular target from its background.

Higher layers capture the semantic concept of object classes, while lower layers encode more discriminant features to capture intra-class variations.

Due to these observations, FCNT designed a feature selection network to select the most relevant feature maps on the conv4-3 and conv5-3 layers of the VGG network. Then, to avoid overfitting the noise, it also designed two additional channels (called SNet and GNet) for the selected features of the two layers. GNet captures the category information of the target, while SNet targets the region of interest (ROI). Finally, through SNet and GNet, the classifier obtains two predicted heat maps, and the tracker decides which heat map to use to generate the final trace result based on whether there is a disturber. The flow of FCNT is as follows.

Unlike FCNT's idea, MD Net uses all sequences of video to track moving objects. The above network uses irrelevant image data to reduce the training requirements of tracking data, and this idea has some deviation from tracking. The target of one class in this video can be the background in another video, so MD Net proposes the concept of multiple domains to independently distinguish between targets and backgrounds in each domain. A domain represents a set of videos that contain targets of the same type.

As shown below, MD Net is divided into two parts: the shared layer and the K branch of a specific domain layer. Each branch contains a binary classification layer with softmax loss, which is used to distinguish between targets and backgrounds in each domain, and the shared layer is shared with all domains to ensure a generic representation.

In recent years, deep learning researchers have experimented with different approaches to adapt to the characteristics of visual tracking tasks. They have explored many directions: applying other network models, such as recursive neural networks and deep confidence networks, designing network structures to accommodate video processing and end-to-end learning, optimizing processes, structures and parameters, or even deep learning with traditional computers Visual methods or methods combining language processing and speech recognition in other fields.

to be continued 

Follow Me

Link:Tenco

twitter:https://twitter.com/TencoTechnology

facebook:https://www.facebook.com/tencotechnology

linkedin:https://www.linkedin.com/in/nicole-song-64400b147/

These five computer vision technologies refresh your worldview（2）

2--target detection

The task of defining a target in an image typically involves the bounding box of a single target and the output of the tag. This differs from the classification/positioning task in that it applies classification and positioning to many goals, not just a dominant goal. You only have 2 types of target categories, the target bounding box and the non-target bounding box. For example, in car detection, you must use its bounding box to detect all cars in a given image.

If we use sliding window technology in a way that classifies and locates images, we need to apply CNN to many different locations of the image. Because CNN classifies each location as a target or background, we need to apply many locations and different scales in CNN, which is computationally expensive!

To deal with this situation, neural network researchers have proposed using regions instead, where we find "blobby" image regions that may contain targets. This area is relatively fast to run. The first compelling model is R-CNN (region-based convolutional neural network). In R-CNN, we first scan the input image using an algorithm called selective search to find possible targets, generating about 2,000 candidate regions. Then we run CNN on a per-regional box basis. Finally, we take the output of each CNN and input it to the SVM to classify the regions and use linear regression to tighten the bounding box of the target.

Basically, we turned target detection into an image classification problem. However, there are some problems - slow training, a lot of disk space, and slow reasoning.

The direct descendant of R-CNN is the fast R-CNN, which improves the detection speed by 2 enhancements: 1) performs feature extraction before the candidate area, so only runs one CNN on the entire image, and 2) replaces with softmax layer SVM, thus extending the prediction of neural networks, rather than creating a new model.

Fast R-CNN performs better in terms of speed because it trains only one CNN for the entire image. However, the selective search algorithm still takes a lot of time to generate candidate regions.

Therefore, a faster R-CNN was discovered, which is now a canonical model for target detection based on deep learning. The region from the feature is predicted by inserting a region generation network (RPN), which replaces the slow selective search algorithm with a fast neural network. The RPN is used to determine "where" to reduce the computational requirements of the entire inference process. The RPN scans each location quickly and efficiently to assess whether further processing is required in a given area. It does this by outputting k bounding box regions, each with 2 scores, indicating the probability of the target at each location.

Once we have our candidate areas, we will provide them directly to the content that is basically a fast R-CNN. We added a pooling layer, some fully connected layers, and finally a softmax classification layer and bounding box regenerator.

All in all, the faster R-CNN achieves better speed and higher accuracy. It's worth noting that although future models do a lot of work to improve detection speed, few models can surpass faster R-CNNs with higher advantages. In other words, a faster R-CNN may not be the easiest or fastest way to detect a target, but it is still one of the best performing methods.

The main target detection trend in recent years has turned to faster, more efficient detection systems. This is seen in methods such as You Only Look Once (YOLO), Single Shot MultiBox Detector (SSD) and Region-based Complete Convolutional Network (R-FCN) as a way to share calculations across the entire image. Therefore, these methods distinguish themselves from the expensive subnets associated with the three R-CNN technologies. The main reason behind these trends is to avoid having separate algorithms focus on their sub-problems in isolation, as this usually increases training time and reduces network accuracy.

to be continued 

Follow Me

Link:Tenco

twitter:https://twitter.com/TencoTechnology

facebook:https://www.facebook.com/tencotechnology

linkedin:https://www.linkedin.com/in/nicole-song-64400b147/

These five computer vision technologies refresh your worldview（1）

Computer vision is one of the most popular research areas in the field of deep learning. It is located at the intersection of many academic subjects, such as computer science (graphics, algorithms, theory, systems, architecture), mathematics (information retrieval, machine learning), engineering (robotics, speech, natural language processing, image processing), Physics (optics), biology (neuroscience) and psychology (cognitive science). Since computer vision represents a relative understanding of the visual environment and its context, many scientists believe that the field paves the way for artificial intelligence because of its cross-domain mastery.

So what is computer vision? Here are some official textbook definitions:

“Building clear, meaningful physical object descriptions from images” (Ballard & Brown, 1982)

"Calculating the properties of a 3D world from one or more digital images" (Trucco & Verri, 1998)

“Make useful decisions about real objects and scenes based on perceived images” (Sockman & Shapiro, 2001)

Why study computer vision? The most obvious answer is a collection of rapidly growing useful applications derived from this area of ​​research. Here is a small part of it:

Face Recognition: Snapchat and Facebook use the Face Detection algorithm to apply a convolution kernel and identify you in the image.

Image Retrieval: Google Images uses content-based queries to search for related images. The algorithm analyzes the content in the query image and returns the result based on the best match.

Game and Control: Microsoft Kinect is a great commercial product in a game that uses stereo vision.

Surveillance: Surveillance cameras are ubiquitous in public places and are used to detect suspicious behavior.

Biometrics: Fingerprint, iris and face matching are still some of the common methods used in biometrics.

Smart cars: Vision is still the primary source of information for detecting traffic signs and lights and other visual features.

I recently completed Stanford University's exciting CS231n course on visual recognition using convolutional neural networks. Visual recognition tasks such as image classification, location and detection are key components of computer vision. Recent developments in neural networks and deep learning methods have greatly improved the performance of these state of the art visual recognition systems. This course is an extraordinary resource that teaches me the details of using a deep learning architecture in top computer vision research. In this article, I want to share the five main computer vision techniques I have learned, as well as the main deep learning models and applications that use each technology.

1--image classification

The problem with image classification is this: Given a set of images of a single category of annotations, we are required to predict these categories for a new set of test sets and test the accuracy of the predictions. There are many challenges with this task, including viewpoint changes, scale changes, intra-class changes, image distortion, image occlusion, conditions, and cluttered backgrounds.

How do we write an algorithm that can divide images into different categories? Computer vision researchers have proposed a data-driven approach to solve this problem. Instead of trying to specify each image category of interest directly in the code, they provide the computer with many samples of each image class, then develop learning algorithms, view the samples and understand the visual appearance of each class. In other words, they first collect a training set with an annotated image and then pass it to the computer to process the data.

Given this fact, the entire image classification process can be formalized as follows:

Our input is a training set that includes N images, each of which is labeled with K different categories.

Then, we use this training set to train the classifier to understand what each category looks like.

Finally, we evaluate the effect of the classifier by having this classifier predict a new set of images that it has not seen before. We then compare the real labels of these images with those predicted by the classifier.

Convolutional neural networks (CNNs) are the most popular architecture for image classification. A typical use case for CNNs is that you pass this network image, and then the network classifies the data. CNNs often start with the input "scanner" and it is not intended to parse all training data at once. For example, to enter a 100 x 100 pixel image, you don't need a layer with 10,000 nodes. Instead, you'll create a 10×10 scan input layer that you can provide for the first 10×10 pixels of the image. After this input, the next 10 × 10 pixels can be input by moving the scanner one pixel to the right. This technique is called a sliding window.

The input data is then fed through the convolutional layer instead of the normal layer. Each node only focuses on the part that is adjacent to itself. These convolutional layers tend to shrink as the network deepens, mainly through the easy decomposition of input. In addition to convolutional layers, they usually have a feature pooling layer. Pooling is a way to filter details: the common pooling technique is maximum pooling, we use 2 x 2 pixels and pass the pixels with the largest amount of specific properties.

At present, most image classification techniques are trained on ImageNet, which is a data set of about 1.2 million high-resolution training images. The test image does not display the initial comment (no split or label), and the algorithm must generate a label that specifies the object that exists in the image. The leading computer vision teams at Oxford, INRIA, XRCE and others use the best computer vision methods available for this data set. Often, computer vision systems are complex, multi-level systems that often require manual tuning to optimize parameters at an early stage.

The winner of the first ImageNet competition Alex Krizhevsky (NIPS 2012) The deep convolutional neural network pioneered by Yann LeCun. Its structure includes 7 hidden layers, excluding some of the largest pooling layers. The first 5 layers are convolutional layers and the last 2 layers are fully connected layers. The activation function for each hidden layer is ReLU. These trainings are faster and more expressive than logic units. In addition, when similar units have stronger activation values, it also uses competitive normalization to suppress hidden activity. This contributes to changes in strength.

In terms of hardware requirements, Alex used a very efficient convolutional network implementation on two Nvidia GTX 580 GPUs (more than 1000 fast small cores). The GPU is well suited for matrix matrix multiplication and has a very high memory bandwidth. This allowed him to train the network within a week and quickly combine the results of 10 patches during the test. If we can communicate state quickly enough, we can spread the network across multiple cores. As cores become cheaper and data sets grow larger, large neural networks will improve faster than older computer vision systems. Since AlexNet, a number of new models have used CNN as their backbone architecture and have achieved outstanding results in ImageNet: ZFNet (2013), GoogLeNet (2014), VGGNet (2014), ResNet (2015), DenseNet (2016) )Wait.

to be continued 

Follow Me

Link:Tenco

twitter:https://twitter.com/TencoTechnology

facebook:https://www.facebook.com/tencotechnology

linkedin:https://www.linkedin.com/in/nicole-song-64400b147/