OpenCV - Separating Items on Store Shelves

In my last post I went through Hough transform and how it can detect straight lines in an image. In this post I will try and use it to create a program that can separate individual items in a store shelf. The idea is simple, store racks are usually horizontal and item boxes placed on the shelves are usually rectangular. So, if we can detect straight lines in an image, we can separate out these edges as straight lines. If we have lines described for all four edges of an object, we can deduce its bounding rectangle.

It is of course, not as trivial as the example we tried in our previous post. There are obvious practical issues that need to be tackled while dealing with real world images:

1. Image quality of the provided images are typically not optimal. Brightness and contrast need to be adjusted first. 
2. The edges in the images are usually not perfect. They would be broken by objects places on them 
3. Some package edges are perfectly straight. 
4. Sometimes the pictures are taken at an angle. 
5. Text and pictures on the item packaging clutter and confuse our algorithms. 

There is quite a bit of pre-processing that is required to be done on the images before they can be used. A few basic steps are:

1. Smoothing to reduce some noise. It may be better to use bilateral smoothing to preserve the edges. 
2. Adjusting brightness and contrast of the image so that the interesting portions of the image are highlighted and distinguishable. 
3. Separating detection of horizontal and vertical edges. Since the horizontal and vertical edges have different degrees of clarity in the image, it helps to detect them separately with different degrees of thresholds (even if they are percentages). To separate the horizontal and vertical edges, and enhance the longer edges with comparison to the small ones, I have used a directional erosion kernel. It also serves to selectively remove vertical edges when we want to detect horizontal edges and vice versa. 

In spite of the above, it is still not easy to accurately detect the boundaries. By accuracy I actually mean robustness. That is, having the same set of algorithms and tuning parameters work over a large variety of images. Lowering the threshold to detect all edges in a certain image sometimes makes it super sensitive and detects spurious lines in some other images. Moreover, it will also detect both upper edge and lower edge of the shelf as two different lines. Taking the spacing between the lines as a known fact (approximately how many shelves and items to expect) I used a simple technique of trimming outliers to filter these out.

Here are the results of using my application on a few real world images. The green and blue lines are the separators. The sliders are the only parameters that I have kept adjustable. The approach is robust enough to have all the other parameters fixed.

As you would notice, it is able to separate out most of the packages, though it is not accurate 100% of the times. Some more detailed pre-processing would make it even more robust and accurate. The sources for this project can be gotten from my github gist here.