|Table of Contents|

Design and implementation of shoes arrangement robot system for unknown environment(PDF)

Journal of Shenzhen University Science and Engineering[ISSN:1000-2618/CN:44-1401/N]

Issue:
2022 Vol.39 No.4(363-488)
Page:
472-479
Research Field:
Electronics and Information Science

Info

Title:
Design and implementation of shoes arrangement robot system for unknown environment
Author(s):
TANG Xiaolong and HUANG Hui
College of Computer Science and Software Engineering, Shenzhen University, Shenzhen 518060, Guangdong Province, P. R. China
Keywords:
artificial intelligence intelligent sensing object detection object matching pose estimation robot
PACS:
TP391;TP242
DOI:
10.3724/SP.J.1249.2022.04472
Abstract:
In recent years, the robot and artificial intelligence technology have developed rapidly, but there still remains a challenging problem to realize the contact operation of robot in home environment. In order to make the robot arrange shoes automatically, we design an autonomous shoes arrangement robot system based on the 3D vision. In this system, the instance segmentation network and minimum enclosing rectangle are used to recognize the shoes and their orientation, and the grasping pose and placing pose of robot are estimated accurately by means of the point cloud information of depth camera. In addition, the convolution neural network and cosine similarity are adopted to match with a pair of shoes. Afterwards, we evaluate the accuracy of shoe orientation recognition and shoe matching in the system, and then carry out a real robot arrangement experiment. The result show that this method could assure 96.2% accuracy of shoe orientation recognition, and the matching accuracy of shoes increases from 62.6% to 87.4% when the VGG16 network is added to the shoes matching algorithm. In conclusion, this method can accurately recognize the shoes and their orientation, and then match with a pair of shoes, meanwhile, improves the stability of the robot shoes arrangement.

References:

[1] DALAL N, TRIGGS B. Histograms of oriented gradients for human detection [C]// 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. San Diego, USA: IEEE, 2005: 886-893.
[2] LOWE D G. Object recognition from local scale-invariant features [C]// Proceedings of the 7th IEEE International Conference on Computer Vision. Kerkyra, Greece: IEEE, 1999: 1150-1157.
[3] LIENHART R, MAYDT J. An extended set of haar-like features for rapid object detection [C]// Proceedings of International Conference on Image Processing. New York, USA: IEEE, 2002: I-900-I-903.
[4] FREUND Y, SCHAPIRE R E. Experiments with a new boosting algorithm [C]// Proceedings of the 13th International Conference on Machine Learning. Bari, Italy: ACM, 1996: 148-156.
[5] LIAW A, WIENER M. Classification and regression by random forest [J]. R News, 2002, 2/3:18-22.
[6] BAY H, ESS A, TUYTELAARS T, et al. Speeded-up robust features (SURF) [J]. Computer Vision and Image Understanding, 2008, 110(3): 346-359.
[7] GALVEZ-L?PEZ D, TARDOS J D. Bags of binary words for fast place recognition in image sequences [J]. IEEE Transactions on Robotics, 2012, 28(5): 1188-1197.
[8] KRIZHEVSKY A, SUTSKEVER I, HINTON G E. ImageNet classification with deep convolutional neural networks [C]// Conference and Workshop on Neural Information Processing Systems. Cambridge, USA: MIT Press, 2012: 1097-1105.
[9] REDMON J, DIVVALA S, GIRSHICK R, et al. You only look once: unified, real-time object detection [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, USA: IEEE, 2016: 779-788.
[10] LIU Wei, ANGUELOV D, ERHAN D, et al. SSD: single shot multibox detector [C]// European Conference on Computer Vision. Cham, Switzerland: Springer, 2016: 21-37.
[11] GIRSHICK R, DONAHUE J, DARRELL T, et al. Rich feature hierarchies for accurate object detection and semantic segmentation [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Columbus, USA: IEEE, 2014: 580-587.
[12] GIRSHICK R. Fast R-CNN [C]// Proceedings of the IEEE International Conference on Computer Vision. Santiago, Chilepp: IEEE, 2015: 1440-1448.
[13] REN Shaoqing, HE Kaiming, GIRSHICK R, et al. Faster R-CNN: towards real-time object detection with region proposal networks [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, 39(6): 1137-1149.
[14] HE Kaiming, GKIOXARI G, DOLL?R P, et al. Mask R-CNN [C]// Proceedings of the IEEE International Conference on Computer Vision. Venice, Italy: IEEE, 2017: 2961-2969.
[15] LONG J, SHELHAMER E, DARRELL T. Fully convolutional networks for semantic segmentation [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Boston, USA: IEEE, 2015: 3431-3440.
[16] MAHLER J, MATL M, LIU X, et al. Dex-net 3.0: computing robust vacuum suction grasp targets in point clouds using a new analytic model and deep learning [C]// IEEE International Conference on Robotics and Automation. Brisbane, Australia: IEEE, 2018: 5620-5627.
[17] JIANG Y, MOSESON S, SAXENA A. Efficient grasping from RGBD images: Learning using a new rectangle representation [C]// IEEE International Conference on Robotics and Automation. Shanghai, China: IEEE, 2011: 3304-3331.
[18] LENZ I, LEE H, SAXENA A, et al. Deep learning for detecting robotic grasps [J]. International Journal of Robotics Research, 2015: 34(3/4): 705-724.
[19] REDMON J, ANGELOVA A. Real-time grasp detection using convolutional neural networks [C]// IEEE International Conference on Robotics and Automation. Seattle, USA: IEEE, 2015: 1316-1322.
[20] LIN T Y, DOLL?R P, GIRSHICK R, et al. Feature pyramid networks for object detection [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, IEEE, 2017: 2117-2125.

Memo

Memo:
-