research-article

Learning to place new objects in a scene

Authors:

Ashutosh SaxenaAuthors Info & Claims

International Journal of Robotics Research, Volume 31, Issue 9

Pages 1021 - 1043

https://doi.org/10.1177/0278364912438781

Published: 01 August 2012 Publication History

Abstract

Placing is a necessary skill for a personal robot to have in order to perform tasks such as arranging objects in a disorganized room. The object placements should not only be stable but also be in their semantically preferred placing areas and orientations. This is challenging because an environment can have a large variety of objects and placing areas that may not have been seen by the robot before. In this paper, we propose a learning approach for placing multiple objects in different placing areas in a scene. Given point-clouds of the objects and the scene, we design appropriate features and use a graphical model to encode various properties, such as the stacking of objects, stability, object-area relationship and common placing constraints. The inference in our model is an integer linear program, which we solve efficiently via an linear programming relaxation. We extensively evaluate our approach on 98 objects from 16 categories being placed into 40 areas. Our robotic experiments show a success rate of 98% in placing known objects and 82% in placing new objects stably. We use our method on our robots for performing tasks such as loading several dish-racks, a bookshelf and a fridge with multiple items.

References

[1]

Koppula HS,Anand A,Joachims T,Saxena A.Semantic labeling of 3D point clouds for indoor scenes.25th annual conference on neural information processing systems; 2011Granada, Spain; 2011. .

[2]

Berenson D,Diankov R,Nishiwaki K,Kagami S,Kuffner J.Grasp planning in complex scenes.7th international IEEE-RAS conference on humanoid robots; 2007Pittsburgh, USA; 2007. 42.

[3]

Berg A,Berg T,Malik J.Shape matching and object recognition using low distortion correspondences.<conftl/>; 2005San Diego, USA; 2005. 26.

Digital Library

[4]

BerkelaarMEiklandKNotebaertP (2004) LP_solve 5.5, open source (mixed-integer) linear programming system. http://lpsolve.sourceforge.net/5.5/.

[5]

Bicchi A,Kumar V.Robotic grasping and contact: a review.IEEE international conference on robotics and automation; 2000San Francisco, USA; 2000. 348.

[6]

Brook P,Ciocarlie M,Hsiao K.Collaborative grasp planning with multiple object representations.IEEE international conference on robotics and automation; 2011Shanghai, China; 2011. 2851.

[7]

Ciocarlie MT,Allen PK.On-line interactive dexterous grasping.<conftl/>; 2008Madrid, Spain; 2008. 104.

Digital Library

[8]

Coffman E,Garey M,Johnson DApproximation algorithms for NP-hard problems. Hochbaum DS, ed. Boston: PWS Publishing Company; 1996:46-93.

Digital Library

[9]

Cortes C,Vapnik V.Support-vector networks.Machine learning. 1995;20 (3): 273-297

Digital Library

[10]

Deng J,Berg A,Fei-Fei L.Hierarchical semantic indexing for large scale image retrieval.IEEE computer society conference on computer vision and pattern recognition; 2011Colorado Springs, USA; 2011. 785.

Digital Library

[11]

Diankov R,Kuffner JPittsburgh, USA: Report, Robotics Institute; 2008:

[12]

Diankov R,Srinivasa S,Ferguson D,Kuffner J.Manipulation planning with caging grasps.8th international IEEE-RAS conference on humanoid robots; 2008Daejeon, South Korea; 2008. 285.

[13]

Divvala S,Hoiem D,Hays J,Efros A,Hebert M.An empirical study of context in object detection.<conftl/>; 2009Miami, USA; 2009. 1271.

[14]

Dogar M,Srinivasa S.A framework for push-grasping in clutter.Robotics: science and systems. 2011 2011;:

[15]

Dy J,Brodley C.Feature selection for unsupervised learning.Journal of Machine Learning Research. 2004;5:845-889

Digital Library

[16]

Edsinger A,Kemp C.Manipulation in human environments.<conftl/>; 2006Genoa, Italy; 2006. 102.

[17]

Fergus R,Perona P,Zisserman A.Object class recognition by unsupervised scale-invariant learning.IEEE computer society conference on computer vision and pattern recognition; 2003Wisconsin, USA; 2003. 264.

[18]

FergusonM (2011) Improved AR markers for topological navigation. ROS 3D Contest. http://www.ros.org/news/2011/01/ros-3d-entries-improved-ar-markers-for-topological-navigation.html.

[19]

Fisher M.The Lagrangian relaxation method for solving integer programming problems.Management Science. 1981;27:1-18

[20]

Fisher M,Hanrahan P.Context-based search for 3D models.ACM Transactions on Graphics. 2010;29 (6): 182

Digital Library

[21]

Fisher M,Savva M,Hanrahan P.Characterizing structural relationships in scenes using graph kernels.ACM Transactions on Graphics. 2011;30 (4): 34

Digital Library

[22]

Fu H,Cohen-Or D,Dror G,Sheffer A.Upright orientation of man-made objects.ACM Transactions on Graphics. 2008;27 (3): 42

Digital Library

[23]

Gelfand N,Mitra NJ,Guibas LJ,Pottmann H.Robust global registration.3rd eurographics symposium on geometry processing; 2005Vienna, Austria 4-6 July 2005; 2005. .

Digital Library

[24]

Globerson A,Jaakkola T.Fixing max-product: convergent message passing algorithms for map LP-relaxations.21st annual conference on neural information processing systems; 2007Vancouver, Canada; 2007. .

[25]

Glover J,Rusu R,Bradski G.Monte carlo pose estimation with quaternion kernels and the Bingham distribution.Robotics: science and systems. 2011 2011;:

[26]

Hanley J,McNeil B.The meaning and use of the area under a receiver operating (ROC) curvel characteristic.Radiology. 1982;143 (1): 29-36

[27]

Hedau V,Hoiem D,Forsyth D.Recovering the spatial layout of cluttered rooms.12th international IEEE conference on computer vision; 2009Kyoto, Japan; 2009. 1849.

[28]

Heitz G,Gould S,Saxena A,Koller D.Cascaded classification models: combining models for holistic scene understanding.22nd annual conference on neural information processing systems; 2008Vancouver, Canada; 2008. .

[29]

Ho ESL,Komura T,Tai CL.Spatial relationship preserving character motion adaptation.ACM Transactions on Graphics. 2010;29 (4): 33

Digital Library

[30]

Hsiao K,Nangeroni P,Huber M,Saxena A,Ng AY.Reactive grasping using optical proximity sensors.IEEE international conference on robotics and automation; 2009Kobe, Japan; 2009. 4230.

Digital Library

[31]

Jain A,Kemp C.Pulling open doors and drawers: coordinating an omni-directional base and a compliant arm with equilibrium point control.IEEE international conference on robotics and automation; 2010Anchorage, USA; 2010. 1807.

[32]

Jalali A,Ravikumar P,Sanghavi S,Ruan C.A dirty model for multi-task learning.24th annual conference on neural information processing systems; 2010Vancouver, Canada; 2010. .

[33]

Jetchev N,Toussaint M.Task space retrieval using inverse feedback control.28th international conference on machine learning; 2011Washington, USA; 2011. .

[34]

Jiang Y,Amend J,Lipson H,Saxena A.Learning hardware agnostic grasps for a universal jamming gripper.IEEE international conference on robotics and automation; 2012aMinnesota, USA; 2012a. .

[35]

Jiang Y,Moseson S,Saxena A.<atl/>.Efficient grasping from RGBD images: learning using a new rectangle representation; 2011aShanghai, China; 2011a. .

[36]

Jiang Y,Zheng C,Lim M,Saxena A.Learning to place new objects.<conftl/>; 2011bLos Angeles, USA; 2011b. .

[37]

Jiang Y,Zheng C,Lim M,Saxena A.Learning to place new objects.IEEE international conference on robotics and automation; 2012bMinnesota, USA; 2012b. .

[38]

Joachims TMaking large-scale SVM learning practical. Cambridge: MIT Press; 1999:

[39]

Johnson A,Hebert M.Using spin images for efficient object recognition in cluttered 3D scenes.IEEE Transactions on Pattern Analysis and Machine Intelligence. 1999;21 (5): 433-449

Digital Library

[40]

Katz D,Brock O.Manipulating articulated objects with interactive perception.IEEE international conference on robotics and automation; 2008Pasadena, USA; 2008. 272.

[41]

Klingbeil E,Saxena A,Ng AY.Learning to open new doors.<conftl/>; 2010Taipei, Taiwan; 2010. 2751.

[42]

Koller D,Friedman NProbabilistic graphical models: principles and techniques. Cambridge: MIT Press; 2009:

Digital Library

[43]

Lai K,Bo L,Ren X,Fox D.Sparse distance learning for object recognition combining RGB and depth information.IEEE international conference on robotics and automation; 2011Shanghai, China; 2011. .

[44]

Le QV,Kamm D,Kara A,Ng AY.Learning to grasp objects with multiple contact points.IEEE international conference on robotics and automation; 2010Anchorage, USA; 2010. 5062.

[45]

Leibe B,Leonardis A,Schiele B.Combined object categorization and segmentation with an implicit shape model.Workshop on statistical learning in computer vision; 2004Prague, Czech Republic; 2004. .

[46]

Li C,Kowdle A,Saxena A,Chen T.Towards holistic scene understanding: Feedback enabled cascaded classification models.24th annual conference on neural information processing systems; 2010Vancouver, Canada; 2010. .

[47]

Li C,Saxena A,Chen T.¿-mrf: capturing spatial and semantic structure in the parameters for scene understanding.25th annual conference on neural information processing systems; 2011Granada, Spain; 2011. .

[48]

Liebelt J,Schmid C,Schertler K.Viewpoint-independent object class detection using 3D feature maps.IEEE computer society conference on computer vision and pattern recognition; 2008Anchorage, USA; 2008. 1.

[49]

Liu H,Yu L.Toward integrating feature selection algorithms for classification and clustering.IEEE Transactions on Knowledge and Data Engineering. 2005;17 (4): 491-502

Digital Library

[50]

Lodi A,Martello S,Vigo D.Heuristic algorithms for the three-dimensional bin packing problem.European Journal of Operational Research. 2002;141 (2): 410-420

[51]

Lozano-Pérez T,Jones J,Mazer E,O'Donnell P.Task-level planning of pick-and-place robot motions.Computer. 2002;22 (3): 21-29

Digital Library

[52]

Maitin-Shepard J,Cusumano-Towner M,Lei J,Abbeel P.Cloth grasp point detection based on multiple-view geometric cues with application to robotic towel folding.IEEE international conference on robotics and automation; 2010Anchorage, USA; 2010. 2308.

[53]

Miller AT,Knoop S,Christensen HI,Allen PK.Automatic grasp planning using shape primitives.IEEE international conference on robotics and automation; 2003Taipei, Taiwan; 2003. 1824.

[54]

Nguyen V.Constructing stable force-closure grasps.<conftl/>; 1986Dallas, USA; 1986. 129.

Digital Library

[55]

Novotni M,Klein R.3D Zernike descriptors for content based shape retrieval.8th ACM symposium on solid modeling and applications; 2003Seattle, USA; 2003. 216.

Digital Library

[56]

Pisinger D,Sigurd M.Using decomposition techniques and constraint programming for solving the two-dimensional bin-packing problem.INFORMS Journal on Computing. 2007;19 (1): 36-51

Digital Library

[57]

Ponce J,Stam D,Faverjon B.On computing two-finger force-closure grasps of curved 2D objects.International Journal of Robotics Research. 1993;12 (3): 263

[58]

Quigley M,Gerkey B,Conley K,Faust J,Foote T,Leibs J,Berger E,Wheeler R,Ng A.ROS: an open-source robot operating system.Workshop on open source software in robotics; 2009Kobe, Japan; 2009. .

[59]

Rao D,Le Q,Phoka T,Quigley M,Sudsang A,Ng A.Grasping novel objects with depth segmentation.IEEE-RSJ international conference on intelligent robots and systems; 2010Taipei, Taiwan; 2010. 2578.

[60]

Rosales C,Porta J,Ros LRobotics: science and systems. USA Cambridge: Los Angeles MIT Press; 2011 2011:1-8.

[61]

Roth D,Yih W.Integer linear programming inference for conditional random fields.<conftl/>; 2005Bonn, Germany; 2005. 736.

Digital Library

[62]

Rusinkiewicz S,Levoy M.Efficient variants of the ICP algorithm.3rd international conference on 3D digital imaging and modeling; 2001Quebec City, Canada; 2001. 145.

[63]

Rusu R,Marton Z,Blodow N,Beetz M.Learning informative point classes for the acquisition of object model maps.10th international conference on control, automation, robotics and vision; 2008Hanoi, Vietnam; 2008. 643.

[64]

Rusu RB,Blodow N,Beetz M.Fast point feature histograms (FPFH) for 3D registration.IEEE international conference on robotics and automation; 2009Kobe, Japan; 2009. 3212.

Digital Library

[65]

Rusu RB,Cousins S.3D is here: point cloud library (PCL).IEEE international conference on robotics and automation; 2011Shanghai, China; 2011. 1.

[66]

Savarese S,Fei-Fei L.3D generic object categorization, localization and pose estimation.11th IEEE international conference on computer vision; 2007Rio de Janeiro, Brazil; 2007. 1.

[67]

Saxena A,Chung SH,Ng AY.3-D depth reconstruction from a single still image.International Journal of Computer Vision. 2008a;76 (1): 53-69

Digital Library

[68]

Saxena A,Driemeyer J,Kearns J,Ng AY.Robotic grasping of novel objects.21st annual conference on neural information processing systems; 2006Vancouver, Canada 3-5 December 2007; 2006. .

[69]

Saxena A,Driemeyer J,Ng A.Robotic grasping of novel objects using vision.International Journal of Robotics Research. 2008b;27 (2): 157

Digital Library

[70]

Saxena A,Driemeyer J,Ng AY.Learning 3-D object orientation from images.IEEE international conference on robotics and automation; 2009aKobe, Japan; 2009a. 4266.

Digital Library

[71]

Saxena A,Sun M,Ng A.Make3D: Learning 3D scene structure from a single still image.IEEE Transactions on Pattern Analysis and Machine Intelligence. 2009b;31 (5): 824-840

Digital Library

[72]

Saxena A,Wong L,Ng AY.Learning grasp strategies with partial shape information.23rd conference on artificial intelligence; 2008cIllinois, USA; 2008c. 1491.

Digital Library

[73]

Schuster M,Okerman J,Nguyen H,Rehg J,Kemp C.Perceiving clutter and surfaces for object placement in indoor environments.10th international IEEE-RAS conference on humanoid robots; 2010Nashville, USA; 2010. 152.

[74]

Sturm J,Konolige K,Stachniss C,Burgard W.3D pose estimation, tracking and model learning of articulated objects from dense depth video using projected texture stereo.Robotics: science and systems. 2010 2010;:

[75]

Sugie H,Inagaki Y,Ono S,Aisu H,Unemi T.Placing objects with multiple mobile robots-mutual help using intention inference.IEEE international conference on robotics and automation; 1995Nagoya, Japan; 1995. 2181.

[76]

Taskar B,Chatalbashev V,Koller D.Learning associative Markov networks.<conftl/>; 2004Banff, Canada; 2004. 102.

Digital Library

[77]

Taskar B,Guestrin C,Koller D.Max-margin Markov networks.17th annual conference on neural information processing systems; 2003Vancouver, Canada; 2003. .

[78]

Thomas A,Ferrar V,Leibe B,Tuytelaars T,Schiel B,Van Gool L.Towards multi-view object class detection.<conftl/>; 2006New York, USA; 2006. 1589.

Digital Library

[79]

Torralba A,Murphy K,Freeman WT.Using the forest to see the trees: object recognition in context.Communications of the ACM. 2010;53 (3): 107-114

Digital Library

[80]

Toussaint M,Plath N,Lang T,Jetchev N.Integrated motor control, planning, grasping and high-level reasoning in a blocks world using probabilistic inference.IEEE international conference on robotics and automation; 2010Anchorage, USA; 2010. 385.

[81]

Winn J,Criminisi A,Minka T.Object categorization by learned universal visual dictionary.10th IEEE international conference on computer vision; 2005Beijing, China; 2005. 1800.

Digital Library

[82]

Xiong X,Huber D.Using context to create semantic 3D models of indoor environments.21st British machine vision conference; 2010Aberystwyth, Wales; 2010. .

[83]

Yanover C,Meltzer T,Weiss Y.Linear programming relaxations and belief propagation-an empirical study.Journal of Machine Learning Research. 2006;7:1887-1907

Digital Library

Cited By

Huang YTaylor NConkey ALiu WHermans T(2024)Latent Space Planning for Multiobject Manipulation With Environment-Aware Relational ClassifiersIEEE Transactions on Robotics10.1109/TRO.2024.336095640(1724-1739)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1109/TRO.2024.3360956
Nath APatel DNiyogi R(2024)Mass Prediction and Analysis of an Object’s Mass from Its Image Using Deep LearningSN Computer Science10.1007/s42979-024-03050-65:6Online publication date: 15-Jul-2024
https://dl.acm.org/doi/10.1007/s42979-024-03050-6
Scalise RThomason JBisk YSrinivasa S(2022)Improving Robot Success Detection using Static Object Data2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)10.1109/IROS40897.2019.8968142(4229-4235)Online publication date: 28-Dec-2022
https://dl.acm.org/doi/10.1109/IROS40897.2019.8968142
Show More Cited By

Recommendations

Manipulator and object tracking for in-hand 3D object modeling

Recognizing and manipulating objects is an important task for mobile robots performing useful services in everyday environments. While existing techniques for object recognition related to manipulation provide very good results even for noisy and ...
Analyzing differences between teachers when learning object affordances via guided exploration

Our work focuses on robots deployed in human environments. These robots, which will need specialized object manipulation skills, should leverage end-users to efficiently learn the affordances of objects in their environment. This approach is promising ...
Learning visual representations for perception-action systems

We discuss vision as a sensory modality for systems that interact flexibly with uncontrolled environments. Instead of trying to build a generic vision system that produces task-independent representations, we argue in favor of task-specific, learn-able ...

Comments

Information & Contributors

Information

Published In

cover image International Journal of Robotics Research

International Journal of Robotics Research Volume 31, Issue 9

August 2012

93 pages

ISSN:0278-3649

Issue’s Table of Contents

Publisher

Sage Publications, Inc.

United States

Publication History

Published: 01 August 2012

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

34
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 12 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Huang YTaylor NConkey ALiu WHermans T(2024)Latent Space Planning for Multiobject Manipulation With Environment-Aware Relational ClassifiersIEEE Transactions on Robotics10.1109/TRO.2024.336095640(1724-1739)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1109/TRO.2024.3360956
Nath APatel DNiyogi R(2024)Mass Prediction and Analysis of an Object’s Mass from Its Image Using Deep LearningSN Computer Science10.1007/s42979-024-03050-65:6Online publication date: 15-Jul-2024
https://dl.acm.org/doi/10.1007/s42979-024-03050-6
Scalise RThomason JBisk YSrinivasa S(2022)Improving Robot Success Detection using Static Object Data2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)10.1109/IROS40897.2019.8968142(4229-4235)Online publication date: 28-Dec-2022
https://dl.acm.org/doi/10.1109/IROS40897.2019.8968142
Haustein JHang KStork JKragic D(2022)Object Placement Planning and optimization for Robot Manipulators2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)10.1109/IROS40897.2019.8967732(7417-7424)Online publication date: 28-Dec-2022
https://dl.acm.org/doi/10.1109/IROS40897.2019.8967732
Liu WPaxton CHermans TFox D(2022)StructFormer: Learning Spatial Structure for Language-Guided Semantic Rearrangement of Novel Objects2022 International Conference on Robotics and Automation (ICRA)10.1109/ICRA46639.2022.9811931(6322-6329)Online publication date: 23-May-2022
https://dl.acm.org/doi/10.1109/ICRA46639.2022.9811931
Xu KYu HHuang RGuo DWang YXiong R(2022)Efficient Object Manipulation to an Arbitrary Goal Pose: Learning-Based Anytime Prioritized Planning2022 International Conference on Robotics and Automation (ICRA)10.1109/ICRA46639.2022.9811547(7277-7283)Online publication date: 23-May-2022
https://dl.acm.org/doi/10.1109/ICRA46639.2022.9811547
You YShao LMigimatsu TBohg J(2021)OmniHang: Learning to Hang Arbitrary Objects using Contact Point Correspondences and Neural Collision Estimation2021 IEEE International Conference on Robotics and Automation (ICRA)10.1109/ICRA48506.2021.9560971(5921-5927)Online publication date: 30-May-2021
https://dl.acm.org/doi/10.1109/ICRA48506.2021.9560971
Niyogi RSharma SVavrecka MMilani A(2020)A Learning Based Approach for Planning with Safe ActionsComputational Science and Its Applications – ICCSA 202010.1007/978-3-030-58814-4_7(93-105)Online publication date: 1-Jul-2020
https://dl.acm.org/doi/10.1007/978-3-030-58814-4_7
Dong HAsadi ESun GPrasad DChen I(2019)Real-Time Robotic Manipulation of Cylindrical Objects in Dynamic Scenarios Through Elliptic Shape PrimitivesIEEE Transactions on Robotics10.1109/TRO.2018.286880435:1(95-113)Online publication date: 1-Feb-2019
https://dl.acm.org/doi/10.1109/TRO.2018.2868804
Castellano GCarolis BMacchiarulo NRossano V(2019)Learning waste Recycling by playing with a Social Robot2019 IEEE International Conference on Systems, Man and Cybernetics (SMC)10.1109/SMC.2019.8914455(3805-3810)Online publication date: 6-Oct-2019
https://dl.acm.org/doi/10.1109/SMC.2019.8914455
Show More Cited By

View Options

View options

Media

Figures

Other

Tables

View Issue’s Table of Contents