Deep Q-Learning for Dry Stacking Irregular Objects

We propose a reinforcement learning approach for automatically building dry stacked (i.e. no mortar) structures with irregular objects. Stacking irregular objects is a challenging problem since each assembly action can be drawn from a continuous space of poses for an object, and several local geometric and physical considerations strongly affect the stability. To tackle this challenge, we concentrate on a simplified 2D version of the problem. We present a reinforcement learning algorithm based on deep $Q$-learning, where the learned $Q$-function, which maps state-action pairs into expected long-term rewards, is represented by a deep neural network. As the action space is continuous the $Q$-network is trained by sampling a finite number of actions that consider both geometric and physical constraints to approximate the target $Q$-values, Experiments show that the proposed method outperforms previous heuristics-based planning, leading to super construction with objects containing a significant amount of variations. We validate the generated stacking plans by executing them using a robot arm and manufactured, irregular objects.