Research Interests

Our research focuses on creating algorithms that allow robots to interact with the world. These general-purpose motion planning, machine learning, and manipulation algorithms can be applied to robots that work in homes, factories, and operating rooms. We are interested in all aspects of algorithm development; including creating efficient algorithms, proving their theoretical properties, validating them on real-world robots and problems, integrating them with sensing and higher-level reasoning, and distributing them to open-source communities. Our lab draws on ideas in search, optimization, machine learning, motion planning, control theory, and topology to develop these algorithms and to prove their properties. We also seek to develop algorithms which can generalize to many types of practical tasks and application areas.

Press

Article about our RSS 2022 paper: Data Augmentation for Manipulation. (link)

Our paper on learning what a robot can and can't do is published in Science Robotics (link)

Glen Chou receives the NDSEG graduate fellowship (link)

Dmitry Berenson receives the NSF CAREER award for research on deformable object manipulation (link)

New ONR grant to support our work on Humanoid locomotion and deformable object manipulation (link)

Announcement of the IEEE RAS Early Career Award for Prof. Berenson (link)

Our soft hand work is featured on IEEE Spectrum Video Friday (link)

Announcement of our project on locomotion planning for shipboard humanoid robots (link)

An article covering our two new NSF grants on manipulating deformable objects and motion planning for soft robots from the Robotics Business Review (link)

Our research is featured in an article about the future of service robots in Robotics Industry Association Industry Insights (link)

Bridging Learning and Motion Planning

Learning algorithms are good at quickly producing solutions to problems that have been encountered before, but they have difficulty generalizing to new situations. Planning algorithms are good at generalizing to new situations but at significant (and sometimes prohibitive) computational cost. This project seeks to combine the best of both approaches.

Expand

The goal of bridging learning and planning is to create algorithms whose performance improves with experience while maintaining generality and reliability. Learning algorithms are good at quickly producing solutions to problems that have been encountered before, but they have difficulty generalizing to new situations. Planning algorithms are good at generalizing to new situations but at significant (and sometimes prohibitive) computational cost. Our work seeks to combine the best of both approaches to create a method whose performance improves as it gains more experience while retaining the ability to generalize to new situations.

A Robot Path Planning Framework that Learns from Experience
Dmitry Berenson, Pieter Abbeel, and Ken Goldberg
IEEE International Conference on Robotics and Automation (ICRA), May, 2012.
Details | PDF | Video

Reproducing Expert-Like Motion in Deformable Environments Using Active Learning and IOC
Calder Phillips-Grafflin and Dmitry Berenson
International Symposium on Robotics Research (ISRR), September, 2015.
PDF

Learning Object Orientation Constraints and Guiding Constraints for Narrow Passages from One Demonstration
Changshuo Li and Dmitry Berenson
International Symposium on Experimental Robotics (ISER), October, 2016.
PDF | Video

Using Previous Experience for Humanoid Navigation Planning
Yu-Chi Lin and Dmitry Berenson
IEEE-RAS International Conference on Humanoid Robots (Humanoids), November, 2016.
PDF | Video

Humanoid Navigation in Uneven Terrain using Learned Estimates of Traversability
Yu-Chi Lin and Dmitry Berenson
IEEE-RAS International Conference on Humanoid Robots (Humanoids), November, 2017.
PDF | Video

Simultaneous learning of hierarchy and primitives for complex robot tasks
Anahita Mohseni-Kabir, Changshuo Li, Victoria Wu, Daniel Miller, Benjamin Hylak, Sonia Chernova, Dmitry Berenson, Candace Sidner, Charles Rich
Autonomous Robots (AuRo), Vol. 43, No. 4, pp. 859-874, April 2019.
Details

Efficient Humanoid Contact Planning using Learned Centroidal Dynamics Prediction
Yu-Chi Lin, Brahayam Ponton, Ludovic Righetti, and Dmitry Berenson
IEEE International Conference on Robotics and Automation (ICRA), May, 2019.
PDF | Video

Inferring Occluded Geometry Improves Performance when Retrieving an Object from Dense Clutter
Andrew Price*, Linyi Jin*, and Dmitry Berenson
International Symposium on Robotics Research (ISRR), October 2019.
Details | PDF | Video

Learning Parametric Constraints in High Dimensions from Demonstrations
Glen Chou, Necmiye Ozay, and Dmitry Berenson
Conference on Robot Learning (CoRL), October, 2019.
Details | PDF

Learning When to Trust a Dynamics Model for Planning in Reduced State Spaces
Dale McConachie, Thomas Power, Peter Mitrano, and Dmitry Berenson
IEEE Robotics and Automation Letters (RA-L) (Presented at ICRA 2020), vol. 5, no. 2, pp. 3540-3547, April 2020.
Details | PDF | Video

Learning Constraints from Locally-Optimal Demonstrations under Cost Function Uncertainty
Glen Chou, Necmiye Ozay, and Dmitry Berenson
IEEE Robotics and Automation Letters (RA-L) (presented at ICRA 2020), vol. 5, no. 2, pp. 3682-3690, April 2020.
Details | PDF | Video

Learning for Humanoid Multi-Contact Navigation Planning
Yu-Chi Lin
Ph.D. dissertation, Robotics Institute, University of Michigan, April 2020.
PDF

Deformable Object Manipulation: Learning While Doing
Dale McConachie
Ph.D. dissertation, Robotics Institute, University of Michigan, April 2020.
PDF

Explaining Multi-stage Tasks by Learning Temporal Logic Formulas from Suboptimal Demonstrations
Glen Chou, Necmiye Ozay, and Dmitry Berenson
Robotics: Science and Systems (RSS), July 2020.
Details | PDF | Video

Uncertainty-Aware Constraint Learning for Adaptive Safe Motion Planning from Demonstrations
Glen Chou, Necmiye Ozay, and Dmitry Berenson
Conference on Robot Learning (CoRL), November 2020.
Details | PDF | Video

Keep it Simple: Data-efficient Learning for Controlling Complex Systems with Simple Models
Thomas Power and Dmitry Berenson
IEEE Robotics and Automation Letters (RA-L) (Presented at ICRA 2021), vol. 6, no. 2, pp. 1184-1191, April 2021.
Details | PDF | Video

TAMPC: A Controller for Escaping Traps in Novel Environments
Sheng Zhong, Zhenyuan Zhang, Nima Fazeli, and Dmitry Berenson
IEEE Robotics and Automation Letters (RA-L) (Presented at ICRA 2021), vol. 6, no. 2, pp. 1447-1454, April 2021.
Details | PDF | Video

Planning with Learned Dynamics: Probabilistic Guarantees on Safety and Reachability via Lipschitz Constants
Craig Knuth*, Glen Chou*, Necmiye Ozay, and Dmitry Berenson
IEEE Robotics and Automation Letters (RA-L) (Presented at ICRA 2021), vol. 6, no. 3, pp. 5129 - 5136, March 2021.
Details | PDF | Video

Learning where to trust unreliable models in an unstructured world for deformable object manipulation
Peter Mitrano, Dale McConachie, and Dmitry Berenson
Science Robotics, vol. 6, no. 54, May 2021.
Details | Video

Long-horizon humanoid navigation planning using traversability estimates and previous experience
Yu-Chi Lin and Dmitry Berenson
Autonomous Robots (AuRo), vol. 45, no. 6, pp. 937-956, June 2021.
Details | PDF | Video

Learning Constraints from Demonstrations with Grid and Parametric Representations
Glen Chou, Dmitry Berenson, and Necmiye Ozay
International Journal of Robotics Research (IJRR), vol. 40, no. 10-11, pp. 1255-1283, September 2021.
Details | PDF

Belief Representations for Planning with Contact Uncertainty
Brad Saund
Ph.D. dissertation, Robotics Institute, University of Michigan, July 2021.
PDF

Learning temporal logic formulas from suboptimal demonstrations: theory and experiments
Glen Chou, Necmiye Ozay, and Dmitry Berenson
Autonomous Robots (AuRo), vol. 46, pp. 149-174, January 2022.
Details | PDF | Video

Model Error Propagation via Learned Contraction Metrics for Safe Feedback Motion Planning of Unknown Systems
Glen Chou, Necmiye Ozay, and Dmitry Berenson
Conference on Decision and Control (CDC), December 2021.
PDF | Video

Data Augmentation for Manipulation
Peter Mitrano and Dmitry Berenson
Robotics Science and Systems (RSS), June 2022.
Details | Video

Variational Inference MPC using Normalizing Flows and Out-of-Distribution Projection
Thomas Power and Dmitry Berenson
Robotics Science and Systems (RSS), June 2022.
Details | Video

Manipulation of Deformable Objects

A major frontier for autonomous manipulation lies in interacting with deformable objects, such as cloth and cooking ingredients (in the context of a domestic robot) and thread and tissue (in the context of surgery). This is an extremely under-explored area in autonomous manipulation, mainly because deformable objects are difficult to model and simulate.

Expand

Our work in deformable object manipulation has explored planning and control methods for elastic objects as well as objects like cloth and rope. These objects have infinite-dimensional configuration spaces and are difficult to model and simulate. The key to our approach is to find appropriate model reductions that make planning and control tractable despite these challenges. We also use machine learning methods to approximate the dynamics of deformable objects and determine where our approximations are valid. The approximate dynamics models are then used in motion planning and control to accomplish manipulation tasks.

Manipulation of Deformable Objects Without Modeling and Simulating Deformation
Dmitry Berenson
IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), November, 2013
PDF | Video

A Representation Of Deformable Objects For Motion Planning With No Physical Simulation
Calder Phillips-Grafflin and Dmitry Berenson
IEEE International Conference on Robotics and Automation (ICRA), May, 2014.
PDF | Video

An Online Method for Tight-tolerance Insertion Tasks for String and Rope
Weifu Wang, Dmitry Berenson, and Devin Balkcom
IEEE International Conference on Robotics and Automation (ICRA), May, 2015.
PDF | Video

Bandit-Based Model Selection for Deformable Object Manipulation
Dale McConachie and Dmitry Berenson
Workshop on the Algorithmic Foundations of Robotics (WAFR), December, 2016.
PDF | Video

Interleaving Planning and Control for Deformable Object Manipulation
Dale McConachie, Mengyao Ruan, and Dmitry Berenson
International Symposium on Robotics Research (ISRR), December, 2017.
PDF | Video

Accounting for Directional Rigidity and Constraints in Control for Manipulation of Deformable Objects without Physical Simulation
Mengyao Ruan, Dale McConachie, and Dmitry Berenson
IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), October, 2018.
PDF | Video

Occlusion-robust Deformable Object Tracking without Physics Simulation
Cheng Chi and Dmitry Berenson
IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), November 2019.
Details | PDF | Video

Manipulating Deformable Objects by Interleaving Prediction, Planning, and Control
Dale McConachie, Andrew Dobson, Mengyao Ruan, and Dmitry Berenson
International Journal of Robotics Research (IJRR), vol. 39, no. 8, pp. 957-982, July 2020.
Details | Video

Deformable Object Manipulation: Learning While Doing
Dale McConachie
Ph.D. dissertation, Robotics Institute, University of Michigan, April 2020.
PDF

Tracking Partially-Occluded Deformable Objects While Enforcing Geometric Constraints
Yixuan Wang, Dale McConachie, and Dmitry Berenson
IEEE International Conference on Robotics and Automation (ICRA), May, 2021.
Details | PDF | Video

Data Augmentation for Manipulation
Peter Mitrano and Dmitry Berenson
Robotics Science and Systems (RSS), June 2022.
Details | Video

Manipulation in Clutter

Perception and planning methods for manipulation in extreme clutter

Expand

This project seeks to develop perception and planning methods that allow robots to manipulate in very cluttered environments. A key challenge for perception is the extreme occlusion and the lack of knowledge about how objects will move when manipulated, which makes it difficult to estimate/track them. We investigate shape completion and tracking methods to mitigate the effects of occlusion. For planning it is difficult to reason about occlusion without being too conservative or too reckless. We investigate ways to respond to unanticipated contacts online and to plan in the presence of uncertainty about the occupancy of the environment.

Motion Planning for Manipulators in Unknown Environments with Contact Sensing Uncertainty
Brad Saund and Dmitry Berenson
International Symposium on Experimental Robotics (ISER), November 2018.
PDF | Video

The Blindfolded Robot: A Bayesian Approach to Planning with Contact Feedback
Brad Saund, Sanjiban Choudhury, Siddhartha Srinivasa, and Dmitry Berenson
International Symposium on Robotics Research (ISRR), October, 2019.
PDF | Video

Diverse Plausible Shape Completions from Ambiguous Depth Images
Brad Saund and Dmitry Berenson
Conference on Robot Learning (CoRL), November 2020.
Details | PDF | Video

Fusing RGBD Tracking and Segmentation Tree Sampling for Multi-Hypothesis Volumetric Segmentation
Andrew Price*, Kun Huang*, and Dmitry Berenson
IEEE International Conference on Robotics and Automation (ICRA), May, 2021.
Details | PDF | Video

CLASP: Constrained Latent Shape Projection for Refining Object Shape from Robot Contact
Brad Saund and Dmitry Berenson
Conference on Robot Learning (CoRL), November 2021.
Details | PDF | Video

Soft Tracking Using Contacts for Cluttered Objects to Perform Blind Object Retrieval
Sheng Zhong, Nima Fazeli, and Dmitry Berenson
IEEE Robotics and Automation Letters (RA-L) (Presented at ICRA 2022), vol. 7, no. 2, pp. 3507-3514, April 2022.
Details | PDF | Video

Tracking Deformable Objects

Perception methods for tracking partially-occluded deformable objects

Expand

This project investigates methods to track deformable objects despite partial (self-)occlusions. Our methods are based on Coherent Point Drift (CPD) and use RGBD data to reason about the location of the deformable object in 3D. A key ability is ensuring that the tracking estimate complies with the geometric constraints of the object and environment.

Humanoid Locomotion Planning

We are creating algorithms that allow humanoids to locomote through difficult environments such as ships or disaster sites while subject to disturbances.

Expand

Because of their human-like morphology and capabilities, humanoid robots offer the promise of taking the place of people operating in shipboard environments and disaster sites. For instance, a shipboard maintenance robot requires the ability to autonomously traverse environments with complex geometry and respond to significant disturbances; i.e., the robot will need to pass through narrow corridors, open doors, climb and descend ladders and stairs, and navigate through clutter. The robot must also be able to operate despite significant disturbance forces arising from e.g. ocean waves. We are developing new locomotion planning algorithms that use previous experience and integrate with perception to quickly compute effective plans for these scenarios.

This project is in collaboration with the ExtReMe Lab at Virginia Tech.

Using Previous Experience for Humanoid Navigation Planning
Yu-Chi Lin and Dmitry Berenson
IEEE-RAS International Conference on Humanoid Robots (Humanoids), November, 2016.
PDF | Video

Integrated Affordance Detection and Humanoid Locomotion Planning
Will Pryor, Yu-Chi Lin, and Dmitry Berenson
IEEE-RAS International Conference on Humanoid Robots (Humanoids), November, 2016.
PDF | Video

Humanoid Navigation Planning in Large Unstructured Environments Using Traversability-Based Segmentation
Yu-Chi Lin and Dmitry Berenson
IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), October, 2018
PDF | Video

Robust Humanoid Contact Planning with Learned Zero- and One-Step Capturability Prediction
Yu-Chi Lin, Ludovic Righetti, and Dmitry Berenson
IEEE Robotics and Automation Letters (RA-L) (Presented at ICRA 2020), vol. 5, no. 2, pp. 2451-2458, April 2020.
Details | PDF | Video

Learning for Humanoid Multi-Contact Navigation Planning
Yu-Chi Lin
Ph.D. dissertation, Robotics Institute, University of Michigan, April 2020.
PDF

Human-Robot Collaboration for Manipulation

Expand

This project addresses a large space of tasks that cannot be fully automated because of either the limitations of current algorithms or prohibitive cost and set-up time. Such tasks generally require humans to collaborate in close proximity and adapt to each other's decisions and motions. This project explores accomplishing these tasks through human-robot collaboration.

Recent hardware developments in robotics have made human-robot collaboration physically possible, but robots still require new algorithms to ensure safety, efficiency, and fluency when working with people. Creating such algorithms is difficult because there can be high uncertainty in what a person is going to do and how they are going to do it. This project explores the integration of reasoning about how a person moves and how he or she makes decisions into a robot motion planning and decision-making framework. The research centers on the development of new algorithmic frameworks for modeling, simulating, and planning for human-robot collaboration, which requires advances in robot training, task modeling, human motion understanding, high-dimensional motion planning with uncertainty, and metrics to assess human-robot joint action.

Human-Robot Collaborative Manipulation Planning Using Early Prediction of Human Motion
Jim Mainprice and Dmitry Berenson
IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), November, 2013
PDF | Video

Predicting Human Reaching Motion in Collaborative Tasks Using Inverse Optimal Control and Iterative Re-planning
Jim Mainprice, Rafi Hayne, and Dmitry Berenson
IEEE International Conference on Robotics and Automation (ICRA), May, 2015.
PDF | Video

A Framework for Unsupervised Online Human Reaching Motion Recognition and Early Prediction
Ruikun Luo and Dmitry Berenson
IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), September, 2015
PDF

Considering Avoidance and Consistency in Motion Planning for Human-Robot Manipulation in a Shared Workspace
Rafi Hayne, Ruikun Luo, and Dmitry Berenson
IEEE International Conference on Robotics and Automation (ICRA), May, 2016.
PDF | Video

Goal Set Inverse Optimal Control and Iterative Re-planning for Predicting Human Reaching Motions in Shared Workspaces
Jim Mainprice, Rafi Hayne, and Dmitry Berenson
IEEE Transactions on Robotics (T-RO), Vol. 32, No. 4, pp. 897-908, August 2016.
Details | PDF

A Framework For Robot-Assisted Doffing of Personal Protective Equipment
Antonio Umali and Dmitry Berenson
IEEE International Conference on Robotics and Automation (ICRA), May, 2017.
PDF | Video

Unsupervised Early Prediction of Human Reaching for Human-robot Collaboration in Shared Workspaces
Ruikun Luo, Rafi Hayne, and Dmitry Berenson
Autonomous Robots (AuRo), Vol. 42, No. 3, pp 631-648, March 2018.
Details | PDF | Video

Soft Pneumatic Hand

A soft penumatic hand that will enable grasping objects with pose, shape, and actuation uncertainty.

Expand

Most robotic systems today are rigid, or composed of parts that are hard and unable to flex, morph, or change other physical characteristics. This can be an issue when a rigid robotic hand is trying to grasp something irregularly shaped. These rigid designs need precise control over position, force, and other senses to successfully pick up objects. Our soft hand design builds on the work of Raphael Deimel and Oliver Brock at TU Berlin in soft pneumatic robot fingers.

The idea behind a soft robotic hand is to use soft and flexible fingers that can adapt to the objects being picked up. A soft hand is far more robust in regards to the uncertainty of the objectsâ€™ placement, size, and shape. Although some certainty is still required, less attention needs to be given to the placement of the object in 3D space. A soft robotic hand can simply morph to the object's shape, and then get a firm grip on it.

The purpose of this project is to research the development, effectiveness, and practicality of a soft robotic hand that can be used on robots that manipulate complex objects. The initial goal established was to construct a hand prototype that could be pneumatically controlled to pick up objects of different sizes, shapes, and weights. This is an ongoing project, however, as there are many areas for expansion. These include sensor integration, stronger fingers, and a more sophisticated control system. Now that a proof of concept has been completed, the doors have been opened for a vast range of ways to improve the quality of the hand.

This project's hardware and software are completely open-source.

Source code available on GitHub.

CAD for the molds and hand.

Documentation for the hand.

Improving Soft Pneumatic Actuator Fingers through Integration of Soft Sensors, Position and Force Control, and Rigid Fingernails
John Morrow, Hee-Sup Shin, Calder Phillips-Grafflin, Sung-Hwan Jang, Jacob Torrey, Riley Larkins, Steven Dang, Yong-Lae Park, and Dmitry Berenson
IEEE International Conference on Robotics and Automation (ICRA), May, 2016.
PDF | Video

User-Guided Manipulation for the DARPA Robotics Challenge

We seek to create a user-guided manipulation framework for High Degree-of-Freedom robots operating in environments with limited communication to apply to the DARPA Robotics Challenge.

Expand

We seek to create a user-guided manipulation framework for High Degree-of-Freedom robots operating in environments with limited communication to apply to the DARPA Robotics Challenge. Our approach consists of three elements: (1) a user-guided perception interface which assists the user to provide task level commands to the robot, (2) planning algorithms that autonomously generate robot motion while obeying relevant constraints, and (3) a trajectory execution and monitoring system. We are prototyping our work on the PR2 but are applying it for the DRC on the Hubo humanoid robot.

We are part of a multi-university Track A DARPA Robotics Challenge Team led by Drexel University.

See a description of the challenge here.

This project is joint work with the Interaction Lab (Prof. Sonia Chernova) and the HIVE lab (Prof. Rob Lindeman) at WPI.

Toward A User-Guided Manipulation Framework for High-DOF Robots with Limited Communication
Nicholas Alunni, Calder Phillips-Grafflin, Halit Bener Suay, Daniel Lofaro, Dmitry Berenson, Sonia Chernova, Robert W. Lindeman, Paul Oh
IEEE International Conference on Technologies for Practical Robot Applications (TePRA), April, 2013
PDF | Video

From Autonomy to Cooperative Traded Control of Humanoid Manipulation Tasks with Unreliable Communication: System Design and Lessons Learned
Jim Mainprice, Calder Phillips-Grafflin, Halit Bener Suay, Nicholas Alunni, Daniel Lofaro, Dmitry Berenson, Sonia Chernova, Robert W. Lindeman, and Paul Oh
IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), November, 2014
PDF | Video

Toward a user-guided manipulation framework for high-DOF robots with limited communication
Calder Phillips-Grafflin, Nicholas Alunni, Halit Bener Suay, Jim Mainprice, Daniel Lofaro, Dmitry Berenson, Sonia Chernova, Robert W. Lindeman, and Paul Oh
Journal of Intelligent Service Robotics, Vol. 7, No. 3, pp. 121-131, July 2014
Details | PDF

Constrained Manipulation Planning

Expand

Overview

Our everyday lives are full of tasks that constrain our movement. Carrying a coffee mug, lifting a heavy object, or sliding a milk jug out of a refrigerator are examples of tasks that involve constraints imposed on our bodies as well as on the manipulated objects. Creating algorithms for general-purpose robots to perform these kinds of tasks also involves computing motions that are subject to multiple simultaneous task constraints.

In general, a robot cannot assume arbitrary joint configurations when performing constrained motion. Instead, the robot must move within a manifold embedded in its high-dimensional configuration space that satisfies both the constraints of the task and the limits of the mechanism. Planning such motion is challenging for two reasons: 1) The space of motions is continuous and high-dimensional. 2) Constraints on the robot's motion (imposed by the task and the environment) often make feasible states difficult to find. For instance, constraints on the pose of the robot's hand can restrict the allowable states to an infinitely thin manifold in the configuration space. Such manifolds cannot be searched using standard sampling-based planning techniques.

My thesis work focused on developing a constrained manipulation planning framework for practical manipulation tasks. The framework has three main components: constraint representation, constraint-satisfaction strategies, and a sampling-based planning strategy. These three components come together to create an efficient and probabilistically-complete manipulation planning algorithm called the Constrained BiDirectional RRT (CBiRRT).

An implementation of the above framework as the open-sourceÂ Constrained Manipulation Planning Suite.

Task Space Regions

Some of the most common constraints in manipulation planning are constraints on the pose of a robot's end-effector.Â They arise in tasks such as reaching to grasp an object, carrying a cup of coffee, or opening a door. These constraints can act on the entire path, such as not spilling a cup of coffee, and/or on the goal of a path, such as reaching to grasp an object. In general a set of constraints for a manipulator's end-effector can consist of an arbitrary number of poses spread in an arbitrary way throughout the task space. However, such a broad representation lacks three fundamental properties that are necessary for sampling-based planning:

The set of poses must be easy to specify.
Sampling from the set of poses must be efficient.
The distance to the set must be fast to compute.

TSRs describe end-effector pose constraints as volumes in SE(3), the space of rigid spatial transformations. These volumes are particularly useful for manipulation tasks such as reaching to grasp an object, manipulating an object with pose constraints, such as a glass of water, or placing an object onto a surface, such as table. TSRs are intuitive to specify, they can be efficiently sampled, and the distance to a TSR can be evaluated very quickly, making them ideal for sampling-based planners such as CBiRRT.

Task Space Region Chains for Whole-Body Manipulation

While TSRs can represent many useful constraints, a single TSR, or even a finite set of TSRs, is sometimes insufficient to capture the pose constraints of a given task.Â To describe more complex constraints such as closed chain kinematics and manipulating articulated objects, we use TSR Chains, which are defined by linking a series of TSRs. A TSR Chain can be thought of as a virtual manipulation whose reachability is the set of allowable poses. Though the sampling for TSR Chains follows clearly from that of TSRs, the distance metric for TSR Chains is radically different. TSR Chains allow high degree-of-freedom systems such as humanoid robots to perform constrained whole-body manipulation tasks while exploiting the redundancy of these systems.

Planning with Pose Uncertainty

A common assumption when planning for robotic manipulation tasks is that the robot has perfect knowledge of the geometry and pose of objects in the environment. For a robot operating in a home environment it may be reasonable to have geometric models of the objects the robot manipulates frequently and/or the robot's work area. However, these objects and the robot frequently move around the environment, introducing uncertainty into the pose of the objects relative to the robot. Laser-scanners, cameras, and sonar sensors can all be used to help resolve the poses of objects in the environment, but these sensors are never perfect and usually localize the objects to be within some hypothetical probability distribution of pose estimates. If this set of probable poses is large, planning with the best hypothesis alone can be unreliable because the robot may fail to complete the task and unsafe because the robot could collide with a poorly-localized object.

Since we would like to compute a plan that is guaranteed to meet task specifications for all hypotheses of object pose, we must modify the TSRs of a given task to account for pose uncertainty and introduce virtual obstacles into the simulation to avoid potential collisions. We start by duplicating the TSRs for each pose hypothesis and computing the intersection of all duplicates. We can then sample from the 6-dimensional polytope of intersection to obtain poses that are guaranteed to obey task constraints despite pose uncertainty. A key advantage of this approach is that if the pose uncertainty is too great to accomplish a certain task, we can quickly reject that task without invoking a planner. This approach can also eliminate grasping strategies which are not robust to uncertainty.

Motion Planning Theory

We were the first to present a proof for the probabilistic completeness of RRT-based algorithms when planning with constraints on end-effector pose. If the manifold of valid configurations corresponding to the pose constraint has non-zero measure in the C-space, it is straightforward to show that algorithms like CBiRRT are probabilistically complete. This is because random sampling in the C-space will eventually place samples inside of the manifold.Â

However, pose constraints can also induce lower-dimensional constraint manifolds in the configuration space, making rejection sampling techniques infeasible. Sampling-based algorithms such as CBiRRT can overcome this problem by using the sample-project method: sampling coupled with a projection operator to move configuration space samples onto the constraint manifold. Before our work, it was not known whether the sample-project method produces adequate coverage of the constraint manifold to guarantee probabilistic completeness. Our proof guarantees probabilistic completeness for a class of RRT-basedÂ algorithms given an appropriate projection operator.Â This proof is valid for constraint manifolds of any fixed dimensionality.

Base Placement for Mobile Manipulators

Path planning for a mobile manipulator involves multiple levels of planning which are often divided into sub-problems to manage complexity. For a pick-and place operation the break-down might be: 1) move the robot to a configuration near the object, 2) grasp the object, 3) move the robot (holding the object) to some configuration which places the object into its goal configuration. Breaking the problem into the above sub-problems reduces the complexity of the overall task by allowing sub-plans to be generated in series. However, ignoring the coupling between sub-problems can turn feasible problems into infeasible ones and introduce unnecessary difficulty for the path planning algorithm.Â We show how to couple sub-problems through choosing optimal grasps and base placements for transition configurations, i.e. configurations where the robot first grasps or releases the object. We find optimal configurations (in terms of grasp quality, manipulability, and clutter) by searching a constrained space of base placements and grasps using a co-evolutionary algorithm. This algorithm significantly out-performs random sampling and is able to generate high-quality configurations in extremely cluttered environments.

Constrained Manipulation Planning
Dmitry Berenson
Ph.D. dissertation, Robotics Institute, School of Computer Science, Carnegie Mellon University, June, 2011.
Details | PDF

Grasping in Cluttered Environments

The goal of a grasping algorithm is to find a pose and configuration of a robot's hand which grasps a given object in a desirable way. Our research in grasping has focused on extracting information about an object's environment to make decisions about how that object should be grasped.

Expand

The goal of a grasping algorithm is to find a pose and configuration of a robot's hand which grasps a given object in a desirable way. Development of grasp quality metrics, such as force-closure, has been intensely studied in robotics for many years; however these metrics neglect a key component of grasp selection: an object's environment. Our research in grasping has focused on extracting information about an object's environment to make decisions about how that object should be grasped. Considering the object's environment allows the robot to quickly eliminate hand poses that place the hand in areas where there are obstacles and to focus on areas where the hand is collision-free and where the fingers make good contact with the object. Our grasping algorithms are able to find grasps in environments where state-of-the-art algorithms cannot, while requiring only several seconds of computation. We have also extended these algorithms to consider two-handed grasps with similar results.

Grasp Synthesis in Cluttered Environments for Dexterous Hands
Dmitry Berenson and Siddhartha Srinivasa
IEEE-RAS International Conference on Humanoid Robots (Humanoids), December, 2008
Details | PDF | Video

Grasp Planning in Complex Scenes
Dmitry Berenson, Rosen Diankov, Koichi Nishiwaki, Satoshi Kagami, and James Kuffner
IEEE-RAS International Conference on Humanoid Robots (Humanoids), December, 2007
Details | PDF | Video

Faculty

Dmitry Berenson

Associate Professor
Robotics Department
EECS Department

dmitryb@umich.edu

PhD Students

Mark Van der Merwe (co-advised with Nima Fazeli)

PhD Student, Robotics

markvdm@umich.edu

Yating Lin

PhD Student, Robotics

yatinlin@umich.edu

Luís Marques

PhD Student, Robotics

lmarques@umich.edu

Fan Yang

PhD Student, Robotics

fanyangr@umich.edu

Zixuan Huang

PhD Student, Robotics

zixuanh@umich.edu

Abhinav Kumar

PhD Student, Robotics

abhin@umich.edu

Undergraduate Students

Advaith Balaji

Robotics Major

advaithb@umich.edu

Adam Hung

Robotics Major

adamhung@umich.edu

Yinong He

CSE Major

heyinong@umich.edu

Alumni

Name

Degree

After Graduation

Shaoxiong Yao

B.S. CS 2021 (UM)

Ph.D. at UIUC

Cheng Chi

B.S. CSE 2019 (UM)

Ph.D. at Columbia

Peter Mitrano

PhD, Robotics 2023

Berkshire Gray

Tom Power

PhD, Robotics 2024 (UM)

Google DeepMind

Erica Tevere

B.S. ME 2019 (UM)

Xucheng Ma

B.S. CSE and ME 2019 (UM)

Chenxi Gu

B.S. CS and Math (UM)

M.S. at Stanford

Kun Huang

B.S. CS 2020 (UM)

M.S. at UPenn

Craig Knuth

M.S. Robotics 2020 (UM)

JHU Applied Physics Lab

Yixuan Wang

B.S. CS and ME 2021 (UM)

Ph.D. at UIUC

Dylan Colli

MS Robotics 2024 (UM)

Berkshire Gray

Hao Wang

B.S. CS and ME 2022 (UM)

Ph.D. at USC

Haoxuan Shan

B.S. CSE and ECE 2022 (UM)

Ph.D. at Duke University

Changyuan (Peter) Qiu

B.S. in CSE 2022 (UM)

Ph.D. at University of Washington

Jiayi Pan

B.S. CSE 2023 (UM)

PhD at UC Berkeley

Ashwin Gupta

B.S. CSE 2022 (UM)

PhD at Johns Hopkins

ShengAo Wang

M.S. Robotics 2022 (UM)

Alison Ryckman

CSE Major

Marco Faroni

1-year Post-doc

Assistant Prof. at Politecnico di Milano

Dale McConachie

PhD, Robotics 2020 (UM)

Toyota Research Institute

Alex Henning

B.S./M.S. RBE 2015 (WPI)

Fetch Robotics

Jim Mainprice

2-year Post-Doc

Post-Doc at Max Planck Institute

Jessica Gwozdz

B.S. RBE/CS 2014 (WPI)

Foliage, Inc.

Rafi Hayne

B.S./M.S. CS 2016 (WPI)

Quan Peng

B.S. RBE 2014 (WPI)

M.S. at UC Berkeley

Nick Morin

B.S. RBE/CS 2014 (WPI)

Wayfair

Ransom Mowris

B.S. RBE/PW 2014 (WPI)

HighRes Biosolutions

Antonio Umali

M.S. CS 2016 (WPI)

Lecturer, Ateneo de Manila University

John Morrow

B.S. RBE 2015 (WPI)

Ph.D. at Oregon State Univ.

Steven Dang

B.S. RBE 2015 (WPI)

Tesla Motors

Will Pryor

B.S./M.S. RBE 2017 (WPI)

Ph.D. at Johns Hopkins University

Nathan Hughes

B.S. RBE/CS 2016 (WPI)

MIT Lincoln Labs

Evan Richard

B.S. RBE/ME 2015 (WPI)

Carnegie Robotics

Linyi Jin

B.S. CSE/ME 2019 (UM)

M.S., Ph.D. at University of Michigan

Yu-Chi Lin

PhD, Robotics 2020 (UM)

Nuro

Changshuo Li

M.S. RBE 2017 (WPI)

Andreas Orthey

1-year Post-Doc

Post-doc at AIST

Vinay Pilania

1-year Post-Doc

Mengyao Ruan

M.S. Robotics 2018 (UM)

AutoX

Yetong Zhang

B.S. CE 2018 (UM)

PhD at Georgia Tech

Andrew Dobson

1-year Post-Doc

Airlitix

Brad Saund

PhD, Robotics 2021 (UM)

Amazon

Glen Chou (co-advised w/ N. Ozay)

PhD, ECE 2022 (UM)

Postdoc at MIT

Calder Phillips-Grafflin

PhD, RBE 2017 (WPI)

Toyota Research Institute

Ruikun Luo

PhD Student

Moved to Jessie Yang's Lab

Andrew Price

2-year Post-doc

Waymo

Sheng Zhong (Johnson)

PhD, Robotics 2024 (UM)

Cruise

Book Chapters

Obeying Constraints During Motion Planning
Dmitry Berenson
Humanoid Robotics: A Reference, A. Goswami and P. Vadakkepat (Eds.), Springer, 2019.
Details | PDF

Refereed Journal Papers

Constraining Gaussian Process Implicit Surfaces for Robot Manipulation via Dataset Refinement
Abhinav Kumar, Peter Mitrano, and Dmitry Berenson
IEEE Robotics and Automation Letters (RA-L), vol. 9, no. 12, pp. 10994-11001, December 2024.
Details | Video
Constrained Stein Variational Trajectory Optimization
Thomas Power and Dmitry Berenson
IEEE Transactions on Robotics (T-RO), vol. 40, pp. 3602-3619, 2024.
Details | PDF | Video
Learning a Generalizable Trajectory Sampling Distribution for Model Predictive Control
Thomas Power and Dmitry Berenson
IEEE Transactions on Robotics (T-RO), vol. 40, pp. 2111-2127, 2024.
Details | Video
Motion Planning as Online Learning: A Multi-Armed Bandit Approach to Kinodynamic Sampling-Based Planning
Marco Faroni and Dmitry Berenson
IEEE Robotics and Automation Letters (RA-L), vol. 8, no. 10, pp. 6651-6658, October 2023.
Details | PDF | Video
The Blindfolded Traveler's Problem: A Search Framework for Motion Planning with Contact Estimates
Brad Saund, Sanjiban Choudhury, Siddhartha Srinivasa, and Dmitry Berenson
International Journal of Robotics Research (IJRR), Vol. 42, No. 4-5, pp. 289-309, May 2023.
Details | PDF
Soft Tracking Using Contacts for Cluttered Objects to Perform Blind Object Retrieval
Sheng Zhong, Nima Fazeli, and Dmitry Berenson
IEEE Robotics and Automation Letters (RA-L) (Presented at ICRA 2022), vol. 7, no. 2, pp. 3507-3514, April 2022.
Details | PDF | Video
Gaussian Process Constraint Learning for Scalable Chance-Constrained Motion Planning from Demonstrations
Glen Chou*, Hao Wang*, and Dmitry Berenson
IEEE Robotics and Automation Letters (RA-L) (Presented at ICRA 2022), vol. 7, no. 2, pp. 3827-3834, April 2022.
Details | PDF | Video
Challenges and Outlook in Robotic Manipulation of Deformable Objects
Jihong Zhu, Andrea Cherubini, Claire Dune, David Navarro-Alarcon, Farshid Alambeigi, Dmitry Berenson, Fanny Ficuciello, Kensuke Harada, Jens Kober, Xiang Li, Jia Pan, Wenzhen Yuan, and Michael Gienger
IEEE Robotics and Automation Magazine (RAM), vol. 29, no. 3, pp. 67-77, Sept. 2022.
Details | PDF
Learning temporal logic formulas from suboptimal demonstrations: theory and experiments
Glen Chou, Necmiye Ozay, and Dmitry Berenson
Autonomous Robots (AuRo), vol. 46, pp. 149-174, January 2022.
Details | PDF | Video
Learning where to trust unreliable models in an unstructured world for deformable object manipulation
Peter Mitrano, Dale McConachie, and Dmitry Berenson
Science Robotics, vol. 6, no. 54, May 2021.
Details | Video
Long-horizon humanoid navigation planning using traversability estimates and previous experience
Yu-Chi Lin and Dmitry Berenson
Autonomous Robots (AuRo), vol. 45, no. 6, pp. 937-956, June 2021.
Details | PDF | Video
Learning Constraints from Demonstrations with Grid and Parametric Representations
Glen Chou, Dmitry Berenson, and Necmiye Ozay
International Journal of Robotics Research (IJRR), vol. 40, no. 10-11, pp. 1255-1283, September 2021.
Details | PDF
Planning with Learned Dynamics: Probabilistic Guarantees on Safety and Reachability via Lipschitz Constants
Craig Knuth*, Glen Chou*, Necmiye Ozay, and Dmitry Berenson
IEEE Robotics and Automation Letters (RA-L) (Presented at ICRA 2021), vol. 6, no. 3, pp. 5129 - 5136, March 2021.
Details | PDF | Video
Keep it Simple: Data-efficient Learning for Controlling Complex Systems with Simple Models
Thomas Power and Dmitry Berenson
IEEE Robotics and Automation Letters (RA-L) (Presented at ICRA 2021), vol. 6, no. 2, pp. 1184-1191, April 2021.
Details | PDF | Video
TAMPC: A Controller for Escaping Traps in Novel Environments
Sheng Zhong, Zhenyuan Zhang, Nima Fazeli, and Dmitry Berenson
IEEE Robotics and Automation Letters (RA-L) (Presented at ICRA 2021), vol. 6, no. 2, pp. 1447-1454, April 2021.
Details | PDF | Video
Manipulating Deformable Objects by Interleaving Prediction, Planning, and Control
Dale McConachie, Andrew Dobson, Mengyao Ruan, and Dmitry Berenson
International Journal of Robotics Research (IJRR), vol. 39, no. 8, pp. 957-982, July 2020.
Details | Video
Learning When to Trust a Dynamics Model for Planning in Reduced State Spaces
Dale McConachie, Thomas Power, Peter Mitrano, and Dmitry Berenson
IEEE Robotics and Automation Letters (RA-L) (Presented at ICRA 2020), vol. 5, no. 2, pp. 3540-3547, April 2020.
Details | PDF | Video
Learning Constraints from Locally-Optimal Demonstrations under Cost Function Uncertainty
Glen Chou, Necmiye Ozay, and Dmitry Berenson
IEEE Robotics and Automation Letters (RA-L) (presented at ICRA 2020), vol. 5, no. 2, pp. 3682-3690, April 2020.
Details | PDF | Video
Robust Humanoid Contact Planning with Learned Zero- and One-Step Capturability Prediction
Yu-Chi Lin, Ludovic Righetti, and Dmitry Berenson
IEEE Robotics and Automation Letters (RA-L) (Presented at ICRA 2020), vol. 5, no. 2, pp. 2451-2458, April 2020.
Details | PDF | Video
Fast Planning Over Roadmaps via Selective Densification
Brad Saund and Dmitry Berenson
IEEE Robotics and Automation Letters (RA-L) (Presented at ICRA 2020), vol. 5, no. 2, pp. 2873-2880, April 2020.
Details | PDF | Video
Asymptotically Near-Optimal Methods for Kinodynamic Planning with Initial State Uncertainty
Kaiwen Liu, Yang Zhang, Andrew Dobson, and Dmitry Berenson
IEEE Robotics and Automation Letters (RA-L), Vol. 4, No. 2, pp. 2124-2131, April 2019.
Details | PDF
Simultaneous learning of hierarchy and primitives for complex robot tasks
Anahita Mohseni-Kabir, Changshuo Li, Victoria Wu, Daniel Miller, Benjamin Hylak, Sonia Chernova, Dmitry Berenson, Candace Sidner, Charles Rich
Autonomous Robots (AuRo), Vol. 43, No. 4, pp. 859-874, April 2019.
Details
Estimating Model Utility for Deformable Object Manipulation Using Multi-Armed Bandit Methods
Dale McConachie and Dmitry Berenson
IEEE Transactions on Automation Science and Engineering (T-ASE), Vol. 15, No. 3, pp. 967-979, July 2018.
Details
Unsupervised Early Prediction of Human Reaching for Human-robot Collaboration in Shared Workspaces
Ruikun Luo, Rafi Hayne, and Dmitry Berenson
Autonomous Robots (AuRo), Vol. 42, No. 3, pp 631-648, March 2018.
Details | PDF | Video
Analysis and Observations From the First Amazon Picking Challenge
Nikolaus Correll, Kostas E. Bekris, Dmitry Berenson, Oliver Brock, Albert Causo, Kris Hauser, Kei Okada, Alberto Rodriguez, Joseph M. Romano, and Peter R. Wurman
IEEE Transactions on Automation Science and Engineering (T-ASE), Vol. 15, No. 1, pp. 172 - 188, January 2018.
Details
Team WPI-CMU: Achieving Reliable Humanoid Behavior in the DARPA Robotics Challenge
Mathew DeDonato, Felipe Polido, Kevin Knoedler, Benzun P. W. Babu, Nandan Banerjee, Christoper P. Bove, Xiongyi Cui, Ruixiang Du, Perry Franklin, Joshua P. Graff, Peng He, Aaron Jaeger, Lening Li, Dmitry Berenson, Michael A. Gennert, Siyuan Feng, Chenggang Liu, X Xinjilefu, Joohyung Kim, Christopher G. Atkeson, Xianchao Long and Taskin Padir
Journal of Field Robotics (JFR), Vol. 34, No. 2, pp. 381-399, March 2017.
Details
Goal Set Inverse Optimal Control and Iterative Re-planning for Predicting Human Reaching Motions in Shared Workspaces
Jim Mainprice, Rafi Hayne, and Dmitry Berenson
IEEE Transactions on Robotics (T-RO), Vol. 32, No. 4, pp. 897-908, August 2016.
Details | PDF
From Autonomy to Cooperative Traded Control of Humanoid Manipulation Tasks with Unreliable Communication
Calder Phillips-Grafflin, Halit Bener Suay, Jim Mainprice, Nicholas Alunni, Daniel Lofaro, Dmitry Berenson, Sonia Chernova, Robert W. Lindeman, and Paul Oh
Journal of Intelligent and Robotic Systems (JINT), Vol. 82, No. 3, pp. 341-361, June 2016.
Details | PDF
Toward a user-guided manipulation framework for high-DOF robots with limited communication
Calder Phillips-Grafflin, Nicholas Alunni, Halit Bener Suay, Jim Mainprice, Daniel Lofaro, Dmitry Berenson, Sonia Chernova, Robert W. Lindeman, and Paul Oh
Journal of Intelligent Service Robotics, Vol. 7, No. 3, pp. 121-131, July 2014
Details | PDF
Robot-Guided Open-Loop Insertion of Skew-Line Needle Arrangements for High Dose Rate Brachytherapy
Animesh Garg, Timmy Siauw, Dmitry Berenson, J. Adam M. Cunha, I-Chow Hsu, Jean Pouliot, Dan Stoianovici, and Ken Goldberg
IEEE Transactions on Automation Science and Engineering (T-ASE), Vol. 10, No. 4, October 2013
Details | PDF
NPIP: A Skew Line Needle Configuration Optimization System for HDR Brachytherapy
Timmy Siauw, Adam Cunha, Dmitry Berenson, Alper Atamturk, I-Chow Hsu, Ken Goldberg, and Jean Pouliot
Medical Physics, Volume 39, Number 7, pp. 4339 - 4346, July, 2012.
Details | PDF
HERB 2.0: Lessons Learned from Developing a Mobile Manipulator for the Home
Siddhartha S. Srinivasa, Dmitry Berenson, Maya Cakmakz, Alvaro Collet, Mehmet R. Dogar, Anca D. Dragan, Ross A. Knepper, Tim Niemueller, Kyle Strabala, Mike Vande Weghe, Julius Ziegler
Proceedings of the IEEE, Vol. 100, No. 8, pp. 2410 - 2428, July, 2012.
Details | PDF
Task Space Regions: A Framework for Pose-Constrained Manipulation Planning
Dmitry Berenson, Siddhartha Srinivasa, and James Kuffner
International Journal of Robotics Research (IJRR), Volume 30, Number 12, pp. 1435 - 1460, October, 2011.
Details | PDF
HERB: A Home Exploring Robotic Butler
Siddhartha Srinivasa, David Ferguson, Casey Helfrich, Dmitry Berenson, Alvaro Collet Romea, Rosen Diankov, Garratt Gallagher, Geoffrey Hollinger, James Kuffner, and J Michael Vandeweghe
Autonomous Robots (AuRo), Vol. 28, No. 1, pp. 5-20, January, 2010.
Details | PDF

Refereed Conference Papers

Subgoal Diffuser: Coarse-To-Fine Subgoal Generation to Guide Model Predictive Control for Robot Manipulation
Zixuan Huang, Yating Lin, Fan Yang, and Dmitry Berenson
IEEE International Conference on Robotics and Automation (ICRA), May, 2024.
Details | Video
The Grasp Loop Signature: A Topological Representation for Manipulation Planning with Ropes and Cables
Peter Mitrano and Dmitry Berenson
IEEE International Conference on Robotics and Automation (ICRA), May, 2024.
Details | Video
Improving Out-Of-Distribution Generalization of Learned Dynamics by Learning Pseudometrics and Constraint Manifolds
Yating Lin, Glen Chou, and Dmitry Berenson
IEEE International Conference on Robotics and Automation (ICRA), May, 2024.
Details | Video
Online Adaptation of Sampling-Based Motion Planning with Inaccurate Models
Marco Faroni and Dmitry Berenson
IEEE International Conference on Robotics and Automation (ICRA), May, 2024.
Details | Video
TactileVAD: Geometric Aliasing-Aware Dynamics for High-Resolution Tactile Control
Miquel Oller, Dmitry Berenson, and Nima Fazeli
Conference on Robot Learning (CoRL), November 2023.
Details | PDF | Video
CHSEL: Producing Diverse Plausible Pose Estimates from Contact and Free Space Data
Sheng Zhong, Nima Fazeli, and Dmitry Berenson
Robotics: Science and Systems (RSS), July 2023.
Details | PDF | Video
Integrated Object Deformation and Contact Patch Estimation from Visuo-Tactile Feedback
Mark Van der Merwe, Youngsun Wi, Dmitry Berenson, Nima Fazeli
Robotics: Science and Systems (RSS), July 2023.
Details | PDF
Focused Adaptation of Dynamics Models for Deformable Object Manipulation
Peter Mitrano, Alex LaGrassa, Oliver Kroemer, and Dmitry Berenson
IEEE International Conference on Robotics and Automation (ICRA), May, 2023.
Details | PDF | Video
Data-Efficient Learning of Natural Language to Linear Temporal Logic Translators for Robot Task Specification
Jiayi Pan, Glen Chou, and Dmitry Berenson
IEEE International Conference on Robotics and Automation (ICRA), May, 2023.
Details | PDF | Video
Learning the Dynamics of Compliant Tool-Environment Interaction for Visuo-Tactile Contact Servoing
Mark Van der Merwe, Dmitry Berenson, and Nima Fazeli
Conference on Robot Learning (CoRL), December 2022.
Details | Video
Manipulation via Membranes: High-Resolution and Highly Deformable Tactile Sensing and Control
Miquel Oller, Mireia Planas, Dmitry Berenson, and Nima Fazeli
Conference on Robot Learning (CoRL), December 2022.
Details | Video
Data Augmentation for Manipulation
Peter Mitrano and Dmitry Berenson
Robotics Science and Systems (RSS), June 2022.
Details | Video
Variational Inference MPC using Normalizing Flows and Out-of-Distribution Projection
Thomas Power and Dmitry Berenson
Robotics Science and Systems (RSS), June 2022.
Details | Video
Safe Output Feedback Motion Planning from Images via Learned Perception Modules and Contraction Theory
Glen Chou, Necmiye Ozay, and Dmitry Berenson
Workshop on the Algorithmic Foundations of Robotics (WAFR), June 2022.
Details | Video
Model Error Propagation via Learned Contraction Metrics for Safe Feedback Motion Planning of Unknown Systems
Glen Chou, Necmiye Ozay, and Dmitry Berenson
Conference on Decision and Control (CDC), December 2021.
PDF | Video
CLASP: Constrained Latent Shape Projection for Refining Object Shape from Robot Contact
Brad Saund and Dmitry Berenson
Conference on Robot Learning (CoRL), November 2021.
Details | PDF | Video
Tracking Partially-Occluded Deformable Objects While Enforcing Geometric Constraints
Yixuan Wang, Dale McConachie, and Dmitry Berenson
IEEE International Conference on Robotics and Automation (ICRA), May, 2021.
Details | PDF | Video
Fusing RGBD Tracking and Segmentation Tree Sampling for Multi-Hypothesis Volumetric Segmentation
Andrew Price*, Kun Huang*, and Dmitry Berenson
IEEE International Conference on Robotics and Automation (ICRA), May, 2021.
Details | PDF | Video
Diverse Plausible Shape Completions from Ambiguous Depth Images
Brad Saund and Dmitry Berenson
Conference on Robot Learning (CoRL), November 2020.
Details | PDF | Video
Uncertainty-Aware Constraint Learning for Adaptive Safe Motion Planning from Demonstrations
Glen Chou, Necmiye Ozay, and Dmitry Berenson
Conference on Robot Learning (CoRL), November 2020.
Details | PDF | Video
Explaining Multi-stage Tasks by Learning Temporal Logic Formulas from Suboptimal Demonstrations
Glen Chou, Necmiye Ozay, and Dmitry Berenson
Robotics: Science and Systems (RSS), July 2020.
Details | PDF | Video
Inferring Obstacles and Path Validity from Visibility-Constrained Demonstrations
Craig Knuth, Glen Chou, Necmiye Ozay, and Dmitry Berenson
Workshop on the Algorithmic Foundations of Robotics (WAFR), 2020.
Details
Occlusion-robust Deformable Object Tracking without Physics Simulation
Cheng Chi and Dmitry Berenson
IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), November 2019.
Details | PDF | Video
Learning Parametric Constraints in High Dimensions from Demonstrations
Glen Chou, Necmiye Ozay, and Dmitry Berenson
Conference on Robot Learning (CoRL), October, 2019.
Details | PDF
Inferring Occluded Geometry Improves Performance when Retrieving an Object from Dense Clutter
Andrew Price*, Linyi Jin*, and Dmitry Berenson
International Symposium on Robotics Research (ISRR), October 2019.
Details | PDF | Video
The Blindfolded Robot: A Bayesian Approach to Planning with Contact Feedback
Brad Saund, Sanjiban Choudhury, Siddhartha Srinivasa, and Dmitry Berenson
International Symposium on Robotics Research (ISRR), October, 2019.
PDF | Video
Efficient Humanoid Contact Planning using Learned Centroidal Dynamics Prediction
Yu-Chi Lin, Brahayam Ponton, Ludovic Righetti, and Dmitry Berenson
IEEE International Conference on Robotics and Automation (ICRA), May, 2019.
PDF | Video
Learning Constraints from Demonstrations
Glen Chou, Dmitry Berenson, Necmiye Ozay
Workshop on the Algorithmic Foundations of Robotics (WAFR), December, 2018.
PDF
Motion Planning for Manipulators in Unknown Environments with Contact Sensing Uncertainty
Brad Saund and Dmitry Berenson
International Symposium on Experimental Robotics (ISER), November 2018.
PDF | Video
Accounting for Directional Rigidity and Constraints in Control for Manipulation of Deformable Objects without Physical Simulation
Mengyao Ruan, Dale McConachie, and Dmitry Berenson
IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), October, 2018.
PDF | Video
Humanoid Navigation Planning in Large Unstructured Environments Using Traversability-Based Segmentation
Yu-Chi Lin and Dmitry Berenson
IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), October, 2018
PDF | Video
Incremental Segmentation of ARX Models
Glen Chou, Necmiye Ozay, and Dmitry Berenson
IFAC Symposium on System Identification (SYSID), July, 2018.
PDF
Interleaving Planning and Control for Deformable Object Manipulation
Dale McConachie, Mengyao Ruan, and Dmitry Berenson
International Symposium on Robotics Research (ISRR), December, 2017.
PDF | Video
Humanoid Navigation in Uneven Terrain using Learned Estimates of Traversability
Yu-Chi Lin and Dmitry Berenson
IEEE-RAS International Conference on Humanoid Robots (Humanoids), November, 2017.
PDF | Video
A Framework For Robot-Assisted Doffing of Personal Protective Equipment
Antonio Umali and Dmitry Berenson
IEEE International Conference on Robotics and Automation (ICRA), May, 2017.
PDF | Video
Bandit-Based Model Selection for Deformable Object Manipulation
Dale McConachie and Dmitry Berenson
Workshop on the Algorithmic Foundations of Robotics (WAFR), December, 2016.
PDF | Video
Planning and Resilient Execution of Policies for Manipulation in Contact with Actuation Uncertainty
Calder Phillips-Grafflin and Dmitry Berenson
Workshop on the Algorithmic Foundations of Robotics (WAFR), December, 2016.
PDF | Video
Using Previous Experience for Humanoid Navigation Planning
Yu-Chi Lin and Dmitry Berenson
IEEE-RAS International Conference on Humanoid Robots (Humanoids), November, 2016.
PDF | Video
Integrated Affordance Detection and Humanoid Locomotion Planning
Will Pryor, Yu-Chi Lin, and Dmitry Berenson
IEEE-RAS International Conference on Humanoid Robots (Humanoids), November, 2016.
PDF | Video
Learning Object Orientation Constraints and Guiding Constraints for Narrow Passages from One Demonstration
Changshuo Li and Dmitry Berenson
International Symposium on Experimental Robotics (ISER), October, 2016.
PDF | Video
Improving Soft Pneumatic Actuator Fingers through Integration of Soft Sensors, Position and Force Control, and Rigid Fingernails
John Morrow, Hee-Sup Shin, Calder Phillips-Grafflin, Sung-Hwan Jang, Jacob Torrey, Riley Larkins, Steven Dang, Yong-Lae Park, and Dmitry Berenson
IEEE International Conference on Robotics and Automation (ICRA), May, 2016.
PDF | Video
Considering Avoidance and Consistency in Motion Planning for Human-Robot Manipulation in a Shared Workspace
Rafi Hayne, Ruikun Luo, and Dmitry Berenson
IEEE International Conference on Robotics and Automation (ICRA), May, 2016.
PDF | Video
Reproducing Expert-Like Motion in Deformable Environments Using Active Learning and IOC
Calder Phillips-Grafflin and Dmitry Berenson
International Symposium on Robotics Research (ISRR), September, 2015.
PDF
A Framework for Unsupervised Online Human Reaching Motion Recognition and Early Prediction
Ruikun Luo and Dmitry Berenson
IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), September, 2015
PDF
Predicting Human Reaching Motion in Collaborative Tasks Using Inverse Optimal Control and Iterative Re-planning
Jim Mainprice, Rafi Hayne, and Dmitry Berenson
IEEE International Conference on Robotics and Automation (ICRA), May, 2015.
PDF | Video
An Online Method for Tight-tolerance Insertion Tasks for String and Rope
Weifu Wang, Dmitry Berenson, and Devin Balkcom
IEEE International Conference on Robotics and Automation (ICRA), May, 2015.
PDF | Video
From Autonomy to Cooperative Traded Control of Humanoid Manipulation Tasks with Unreliable Communication: System Design and Lessons Learned
Jim Mainprice, Calder Phillips-Grafflin, Halit Bener Suay, Nicholas Alunni, Daniel Lofaro, Dmitry Berenson, Sonia Chernova, Robert W. Lindeman, and Paul Oh
IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), November, 2014
PDF | Video
A Representation Of Deformable Objects For Motion Planning With No Physical Simulation
Calder Phillips-Grafflin and Dmitry Berenson
IEEE International Conference on Robotics and Automation (ICRA), May, 2014.
PDF | Video
Human-Robot Collaborative Manipulation Planning Using Early Prediction of Human Motion
Jim Mainprice and Dmitry Berenson
IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), November, 2013
PDF | Video
Manipulation of Deformable Objects Without Modeling and Simulating Deformation
Dmitry Berenson
IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), November, 2013
PDF | Video
Toward A User-Guided Manipulation Framework for High-DOF Robots with Limited Communication
Nicholas Alunni, Calder Phillips-Grafflin, Halit Bener Suay, Daniel Lofaro, Dmitry Berenson, Sonia Chernova, Robert W. Lindeman, Paul Oh
IEEE International Conference on Technologies for Practical Robot Applications (TePRA), April, 2013
PDF | Video
Initial Experiments toward Automated Robotic Implantation of Skew-Line Needle Arrangements for HDR Brachytherapy
Animesh Garg, Timmy Siauw, Dmitry Berenson, Adam Cunha, I-Chow Hsu, Jean Pouliot, Dan Stoianovici, Ken Goldberg
IEEE International Conference on Automation Science and Engineering (CASE), August, 2012. Best Application Paper Award
PDF
Estimating Part Tolerance Bounds Based on Adaptive Cloud-Based Grasp Planning with Slip
Ben Kehoe, Dmitry Berenson, and Ken Goldberg
IEEE International Conference on Automation Science and Engineering (CASE), August, 2012.
PDF
A Constraint-Aware Motion Planning Algorithm for Robotic Folding of Clothes
Karthik Lakshmanan, Apoorva Sachdev, Ziang Xie, Dmitry Berenson, Ken Goldberg, and Pieter Abbeel
International Symposium on Experiment Robotics (ISER), June, 2012.
PDF | Video
A Robot Path Planning Framework that Learns from Experience
Dmitry Berenson, Pieter Abbeel, and Ken Goldberg
IEEE International Conference on Robotics and Automation (ICRA), May, 2012.
Details | PDF | Video
Constellation - An Algorithm for Finding Robot Configurations that Satisfy Multiple Constraints
Peter Kaiser, Dmitry Berenson, Nikolaus Vahrenkamp, Tamim Asfour, Rudiger Dillmann, Siddhartha Srinivasa
IEEE International Conference on Robotics and Automation (ICRA), May, 2012.
PDF
Toward Cloud-Based Grasping with Uncertainty in Shape: Estimating Lower Bounds on Achieving Force Closure with Zero-Slip Push Grasps
Ben Kehoe, Dmitry Berenson, and Ken Goldberg
IEEE International Conference on Robotics and Automation (ICRA), May, 2012.
PDF
Addressing Cost-Space Chasms for Manipulation Planning
Dmitry Berenson, Thierry Simeon, Siddhartha Srinivasa
IEEE International Conference on Robotics and Automation (ICRA), May, 2011.
Details | PDF | Video
People Helping Robots Helping People: Crowdsourcing for Grasping Novel Objects
Alexander Sorokin, Dmitry Berenson, Siddhartha Srinivasa, and Martial Hebert
IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), October, 2010
Details | PDF | Video
Probabilistically Complete Planning with End-Effector Pose Constraints
Dmitry Berenson and Siddhartha Srinivasa
IEEE International Conference on Robotics and Automation (ICRA), May, 2010
Details | PDF
Pose-Constrained Whole-Body Planning using Task Space Region Chains
Dmitry Berenson, Joel Chestnutt, Siddhartha Srinivasa, James Kuffner, and Satoshi Kagami
IEEE-RAS International Conference on Humanoid Robots (Humanoids), December, 2009
Details | PDF | Video
Addressing Pose Uncertainty in Manipulation Planning Using Task Space Regions
Dmitry Berenson, Siddhartha Srinivasa, and James Kuffner
IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), October, 2009
Details | PDF
Humanoid Motion Planning for Dual-Arm Manipulation and Re-Grasping Tasks
Nikolaus Vahrenkamp, Dmitry Berenson, Tamim Asfour, James Kuffner, and Rudiger Dillmann
IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), October, 2009
Details | PDF
Manipulation Planning on Constraint Manifolds
Dmitry Berenson, Siddhartha Srinivasa, David Ferguson, and James Kuffner
IEEE International Conference on Robotics and Automation (ICRA), May, 2009
Details | PDF | Video
Manipulation Planning with Workspace Goal Regions
Dmitry Berenson, Siddhartha Srinivasa, David Ferguson, Alvaro Collet Romea, and James Kuffner
IEEE International Conference on Robotics and Automation (ICRA), May, 2009
Details | PDF
Object Recognition and Full Pose Registration from a Single Image for Robotic Manipulation
Alvaro Collet Romea, Dmitry Berenson, Siddhartha Srinivasa, and David Ferguson
IEEE International Conference on Robotics and Automation (ICRA), May, 2009
Details | PDF
Grasp Synthesis in Cluttered Environments for Dexterous Hands
Dmitry Berenson and Siddhartha Srinivasa
IEEE-RAS International Conference on Humanoid Robots (Humanoids), December, 2008
Details | PDF | Video
The Robotic Busboy: Steps Towards Developing a Mobile Robotic Home Assistant
Siddhartha Srinivasa, David Ferguson, J Michael Vandeweghe, Rosen Diankov, Dmitry Berenson, Casey Helfrich, and Hauke Strasdat
International Conference on Intelligent Autonomous Systems (IAS), July, 2008
Details | PDF
An Optimization Approach to Planning for Mobile Manipulation
Dmitry Berenson, Howie Choset, and James Kuffner
IEEE International Conference on Robotics and Automation (ICRA), May, 2008
Details | PDF
Grasp Planning in Complex Scenes
Dmitry Berenson, Rosen Diankov, Koichi Nishiwaki, Satoshi Kagami, and James Kuffner
IEEE-RAS International Conference on Humanoid Robots (Humanoids), December, 2007
Details | PDF | Video
Hardware Evolution of Analog Circuits for In-situ Robotic Fault-Recovery
Dmitry Berenson, Nicholas Esteves, and Hod Lipson
NASA/DoD Conference on Evolvable Hardware, June, 2005
Details | PDF | Video

Refereed Demonstrations and Videos

A Demonstration of Planar Dragging of a Hose with Obstacles
Peter Mitrano, Alison Ryckman, and Dmitry Berenson
IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) Demonstration Sessions, October 2023.
PDF | Video
SLHAP: Simultaneous Learning of Hierarchy and Primitives
Anahita Mohseni-Kabir, Changshuo Li, Victoria Wu, Daniel Miller, Benjamin Hylak, Sonia Chernova, Dmitry Berenson, Candace Sidner, and Charles Rich
ACM/IEEE International Conference on Human-Robot Interaction (HRI) Videos, March, 2017.
PDF | Video
DARPA Robotics Challenge: Towards a User-Guided Manipulation Framework for High-DOF Robots
Nicholas Alunni, Halit Bener Suay, Calder Phillips-Grafflin, Jim Mainprice, Dmitry Berenson, Sonia Chernova, Robert W. Lindeman, Daniel Lofaro, and Paul Oh
IEEE International Conference on Robotics and Automation (ICRA) Video Proceedings, May, 2014.
PDF | Video

Refereed Workshop Papers

Improving Path Execution in Deformable Environments Using Reactive Cost-space Control
Calder Phillips-Grafflin and Dmitry Berenson
IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) Workshop on Robot Manipulation: What Has Been Achieved and What Remains to Be Done?, Chicago, September 2014.
PDF
Using Task Symmetry for Human-Robot Collaborative Manipulation of Deformable Objects Without Modeling Deformation
Dmitry Berenson
EEE/RSJ International Conference on Intelligent Robots and Systems (IROS) Workshop on Cognitive Surgical Robotics: From Virtual Fixtures to Advanced Cooperative Control, November, 2013. Best Poster Award
PDF
Grasp Synthesis in Cluttered Environments for Dexterous Hands
Dmitry Berenson and Siddhartha Srinivasa
Robotics Science and Systems (RSS) Workshop on Robot Manipulation: Intelligence in Human Environments, June, 2008
Details | PDF

Theses

Contact-based Perception and Planning for Robotic Manipulation in Novel Environments
Sheng Zhong
Ph.D. dissertation, Robotics Department, University of Michigan, July 2024.
PDF
Learning and Inference for Adaptable Manipulation Planning
Thomas Power
Ph.D. dissertation, Robotics Department, University of Michigan, June 2024.
PDF
Data-Efficient Robotic Manipulation of Deformable One-dimensional Objects with Unreliable Dynamics
Peter Mitrano
Ph.D. dissertation, Robotics Department, University of Michigan, December 2023.
PDF
Safe End-to-end Learning-based Robot Autonomy via Integrated Perception, Planning, and Control
Glen Chou
Ph.D. dissertation, EECS Department, University of Michigan, August 2022.
Details | PDF
Belief Representations for Planning with Contact Uncertainty
Brad Saund
Ph.D. dissertation, Robotics Institute, University of Michigan, July 2021.
PDF
Deformable Object Manipulation: Learning While Doing
Dale McConachie
Ph.D. dissertation, Robotics Institute, University of Michigan, April 2020.
PDF
Learning for Humanoid Multi-Contact Navigation Planning
Yu-Chi Lin
Ph.D. dissertation, Robotics Institute, University of Michigan, April 2020.
PDF
Enabling Motion Planning and Execution for Tasks Involving Deformation and Uncertainty
Calder Phillips-Grafflin
Ph.D. dissertation, Robotics Engineering, Worcester Polytechnic Institute, June, 2017.
PDF
Constrained Manipulation Planning
Dmitry Berenson
Ph.D. dissertation, Robotics Institute, School of Computer Science, Carnegie Mellon University, June, 2011.
Details | PDF

Visit us

We are happy to arrange visits to the lab for K-12 student groups during the school year or summer. Please contact Prof. Dmitry Berenson (dmitryb [at] umich.edu) to arrange a visit.

Visit the ARM Lab on facebook!

Visit our YouTube channel for the latest videos.

Visit our twitter page @umicharmlab for the latest news.

Where is the ARM lab?

Our lab is located at 2140 Ford Motor Company Robotics Building (FMCRB) on North Campus, University of Michigan, Ann Arbor, MI. Please see this map for the location of the building.

See photos from recent outreach events here.

PyTorch Robot Kinematics

Parallel and differentiable forward kinematics (FK) and Jacobian calculation

Expand

A kinematics library that uses PyTorch to compute forward kinematics for multiple configurations in parallel. The computation is differentiable and the library loads robot descriptions from the URDF, SDF, and MJCF formats.

https://github.com/UM-ARM-Lab/pytorch_kinematics

PyTorch MPPI Implementation

This repository implements Model Predictive Path Integral (MPPI) with approximate dynamics in pytorch

Expand

This repository implements Model Predictive Path Integral (MPPI) with approximate dynamics in pytorch.

https://github.com/UM-ARM-Lab/pytorch_mppi

Constrained Deformable Coherent Point Drift (CDCPD)

An occlusion-robust deformable object tracker

Expand

The latest version of our deformable object tracker, CDCPD2, is now available here:

https://github.com/UM-ARM-Lab/cdcpd/tree/CDCPD2

it implements the method from this paper:
Tracking Partially-Occluded Deformable Objects while Enforcing Geometric Constraints
Yixuan Wang, Dale McConachie, and Dmitry Berenson
IEEE International Conference on Robotics and Automation (ICRA), May, 2021.
Details | PDF | Video

The original CDCPD is an implementation of

Occlusion-robust Deformable Object Tracking without Physics Simulation
Cheng Chi and Dmitry Berenson
IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), November 2019.

This library includes object-oriented API for tracker, visualization utilities and a ROS node that subscribes and publishes PointCloud2.

https://github.com/UM-ARM-Lab/cdcpd

Traversability-Based Contact-Space Planner

A planner that uses a learned estimate of traversability to find contact sequences for humanoids

Expand

Given an environment specified as a set of polygonal surface and a robot model, the code generates contact sequences for a humanoid robot using both arms and legs. We provide an example with the Escher humanoid robot model. The code is written in Python 2.7 and tested in Ubuntu 14.04 with ROS Indigo

https://github.com/UM-ARM-Lab/Traversability-Based-Contact-Space-Planner

CAD Files

Our open-source CAD repository.

Expand

Our open source CAD repository is here: https://github.com/UM-ARM-Lab/CAD/

ARM Lab Github Page

The latest, most bleeding-edge software developed in the lab.

Expand

Visit our Github page to get the latest code developed in the lab. Watch out, it's a jungle in there!

Datalink Toolkit

A ROS package designed for remote operation of a robot over a high-latency and low-bandwidth datalink

Expand

The Datalink Toolkit is a ROS package designed for remote operation of a robot over a high-latency and low-bandwidth datalink. The package allows the user to easily set up relays and compression methods for a single-master system. These relays avoid duplicating data sent over the datalink while compressing common datatypes (i.e. point-clouds and images) to minimize bandwidth usage. The package was developed and extensively tested as part of the DARPA robotics challenge, though it is not specific to a type of robot.

The package allows the user to easily set up relays and compression methods for a single-master system. These relays avoid duplicating data sent over the datalink while compressing common datatypes (i.e. point-clouds and images) to minimize bandwidth usage.

The toolkit includes both message-based and service-based relays so that data can be sent on-demand or at a specified frequency. The service-based relays are more robust in low-bandwidth conditions, guaranteeing the synchronization of camera images and camera info messages, and allow more reconfiguration while running.

The key features of the package are:

Generic relays with integrated rate throttling for all message types
Dedicated relays with rate throttling for images and pointclouds
Generic service-based relays with integrated rate throttling for all message types
Dedicated service-based relays with integrated rate throttling for images and pointclouds
Image resizing and compression using methods from OpenCV and image_transport
Pointcloud voxel filtering and compression using methods from PCL, Zlib, and other algorithms. (Note: pointcloud compression is provided in a separate library that can be easily integrated with other projects)
Launch files for easy use of the datalink software with RGBD cameras
Works with ROS Hydro

For more information, please see the wiki.

Get the package from our git repository.

Constrained Manipulation Planning Suite (CoMPS)

Expand

The Constrained Manipulation Planning Suite (CoMPS) consists of three openrave plugins and associated data files. The planning and inverse kinematics algorithms in this suite are designed for articulated robots like robotic arms and humanoids.

CoMPS is implemented in C++ and compiles in linux only. There are also several examples in python and matlab that show how to interface with openrave to use the plugins in CoMPS.

This package is available on SourceForge.

LightningROS

Expand

LightningROS is a ROS package implementing the Lightning Path Planning Framework. This approach uses a path library to store previous experience while allowing generality by also planning from scratch. Please see this paper for more details.

This package uses OMPL planners to implement each component in Lightning and can be called the same way as any other OMPL planner.

This package is available for ROS Feurte here. For later versions of ROS, see here.

Github Page for Projects at WPI

Contains the archive of projects at WPI from 2012 to 2016.

Expand

Github archive of projects at WPI 2012-2016.

Autonomous Robotic Manipulation Lab

University of Michigan

People

Welcome

Recent papers:

Research Interests

Press

Faculty

PhD Students

Undergraduate Students

Alumni

Visit us

Where is the ARM lab?

© 2023 Dmitry Berenson. All Rights Reserved.