Bilateral shared control of mobile robots

We address the problem of the interaction (see below) between humans and groups of robots whose local synergy is exploited to accomplish complex tasks. Multi-robot systems possess several advantages w.r.t. single robots, e.g., higher performance in simultaneous spatial domain coverage, better affordability as compared to a single/bulky system, robustness against single point failures.

As multi-robot platform, we considered the case of a group of Unmanned Aerial Vehicles (UAVs), because of their high motion flexibility and potential pervasivity in dangerous or unaccessible locations. We envision a scenario where the UAVs possess some level of local autonomy and act as a group, e.g., by maintaining a desired formation, by avoiding obstacles, and by performing additional local tasks. At the same time, the remote human operator is in control of the overall UAV motion and receives, through haptic feedback, suitable cues informative enough of the remote UAV/environment state. We addressed two distinct possibilities for the human/multi-robot teleoperation: a top-down approach, and a bottom-up approach, mainly differing in the way the local robot interactions and desired formation shape are treated.

Fixed-topology approach

In [1] the N UAVs are abstracted as 3-DOF first-order kinematic VPs (virtual points): the remote human user teleoperates a subset of these N VPs, while the real UAV's position tracks the trajectory of its own VP. The VPs collectively move as an N-nodes deformable flying object, whose shape (chosen beforehand) autonomously deforms, rotates and translates reacting to the presence of obstacles (to avoid them), and the operator commands (to follow them). The operator receives a haptic feedback informing him about the motion state of the real UAVs, and about the presence of obstacles via their collective effects on the VPs. Passivity theory is exploited to prove stability of the overall teleoperation system. A video showing the top-down approach is here.

In [2],[3] the approach has been extended to the case where bearing measurements are the only available. The main focus in that case have been the design of a novel decentralized and minimum-complexity bearing-only formation controller for the slave side.

In [4] we describe an experimental testbed for the intercontinental teleoperation of multiple UAVs and we demonstrate its feasibility employing a communication channel going from Germany to South Korea. 

An updated and comprehensive collection of videos is available at the video page

top-down_cube

Fig. 1 An example of 8 UAVs arranged in a cubic shape controlled with an haptic device.

Time-varying topology approach

In [5]  the N UAVs are abstracted as 3-DOF second-order VPs: the remote human user teleoperates a single leader, while the remaining followers motion is determined by local interactions (modeled as spring/damper couplings) among themselves and the leader, and repulsive interactions with the obstacles. The overall formation shape is not chosen beforehand but is a result of the UAVs motion. Split and rejoin decisions are allowed depending on any criterion, e.g., the UAVs relative distance and their relative visibility (i.e., when two UAVs are not close enough or obstructed by an obstacle, they split their visco-elastic coupling). The operator receives a haptic feedback informing him about the motion state of the leader which is also influenced by the motion of its followers and their interaction with the obstacles. Passivity theory is exploited to prove stability of the overall teleoperation system. The first journal paper about the time-varying topology approach is [6].

In [7] the approach has been extended by explicitly consider the presence of time delays, both among the haptic device and the multi-robot system, and within the robots composing the multi-robot system.

In  [8] we present a decentralized passivity-based control strategy for the bilateral teleoperation of a fleet of Unmanned Aerial Vehicles (UAVs). The human operator at the master side can command the fleet motion and receive suitable force cues informative about the remote environment. By properly controlling the energy exchanged within the slave side (the UAV fleet), we guarantee that the connectivity of the fleet is preserved and we prevent inter-agent and obstacle collisions. At the same time, we allow the behavior of the UAVs to be as flexible as possible with arbitrary split and join maneuvers. The results of the paper are validated through semi-experiments.

An experimental validation of the bottom-up approach has been presented in [9], while in [10] an extension of the approach where the group autonomously changes the leader in order to maximize the performances is presented.

Watch the video page to have an updated and comprehensive collection of videos about this topic.

bott_up

Fig. 2 The haptic interface used to control the motion of the leader plus  8 follower UAVs in a cluttered environment. The connectivity graph is shown with blue lines and the leader UAV is surrounded by a transparent red sphere.

Psychophysical Evaluation

A psychophysical evaluation study in the bilateral teleoperation of multiple mobile robots have been conducted in [11] and [12].

The perceptual awareness of the operator is investigated in [11], while in [12] the maneuverability metric is studied.

Human-robot interaction

Human-robot interaction is a very active research area which spans a big variety of topics. A nonexhaustive list includes robot mechanical design and low-level control, higher-level control and planning, learning approaches, cognitive and/or physical interaction, and human intention/emotion interpretation. These efforts are guided by the accepted vision that in the future humans and robots will seamlessly cooperate in shared or remote spaces, thus becoming an integral part of our daily life. For instance, robots are expected to relieve us from monotonous and physically demanding work in industrial settings, or help humans in dealing with complex/dangerous tasks, thus augmenting their capabilities.

  • Coexisting interaction: the robots share the environment with humans not directly involved in their task. Examples are: housekeeping, city cleaning, or navigation in crowded areas. In these scenarios, the robots should minimize the interferences with any human activity, but also be prepared to successfully solve unexpected conflicts with humans;
  • Conditional interaction: the robots need to be guided/helped by expert human operators in tasks which are either too sensitive or hard to be solved given the existing ethical/technological situation. Examples are: cooperation with a surgeon in medical operations, or with firefighters in search and rescue tasks. In these scenarios, the robots should perfectly complete the human skills/roles in order to maximize the probability of task achievement as in, e.g., telepresence applications where the human operator capabilities are mediated/magnified by the remote robots;
  • Essential interaction: the robots are asked to accomplish tasks in which the humans take a passive, but essential, role. Examples are: actively assisting patients in a medical operation, or passengers during their transportation. In these scenarios, the robots should be able to interpret the actions of humans who are assumed as being not specifically trained in interacting with robots.

In all interaction cases (which may also happen simultaneously), it is interesting to study what is the best level of autonomy expected in the robots, and what is the best sensory feedback needed by the humans to take an effective role in the interaction.

Additional Work

Additional work on this topic has been published in:
[13] (Fixed topology approach), [14] and [15] (integral haptic feedback), [16] (the flying hand), [17] (connectivity maintenance with haptic feedback), [18] and [19] (haptic teleoperation with fully-autonomous onboard-only measurements), [20] (summary framework on shared control of multiple UAVs), [21] (bilateral teleoperation of aerial robots in contact with the enviornment).

Credits

Some parts of this page are an excerpt of [22].


References

  1. Lee D, Franchi A, Robuffo Giordano P, Son HI, Bülthoff HH. Haptic Teleoperation of Multiple Unmanned Aerial Vehicles over the Internet. In: 2011 IEEE Int. Conf. on Robotics and Automation. 2011 IEEE Int. Conf. on Robotics and Automation. Shanghai, China; 2011. pp. 1341-1347.
  2. Franchi A, Masone C, Bülthoff HH, Robuffo Giordano P. Bilateral Teleoperation of Multiple UAVs with Decentralized Bearing-only Formation Control. In: 2011 IEEE/RSJ Int. Conf. on Intelligent Robots and Systems. 2011 IEEE/RSJ Int. Conf. on Intelligent Robots and Systems. San Francisco, CA; 2011. pp. 2215-2222.
  3. Franchi A, Masone C, Grabe V, Ryll M, Bülthoff HH, Robuffo Giordano P. Modeling and Control of UAV Bearing-Formations with Bilateral High-Level Steering. The International Journal of Robotics Research, Special Issue on 3D Exploration, Mapping, and Surveillance. 2012 ;31(12):1504-1525.
  4. Riedel M, Franchi A, Bülthoff HH, Robuffo Giordano P, Son HI. Experiments on Intercontinental Haptic Control of Multiple UAVs. In: 12th Int. Conf. on Intelligent Autonomous Systems. 12th Int. Conf. on Intelligent Autonomous Systems. Jeju Island, Korea; 2012. pp. 227-238.
  5. Franchi A, Robuffo Giordano P, Secchi C, Son HI, Bülthoff HH. A Passivity-Based Decentralized Approach for the Bilateral Teleoperation of a Group of UAVs with Switching Topology. In: 2011 IEEE Int. Conf. on Robotics and Automation. 2011 IEEE Int. Conf. on Robotics and Automation. Shanghai, China; 2011. pp. 898-905.
  6. Franchi A, Secchi C, Son HI, Bülthoff HH, Robuffo Giordano P. Bilateral Teleoperation of Groups of Mobile Robots with Time-Varying Topology. IEEE Transaction on Robotics. 2012 ;28(5):1019 -1033.
  7. Secchi C, Franchi A, Bülthoff HH, Robuffo Giordano P. Bilateral Teleoperation of a Group of UAVs with Communication Delays and Switching Topology. In: 2012 IEEE Int. Conf. on Robotics and Automation. 2012 IEEE Int. Conf. on Robotics and Automation. St. Paul, MN; 2012.
  8. Robuffo Giordano P, Franchi A, Secchi C, Bülthoff HH. Bilateral Teleoperation of Groups of UAVs with Decentralized Connectivity Maintenance. In: 2011 Robotics: Science and Systems Conference. 2011 Robotics: Science and Systems Conference. Los Angeles, CA; 2011.
  9. Robuffo Giordano P, Franchi A, Secchi C, Bülthoff HH. Experiments of Passivity-Based Bilateral Aerial Teleoperation of a Group of UAVs with Decentralized Velocity Synchronization. In: 2011 IEEE/RSJ Int. Conf. on Intelligent Robots and Systems. 2011 IEEE/RSJ Int. Conf. on Intelligent Robots and Systems. San Francisco, CA; 2011. pp. 163-170.
  10. Franchi A, Bülthoff HH, Robuffo Giordano P. Distributed Online Leader Selection in the Bilateral Teleoperation of Multiple UAVs. In: 50th IEEE Conference on Decision and Control . 50th IEEE Conference on Decision and Control . Orlando, FL; 2011. pp. 3559-3565.
  11. Son HI, Kim J, Chuang LL, Franchi A, Robuffo Giordano P, Lee D, Bülthoff HH. An Evaluation of Haptic Cues on the Tele-Operator’s Perceptual Awareness of Multiple UAVs’ Environments. In: IEEE – World Haptics Conference. IEEE – World Haptics Conference. Istanbul, Turkey; 2011. pp. 149-154.
  12. Son HI, Chuang LL, Franchi A, Kim J, Lee D, Lee S-W, Bülthoff HH, Robuffo Giordano P. Measuring an Operator's Maneuverability Performance in the Haptic Teleoperation of Multiple Robots. In: 2011 IEEE/RSJ Int. Conf. on Intelligent Robots and Systems. 2011 IEEE/RSJ Int. Conf. on Intelligent Robots and Systems. San Francisco, CA; 2011. pp. 3039-3046.
  13. Lee D, Franchi A, Son HI, Bülthoff HH, Robuffo Giordano P. Semi-Autonomous Haptic Teleoperation Control Architecture of Multiple Unmanned Aerial Vehicles. IEEE/ASME Transaction on Mechatronics, Focused Section on Aerospace Mechatronics. 2013 ;18(4):1334-1345.
  14. Masone C, Franchi A, Bülthoff HH, Robuffo Giordano P. Interactive Planning of Persistent Trajectories for Human-Assisted Navigation of Mobile Robots. In: 2012 IEEE/RSJ Int. Conf. on Intelligent Robots and Systems. 2012 IEEE/RSJ Int. Conf. on Intelligent Robots and Systems. Vilamoura, Portugal; 2012. pp. 2641-2648.
  15. Masone C, Robuffo Giordano P, Bülthoff HH, Franchi A. Semi-autonomous Trajectory Generation for Mobile Robots with Integral Haptic Shared Control. In: 2014 IEEE Int. Conf. on Robotics and Automation. 2014 IEEE Int. Conf. on Robotics and Automation. Hong Kong, China; 2014. pp. 6468-6475.
  16. Gioioso G, Franchi A, Salvietti G, Scheggi S, Prattichizzo D. The Flying Hand: a Formation of UAVs for Cooperative Aerial Tele-Manipulation. In: 2014 IEEE Int. Conf. on Robotics and Automation. 2014 IEEE Int. Conf. on Robotics and Automation. Hong Kong, China; 2014. pp. 4335-4341.
  17. Robuffo Giordano P, Franchi A, Secchi C, Bülthoff HH. A Passivity-Based Decentralized Strategy for Generalized Connectivity Maintenance. The International Journal of Robotics Research. 2013 ;32(3): 299-323.
  18. Stegagno P, Basile M, Bülthoff HH, Franchi A. A Semi-autonomous UAV Platform for Indoor Remote Operation with Visual and Haptic Feedback. In: 2014 IEEE Int. Conf. on Robotics and Automation. 2014 IEEE Int. Conf. on Robotics and Automation. Hong Kong, China; 2014. pp. 3862-3869.
  19. Stegagno P, Basile M, Bülthoff HH, Franchi A. RGB-D based Haptic Teleoperation of UAVs with Onboard Sensors: Development and Preliminary Results. In: 2013 IROS Work. on Vision-based Closed-Loop Control and Navigation of Micro Helicopters in GPS-denied Environments. 2013 IROS Work. on Vision-based Closed-Loop Control and Navigation of Micro Helicopters in GPS-denied Environments. Tokyo, Japan; 2013.
  20. Franchi A, Secchi C, Ryll M, Bülthoff HH, Robuffo Giordano P. Shared Control: Balancing Autonomy and Human Assistance with a Group of Quadrotor UAVs. IEEE Robotics and Automation Magazine, Special Issue on Aerial Robotics and the Quadrotor Platform. 2012 ;19(3):57-68.
  21. Gioioso G, Mohammadi M, Franchi A, Prattichizzo D. A Force-based Bilateral Teleoperation Framework for Aerial Robots in Contact with the Environment. In: 2015 IEEE Int. Conf. on Robotics and Automation. 2015 IEEE Int. Conf. on Robotics and Automation. Seattle, WA; 2015. pp. 318-324.
  22. Robuffo Giordano P, Franchi A, Son HI, Secchi C, Lee D, Bülthoff HH. Towards Bilateral Teleoperation of Multi-Robot Systems. 3rd Int. Work. on Human-Friendly Robotics. 2010 .

Research Topics: