5 Implementation and Experiments

5.1 Equipment

The system was implemented on a Sun SPARCstation 10 with a Data Cell S2200 frame grabber. The manipulator is a Scorbot ER-7 robot arm, which has 5 degrees of freedom and a parallel-jawed gripper. The robot has its own 68000-based controller which implements the low-level control loop and provides a Cartesian kinematic model. Images are obtained from two inexpensive CCD cameras placed 1m-3m from the robot's workspace. The angle between the cameras is in the range of 15-30 degrees (figure 4).

Figure 4: The experimental setup. Uncalibrated stereo cameras viewing a robot gripper and target object.

5.2 Implementation

When the system is started up, it begins by opening and closing the jaws of the gripper. By observing the image difference, it is able to locate the gripper and set up a pair of affine trackers as instances of a hand-made 2D template. The trackers will then follow the gripper's movements continuously. Stereo tracking can be implemented on the Sun at over 10 Hz. The robot moves to four preset points to calibrate the system in terms of the controller's coordinate space.

A target object is found by similar means - observing the image changes when it is placed in the manipulator's workspace. Alternatively it may be selected from a monitor screen using the mouse. There is no pre-defined model of the target shape, so a pair of `expanding' B-spline snakes [2] are used to locate the contours delimiting the target surface in each of the images. The snakes are then converted to a pair of affine trackers. The target surface is then tracked, to compensate for unexpected motions of either the target or the two cameras.

By introducing modifications and offsets to the feedback mechanism (which would otherwise try to superimpose the gripper and the target), two `behaviours' have been implemented. The tracking behaviour causes it to follow the target continuously, hovering a few centimetres above it (figure 5). The grasping behaviour causes the gripper to approach the target from above (to avoid collisions) with the gripper turned through an angle of 90 degrees, to grasp it normal to its visible surface (figure 6).

Figure 5: The robot is tracking its quarry, guided by the position and orientation of the target contour (view through left camera). On the target surface is an affine snake - an affine tracker obtained by `expanding' a B-spline snake from the centre of the object. A slight offset has been introduced into the control loop to cause the gripper to hover above it. Last frame: one of the cameras has been rotated and zoomed, but the system continues to operate successfully with visual feedback.

Figure 6: Affine stereo and visual feedback used to grasp a planar face.

5.3 Results

Without feedback control, the robot locates its target only approximately (typically to within 5cm in a 50cm workspace) reflecting the approximate nature of affine stereo and calibration from only four points. With a feedback gain of 0.75 the gripper converges on its target in three or four control iterations. If the system is not disturbed it will take a straight-line path. The system has so far demonstrated its robustness by continuing to track and grasp objects despite:
  • Kinematic errors. Linear offsets or scalings of the controller's coordinate system are absorbed by the self-calibration process with complete transparency. Slight nonlinear distortions to the kinematics are corrected for by the visual feedback loop, though large errors introduce a risk of ringing and instability unless the gain is reduced.
  • Camera disturbances. The system continues to function when its cameras are subjected to small translations (e.g. 20cm), rotations (e.g. 30 degrees) and zooms (e.g. 200% change in focal length), even after it has self-calibrated. Large disturbances to camera geometry cause the gripper to take a curved path towards the target, and require more control iterations to get there.
  • Strong perspective. The condition of weak perspective throughout the robot's workspace does not seem to be essential for image-based control and the system can function when the cameras are as close as 1.5 metres (the robot's reach is a little under 1 metre). However the feedback gain must be reduced or the system will overshoot on motions towards the cameras. Figure 5 shows four frames from a tracking sequence (all taken through the same camera). The cameras are about two metres from the workspace. Tracking of position and orientation is maintained even when one of the cameras is rotated about its optical axis and zoomed (figure 5, bottom right).

  • Next
  • Contents