Teleoperated system performance evaluation

Measures for the evaluation of teleoperated system performance are considered. The paper discusses the predominant use of time to task completion for performance evaluation and proposes three new measures based on an analysis of manipulator motions. The new measures include manipulator joint effort, manipulator dexterity and end effector motion effort. Data from over 900 trials of a task performed by 36 operators is used to empirically compare these measures. The task involved insertion of a peg in a hollow cylinder mounted on a rotating circular table, with one of three visual feedback modalities used by each operator. These included: direct vision; three dimensional stereo video and monocular video. The performance measures based on manipulator joint effort and end effector motion effort are found to exhibit the same learning curve trends, similar ability to show performance difference between the visual modalities used and to possess good correlation with the traditional measures of time to completion and of operator subjective difficulty rating. The measure of manipulator dexterity shows similar learning trends but was unable to discriminate any performance difference between tests completed with the three different visual modalities. The joint effort and end effector effort measures are applicable to telerobotic (no operator) situations and do not require the completion of a `calibration task´ for evaluation