Action Cue Assistant
Advanced AutonomyIntermediate Robotics plus perception pipelines, control loops, coordinate frames, and real-time behavior.
Before you start
- Complete Camera Module
- Understand body keypoints, confidence thresholds, and short motion histories
- Use an instructor-approved pose or action model
Build a camera project that recognizes a small set of actions such as wave, stop, and point, then reports the matching response.
Project Outcome
The first version prints the detected action and confidence. A later supervised version can map actions to lights or other instructor-approved robot behavior.
Camera Input
- Upper-body keypoints or tracked joint positions
- A short history of keypoint movement
- Detection confidence and target visibility
Build Stages
- Capture frames and confirm the person remains visible.
- Record or obtain body keypoints from an approved perception model.
- Define one action using a small, explainable rule.
- Store recent observations in a fixed-length history.
- Add confidence and cooldown rules to prevent repeated triggers.
- Print the selected response without moving the robot.
First Prototype
Recognize wave and stop. Print hello for a wave and motion inhibited for a stop gesture.
What To Measure
- Correct actions versus false triggers
- Recognition delay
- Minimum useful confidence
- Performance when a hand or arm is briefly hidden
Extensions
- Add pointing direction.
- Display keypoints and confidence over the image.
- Compare rule-based recognition with a small temporal classifier.
- Add a supervised light response after the camera-only version is reliable.
Safety Boundary
Keep the first prototype camera-only. Gesture recognition must never be the only emergency-stop mechanism, and generated code must not initiate robot motion.