Early computer vision handled rigid objects like cars and cups well. Humans and animals are different. They bend, twist, and occlude their own limbs.
A bounding box tells an object detection system that a person exists somewhere in the frame. It cannot tell the system whether that person is running, falling, reaching, or throwing a punch.
Algorithm engineers need a mathematical way to represent joint motion. They need a structural constraint graph that knows an elbow connects a shoulder to a wrist, and it only moves within a limited range.
This need gave rise to keypoint and skeleton methods, which push past what classic object detection can express.
The concept actually dates back to 1973. Fischler and Elschlager proposed Pictorial Structures, representing objects as collections of parts connected by flexible springs. That work set the conceptual foundation for today’s keypoint-and-skeleton representations.
Now, teams across industries rely on structured keypoint data to train models that capture geometric relationships in visual scenes.
In this article, we'll explain what keypoint and skeleton annotation mean, where they are used, common dataset conventions, and a practical workflow you can follow.
What are keypoint and skeleton annotation in computer vision?
Keypoint annotation means marking specific feature points on objects in images or video frames. Each point corresponds to a meaningful location such as a body joint, a facial landmark, or a corner of an object.
These points are represented as 2D (or 3D) coordinates, typically stored as pixel locations (x,y). Unlike bounding boxes that only provide regions, keypoints capture precise positional information at specific anatomical or structural positions.
Skeleton annotation adds connectivity between keypoints. It defines which points are linked, forming a structured representation, such as shoulder–elbow–wrist.
This connected representation enables models to learn not just where individual points are, but also the spatial relationships and geometric constraints between them.
Keypoints and skeletons solve slightly different problems.
Keypoint annotation works best when you need to track specific features or joints without full shape information. Skeleton data becomes necessary when understanding relationships between points matters more than tracking precise boundaries. Motion analysis and gesture recognition are typical tasks.
What AI models and applications benefit most from keypoint and skeleton data?
Over the past decade, human pose estimation has been the primary application domain for keypoint and skeleton annotation. It's for identifying the spatial positions of body joints from images or video sequences.
Models trained on keypoint data detect and track body parts including shoulders, elbows, wrists, hips, knees, and ankles. This enables systems to infer a person's movement or posture.
In fitness and rehabilitation, pose estimation models powered by keypoint data analyze exercise form in real time, providing corrective feedback to users during workouts.
Facial expression analysis is another major area. Facial landmarks help models detect emotion, recognize individuals, and support applications ranging from user authentication to accessibility features in human-computer interfaces.
Autonomous driving perception also benefits. When tracking pedestrians and cyclists, keypoints can help infer whether someone is walking, standing still, or preparing to cross. That added context supports behavior prediction and can improve safety decisions.
Keypoints are not limited to humans. Defining a fixed set of structural points on vehicles creates a shape prior that can improve generalization and robustness. This supports use cases such as intelligent traffic monitoring, trajectory prediction, and accident reconstruction.
More broadly, keypoints extend detection and tracking into manufacturing, robotics, and agriculture.
In quality control, keypoints localize specific part features for defect detection and dimensional verification. In robotics, keypoints help a robot understand object geometry and candidate grasp points. In livestock monitoring, pose estimation can track behaviors like standing, lying, or grazing, which can indicate health or feeding patterns.
Are there any notable keypoint and skeleton datasets and industry standards?
Annotation standards often follow established conventions so models and tooling can interoperate, and so results can be compared fairly.
The COCO dataset is a foundational benchmark for keypoint detection. COCO Keypoints 2017 provides 17 keypoints per person across more than 56,000 training images.
These 17 keypoints define a standard human skeleton:
nose; left/right eye; left/right ear; left/right shoulder; left/right elbow; left/right wrist; left/right hip; left/right knee; left/right ankle.This convention is widely adopted in practice. Many computer vision frameworks, including Ultralytics YOLO26, use the 17-point COCO scheme as a default for human pose estimation.
Hand keypoint datasets have evolved alongside the growing importance of hand pose estimation in gesture recognition and VR applications.
The Hand Keypoints Dataset contains 26,768 images with 21 hand keypoints annotated per hand. These 21 keypoints include the wrist position plus four joints for each of the five fingers, providing enough detail to recognize complex hand poses and gestures.
Facial landmark datasets typically use 68-point or 104-point annotation schemes, depending on required granularity. The 68-point standard covers key positions around the eyes, nose, mouth, and jawline. The expanded 104-point scheme adds landmarks on areas such as the ears, eyebrows, and facial contours for a finer description of facial geometry.
When designing annotation schemes, you need to decide point ordering, attribute definitions, and occlusion rules.
Ordering is critical for skeleton-style labels because it determines the logical structure and how connections are built. Attributes can be attached to keypoints to store extra information, such as visibility state (visible, occluded, or outside the frame) or body-part type.
Visibility attributes are particularly important. They allow annotators to mark occluded points while still preserving anatomically consistent estimated locations when your task requires them.
How to perform keypoint and skeleton annotation?
The workflow starts after data collection and preparation, once you have a set of images or extracted video frames ready to label.
Here we use BasicAI Data Annotation Platform* as an example. We have prepared a video guide to help you understand the process more intuitively.
In this guide, we demonstrate 5-point facial keypoint annotation and a simplified human skeleton annotation.
The first step is creating a new dataset in the platform and uploading your data. Next, you must define the ontology, which specifies the annotation structure and the classes you will create.
For keypoint classes, ontology definition involves class name, numbering, attached attributes, and display style.
For skeleton annotation, you must build a skeleton template that defines point structure and connectivity. BasicAI Data Annotation Platform* allows you to upload a reference image and draw the skeleton template directly on it.
Then, select all data and enter the annotation interface. For keypoint-only annotation, select the keypoint tool and click each relevant location in the image, placing points at precise coordinates.
For skeleton annotation, the process is more structured. Annotators must place keypoints following the predefined template order so the system can maintain the intended structure.
If connections were defined in the template, the platform draws edges automatically as points are placed, creating a visible skeleton that guides the annotator toward anatomically reasonable configurations.
When points are occluded or invisible, follow your annotation guidelines. If you added visibility attributes during ontology setup, you can estimate possible keypoint positions based on visible neighboring points and anatomical constraints, then add the appropriate attributes.
After labeling, save your work through the interface and exit. The final step is creating an export task to prepare the annotated data for model training. The export process selects which annotations to include, specifies output format, and generates the dataset in your chosen standard (such as COCO JSON).
*Contact us here to customize your privately deployed annotation platform.
How to build high-quality keypoint and skeleton datasets efficiently?
Keypoint labeling is less forgiving than many other annotation tasks. Precision and consistency matter because small point errors propagate into downstream training and can degrade performance across tasks.
If you are preparing training data for pose estimation or similar models, here are several recommendations based on our experience.
These should be addressed in your data annotation guidelines:
Clearly define what each keypoint represents and how to position it in ambiguous cases. For a shoulder keypoint, guidelines should specify whether the point marks the shoulder joint center, the top of the shoulder, or the outermost point of the scapula. Small differences like this are a common source of annotator disagreement.
The human skeleton annotation task needs a strict left/right convention. The guide should state how “left” and “right” map to image coordinates across viewpoints, including frontal views and rotated poses.
Skeleton annotation guidelines should emphasize point ordering. Annotators should know whether they are expected to label the upper body before the lower body, or complete one side of the body before the other, and why that order exists.
Fully visible limbs are rare in real-world footage, so occlusion handling must be standardized. Define when annotators should estimate a point using anatomical constraints and when they should mark it as not visible. The best choice depends on the application. Pose optimization algorithms may need explicit visibility tags, while other systems work better with estimated positions.
Accuracy drives model quality. Efficiency gains mean nothing if they come at the cost of accuracy. For projects building in-house annotation teams, choose a professional annotation platform designed for this type of work.
BasicAI Data Annotation Platform is the tool of choice for many teams like yours. It offers:
Standalone keypoint annotation and structured skeleton annotation tools;
Multi-level attribute labels for building complex ontologies;
Role-based access control for team collaboration;
Batch automated quality checks that flag error annotations;
Private deployment option for maximized data security; and
Export in standard formats like COCO JSON to ensure compatibility with common training frameworks.
For large projects, external vendors can improve throughput. In practice, providers with dedicated annotation teams often deliver higher accuracy than open crowdsourcing, and they tend to handle domain-specific cases better.
When selecting a vendor, verify their accuracy on similar projects, their experience with keypoint labeling in particular, and their ability to keep consistency across large datasets.
