Skip to content

Skeleton system

What is a MANUS Skeleton

When using glove data to drive the animation of a model, you commonly run into mismatches. Be it, the dimensions of the model don't match, the bone names don't match etcetera. The skeleton system is our answer to tackle that issue. It's a system to retarget our data (be it glove or body) onto the model of your choosing.

Structure

Skeleton Types

There are several types of skeletons, the skeleton type will determine what type of animation can drive it.

  • Hand, a hand skeleton will only ever animate the hands, useful when only animating hands or using a different animation system for driving the body.
  • Body, a body skeleton will only ever animate the body without hands or face.
  • Both, this skeleton type will animate the entire body, hands and body. Facial animation is not something handled by the skeleton system.

Nodes

Nodes are the joints of the skeleton (the point around which your bones rotate). A node consists of the following properties.

Property Description
ID Unique identifier of the node
Name Name of the node, only relevant for human readability
Position Position of the node
Rotation Rotation of the node represented as a Quaternion
ParentID ID of the parent node (0 or its own ID when no parent)

Coordinate-system

Position and rotation will use the coordinate system based on the type specified by the Session (World / Local). This is specified when creating the session.

Chains

Chains are a combination of nodes, representing a body part. For instance an arm chain consists of the shoulder, elbow and wrist nodes, creating an ik-chain that can then be used to drive the animation.

Types of chains

The table below describes all the chain types. Chain types can also have a side to them.

Chain type Description Full body requirement Hands only requirement Sidedness
Hand Node containing hand/wrist , usually only one
FingerThumb All nodes representing the thumb
FingerIndex All nodes representing the index finger
FingerMiddle All nodes representing the middle finger
FingerRing All nodes representing the ring finger
FingerPinky All nodes representing the pinky finger
Head Node containing the head, usually only one
Neck Nodes containing the neck, usually the neck and head nodes
Shoulder Contains the shoulder bones, from clavicle to upper arm
Arm Contains the arm nodes, starts at the upper arm and ends at the hand
Spine Contains the spine nodes, starts at the pelvis and ends at the neck
Pelvis Node containing the pelvis, usually only one
Leg Contains the leg nodes, starts at thigh and ends at the foot
Foot Node containing the foot, usually only one
Toe Node containing the toe, usually only one

Finger-requirements

A minimum of two finger chains and a thumb chain are required for hand animation, it does not matter which fingers specifically.

Setting up a skeleton

Now that we've covered the basics requirements of a Skeleton it's time to set one up. For this there are two methods, manually defining the entire structure or using the DevTools in conjunction with our Unity or Unreal plugins.

Code (SDK)

The SDK Client example has an entire chapter on how to set up and use skeletons. This method of implementation is supported by all plugins (SDK, Unreal, Unity).

DevTools (Unity / Unreal)

When setting up a 3D model for animation in combination with our Unity and Unreal plugins, it's however highly advisable to use our DevTools application. This automates a big part of the process. Please refer to its documentation for specific steps how to use the DevTools to configure skeletons.

Note

Currently loading FBX files directly into the DevTools for skeleton creation is not supported.

Target types

There are several ways to specify how the skeleton is driven. Directly via DeviceID or via the User system.

Target types Parameter Description
User Data UserID Data is applied based on the specified UserID, assigned glove data is used to drive the skeleton
User Index Data User Index Identical to UserID but the UserID is based off the Index (0 = first user, 1 = second user)
Glove Data GloveID Data is applied based on deviceID

Note

In the past it was also possible to drive the skeleton based off recorded animation data. This is no longer supported.

Settings

After defining all the skeleton's nodes and chains there are several settings that allow you to tweak the retargeting based on your intended goal and target model.

Skeleton

Use end point approximation (bool)

Use end-point approximation increases the weight on the solve towards accurate finger end-point precision over joint angle accuracy. Leaving this off will give you the most realistic joint angles and turning this on will give you the most true to life finger tip positions.

Note

For accurate end-to-end finger positions, leaf bones are a requirement.

Scale skeleton (bool)

Scale skeleton will scale the bones in the skeleton to match the user's dimensions. This will size the skeleton more true to the user's actual dimensions but might result in scaling issues when the model is stretched or squashed beyond what is acceptable.

Hand

Finger bones

Hand chain ID

uint ID of hand chain

Metacarpal bones

bool First node of the finger chain is metacarpal bone

Metacarpal bone ID

uint Node ID of metacarpal bone

Use leaf at end

bool Will use the last node in the chain and read its leaf settings. The leaf direction and leaf

Leaf direction

Vector3 Directional vector of the fingertip in local space.

Leaf length

float Length of fingertip in the integration's unit scale. This will be meters for SDK and Unity and centimeters for Unreal.

Hand motion

Specifies what drives wrist rotation and positioning.

Hand Motion Description
None Hand will not transform
IMU Hand / Wrist rotates based on glove device IMU data
Tracker Hand / Wrist rotates and positions based on assigned tracker data
TrackerRotationOnly Hand / Wrist rotates based on assigned tracker data
Auto Hand / Wrist switches between IMU and Tracker based on data availability
Head pitch offset (float)

float Changes the head node pitch offset, value is in degrees.

Head yaw offset

float Changes the head node yaw offset, value is in degrees.

Head tilt offset

float Changes the head tilt/roll offset, value is in degrees.

Use leaf at end

bool Similarly to how this works for fingers, the head "tip" node is added. This data is to match the scale of the head to the user's dimensions. Only works if scale to target is enabled.

Neck

Neck bend offset

float Changes the neck tilt offset, value is in degrees.

Shoulder

Forward offset

float Rotates the shoulders into the forward direction, value is in degrees.

Shrug offset

float Rotates the shoulders into the up direction, value is in degrees.

Forward multiplier

float Allows for fine tuning of shoulder forward rotation severity. Default is 1, value at 0 will lock the forward rotation in place.

Shrug multiplier

float Allows for fine tuning of shoulder up rotation severity. Default is 1, value at 0 will lock the forward rotation in place.

Arm

Elbow rotation offset

float Rotates the elbow around the directional axis of the arm, value is in degrees.

Arm length multiplier

float Scales the end position of the arm. This does not scale the model / bones themselves but modifies the reach of the arm. Default is 1 and multiplies accordingly.

Arm rotation offset

Vector3 Modifies the rotation of the arm, X represents the yaw, Y represents the pitch and Z represents the roll. Expressed in degrees.

Position offset

Vector3 Offsets the hand target position, value is in the integration's unit scale. This will be meters for SDK and Unity and centimeters for Unreal.

Position multiplier

float Multiplier for the endpoint direction. Default is 1 and multiplies accordingly.

Spine

Spine bend offset

float Changes the spine tilt offset, value is in degrees.

Pelvis

Hip Height multiplier

float Changes the hip height, does not change the leg length. Default is 1 and multiplies accordingly.

Hip bend offset

float Changes the hip tilt offset, value is in degrees.

Thickness multiplier

float Changes the forward and up scales for the model. Only works if scale to target is enabled. Default is 1 and multiplies accordingly.

Leg

Reverse knee direction

bool Reverses the direction the knee bends.

Knee rotation offset

float Rotates the knee around the directional axis of the leg, value is in degrees.

Foot forward offset

float Offsets the foot forward position, value is in the integration's unit scale.

Foot side offset

float Offsets the foot sideways' position, value is in the integration's unit scale.

Finger leaf bones

It is highly advised for the finger skeletal structure to also contain tip leaf bones. For our skeletal solver to correctly solve for finger point interactions it requires tip bones to know where the finger ends. The finger chain settings do allow for the addition of tip leaf bones when the model does not have them.

alt text

Skeleton types

There are three different skeletons in MANUS Core: Retargeted Skeletons, Raw Skeletons and Temporary Skeletons.

Retargeted Skeletons

Retargeted skeletons are the skeletons this article has been mainly about. The data that is streamed out to the plugins and animates your models are these skeletons.

Raw Skeletons

The Raw skeletons are Core's internal representations of the glove data. Internally a skeleton is made using the device's sensor data and calibration values. This data is streamed out per device over the so called RawSkeletonStream. Please refer to the SDK Client documentation for more in depth information.

Temporary Skeletons

Temporary skeletons are the temporary definitions of a skeleton while they're still being built up and not yet ready to be animated.

mskl files

Skeletons can be saved to our mskl (Manus Skeleton) file format. These files can then be loaded and re-used for models. Opening these files will open the MANUS DevTools for you to edit them, and they can be saved back to mskl files.

These files can be loaded from SDK, Unity and Unreal plugins to improve the flow when re-using models.

Loading in Unreal

In Unreal open your MANUS Skeleton, there you are able to import and export an mskl file for the used skeletal mesh.

alt text

Loading in Unity

In Unity the MANUS skeleton component has the ability to load and save mskl files.

alt text

Loading in SDK

Loading mskl files in the SDK is a bit more involved. The SDK Client has example code how to load mskl files.

Load skeleton from MSKL file
void SDKClient::GetTemporarySkeletonFromFile()
{
    // this example shows how to load a temporary skeleton data from a file

    // as an example we try to get the temporary skeleton data previously saved as .mskl file in directory Documents/ManusTemporarySkeleton
    // get the path for the documents directory
    std::string t_DirectoryPathString = GetDocumentsDirectoryPath_UTF8();

    // check if directory exists
    std::string t_DirectoryPath =
        t_DirectoryPathString
        + s_SlashForFilesystemPath
        + "ManusTemporarySkeleton";

    if (!DoesFolderOrFileExist(t_DirectoryPath))
    {
        SPDLOG_WARN("Failed to read from client file, the mentioned directory does not exist");
        return;
    }

    // create string with file name
    std::string t_DirectoryPathAndFileName =
        t_DirectoryPath
        + s_SlashForFilesystemPath
        + "TemporarySkeleton.mskl";

    // read from file
    std::ifstream t_File = GetInputFileStream(t_DirectoryPathAndFileName);

    if (!t_File)
    {
        SPDLOG_WARN("Failed to read from client file, the file does not exist in the mentioned directory");
        return;
    }

    // get file dimension
    t_File.seekg(0, t_File.end);
    int t_FileLength = (int)t_File.tellg();
    t_File.seekg(0, t_File.beg);

    // get temporary skeleton data from file
    unsigned char* t_TemporarySkeletonData = new unsigned char[t_FileLength];
    t_File.read((char*)t_TemporarySkeletonData, t_FileLength);
    t_File.close();


    // save the zipped temporary skeleton information, they will be used internally for sending the data to Core
    uint32_t t_TemporarySkeletonLengthInBytes = t_FileLength;

    if (t_TemporarySkeletonData == nullptr)
    {
        SPDLOG_WARN("Failed to read the compressed temporary skeleton data from file");
        delete[] t_TemporarySkeletonData;
        return;
    }

    // create a skeleton setup where we will store the temporary skeleton retrieved from file
    SkeletonSetupInfo t_SKL;
    SkeletonSetupInfo_Init(&t_SKL);
    uint32_t t_SklIndex = 0;
    SDKReturnCode t_Res = CoreSdk_CreateSkeletonSetup(t_SKL, &t_SklIndex);
    if (t_Res != SDKReturnCode::SDKReturnCode_Success)
    {
        spdlog::error("Failed to Create Skeleton Setup. The error given was {}.", t_Res);
        return;
    }
    m_TemporarySkeletons.push_back(t_SklIndex);

    // associate the retrieved temporary skeleton to the current session id
    uint32_t t_SessionId = m_SessionId;

    // load the temporary skeleton data retrieved from the zipped file and save it with index t_SklIndex and session id of the current session
    SDKReturnCode t_Result = CoreSdk_GetTemporarySkeletonFromCompressedData(t_SklIndex, t_SessionId, t_TemporarySkeletonData, t_TemporarySkeletonLengthInBytes);
    if (t_Result != SDKReturnCode::SDKReturnCode_Success)
    {
        SPDLOG_WARN("Failed to load temporary skeleton data from client file in Core, the error code was: {}.", t_Result);
        return;
    }

    delete[] t_TemporarySkeletonData;
}

Troubleshooting

Skeleton does not animate

If the skeleton does not animate. You can verify if a skeleton is setup correctly within the DevTools. In the left bottom corner of the viewport you are able to playback several animations. If the skeleton does animate in this view but not in your other implementation, make sure the target type is set correctly. If your skeleton also does not animate within the DevTools animation view it means the chain setup is probably configured incorrectly. Please make sure you minimally have all the chains required for your use case.

Pinches don't look correct

If the finger end point interations do not seem to be lining up correctly but they are fine within MANUS Core Dashboard. Make sure your skeleton has finger tips setup correctly, to improve end-point interactions, end point approximation and scale to target can be enabled.