License requirement
The functionality described requires a MANUS Bodypack or a MANUS license dongle containing one of these licenses to be connected:
Core Pro Core XR Core Xsens Pro Core Qualisys Pro Core OptiTrack Pro Demo
SDK Client
Introduction
This article will describe the more important functions of the SDK client and where needed go into detail about the code behind the function. It will however not describe the complete program layout as it is not representative of a normal client program using the SDK, nor always compatible with different program architectures. In fact, it is written to demonstrate the function usage instead of being the optimal way of using those functions. Due to this there is no graphical UI beyond the simple console output as that would require more complexity that may obfuscate the SDK usage.
The inner functions of the SDK are outside of this document's scope and the sample implementation is only used as a demonstration on how to use the SDK. What you will find in this document are the most common functions to set up a (temporary) skeleton, process the landscape, handle haptic feedback, handle trackers, gestures, and handle time tracking.
If you want a simple quick start guide for the basics required to get data, the SDK minimal client quick start guide might be a better choice, but for more in-depth details look at this documentation.
SDK Callbacks
All the data streams MANUS Core provides are structured as callbacks. A callback is a function that you can pass into a CoreSdk_RegisterCallback... function. These functions call them at another point (on another thread) when a certain event occurs. This way users do not need to poll the SDK to figure out if there is new data available.
It is good practice to register all the callbacks you require after initializing the SDK and before connecting to MANUS Core. You only need to register the callbacks which you intend to use, if you do not plan to use certain data streams this could decrease the traffic from MANUS Core.
void SDKClient::OnErgonomicsCallback(const ErgonomicsStream* const p_Ergo)
{
if (s_Instance)
{
for (uint32_t i = 0; i < p_Ergo->dataCount; i++)
{
if (p_Ergo->data[i].isUserID)continue;
ErgonomicsData* t_Ergo = nullptr;
if (p_Ergo->data[i].id == s_Instance->m_FirstLeftGloveID)
{
t_Ergo = &s_Instance->m_LeftGloveErgoData;
}
if (p_Ergo->data[i].id == s_Instance->m_FirstRightGloveID)
{
t_Ergo = &s_Instance->m_RightGloveErgoData;
}
if (t_Ergo == nullptr)continue;
CoreSdk_GetTimestampInfo(p_Ergo->publishTime, &s_Instance->m_ErgoTimestampInfo);
t_Ergo->id = p_Ergo->data[i].id;
t_Ergo->isUserID = p_Ergo->data[i].isUserID;
for (int j = 0; j < ErgonomicsDataType::ErgonomicsDataType_MAX_SIZE; j++)
{
t_Ergo->data[j] = p_Ergo->data[i].data[j];
}
}
}
}
This callback is called whenever the SDK receives new ergonomic data from MANUS Core. A best practice for data being received on another thread (which happens with the SDK's callback mechanism) is to save the data somewhere and then process whatever you want to do with the data on your own thread. This way you do not block/delay the callback thread, which could lead to delayed/laggy data transfer.
In several of our streams, we add timestamps to show when the data was captured. To get a more readable timestamp you need to call the CoreSdk_GetTimestampInfo on the timestamp value, this will translate the timestamp value to a date & time format.
Client Connection
There are several options for connecting to MANUS Core. You usually have MANUS Core running on your local machine, which makes connecting simpler than via a network connection. Selecting local by pressing the L key will connect you locally. You may potentially have certain network security settings that may block certain connections , this may be a first indicator that Manus Core could have trouble making a connection.
If you opt for a remote connection, press the H key. It will now search all available networks for MANUS Core hosts and display them in a simple selection menu.
Once a connection is made the Main Menu screen will be displayed which allows you to look at the various kinds of data available via the SDK.
To dive into the code behind our example client a bit, when looking at the code for finding hosts, you will see something like the following:
ClientReturnCode SDKClient::LookingForHosts()
{
spdlog::info("Looking for hosts...");
// Underlying function will sleep for m_SecondsToFindHosts to allow servers to reply.
const SDKReturnCode t_StartResult = CoreSdk_LookForHosts(m_SecondsToFindHosts, m_ShouldConnectLocally);
if (t_StartResult != SDKReturnCode::SDKReturnCode_Success)
{
spdlog::error("Failed to look for hosts. The error given was {}.", t_StartResult);
return ClientReturnCode::ClientReturnCode_FailedToFindHosts;
}
m_NumberOfHostsFound = 0;
const SDKReturnCode t_NumberResult = CoreSdk_GetNumberOfAvailableHostsFound(&m_NumberOfHostsFound);
if (t_NumberResult != SDKReturnCode::SDKReturnCode_Success)
{
spdlog::error("Failed to get the number of available hosts. The error given was {}.", t_NumberResult);
return ClientReturnCode::ClientReturnCode_FailedToFindHosts;
}
if (m_NumberOfHostsFound == 0)
{
spdlog::warn("No hosts found.");
m_State = ClientState::ClientState_NoHostsFound;
return ClientReturnCode::ClientReturnCode_FailedToFindHosts;
}
m_AvailableHosts.reset(new ManusHost[m_NumberOfHostsFound]);
const SDKReturnCode t_HostsResult = CoreSdk_GetAvailableHostsFound(m_AvailableHosts.get(), m_NumberOfHostsFound);
if (t_HostsResult != SDKReturnCode::SDKReturnCode_Success)
{
spdlog::error("Failed to get the available hosts. The error given was {}.", t_HostsResult);
return ClientReturnCode::ClientReturnCode_FailedToFindHosts;
}
if (m_ShouldConnectLocally)
{
m_State = ClientState::ClientState_ConnectingToCore;
return ClientReturnCode::ClientReturnCode_Success;
}
m_State = ClientState::ClientState_PickingHost;
return ClientReturnCode::ClientReturnCode_Success;
}
To find all the available MANUS Core instances, the CoreSdk_LookForHosts function needs to be called. This function call is a blocking function and therefore will block the calling thread for the specified duration. We advise not running any time sensitive functions or UI in the same thread as the Manus SDK connection.
After this function has completed you can get the number of hosts found by calling the CoreSdk_GetNumberOfAvailableHostsFound function. If this function returns 0 hosts, either no MANUS core is available, or the network is being blocked by a security system or another program.
The next step is creating an array at least the size of the number of found hosts and passing this array into the CoreSdk_GetAvailableHostsFound function. This function will put information about all the hosts that are found on the network into the array.
Once you have at least one host to connect to you can connect to it with code like this:
ClientReturnCode SDKClient::ConnectingToCore()
{
SDKReturnCode t_ConnectResult = SDKReturnCode::SDKReturnCode_Error;
if (m_ShouldConnectGRPC)
{
t_ConnectResult = CoreSdk_ConnectGRPC();
}
else
{
if (m_ShouldConnectLocally) { m_HostToConnectTo = 0; }
t_ConnectResult = CoreSdk_ConnectToHost(m_AvailableHosts[m_HostToConnectTo]);
}
if (t_ConnectResult == SDKReturnCode::SDKReturnCode_NotConnected)
{
m_State = ClientState::ClientState_NoHostsFound;
return ClientReturnCode::ClientReturnCode_Success; // Differentiating between error and no connect
}
if (t_ConnectResult != SDKReturnCode::SDKReturnCode_Success)
{
spdlog::error("Failed to connect to Core. The error given was {}.", t_ConnectResult);
return ClientReturnCode::ClientReturnCode_FailedToConnect;
}
m_State = ClientState::ClientState_DisplayingData;
// Note: a log message from somewhere in the SDK during the connection process can cause text
// to permanently turn green after this step. Adding a sleep here of 2+ seconds "fixes" the
// issue. It seems to be caused by a threading issue somewhere, resulting in a log call being
// interrupted while it is printing the green [info] text. The log output then gets stuck in
// green mode.
return ClientReturnCode::ClientReturnCode_Success;
}
The chosen host structure needs to be put into the CoreSdk_ConnectToHost function. This function then attempts to connect to the given host and will trigger the OnConnectedCallback callback when it is successful. This callback is registered via the CoreSdk_RegisterCallbackForOnConnect function.
If a connection to MANUS Core is lost the OnDisconnected callback is invoked. This one can be registered via the CoreSdk_RegisterCallbackForOnDisconnect function.
Main menu
To make the example client easier to navigate and our data easier to observe, we made a menu system to view various parts of the data available. None of the code used to display or navigate through the menus are needed to use the SDK, all data can be accessed simultaneously.
Landscape
The landscape callback is one that happens every second, this callback gives information about various parts of MANUS Core, for example which gloves and dongles are connected, which users exist, gestures, etc. Our sample client does not use all the information given via the landscape to keep the example a bit easier to follow.
void SDKClient::OnLandscapeCallback(const Landscape* const p_Landscape)
{
if (s_Instance == nullptr)return;
Landscape* t_Landscape = new Landscape(*p_Landscape);
s_Instance->m_LandscapeMutex.lock();
if (s_Instance->m_NewLandscape != nullptr) delete s_Instance->m_NewLandscape;
s_Instance->m_NewLandscape = t_Landscape;
s_Instance->m_NewGestureLandscapeData.resize(t_Landscape->gestureCount);
CoreSdk_GetGestureLandscapeData(s_Instance->m_NewGestureLandscapeData.data(), (uint32_t)s_Instance->m_NewGestureLandscapeData.size());
s_Instance->m_LandscapeMutex.unlock();
}
If you plan to use the Gestures and need the landscape data related to these make sure to call the CoreSdk_GetGestureLandscapeData to retrieve all the data needed for the gestures from within the callback to ensure that the entirety of the landscape is stored correctly.
An overview of most of the landscape is displayed in our landscape viewer tool:
The entire Landscape structure is self-explanatory, the gloves and dongle information can be found under the gloveDevices variable. You can find the battery information, the pairing state of the glove, the type of glove, haptics, etc. inside the GloveLandscapeData structure. In the DongleLandscapeData structure you can find information such as the dongle type, firmware versions, which gloves are paired to it, etc. In the users you can find information such as which dongle and gloves are assigned to which user.
Gloves & Dongles
The code in the DisplayingDataGlove function handles the console logging, the HandleHapticCommands function shows the way haptics are handled when using a glove ID to identify which glove needs to vibrate.
In this example 2 gloves are connected, and their ergonomics data (received via the callback) is shown with a timestamp. If the gloves are haptic enabled, you can use the number keys to trigger individual haptic actuators on the fingers.
This view also shows which dongles are present and the licenses on them.
void SDKClient::OnErgonomicsCallback(const ErgonomicsStream* const p_Ergo)
{
if (s_Instance)
{
for (uint32_t i = 0; i < p_Ergo->dataCount; i++)
{
if (p_Ergo->data[i].isUserID)continue;
ErgonomicsData* t_Ergo = nullptr;
if (p_Ergo->data[i].id == s_Instance->m_FirstLeftGloveID)
{
t_Ergo = &s_Instance->m_LeftGloveErgoData;
}
if (p_Ergo->data[i].id == s_Instance->m_FirstRightGloveID)
{
t_Ergo = &s_Instance->m_RightGloveErgoData;
}
if (t_Ergo == nullptr)continue;
CoreSdk_GetTimestampInfo(p_Ergo->publishTime, &s_Instance->m_ErgoTimestampInfo);
t_Ergo->id = p_Ergo->data[i].id;
t_Ergo->isUserID = p_Ergo->data[i].isUserID;
for (int j = 0; j < ErgonomicsDataType::ErgonomicsDataType_MAX_SIZE; j++)
{
t_Ergo->data[j] = p_Ergo->data[i].data[j];
}
}
}
}
In the ergonomics callback we filter on the first user's gloves so as not to bloat the sample code, in normal cases you usually want all the information contained in this callback when using ergonomics.
Skeletons
MANUS Core uses something we've named skeletons to animate the hands and bodies. An important part of the skeleton setup are nodes, which you can consider as being the joints (the point around which your finger bone rotates) of your fingers.
Another important part of the skeleton setup are the chains, these define which nodes belong to which part of the skeleton. For example, your thumb chain would contain all the nodes that make up your thumb's bones. There are three different skeletons in MANUS Core, Temporary Skeletons, Retargeted Skeletons and Raw Skeletons. Temporary Skeletons are the skeletons that you create and send to the development tools for verification, but you do not load them into MANUS Core's retargeting system. Retargeted Skeletons are usually called Skeletons in the SDK, which are loaded into MANUS Core with the animation data applied. The Raw Skeletons are hand models on which MANUS Core applies the hand data without doing any retargeting, these are not necessarily using the same bone hierarchy structure as the skeletons that the user loads into MANUS Core.
Skeletal Data
MANUS Core supplies two streams of skeletal data. The first being retargeted skeletal data, this is the glove data applied onto a skeleton of the user's choosing. The second stream is the raw skeletal data, this is the glove data applied onto a standardized MANUS hand skeleton. Depending on your application's needs you can use one stream or the other, or in some cases even both.
The retargeted skeletal data is useful when you have your own skeleton or hand model, and you would like things like pinches to look better than they would when simply applying rotations from the raw skeleton stream. Using retargeted skeletal data allows you to create hands that have strange dimensions, or a different number of fingers compared to a real-life hand.
The raw skeletal data is generally the more accurate representation of the glove wearer's hand. The rotations and positions output in the raw skeleton stream are usually closer to the real world hand of a person, but applying this data directly to your desired skeleton or hand model can be a much more challenging task and will not guarantee lifelike animation . Another important detail to keep in mind is that our raw skeletal data does not contain any wrist positioning or orientation, if these are needed then it is advised to use the retargeted skeletal data.
The closer the dimensions are to a real-life hand the smaller the difference is between the retargeted and the raw skeletal data, at which point it tends to be much easier to just use the retargeted skeletal data for your skeleton or hand model.
Both skeletal stream callbacks function in a similar way, and can be seen in the example client:
void SDKClient::OnSkeletonStreamCallback(const SkeletonStreamInfo* const p_SkeletonStreamInfo)
{
if (s_Instance)
{
ClientSkeletonCollection* t_NxtClientSkeleton = new ClientSkeletonCollection();
t_NxtClientSkeleton->skeletons.resize(p_SkeletonStreamInfo->skeletonsCount);
for (uint32_t i = 0; i < p_SkeletonStreamInfo->skeletonsCount; i++)
{
CoreSdk_GetSkeletonInfo(i, &t_NxtClientSkeleton->skeletons[i].info);
t_NxtClientSkeleton->skeletons[i].nodes.resize(t_NxtClientSkeleton->skeletons[i].info.nodesCount);
t_NxtClientSkeleton->skeletons[i].info.publishTime = p_SkeletonStreamInfo->publishTime;
CoreSdk_GetSkeletonData(i, t_NxtClientSkeleton->skeletons[i].nodes.data(), t_NxtClientSkeleton->skeletons[i].info.nodesCount);
}
s_Instance->m_SkeletonMutex.lock();
if (s_Instance->m_NextSkeleton != nullptr) delete s_Instance->m_NextSkeleton;
s_Instance->m_NextSkeleton = t_NxtClientSkeleton;
s_Instance->m_SkeletonMutex.unlock();
}
}
void SDKClient::OnRawSkeletonStreamCallback(const SkeletonStreamInfo* const p_RawSkeletonStreamInfo)
{
if (s_Instance)
{
ClientRawSkeletonCollection* t_NxtClientRawSkeleton = new ClientRawSkeletonCollection();
t_NxtClientRawSkeleton->skeletons.resize(p_RawSkeletonStreamInfo->skeletonsCount);
for (uint32_t i = 0; i < p_RawSkeletonStreamInfo->skeletonsCount; i++)
{
CoreSdk_GetRawSkeletonInfo(i, &t_NxtClientRawSkeleton->skeletons[i].info);
t_NxtClientRawSkeleton->skeletons[i].nodes.resize(t_NxtClientRawSkeleton->skeletons[i].info.nodesCount);
t_NxtClientRawSkeleton->skeletons[i].info.publishTime = p_RawSkeletonStreamInfo->publishTime;
CoreSdk_GetRawSkeletonData(i, t_NxtClientRawSkeleton->skeletons[i].nodes.data(), t_NxtClientRawSkeleton->skeletons[i].info.nodesCount);
}
s_Instance->m_RawSkeletonMutex.lock();
if (s_Instance->m_NextRawSkeleton != nullptr) delete s_Instance->m_NextRawSkeleton;
s_Instance->m_NextRawSkeleton = t_NxtClientRawSkeleton;
s_Instance->m_RawSkeletonMutex.unlock();
}
}
Getting the basic skeleton information such as the ID and node count ca n be done via the CoreSdk_GetSkeletonInfo or the CoreSdk_GetRawSkeletonInfo respectively. In this info you can get the ID of the skeleton, which is either the id of the retargeted skeleton or of the glove ID , depending on which of the two streams you are looking at.
Using the CoreSdk_GetSkeletonData or the CoreSdk_GetRawSkeletonData you can get all the node data for the skeleton. The ID of the node is either the ID you gave it when setting up the skeleton for the retargeting or the ID that we gave the raw skeleton in MANUS Core.
The difficult part about using the Raw Skeletons compared to the Retargeting Skeleton is that you will need to reconstruct the hierarchy of that skeleton to be able to apply the data gotten from the stream. In the PrintRawSkeletonData function we demonstrate how to get the hierarchal information for the raw skeleton.
void SDKClient::PrintRawSkeletonData()
{
if (m_RawSkeleton == nullptr || m_RawSkeleton->skeletons.size() == 0)
{
return;
}
if (m_FirstLeftGloveID == 0 && m_FirstRightGloveID == 0) return; // no gloves connected to core
uint32_t t_NodeCount = 0;
uint32_t t_GloveId = 0;
if (m_FirstLeftGloveID != 0)
{
t_GloveId = m_FirstLeftGloveID;
}
else
{
t_GloveId = m_FirstRightGloveID;
}
SDKReturnCode t_Result = CoreSdk_GetRawSkeletonNodeCount(t_GloveId, t_NodeCount);
if (t_Result != SDKReturnCode::SDKReturnCode_Success)
{
spdlog::error("Failed to get Estimation Node Count. The error given was {}.", t_Result);
return;
}
// now get the hierarchy data, this can be used to reconstruct the positions of each node in case the user set up the system with a local coordinate system
// having a node position defined as local means that this will be related to its parent
NodeInfo* t_NodeInfo = new NodeInfo[t_NodeCount];
t_Result = CoreSdk_GetRawSkeletonNodeInfo(t_GloveId, t_NodeInfo);
if (t_Result != SDKReturnCode::SDKReturnCode_Success)
{
spdlog::error("Failed to get Estimation Hierarchy. The error given was {}.", t_Result);
return;
}
spdlog::info("Received Skeleton glove data from the estimation system. skeletons:{} first skeleton glove id:{}", m_RawSkeleton->skeletons.size(), m_RawSkeleton->skeletons[0].info.gloveId);
AdvanceConsolePosition(2);
}
The CoreSdk_GetRawSkeletonNodeCount function can be used to get the number of nodes that exist in the raw skeleton of a certain glove. Using the CoreSdk_GetRawSkeletonNodeInfo you can get all the information for every node in the Raw Skeleton. This NodeInfo gives you information such as the parent's ID and the side. Using this information you can reconstruct the skeleton. When using a relative coordinate system (a not world space coordinate system) you will need to keep the hierarchy into account to be able to accurately reconstruct the skeleton.
Skeletons in the Sample Client
This menu shows a few options. The first two are to load or unload a skeleton to MANUS Core, which we will talk about later in this article.
The other options are to activate haptics based on the skeleton id instead of a glove id. This is an easier way to do haptics since you do not have to manually keep track of which glove belongs to what skeleton.
Setup a skeleton
To setup a hand skeleton and make sure it animates correctly, the following steps are required:
- Setup the basic skeleton information
- Create and add nodes for the skeleton
- Create and add chains for the skeleton
The first step is creating a SkeletonSetupInfo structure and setting the data you wish to adjust. For a hand skeleton the type needs to be set to the SkeletonType_Hand enumerator.
void SDKClient::LoadTestSkeleton()
{
uint32_t t_SklIndex = 0;
SkeletonSetupInfo t_SKL;
SkeletonSetupInfo_Init(&t_SKL);
t_SKL.type = SkeletonType::SkeletonType_Hand;
t_SKL.settings.scaleToTarget = true;
t_SKL.settings.useEndPointApproximations = true;
t_SKL.settings.targetType = SkeletonTargetType::SkeletonTargetType_UserIndexData;
//If the user does not exist then the added skeleton will not be animated.
//Same goes for any other skeleton made for invalid users/gloves.
t_SKL.settings.skeletonTargetUserIndexData.userIndex = 0;
CopyString(t_SKL.name, sizeof(t_SKL.name), std::string("LeftHand"));
SDKReturnCode t_Res = CoreSdk_CreateSkeletonSetup(t_SKL, &t_SklIndex);
if (t_Res != SDKReturnCode::SDKReturnCode_Success)
{
spdlog::error("Failed to Create Skeleton Setup. The error given was {}.", t_Res);
return;
}
m_TemporarySkeletons.push_back(t_SklIndex);
// setup nodes and chains for the skeleton hand
if (!SetupHandNodes(t_SklIndex)) return;
if (!SetupHandChains(t_SklIndex)) return;
// load skeleton
uint32_t t_ID = 0;
t_Res = CoreSdk_LoadSkeleton(t_SklIndex, &t_ID);
if (t_Res != SDKReturnCode::SDKReturnCode_Success)
{
spdlog::error("Failed to load skeleton. The error given was {}.", t_Res);
return;
}
RemoveIndexFromTemporarySkeletonList(t_SklIndex);
if (t_ID == 0)
{
spdlog::error("Failed to give skeleton an ID.");
}
m_LoadedSkeletons.push_back(t_ID);
}
To have the skeleton move according to the hand movement of the first user you can set the settings.targetType to SkeletonTarget_UserIndexData and the settings.SkeletonTargetUserIndexData to 0. This means it will look at all the users in MANUS Core and take the first user's hands to animate the skeleton.
typedef struct SkeletonSettings
{
bool scaleToTarget; //default = false;
bool useEndPointApproximations; //default = false;
CollisionType collisionType; //default = CollisionType::CollisionType_None
SkeletonTargetType targetType;
SkeletonTargetUserData skeletonTargetUserData;
SkeletonTargetUserIndexData skeletonTargetUserIndexData;
SkeletonTargetAnimationData skeletonTargetAnimationData;
SkeletonTargetGloveData skeletonGloveData;
} SkeletonSettings;
There are several different target types which allow the user to choose how the skeleton is animated. The simplest is the user index, which we explained above, it requires very little knowledge of which gloves or users are available. When changing the target type make sure to match the targetType with the structure of the data, for example the SkeletonTargetType_GloveData enumerator requires the use of the SkeletonTargetGloveData structure.
The user data target allows you to specify a user ID which will enable the skeleton to be animated by the user specified, this does require you to know which users are available. Please keep in mind that the ID is not the same as the index, and you would need to find a certain user's ID in the landscape. The glove data target allows you to specify a glove ID. This ID will be used to find the correct glove and animate your skeleton according to the given glove. If this glove does not exist, the skeleton will not be animated.
To start adding the nodes and the chains to the skeleton setup we need to make it into a Temporary Skeleton by calling the CoreSdk_CreateSkeletonSetup function, which returns an index value which we can use to modify and load the skeleton.
Every skeleton needs a root node to define a point where everything originates from. The ID of the root node should be 0, and the parent of every other node should have a parent id.
In our example we setup our skeleton nodes with the following positions:
bool SDKClient::SetupHandNodes(uint32_t p_SklIndex)
{
// Define number of fingers per hand and number of joints per finger
const uint32_t t_NumFingers = 5;
const uint32_t t_NumJoints = 4;
// Create an array with the initial position of each hand node.
// Note, these values are just an example of node positions and refer to the hand laying on a flat surface.
ManusVec3 t_Fingers[t_NumFingers * t_NumJoints] = {
CreateManusVec3(0.024950f, 0.000000f, 0.025320f), //Thumb CMC joint
CreateManusVec3(0.000000f, 0.000000f, 0.032742f), //Thumb MCP joint
CreateManusVec3(0.000000f, 0.000000f, 0.028739f), //Thumb IP joint
CreateManusVec3(0.000000f, 0.000000f, 0.028739f), //Thumb Tip joint
//CreateManusVec3(0.011181f, 0.031696f, 0.000000f), //Index CMC joint // Note: we are not adding the matacarpal bones in this example, if you want to animate the metacarpals add each of them to the corresponding finger chain.
CreateManusVec3(0.011181f, 0.000000f, 0.052904f), //Index MCP joint, if metacarpal is present: CreateManusVec3(0.000000f, 0.000000f, 0.052904f)
CreateManusVec3(0.000000f, 0.000000f, 0.038257f), //Index PIP joint
CreateManusVec3(0.000000f, 0.000000f, 0.020884f), //Index DIP joint
CreateManusVec3(0.000000f, 0.000000f, 0.018759f), //Index Tip joint
//CreateManusVec3(0.000000f, 0.033452f, 0.000000f), //Middle CMC joint
CreateManusVec3(0.000000f, 0.000000f, 0.051287f), //Middle MCP joint
CreateManusVec3(0.000000f, 0.000000f, 0.041861f), //Middle PIP joint
CreateManusVec3(0.000000f, 0.000000f, 0.024766f), //Middle DIP joint
CreateManusVec3(0.000000f, 0.000000f, 0.019683f), //Middle Tip joint
//CreateManusVec3(-0.011274f, 0.031696f, 0.000000f), //Ring CMC joint
CreateManusVec3(-0.011274f, 0.000000f, 0.049802f), //Ring MCP joint, if metacarpal is present: CreateManusVec3(0.000000f, 0.000000f, 0.049802f),
CreateManusVec3(0.000000f, 0.000000f, 0.039736f), //Ring PIP joint
CreateManusVec3(0.000000f, 0.000000f, 0.023564f), //Ring DIP joint
CreateManusVec3(0.000000f, 0.000000f, 0.019868f), //Ring Tip joint
//CreateManusVec3(-0.020145f, 0.027538f, 0.000000f), //Pinky CMC joint
CreateManusVec3(-0.020145f, 0.000000f, 0.047309f), //Pinky MCP joint, if metacarpal is present: CreateManusVec3(0.000000f, 0.000000f, 0.047309f),
CreateManusVec3(0.000000f, 0.000000f, 0.033175f), //Pinky PIP joint
CreateManusVec3(0.000000f, 0.000000f, 0.018020f), //Pinky DIP joint
CreateManusVec3(0.000000f, 0.000000f, 0.019129f), //Pinky Tip joint
};
// skeleton entry is already done. just the nodes now.
// setup a very simple node hierarchy for fingers
// first setup the root node
//
// root, This node has ID 0 and parent ID 0, to indicate it has no parent.
SDKReturnCode t_Res = CoreSdk_AddNodeToSkeletonSetup(p_SklIndex, CreateNodeSetup(0, 0, 0, 0, 0, "Hand"));
if (t_Res != SDKReturnCode::SDKReturnCode_Success)
{
spdlog::error("Failed to Add Node To Skeleton Setup. The error given was {}.", t_Res);
return false;
}
// then loop for 5 fingers
int t_FingerId = 0;
for (uint32_t i = 0; i < t_NumFingers; i++)
{
uint32_t t_ParentID = 0;
// then the digits of the finger that are linked to the root of the finger.
for (uint32_t j = 0; j < t_NumJoints; j++)
{
t_Res = CoreSdk_AddNodeToSkeletonSetup(p_SklIndex, CreateNodeSetup(1 + t_FingerId + j, t_ParentID, t_Fingers[i * 4 + j].x, t_Fingers[i * 4 + j].y, t_Fingers[i * 4 + j].z, "fingerdigit"));
if (t_Res != SDKReturnCode::SDKReturnCode_Success)
{
printf("Failed to Add Node To Skeleton Setup. The error given %d.", t_Res);
return false;
}
t_ParentID = 1 + t_FingerId + j;
}
t_FingerId += t_NumJoints;
}
return true;
}
We use these positions when creating a NodeSetup for each of the nodes, as seen in the CreateNodeSetup function. The resulting structure is then passed into the CoreSdk_AddNodeToSkeletonSetup function which adds it to the Temporary Skeleton.
To have a functional hand skeleton we need to have a Hand chain which contains the ids of all finger chains, and it should have the node ID of a wrist along with which side the hand is from. The handMotion setting in the hand chain specifies if the wrist should move via IMU data or via tracker data, or not at all. Each of the needed chains can be added with the CoreSdk_AddChainToSkeletonSetup function. It is also possible to assign nodes to chains in the Dev Tools, we will explain this in more details during the Temporary Skeletons part of this article.
In our example we add a wrist and all 5 fingers:
bool SDKClient::SetupHandChains(uint32_t p_SklIndex)
{
// Add the Hand chain, this identifies the wrist of the hand
{
ChainSettings t_ChainSettings;
ChainSettings_Init(&t_ChainSettings);
t_ChainSettings.usedSettings = ChainType::ChainType_Hand;
t_ChainSettings.hand.handMotion = HandMotion::HandMotion_IMU;
t_ChainSettings.hand.fingerChainIdsUsed = 5; //we will have 5 fingers
t_ChainSettings.hand.fingerChainIds[0] = 1; //links to the other chains we will define further down
t_ChainSettings.hand.fingerChainIds[1] = 2;
t_ChainSettings.hand.fingerChainIds[2] = 3;
t_ChainSettings.hand.fingerChainIds[3] = 4;
t_ChainSettings.hand.fingerChainIds[4] = 5;
ChainSetup t_Chain;
ChainSetup_Init(&t_Chain);
t_Chain.id = 0; //Every ID needs to be unique per chain in a skeleton.
t_Chain.type = ChainType::ChainType_Hand;
t_Chain.dataType = ChainType::ChainType_Hand;
t_Chain.side = Side::Side_Left;
t_Chain.dataIndex = 0;
t_Chain.nodeIdCount = 1;
t_Chain.nodeIds[0] = 0; //this links to the hand node created in the SetupHandNodes
t_Chain.settings = t_ChainSettings;
SDKReturnCode t_Res = CoreSdk_AddChainToSkeletonSetup(p_SklIndex, t_Chain);
if (t_Res != SDKReturnCode::SDKReturnCode_Success)
{
spdlog::error("Failed to Add Chain To Skeleton Setup. The error given was {}.", t_Res);
return false;
}
}
// Add the 5 finger chains
const ChainType t_FingerTypes[5] = { ChainType::ChainType_FingerThumb,
ChainType::ChainType_FingerIndex,
ChainType::ChainType_FingerMiddle,
ChainType::ChainType_FingerRing,
ChainType::ChainType_FingerPinky };
for (int i = 0; i < 5; i++)
{
ChainSettings t_ChainSettings;
ChainSettings_Init(&t_ChainSettings);
t_ChainSettings.usedSettings = t_FingerTypes[i];
t_ChainSettings.finger.handChainId = 0; //This links to the wrist chain above.
//This identifies the metacarpal bone, if none exists, or the chain is a thumb it should be set to -1.
//The metacarpal bone should not be part of the finger chain, unless you are defining a thumb which does need it.
t_ChainSettings.finger.metacarpalBoneId = -1;
t_ChainSettings.finger.useLeafAtEnd = false; //this is set to true if there is a leaf bone to the tip of the finger.
ChainSetup t_Chain;
ChainSetup_Init(&t_Chain);
t_Chain.id = i + 1; //Every ID needs to be unique per chain in a skeleton.
t_Chain.type = t_FingerTypes[i];
t_Chain.dataType = t_FingerTypes[i];
t_Chain.side = Side::Side_Left;
t_Chain.dataIndex = 0;
if (i == 0) // Thumb
{
t_Chain.nodeIdCount = 4; //The amount of node id's used in the array
t_Chain.nodeIds[0] = 1; //this links to the hand node created in the SetupHandNodes
t_Chain.nodeIds[1] = 2; //this links to the hand node created in the SetupHandNodes
t_Chain.nodeIds[2] = 3; //this links to the hand node created in the SetupHandNodes
t_Chain.nodeIds[3] = 4; //this links to the hand node created in the SetupHandNodes
}
else // All other fingers
{
t_Chain.nodeIdCount = 4; //The amount of node id's used in the array
t_Chain.nodeIds[0] = (i * 4) + 1; //this links to the hand node created in the SetupHandNodes
t_Chain.nodeIds[1] = (i * 4) + 2; //this links to the hand node created in the SetupHandNodes
t_Chain.nodeIds[2] = (i * 4) + 3; //this links to the hand node created in the SetupHandNodes
t_Chain.nodeIds[3] = (i * 4) + 4; //this links to the hand node created in the SetupHandNodes
}
t_Chain.settings = t_ChainSettings;
SDKReturnCode t_Res = CoreSdk_AddChainToSkeletonSetup(p_SklIndex, t_Chain);
if (t_Res != SDKReturnCode::SDKReturnCode_Success)
{
return false;
}
}
return true;
}
Adding the finger chains goes in a similar process to the hand chain. We need to make sure to add all the node ids and set the handChainId to the same id as the one set in the hand chain, which is 0 in our example. Do not forget to set the correct side for the finger chains as well.
At this point we have a fully functioning Temporary Skeleton. This skeleton can either be loaded into the retargeting system of MANUS Core or sent to the Dev Tools for verification via the CoreSdk_SaveTemporarySkeleton function, which we will get more into in the Temporary Skeletons section of this article.
Retargeted Skeletons
After setting up a Temporary Skeleton via the method explained above the user can call the CoreSdk_LoadSkeleton function. This will load the Temporary Skeleton into MANUS Core's retargeting system and remove it from the list of Temporary Skeletons (because it is no longer temporary at this point). The ID returned from the load function is the one you should keep track of to match the skeleton stream data to the skeleton you loaded in.
It is also possible to unload a skeleton when you no longer need it to be animated. This prevents unnecessary calculations being done for a skeleton that may no longer be in use.
void SDKClient::UnloadTestSkeleton()
{
if (m_LoadedSkeletons.size() == 0)
{
spdlog::error("There was no skeleton for us to unload.");
return;
}
SDKReturnCode t_Res = CoreSdk_UnloadSkeleton(m_LoadedSkeletons[0]);
m_LoadedSkeletons.erase(m_LoadedSkeletons.begin());
if (t_Res != SDKReturnCode::SDKReturnCode_Success)
{
spdlog::error("Failed to unload skeleton. The error given was {}.", t_Res);
return;
}
}
In this example, if there are skeletons loaded, we can unload a skeleton by calling CoreSdk_UnloadSkeleton and passing the skeleton id.
When a client application is closed, the session will be closed, and MANUS Core will automatically unload all skeletons that belong to that session.
This can also happen when a connection is lost for any reason, but in such a case you can simply reload the skeletons.
Temporary Skeletons
When a skeleton is still being defined it cannot be animated fully yet, but it can be sent via Manus Core to the Dev Tools for ease of verification. The Dev Tools will automatically launch and load the temporary skeleton. For in-depth information about how to use the Dev Tools, you can look at the Dev Tools article in our knowledge center.
To do this a Temporary Skeleton should be set up, which is like a regular skeleton, however not all the chains need to be defined.
The Dev Tools can send an updated Temporary Skeleton back (depending on what the user did in the Dev Tools) which can be used to create a regular skeleton for animation.
You can see this functionality in both the Unreal Engine and Unity plugins, where this is used to verify the animation.
If the Temporary Skeleton is not destroyed, it can be updated easily by the Dev Tools. If a network connection with MANUS core is lost, the temporary skeleton needs to be resent for the plugins to be able to work on it.
If the skeleton is well defined, you can also try to automatically allocate the chains with the CoreSdk_AllocateChainsForSkeletonSetup function, however this can be quite opaque without a visualization, so i t is not advised to do this outside the Dev Tools.
See the BuildTemporarySkeleton function in the sample client for an example of this process.
A temporary skeleton can also be transformed into a file and vice versa for ease of handling, these files are saved as an mskl file. These binary files can be loaded into MANUS Core, this might make it easier for you to adjust the data later.
An example of saving and loading mskl can be found in the SaveTemporarySkeletonToFile and GetTemporarySkeletonFromFile functions.
void SDKClient::SaveTemporarySkeletonToFile()
{
// this example shows how to save a temporary skeleton to a file
// first create a temporary skeleton:
// define the session Id for which we want to save
uint32_t t_SessionId = m_SessionId;
bool t_IsSkeletonModified = false; // setting this bool to true is not necessary here, it is mostly used by the Dev Tools
// to notify the SDK sessions about their skeleton being modified.
// first create a skeleton setup
uint32_t t_SklIndex = 0;
SkeletonSetupInfo t_SKL;
SkeletonSetupInfo_Init(&t_SKL);
t_SKL.type = SkeletonType::SkeletonType_Hand;
t_SKL.settings.scaleToTarget = true;
t_SKL.settings.targetType = SkeletonTargetType::SkeletonTargetType_GloveData;
t_SKL.settings.skeletonTargetUserIndexData.userIndex = 0;
CopyString(t_SKL.name, sizeof(t_SKL.name), std::string("LeftHand"));
SDKReturnCode t_Res = CoreSdk_CreateSkeletonSetup(t_SKL, &t_SklIndex);
if (t_Res != SDKReturnCode::SDKReturnCode_Success)
{
spdlog::error("Failed to Create Skeleton Setup. The error given was {}.", t_Res);
return;
}
m_TemporarySkeletons.push_back(t_SklIndex);
// setup nodes and chains for the skeleton hand
if (!SetupHandNodes(t_SklIndex)) return;
if (!SetupHandChains(t_SklIndex)) return;
// save the temporary skeleton
t_Res = CoreSdk_SaveTemporarySkeleton(t_SklIndex, t_SessionId, t_IsSkeletonModified);
if (t_Res != SDKReturnCode::SDKReturnCode_Success)
{
spdlog::error("Failed to save temporary skeleton. The error given was {}.", t_Res);
return;
}
// now compress the temporary skeleton data and get the size of the compressed data:
uint32_t t_TemporarySkeletonLengthInBytes;
t_Res = CoreSdk_CompressTemporarySkeletonAndGetSize(t_SklIndex, t_SessionId, &t_TemporarySkeletonLengthInBytes);
if (t_Res != SDKReturnCode::SDKReturnCode_Success)
{
spdlog::error("Failed to compress temporary skeleton and get size. The error given was {}.", t_Res);
return;
}
unsigned char* t_TemporarySkeletonData = new unsigned char[t_TemporarySkeletonLengthInBytes];
// get the array of bytes with the compressed temporary skeleton data, remember to always call function CoreSdk_CompressTemporarySkeletonAndGetSize
// before trying to get the compressed temporary skeleton data
t_Res = CoreSdk_GetCompressedTemporarySkeletonData(t_TemporarySkeletonData, t_TemporarySkeletonLengthInBytes);
if (t_Res != SDKReturnCode::SDKReturnCode_Success)
{
spdlog::error("Failed to get compressed temporary skeleton data. The error given was {}.", t_Res);
return;
}
// now save the data into a .mskl file
// as an example we save the temporary skeleton in a folder called ManusTemporarySkeleton inside the documents directory
// get the path for the documents directory
std::string t_DirectoryPathString = GetDocumentsDirectoryPath_UTF8();
// create directory name and file name for storing the temporary skeleton
std::string t_DirectoryPath =
t_DirectoryPathString
+ s_SlashForFilesystemPath
+ "ManusTemporarySkeleton";
CreateFolderIfItDoesNotExist(t_DirectoryPath);
std::string t_DirectoryPathAndFileName =
t_DirectoryPath
+ s_SlashForFilesystemPath
+ "TemporarySkeleton.mskl";
// write the temporary skeleton data to .mskl file
std::ofstream t_File = GetOutputFileStream(t_DirectoryPathAndFileName);
t_File.write((char*)t_TemporarySkeletonData, t_TemporarySkeletonLengthInBytes);
t_File.close();
t_Res = CoreSdk_ClearTemporarySkeleton(t_SklIndex, t_SessionId);
if (t_Res != SDKReturnCode::SDKReturnCode_Success)
{
spdlog::error("Failed to Clear Temporary Skeleton after saving. The error given was {}.", t_Res);
return;
}
RemoveIndexFromTemporarySkeletonList(t_SklIndex);
}
The example first sets up a temporary skeleton and saves it to Manus Core via the CoreSdk_SaveTemporarySkeleton function. After that it can get the number of bytes needed to save the data from CoreSdk_CompressTemporarySkeletonAndGetSize and you can call the CoreSdk_GetCompressedTemporarySkeletonData function to get the actual data into the byte array. In the end it saves it to file like a normal byte array. In the example the temporary skeleton is also cleared and removed, but this is only for illustration purposes and not required if you plan on using the skeleton.
Loading a temporary skeleton from a file is like saving one, in the example function GetTemporarySkeletonFromFile you can see the process of reading a file out and sending it to MANUS Core.
The byte array you read out of the file can be sent to MANUS Core via the CoreSdk_GetTemporarySkeletonFromCompressedData function, which will create a new temporary skeleton and return the ID of that skeleton.
Trackers
Manus Core is able to ingest positional and rotation data from various tracking systems (for example ART and SteamVR) and can be used to set up a skeleton for the whole body. For this, they must be assigned to a user and configured to represent a limb or body part location. For more details on this, please check https://www.manus-meta.com/resources/knowledge-base. However, getting the data for trackers can still be useful for props etc. You may also want to add your own tracker system and be able to stream the tracker data back into MANUS core.
You can access tracker information via the data gotten via the landscape callback or via a few functions we made specifically for the tracker information.
Showing trackers per user or globally:
void SDKClient::PrintTrackerDataPerUser()
{
uint32_t t_NumberOfAvailableUsers = 0;
SDKReturnCode t_UserResult = CoreSdk_GetNumberOfAvailableUsers(&t_NumberOfAvailableUsers);
if (t_UserResult != SDKReturnCode::SDKReturnCode_Success)
{
spdlog::error("Failed to get user count. The error given was {}.", t_UserResult);
return;
}
if (t_NumberOfAvailableUsers == 0) return; // nothing to get yet
for (uint32_t i = 0; i < t_NumberOfAvailableUsers; i++)
{
uint32_t t_NumberOfAvailabletrackers = 0;
SDKReturnCode t_TrackerResult = CoreSdk_GetNumberOfAvailableTrackersForUserIndex(&t_NumberOfAvailabletrackers, i);
if (t_TrackerResult != SDKReturnCode::SDKReturnCode_Success)
{
spdlog::error("Failed to get tracker data. The error given was {}.", t_TrackerResult);
return;
}
if (t_NumberOfAvailabletrackers == 0) continue;
spdlog::info("received available trackers for user index[{}] :{} ", i, t_NumberOfAvailabletrackers);
if (t_NumberOfAvailabletrackers == 0) return; // nothing else to do.
TrackerId* t_TrackerId = new TrackerId[t_NumberOfAvailabletrackers];
t_TrackerResult = CoreSdk_GetIdsOfAvailableTrackersForUserIndex(t_TrackerId, i, t_NumberOfAvailabletrackers);
if (t_TrackerResult != SDKReturnCode::SDKReturnCode_Success)
{
spdlog::error("Failed to get tracker data. The error given was {}.", t_TrackerResult);
return;
}
}
}
In this code example all users are first gathered and then the tracker IDs per user are counted and displayed.
For normal body tracking the trackers are usually assigned to a user. Unassigned trackers will need to be found with the global function CoreSdk_GetNumberOfAvailableTrackers followed by the CoreSdk_GetIdsOfAvailableTrackers function.
void SDKClient::PrintTrackerDataGlobal()
{
uint32_t t_NumberOfAvailabletrackers = 0;
SDKReturnCode t_TrackerResult = CoreSdk_GetNumberOfAvailableTrackers(&t_NumberOfAvailabletrackers);
if (t_TrackerResult != SDKReturnCode::SDKReturnCode_Success)
{
spdlog::error("Failed to get tracker data. The error given was {}.", t_TrackerResult);
return;
}
spdlog::info("received available trackers :{} ", t_NumberOfAvailabletrackers);
if (t_NumberOfAvailabletrackers == 0) return; // nothing else to do.
TrackerId* t_TrackerId = new TrackerId[t_NumberOfAvailabletrackers];
t_TrackerResult = CoreSdk_GetIdsOfAvailableTrackers(t_TrackerId, t_NumberOfAvailabletrackers);
if (t_TrackerResult != SDKReturnCode::SDKReturnCode_Success)
{
spdlog::error("Failed to get tracker data. The error given was {}.", t_TrackerResult);
return;
}
}
No actual tracker data or meta data is being shown here, but it can be retrieved with the CoreSdk_GetDataForTracker_UsingTrackerId function. Using the meta data, you can determine if it is an unassigned tracker or not in case you want to use a tracker for something else than body tracking.
Adding and removing custom trackers
void SDKClient::HandleTrackerCommands()
{
if (GetKeyDown('O'))
{
m_TrackerTest = !m_TrackerTest;
}
if (GetKeyDown('G'))
{
m_TrackerDataDisplayPerUser = !m_TrackerDataDisplayPerUser;
}
if (m_TrackerTest)
{
m_TrackerOffset += 0.0005f;
if (m_TrackerOffset >= 10.0f)
{
m_TrackerOffset = 0.0f;
}
TrackerId t_TrackerId;
CopyString(t_TrackerId.id, sizeof(t_TrackerId.id), std::string("Test Tracker"));
TrackerData t_TrackerData = {};
t_TrackerData.isHmd = false;
t_TrackerData.trackerId = t_TrackerId;
t_TrackerData.trackerType = TrackerType::TrackerType_Unknown;
t_TrackerData.position = { 0.0f, m_TrackerOffset, 0.0f };
t_TrackerData.rotation = { 1.0f, 0.0f, 0.0f, 0.0f };
t_TrackerData.quality = TrackerQuality::TrackingQuality_Trackable;
TrackerData t_TrackerDatas[MAX_NUMBER_OF_TRACKERS];
t_TrackerDatas[0] = t_TrackerData;
const SDKReturnCode t_TrackerSend = CoreSdk_SendDataForTrackers(t_TrackerDatas, 1);
if (t_TrackerSend != SDKReturnCode::SDKReturnCode_Success)
{
spdlog::error("Failed to send tracker data. The error given was {}.", t_TrackerSend);
return;
}
}
}
In this code example a simple TestTracker is set up and passed into MANUS Core by passing an array of trackers into the CoreSdk_SendDataForTrackers function. As a test, the position is slightly altered every update to show it moving and is visualized in the MANUS Core Dashboard if it is also running. When you make your own trackers make sure every tracker has a unique ID and is trackable.
In this test only one tracker is being sent, but it is advisable for synchronization to send all custom trackers in the same array at the same time.
Timecode
A timecode generator is best configured via the MANUS Core Dashboard . However, you might still be interested in the timecode settings in your own client. For this you can use the functions shown in the PrintLandscapeTimeData function of the example client.
void SDKClient::PrintLandscapeTimeData()
{
spdlog::info("Total count of Interfaces: {}", m_Landscape->time.interfaceCount);
spdlog::info("Current Interface: {} {} at index {}", m_Landscape->time.currentInterface.name, m_Landscape->time.currentInterface.api, m_Landscape->time.currentInterface.index);
spdlog::info("FPS: {}", GetFPSEnumName(m_Landscape->time.fps));
spdlog::info("Fake signal: {} | Sync Pulse: {} | Sync Status: {}", m_Landscape->time.fakeTimecode, m_Landscape->time.useSyncPulse, m_Landscape->time.syncStatus);
spdlog::info("Device keep alive: {} | Timecode Status: {}", m_Landscape->time.deviceKeepAlive, m_Landscape->time.timecodeStatus);
AdvanceConsolePosition(6);
}
If the landscape callback is registered, you can read out the landscape data to retrieve the time landscape with timecode information.
Due to thread safety, this is copied during a mutex to a local copy of the data and eventually printed in the UI thread via the PrintLandscapeTimeData function.
For more information on timecode, please consult your timecode device manual.
Gestures
MANUS Core has a built-in gesture system. Data from this gesture system can be accessed via the Gesture Stream. In our example in the function OnGestureStreamCallback you can see how to receive the data:
void SDKClient::OnGestureStreamCallback(const GestureStreamInfo* const p_GestureStream)
{
if (s_Instance)
{
for (uint32_t i = 0; i < p_GestureStream->gestureProbabilitiesCount; i++)
{
GestureProbabilities t_Probs;
CoreSdk_GetGestureStreamData(i, 0, &t_Probs);
if (t_Probs.isUserID)continue;
if (t_Probs.id != s_Instance->m_FirstLeftGloveID && t_Probs.id != s_Instance->m_FirstRightGloveID)continue;
ClientGestures* t_Gest = new ClientGestures();
t_Gest->info = t_Probs;
t_Gest->probabilities.reserve(t_Gest->info.totalGestureCount);
uint32_t t_BatchCount = (t_Gest->info.totalGestureCount / MAX_GESTURE_DATA_CHUNK_SIZE) + 1;
uint32_t t_ProbabilityIdx = 0;
for (uint32_t b = 0; b < t_BatchCount; b++)
{
for (uint32_t j = 0; j < t_Probs.gestureCount; j++)
{
t_Gest->probabilities.push_back(t_Probs.gestureData[j]);
}
t_ProbabilityIdx += t_Probs.gestureCount;
CoreSdk_GetGestureStreamData(i, t_ProbabilityIdx, &t_Probs); //this will get more data, if needed for the next iteration.
}
s_Instance->m_GestureMutex.lock();
if (t_Probs.id == s_Instance->m_FirstLeftGloveID)
{
if (s_Instance->m_NewFirstLeftGloveGestures != nullptr) delete s_Instance->m_NewFirstLeftGloveGestures;
s_Instance->m_NewFirstLeftGloveGestures = t_Gest;
}
else
{
if (s_Instance->m_NewFirstRightGloveGestures != nullptr) delete s_Instance->m_NewFirstRightGloveGestures;
s_Instance->m_NewFirstRightGloveGestures = t_Gest;
}
s_Instance->m_GestureMutex.unlock();
}
}
}
The stream info gives you the number of gestures available per glove. Using the CoreSdk_GetGestureStreamData function you can get the specific gesture information for a given glove. Due to the amount of potential gesture data in the future, it is advisable to call this function in the callback to gather all the gesture data. This way it becomes impossible for the data to be modified by the SDK during this function call. In our example we gather the gesture data for the first left and right glove, we display this data to the console:
void SDKClient::PrintGestureData()
{
ClientGestures* t_Gest = m_FirstLeftGloveGestures;
std::string t_Side = "Left";
if (!m_ShowLeftGestures)
{
t_Side = "Right";
t_Gest = m_FirstRightGloveGestures;
}
if (t_Gest == nullptr)
{
spdlog::info("No Gesture information for first {} glove.", t_Side);
AdvanceConsolePosition(3);
return;
}
spdlog::info("Total count of gestures for the {} glove: {}", t_Side, t_Gest->info.totalGestureCount);
uint32_t t_Max = t_Gest->info.totalGestureCount;
if (t_Max > 20) t_Max = 20;
spdlog::info("Showing result of first {} gestures.", t_Max);
for (uint32_t i = 0; i < t_Max; i++)
{
char* t_Name = "";
for (uint32_t g = 0; g < m_GestureLandscapeData.size(); g++)
{
if (m_GestureLandscapeData[g].id == t_Gest->probabilities[i].id)
{
t_Name = m_GestureLandscapeData[g].name;
}
}
spdlog::info("Gesture {} ({}) has a probability of {}%.", t_Name,t_Gest->probabilities[i].id, t_Gest->probabilities[i].percent * 100.0f);
}
AdvanceConsolePosition(6);
}
The gesture data you receive contains ID and normalized percentage numbers (from 0.0 to 1.0). The gesture ID can be matched to the gesture name shown in the landscape data for gestures, which will allow you to see what gesture it is.
Currently it is not supported to add custom gestures to MANUS Core , when this does become possible it is best to simply remember what IDs are assigned to which gesture. In predefined gestures, the IDs will remain the same throughout the application runtime, so you will only need to match the gesture's name to its ID once if you wish to use those names for identification.