License requirement
The functionality described requires a MANUS Bodypack or a MANUS license dongle with one of the following licenses:
Core Pro Core XR Core Xsens Pro Core Qualisys Pro Core OptiTrack Pro Demo, or a Feature license with the SDK feature enabled.
SDK Client example
Introduction
The SDK Client example functions as a demonstration of all major SDK functions. It is a simple console application that shows how to connect to MANUS Core, how to set up the coordinate system and how to handle the landscape, gloves, dongles, skeletons, gestures, trackers and time tracking. The example client is not meant to be a full-fledged application, but rather a demonstration of how to use the SDK functions. The example client is written in C++ and uses the MANUS Core SDK.
SDK Callbacks
All the data streams MANUS Core provides are structured as callbacks. A callback is a function that you can pass into a CoreSdk_RegisterCallback... function. These functions call them at another point (on another thread) when a certain event occurs. This way users do not need to poll the SDK to figure out if there is new data available.
It is good practice to register all the callbacks you require after initializing the SDK and before connecting to MANUS Core. You only need to register the callbacks which you intend to use, if you do not plan to use certain data streams this could decrease the traffic from MANUS Core.
void SDKClient::OnErgonomicsCallback(const ErgonomicsStream* const p_Ergo)
{
if (s_Instance)
{
for (uint32_t i = 0; i < p_Ergo->dataCount; i++)
{
if (p_Ergo->data[i].isUserID)continue;
ErgonomicsData* t_Ergo = nullptr;
if (p_Ergo->data[i].id == s_Instance->m_FirstLeftGloveID)
{
t_Ergo = &s_Instance->m_LeftGloveErgoData;
}
if (p_Ergo->data[i].id == s_Instance->m_FirstRightGloveID)
{
t_Ergo = &s_Instance->m_RightGloveErgoData;
}
if (t_Ergo == nullptr)continue;
CoreSdk_GetTimestampInfo(p_Ergo->publishTime, &s_Instance->m_ErgoTimestampInfo);
t_Ergo->id = p_Ergo->data[i].id;
t_Ergo->isUserID = p_Ergo->data[i].isUserID;
for (int j = 0; j < ErgonomicsDataType::ErgonomicsDataType_MAX_SIZE; j++)
{
t_Ergo->data[j] = p_Ergo->data[i].data[j];
}
}
}
}
This callback is called whenever the SDK receives new ergonomic data from MANUS Core. A best practice for data being received on another thread (which happens with the SDK's callback mechanism) is to save the data somewhere and then process whatever you want to do with the data on your own thread. This way you do not block/delay the callback thread, which could lead to delayed data transfer.
In several of our streams, we add timestamps to show when the data was captured. To get a more readable timestamp you need to call the CoreSdk_GetTimestampInfo on the timestamp value, this will translate the timestamp value to a date & time format.
Client Connection
There are two modes to connect to your glove devices. What we call integrated and what we call remote. The integrated method will talk directly to the gloves without the need for a MANUS Core instance. The remote method will talk to a MANUS Core instance which will then talk to the gloves. The integrated method is usually used for applications that want to handle all glove functionality themselves and deeply integrate the MANUS gloves into their application. The remote method is usually used for applications that want to use the MANUS Core Dashboard to handle the gloves and dongles and only want to receive the data from the gloves.
On startup the choice is given between:
- Core Integrated mode: The SDK is integrated into the client application.
- Core Local mode: The SDK will connect to a MANUS Core running locally on this machine.
- Core Remote mode: The SDK will search and connected to a MANUS Core instance on the network.
When using Remote, the network is scanned, and a list of available MANUS Core instances is returned. To explore the code behind our example client, consider the following snippet for finding hosts:
ClientReturnCode SDKClient::LookingForHosts()
{
ClientLog::print("Looking for hosts...");
// Underlying function will sleep for m_SecondsToFindHosts to allow servers to reply.
bool t_ConnectLocally = m_ConnectionType == ConnectionType::ConnectionType_Local;
const SDKReturnCode t_StartResult = CoreSdk_LookForHosts(m_SecondsToFindHosts, t_ConnectLocally);
if (t_StartResult != SDKReturnCode::SDKReturnCode_Success)
{
ClientLog::error("Failed to look for hosts. The error given was {}.", (int32_t)t_StartResult);
return ClientReturnCode::ClientReturnCode_FailedToFindHosts;
}
m_NumberOfHostsFound = 0;
const SDKReturnCode t_NumberResult = CoreSdk_GetNumberOfAvailableHostsFound(&m_NumberOfHostsFound);
if (t_NumberResult != SDKReturnCode::SDKReturnCode_Success)
{
ClientLog::error("Failed to get the number of available hosts. The error given was {}.", (int32_t)t_NumberResult);
return ClientReturnCode::ClientReturnCode_FailedToFindHosts;
}
if (m_NumberOfHostsFound == 0)
{
ClientLog::warn("No hosts found.");
m_State = ClientState::ClientState_NoHostsFound;
return ClientReturnCode::ClientReturnCode_FailedToFindHosts;
}
m_AvailableHosts.reset(new ManusHost[m_NumberOfHostsFound]);
const SDKReturnCode t_HostsResult = CoreSdk_GetAvailableHostsFound(m_AvailableHosts.get(), m_NumberOfHostsFound);
if (t_HostsResult != SDKReturnCode::SDKReturnCode_Success)
{
ClientLog::error("Failed to get the available hosts. The error given was {}.", (int32_t)t_HostsResult);
return ClientReturnCode::ClientReturnCode_FailedToFindHosts;
}
if (t_ConnectLocally)
{
m_State = ClientState::ClientState_ConnectingToCore;
return ClientReturnCode::ClientReturnCode_Success;
}
m_State = ClientState::ClientState_PickingHost;
return ClientReturnCode::ClientReturnCode_Success;
}
To find all the available MANUS Core instances, the CoreSdk_LookForHosts function needs to be called. This function call is a blocking function and therefore will block the calling thread for the specified duration. We advise not running any time sensitive functions or UI in the same thread as the Manus SDK connection.
After this function has completed you can get the number of hosts found by calling the CoreSdk_GetNumberOfAvailableHostsFound function. If this function returns 0 hosts, either no MANUS core is available, or the network is being blocked by a security system or another program.
The next step is creating an array at least the size of the number of found hosts and passing this array into the CoreSdk_GetAvailableHostsFound function. This function will put information about all the hosts that are found on the network into the array.
Once you have at least one host to connect to you can connect to it with code like this:
ClientReturnCode SDKClient::ConnectingToCore()
{
SDKReturnCode t_ConnectResult = SDKReturnCode::SDKReturnCode_Error;
switch (m_ConnectionType)
{
case ConnectionType::ConnectionType_Integrated:
{
ManusHost t_Empty;
ManusHost_Init(&t_Empty);
t_ConnectResult = CoreSdk_ConnectToHost(t_Empty);
break;
}
case ConnectionType::ConnectionType_Local:
{
t_ConnectResult = CoreSdk_ConnectToHost(m_AvailableHosts[0]);
break;
}
case ConnectionType::ConnectionType_Remote:
{
t_ConnectResult = CoreSdk_ConnectToHost(m_AvailableHosts[m_HostToConnectTo]);
break;
}
}
if (t_ConnectResult == SDKReturnCode::SDKReturnCode_NotConnected)
{
m_State = ClientState::ClientState_NoHostsFound;
return ClientReturnCode::ClientReturnCode_Success; // Differentiating between error and no connect
}
if (t_ConnectResult != SDKReturnCode::SDKReturnCode_Success)
{
ClientLog::error("Failed to connect to Core. The error given was {}.", (int32_t)t_ConnectResult);
return ClientReturnCode::ClientReturnCode_FailedToConnect;
}
m_State = ClientState::ClientState_DisplayingData;
// Note: a log message from somewhere in the SDK during the connection process can cause text
// to permanently turn green after this step. Adding a sleep here of 2+ seconds "fixes" the
// issue. It seems to be caused by a threading issue somewhere, resulting in a log call being
// interrupted while it is printing the green [info] text. The log output then gets stuck in
// green mode.
return ClientReturnCode::ClientReturnCode_Success;
}
The chosen host structure needs to be put into the CoreSdk_ConnectToHost function. This function then attempts to connect to the given host and will trigger the OnConnectedCallback callback when it is successful. This callback is registered via the CoreSdk_RegisterCallbackForOnConnect function.
If a connection to MANUS Core is lost the OnDisconnected callback is invoked. This one can be registered via the CoreSdk_RegisterCallbackForOnDisconnect function.
Coordinate system Initialization
After registering the callbacks, the coordinate system used by the client is set using the CoreSdk_InitializeCoordinateSystemWithVUH() function. In this example, a z-up, x-positive, right-handed coordinate system is used. It is recommended to align the coordinate system with the application being developed, as this will make it easier to work with the data by letting MANUS Core handle the conversion.
There are two ways to set up the coordinate system: using the VUH (View, Up, Handedness) system or using the Direction system.
CoordinateSystemVUH t_VUH;
CoordinateSystemVUH_Init(&t_VUH);
t_VUH.handedness = Side::Side_Right;
t_VUH.up = AxisPolarity::AxisPolarity_PositiveZ;
t_VUH.view = AxisView::AxisView_XFromViewer;
t_VUH.unitScale = 1.0f; //1.0 is meters, 0.01 is cm, 0.001 is mm.
// The above specified coordinate system is used to initialize and the coordinate space is specified (world vs local).
const SDKReturnCode t_CoordinateResult = CoreSdk_InitializeCoordinateSystemWithVUH(t_VUH, true);
// this is an example of an alternative way of setting up the coordinate system instead of VUH (view, up, handedness)
CoordinateSystemDirection t_Direction;
t_Direction.x = AxisDirection::AD_Right;
t_Direction.y = AxisDirection::AD_Up;
t_Direction.z = AxisDirection::AD_Forward;
const SDKReturnCode t_InitializeResult = CoreSdk_InitializeCoordinateSystemWithDirection(t_Direction, true);
The unit scale is specified as a float value. A scale of 1 represents meters, 0.01 represents centimeters, and 0.001 represents millimeters.
The second parameter of the CoreSdk_InitializeCoordinateSystemWithVUH() function indicates whether the coordinates should be set as world coordinates or relative coordinates. In this case, it is set to false to use relative coordinates.
Main menu
To make the example client easier to navigate and our data easier to observe, we made a menu system to view various parts of the data available. None of the code used to display or navigate through the menus are needed to use the SDK, all data can be accessed simultaneously.
Landscape
The landscape is a record of the current state of devices, users, trackers, gestures etcetera. The callback happens once every second. Our sample client does not use all the information given via the landscape to keep the example a bit easier to follow.
void SDKClient::OnLandscapeCallback(const Landscape* const p_Landscape)
{
if (s_Instance == nullptr)return;
Landscape* t_Landscape = new Landscape(*p_Landscape);
s_Instance->m_LandscapeMutex.lock();
if (s_Instance->m_NewLandscape != nullptr) delete s_Instance->m_NewLandscape;
s_Instance->m_NewLandscape = t_Landscape;
s_Instance->m_NewGestureLandscapeData.resize(t_Landscape->gestureCount);
CoreSdk_GetGestureLandscapeData(s_Instance->m_NewGestureLandscapeData.data(), (uint32_t)s_Instance->m_NewGestureLandscapeData.size());
s_Instance->m_LandscapeMutex.unlock();
}
If you plan to use the gestures and need the landscape data related to these, make sure to call the CoreSdk_GetGestureLandscapeData to retrieve all the data needed for the gestures from within the callback to ensure that the entirety of the landscape is stored correctly.
An overview of most of the landscape is displayed in our landscape viewer tool, which can be found in the MANUS Core installation directory as ManusLandscapeView.exe.
The entire Landscape structure is self-explanatory, the gloves and dongle information can be found under the gloveDevices variable. You can find the battery information, the pairing state of the glove, the type of glove, haptics, etc. inside the GloveLandscapeData structure. In the DongleLandscapeData structure you can find information such as the dongle type, firmware versions, which gloves are paired to it, etc. In the users you can find information such as which dongle and gloves are assigned to which user.
Gloves & Dongles
The code in the DisplayingDataGlove function handles the console logging, the HandleHapticCommands function shows the way haptics are handled when using a glove ID to identify which glove needs to vibrate.
In this example two gloves are connected, and their ergonomics data (received via the callback) is shown with a timestamp. If the gloves are haptic enabled, you can use the number keys to trigger the individual haptic actuators on the fingers.
This view also shows which dongles are present and the licenses on them.
void SDKClient::OnErgonomicsCallback(const ErgonomicsStream* const p_Ergo)
{
if (s_Instance)
{
for (uint32_t i = 0; i < p_Ergo->dataCount; i++)
{
if (p_Ergo->data[i].isUserID)continue;
ErgonomicsData* t_Ergo = nullptr;
if (p_Ergo->data[i].id == s_Instance->m_FirstLeftGloveID)
{
t_Ergo = &s_Instance->m_LeftGloveErgoData;
}
if (p_Ergo->data[i].id == s_Instance->m_FirstRightGloveID)
{
t_Ergo = &s_Instance->m_RightGloveErgoData;
}
if (t_Ergo == nullptr)continue;
CoreSdk_GetTimestampInfo(p_Ergo->publishTime, &s_Instance->m_ErgoTimestampInfo);
t_Ergo->id = p_Ergo->data[i].id;
t_Ergo->isUserID = p_Ergo->data[i].isUserID;
for (int j = 0; j < ErgonomicsDataType::ErgonomicsDataType_MAX_SIZE; j++)
{
t_Ergo->data[j] = p_Ergo->data[i].data[j];
}
}
}
}
In the ergonomics callback we filter on the first user`s gloves so as not to bloat the sample code, in normal cases you usually want all the information contained in this callback when using ergonomics.
Skeletons
MANUS Core uses something we've named skeletons to animate the hands and bodies. An important part of the skeleton setup are nodes, which you can consider as being the joints (the point around which your finger bone rotates) of your fingers.
Another important part of the skeleton setup are the chains, these define which nodes belong to which part of the skeleton. For example, your thumb chain would contain all the nodes that make up your thumb's bones. There are three different skeletons in MANUS Core, Temporary Skeletons, Retargeted Skeletons and Raw Skeletons. Temporary Skeletons are the skeletons that you create and send to the development tools for verification, but you do not load them into MANUS Core's retargeting system. Retargeted Skeletons are usually called Skeletons in the SDK, which are loaded into MANUS Core with the animation data applied. The Raw Skeletons are hand models on which MANUS Core applies the hand data without doing any retargeting, these are not necessarily using the same bone hierarchy structure as the skeletons that the user loads into MANUS Core.
Skeletal Data
MANUS Core supplies two streams of skeletal data. The first being retargeted skeletal data, this is the glove data applied onto a skeleton of the user's choosing. The second stream is the raw skeletal data, this is the glove data applied onto a standardized MANUS hand skeleton. Depending on your application's needs you can use one stream or the other, or in some cases even both.
The retargeted skeletal data is useful when you have your own skeleton or hand model, and you would like things like pinches to look better than they would when simply applying rotations from the raw skeleton stream. Using retargeted skeletal data allows you to create hands that have varying sizings and structures compared to the glove wearer's hand.
The raw skeletal data is generally the more accurate representation of the glove wearer's hand. The rotations and positions output in the raw skeleton stream are usually closer to the real world hand of a person, but applying this data directly to your desired skeleton or hand model can be a much more challenging task and will not guarantee lifelike animation. When wanting to include any wrist positioning and orientation in the raw skeleton stream the global source can be specified using the CoreSDK_SetRawSkeletonHandMotion() function. This can be set to any of the HandMotion enum values (HandMotion_None, HandMotion_Auto, HandMotion_Tracker, HandMotion_Tracker_RotationOnly, HandMotion_IMU).
HandMotion_None: No hand motion data is used.HandMotion_Auto: Automatically uses tracker data if a tracker is assigned to that hand in MANUS Core Dashboard; otherwise, it uses glove IMU data.HandMotion_Tracker: Uses a tracker's orientation and position if available.HandMotion_Tracker_RotationOnly: Uses the tracker's orientation without its position.HandMotion_IMU: Uses the gloves' IMU orientation.
The closer the dimensions are to a real-life hand, the smaller the difference is between the retargeted and the raw skeletal data, at which point it tends to be much easier to just use the retargeted skeletal data for your skeleton or hand model. Retargeted skeletal data needs to be set up using the Skeleton system which requires more effort.
Both skeletal stream callbacks function in a similar way, and can be seen in the example client:
/// @brief This gets called when the client is connected to manus core
/// @param p_SkeletonStreamInfo contains the meta data on how much data regarding the skeleton we need to get from the SDK.
void SDKClient::OnSkeletonStreamCallback(const SkeletonStreamInfo* const p_SkeletonStreamInfo)
{
if (s_Instance)
{
ClientSkeletonCollection* t_NxtClientSkeleton = new ClientSkeletonCollection();
t_NxtClientSkeleton->skeletons.resize(p_SkeletonStreamInfo->skeletonsCount);
for (uint32_t i = 0; i < p_SkeletonStreamInfo->skeletonsCount; i++)
{
CoreSdk_GetSkeletonInfo(i, &t_NxtClientSkeleton->skeletons[i].info);
t_NxtClientSkeleton->skeletons[i].nodes.resize(t_NxtClientSkeleton->skeletons[i].info.nodesCount);
t_NxtClientSkeleton->skeletons[i].info.publishTime = p_SkeletonStreamInfo->publishTime;
CoreSdk_GetSkeletonData(i, t_NxtClientSkeleton->skeletons[i].nodes.data(), t_NxtClientSkeleton->skeletons[i].info.nodesCount);
}
s_Instance->m_SkeletonMutex.lock();
if (s_Instance->m_NextSkeleton != nullptr) delete s_Instance->m_NextSkeleton;
s_Instance->m_NextSkeleton = t_NxtClientSkeleton;
s_Instance->m_SkeletonMutex.unlock();
}
}
void SDKClient::OnRawSkeletonStreamCallback(const SkeletonStreamInfo* const p_RawSkeletonStreamInfo)
{
if (s_Instance)
{
ClientRawSkeletonCollection* t_NxtClientRawSkeleton = new ClientRawSkeletonCollection();
t_NxtClientRawSkeleton->skeletons.resize(p_RawSkeletonStreamInfo->skeletonsCount);
for (uint32_t i = 0; i < p_RawSkeletonStreamInfo->skeletonsCount; i++)
{
CoreSdk_GetRawSkeletonInfo(i, &t_NxtClientRawSkeleton->skeletons[i].info);
t_NxtClientRawSkeleton->skeletons[i].nodes.resize(t_NxtClientRawSkeleton->skeletons[i].info.nodesCount);
t_NxtClientRawSkeleton->skeletons[i].info.publishTime = p_RawSkeletonStreamInfo->publishTime;
CoreSdk_GetRawSkeletonData(i, t_NxtClientRawSkeleton->skeletons[i].nodes.data(), t_NxtClientRawSkeleton->skeletons[i].info.nodesCount);
}
s_Instance->m_RawSkeletonMutex.lock();
if (s_Instance->m_NextRawSkeleton != nullptr) delete s_Instance->m_NextRawSkeleton;
s_Instance->m_NextRawSkeleton = t_NxtClientRawSkeleton;
s_Instance->m_RawSkeletonMutex.unlock();
}
}
Getting the basic skeleton information such as the ID and node count can be done via the CoreSdk_GetSkeletonInfo or the CoreSdk_GetRawSkeletonInfo respectively. In this info you can get the ID of the skeleton, which is either the id of the retargeted skeleton or of the glove ID, depending on which of the two streams you are looking at.
Using the CoreSdk_GetSkeletonData or the CoreSdk_GetRawSkeletonData you can get all the node data for the skeleton. The ID of the node is either the ID you gave it when setting up the skeleton for the retargeting or the ID that we gave the raw skeleton in MANUS Core.
The difficult part about using the Raw Skeletons compared to the Retargeting Skeleton is that you will need to reconstruct the hierarchy of that skeleton to be able to apply the data gotten from the stream. In the PrintRawSkeletonData function we demonstrate how to get the hierarchal information for the raw skeleton.
void SDKClient::PrintRawSkeletonData()
{
if (m_RawSkeleton == nullptr || m_RawSkeleton->skeletons.size() == 0)
{
return;
}
ClientLog::print("Received Skeleton glove data from Core. skeletons:{} first skeleton glove id:{}", m_RawSkeleton->skeletons.size(), m_RawSkeleton->skeletons[0].info.gloveId);
if (m_FirstLeftGloveID == 0 && m_FirstRightGloveID == 0) return; // no gloves connected to core
uint32_t t_NodeCount = 0;
uint32_t t_GloveId;
if (m_FirstLeftGloveID != 0)
{
t_GloveId = m_FirstLeftGloveID;
}
else
{
t_GloveId = m_FirstRightGloveID;
}
// we need to know the number of nodes in the hierarchy, so we can initialize an array with the correct size.
SDKReturnCode t_Result = CoreSdk_GetRawSkeletonNodeCount(t_GloveId, t_NodeCount);
if (t_Result != SDKReturnCode::SDKReturnCode_Success)
{
ClientLog::error("Failed to get Raw Skeleton Node Count. The error given was {}.", (int32_t)t_Result);
return;
}
// now get the hierarchy data, this can be used to reconstruct the positions of each node in case the user set up the system with a local coordinate system, and to know what each node is exactly.
// having a node position defined as local means that this will be related to its parent
NodeInfo* t_NodeInfo = new NodeInfo[t_NodeCount];
t_Result = CoreSdk_GetRawSkeletonNodeInfoArray(t_GloveId, t_NodeInfo, t_NodeCount);
if (t_Result != SDKReturnCode::SDKReturnCode_Success)
{
ClientLog::error("Failed to get Raw Skeleton Hierarchy. The error given was {}.", (int32_t)t_Result);
return;
}
ClientLog::print("Fetched array with Raw Skeleton Hierarchy with {} nodes.", (int32_t)t_NodeCount);
AdvanceConsolePosition(2);
}
/// @brief Prints the tracker data
/// Since our console cannot render this data visually in 3d (its not in the scope of this client)
/// we only display a little bit of data of the trackers we receive.
void SDKClient::PrintTrackerData()
{
ClientLog::print("Tracker test active: {}.", m_TrackerTest); //To show that test tracker is being sent to core
ClientLog::print("Per user tracker display: {}.", m_TrackerDataDisplayPerUser);
AdvanceConsolePosition(2);
if (m_TrackerDataDisplayPerUser)
{
PrintTrackerDataPerUser();
AdvanceConsolePosition(10);
}
else
{
PrintTrackerDataGlobal();
AdvanceConsolePosition(3);
}
// now, as a test, print the tracker data received from the stream
if (m_TrackerData == nullptr || m_TrackerData->trackerData.size() == 0)
{
return;
}
ClientLog::print("Received Tracker data. number of received trackers:{} first tracker type:{}", m_TrackerData->trackerData.size(), (int32_t)m_TrackerData->trackerData[0].trackerType);
}
The CoreSdk_GetRawSkeletonNodeCount function can be used to get the number of nodes that exist in the raw skeleton of a certain glove. Using the CoreSdk_GetRawSkeletonNodeInfo you can get all the information for every node in the Raw Skeleton. This NodeInfo gives you information such as the parent's ID and the side. Using this information you can reconstruct the skeleton. When using a relative coordinate system (a not world space coordinate system) you will need to keep the hierarchy into account to be able to accurately reconstruct the skeleton.
Skeletons in the Sample Client
This menu shows a few options. The first two are to load left[B] or right[N] hand skeletons. The third one[M] will unload the hand skeleton from MANUS Core, which we will talk about later in this article.
It's also possible to toggle "Send to DevTools"[D] which when true, will open Manus DevTools and load the skeleton for 3D visualization.
The other options are to activate haptics based on the skeleton id instead of a glove id. This is an easier way to do haptics since you do not have to manually keep track of which glove belongs to what skeleton.
Setup a skeleton
To setup a hand skeleton and make sure it animates correctly, the following steps are required:
- Setup the basic skeleton information
- Create and add nodes for the skeleton
- Create and add chains for the skeleton
The first step is creating a SkeletonSetupInfo structure and setting the data you wish to adjust. For a hand skeleton the type needs to be set to the SkeletonType_Hand enumerator.
Please note that this example builds both Left or Right hand skeleton, depending on the side passed as a parameter (Side p_Side).
void SDKClient::LoadTestSkeleton(Side p_Side)
{
uint32_t t_SklIndex = 0;
SkeletonSetupInfo t_SKL;
SkeletonSetupInfo_Init(&t_SKL);
t_SKL.type = SkeletonType::SkeletonType_Hand;
t_SKL.settings.scaleToTarget = true;
t_SKL.settings.useEndPointApproximations = true;
t_SKL.settings.targetType = SkeletonTargetType::SkeletonTargetType_UserIndexData;
//If the user does not exist then the added skeleton will not be animated.
//Same goes for any other skeleton made for invalid users/gloves.
t_SKL.settings.skeletonTargetUserIndexData.userIndex = 0;
CopyString(t_SKL.name, sizeof(t_SKL.name), p_Side == Side::Side_Left ? std::string("LeftHand") : std::string("RightHand"));
SDKReturnCode t_Res = CoreSdk_CreateSkeletonSetup(t_SKL, &t_SklIndex);
if (t_Res != SDKReturnCode::SDKReturnCode_Success)
{
ClientLog::error("Failed to Create Skeleton Setup. The error given was {}.", (int32_t)t_Res);
return;
}
m_TemporarySkeletons.push_back(t_SklIndex);
// setup nodes and chains for the skeleton hand
if (!SetupHandNodes(t_SklIndex, p_Side)) return;
if (!SetupHandChains(t_SklIndex, p_Side)) return;
if (m_SendToDevTools)
{
SendLoadedSkeleton(t_SklIndex);
}
// load skeleton
uint32_t t_ID = 0;
t_Res = CoreSdk_LoadSkeleton(t_SklIndex, &t_ID);
if (t_Res != SDKReturnCode::SDKReturnCode_Success)
{
ClientLog::error("Failed to load skeleton. The error given was {}.", (int32_t)t_Res);
return;
}
RemoveIndexFromTemporarySkeletonList(t_SklIndex);
if (t_ID == 0)
{
ClientLog::error("Failed to give skeleton an ID.");
}
m_LoadedSkeletons.push_back(t_ID);
}
To have the skeleton move according to the hand movement of the first user you can set the settings.targetType to SkeletonTarget_UserIndexData and the settings.SkeletonTargetUserIndexData to 0. This means it will look at all the users in MANUS Core and take the first user`s hands to animate the skeleton.
/// @brief Stores all the possible skeleton settings.
typedef struct SkeletonSettings
{
bool scaleToTarget;
bool useEndPointApproximations;
CollisionType collisionType;
SkeletonTargetType targetType;
SkeletonTargetUserData skeletonTargetUserData;
SkeletonTargetUserIndexData skeletonTargetUserIndexData;
SkeletonTargetAnimationData skeletonTargetAnimationData;
SkeletonTargetGloveData skeletonGloveData;
} SkeletonSettings;
There are several different target types which allow the user to choose how the skeleton is animated. The simplest is the user index, which we explained above, it requires very little knowledge of which gloves or users are available. When changing the target type make sure to match the targetType with the structure of the data, for example the SkeletonTargetType_GloveData enumerator requires the use of the SkeletonTargetGloveData structure.
The user data target allows you to specify a user ID which will enable the skeleton to be animated by the user specified, this does require you to know which users are available. Please keep in mind that the ID is not the same as the index, and you would need to find a certain user's ID in the landscape. The glove data target allows you to specify a glove ID. This ID will be used to find the correct glove and animate your skeleton according to the given glove. If this glove does not exist, the skeleton will not be animated.
To start adding the nodes and the chains to the skeleton setup we need to make it into a Temporary Skeleton by calling the CoreSdk_CreateSkeletonSetup function, which returns an index value which we can use to modify and load the skeleton.
Every skeleton needs a root node to define a point where everything originates from. The ID of the root node should be 0, and the parent of every other node should have a parent id.
In our example we setup our skeleton nodes with the following positions:
Note
These values are only illustrative and just show and example of how you could set up the nodes for a hand skeleton. The actual values will depend on the hand model you are using.
bool SDKClient::SetupHandNodes(uint32_t p_SklIndex, Side p_Side)
{
// Define number of fingers per hand and number of joints per finger
const uint32_t t_NumFingers = 5;
const uint32_t t_NumJoints = 4;
static ManusVec3 s_LeftHandPositions[t_NumFingers * t_NumJoints]
{
CreateManusVec3(0.024950f, 0.000000f, 0.025320f), //Thumb CMC joint
CreateManusVec3(0.000000f, 0.000000f, 0.032742f), //Thumb MCP joint
CreateManusVec3(0.000000f, 0.000000f, 0.028739f), //Thumb IP joint
CreateManusVec3(0.000000f, 0.000000f, 0.028739f), //Thumb Tip joint
//CreateManusVec3(0.011181f, 0.031696f, 0.000000f), //Index CMC joint // Note: we are not adding the matacarpal bones in this example, if you want to animate the metacarpals add each of them to the corresponding finger chain.
CreateManusVec3(0.011181f, 0.000000f, 0.052904f), //Index MCP joint, if metacarpal is present: CreateManusVec3(0.000000f, 0.000000f, 0.052904f)
CreateManusVec3(0.000000f, 0.000000f, 0.038257f), //Index PIP joint
CreateManusVec3(0.000000f, 0.000000f, 0.020884f), //Index DIP joint
CreateManusVec3(0.000000f, 0.000000f, 0.018759f), //Index Tip joint
//CreateManusVec3(0.000000f, 0.033452f, 0.000000f), //Middle CMC joint
CreateManusVec3(0.000000f, 0.000000f, 0.051287f), //Middle MCP joint
CreateManusVec3(0.000000f, 0.000000f, 0.041861f), //Middle PIP joint
CreateManusVec3(0.000000f, 0.000000f, 0.024766f), //Middle DIP joint
CreateManusVec3(0.000000f, 0.000000f, 0.019683f), //Middle Tip joint
//CreateManusVec3(-0.011274f, 0.031696f, 0.000000f), //Ring CMC joint
CreateManusVec3(-0.011274f, 0.000000f, 0.049802f), //Ring MCP joint, if metacarpal is present: CreateManusVec3(0.000000f, 0.000000f, 0.049802f),
CreateManusVec3(0.000000f, 0.000000f, 0.039736f), //Ring PIP joint
CreateManusVec3(0.000000f, 0.000000f, 0.023564f), //Ring DIP joint
CreateManusVec3(0.000000f, 0.000000f, 0.019868f), //Ring Tip joint
//CreateManusVec3(-0.020145f, 0.027538f, 0.000000f), //Pinky CMC joint
CreateManusVec3(-0.020145f, 0.000000f, 0.047309f), //Pinky MCP joint, if metacarpal is present: CreateManusVec3(0.000000f, 0.000000f, 0.047309f),
CreateManusVec3(0.000000f, 0.000000f, 0.033175f), //Pinky PIP joint
CreateManusVec3(0.000000f, 0.000000f, 0.018020f), //Pinky DIP joint
CreateManusVec3(0.000000f, 0.000000f, 0.019129f), //Pinky Tip joint
};
static ManusVec3 s_RightHandPositions[t_NumFingers * t_NumJoints]
{
CreateManusVec3(-0.024950f, 0.000000f, 0.025320f), //Thumb CMC joint
CreateManusVec3(-0.000000f, 0.000000f, 0.032742f), //Thumb MCP joint
CreateManusVec3(-0.000000f, 0.000000f, 0.028739f), //Thumb IP joint
CreateManusVec3(-0.000000f, 0.000000f, 0.028739f), //Thumb Tip joint
//CreateManusVec3(0.011181f, 0.031696f, 0.000000f), //Index CMC joint // Note: we are not adding the matacarpal bones in this example, if you want to animate the metacarpals add each of them to the corresponding finger chain.
CreateManusVec3(-0.011181f, 0.000000f, 0.052904f), //Index MCP joint, if metacarpal is present: CreateManusVec3(0.000000f, 0.000000f, 0.052904f)
CreateManusVec3(-0.000000f, 0.000000f, 0.038257f), //Index PIP joint
CreateManusVec3(-0.000000f, 0.000000f, 0.020884f), //Index DIP joint
CreateManusVec3(-0.000000f, 0.000000f, 0.018759f), //Index Tip joint
//CreateManusVec3(0.000000f, 0.033452f, 0.000000f), //Middle CMC joint
CreateManusVec3(-0.000000f, 0.000000f, 0.051287f), //Middle MCP joint
CreateManusVec3(-0.000000f, 0.000000f, 0.041861f), //Middle PIP joint
CreateManusVec3(-0.000000f, 0.000000f, 0.024766f), //Middle DIP joint
CreateManusVec3(-0.000000f, 0.000000f, 0.019683f), //Middle Tip joint
//CreateManusVec3(-0.011274f, 0.031696f, 0.000000f), //Ring CMC joint
CreateManusVec3(0.011274f, 0.000000f, 0.049802f), //Ring MCP joint, if metacarpal is present: CreateManusVec3(0.000000f, 0.000000f, 0.049802f),
CreateManusVec3(-0.000000f, 0.000000f, 0.039736f), //Ring PIP joint
CreateManusVec3(-0.000000f, 0.000000f, 0.023564f), //Ring DIP joint
CreateManusVec3(-0.000000f, 0.000000f, 0.019868f), //Ring Tip joint
//CreateManusVec3(-0.020145f, 0.027538f, 0.000000f), //Pinky CMC joint
CreateManusVec3(0.020145f, 0.000000f, 0.047309f), //Pinky MCP joint, if metacarpal is present: CreateManusVec3(0.000000f, 0.000000f, 0.047309f),
CreateManusVec3(-0.000000f, 0.000000f, 0.033175f), //Pinky PIP joint
CreateManusVec3(-0.000000f, 0.000000f, 0.018020f), //Pinky DIP joint
CreateManusVec3(-0.000000f, 0.000000f, 0.019129f), //Pinky Tip joint
};
// Create an array with the initial position of each hand node.
// Note, these values are just an example of node positions and refer to the hand laying on a flat surface.
ManusVec3* t_Fingers;
if (p_Side == Side::Side_Left)
{
t_Fingers = s_LeftHandPositions;
}
else
{
t_Fingers = s_RightHandPositions;
}
// skeleton entry is already done. just the nodes now.
// setup a very simple node hierarchy for fingers
// first setup the root node
//
// root, This node has ID 0 and parent ID 0, to indicate it has no parent.
SDKReturnCode t_Res = CoreSdk_AddNodeToSkeletonSetup(p_SklIndex, CreateNodeSetup(0, 0, 0, 0, 0, "Hand"));
if (t_Res != SDKReturnCode::SDKReturnCode_Success)
{
ClientLog::error("Failed to Add Node To Skeleton Setup. The error given was {}.", (int32_t)t_Res);
return false;
}
// then loop for 5 fingers
int t_FingerId = 0;
for (uint32_t i = 0; i < t_NumFingers; i++)
{
uint32_t t_ParentID = 0;
// then the digits of the finger that are linked to the root of the finger.
for (uint32_t j = 0; j < t_NumJoints; j++)
{
t_Res = CoreSdk_AddNodeToSkeletonSetup(p_SklIndex, CreateNodeSetup(1 + t_FingerId + j, t_ParentID, t_Fingers[i * 4 + j].x, t_Fingers[i * 4 + j].y, t_Fingers[i * 4 + j].z, "fingerdigit"));
if (t_Res != SDKReturnCode::SDKReturnCode_Success)
{
printf("Failed to Add Node To Skeleton Setup. The error given %d.", t_Res);
return false;
}
t_ParentID = 1 + t_FingerId + j;
}
t_FingerId += t_NumJoints;
}
return true;
}
We use these positions when creating a NodeSetup for each of the nodes, as seen in the CreateNodeSetup function. The resulting structure is then passed into the CoreSdk_AddNodeToSkeletonSetup function which adds it to the Temporary Skeleton.
To have a functional hand skeleton we need to have a Hand chain which contains the ids of all finger chains, and it should have the node ID of a wrist along with which side the hand is from. The handMotion setting in the hand chain specifies if the wrist should move via IMU data or via tracker data, or not at all. Each of the needed chains can be added with the CoreSdk_AddChainToSkeletonSetup function. It is also possible to assign nodes to chains in the Dev Tools, we will explain this in more details during the Temporary Skeletons part of this article.
In our example we add a wrist and all 5 fingers:
bool SDKClient::SetupHandChains(uint32_t p_SklIndex, Side p_Side)
{
// Add the Hand chain, this identifies the wrist of the hand
{
ChainSettings t_ChainSettings;
ChainSettings_Init(&t_ChainSettings);
t_ChainSettings.usedSettings = ChainType::ChainType_Hand;
t_ChainSettings.hand.handMotion = HandMotion::HandMotion_IMU;
t_ChainSettings.hand.fingerChainIdsUsed = 5; //we will have 5 fingers
t_ChainSettings.hand.fingerChainIds[0] = 1; //links to the other chains we will define further down
t_ChainSettings.hand.fingerChainIds[1] = 2;
t_ChainSettings.hand.fingerChainIds[2] = 3;
t_ChainSettings.hand.fingerChainIds[3] = 4;
t_ChainSettings.hand.fingerChainIds[4] = 5;
ChainSetup t_Chain;
ChainSetup_Init(&t_Chain);
t_Chain.id = 0; //Every ID needs to be unique per chain in a skeleton.
t_Chain.type = ChainType::ChainType_Hand;
t_Chain.dataType = ChainType::ChainType_Hand;
t_Chain.side = p_Side;
t_Chain.dataIndex = 0;
t_Chain.nodeIdCount = 1;
t_Chain.nodeIds[0] = 0; //this links to the hand node created in the SetupHandNodes
t_Chain.settings = t_ChainSettings;
SDKReturnCode t_Res = CoreSdk_AddChainToSkeletonSetup(p_SklIndex, t_Chain);
if (t_Res != SDKReturnCode::SDKReturnCode_Success)
{
ClientLog::error("Failed to Add Chain To Skeleton Setup. The error given was {}.", (int32_t)t_Res);
return false;
}
}
// Add the 5 finger chains
const ChainType t_FingerTypes[5] = { ChainType::ChainType_FingerThumb,
ChainType::ChainType_FingerIndex,
ChainType::ChainType_FingerMiddle,
ChainType::ChainType_FingerRing,
ChainType::ChainType_FingerPinky };
for (int i = 0; i < 5; i++)
{
ChainSettings t_ChainSettings;
ChainSettings_Init(&t_ChainSettings);
t_ChainSettings.usedSettings = t_FingerTypes[i];
t_ChainSettings.finger.handChainId = 0; //This links to the wrist chain above.
//This identifies the metacarpal bone, if none exists, or the chain is a thumb it should be set to -1.
//The metacarpal bone should not be part of the finger chain, unless you are defining a thumb which does need it.
t_ChainSettings.finger.metacarpalBoneId = -1;
t_ChainSettings.finger.useLeafAtEnd = false; //this is set to true if there is a leaf bone to the tip of the finger.
ChainSetup t_Chain;
ChainSetup_Init(&t_Chain);
t_Chain.id = i + 1; //Every ID needs to be unique per chain in a skeleton.
t_Chain.type = t_FingerTypes[i];
t_Chain.dataType = t_FingerTypes[i];
t_Chain.side = p_Side;
t_Chain.dataIndex = 0;
if (i == 0) // Thumb
{
t_Chain.nodeIdCount = 4; //The amount of node id's used in the array
t_Chain.nodeIds[0] = 1; //this links to the hand node created in the SetupHandNodes
t_Chain.nodeIds[1] = 2; //this links to the hand node created in the SetupHandNodes
t_Chain.nodeIds[2] = 3; //this links to the hand node created in the SetupHandNodes
t_Chain.nodeIds[3] = 4; //this links to the hand node created in the SetupHandNodes
}
else // All other fingers
{
t_Chain.nodeIdCount = 4; //The amount of node id's used in the array
t_Chain.nodeIds[0] = (i * 4) + 1; //this links to the hand node created in the SetupHandNodes
t_Chain.nodeIds[1] = (i * 4) + 2; //this links to the hand node created in the SetupHandNodes
t_Chain.nodeIds[2] = (i * 4) + 3; //this links to the hand node created in the SetupHandNodes
t_Chain.nodeIds[3] = (i * 4) + 4; //this links to the hand node created in the SetupHandNodes
}
t_Chain.settings = t_ChainSettings;
SDKReturnCode t_Res = CoreSdk_AddChainToSkeletonSetup(p_SklIndex, t_Chain);
if (t_Res != SDKReturnCode::SDKReturnCode_Success)
{
return false;
}
}
return true;
}
Adding the finger chains goes in a similar process to the hand chain. We need to make sure to add all the node ids and set the handChainId to the same id as the one set in the hand chain, which is 0 in our example. Do not forget to set the correct side for the finger chains as well.
At this point we have a fully functioning Temporary Skeleton. This skeleton can either be loaded into the retargeting system of MANUS Core or sent to the Dev Tools for verification via the CoreSdk_SaveTemporarySkeleton function, which we will get more into in the Temporary Skeletons section of this article.
Retargeted Skeletons
After setting up a Temporary Skeleton via the method explained above the user can call the CoreSdk_LoadSkeleton function. This will load the Temporary Skeleton into MANUS Core's retargeting system and remove it from the list of Temporary Skeletons (because it is no longer temporary at this point). The ID returned from the load function is the one you should keep track of to match the skeleton stream data to the skeleton you loaded in.
It is also possible to unload a skeleton when you no longer need it to be animated. This prevents unnecessary calculations being done for a skeleton that may no longer be in use.
/// @brief This support function is used to unload a skeleton from Core.
void SDKClient::UnloadTestSkeleton()
{
if (m_LoadedSkeletons.size() == 0)
{
ClientLog::error("There was no skeleton for us to unload.");
return;
}
SDKReturnCode t_Res = CoreSdk_UnloadSkeleton(m_LoadedSkeletons[0]);
m_LoadedSkeletons.erase(m_LoadedSkeletons.begin());
if (t_Res != SDKReturnCode::SDKReturnCode_Success)
{
ClientLog::error("Failed to unload skeleton. The error given was {}.", (int32_t)t_Res);
return;
}
}
In this example, if there are skeletons loaded, we can unload a skeleton by calling CoreSdk_UnloadSkeleton and passing the skeleton id.
When a client application is closed, the session will be closed, and MANUS Core will automatically unload all skeletons that belong to that session.
This can also happen when a connection is lost for any reason, but in such a case you can simply reload the skeletons.
Temporary Skeletons
When defining a skeleton, it cannot be fully animated yet but can be sent to Manus Core's Dev Tools for verification. The Dev Tools will automatically launch and load the temporary skeleton. For detailed usage instructions, refer to the Dev Tools article in our knowledge center.
To set up a Temporary Skeleton, which is similar to a regular skeleton but does not require all chains to be defined, follow these steps. The Dev Tools can send back an updated Temporary Skeleton, which can then be used to create a fully animated skeleton. This functionality is demonstrated in both the Unreal Engine and Unity plugins for animation verification.
If the Temporary Skeleton is not destroyed, it can be easily updated by the Dev Tools. However, if the network connection with MANUS Core is lost, the temporary skeleton must be resent for the plugins to continue working.
For well-defined, typically structured skeletons, you can use the CoreSdk_AllocateChainsForSkeletonSetup function to automatically allocate chains. However, this process can be opaque without visualization, so it is recommended to use the Dev Tools for this purpose.
Refer to the BuildTemporarySkeleton function in the sample client for an example of this process. Temporary skeletons can also be saved to and loaded from .mskl files for easier handling. Examples of saving and loading .mskl files can be found in the SaveTemporarySkeletonToFile and GetTemporarySkeletonFromFile functions.
void SDKClient::SaveTemporarySkeletonToFile()
{
// this example shows how to save a temporary skeleton to a file
// first create a temporary skeleton:
// define the session Id for which we want to save
uint32_t t_SessionId = m_SessionId;
bool t_IsSkeletonModified = false; // setting this bool to true is not necessary here, it is mostly used by the Dev Tools
// to notify the SDK sessions about their skeleton being modified.
// first create a skeleton setup
uint32_t t_SklIndex = 0;
SkeletonSetupInfo t_SKL;
SkeletonSetupInfo_Init(&t_SKL);
t_SKL.type = SkeletonType::SkeletonType_Hand;
t_SKL.settings.scaleToTarget = true;
t_SKL.settings.targetType = SkeletonTargetType::SkeletonTargetType_GloveData;
t_SKL.settings.skeletonTargetUserIndexData.userIndex = 0;
CopyString(t_SKL.name, sizeof(t_SKL.name), std::string("LeftHand"));
SDKReturnCode t_Res = CoreSdk_CreateSkeletonSetup(t_SKL, &t_SklIndex);
if (t_Res != SDKReturnCode::SDKReturnCode_Success)
{
ClientLog::error("Failed to Create Skeleton Setup. The error given was {}.", (int32_t)t_Res);
return;
}
m_TemporarySkeletons.push_back(t_SklIndex);
// setup nodes and chains for the skeleton hand
if (!SetupHandNodes(t_SklIndex, Side_Left)) return;
if (!SetupHandChains(t_SklIndex, Side_Left)) return;
// save the temporary skeleton
t_Res = CoreSdk_SaveTemporarySkeleton(t_SklIndex, t_SessionId, t_IsSkeletonModified);
if (t_Res != SDKReturnCode::SDKReturnCode_Success)
{
ClientLog::error("Failed to save temporary skeleton. The error given was {}.", (int32_t)t_Res);
return;
}
// now compress the temporary skeleton data and get the size of the compressed data:
uint32_t t_TemporarySkeletonLengthInBytes;
t_Res = CoreSdk_CompressTemporarySkeletonAndGetSize(t_SklIndex, t_SessionId, &t_TemporarySkeletonLengthInBytes);
if (t_Res != SDKReturnCode::SDKReturnCode_Success)
{
ClientLog::error("Failed to compress temporary skeleton and get size. The error given was {}.", (int32_t)t_Res);
return;
}
unsigned char* t_TemporarySkeletonData = new unsigned char[t_TemporarySkeletonLengthInBytes];
// get the array of bytes with the compressed temporary skeleton data, remember to always call function CoreSdk_CompressTemporarySkeletonAndGetSize
// before trying to get the compressed temporary skeleton data
t_Res = CoreSdk_GetCompressedTemporarySkeletonData(t_TemporarySkeletonData, t_TemporarySkeletonLengthInBytes);
if (t_Res != SDKReturnCode::SDKReturnCode_Success)
{
ClientLog::error("Failed to get compressed temporary skeleton data. The error given was {}.", (int32_t)t_Res);
return;
}
// now save the data into a .mskl file
// as an example we save the temporary skeleton in a folder called ManusTemporarySkeleton inside the documents directory
// get the path for the documents directory
std::string t_DirectoryPathString = GetDocumentsDirectoryPath_UTF8();
// create directory name and file name for storing the temporary skeleton
std::string t_DirectoryPath =
t_DirectoryPathString
+ s_SlashForFilesystemPath
+ "ManusTemporarySkeleton";
CreateFolderIfItDoesNotExist(t_DirectoryPath);
std::string t_DirectoryPathAndFileName =
t_DirectoryPath
+ s_SlashForFilesystemPath
+ "TemporarySkeleton.mskl";
// write the temporary skeleton data to .mskl file
std::ofstream t_File = GetOutputFileStream(t_DirectoryPathAndFileName);
t_File.write((char*)t_TemporarySkeletonData, t_TemporarySkeletonLengthInBytes);
t_File.close();
t_Res = CoreSdk_ClearTemporarySkeleton(t_SklIndex, t_SessionId);
if (t_Res != SDKReturnCode::SDKReturnCode_Success)
{
ClientLog::error("Failed to Clear Temporary Skeleton after saving. The error given was {}.", (int32_t)t_Res);
return;
}
RemoveIndexFromTemporarySkeletonList(t_SklIndex);
}
-
Create a Temporary Skeleton: Use the
CoreSdk_SaveTemporarySkeletonfunction to save the temporary skeleton to Manus Core. -
Compress the Skeleton Data: Call the
CoreSdk_CompressTemporarySkeletonAndGetSizefunction to get the number of bytes needed to save the data. -
Retrieve the Compressed Data: Use the
CoreSdk_GetCompressedTemporarySkeletonDatafunction to get the actual data into a byte array. -
Save to File: Save the byte array to a file as you would with any normal byte array.
-
Clear and Remove (Optional): In the example, the temporary skeleton is cleared and removed for illustration purposes. This step is not required if you plan on using the skeleton.
To load a temporary skeleton from a file:
-
Read the File: Use the example function
GetTemporarySkeletonFromFileto read the file. -
Send to Manus Core: Send the byte array read from the file to Manus Core using the
CoreSdk_GetTemporarySkeletonFromCompressedDatafunction. This will create a new temporary skeleton and return its ID.
Trackers
Manus Core is able to ingest positional and rotation data from various tracking systems (for example ART and SteamVR) and can be used to set up a skeleton for the whole body. For this, they must be assigned to a user and configured to represent a limb or body part location. However, getting the data for trackers can still be useful for props etc. You may also want to add your own tracker system and be able to stream the tracker data back into MANUS core.
You can access tracker information via the data gotten via the landscape callback or via a few functions we made specifically for the tracker information.
Showing trackers per user or globally:
void SDKClient::PrintTrackerDataPerUser()
{
uint32_t t_NumberOfAvailableUsers = 0;
SDKReturnCode t_UserResult = CoreSdk_GetNumberOfAvailableUsers(&t_NumberOfAvailableUsers);
if (t_UserResult != SDKReturnCode::SDKReturnCode_Success)
{
ClientLog::error("Failed to get user count. The error given was {}.", (int32_t)t_UserResult);
return;
}
if (t_NumberOfAvailableUsers == 0) return; // nothing to get yet
for (uint32_t i = 0; i < t_NumberOfAvailableUsers; i++)
{
uint32_t t_NumberOfAvailabletrackers = 0;
SDKReturnCode t_TrackerResult = CoreSdk_GetNumberOfAvailableTrackersForUserIndex(&t_NumberOfAvailabletrackers, i);
if (t_TrackerResult != SDKReturnCode::SDKReturnCode_Success)
{
ClientLog::error("Failed to get tracker data. The error given was {}.", (int32_t)t_TrackerResult);
return;
}
if (t_NumberOfAvailabletrackers == 0) continue;
ClientLog::print("received available trackers for user index[{}] :{} ", i, t_NumberOfAvailabletrackers);
if (t_NumberOfAvailabletrackers == 0) return; // nothing else to do.
TrackerId* t_TrackerId = new TrackerId[t_NumberOfAvailabletrackers];
t_TrackerResult = CoreSdk_GetIdsOfAvailableTrackersForUserIndex(t_TrackerId, i, t_NumberOfAvailabletrackers);
if (t_TrackerResult != SDKReturnCode::SDKReturnCode_Success)
{
ClientLog::error("Failed to get tracker data. The error given was {}.", (int32_t)t_TrackerResult);
return;
}
}
}
In this code example all users are first gathered and then the tracker IDs per user are counted and displayed.
For normal body tracking the trackers are usually assigned to a user. Unassigned trackers will need to be found with the global function CoreSdk_GetNumberOfAvailableTrackers followed by the CoreSdk_GetIdsOfAvailableTrackers function.
void SDKClient::PrintTrackerData()
{
ClientLog::print("Tracker test active: {}.", m_TrackerTest); //To show that test tracker is being sent to core
ClientLog::print("Per user tracker display: {}.", m_TrackerDataDisplayPerUser);
AdvanceConsolePosition(2);
if (m_TrackerDataDisplayPerUser)
{
PrintTrackerDataPerUser();
AdvanceConsolePosition(10);
}
else
{
PrintTrackerDataGlobal();
AdvanceConsolePosition(3);
}
// now, as a test, print the tracker data received from the stream
if (m_TrackerData == nullptr || m_TrackerData->trackerData.size() == 0)
{
return;
}
ClientLog::print("Received Tracker data. number of received trackers:{} first tracker type:{}", m_TrackerData->trackerData.size(), (int32_t)m_TrackerData->trackerData[0].trackerType);
AdvanceConsolePosition(1);
}
No actual tracker data or meta data is being shown here, but it can be retrieved with the CoreSdk_GetDataForTracker_UsingTrackerId function. Using the meta data, you can determine if it is an unassigned tracker or not in case you want to use a tracker for something else than body tracking.
Adding and removing custom trackers
/// @brief This support function is used to set a test tracker and add it to the landscape.
void SDKClient::HandleTrackerCommands()
{
if (GetKeyDown('O'))
{
m_TrackerTest = !m_TrackerTest;
}
if (GetKeyDown('G'))
{
m_TrackerDataDisplayPerUser = !m_TrackerDataDisplayPerUser;
}
if (m_TrackerTest)
{
m_TrackerOffset += 0.0005f;
if (m_TrackerOffset >= 10.0f)
{
m_TrackerOffset = 0.0f;
}
TrackerId t_TrackerId;
CopyString(t_TrackerId.id, sizeof(t_TrackerId.id), std::string("Test Tracker"));
TrackerData t_TrackerData = {};
t_TrackerData.isHmd = false;
t_TrackerData.trackerId = t_TrackerId;
t_TrackerData.trackerType = TrackerType::TrackerType_Unknown;
t_TrackerData.position = { 0.0f, m_TrackerOffset, 0.0f };
t_TrackerData.rotation = { 1.0f, 0.0f, 0.0f, 0.0f };
t_TrackerData.quality = TrackerQuality::TrackingQuality_Trackable;
TrackerData t_TrackerDatas[MAX_NUMBER_OF_TRACKERS];
t_TrackerDatas[0] = t_TrackerData;
const SDKReturnCode t_TrackerSend = CoreSdk_SendDataForTrackers(t_TrackerDatas, 1);
if (t_TrackerSend != SDKReturnCode::SDKReturnCode_Success)
{
ClientLog::error("Failed to send tracker data. The error given was {}.", (int32_t)t_TrackerSend);
return;
}
}
}
In this code example a simple TestTracker is set up and passed into MANUS Core by passing an array of trackers into the CoreSdk_SendDataForTrackers function. As a test, the position is slightly altered every update to show it moving and is visualized in the MANUS Core Dashboard if it is also running. When you make your own trackers make sure every tracker has a unique ID and is trackable.
In this test only one tracker is being sent, but it is advisable for synchronization to send all custom trackers in the same array at the same time.
Setting Tracker Offset
To set the tracker offset, you can use the CoreSdk_SetTrackerOffset function. This function is used to set the tracker offset for a specific user. In the example below, the tracker offset is set for the left hand tracker to the wrist. The tracker offset is set to 3mm right, 58mm up, and 43mm forward relative to the wrist. This value is illustrative and should be adjusted to the actual tracker position.
/// @brief This support function is used to set the tracker offset. It sets the tracker offset for the left hand tracker to the wrist.
void SDKClient::SetTrackerOffset() {
TrackerOffset t_TrackerOffset;
//Positions the tracker 3mm right, 58mm up and 43mm forward relative to the wrist. This value is illustrative and should be adjusted to the actual tracker position.
t_TrackerOffset.translation = { 0.003f, -0.058f, -0.043f };
t_TrackerOffset.rotation = { 1.0f, 0.0f, 0.0f, 0.0f };
t_TrackerOffset.entryType = TrackerOffsetType::TrackerOffsetType_LeftHandTrackerToWrist;
auto t_UserId = m_Landscape->users.users[0].id;
SDKReturnCode t_SetTrackerReturn = CoreSdk_SetTrackerOffset(t_UserId, &t_TrackerOffset);
if (t_SetTrackerReturn != SDKReturnCode::SDKReturnCode_Success)
{
ClientLog::error("Failed to set tracker offset. The error given was {}.", (int32_t)t_SetTrackerReturn);
return;
}
}
Timecode
A timecode generator is best configured via the MANUS Core Dashboard . However, you might still be interested in the timecode settings in your own client. For this you can use the functions shown in the PrintLandscapeTimeData function of the example client.
void SDKClient::PrintLandscapeTimeData()
{
ClientLog::print("Total count of Interfaces: {}", m_Landscape->time.interfaceCount);
ClientLog::print("Current Interface: {} {} at index {}", m_Landscape->time.currentInterface.name, m_Landscape->time.currentInterface.api, m_Landscape->time.currentInterface.index);
ClientLog::print("FPS: {}", GetFPSEnumName(m_Landscape->time.fps));
ClientLog::print("Fake signal: {} | Sync Pulse: {} | Sync Status: {}", m_Landscape->time.fakeTimecode, m_Landscape->time.useSyncPulse, m_Landscape->time.syncStatus);
ClientLog::print("Device keep alive: {} | Timecode Status: {}", m_Landscape->time.deviceKeepAlive, m_Landscape->time.timecodeStatus);
AdvanceConsolePosition(6);
}
If the landscape callback is registered, you can read out the landscape data to retrieve the time landscape with timecode information.
Due to thread safety, this is copied during a mutex to a local copy of the data and eventually printed in the UI thread via the PrintLandscapeTimeData function.
For more information on timecode, please consult your timecode device manual.
Gestures
MANUS Core has a built-in gesture system. Data from this gesture system can be accessed via the Gesture Stream. In our example in the function OnGestureStreamCallback you can see how to receive the data:
/// @brief This gets called when receiving gesture data from Manus Core
/// In our sample we only save the first glove's gesture data.
/// Gesture data gets generated and sent when glove data changes, this means that the stream
/// does not always contain ALL of the devices, because some may not have had new data since
/// the last time the gesture data was sent.
/// @param p_GestureStream contains the basic info to retrieve gesture data.
void SDKClient::OnGestureStreamCallback(const GestureStreamInfo* const p_GestureStream)
{
if (s_Instance)
{
for (uint32_t i = 0; i < p_GestureStream->gestureProbabilitiesCount; i++)
{
GestureProbabilities t_Probs;
CoreSdk_GetGestureStreamData(i, 0, &t_Probs);
if (t_Probs.isUserID)continue;
if (t_Probs.id != s_Instance->m_FirstLeftGloveID && t_Probs.id != s_Instance->m_FirstRightGloveID)continue;
ClientGestures* t_Gest = new ClientGestures();
t_Gest->info = t_Probs;
t_Gest->probabilities.reserve(t_Gest->info.totalGestureCount);
uint32_t t_BatchCount = (t_Gest->info.totalGestureCount / MAX_GESTURE_DATA_CHUNK_SIZE) + 1;
uint32_t t_ProbabilityIdx = 0;
for (uint32_t b = 0; b < t_BatchCount; b++)
{
for (uint32_t j = 0; j < t_Probs.gestureCount; j++)
{
t_Gest->probabilities.push_back(t_Probs.gestureData[j]);
}
t_ProbabilityIdx += t_Probs.gestureCount;
CoreSdk_GetGestureStreamData(i, t_ProbabilityIdx, &t_Probs); //this will get more data, if needed for the next iteration.
}
s_Instance->m_GestureMutex.lock();
if (t_Probs.id == s_Instance->m_FirstLeftGloveID)
{
if (s_Instance->m_NewFirstLeftGloveGestures != nullptr) delete s_Instance->m_NewFirstLeftGloveGestures;
s_Instance->m_NewFirstLeftGloveGestures = t_Gest;
}
else
{
if (s_Instance->m_NewFirstRightGloveGestures != nullptr) delete s_Instance->m_NewFirstRightGloveGestures;
s_Instance->m_NewFirstRightGloveGestures = t_Gest;
}
s_Instance->m_GestureMutex.unlock();
}
}
}
The stream info gives you the number of gestures available per glove. Using the CoreSdk_GetGestureStreamData function you can get the specific gesture information for a given glove. Due to the amount of potential gesture data in the future, it is advisable to call this function in the callback to gather all the gesture data. This way it becomes impossible for the data to be modified by the SDK during this function call. In our example we gather the gesture data for the first left and right glove, we display this data to the console:
void SDKClient::PrintGestureData()
{
ClientGestures* t_Gest = m_FirstLeftGloveGestures;
std::string t_Side = "Left";
if (!m_ShowLeftGestures)
{
t_Side = "Right";
t_Gest = m_FirstRightGloveGestures;
}
if (t_Gest == nullptr)
{
ClientLog::print("No Gesture information for first {} glove.", t_Side);
AdvanceConsolePosition(1);
return;
}
ClientLog::print("Total count of gestures for the {} glove: {}", t_Side, t_Gest->info.totalGestureCount);
uint32_t t_Max = t_Gest->info.totalGestureCount;
if (t_Max > 20) t_Max = 20;
ClientLog::print("Showing result of first {} gestures.", t_Max);
for (uint32_t i = 0; i < t_Max; i++)
{
char* t_Name = "";
for (uint32_t g = 0; g < m_GestureLandscapeData.size(); g++)
{
if (m_GestureLandscapeData[g].id == t_Gest->probabilities[i].id)
{
t_Name = m_GestureLandscapeData[g].name;
}
}
ClientLog::print("Gesture {} ({}) has a probability of {}%.", t_Name, t_Gest->probabilities[i].id, t_Gest->probabilities[i].percent * 100.0f);
}
AdvanceConsolePosition(t_Max + 2);
}
The gesture data you receive contains ID and normalized percentage numbers (from 0.0 to 1.0). The gesture ID can be matched to the gesture name shown in the landscape data for gestures, which will allow you to see what gesture it is.
Currently it is not supported to add custom gestures to MANUS Core , when this does become possible it is best to simply remember what IDs are assigned to which gesture. In predefined gestures, the IDs will remain the same throughout the application runtime, so you will only need to match the gesture's name to its ID once if you wish to use those names for identification.
Calibrating gloves
To calibrate a glove, a specific series of steps must be followed. The number of steps can vary between different glove models. To determine how many steps are required for a specific glove, you can use the CoreSdk_GloveCalibrationGetNumberOfSteps function, passing an argument of type GloveCalibrationArgs.
For detailed information about each calibration step, the CoreSdk_GloveCalibrationGetStepData function can be used. This method takes a GloveCalibrationStepArgs parameter and returns a GloveCalibrationStepData object. The returned data includes an index, title, description, and duration. If the duration is an estimated value, it will be negative.
To start the calibration process, call the CoreSdk_GloveCalibrationStart method, and to stop it, use the CoreSdk_GloveCalibrationStop method. Each calibration step must be performed in sequential order using the CoreSdk_GloveCalibrationStartStep method, the GloveCalibrationStepArgs contains information on which step to start. After all steps are completed, call CoreSdk_GloveCalibrationFinish to save and apply the calibration.
In this sample, the gloves for the user at index 0 can be calibrated.
void SDKClient::ExecuteGloveCalibrationStep(GloveCalibrationStepArgs p_Args)
{
bool t_Result;
auto t_Res = CoreSdk_GloveCalibrationStartStep(p_Args, &t_Result);
if (t_Res == SDKReturnCode_Success && t_Result)
m_CalibrationMessage = "Step finished";
else
m_CalibrationMessage = "Step failed";
m_IsCalibrationInProgress = false;
}
/// @brief Handles the console commands for the glove calibration.
void SDKClient::HandleGloveCalibrationCommands()
{
// Reset data
m_NumberOfCalibrationSteps = 0;
m_CalibrationGloveId = 0;
m_ValidStepData = false;
GloveCalibrationStepData_Init(&m_StepData);
if (GetKeyDown('H'))
{
m_CalibrateLeftHand = !m_CalibrateLeftHand;
}
// Get glove id
if (m_Landscape == nullptr || m_Landscape->users.userCount == 0)
return;
uint32_t t_GloveID = m_Landscape->users.users[0].leftGloveID;
if (!m_CalibrateLeftHand)
t_GloveID = m_Landscape->users.users[0].rightGloveID;
m_CalibrationGloveId = t_GloveID;
// Check if glove ID is valid, if not return
if (m_CalibrationGloveId == 0)
return;
GloveCalibrationArgs t_Args;
t_Args.gloveId = m_CalibrationGloveId;
// Get the number of calibration steps
auto t_Res = CoreSdk_GloveCalibrationGetNumberOfSteps(t_Args, &m_NumberOfCalibrationSteps);
if (t_Res != SDKReturnCode_Success || m_NumberOfCalibrationSteps == 0)
return;
if (GetKeyDown('P') && m_CalibrationStep < 9)
{
m_CalibrationStep++;
}
if (GetKeyDown('M') && m_CalibrationStep != 0)
{
m_CalibrationStep--;
}
// Clamp current step to max number of steps
if (m_CalibrationStep >= m_NumberOfCalibrationSteps)
m_CalibrationStep = m_NumberOfCalibrationSteps - 1;
// Get step data
GloveCalibrationStepArgs t_StepArgs;
t_StepArgs.gloveId = t_Args.gloveId;
t_StepArgs.stepIndex = m_CalibrationStep;
t_Res = CoreSdk_GloveCalibrationGetStepData(t_StepArgs, &m_StepData);
if (t_Res != SDKReturnCode_Success)
return;
// Dont allow starting/stopping/etc while in progress
if (m_IsCalibrationInProgress)
return;
if (GetKeyDown('S')) // Start
{
bool t_Result = false;
t_Res = CoreSdk_GloveCalibrationStart(t_Args, &t_Result);
if (t_Res == SDKReturnCode::SDKReturnCode_Success && t_Result)
m_CalibrationMessage = "Glove calibration started";
else
m_CalibrationMessage = "Failed to start glove calibration";
}
if (GetKeyDown('C')) // Cancel
{
bool t_Result = false;
t_Res = CoreSdk_GloveCalibrationStop(t_Args, &t_Result);
if (t_Res == SDKReturnCode::SDKReturnCode_Success && t_Result)
m_CalibrationMessage = "Glove calibration stopped";
else
m_CalibrationMessage = "Failed to stop glove calibration";
}
if (GetKeyDown('F')) // Finish
{
bool t_Result = false;
t_Res = CoreSdk_GloveCalibrationFinish(t_Args, &t_Result);
if (t_Res == SDKReturnCode::SDKReturnCode_Success && t_Result)
m_CalibrationMessage = "Glove calibration finished";
else
m_CalibrationMessage = "Failed to finish glove calibration";
}
if (GetKeyDown('E')) // Execute step
{
m_CalibrationMessage = "Step " + std::to_string(m_CalibrationStep) + " in progress";
m_IsCalibrationInProgress = true;
std::thread t_Thread(&SDKClient::ExecuteGloveCalibrationStep, this, t_StepArgs);
t_Thread.detach();
}
}
Pairing / Unpairing
To pair or unpair a glove, you can use the CoreSdk_PairGlove or CoreSdk_UnpairGlove functions. Both functions require the glove’s ID and a pointer to a boolean value, which will indicate whether the operation was successful. A true value means the glove was successfully paired or unpaired.
When pairing, the SDK will automatically use the first available dongle. However, if you want to pair the glove with a specific dongle, ensure that all other dongles are disconnected from the PC, and that all network devices (except the preferred one) are disconnected from MANUS Core.
In the SDK client pairing example, the first unpaired glove ID is used as the argument for the pairing function.
void SDKClient::PairGlove()
{
bool t_GloveFound = false;
uint32_t t_GloveToPairId = -1;
for (size_t i = 0; i < m_Landscape->gloveDevices.gloveCount; i++)
{
if (m_Landscape->gloveDevices.gloves[i].pairedState == DevicePairedState_Unpaired)
{
t_GloveToPairId = m_Landscape->gloveDevices.gloves[i].id;
t_GloveFound = true;
break;
}
}
if (!t_GloveFound)
{
ClientLog::warn("No glove available for pairing found.");
return;
}
bool t_Paired;
SDKReturnCode t_PairGloveResult = CoreSdk_PairGlove(t_GloveToPairId, &t_Paired);
if (t_PairGloveResult != SDKReturnCode::SDKReturnCode_Success)
{
ClientLog::error("Failed to pair glove with ID: {}. The error given was {}", t_GloveToPairId, (int32_t)t_PairGloveResult);
return;
}
if (!t_Paired)
{
ClientLog::error("Failed to pair glove with ID: {}", t_GloveToPairId);
return;
}
ClientLog::print("Succesfully paired glove with ID: {}", t_GloveToPairId);
return;
}
In the SDK client unpairing example, the ID of the first paired glove is passed as the parameter to the unpair function.
void SDKClient::UnpairGlove()
{
bool t_GloveFound = false;
uint32_t t_GloveToUnPairId = -1;
for (size_t i = 0; i < m_Landscape->gloveDevices.gloveCount; i++)
{
if (m_Landscape->gloveDevices.gloves[i].pairedState == DevicePairedState_Paired)
{
t_GloveToUnPairId = m_Landscape->gloveDevices.gloves[i].id;
t_GloveFound = true;
break;
}
}
if (!t_GloveFound)
{
ClientLog::error("No glove to unpair found");
return;
}
bool t_Unpaired;
SDKReturnCode t_UnpairGloveResult = CoreSdk_UnpairGlove(t_GloveToUnPairId, &t_Unpaired);
if (t_UnpairGloveResult != SDKReturnCode::SDKReturnCode_Success)
{
ClientLog::error("Failed to unpair glove with ID: {}. The error given was ", t_GloveToUnPairId, (int32_t)t_UnpairGloveResult);
return;
}
ClientLog::print("Succesfully unpaired glove with ID: {}", t_GloveToUnPairId);
return;
}