Summary: In this blog post, we discuss how to integrate synchronized signals from peripheral devices with your Theia3D kinematics.
As the third and final installment in our series of blog posts related to working with Theia3D data in Visual3D, this post aims to provide guidance on how to integrate signals from peripheral devices with your markerless kinematic signals from Theia3D. These signals may be from force platforms, an instrumented treadmill, EMGs, or any other supported external device which can record data that can be imported into Visual3D. The previous two blog posts covered the contents and interpretation of signals in Theia-output .C3D files and basic post-processing steps often performed in Visual3D.
When it comes to integrating external devices with markerless kinematics, it is absolutely crucial that the signals recorded by those devices are synchronized with the videos used to obtain the markerless kinematics. Therefore, only camera systems that support synchronization of external devices can be used to integrate such signals (e.g. Qualisys Miqus, Vicon Vue/FLIR Blackfly S, OptiTrack Prime Color). It’s also important to note that Theia3D itself does not handle the .C3D files from external devices - all integration of these signals with markerless kinematics is performed as a post-processing step in Visual3D. However, due to the vast functionality of Visual3D and its careful development alongside Theia3D, the process of merging your kinematic and other signals is relatively straightforward.
Figure 1: Integrating signals from external devices with kinematics from Theia3D is performed as a post-processing step in Visual3D, where two separate .C3D files containing the different sets of signals are merged into one.
There are two main approaches that can be used, each being a specific pipeline command within Visual3D. These two approaches are both acceptable and capable of importing external signals into markerless kinematic .C3D files, but differ in how they are used and their flexibility. For further reading and a tutorial on these methods, check out the Visual3D tutorial: Theia3D Merge.
The first approach is using the Manage_Theia3D_Merge pipeline command. Of the two approaches presented here, this one is more structured and has some requirements for how your data is organized and named. However, the benefit of this approach is its speed and comprehensiveness, as it is an all-in-one approach to merge analog signals from separate .C3D files with kinematic .C3D files from Theia3D. This section briefly describes one way this command can be used, including how the data needs to be organized and some of the options you can change. Users are also encouraged to review the Visual3D tutorial for this process linked above and try it out for themselves.
For this example of how to use Manage_Theia3D_Merge to import analog signals into Theia kinematic .C3D files, we have organized our data as shown in Figure 2. The naming convention used for the kinematic (‘THEIA’) files is not required, but this is how the output would look if you used the Use Hierarchical Names option set to Level: 3. However, the naming of the analog files is important and should be similar to the Theia-output trial path name, as is the case here. Also, note that the ANALOG data is organized into action folders, but does not have the lowest trial-level folders present in the THEIA data structure - this is another requirement for the data structure.
Figure 2: Example data structure for merging analog and Theia3D kinematic .C3D files. This example dataset shows 10 subjects’ data, organized into [subject]_THEIA and [subject]_ANALOG folders. The structure and contents of these folders are shown on the right side for one subject.
With our data organized in this fashion, we can configure the Manage_Theia3D_Merge command. The interface for this command is shown in Figure 3. The Manage_Theia3D_Merge pipeline command can perform three types of merges with Theia3D kinematics:
Merge Theia and Other Files
Import Analog Files
Merge Theia Files
The first option, Merge Theia and Other Files, allows you to merge files containing marker-based data from the same trial, analog signals, and build a model from a specified static trial and model template.
The second option, Import Analog Files, allows you to import analog signals from the same trial. This option is the one we will be using.
The third option, Merge Theia Files, allows you to merge files containing markerless kinematics from the same trial, such as for more than one tracked person.
Figure 3: Visual3D’s Manage_Theia3D_Merge pipeline command interface.
As you can see, the interface for this command is quite comprehensive, and may be intimidating at first glance. However, there are only a few required entries in order to use this tool to import our analog data:
Root path to search: folder path containing all of the [subject]_THEIA and [subject]_ANALOG folders, in this case D:\Data\.
Automatically save default CMZ files: whether to save .CMZ workspaces, using the folder-level selected in the dropdown to the right. <SESSION> in this case refers to the subject-level folders.
File Tags: whether to tag each .C3D file in the workspace using the selected folder level details, in this case: ACTION, which refers to the action-level folder names.
Primary Data Folder: DEFAULT.
Theia3D Session Folder Mask: mask to select the Theia kinematic folders, in this case: *_THEIA
Theia File Mask: mask to select the desired Theia kinematic files, in this case: *pose_filt_0.c3d
Import Analog Files: [selected]
Secondary Data Folder: the folder level of the secondary data folders, in this case: SESSION, which refers to the subject-level folders.
Other Session Folder Mask: mask to select the analog folders, in this case: *_ANALOG
Other File Mask: mask to select the analog .C3D files, in this case: *.c3d
With the Manage_Theia3D_Merge command configured as shown above, for the data structure shown above, the command can be executed and the merge will be performed. This will create subject-level .CMZ workspaces that contain the merged kinematic and analog signals for each trial within the subject folders. These .CMZ workspaces will be saved within the [subject]_THEIA folders, and are named based on the dropdown selection, as shown in Figure 4 below. The execution of this command is quite fast, and performs the merge for all trials at once, requiring only one call of the command to achieve the merge.
Figure 4: Updated data structure for one subject showing the newly created .CMZ workspaces with merged kinematic and analog signals, saved within the [subject]_THEIA folder.
The second approach is using the Import_Signals_From_C3D_File pipeline command. This approach is less structured and therefore a bit more flexible, but also more complicated. If you are importing signals into your markerless kinematic .C3D files in the midst of performing other processing steps in the pipeline, this may be a better approach in this situation. However, this command must be executed separately for every individual trial that requires merging. So, this can require setting up a more complex pipeline that iterates through a list of .C3D files, importing signals from separate but corresponding analog .C3D files for each import. An example of this method is included in a Visual3D pipeline script available upon request at the end of this blog post.
The example pipeline script is written to use data that is structured as shown in Figure 5 below, but since the reading and merging is performed by this pipeline it could be rewritten to use data in any number of formats. As above, the THEIA_c3d folder is structured as it would be output from batch processing in TMBatch using the Use Hierarchical Names option set to Level: 3. The ANALOG_c3d folder is similarly structured, and has .C3D files within the lowest trial-level folders that are identical to those in the THEIA_c3d folder, excluding the “pose_0” and “pose_filt_0” suffixes.
Figure 5: Example data structure used by the example pipeline that utilizes the Import_Signals_From_C3D_File pipeline command. This data structure is not necessary for this specific pipeline command, as the pipeline itself could be rewritten to use data in any number of structures. This example dataset shows 10 subjects’ data, organized into ANALOG_c3d and THEIA_c3d folders. The structure and contents of these folders are shown on the right side for one subject.
The Import_Signals_From_C3D_File command does not have a graphical interface like the Manage_Theia3D_Merge command, and instead needs to be modified directly. There are several entries that can be used, but only a few are required to be able to use it to merge data. The pipeline command is copied below, with relevant parameters described:
Import_Signals_From_C3D_File
/FILE_NAME= [.C3D file within the workspace where signals should be imported into]
/IMPORT_FILE_NAME= [.C3D file from which signals should be imported from]
! /REPLACE_EXISTING_SIGNALS=FALSE
/SYNCH_OFFSET= [0]
! /IMPORT_ANALOG_SYNCH=
! /IMPORT_ANALOG_THRESHOLD=0.0
! /IMPORT_ANALOG_ASCENDING=TRUE
! /ANALOG_RATE= [integer value of the analog signal sampling rate; can only use one of ANALOG_RATE or ANALOG_RATIO]
/ANALOG_RATIO= [integer value of the ratio between analog signals / kinematic signals; can only use one of ANALOG_RATE or ANALOG_RATIO]
! /ALLOW_RESAMPLE=FALSE
! /INTERPOLATION_METHOD=CUBIC_SPLINE
! /IMPORT_AS_DERIVED=FALSE
! /DERIVED_FOLDER=EMG
! /IMPORT_FORCE_PLATFORM_PARAMETERS= [TRUE, if importing analog force signals]
! /ADD_PREFIX_TO_IMPORT=
;
With the data structured as shown in Figure 5 above and using the example pipeline script, the pipeline can be executed to begin the merge.The user is prompted to select the THEIA_c3d folder, and the merge proceeds. Each analog .C3D file is imported into the corresponding kinematic .C3D file, and subject-specific .CMZ workspaces are saved within the subject-level folders in the THEIA_c3d folder. All-in-all, this achieves a nearly identical merge as the Manage_Theia3D_Merge pipeline command, but requires a custom-written pipeline to achieve. However, writing your own pipeline to suit your specific data structure needs can provide greater flexibility and control over your data processing.
Conclusion
Being able to integrate signals from external devices with markerless kinematics from Theia allows you to include diverse, useful signals such as ground reaction forces or EMG activity with your markerless data. This is achieved as a post-processing step in Visual3D, and can be performed using the more comprehensive Manage_Theia3D_Merge pipeline command which requires your data to be organized in a specific fashion, or the more flexible Import_Signals_From_C3D_File pipeline command which requires being written into a custom pipeline script to achieve batch-integration of signals. With these two approaches, you should be able to achieve your goals of analyzing synchronized markerless kinematics with other important signals from external devices.
Click here to request the example Visual3D pipeline script referenced in the second merging option.
To learn more about Theia, click here to book a demo.