Independent of collection modality, occlusions are a common occurrence when measuring human motion. Even when only a single person is being measured, and without any object obstructions, the body itself will occlude the contralateral side. Moreover, there are other types of occlusions and they are all handled somewhat uniquely by Theia3D. The objective of this blog post is to provide some background on the different types of occlusions, how they are managed, and how to limit their effect on the accuracy of your data.
Type 1 - Self and Partial Occlusions
The first type of occlusion, which we call self or partial occlusion, occurs in every single data collection. If you consider a fully sagittal view of a person, you may be able to see some of the contralateral side, but it is predominantly not visible. With traditional motion capture, the markers themselves would not be seen, and in order to measure that side, other cameras would have to see those markers. Markerless measurements by Theia are a little different. Regardless of the angle, we infer all 124 key points on the body. Basically, this is their location in the 2D view. Now in this case, for the contralateral side, the inferred (or estimated) locations may be quite close, but since they can’t really be seen, it may not produce 2D locations that are very accurate. So, in addition to the 2D locations, we also produce a probability associated with each 2D detection, with highly probable detections occurring for key points that are clearly visible. When we consolidate all the data from the 2D views into 3D, we consider the key point location as well as the probability; in this case, the contralateral probabilities will likely be lower and as a result, may not be included in the final 3D solution (and they shouldn’t be!). So the distinction here is that Theia3D has to run some statistical tests to determine which key points are high quality and which ones are lower quality - and in some cases, there are low probability points that are in fact high quality. Effectively, we employ a variety of complex statistical tests to produce results that yield the most accurate skeleton in 3D. I find it interesting to estimate “easy locations” for occluded views by eye and see how well I do - and most of the time my guesses are pretty bad. It’s really hard to guess well in 3D!
As I mentioned above, there is no great way to fully avoid this type of occlusion in a collection, because it's just the nature of being human and measuring key points in 3D - depending on the angle, parts of the body just won’t be visible. However, it is possible to limit how it affects your data. Using the description above, one key element is to ensure that all limbs have great visibility in at least three camera views, and ideally five. So, if your camera setup meets these requirements, this likely won’t be a problem. The challenge with this statement is that it varies depending on where the person is located within the volume. For instance, in a volume with the cameras mounted to the walls, this assumption likely holds in the center of the volume, however as the individual approaches the edge of the volume, the entire anterior side of the body is occluded for all views, and while the person may be visible in multiple views, the tracking on the anterior side may be questionable. Avoiding this just means knowing your volume and, as I mention in every blog, running a pilot test to understand where you are able to collect high quality data within your space.
One additional note on this is that all camera setups are not equal - so when we mention 3-5 cameras seeing the limbs, it means 3-5 cameras in appreciably different locations within the volume that are conducive to accurate triangulation. For example, if a hand is seen in 4 cameras, but they are all very close together, this is basically equal to one camera seeing the limb. Avoiding this requires variety in camera locations - it is not recommended to cluster cameras close to each other because they see such similar images, and triangulating in these cases may yield low residuals but inaccurate overall 3D location of that location.
Type 2 - Full Occlusion
The second type of occlusion that happens less frequently is a full occlusion, which typically occurs when interacting with objects, or when multiple people are in the scene. When the entire person is not visible, there isn’t a whole lot we can do. However, sometimes only one particular limb is obscured in all cameras. For example, if someone is reaching into a mailbox to grab something, while in the mailbox, the hand is fully obscured. Using the description above, the locations of the hand will still be guessed, and those guesses will probably be pretty good! However, it’s difficult to tell what will happen to the tracking based on just this description - the statistical process we use to assess the quality of those inferred locations may or may not reject them as high quality. If this occurs in most views, that particular limb (and its parent segments) will drop out of tracking - which would result in the skeleton disappearing for that particular part of the trial. Now, I say this often, but this isn’t magic! If we can’t see it, we can’t measure it accurately. However, our algorithm is pretty good at guessing, so in some cases, these guesses will result in tracking that is consistent in 3D and passes our statistical tests, yielding a tracked skeleton despite full occlusion. The likelihood of this occurring increases as the number of cameras that are guessing at those locations increases, as well as the difficulty of the guesses. For example, using the mailbox example above, if the cameras can see the whole arm, and are just missing the hand, the guess will be a lot better than estimating from behind where the entire arm is not visible. Determining the difficulty of these guesses is pretty easy - it will guess a little bit better than what you can do by eye. If you are having a lot of trouble guessing the joint location, so will we.
In these shots, you can see how Theia3D can still track the hands despite being occluded by baseball gloves.
Type 3 - Anatomical Occlusion
The final type of occlusion that we encounter somewhat rarely is anatomical occlusion, where a limb is atypical compared to normal. Though this isn’t really an occlusion per se, it’s handled in a similar way. Consider a person with a foot amputation where two trials have been measured, one with an orthosis and one without (participant using crutches in second trial). For the trial with the orthosis - as long as the orthosis looks like a foot - the analysis will likely produce accurate results. However if the orthosis looks very different from a foot, it probably won’t track well. This is a consequence of how we train our algorithms, which are effectively trained to track humanoids. Though humans come in all shapes and sizes, all feet/limbs look similar compared to a blade orthosis. In the case where the orthosis is not used and the foot isn’t present, this would likely result in inaccurate and low quality foot guesses - with empty 3D trajectories. The foot, as well as the parent segments in the IK chain would not be tracked - however if free feet are being used, only the foot would drop out and the remaining lower body would likely track.
Here is a right leg occluded by pants that demonstrates how certain anatomical occlusions do not pose an issue for Theia3D's tracking.
This isn’t really something that you can avoid depending on the collection - however, you can test this ahead of time to determine whether the measured results are consistent with your expectations. There are so many varieties to this that it once again comes down to pilot testing to see if the system and the algorithms can measure the motion and person of interest.
Overall, most occlusions are handled automatically and given a camera system that is set up according to our recommendations, will not be an issue for your collections (for more context, refer to our General Framework for Data Collections blog post). In any case, running pilot testing and knowing your capture volume will allow you to better understand what is possible.
To learn more about Theia, click here to book a demo.