Summary: It’s important and needs to be thought through. Doing a literature review, equipment assessment, pilot study, and more testing is a great starting point to find a frequency that best fits your experiment.
When designing a data collection, one critical factor is the frame rate of the recording. Regardless of the collection modality, the frame rate needs to be selected according to the speed of the movements being measured. Selecting a value that is too high may have an impact on resolution, video quality, as well as unnecessarily increase the size of the data files, whereas recording at a frequency that is too low may miss some subtleties of the motion being measured. The objective of this blog is to provide some information regarding the effect of frame rate, and provide additional context on how to select a frame rate that meets the requirements of your collection.
Before collecting any data, I strongly recommend doing a literature review to see what others have done. Even if the protocol itself is different, a literature review will help you become familiar with other related protocols and can be a good starting point. This research should provide a reasonable range of values that may suit your collection.
With this range in mind, it’s time to review the hardware. The specifications for video cameras can be quite different, and understanding what your hardware is capable of before initiating an experiment is an important consideration. For instance, if the hardware has a maximum recording rate of 60 FPS, and the range of frame rates you found in literature are quite a bit higher, it may be time to either look for different hardware, or look at different questions that are more in line with the capabilities of the system. In a lot of cases, this is a tough pill to swallow, but it’s better to deal with now rather than later when all the data is recorded and you discover it’s unusable.
Some hardware can perform ‘windowing’ which can increase the maximum collection frequency. Windowing only looks at a portion of the sensor, effectively capturing a sub part of the view. In the image below the red outline shows the windowed area that is quite a bit smaller than the full area. As long as the windowed area fully contains the subject (i.e. there is quite a bit of unused space near the edges) it's a good option that allows you to increase the frame rate. A few words of caution - the caveat above is important so make sure you check this - if windowing provides the frame rate you need, but prevents the whole subject from being recorded, it should not be used. As you can see this technique has its own set of tradeoffs.
Windowing can be an effective tool for obtaining an appropriate frame rate for your collection. Here you can see that the window easily captures the entirety of the subject.
Assuming your hardware meets the requirements you found in literature, it’s time to pilot the protocol. This step is critical because theoretical frame rates and what you can actually measure are sometimes different. Moreover, reviewing a small portion of the data to ensure image clarity and quality of results costs you very little time and will pay off in the end. For example, if in theory everything looks good, and you don’t pilot, on the day of collection you may see blurry images, unclear people, or images that are too dark given the lighting in the space. Some of these issues are correctable by tuning the cameras or adding more light, others are not. If you pilot and realize some things need to be corrected, that’s not a big deal! It’s also possible that you realize collecting high quality, clear images will not be possible based on the tasks you wish to collect and your equipment. This may lead you back to the drawing board, or to consider different questions, but again, it’s far better that this happens during the pilot stage before a ton of effort is put into collecting useless data. I sometimes feel like a broken record when talking about piloting, but I wish I had this type of advice when I was more focused on research - it would have saved me a lot of time and frustration.
In some cases, there is a temptation to collect at frame rates that are too high for the movement being measured. For example, collecting at 500 FPS for a walking trial is simply unnecessary. There are other consequences to choosing a frame rate that is too high. Firstly, as the rate increases, the likelihood of frame drops also increases because of higher data throughput demands. Secondly, the quality of the collection can also degrade as the frame rate increases, and additional lighting is often required at rates higher than 100 FPS. Though I’m less concerned about data size - increasing frame rate can also lead to larger files (see our blog post on Data Management). These, in turn, increase storage requirements and analysis time - neither are too bad, but entirely avoidable. As our algorithms change and improve, we recommend reprocessing data, so having large files that take more time to analyze can be a bit frustrating.
Now that we have covered the basics on frame rate selection (literature review, hardware review, and pilot testing), let’s see how frame rate actually affects data. Here we have the same pitch that was originally recorded at 500 FPS. By keeping every other frame, and every fourth frame, we can synthetically generate identical videos of the movement that are recorded at different frequencies.
Here is the original video recorded at 500 FPS.
Here is the same video but with half of the frames (250 FPS).
At a glance, the videos look pretty similar, but when we look at the joint angles, there are obvious differences. When we overlay the lower arm relative to the thorax for 250 FPS and 500 FPS, they are mostly similar, however there are distinct peak differences:
Even though the pitching videos shown previously are practically identical to the human eye, reducing the FPS clearly impacts our joint analysis results.
As we discussed earlier, lower frame rates may not be able to capture subtle motions that occur at high speeds, which are very much present during pitching. The differences between 125 and 500 are egregious, however that is not surprising. This type of analysis is important when looking at the effect of frame rate, but it doesn’t necessarily require removal of frames. This isolated example helps us understand the effects but there are other factors present in a real collection, such as changes in quality and lighting when adjusting the rate that wouldn’t be captured in this situation (everything is constant but the frame rate here). So, if you are collecting something that is highly repeatable (like running), and are unsure what frame rate to use after reading this - try it out in your pilot study! Record at a few different rates, go from start to finish with your analysis (see our blog post, General Framework for Data Collections for reference) and review the results. This is a great litmus test to reinforce the critical thinking put into this small, but important aspect of any collection.
In summary, it’s pretty simple to find an appropriate recording frequency for your experiment. It takes a bit of organization, research, time, and testing, however, the small amount of work you put into this will pay dividends when you go to collect data.
To learn more about Theia, click here to book a demo.