David Musgrave is a man with a problem.
His latest project is undergoing field testing and he’s just not getting the data he wants on how the weapon system he helped to develop is being used.
The solution to his problem just might come from the video game industry.
“We were encountering some problems with firing tests,” Musgrave said. Musgrave serves as the project lead for fire control software development on the M109 Paladin Integrated Management, the U.S. Army’s latest iteration of its self-propelled 155 mm howitzer.
“I started asking questions looking for objective use data. How often does subsystem X fail? When it does fail, what was the user trying to do at the time? How often does a user perform Y task? The truth was I couldn’t get any decent answers,” said Musgrave who works at the Armament Research, Development and Engineering Center’s Weapons Software Engineering Center.
“I was frustrated that there was a very limited information channel from our system back to us while it was being used,” he said. “I felt like I had helped create a system and then threw it over a brick wall for someone else to use, and my only visibility was to try to peer through cracks in the mortar.”
But something about video games planted an idea in his mind that promised to smash through that brick wall like Super Mario.
PRESS START TO BEGIN
“My background is in software development and I’ve always looked to the video game industry for potential cutting edge ideas; sort of like how an automotive engineer at Ford may follow Formula 1 racing,” he said.
“I do really enjoy playing video games and I’ve been active since the days of Jumpman on my dad’s Commodore 64,” said Musgrave. “I don’t have as much time to play as I once did but I try to get in a few hours a week as a way to relax once the wife and kids go to sleep.”
It was that enduring enjoyment which started as a child that led to his latest idea.
At the same time as the testing was going on, Musgrave was watching a weekly web series called ExtraCredits that focuses on video game development technologies.
“One of their episodes discussed how user telemetry and the resulting metrics were changing the way games were developed,” said Musgrave.
According to Musgrave, user telemetry is the concept of recording data about the use of a deployed system and then transmitting that data back to a central point where it can be analyzed.
This data could consist of fault data, system usage data (how the system is used), performance data (how well the system is used) or anything else needed.
“For example let’s say you had a racing game where the user telemetry was collected and tracked. From a general perspective, the developer would know which tracks are the most popular, which cars are most used and if there are any parts of a track that are more difficult than others,” he explained.
“They could also drill down to see individual users and monitor your favorite track, your favorite car, and how often you crash or leave the pavement. They can then decide how they want to improve for the next game,” he continued.
“Let’s say the Ferrari’s are the most popular car and Honda’s are largely ignored. For the next game they may decide to expand the selection of available Ferrari’s or maybe take a second look at their Honda’s to see why nobody drives them,” said Musgrav.
“Or maybe one hairpin turn is too tough and 95 percent of the players can’t make it without smacking into a wall. That data could lead to a subsequent patch that would widen the turn and make it easier but still challenging.”
“My current favorite is the Mass Effect series from BioWare. Despite the recent controversy with the ending cinematics, the game is a prime example of how user data feeds back into the development cycle,” he said. “BioWare took great pains to listen to their user community and keep the parts of the game that worked and fix what was annoying from one game to the next.
“I dug more into the video game industry and I found a wonderful presentation by Georg Zoller who was a lead technical designer at BioWare,” Musgrave said. “He worked on BioWare’s user telemetry system used on games such as Mass Effect 2 and Dragon Age. I saw the amazing insight that system gave the developers and became excited about what a similar system could mean for the military.”
Musgrave said he believes that many different people in the Army could benefit from the collection and analysis of user telemetry.
“The program manager of a system would benefit from seeing objective fault information including types of faults along with frequency and associated data that would help them re-create the fault and come up with a solution,” he said. “The program manager would also be able to see which functions of the system are being used and how often. This would provide objective data to help them decide where to focus product improvement to get ‘the best bang for the buck.'”
“For instance let’s say we had a weapon system that provides direct and indirect fire modes; and through user telemetry data we determine that 87.5 percent of all engagements are direct fire,” he explained.
“The program manager can then determine if improvements to the indirect fire capability are a worthwhile investment since it is rarely used.”
Another potential beneficiary of a user telemetry program Musgrave identified would be schoolhouses or other training groups.
“If performance data is collected and tracked, they can see which units are performing well and which units may need some help,” said Musgrave. “They can also track training effectiveness by comparing performance data before and after an exercise or introduction of a new teaching tool.”
The final, and somewhat indirect, beneficiary Musgrave identified is the Soldier.
“It will relieve the burden of reporting problems up the chain of command when they are encountered,” Musgrave said.
“A Soldier in the field is not an IT guy and he has better things to do than try to fill out a trouble ticket.”
PITFALLS
Even small ideas can sometimes impose enormous challenges.
“I think there are three big pitfalls that we need to avoid if we want to successfully implement this into fielded weapon systems,” Musgrave said.
“First, we have to make sure that whatever telemetry system we can implement, and that has to be specifically tailored to each weapon, does not impact the existing functionality of that system. The worst thing we could do would be to take a reliable, working system and gum it up with something like this that’s non-critical.”
“The next challenge would be to keep the system small and simple,” Musgrave cautioned. “Far too often concepts like this get expanded quickly to be a leviathan that tries to do everything, for everybody, all the time. The system becomes huge, expensive and ultimately cancelled. We need to start any system small, see what works and build in increments from that point.”
“The truth is that while it may be ideal to collect and monitor 300 variables about a system, it’s probably far too cumbersome to implement properly,” explained Musgrave.
“Pulling only five variables instead may sound limiting, but considering we’re collecting nothing now, it’s an infinite improvement.”
“The final challenge, and the most technical, is developing a method to get the data from the weapon system onto a network so it can eventually make it back to a central server,” he concluded.
“There are some technologies out there that can do this but any solution will have to be heavily tailored to the target platform. For instance pulling data from a rifle scope would be far more difficult than a semi-fixed missile system,” he said.
Despite the challenges that may come with putting together such a system of user telemetry on current and future Army weapons systems, Musgrave says he believes that potential long-term benefits could outweigh the initial costs in time and effort and improve the bottom line for both the troops who use the weapon and the taxpayers who fund it.
“The data collected will eventually lead to a higher quality, more affordable system,” Musgrave added.
