“Fixer-upper” isn’t a term anyone wants to apply to their software.
Much like your average homebuyer, software end users want a product that promises safety, security and a lack of bugs before they ever commit to purchase, and they rely on the products that deliver on that promise.
As any software engineer will tell you, well-oiled, bug-free software is no easy task. It requires rigorous testing to ensure the satisfaction and safety of the customers who use it.
Testing takes time, and how an engineering team spends its time testing in the building process says a lot about the organization’s holistic approach to every product. Organizations that deliver a sense of reliability and security for their customers know to keep safety in mind from the start of the development process.
“At Motorola Solutions, we believe safety is the foundation on which everything else is built, and this principle lies at the heart of our engineering.” said Daniel Wilburn, director of product management and device development, vehicle intelligence.
Automated testing, while sometimes tedious to maintain throughout the building process, can prove vital in more ways than one. Teams like Intercept Games embrace their automated testing upkeep as an opportunity to sharpen their newer engineers.
“For someone new to the code, there is an opportunity to understand what the code does by how it is tested,” said Engineering Director Jeff Goldian.
By testing their software, teams like Intercept Games can likewise keep their engineers’ skills up to par for their next big undertaking.
And for that next big undertaking, each engineering team brings their own experience and lessons learned along with them. Built In sat down with Goldian and Wilburn to learn more about how their engineers overcome obstacles and stay on top of safety while delivering reliable, top-tier software.
Intercept Games is a development studio with a deep passion for space, simulation, strategy and all things video games.
Tell us about a time when your engineering team ran into a software issue. What was the problem, and how did your team solve it? What is one lesson your team learned from the experience?
When Kerbal Space Program 2, or KSP2, shipped, the performance of the game was problematic on multiple fronts. The requirements to run the game were higher than we wanted. Even on better equipment, the game would not consistently run well and we would drop frames.
The first part of having a problem is to admit you have a problem and acknowledge its importance, which is what we did. We actively spoke about performance problems as a first-class issue and looked to address those at the same level of importance as game issues. Our next focus was to start to get consistent with how we measure performance, where we measure it and how we understand where the issues are. This led to multiple improvements across our graphics pipelines, simulation and gameplay.
“We actively spoke about performance problems as a first-class issue and looked to address those at the same level of importance as game issues.”
We learned that we needed to keep this level of focus and measurements, even while building new features or making the game better. This seems obvious in retrospect, but when you are pushing hard to get the functionality and fun of a game in, you can lose track of other areas. Having a clear set of priorities and best practices becomes even more important.
What is one way your engineering team reduces risk and ensures that your software is safe and reliable?
Automated testing as a form of documentation. Ultimately, every written document I have ever written or read was — or soon became — out of date. The nice thing about automated testing is that it is always running, so if something changes, your tests need to be updated. Testing gives you the edges of the function and how it should be used. By its very nature, it helps set up a contract on how a function or set of functions should work, which is incredibly useful for learning and regressions.
What is one best practice you think every engineering team should follow when it comes to software development safety?
Have regular conversations about what you are planning before you build. When you are crunched for time, the conversations are often the first thing to go, and you end up building something multiple times with more errors. You have to go slow to go fast. Ensuring you have the time and space early on to discuss the approach of how things will work and talking it through with other dependent teams is incredibly useful, but often overlooked.
Motorola Solutions provides products and services for mobile, wireline digital communication, information and entertainment applications.
Tell us about a time when your engineering team ran into a software issue. What was the problem, and how did your team solve it? What is one lesson your team learned from the experience?
Our software team builds technologies that help keep people safe and secure everywhere, and with this imperative, availability and reliability are critical. There was a time when we found ourselves facing a significant data load increase that was not forecasted. The new volume of data could create workflow challenges during peak times and maintenance routines or max out storage.
Multiple engineering teams collaborated to isolate the bottlenecks and enable scale, all while making minimal changes to our core code. We were able to remove all bottlenecks and implement a plan that safely and securely kept data flowing and accessible with minimal downtime to the end user.
This experience highlights how critically important it is to have a scalability plan in place while taking the time to allocate development resources for a large migration project. We also learned that rapid innovation is best when it comes from all team members bringing ideas to the table, openly collaborating and ideating — all backed by our decades of experience working with our customers to support confident operational performance.
What is one way your engineering team reduces risk and ensures that your software is safe and reliable?
To reduce risk and ensure reliability, we’re always looking for opportunities to simplify and limit code changes. We know that the more complex a solution is, the more it can be prone to bugs and defects.
“Rapid innovation is best when it comes from all team members bringing ideas to the table, openly collaborating and ideating.”
We also understand the importance of the validation of code in development environments while having multiple engineers review changes. This means scanning code before each deployment and having other teams assess the solution. By doing this, we help ensure the software is safe and reliable.
Additionally, we conduct extensive field tests following significant software changes or enhancements. For example, we recently introduced a new performance mode on one of our cameras which was a significant change, touching multiple aspects of our firmware. This led to field tests for multiple cameras running live traffic for several weeks to verify that the feature met our standards.
This practice enables us to iterate and release quickly, with the safeguards in place to continuously enable our customers with technology advancements.
What is one best practice you think every engineering team should follow when it comes to software development safety?
Our engineers are steadfast in our never-ending pursuit to help keep people safer everywhere. And given the growing complexity of threats and challenges, it’s vital that we are constantly evolving and learning together.
As a team, we’ve adopted best practices of code review and quality control methodology. This means we ensure multiple developers review code changes, and we expand this protocol to introduce additional review cycles commensurate with the criticality or risk profile of the update. Our leadership team also highlights when risk warrants assessment, which is key in steering a release or change.
I’m incredibly proud of our impressive engineering team, building software to help to protect people, property and places so everyone everywhere can thrive.
