In the ASTC Embedded Systems Group (ESG), we like a lot of development teams work with multiple generations of evolving software and hardware platforms. We manage this challenge by seeking to reuse and build up libraries of common software. As you can imagine, verification and validation against a constantly growing matrix of configurations is complex and time consuming.

Continuous Integration of Embedded Software

Some time ago we joined the growing adoption of Continuous Integration, demonstrated by our push into continual regression testing. With this approach we maintain visibility into the status of early generation components that share code with current generation components under active development.

While that sounds easy to say, for embedded software that is reliant on target hardware, maintaining test environments for multiple generations of hardware is hard. Sacrificing desk or lab space inevitably leads to taking your laptop to a common area to work. Again, with access to VLAB VDMs, our test environments are all virtual and available on demand. With licenses and compute resources the only limitation, we scale according to our needs and share the cost across teams.

In a recent review we compared our process with the manual process. Setup and/or managing hardware test environments, executing the tests and collating results… Our estimate is that our approach takes less than 1/2 the time.

For our purposes, we find Jenkins a great way to manage and run our regression test runs. Triggered on demand, periodically or in response to commit hooks, we maintain oversight over current development. The most important point, ensuring the ongoing integrity of deployed solutions.

Embedded Testing with Continuous Integration

Typically we use py.test with its extensive features and ongoing support to capture, execute and validate our test specifications. Using VLAB Batch which has been designed specifically for regression testing, tests can be run in parallel. Each simulation instance creates the required test environment, loads the software under test and validates the test conditions described.

This approach has served us well, however over time, we continue to deal with increasingly complex test scenarios. In response, we identified subsets of tests that could be safely executed as part of the same simulation. The result, another saving of time when executing the entire suite.

To take advantage of this situation, we turned to VLAB with it’s builtin py unit test features. In the example below, VLAB creates a virtual test environment, loads target software, then detects and run the tests.