What to model?

In today’s blog we continue to look at how the ASTC Embedded Systems Group (ESG) uses simulation and virtual platforms in our everyday workflow. As is the case for many VLAB customers, the ESG group are working with increasingly powerful and complex hardware. The obvious flow on effect …. simulators need to match this increase in scope and complexity, which creates the following problems:

• Accessing or creating new models can be an expensive and / or time consuming exercise, which delays when testing can start.
• Bigger and more complex systems increases the volume of simulation required, impacting performance and slowing execution of tests.

What is the solution?

Over time we have found that different development and testing scenarios can use various simulation abstractions. With this approach we have answered “What to model” and found it easier to achieve our goals with the minimum required effort. Using firmware testing as an example, we use hardware models that include:

• Function stubs
• Register and interrupt stubs
• Simple ‘passthrough’ models
• Detailed stateful models

Typically, models are written in either Python or C++ depending on the needs of the use case. Simple system models can be completed in a few days with Python. By contrast, a peripheral model developed in C++ will generally be more detailed but take longer.

Function Stubs

In some of our testing scenarios it is beneficial to stub the target software, which means that hardware models are unnecessary. Driver functions are intercepted and completed by the simulator and interrupt handlers are directly called by test environment. Similar to host emulation, VLAB’s function stubbing allows us to use the target binary as-is.

Register and Interrupt Stub Models

This class of models provide limited representations of the hardware, where modelling is restricted to registers and interrupt signals with no functional behaviour. To create these models we use VLAB to synthesise high level descriptions.

The advantage of using these models, other than being quick to create, is that values the firmware reads or writes to a register are available to a test script. In fact the test scripts are required to provide any functional behaviour required by the firmware. Typically we write tests that manipulate register values for the firmware to read and assert interrupts to alert the firmware. Using this test methodology we can define and validate the system interaction for the firmware.

As mentioned, to create such models is generally very quick. Take for example ARM Cortex or RISC-V, which we can generate a complete register stub model from a system descriptions such as CMSIS-SVD or IPXACT. In contrast, coding of test cases and scripts is dependent on the level of behaviour required by the test scenario and tends to be longer. 

For those wondering what type of functionality can be included in a test script, we have modelled:

• The pre-amble for each test writing to a PLL register to set the LOCK bit at a pre defined time.
• An ADC input provided by the test script writing to the ADC register.
• A communications interface by directly populating the area in memory used as a receive buffer and verifying the contents of the transmit buffer directly.
• Clearing interrupt status bits..

Simple ‘Passthrough’ Models

These models are an extension of the Register Stub Models. The same synthesised base class defines all registers and ports. Ports are defined to allow system inputs and outputs. In this case we add some event call-backs to register read/writes and ports. However detailed state, behaviour and timing are only modelled where required.

The choice to create such a ‘Passthrough’ model is usually determined by what the test case(s) require. In our work we have selected such models for the following scenarios:

• Modelling a PLL to assert the ‘lock’ bit as soon the firmware writes to a given enable control register. Often this is needed for a firmware start-up routine.
• An ADC may be modelled as a port event call-back that converts a voltage level to an integer and populates a register.
• A communications interface might be modelled by translating register read/writes directly to a port interface. If a FIFO is needed for RX it could be ‘infinitely’ sized with no bandwidth limit.
• An interrupt register model will implement a call-back to clear status bits and de-assert interrupts.

This methodology typically results in a broader system model. To manage the creation and connection of such system models we use high level python components in VLAB to represent the system.

Detailed Stateful Models

These are the final evolution of our models which again build on top of simple ‘passthrough’ models. In these models, register call-backs are extended to add state and timing behaviour. Importantly we still avoid modelling details that are not visible to the firmware.

As a demonstration of how we often build upon and reuse models to suit our needs, we have extended the previously mentioned ‘Passthrough’ models from above:

• The PLL now includes a timer and state machine, which on a transition to the on state, waits for a delay to represent the time to achieve lock and then sets the lock bit. With this model we have tested if firmware is polling the register a number of times and waiting for this to happen.
• A communications interface will now include a realistic sized FIFO, with watermark interrupt supports and a RX/TX rates matched to the hardware interface bandwidth.

What type of model… What works for you!

Our experience has been that you CAN test firmware early using a minimal initial investment in modelling. As the project progresses, models are built up to match growing firmware functionality. The most important lesson learnt has been that you only need to model enough of a system to enable testing.