Testing
We have invested in testing frameworks and software which can help you test all aspects of the embedded firmware in your devices. Testing ranges from very simple checks like verifying that the source compiles. Or more complex tests which verify if the system ‘as a whole’ functions as specified.
All components we have or can supply you with, are always tested in a CICD matter. Any change made to one of those components (backends, reference distribution, device agent, customer project implementation) will result in that component being unit-tested but will also trigger an integration test. A minimal integration test will test if the firmware can boot and talk to the backend and is being able to update itself to a new version after which it must keep functioning. This will ensure that any customer made changes will be tested against the complete system but also the other way around: any changes we make are also tested against all customer projects.
If a problem arises we will most probably detect it before it gets deployed to any device.
Simulator testing
Software can often also be tested in a simulated environment. This saves you space, energy and obviously, you do not have to have the hardware available for it. Simple tests like if the firmware is still being able to connect to the backend, the device agent will keep contacting the backend and firmware upgrades can be tested in a simulator like QEMU. This can also be scripted automatically on the test build server.
On target testing
The most complete and representative test can also be executed on the hardware itself. We have developed both hardware and software for that purpose. This functionality can also be offered as a service in which case the physical hardware will be placed in our testing room. On request, this can also be installed on premises.
(*) can be securely connected over the internet; the devices do not have to be on premises.
Test cases
When a developer changes something in the (any) source repository, this is detected and fetched by the testing build server. Such a change can also be any change made in any of the used opensource components. This allows us to test any Linux kernel or userspace/application change which is made by the community to be tested automatically. When security issues are to be found and fixed they are tested very soon which allows you to get those in your devices quickly.
When the changes are collected, a firmware image will be built by the build server. After the building is completed, it will be transferred to the ‘device-under-test-controller’ slave device. This is a network multi-homed embedded target which is connected to the ‘device-under-test’ (your device). The controller device can power-cycle and flash the DUT to allow it to be used even after having being bricked by a possible problematic firmware package earlier.
After the triggering firmware package is flashed into the DUT, it will also be tested by special test software running on the DUT controller. That software is highly configurable and can test various functions on the system. It can test both things in the DUT as well as services externally (backend) to which the DUT should connect. If all testing goes well, also the OTA functionality will be tested by flashing another firmware package into the device. But this time the DUT has to do it itself, just like how any device in the field will do it.
This will verify if any firmware package ever being built by the system can still be updated using the OTA system.