The source code for Linphone and Flexisip is evolving quickly. Every few months, new features are added, and every day or so, patch releases are pushed to fix issues reported by our customers via our support service. Source code is regularly refactored to enable further progress in terms of compactness, readability and performance optimisation.
Doing all this while avoiding the risk of introducing a regression or a new defect represents major challenge.
We estimate that about one third of our developers’ efforts are focused solely on enhancing our non-regression procedures and maintaining our test infrastructure to ensure it is reliable and efficient.
In this article, we present an overview of our company strategy in terms of non-regression testing and quality procedures, which can be divided into the following categories:
- Low-level, per component automated testing
- Sophisticated (integration) automated testing (for our Linphone SDK and Flexisip products)
- UI automated testing (for applications)
- Manual testing (by real humans :-) for Linphone applications and server products.
All automated tests are conducted using Gitlab each time a developer makes a modification to the source, via a "Merge request" process, which also includes a code review by a peer developer.
We give a bit more detail on each category below.
1. Low level, per component, automated tests
Linphone SDK and Flexisip are made from several software components that are assembled together. For example, oRTP is our library for implementation of the Real Time Transport protocol, belr is a small library used for parsing protocol languages, and mediastreamer2 is our library for streaming audio and video over the network. These software components have their own test suites as part of their source code, which are designed and maintained by developers to assert that their basic functionality is achieved. When networking is involved, we use a local loopback network, which is simple to understand for anyone familiar with software development.
2. Sophisticated automated tests
Liblinphone is a library whose purpose is to realise VoIP calls, conferences and IM. A real-world VoIP call involves many different aspects, such as signalling protocols, media streaming, network, routers/firewall, and servers. Our objective is to create automated tests that match the real-world usage of Liblinphone as far as possible, which is why we built not only tests on top of the Liblinphone API, but also a complete infrastructure comprising our server products, dedicated to automated testing. This test infrastructure includes:
- Flexisip Account Manager, which allows the test suites to register their own randomised SIP users;
- Flexisip Proxy, which allows test suites to make calls that are actually authenticated and routed through the network.
- Flexisip Conference Server, which allows the test suites to create group chats and audio/video conferences.
The test server infrastructure is deployed based on Ansible scripts.
We developed specific tools together with the Liblinphone test suites to assess the performance and quality of the calls/conference/IM flows:
- A network conditioner component, which simulates packet loss, jitter, and network transmission delay;
- An audio file comparison tool, which allows us to ensure that audio content is streamed from one virtual participant to another without perceptible degradation.
The Liblinphone test suite runs approximately 1700 tests, which include real calls, conference and IM flows in real time, and which cover both typical end-user usage and more advanced usage such as call transfer and customer-specific SIP flows.
Since a call typically lasts between five and 10 seconds, tests are parallelised over multiple threads to prevent the suite from taking too much time. The full test suite can be executed in around 20 minutes, which is compatible with our strategy of agile development.
The test results for each release (including minor ones) are published on our download page (for example, see one of our latest Liblinphone version 5.2 reports).
The principle of the Flexisip test suite is similar: SIP call flows generated by Liblinphone are passed to the Flexisip instance under test, which allows Flexisip to be tested in scenarios that include the entire network stack. In addition, the test suite models the use of Flexisip in a real system comprising the MariaDb and Redis databases, so that Flexisip can be tested automatically in scenarios that are very similar to real-world deployments.
Our testing infrastructure is constantly evolving: for example, a compilation with an address sanitiser is in the process of being added, which will mean that the test results can immediately highlight memory faults or leaks.
3. UI automated tests
Our UI team has developed a framework for the automation of application-level tests by simulating taps on real or emulated Android & iOS devices, and making asserts on the rendered user interface. The test scenarios represent typical user scenarios, and involve creating or configuring a SIP account, making a call, receiving a call, sending an IM, etc.
4. Manual tests
Although most things can potentially be automated, at some point the cost of automation exceeds the cost of manual testing. Our app release process also includes a two-day intense testing program using multiple devices: iPhones, Android phones from various manufacturers, laptops with the Mac, Windows and Linux operating systems, and Bluetooth peripherals.
Conclusion
The extremely large combinatory inherent to communications scenarios means that automated test procedures are required for all of our products. Over the years, our company has built a complete test infrastructure, which enables an impressive number of test cases that reproduce real-world conditions and usage to be executed in less than half an hour. This test infrastructure is the key to ensuring strong quality assessment and quick delivery of our releases, especially in the context of our support service.
Don’t hesitate to contact our team for more information!
