This post is a part of our series about Microservices testing:
In this post, we’re putting it all together and turning the theory into a real practicing.
Here is a list of the most commonly used tools for the microservices testing, along with their prominent features, and pricing.
|Katalon||– Support both SOAP and Rest|
– All-in-one shop for API, WebUI and Mobile testing and the combined capabilities among those.
– Support data-driven approach.
– Can be used for automated and exploratory testing
– Support CI/CD integration.
– Support AssertJ, one of the most potent assertion library, to create fluent assertion with BDD style
– Suitable for both non-techies and pros via Manual and Groovy Scripting modes.
|Postman||– Easy-to-use REST client|
– Rich interface which makes it easy to use
– Can be used for both automated and exploratory testing
– Can be run on Mac, Windows, Linux & Chrome Apps
– Has a bunch of integrations like support for Swagger & RAML formats
– Has Run, Test, Document and Monitoring Features
– Doesn’t require learning a new language
– Enable users to easily share the knowledge with the team as they can package up all the requests and expected responses, then send it to their colleagues.
|Free – $21/user/month|
– Create test quickly and easily with Drag and drop, Point-and-click
– Reusability of Scripts: load tests and security scans can be reused for functional test cases in just several steps
– Powerful data-driven testing: Data loaded from files, and databases, and Excel so that they can simulate how consumers interact with the APIs
– Support native CI/CD integrations, asynchronous testing
|Free – $659/year|
|Pact||– Contract Testing||Free|
|Pacto||– Contract Testing||Free|
|Dredd||Validates whether your API implementation replies with responses as they are described in the documentation.||Free|
|API Blueprint||A high-level API description language for web APIs.||Free|
|Swagger||– API Design according to specification|
– Build stable, reusable code for your API in almost any language
– Interactive API documentation
– Perform simple functional tests on your APIs
|Free – $30/Month|
|Hoverfly||– Create API simulations.|
– Cross platforms.
|JMeter||– Supports replaying of test results|
– Automatically work with CSV files, allowing the team to quickly create unique parameter values for the API tests.
– Users can include the API tests in CI pipelines thank to the integration between JMeter and Jenkins
– It can be used for both static as well as dynamic resources performance testing
|Rest-Assured||– Have a bunch of baked-in functionalities, which means users don’t have to code things from scratch.|
– Integrates seamlessly with the Serenity automation framework, so that users can combine the UI and REST tests all in one framework that generates awesome reports.
– Support BDD Given/When/Then syntax
– Users don’t necessarily need to be an HTTP expert
|Karate DSL||– Build on top of Cucumber-JVM|
– Can run a test and generate reports like any standard Java project
– A test can be written without any Java knowledge required
– Tests are easy to write even for non-programmers
– Supports configuration switching/staging, multi-threaded parallel execution
Tools in Context
In addition to the traditional tests needed for a monolith application e.g. unit tests, component tests, etc. for microservices testing we have contract testing and integration testing. These tests ensure that the various services are able to communicate and deliver outcomes as expected.
In this section, we’re exploring the usage of some testing tools in specific test contexts.
Whenever some consumer couples to the interface of a component to make use of its behavior, a contract is formed between them.
Integration Contract Testing
An integration contract test is a test at the boundary of an external service verifying that it meets the contract expected by a consuming service.
Consumer-Driven Contract Testing
In Consumer-Driven Contracts, each consumer captures their expectations of the provider in a separate contract. All of these contracts are shared with the provider so they gain insight into the obligations they must fulfill for each individual client.
An integration test verifies the communication paths and interactions between components to detect interface defects.
Tests the data passed between two integrated services within your system.
Tests test data shared between services that are integrated with other external or third-party services:
Some useful tools: Hoverfly
Testing Practical Scenarios
In order to get a clear understanding of how testing can be carried out in different scenarios, let us look at a few practical examples.
Testing between microservices internal to an application or residing within the same application
Is one of the most commonly encountered scenario.
Let’s consider an e-commerce application that has two services:
- Selecting an item.
- Reserving an item.
We also assume there is a close interaction between these two services.
- Unit testing: it would be ideal to use any unit testing framework for this goal. The change in data internal to the application needs to be verified. For example, if reserving an item provides a reservation ID on success in the response to a REST call, the same needs to be verified within the service for persistence during unit testing.
- Contract testing: In case there are several dissimilar consumers of the service within the application, it is recommended to use a tool that can enable consumer-driven contract testing. Tools like Pact, Pacto can be used. In the context of this example, we assume that there is only a single consumer of the service. For such a condition, a test stub or a mock can be used for testing.
- Internal integration testing: Data being passed between the services needs to be verified and validated using tools like Katalon, Postman, SoapUI, ect. For example, an item number being passed between the services that select it to the one that reserves it.
- E2E testing: should ensure that dependency between microservices is tested at least in one flow, though extensive testing is not necessary. For example, an item being purchased should trigger both the ‘select’ and ‘reserve’ microservices.
Testing between internal microservices and a third-party service
Here we have a service with an application consumes or interacts with an external API.
In this example, we have considered a retail application, where paying for an item is modeled as microservices and interacts with the PayPal API that is exposed to authenticating the purchase.
- Unit testing: should ensure that the service model is meeting requirements defined for interacting with the external service, while also ensuring that internal logic is maintained. Since there is an external dependency, there exists a need to ensure that requirements are clearly defined and hence, documenting them remains key. TDD approach is suggested where possible and any of the popular unit test frameworks chosen for this.
Consider reading our previous post about Test-Driven Development (TDD).
- Contract testing: can be used in this case to test the expectations from consumer microservices, that is, the application’s internal service, decoupling it from the dependency on the external web service to be available. In this context, test doubles can be used to define the PayPal API’s implementation and testing. This is essentially integration contract testing and needs to be verified with a live instance of the external service periodically, to ensure that there is no change to the external service that has been published and consumed by the consumer.
- Integration testing: can be executed if the third-party application developer provides a sandbox (e.g. PayPal’s Sandbox APIii ) for testing. Live testing for integration is not recommended. If there is no availability of a sandbox, integration contract testing needs to be exercised thoroughly for verification of integration.
- E2E tests should ensure that there are no failures in other workflows that might integrate with the internal service. Also, a few monitoring tests can be set up to ensure that there are no surprises. In this example, selecting and purchasing an item (including payment) can be considered an E2E test that can run at regular and pre-defined intervals to spot any changes or breaks.
Testing for a microservice that is to be exposed to the public domain.
Consider an e-commerce application where retailers can check for the availability of an item by invoking a Web API.
- Unit testing: should cover testing for the various functions that the service defines. Including a TDD development can help here to ensure that the requirements are clearly validated during unit testing. A unit test should also ensure that data persistence within the service is taken care of and passed on to other services that it might interact with.
- Contract testing: In this example, consumers need to be set up by using tools that help define contracts. Also, the expectations from a consumer’s perspective need to be understood. The consumer should be well-defined and in line with the expectations in the live situation and contracts should be collated and agreed upon.
Once the consumer contracts are validated, a consumer-driven contract approach to testing can be followed. It is assumed that in this scenario, there would be multiple consumers and hence, individual consumer contracts for each of them. For example, in the above context, a local retailer and an international retailer can have different methods and parameters of invocation. Both need to be tested by setting up contracts accordingly. It is also assumed that consumers subscribe to the contract method of notifying the provider on the way they would consume the service and the expectations they have from it via consumer contracts.
- E2E tests: a minimal set of E2E tests would be expected in this case, since interactions with external third parties are key here.
Real-Life Example Testing Strategies
In this section, we’re describing four known strategies for testing microservices.
The Documentation-First Strategy
- Following a documentation first approach so, all of the documentation is in markdown in GitHub.
- Before anybody writes any API changes or either a new API, they should update the documentation first.
- Have that change reviewed to make sure that it conforms with the API conventions and standards which are all documented, and make sure that there’s no breaking change introduced. Make sure it conforms with the naming conventions and so forth as well.
A further step would be performing API contract testing.
The Full-Stack in-a-Box Strategy
Replicating a cloud environment locally and testing everything all in one vagrant instance.
Has a significant downside:
- Hard to setup.
- A huge difference between the test and production environments.
Personal Deployment Strategy
Spinning up an Amazon Web Services (AWS) infrastructure for each engineer to deploy and run tests on. This is a more scalable approach to the full stack in-a-box strategy discussed above.
The Shared Testing Instances Strategy
It involves developers working from their own, local station, while leveraging a separate, shared instance of a microservice to point their local environment during testing.
It’s how Amazon develop and deploy new services to AWS. Was proposed in the AWS summit in Berlin 2018.
In this strategy:
- The new service first goes through the usual unit tests.
- Then it’s deployed to the first staging environment (beta) where functional testing happens.
- Then it’s deployed to a different staging environment (gamma) where integration and load testing can happen.
- Finally, it is deployed in a very careful manner, first to a single machine then to an availability zone, then to several availability zones.
- This process is repeated across the whole of AWS, with close monitoring of error rates, load, etc. Any anomaly that is spotted will cause the service to be automatically rolled back.
The service is tested in multiple staging environments, which helps ensure that it’s robust against unexpected differences in configuration. The different tests are still essentially the sort that you might traditionally conduct (functional, integration and performance testing), but they are being divided across different setups.
A thing to note is that they are using what is called Canary testing. This involves rolling out a change to a small number of users to start with and using the extensive instrumentation and monitoring capabilities of most microservice environments to ensure that the code is behaving as expected without bad things happening (that is, the error rates aren’t spiking, the performance isn’t dropping and the load on the system isn’t skyrocketing).
What is the Best Testing Strategy?
There is no absolute best strategy. A strategy that may seem flawed or old-fashioned in one situation may be the best solution for another. In your case, it might just fit.
You are the one that should design the best strategy that suits your situation.
In this post, we provided you with a list of the best testing tools for microservices, proposed example scenarios for pre-production testing, and real-life example testing strategies.
Do you know that we use all this and other AI technologies in our app? Look at what you’re reading now applied in action. Try our Almeta News app. You can download it from google play: https://play.google.com/store/apps/details?id=io.almeta.almetanewsapp&hl=ar_AR”