Communication/Messaging Tools and Patterns between Microservices

When it is designed, microservice, required to remember that other services will need to integrate with it. There is no general best style of communication that should be used. In practice, we need to find the best solutions for the problem at hand.

In this post, we’re discussing different approaches and technologies used in designing the communication between microservices, shading light on the most common communication services provided by AWS, trying to make preferences towards the different patterns and services taking different factors into the consideration.

Communication Patterns

In this section, let’s introduce you to the two major communication patterns namely: sync & async patterns.

Synchronous Communication

Synchronous means that all parties need to be active at the same time to be able to communicate.
The simplest form is service A and service B doing direct remote procedure calls (RPC), by invoking service B’s HTTP REST endpoint from service A.

Cons:

  • If service B went offline, service A would not be able to communicate with B.
  • Service A would need to implement an internal failure recovery procedure.

Asynchronous Communication

Asynchronous means communication can still happen even if not all participants are present at the same time.

To enable this pattern, persisting messages is mandatory, otherwise, there would be no way to guarantee delivery in the face of failures.

Asynchronous Communication Patterns

For async communication, two common different patterns can be defined:

Queue-based Async Communication

This is the “traditional” way of doing asynchronous communication. The idea is that when a service needs to communicate with another, it leaves a message in a central system that the other service will pick up later. In practice, these message inboxes are like task queues.

Stream-based Async Communication

Streams are an immutable, append-only series of time-directed entries, and many types of data fit naturally into that format. They also have fairly regular structure (since they tend to keep the same set of fields).

All streams implementations also allow clients to tail a stream, receiving live updates as new entries get added. This is sometimes called observing or subscribing to the stream. The simplest way to use Streams as a communication tool is to push to a stream what you would otherwise publish over Pub/Sub, basically creating a resumable Pub/Sub. Every subscriber just needs to remember the last entry-id it processed, so it can easily resume if there’s a crash or disconnection.

Messaging Patterns

A messaging service can perform its job following one of two common messaging patterns:

Message Queuing

A message queue receives incoming messages and ensures that each message for a given topic or channel is delivered to and processed by exactly one consumer.

Message queues can support high rates of consumption by adding multiple consumers for each topic, but only one consumer will receive each message on the topic.

To ensure that a message is only processed by one consumer, each message is deleted from the queue once it has been received and processed by a consumer (i.e. once a consumer has acknowledged consumption of the message to the messaging system).

In the case of network or consumer failures, the message queue will try resending the message at a later time (not necessarily to the same consumer)

Publish-Subscribe Messaging

Publish-subscribe messaging moves information from senders to consumers. However, in contrast, to message queuing, publish-subscribe messaging allows multiple consumers to receive each message in a topic. Further, pub-sub messaging ensures that each consumer receives messages in a topic in the exact order in which they were received by the messaging system.

Publish-subscribe messaging systems support use cases that require that multiple consumers receive each message and/or that messages are received in order by each consumer.

Communication Tools

The AWS Marketplace offers a large variety of commercial services to augment cloud messaging and collaboration within a microservices system. In this section we will give you an overview at many of these services, its use cases, billing patten, and its compatability with other AWS services focusing especially on the AWS Lambda Functions.

Amazon MQ

Overview
Amazon MQ is a managed Apache ActiveMQ that makes it easy to migrate to a message broker in the cloud. 

When to use it?
Amazon MQ is suitable for enterprise IT pros, developers, and architects who are managing a message broker themselves–whether on-premises or in the cloud–and want to move to a fully managed cloud service without rewriting the messaging code in their applications.

Lambda event source? NO

How to use it with lambda?
One way is using CloudWatch Events as described here.

Billing:
You are charged for broker instance and storage usage, and standard data transfer fees.

Amazon SQS

Overview
Amazon SQS is a message queue service used by distributed applications to exchange messages through a polling model, and can be used to decouple sending and receiving components.

SQS Types
SQS offers two types of message queues. Standard queues offer maximum throughput, best-effort ordering, and at-least-once delivery. SQS FIFO queues are designed to guarantee that messages are processed exactly once, in the exact order that they are sent.

Long/Short Polling
The process of consuming messages from a queue depends on whether you use short or long polling. By default, Amazon SQS uses short polling. When the wait time between the tries to consume the queue is greater than 0, long polling is in effect. The maximum long polling wait time is 20 seconds. Long polling helps reduce the cost of using Amazon SQS by eliminating the number of empty responses when there are no messages available in the queue to be consumed.

Lambda event source? YES

The consumption mechanism with Lambda: polling

Billing:
The cost of Amazon SQS is calculated per request, plus data transfer charges for data transferred out of Amazon SQS (unless data is transferred to Amazon EC2 instances or to AWS Lambda functions within the same region).

Amazon SNS

Overview
Pub/Sub messaging service. Using Amazon SNS topics, your publisher systems can fan-out messages to a large number of subscriber endpoints for parallel processing. Additionally, SNS can be used to fan out notifications to end users using mobile push, SMS, and email. 

Some features:

  • Reliability: In case the subscribed endpoint isn’t available, Amazon SNS executes message delivery retry policies and can also move messages to dead-letter queues (DLQ).
  • Message filtering: Message filtering empowers the subscriber to create a filter policy, so that it only gets the notifications it is interested in.
  • Instantaneous, push-based delivery (no polling).

When to use it?

  • Amazon SNS can be used in workflow systems to relay events among distributed computer applications, move data between data stores or update records in business systems.
  • A common pattern is to use SNS to publish messages to Amazon SQS message queues to reliably send messages to one or many system components asynchronously.

Lambda event source? YES

Amazon MSK

Overview
AWS streaming data service that manages Apache Kafka infrastructure and operations, making it easy for developers and DevOps managers to run Apache Kafka applications on AWS without the need to become experts in operating Apache Kafka clusters. Amazon MSK is an ideal place to run existing or new Apache Kafka applications in AWS.

Lambda event source? NO

Billing:
Pricing is per Apache Kafka broker-hour, and per provisioned storage-hour. AWS data transfer rates apply for data transfer in and out of Amazon MSK.

Amazon KDS

Overview
Amazon Kinesis Data Streams enables you to build custom applications that process or analyze streaming data for specialized needs. You can continuously add various types of data such as clickstreams, application logs, and social media to an Amazon Kinesis data stream from hundreds of thousands of sources. Within seconds, the data will be available for your Amazon Kinesis Applications to read and process from the stream.

We can use it in two ways

  • Send stream records directly to services such as Amazon S3, Amazon ES, etc. We can use a Kinesis Data Firehose delivery stream instead of creating a consumer application, which will deliver the stream to the chosen destination.
  • If we need to process data records in a custom way we need to create a consumer which is an application that processes all data from a Kinesis data stream.

Concepts

  • Shard is the base throughput unit of an Amazon Kinesis data stream. One shard provides a capacity of 1MB/sec data input and 2MB/sec data output. One shard can support up to 1000 PUT records per second. You will specify the number of shards needed when you create a data stream. You can monitor shard-level metrics in Amazon Kinesis Data Streams and add or remove shards from your data stream dynamically as your data throughput changes by resharding the data stream.
  • A record is the unit of data stored in an Amazon Kinesis data stream. A record is composed of a sequence number, partition key, and data blob. Data blob is the data of interest your data producer adds to a data stream. The maximum size of a data blob (the data payload before Base64-encoding) is 1 megabyte (MB).

Lambda event source? YES

The consumption mechanism with Lambda: polling

Billing:

  • Hourly Shard cost determined by the number of shards within your Amazon Kinesis data stream.
  • PUT Payload Unit cost determined by the number of 25KB payload units that your data producers add to your data stream.

DynamoDB Streams

Overview
A DynamoDB stream is an ordered flow of information about changes to items in a DynamoDB table. When you enable a stream on a table, DynamoDB captures information about every modification to data items in the table.
Whenever an application creates, updates, or deletes items in the table, DynamoDB Streams writes a stream record with the primary key attributes of the items that were modified. A stream record contains information about a data modification to a single item in a DynamoDB table. You can configure the stream so that the stream records capture additional information, such as the “before” and “after” images of modified items.

DynamoDB Streams helps ensure the following

  • Each stream record appears exactly once in the stream.
  • For each item that is modified in a DynamoDB table, the stream records appear in the same sequence as the actual modifications to the item.

Lambda event source? YES

The consumption mechanism with Lambda: polling

DynamoDB streams VS. Kinesis Streams:

  • Kinesis Streams allows you to produce and consume large volumes of data(logs, web data, etc), where DynamoDB Streams is a feature local to DynamoDB that allows you to see the granular changes to your DynamoDB table items.
  • DynamoDB Streams auto-scales the number of shards.

Billing:
if you’re processing DynamoDB Streams with AWS Lambda then you don’t pay for the reads from DynamoDB Streams (but you still pay for the read & write capacity units for the DynamoDB table itself)

Amazon ElasticCache for Redis

Overview
Amazon ElastiCache for Redis supports the PUB/SUB standard with pattern matching. This allows ElastiCache for Redis to support high-performance chat rooms, real-time comment streams, and server intercommunication. You can also use PUB/SUB to trigger actions based on published events.

Lambda event source? NO

Billing:
Nodes are described by: vCpus, Mem, Network Performance.

  • On-Demand Nodes: On-Demand Nodes let you pay for memory capacity by the hour your node runs with no long-term commitments.
  • Reserved Nodes: you can choose to make a one-time up-front payment.

Takeaways

Always try to follow the asynchronous communication pattern.

Use queues when:

  • The order of the arrived messages doesn’t matter a lot.
  • You need to consume the message only once.

Use streams when:

  • The order of the arrived data really matters.
  • You need to process the same message in parallel with multiple applications.

SQS

  • Respects the arriving order by the FIFO queue.
  • Supports dead letter queue.
  • SQS can trigger lambda using the event source mapping.
  • SQS uses the polling mechanism, the only way to consume the SQS is by polling it. Event source mapping acts as a scheduler; checking the queue for any new data at a high rate.
  • You can consume the messages in the SQS using long-polling
  • The maximum long polling wait time for the SQS is 20 seconds. 

MQ

  • MQ can’t trigger Lambda.
  • Lambda can use a polling mechanism to consume the MQ.
  • Don’t use Amazon MQ if you’re building a new application, consider it only when you’re using messaging with existing applications and want to move your messaging to the cloud quickly.

SNS

  • Use SNS when you need to send time-critical messages to multiple subscribers through a “push” mechanism, eliminating the need to periodically check or “poll” for updates.
  • Use SNS to push notifications for the end-users.
  • SNS supports the dead letter queue.
  • SNS executes retry policies.
  • SNS can trigger Lambda.
  • A common pattern is to use SNS to publish messages to Amazon SQS queues to reliably send messages to one or many system components asynchronously.

MSK

  • Don’t use Amazon MSK but use Amason KDS instead, which is the equal Amazon solution to Kafka and is more compatible with other Amazon services, unless you have an existed application that is using Kafka.
  • Amazon MSK can’t trigger lambda.
  • Lambda can use a poll mechanism to consume the MSK.

KDS

  • KDS allows you to produce and consume large volumes of data(logs, web data, etc)
  • There is no built-in auto-scaling capability for Amazon KDS, you’ll need to provision enough shards ahead of time.
  • Amazon KDS can trigger lambda.
  • Lambda uses a poll mechanism to consume KDS.
  • Amazon KDS can keep the messages from hours up to 7 days.
  • For Amazon KDS the maximum size of a data blob (the data payload before Base64-encoding) within one record is 1 megabyte (MB).
  • To avoid invoking Lambda with a small number of records, you can tell the event source to buffer records for up to 5 minutes by configuring a batch window.

DynamoDB streams

  • DynamoDB Streams auto-scales the number of shards
  • if you’re processing DynamoDB Streams with AWS Lambda then you don’t pay for the reads from DynamoDB Streams (but you still pay for the read & write capacity units for the DynamoDB table itself)
  • Your lambda will be triggered for every DB operation.
  • Lambda uses a poll mechanism to consume the stream.
  • the records describe DynamoDB events and not events from your domain, but they are limited to events from one table.
  • To avoid invoking Lambda with a small number of records, you can tell the event source to buffer records for up to 5 minutes by configuring a batch window.
  • Does a row being written in DynamoDB make it canon to the state of your system? If yes you can use DDB streams otherwise consider using Kinses.

ElasticCache for Redis

  • It’s a cache, because of its purpose, the information stored in a cache is typically not considered durably stored. A node fails, or the data is evicted, and what you stored is not there anymore. 
  • If you need to cache information for random, quick, and repeated retrieval, and the information is disposable and recreatable, then, of course, ElastiCache is the choice.
  • Can’t trigger Lambda.
  • Lambda will use a poll mechanism to consume Redis.

Pricing estimation for the remarkable Pub/Sub services


Kinesis StreamsDynamoDB StreamsSNS
Monthly 1 message/s$ 10.836$ 0.47$ 1.296
Monthly 1,000 msg/s
$ 47.088$ 1296.000$ 470.000

While there’s a baseline cost for using Kinesis Streams, the cost grows much slower with scale compared to SNS and DynamoDB Streams, thanks to the significantly lower cost per million requests.

Do you know that we use all this and other AI technologies in our app? Look at what you’re reading now applied in action. Try our Almeta News app. You can download it from Google Play or Apple’s App Store

Leave a Reply

Your email address will not be published. Required fields are marked *