One anti-pattern I’ve observed in networked Akka applications is a tendency to forget that it is, well, networked and to treat the network as a friendly place.

Trusting the network to be reliable

How does that look like? Well, consider the following piece of code:

1

actorOnDifferentNode.tell(veryImportantMessage, sender());

Akka’s default message sending mechanism is the fire-and-forget tell method. And that works great - until it doesn’t. See, tell has at-most-once message delivery semantics. This means that in theory (and in practice!) messages may not make it to their destination.

So what do do?

Alternative 1: Ask and Pipe

A first approach to not blindly trusting the network is to use a combination of the ask and pipe patterns. Whilst ask is not usually used all too much within the boundaries of an actor systems, this is a valid use-case.

The idea here is to apply a boundary as to how long a message can take to reply and to consider it as lost otherwise. At the same time, we do wish to continue operating in the message-sending paradigm and not switch back to callbacks, which is where pipe comes in. See for yourselves:

1
2
3
4
5

import static akka.pattern.Patterns.ask;
import static akka.pattern.Patterns.pipe;

final Future<Object> call = ask(actorOnDifferentNode, veryImportantMessage, Timeout.apply("3 seconds"));
pipe(call, context().dispatcher()).to(self());

At this point, as with any usage of pipe you will get either:

  • the result of the successfully completed future - in other words, the message reply of the actorOnDifferentNode
  • an akka.actor.Status.Failure message that will contain the throwable caused by the failed execution of the future - in this case, it will be an akka.pattern.AskTimeoutException

From there on you can act upon receiving a Status.Failure message and take appropriate actions.

Alternative 2: At-least-once delivery semantics

If you are looking for strong message delivery semantics (think e.g. transactions of some kind) then what you’re really after in the realm of a distributed systems is at-least-once delivery semantics.

Before going further though, let me just say this: there is no such thing as exactly-once delivery semantics. That’s a lie. Whoever claims their system supports this is lying to you - don’t listen to them! What is possible to achieve is exactly-once effects by combining at-least-once delivery with an idempotent reception. The important part here is that there’s no magic involved - the receiving end will need to be idempotent for this to work.

Akka’s built-in at-least-once delivery of messages works in combination with Akka Persistance. The sending side could also crash at any moment and therefore any undelivered message would’ve been lost, were the sender not capable of remembering its state.

Akka provides the facility for retrying delivery so long as no confirmation has been received:

1

deliver(actorOnDifferentNode, deliveryId -> new VeryImportantMessage(deliveryId, transaction));

When the acknowledgment is received you inform Akka about it:

1

confirmDelivery(evt.deliveryId);

I invite you to read more about this facility in the Akka documentation.

Bonus question: does this not break referential transparency?

A question I’m getting from time to time when teaching the Lightbend Akka - Expert course is whether the special treatment of remote ActorRef-s does not in a way break referential transparency. In that course we take a monolithical, one-JVM application and turn it into a multi-JVM, networked application, exploring the tools Akka has to offer so as to make components resilient and elastic. So when a previously local call happens over the wire, one could argue that the meaning of the underlying ActorRef has changed, giving weaker delivery semantics when deployed over the network.

But that’s not the case. See, Akka’s message delivery guarantees are such that tell has at-most-once delivery semantics. And whilst it is true that for local, intra-JVM calls, calls to tell get a lot more reliable, there still is no 100% guarantee of delivery.