Race Condition in Persistent FSM Actors During Deployment with Multiple Instances

[simonlucalandi]

Hello Pekko community,
I am encountering a race condition issue in my application during deployments and would appreciate your help.

Background:

• My application uses classic persisted FSM Pekko actors, extending AbstractPersistentLoggingFSM, for managing stateful entities.
• The application runs in a single docker container, and the actors are not yet managed through Pekko Cluster or similar mechanisms.

Issue:

• During a deployment, the new application version starts while the old version is still running for a few minutes to allow for a graceful transition.
• Both instances (old and new) share the same event log storage.
• When the new application starts, actors not in a "final" state are "resurrected", leading sometimes to a race condition, as actors for the same persistentId are running on both application
• This leads to inconsistent behavior, due to conflicting writes in the events storage

I want to avoid downtime during deployments, which is why the old and new instances overlap briefly.

Is there a recommended pattern to prevent the race condition during overlapping deployments in a multi-instance setup?
Would Cluster Sharding be the appropriate solution for this issue, or is there an alternative (simpler) approach that could work with classic FSM actors?

Any suggestions, best practices, or guidance would be greatly appreciated.

Thank you in advance for your help!

[raboof]

It is indeed the case that Pekko Persistence works on the 'single writer' principle, so having new and old versions of the same persistence ID writing at the same time is not allowed.

The solution here is indeed to introduce Pekko Cluster and use Cluster Sharding to manage the persistent entities: Cluster Sharding can make sure persistent entities are gracefully migrated to the new nodes during a deployment.