Evaluation Criteria Catalog
In order to evaluate different technological approaches concerning the Saga pattern, some criteria have to be defined against which the evaluation can be performed. The following sections consider several areas of interest related to characteristics of microservices and the Saga pattern as well as some quality attributes of the ISO/IEC 25010 Quality Models1. For each area, the aspects that an evaluator should examine are explained and described. Some of the described criteria have been taken from the previous paper published by Dürr et al. [3]. The aim is to create a criteria catalog that can be used to analyze other technologies in this context.
Fault Tolerance
Since the MSA represents a distributed system, failures have to be expected and the different services have to tolerate them. Therefore, the following criteria concentrate on determining the behaviour of a technology when failures of either the orchestrator or the participant occur. For this purpose, corresponding error scenarios should be simulated with the technology to be evaluated. More information on how such error scenarios were simulated with the different technologies can be found within the respective fault tolerance implementations.
Execution timeouts
The definition of execution timeouts for individual transactions or even for the entire Saga can be a desirable feature to prevent the possibility of infinite waiting. Then again, enforced timeouts are problematic when timeouts should not abort Sagas. This criterion relates to "Enforced execution timeouts" in [3, p. 79 ff.].
Reaction to participant fault
An examination of the behaviour of the orchestrator and the participant if a failure arises within a participant. Several failure scenarios are possible: the participant fails before receiving a message or while executing a local transaction. In addition, a failure during execution can either be a sudden failure like the service crashes or unexpected behaviour such as throwing an exception. These failures happen before the service could return its answer to the orchestrator. The criterion is based upon the "Retry of failing participant without restart" criterion in [3, p. 79 ff.].
Saga continuation trigger after orchestrator crash
Since not only a participant can fail, faults have to be also taken into account for the orchestrator. An orchestrator should be able to continue a Saga where it left off, even if it crashed during its execution. Particularly interesting here is the trigger of such a continuation after the orchestrator crashed and becomes available again. This criterion is related to the "Auto–continuation after orchestrator crash" criterion in [3, p. 79 ff.].
New Sagas while orchestrator unavailable
Depending on who orchestrates the Saga execution and how the communication between the components is realized, it might be possible to start new Sagas even though the orchestrator is temporarily unavailable. This criterion was also part of Dürr et al.’s evaluation [3, p. 79 ff.].
Independent compensating transactions
This criterion is related to the Compensation transaction allocation criterion in the General Saga Characteristics. If compensating transactions can only be defined for the whole Saga, the compensation process needs all services listed in this process available to finish. Whereas, when only needed compensations are executed, only the services that already participated in the Saga have to be available during compensation. This criterion can also be found in [3, p. 79 ff.].
Orchestrator reaction to duplicate messages
Another interesting aspect is how the technology reacts to non-compliance with the protocol. Therefore, this criterion aims to observe what happens when a participant sends duplicate messages to the orchestrator. This can be provoked by causing a participant to send a message twice in immediate succession.
Orchestrator reaction to old messages
Following on from the previous criterion, this time, old messages are resent to the orchestrator. A participant can simulate this by starting a new thread that resends the same message it is about to send to the orchestrator after some time, for example after five minutes.
High availability
This criterion considers if the technology allows achieving highly available systems, for example, by replicating the orchestrating engine. Analyzing High availability was also part of the evaluation done in [3, p. 79 ff.].
References
-
https://iso25000.com/index.php/en/iso-25000-standards/iso-25010?start=0, last accessed 2021-07-06 ↩