Skip to content

Evaluation Criteria Catalog

In order to evaluate different technological approaches concerning the Saga pattern, some criteria have to be defined against which the evaluation can be performed. The following sections consider several areas of interest related to characteristics of microservices and the Saga pattern as well as some quality attributes of the ISO/IEC 25010 Quality Models1. For each area, the aspects that an evaluator should examine are explained and described. Some of the described criteria have been taken from the previous paper published by Dürr et al. [3]. The aim is to create a criteria catalog that can be used to analyze other technologies in this context.

Expandability

Saga definitions might change over time, for example, when another service is being offered to the user or some functionality is no longer supported. This creates the necessity to update or extend existing definitions. Therefore, the following criteria examine different aspects that are relevant when modifying an existing Saga. Again, the used criteria originate from the evaluation done by Dürr et al. [3, p. 79 f.].

Terminating/ Pausing running Sagas

Sagas for complex business workflows can potentially take days to complete. Thus, this criterion assesses whether the technology offers actions to deal with this situation, like stopping or pausing all ongoing Sagas.

Versioning Sagas

Instead of waiting for a Saga to finish or aborting it, the definition of different versions allows finishing Sagas of an old version while already starting new Sagas with the updated definition. This criterion analyzes if the technology offers this possibility.

Built–in language support

This criterion comprises a listing of the programming languages the evaluated technology supports, for example, by offering a corresponding client or DSL.

Any language for orchestrator

Investigating whether the orchestrator can only be written using the supported built–in language or any other programming language, for example, due to a provided API.

Any language for participant

One characteristic of microservices is that services can be realized using different technologies [2, 7]. This criterion is similar to the one before. However, it determines if a participant can be written in any programming language and still participate in the Saga using the evaluated technology.

References

[1] N. Alshuqayran, N. Ali, and R. Evans, "A Systematic Mapping Study in Microservice Architecture." IEEE Computer Society, 2016, pp. 44–51. [Online]. Available: https://dx.doi.org/10.1109/SOCA.2016.15

[2] S. Newman, Building Microservices – Designing Fine–Grained Systems, 1st ed. O’Reilly Media, Inc., 2015, ISBN: 9781491950357.

[3] K. Dürr, R. Lichtenthäler, and G. Wirtz, "An Evaluation of Saga Pattern Implementation Technologies," in Proceedings of the 13th European Workshop on Services and their Composition (ZEUS 2021), Bamberg, Germany, February 25–26, 2021, ser. CEUR Workshop Proceedings, vol. 2839. CEUR-WS.org, 2021, pp. 74–82. [Online]. Available: http://ceur-ws.org/Vol-2839/paper12.pdf

[4] D. Cruz, T. Wieland, and A. Ziegler, "Evaluation Criteria for Free/Open Source Software Products Based on Project Analysis," Software Process: Improvement and Practice, vol. 11, no. 2, pp. 107–122, 2006. [Online]. Available: https://doi.org/10.1002/spip.257

[5] J. P. Confino and P. A. Laplante, "An Open Source Software Evaluation Model," Int. J. Strateg. Inf. Technol. Appl., vol. 1, no. 1, pp. 60–77, 2010. [Online]. Available: https://doi.org/10.4018/jsita.2010101505

[6] T. Cerny, M. J. Donahoo, and M. Trnka, "Contextual Understanding of Microservice Architecture: Current and Future Directions," ACM SIGAPP Applied Computing Review, vol. 17, no. 4, pp. 29–45, 2018. [Online]. Available: https://dx.doi.org/10.1145/3183628.3183631

[7] O. Zimmermann, "Microservices Tenets," Computer Science – Research and Development, vol. 32, no. 3–4, pp. 301–310, 2016. [Online]. Available: https://dx.doi.org/10.1007/s00450-016-0337-0

  1. https://iso25000.com/index.php/en/iso-25000-standards/iso-25010?start=0, last accessed 2021-07-06 ↩

Last update: 2022-02-15
Back to top