java - A Scalable Architecture for Reconstructing events -
i have been tasked develop architecture data transformation pipeline. essentially, data comes in @ 1 end , routed through various internal systems acquiring different forms before ending in destination.
the main objectives -
- fault tolerant. message should recoverable if 1 of intermediate systems down.
- replay/ resequence - message can replayed stage , should possible recreate events in idempotent manner.
i have few custom solutions in mind address
- implement checkpoint system message can logged @ both entry , exit points @ each checkpoint know failure happens.
- implement recovery mechanism can go logged storage ( database, log file etc.. ) , reconstruct events programmatically.
however, have feeling standard problem defined solutions.
so, welcome thoughts on suitable architecture go with, tools/packages/patterns refer etc..
thanks
akka obvious choice. of course scala version more powerful, java bindings can achieve lot.
i think can follow cqrs approach , use akka persistence module. in case it's easy replay sequence of events, because have persistent journal.
generally actor model provides fault-tolerance using supervision.
akka clustering give scalability need.
really awesome example of using akka clustering akka persistence , cassandra - https://github.com/boldradius/akka-dddd-template (only scala unfortunately).