Why do I need event sourcing?

If you are working with real people, then you need to save all the actions for the subsequent “debriefing”. For many, this is the main business goal of event sourcing.

If we look to the future, then, in other words, approximately all business applications will sooner or later fully or partially switch to this architecture. Then why aren’t existing solutions being massively used? It still requires additional design and code, but with the constant increase in software requirements, it is already paying off in theory.

There are additional pros and cons, but they are secondary (they influence the choice of implementation, not the concept).

It is worth noting that this is a rather revolutionary deviation from the usual architecture. In other words, additional training for programmers and new best practices will be required. For an example of a negative experience, you can read, for example, this thread: https://news .ycombinator.com/item?id=13339972

According to the theory, you can read the previous note “Classic event sourcing”.

The main problems

  • Eventual Consistency or old data is returned when reading.
  • How to deal with errors that occurred somewhere in the middle of processing?
  • Slow construction of online projections (when there are many events in the aggregate)
  • When storing events in a regular database with auto-increment, events are sometimes lost during processing, because the database allocates blocks of auto-increment ids under load, which are stored in parallel, and the event handler usually searches for the latest ids.

Proposed solution

To solve these problems, the following architecture changes are expected relative to the classical one:

  • The concept of “main projection” is introduced - this is a projection located in the same database as the event store.
  • The main projection must contain all the data necessary for the work of the teams, the built-in projectors of the main database and reactors. Accordingly, if necessary, data is taken from the main projection, and not from the event store.
  • The event store is used to get the history and, if necessary, rebuild the main projection (full or partial).
  • The command, event saving and embedded projectors of the main database work in a single transaction.
  • The reactors operate in a separate transaction.
  • Live projections are not recommended (only for specific cases, taking into account the potential for delay). In fact, there is no such thing as an aggregate class or an aggregate in memory. Work is underway at the command and event level, and the aggregate as an id is only for transactions and parallelism.
  • If limited asynchrony is required, then you can fasten the sending of events via NATS (or another queue).

How are problems solved?

  • There is no eventual consistency, because there is a classic work with data.
  • Error handling is again classic – if everything went wrong, the transaction is rolled back, and the client receives an error message.
  • There is no slow processing of events when receiving data, because for this, the main projection is mainly used.
  • There is no problem with auto-increment ranges, because we do not rely on these ids when processing events.

What is being lost?

  • A completely asynchronous approach (which potentially facilitates scaling) – it can be applied, but only partially.
  • Distribution - it’s not out of the box, you need to think separately
  • Scaling - it’s not out of the box, you need to think separately

In general, based on the “principle of simplicity”, we usually do not need them. If you do need them, then you should probably use the classic approach to event sourcing.

Someone may even say that this is not Event sourcing at all, if we do not have an aggregate in memory that calculates its state based on events. But we have an aggregate in the database that is created by events (which is not memory, just not operational). So it’s still Event sourcing, but I’m pretty glad it’s not classic, because classic has too many limitations for regular applications. I would say that the proposed approach is not event driven (EVA), but event sourcing.

As a result, there is a rejection of the separation of the event base and the main projection. This solves the problems of distribution and asynchrony (there are none). When scaling a solution, I would rather use partitioning of the solution than asynchrony and separate services. But, of course, it depends on the specifics of the solution: in different cases, there may be different answers.

On the one hand, there is nothing supernatural in the approach. On the other hand, I have not met any description or term for such an approach. I suggest using Inline event sourcing, because it is based on an embedded projection.