Event Sourcing: what it is, why it’s interesting and why we probably won’t use it (this time)

George

When working with a digital system of record, it’s often important to know not just the current state of the system, but how it got there

When working with a digital system of record, it’s often important to know not just the current state of the system, but how it got there. Whether it’s making decisions in the code based on a sequence of past actions or investigating what happened, having a reliable audit trail can be crucial.

At the British Business Bank (BBB), we’ve been exploring architectural options for a new application that helps small UK businesses access asset finance. As with many financial systems, this raised questions about auditability, traceability, and how best to model change over time.

One option that was suggested early on was “Event Sourcing”, an architecture that, on paper, seemed like the perfect fit for these requirements. To properly understand it, we spent a couple of hack days working through the fundamentals with Ismael Celis, a consultant whose experience working with Event Sourcing guided this exploration. Here’s what we learned.

What is Event Sourcing?

When I first Googled Event Sourcing, I was immediately presented with a number of highly technical blog posts filled with Domain-Driven Design jargon like “Aggregate Root”, “Bounded Contexts”, and concepts like “Command Query Responsibility Segregation” (CQRS). My eyes quickly glazed over and I shut my laptop for a bit.

When you take the jargon away and separate out those orthogonal concepts though, Event Sourcing is a lot easier to digest.

Event Sourcing is about how you store data, persisting events as the source of truth. This is different to Event-Driven Architecture which is about how systems communicate, using events to trigger actions between services.

The most commonly used example to explain the benefits of Event Sourcing revolves around a bank account with a balance field. In the traditional CRUD (Create, Read, Update, Destroy) applications that we all know and love, everything is modelled around “state”: when money is deposited or withdrawn that balance state is updated and the previous value is lost. At any point, you only know the current state. Here’s how a simple class might look:

class Account
  def initialize(balance)
    @balance = balance
  end

  def deposit(amount)
    @balance += amount
  end

  def withdraw(amount)
    @balance -= amount
  end
end

The @balance variable is overwritten with a new amount every time a deposit or withrawal is made.

In an Event-sourced architecture, instead of storing the final balance, you store the events that led to it, such as:

Each event is immutable and appended to a log. To determine the current balance, you replay those events and calculate a running total. This process is known as building a projection. In other words, state is derived from events, not stored directly. Here’s how a (simplified) version might look:

class Account 
  DepositedAmount = Data.define(:account_id, :amount)
  WithdrewAmount = Data.define(:account_id, :amount)

  def deposit(amount)
    DepositedAmount.new(account_id: self.id, amount:)
  end

  
  def withdraw(amount)
    WithdrewAmount.new(account_id: self.id, amount:)
  end

  
  def balance
    events.reduce(0) do |final_balance, event|
      BalanceProjector.call(final_balance)
    end
  end

  private

  def events
    Event.where(account_id: self.id).order(:created_at)
  end
end

And here’s how that Balance Projector might look:

BalanceProjector = proc do |balance, event|  
  case event
  when DepositedAmount
    final_balance += event.amount
  when WithdrewAmount
    final_balance -= event.amount
  end
end

As you can see, in a CRUD app, the focus is around objects and their final state, while in an Event-sourced application, events are treated as the main focus, while state is derived by playing them out in sequence.

The benefits of Event Sourcing

Storing events as the main source of truth means that by default, you have an audit log of everything that happened in your application. Whether that’s amounts being deposited, users creating something, updating a value, or changing their password, you’ll have a record of that event occurring at a specific point in time. This means you have:

  1. An audit trail: every change is recorded with a timestamp.
  2. The ability to time travel: you can go back to any point in time and replay the events that got you there. This would be incredibly handy for investigating when something goes wrong.
  3. Flexibility: you can do whatever you want with the granular data you’ve collected, creating a wide variety of projections to arrive at the results you need.

On the surface, this feels like the ideal architecture for a financial system of record, allowing us to keep track of every transaction that took place in the service and keep a clear audit trail.

The downsides of Event Sourcing

While it may feel tempting to build any new app with an Event-sourced architecture given the benefits listed above, there are a number of tradeoffs to consider.

It’s complex

Working with an Event-sourced architecture requires a completely different way of thinking, as you’re forced to think about events rather than state. Particularly for Ruby developers who are used to Rails’ convention over configuration, there’s a lot to get on board with. Building in an Event-sourced manner requires you to go against Rails’s standard way of building things. Even though there are handy libraries like the Rails Event Store, they still introduce a level of complexity far and above what’s expected in a Rails application, leading to a fair amount of head scratching for any new developers joining the team.

As I mentioned when I first encountered Event Sourcing and read up on it, there’s often a lack of distinction between its core concepts and those in Domain-Driven Design, and CQRS (Command Query Responsibility Segregation, essentially an architecture where logic for reading data – querying – is kept separate from logic for writing it – commands). This only adds to the list of new concepts to learn, as a lot of the libraries conflate these concepts in their documentation.

This complexity only feels warranted if the service really needs such a detailed audit trail, and any alternatives (I’ll get to these in a minute) won’t do. Look again at the CRUD and Event-sourced examples above. I know which I’d prefer to work with, particularly when an Event-sourced example using the Rails Event Store looks something like this:

class Account < ApplicationRecord
  def deposit(amount)
    Rails.configuration.event_store.publish(
      MoneyDeposited.new(data: {account_id: self.id, amount: amount }),
      stream_name: "account_#{self.id}"
    )
  end

  def withdraw(amount)
    Rails.configuration.event_store.publish(
      MoneyWithdrawn.new(data: {account_id: self.id, amount: amount }),
      stream_name: "account_#{self.id}"
    )
  end

  def balance
    account_balance =
      RailsEventStore::Projection
        .from_stream("account_#{self.id}")
        .init(-> { { total: 0 } })
        .when(MoneyDeposited, ->(state, event) { state[:total] += event.data[:amount] })
        .when(MoneyWithdrawn, ->(state, event) { state[:total] -= event.data[:amount] })
        .run(Rails.configuration.event_store)

    account_balance[:total]
  end
end

It’s hard to undo

Given the architectural shift required from the usual State-stored architecture, it’s hard to back out of an Event-sourced system once you’ve started. Events are your source of truth, so some complex data migrations would be required to create projections out of all your events and derive a final state. You’d also need to fundamentally rewrite your application to deal with a very different data structure.

So, what alternatives are there?

If you need an audit trail but don’t want to fully commit to the complexity required for Event Sourcing, there are a number of auditing libraries you can make use of, such as a gem like PaperTrail, which we’ve used in the past. This allows you to keep your State-stored CRUD architecture but capture any changes to your entities, albeit in far less detail. 

During our hack days, we also explored what Ismael termed the “hybrid approach”, which is a way of passing around Event-shaped entities while not committing to a fully Event-sourced architecture or complex libraries. You’d respond to these Events by creating the usual database entities, as well as storing a record of events, a bit like this:

def process_event(event)
  Order.transaction do
    case event
    when ProductAdded
      create(:order, amount: event.amount, name: event.order_name)
    end

    Event.new(:product_added)
  end
end
event = ProductAdded.new(item_params)
item = order.process_event(event)

OrderItemProcessingJob.perform_later(item)

What we decided

We ultimately decided against building a service with an Event-sourced architecture for the new project, at least for the time being. Although we were keen to leverage the auditability that storing Events would give us, we don’t currently have all the requirements for the new service. Committing to building a service with such a fundamentally different architecture at these early stages feels risky given its complexity and the difficulties in migrating away from it. If we were to commit to Event-sourcing, we’d have to dedicate a lot of time up front to understanding best practices and design patterns.

Despite this, there were some useful takeaways from spending 2 days exploring Event-sourcing:

Further reading

If you’re interested in learning more about Event Sourcing, here are a few blog posts and videos I found useful: