Taking a step back

When we set out to build Nuon, we opted to build the most straightforward API we could — and our API was starting to show its cracks. At the time, we were struggling to meet the needs of our customers and ship — even the simplest features became challenging to ship, and we were unable to move fast. As we started to diagnose and understand the problems, we identified three key areas:

• Our API was designed to be as CRUD-like as possible, making modeling the state of infrastructure, deployments, and state out in the wild very hard — and often, our API was outdated.
• How we treated workflows as “side-effects” instead of core concepts in the product meant hiding a lot of important visibility from our customers/product.
• Our data model needed to match our product.

Broadly, we realized that while we had an excellent provisioning layer — we had no orchestration layer in our product. Taking a step back, we realized that continuing to invest in our provisioning layer without revisiting how our API worked from first principles would lead to diminishing returns and would continue to slow us down.

Since we already used Temporal for our provisioning layer, we started by asking, “Is Temporal our source of truth?” and began working backward.

We set out to build a new API that solved the core problems we were having and set out with a few design goals for our API:

• must more easily maintain an accurate state for our infrastructure objects by modeling them more realistically
• should enable us to build auto-remediation, orchestration, and more automation in the platform
• should be at least 2/3x quicker to ship new features

Introducing Temporal Long-Lived Workflows

Even though we had been using Temporal for almost a year when we set out to build our new API, we knew we were leaving some capabilities on the table. We were aware of how Temporal Signals and Queries could allow you to pass state around and ask the status of workflows — but we had not evaluated long-lived workflows seriously in our product.

Long-lived Temporal workflows enable you to create an actor-based system to model real-life scenarios. A handful of great talks, such as this one, explain how Uber initially used Cadence (now Temporal) to treat different drivers as actors. Long-lived workflows enable you to create a process that models real-world systems or interactions and then send + receive messages to those actors to coordinate workflows.

Temporal long-lived workflows rely on three concepts:

• Signals — messages that tell a workflow to do something (similar to a write operation)
• Queries — messages that ask a workflow for statuses, state, and data (similar to a read operation)
• Signal Channels + Selectors — primitives to route and receive signals (similar to a queue)

With Temporal long-lived workflows, you can create a queue for each object in your database and then pass messages to and from it, modeling real-life interactions.

Once you have long-lived workflows in place, you can register different signals and their “side-effects” in each workflow worker:

func (a *AwaitSignals) Listen(ctx workflow.Context) {
  log := workflow.GetLogger(ctx)
  for {
    selector := workflow.NewSelector(ctx)
    selector.AddReceive(workflow.GetSignalChannel(ctx, "Signal1"), func(c workflow.ReceiveChannel, more bool) {
      c.Receive(ctx, nil)
      a.Signal1Received = true
      log.Info("Signal1 Received")
    })

Long Lived Workflows as Event Loops

As we evaluated long-lived workflows, it became clear we could use this programming model to power a new API that treated each of our core product objects as event loops. This would allow us to run a long-lived actor for each part of our system and send messages to/from each while giving us the primitives to synchronize jobs and maintain state.

This new programming model would allow us to build a better data model and, overnight, would remove most of the work we had to do to coordinate changes on individual installs and apps manually. By treating each object as its own event loop, we could build simple logic to ensure that only a single operation per object happened at a time and could queue other jobs.

We started by creating an event loop for each of the core objects in our system:

• orgs — event loop that handled provisioning data plane
• apps — event loop that handled managing resources for the app
• installs — event loop for managing provisioning, de-provisioning, and deployments
• components — event loop for managing builds, updates, and configs

Whenever a new object gets created in our API, we create a corresponding long-lived workflow that handles signals for that object and manages its lifecycle.

func (w *Workflows) InstallEventLoop(ctx workflow.Context, req InstallEventLoopRequest) error {
  l := workflow.GetLogger(ctx)

  signalChan := workflow.GetSignalChannel(ctx, req.InstallID)
  selector := workflow.NewSelector(ctx)
  selector.AddReceive(signalChan, func(channel workflow.ReceiveChannel, _ bool) {
    var signal Signal
    channelOpen := channel.Receive(ctx, &signal)
    if !channelOpen {
      l.Info("channel was closed")
      return
    }

    var err error
    switch signal.Operation {
    case OperationProvision:
      err = w.provision(ctx, req.InstallID, req.SandboxMode)
      if err != nil {
        l.Error("unable to provision", zap.Error(err))
        return
      }
    case OperationReprovision:

With this event loop primitive in place, we unlocked an entirely different programming model. We’ve moved from a CRUD first API where most of the state in our product had to be fetched at read time, creating a poor UX and complicating our API, to a system where long-lived workflows are modeling real world objects, and acting as the source of truth.

Managing state + tracking real-world infrastructure

When building our new API, we set out to ensure that each piece of infrastructure we managed could easily be monitored and the health of any object could be easily understood. Previously, this had meant asking the “source” whenever we were querying a status — something that was prone to failure, slow, and required a lot of work.

Today, our event loops are responsible for directly managing state in our database. Our event loops are modeled alongside our API and share the same database. Since each object has a corresponding event loop, that event loop can easily monitor status changes and keep the database up to date.

type UpdateStatusRequest struct {
  InstallID    string `validate:"required"`
  Status      string `validate:"required"`
  StatusDescription string `validate:"required"`
}

func (a *Activities) UpdateStatus(ctx context.Context, req UpdateStatusRequest) error {
  install := app.Install{
    ID: req.InstallID,
  }
  res := a.db.WithContext(ctx).Model(&install).Updates(app.Install{
    Status:       req.Status,
    StatusDescription: req.StatusDescription,
  })
  if res.Error != nil {
    return fmt.Errorf("unable to update install: %w", res.Error)
  }

After the initial “write” to the database, the event loop takes over and is the primary write path to update its object. This allows the natural cadence of retries, background jobs, and long-lived workflows that make up these event loops to emit and update state as they go.

Furthermore, as these event loops monitor and manage the status of different objects in the wild, they can also notify other parts of the stack using signals when failures, remediations, or additional provisioning happens.

Dependency management

One of our biggest challenges with our previous API was dependency management. Since our product comprises several objects that depend on one another but also take time to provision or de-provision, it is essential to ensure that each parent/child object waits for its dependencies when created or destroyed.

With our event-loop driven API, we now use workflow steps to ensure that we do not inadvertently delete or create objects before their parent or child objects have been created/destroyed. Using workflows, we can do this with a simple, imperative programming model:

func (w *Workflows) pollDependencies(ctx workflow.Context, installID string) error {
  for {
    var install app.Install
    if err := w.defaultExecGetActivity(ctx, w.acts.Get, activities.GetRequest{
      InstallID: installID,
    }, &install); err != nil {
      w.updateStatus(ctx, installID, StatusError, "unable to get install from database")
      return fmt.Errorf("unable to get install and poll: %w", err)
    }

    if install.App.Status == "active" {
      return nil
    }
    if install.App.Status == "error" {
      w.updateStatus(ctx, installID, StatusError, "app is in failure state, waiting for remediation")
    }

    workflow.Sleep(ctx, defaultPollTimeout)
  }
}

While this was possible to work around manually in our old API, it became nearly impossible to manage when things failed. This led to many situations where we could “leak” infrastructure,” allowing it to be deleted from the database without being properly cleaned up or destroyed. With these primitives in place, we can now poll for statuses of child or parent objects or use queries to ask for statuses.****

Coordination + queueing

We quickly learned that one of the most complex challenges we would face building Nuon would be managing the lifecycle of each install independently. Users should be able to push to Nuon without manually addressing each customer update requirement.

To correctly model this type of update lifecycle across thousands or millions of installs, we need to be able to coordinate + queue on each install. While most customers want the latest version of your product, others only want updates during specific days and times, and some enterprise customers even require that all updates be manually opted into and reviewed.

Each of our long-lived event loop workflows declares its own signal channel and processes signals synchronously while queueing the rest.

Channels and signals allow us to automatically queue jobs on each install and use different rules based on the install to decide when to apply those deploys.

Sandbox Mode

Using long-lived workflows, we created a “sandbox mode” for Nuon, enabling our users to use the entire product without actually modifying or building infrastructure. Sandbox mode allows a faster feedback loop and enables someone to setup their app with Nuon much faster, without having to worry about provisioning, destroying and managing real infrastructure.

Each long-lived workflow in our system is started with a sandboxMode flag, which is used to control whether the downstream provisioning layer is called:

This allows us to easily mimic provisioning, by simply sleeping when sandbox-mode is invoked.

func (w *Workflows) execCreatePlanWorkflow(
  ctx workflow.Context,
  sandboxMode bool,
  workflowID string,
  req *planv1.CreatePlanRequest,
) (*planv1.CreatePlanResponse, error) {
  l := workflow.GetLogger(ctx)

  if sandboxMode {
    l.Info("sandbox-mode enabled, sleeping for to mimic executing plan", zap.String("duration", w.cfg.SandboxSleep.String()))
    workflow.Sleep(ctx, w.cfg.SandboxSleep)
    return generics.GetFakeObj[*planv1.CreatePlanResponse](), nil
  }
...

When sandbox mode is disabled, the downstream workflows are invoked as child workflows. With sandbox mode disabled, the workflow executes the actual provisioning step:

cwo := workflow.ChildWorkflowOptions{
    TaskQueue:                workflows.ExecutorsTaskQueue,
    WorkflowID:               workflowID,
    WorkflowExecutionTimeout: time.Minute * 60,
    WorkflowTaskTimeout:      time.Minute * 30,
    WorkflowIDReusePolicy:    enumsv1.WORKFLOW_ID_REUSE_POLICY_TERMINATE_IF_RUNNING,
    WaitForCancellation:      true,
  }
  ctx = workflow.WithChildOptions(ctx, cwo)

  var resp planv1.CreatePlanResponse
  fut := workflow.ExecuteChildWorkflow(ctx, "CreatePlan", req)

Operational tooling

Since each object in our system is modeled via a long-lived workflow, we add new operations to create “side effects” to update, create, and destroy infrastructure using signals. Part of our API design depends on developing these signals and implementing handlers that act upon them.

case OperationPollDependencies:
      err = w.pollDependencies(ctx, req.InstallID)
      if err != nil {
        l.Error("unable to poll dependencies", zap.Error(err))
        return
      }
    case OperationProvision:
      err = w.provision(ctx, req.InstallID, req.SandboxMode)
      if err != nil {
        l.Error("unable to provision", zap.Error(err))
        return
      }
    case OperationReprovision:
      err = w.reprovision(ctx, req.InstallID, req.SandboxMode)
      if err != nil {
        l.Error("unable to reprovision", zap.Error(err))
        return
      }
    case OperationDelete:

Previously, we had to model these operations using scripts or CLIs. Now, we can expose them directly in our internal API and occasionally in our public API for customers.****