Customer configuration

The first challenge that companies building for BYOC need to solve is creating a flexible installation strategy — this requires taking your existing infrastructure, application, and automation code and re-configuring it to work in multiple deployment environments, as well as to be portable and work from first install. Not only are multiple deployment options required, but additional customer configuration inputs and controls are required, adding additional complexity.

IAC is for two types of infrastructure — the “base layer” and “application layer.” Base layer IAC is used to provision networks and compute clusters, permissions, and tertiary resources such as log tooling, IAM, and security groups. The second group is the “application layer” IAC, which is generally managed by application developers and used to provision specific resources for services such as databases, cloud storage buckets, etc.

The base layer infrastructure for BYOC deployments requires reconfigurability for customers that have the following requirements:

• want a turnkey solution (deploy everything into a new account)
• want to pick between platforms (ECS vs EKS) or clouds (AWS vs Azure)
• want a turnkey solution but want to customize things like network, compute type, or version
• want to use an existing network or compute clusters

Supporting these permutations requires you to configure your base layer infrastructure to plug and plug different resources, and supporting multiple platforms often requires building IAC that deploys your app in various ways. Customer-provided infrastructure introduces its own challenges since customer configurations must be compatible with your product.

Since most application layer infrastructure depends on base layer infrastructure, this often requires sharing managed resources such as the network with higher level IAC. For instance, to deploy a database, the IAC must know what network to deploy it into. This also means an implicit ordering is required between these components in the provisioning lifecycle.

Supporting these different permutations quickly adds complexity to your IAC footprint, requiring you to reason about an entirely new matrix of configuration options — maintaining, evolving, and testing this becomes a significant investment. Often, this can be at odds with multi-tenant tooling and requires maintaining two bespoke paths to support BYOC customers.

Long tail automation and installers

Once your team has created IAC that is customer-cloud compatible, the next step is automating the various steps of the installation into a single workflow — often called an installer. This allows customers to quickly install your product, even if they are not technical or part of an IT/SRE team.

Often, the first few BYOC installations are a long, drawn-out manual process that goes something like this:

• First, you get on a call with your customer and share access to your configuration repositories.
• The customer starts running IAC and deployment scripts.
• Often, roadblocks are hit — whether another team needs to procure something or the system is incompatible with the account.

This process is not only manual and error-prone but requires a technical user. Since it is not fully automated, the installs can not be programmatically tested, leading to more customer issues.

An automated installer will enable automated testing, a self-serve offering, and faster install times for customers or POCs. Creating a fully automated install experience requires the team to orchestrate their IAC and deployment tooling and create turnkey installer options that package the infrastructure and application code in one.

Artifact management

Beyond infrastructure management, artifact management requires automation. While an installer provisions the installation, most teams will want more control over deployable artifacts — specifically container artifacts. These artifacts change much more frequently than IAC and often need to be updated several times a day.

Most teams use CI to build images, and as their product grows in complexity, they prefer to develop images once and publish them to a private registry that different systems can access. In a multi-tenant environment, granting access to a handful of accounts requiring access to a container image registry is trivial.

However, once your application is being run in foreign customer accounts, you have to solve for the following:

• custom permissions for external accounts
• cleaning up old images and updating to newer versions
• enabling image scanning for enterprise customers
• syncing images into customer accounts to prevent being a SPOF (single point of failure) and avoiding egress costs

While granting third-party access for customer installs to pull and run images directly is a starting point, it is broadly a non-starter once you pass a handful of installs or start deploying into more sensitive customers. It also requires you to be very thoughtful about tagging images because simply tagging an image “latest” can cause inadvertent updates and outages.

Furthermore, with direct customer access to installs, any outage in your image registry can become an outage for all your customers.

Syncing container images into customer accounts before they are used is the most secure and safest way to manage container images. By first copying image artifacts to a customer registry, the customer can enable container image scanning and ensure their installation only relies on local images. Nodes restarting and re-fetching images no longer depend on a third-party vendor-managed registry. Images can also be cleaned up safely whenever a new update is released in the customer account, ensuring no old container images are leaked.

Application change management

To continually update your application in customer environments, teams invest in application rollouts that allow them to update each customer. While updating a hosted SaaS or self-hosted application is as easy as publishing a new container artifact or deploying to one or more environments — updating a BYOC application comes with additional concerns about how customers get updated.

Since each BYOC install is an entirely different environment, deployments must be designed to roll out to each customer individually. This is challenging because application errors might not be immediately apparent, unlike a hosted application with many tenants in the same environment — inherently, each install will have different usage, making pattern matching harder. Properly rolling out changes requires a combination of testing internally and then rolling out and “baking” (letting an update run for some time before continuing the rollout) before updating more customers.

Furthermore, while some customers will want the latest changes at all times, other customers might want to enforce release windows or manually opt into new changes/versions of the software. Many enterprise customers will require contractual agreements controlling how and when the application is updated.

Finally, since deploying application changes usually requires updating running containers, this can also require syncing new container versions, re-scanning container images, and removing unused artifacts.

Migrations

Two common types of migrations are database migrations and infrastructure migrations. Teams building for BYOC need to account for both, as their products will inevitably change after being installed in customer accounts. New services or dependencies will be added, schemas will change, and data will inevitably need to be modified.

Migrations tend to be the riskiest changes to roll out because they often involve updating actual infrastructure (such as clusters, networks, etc) or modifying data and can lead to outages, data breaches, or leaks. Unsurprisingly, BYOC customers are hyper-sensitive to these updates and generally want visibility into what is changing.

Teams building for BYOC often end up building automated tooling that allows them to:

• Automate infrastructure migrations and schedule them for rollout based on customer requirements and release windows.
• Create bespoke migrations to migrate specific vendor data when needed.

Finally, for customers that demand specific update windows, migrations can add additional complexity. Given the risk of infrastructure or data migrations, these customers may ask to audit changes ahead of rollout or limit them to a lower cadence than application updates.

Observability

To ensure quality of service for BYOC customers, observability must be implemented to work across many environments and enable “drilling down” into a single install. Whereas most regressions or problems in a hosted environment can be identified quickly by looking at aggregate metrics and alerts — BYOC adds a combination of release windows, different versions, and different environments.

Further complicating things — while most customers will allow primary telemetry data to leave their installs (think OTEL metrics or product analytics), logs are considered highly sensitive, and most customers will ask that vendors only access them on demand (with proper auditing in place). Sometimes, viewing application logs requires reaching out to customers and asking them to capture and share logs while debugging.

Setting up observability tooling for BYOC offerings requires:

• building an automated way to capture logs with customer approval or allow a customer to export and share logs
• sending product metrics and analytics back to the vendor’s observability tooling
• reworking monitoring/alerting to monitor both aggregate (across install) events, as well as per install events

Finally, it goes without saying that due to the sensitive nature of BYOC customers, teams must proactively ensure that no sensitive data is leaked via their observability stack.

When things go wrong

As with all software — things are going to go wrong. From introducing bugs in new releases, to failed migrations, to security patches that need to be applied quickly, teams building for BYOC are at both an advantage and a disadvantage operationally.

On one hand, in a hosted environment where the team owns the underlying cloud account — they always have access to investigate and remediate when things go wrong. However, in a hosted SaaS, the blast radius is implied to be much larger and can affect more customers. BYOC has the inverse problem; teams must plan for mitigations without direct access since each install is independent, so the blast radius should be smaller.

When building for BYOC, teams must invest in a rollback and escalation strategy early on and set themselves up with the right tools to run operational commands when necessary. Release windows and procedural rollouts help but are not the only solutions required.

Self-serve

An automated installer experience can enable self-serve once the application can be fully provisioned with minimal customer manual intervention. Companies often start by building a self-serve option into their dashboard or leveraging IAC to provision their app (such as creating a custom Terraform provider). Creating these options allows more customers to install and integrate the product.

To offer a self-serve offering, customers often provide cross-account access (usually using IAM), granting access for the initial install. Then, an onboarding experience that leverages the installer will walk the customer through setup.

Often, this flow will require the following steps:

• configure any certificates or secrets
• setup DNS
• connect the product with an existing VPN for access control

Self-serve requires a high level of automation, but once in place, it allows teams to offer their BYOC version to all customers.

Management tooling for customers

As part of self-serve BYOC offerings, management tooling is required to allow customers to manage and maintain their installation without manual intervention.

Teams often build in automation and configuration knobs to allow customers to manage their installs; these include:

• ability to autoscale/manage instance types as the product usage grows
• the ability to manage changes — either audit automatic updates or approve manual updates
• monitoring to ensure the application is healthy

A fully manageable experience for BYOC allows a company to scale its offering and build a better product experience that meets the needs of its earlier and enterprise customers. All while running the app in the customer’s account without significant investment by the customer to manage it.