BYOC Is Not More Secure by Default

In a recent conversation with a prospective client, I told him "Just because you deploy your application in your customer's cloud account, that doesn't mean it's more secure." He was surprised. “Jon,” he said, “you are the last person I would ever expect to say this."

It's true — done right, Bring Your Own Cloud (BYOC) has all of the benefits of SaaS, and the security, peace of mind, and isolation of self-hosted or on-premise software. But what about when it's not done right? Does it actually bring customers greater benefit than their other options?

At worst, BYOC is remote-code-execution directly in a customer's account (after all, you have to deploy your app somehow). At best, BYOC is a more secure, and lower cost alternative to self-hosted or SaaS with the benefits of both.

In between those extremes lies a spectrum — from giving your customer god-mode access to your account with no controls (don't do that, seriously), to having to send your entire observability stack home so you can manage the thing. Wrongly assuming that BYOC is more secure by default allows for problems to arise such as accidentally destroying mutating customer infrastructure, breaking networking, or unleashing a Trojan horse. No one wants that.

We've spent four years talking to hundreds of software vendors and customers about these problems, so I promise there's a light at the end of the tunnel.

What Is BYOC?

Bring Your Own Cloud (BYOC) brings together the security of self-hosted/on-prem software with the ease of SaaS. Basically, as a developer or vendor, you deploy your application into a customer's cloud account and remotely manage the application. The software is updated, patched, and evolves just like a SaaS, but it's owned by the customer and integrated with their existing infrastructure.

As a customer of BYOC software, you can run applications directly alongside your existing databases, infrastructure, and networks. Since the vendor is managing the application for you, you get the isolation and integrate-ability of running in your account without many of the downfalls of using a remotely hosted SaaS application. Since the application is in your account, you can more deeply connect it to existing data, and can optimize away slow and expensive network reliance.

Done right, this deployment option is not only poised to become the default way to deploy software used by other businesses, but opens up entirely new architectures. In other words, software designed from first principles to run in customers’ cloud accounts can leverage where it's deployed to open up new integrations, design patterns, and order-of-magnitude optimizations that are simply not possible with SaaS or self-hosted.

So, about that “done right” part? What exactly do we mean? Well, most of it comes down to answering one simple question: how do I run my software in my customer's account, securely?

Let's talk about all the things that can go wrong, that you need to design around when offering BYOC.

Permissions (or Lack Thereof)

Permissions are the first thing that goes wrong with BYOC. The minute you ask an enterprise customer for the permissions you need to set up your app, they will start asking questions and pushing back. To them, the only thing worse than using your SaaS is giving someone on your side unrestricted access to their account to set up your application.

Cross-Account Does Not Work

Most clouds have some form of cross-account access, meaning that as an account manager/owner, you can share or delegate access to a cloud account you own to an external party.

Cross-account permissions are a common starting point for software vendors thinking about BYOC. As a vendor, you can define the permissions you need from a customer, and the customer can create a new sub-account to grant those permissions to your external identity. If you are a SaaS provider, you can create a remote environment to manage your stack in a customer's account on their behalf.

While this approach sounds easy, in practice it's a nonstarter for almost every customer who is requesting a BYOC deployment. The problem lies in the fact that some customers — especially those coming from SaaS — feel as if they have no control once they give you access to their account, and that it will require extensive permissions. The reality is that the day-to-day permissioning needs are actually pretty minimal, and only get elevated in special circumstances such as initial setup and triaging issues.

Furthermore, many customers who are consuming BYOC software want to connect these applications alongside existing infrastructure, accounts, and networks. In this context, even if the customer creates a sub-account to give you access, it would only get you so far. What if you are a data streaming product and need to run inside their existing production VPC? What about connecting to their SnowFlake instance?

Another issue with cross-account permissions is that the customer ultimately has no auditability or control over what you are doing with those permissions. We know cloud permissions can be complex (looking at you IAM or Kubernetes RBAC), but that complexity doesn’t necessarily translate to control over the blast radius of certain changes. Snafus like preventing a public Ingress, or accidentally resizing your customer's production database because they gave you the wrong ID are all within the realm of possibility when you have an external actor using direct permissions into your cloud account.

Modal Application Lifecycles

One lesson we've learned from handling many BYOC customer installs and developing our permissionless architecture is that software run in a customer's account goes through a series of "modes." We call this the application lifecycle, and it’s pretty much the same for most apps:

• Setup: the initial installation and configuration of the app.
• Maintenance: day-2 updates, patches, and deployments of the app during normal modes.
• Break Glass: escalated permissions for when things go wrong.
• Teardown: uninstallation and removing an application.

It’s useful to think about systems in modes. This enables you, the vendor, to think about what permissions you need and when — and give your customer more peace of mind. When you are talking to a customer, just accept that when the permissions convo comes up, sticker shock is going to happen, so be prepared to justify every permission you need in their account.

When you are setting up your application, you need a whole bunch more permissions than you'd otherwise need. During setup, sure, you need to create clusters, maybe modify an existing network, and build lots of resources. Know what you don't need to do during setup? Modify an existing database or bucket. Creating IAM roles for service accounts that run in the application are totally reasonable.

Similarly, once your app is set up, you need many fewer permissions (we hope) than setup. We call this “maintenance mode.” For example, while creating additional IAM roles in an AWS account is required for setup, it’s basically a non-starter in maintenance mode, whereas modifying a managed database might be an accepted operational task that your managed product needs to take.

Your maintenance mode permissions probably grant enough access to let you know that something has gone wrong. But then how do you fix the issue? Just as a fire extinguisher is stored behind a glass pane that you would smash open during a fire, the so-called “break glass” mode allows the customer to grant you elevated permissions in case of data loss, leakage, or anything else that needs to be mitigated. These elevated permissions might involve reading data from an S3 bucket, accessing the database temporarily, or maybe even patching a CRD that has not been updated since the application was originally set up.

Teardown permissions should only be needed when a customer is attempting to uninstall your application. These permissions are extremely destructive (duh, they are literally removing an entire system), and have the potential to not only blow up application data, but pose risk to customer infrastructure.

In the wild, we find that thinking through these modes and outlining not just the permissions you need, but when you need them, helps to create a better contract between you and your customer.

Customer Managed Permissions

Permissions have to be managed and granted by the customer directly. It's a nonstarter for the customer to grant you access to manage the permissions directly.

As a vendor, you can define the permissions that are needed for each mode, and then the customer can grant or revoke permissions as needed based on which part of the app lifecycle you are in.

Permissions in Layers

I've talked about this at length, but modern software permissioning needs to happen in layers. Giving your customer knobs to grant or revoke permissions during different modes of operation is just the starting point.

In Nuon's BYOC model, we have found that giving the customer tools to control the entire stack of permissions is critical to defining and maintaining a secure contract between customer and vendor. We deploy a runner inside each customer's cloud account which is responsible for managing the application from within.

The runner architecture enables security in layers, giving customers more control over the management of their application. The layers of control are:

• Cloud Controls: the customer can grant or revoke permissions to the runner in their account at any time.
• Network Controls: since the runner is deployed into their account, they can turn off all external management capabilities of the runner, and control it via the network. They can also turn off network access to the application.
• Cluster Controls: the customer can control what permissions in the cluster the runner has, and grant or revoke them during any mode.
• Application Controls: the runner can govern at the application level and cut off updates or patches to specific parts of a system. For instance, you can make decisions such as not allowing any infrastructure patches or upgrades during maintenance mode.

We'll talk more about how this works in our follow-up post about Nuon's change controls and how our security model helps solve many of these problems.

Safely Updating Your Application

We keep mentioning this premise of BYOC — it's in the customer’s account, but it feels like SaaS, right? Well, outside of setting up an application, the most important part of BYOC is updating that application. As the app evolves – new features are added, critical bugs are fixed, or patches are applied – how does the app continue to be updated?

The vendor is responsible for this in the BYOC model, so rather than just blindly pushing updates and hoping for the best, we want to establish controls or guardrails for updates.

Approvals

In SaaS, when a vendor pushes a new version of their application, the customer simply logs in and gets the latest version. With on-premise or self-hosted, the vendor pushes a new release and asks customers to update as soon as possible. As you can imagine, BYOC is a bit of a hybrid. The vendor should be able to automatically push some updates to customers’ environments such as new app versions, UI bug fixes, or critical patches.

As for cluster upgrades, new infrastructure, or other changes that have the potential to change or break an install — those should not be automatically applied. The customer should have some involvement in the critical updates, and no involvement when not needed.

By default, changes to the app should have to be manually approved by customers. But, if the customer is comfortable doing so, they can opt in to automatic app updates.

Modal Updates

We’ve already established that the different modes of consuming software warrant different permissions. When it comes to updates, similar principles apply — different parts of the application or infrastructure should be updated at different times.

A simple update to an application container or to the UI is low risk enough to be allowed in maintenance mode. But other changes, such as modifying infrastructure like a database or cluster, should require the additional permissions that are only granted in provisioning or break-glass mode.

Since the software in a customer environment is modal, thinking through which updates can happen during different modes, along with the permissions required, enables a more effective balance between SaaS-like and secure.

Governing with Policies

While customers understand their environment, and vendors understand their application, in BYOC they don't always understand each other. Asking a customer to understand your application's architecture well enough to determine if they can safely and reliably run it is not realistic.

This is where customer governance, and ultimately infrastructure policies play a critical role in BYOC software. As an enterprise, your customer knows the do’s and don’ts of their environment. For instance, some organizations do not allow any external traffic, limit usage of certain cloud-managed services, and enforce strict networking boundaries.

Policy tools like Kyverno, Kubewarden, and Sentinel give customers the ability to enforce these rules on foreign applications. Instead of manually scanning and auditing each version of an application, they can define policies that limit potentially problematic or non-compliant resources.

Release Windows

When servicing these enterprise customers, the premise is to deliver a SaaS-like experience, but on their terms. Some customers might be okay with more permissive updates and less strict processes on when and where their applications are updated.

Other customers will skew further from SaaS and be more risk averse. These same customers might ask for reduced permissions during normal operations, and request that all updates go through release windows.

Let's Talk About Remote Code Execution

I would be remiss if I wrote an entire blog post about the fundamental challenges of BYOC security and didn’t mention remote code execution.

We’ve unpacked the challenges and security problems of using cross-account permissions to orchestrate software in a remote account. While cross-account permissions can enable you to quickly reuse orchestration and management tools from an existing SaaS, they open customers up to what amounts to remote cloud execution, where another vendor whom you have no visibility into — or control over — can operate and update your system. Lacking visibility and controls at the network, cloud, and infrastructure layers can make consuming software like this more dangerous than the SaaS counterpart, and comes with a much higher blast radius when things go wrong.

A secure BYOC application performs remote code execution to set up, maintain, and operate your app in a customer's account. By combining guardrails to monitor at the cloud, application, network, and infrastructure levels using policies, configurable permissions, and a permission model based on modes, applications can more securely operate like a SaaS in customer accounts.

While this is the topic of another post altogether, we believe that remote code execution with guardrails is a much better approach to BYOC than granting remote access to a cloud account.

In Summary

If there's one thing we've learned talking to and helping hundreds of SaaS vendors with BYOC, it's that the security problems don't just go away with BYOC. It’s tricky to do correctly (cough, cough ... this is kind of why we exist, after all). It's not just packaging your app, and throwing it over the fence to a customer. There's real risk involved, and to do it right you need to take these concerns seriously.

BYOC, when done right, is more secure, feels like SaaS, and brings your app to the customer all while integrating with customer data and giving the customer ownership and control. Here’s to running in our customers’ cloud accounts, safely and securely!