Crossplane & Composition: Taming Secrets at Scale

In one of our client engagements, the development team found themselves in a bind. Running Kubernetes on AWS, they had to juggle dozens of apps — -each needing its own set of secrets, each demanding fresh databases on demand, and all under the watchful eyes of policy restrictions.

Keeping secrets in Git was a non-starter: a recipe for disaster.

Meanwhile, spinning up new databases meant repeatedly looping through AWS consoles or APIs, creating credentials, and somehow piping them back to the cluster. Editing secrets was another battle — -devs lacked direct cluster access, and ops didn’t want to hand out broad AWS permissions.

Enter Crossplane

Crossplane, in essence, is Lego for Kubernetes. Install it once, bring in the providers you need, and suddenly your cluster speaks the language of AWS, GCP, Azure — -whatever cloud you work with. Each provider extends Kubernetes with new CRDs (custom resource definitions). Want an RDS instance? Declare it. Need an S3 bucket? Apply a manifest.

But raw building blocks are not enough. You don’t want every developer hand-crafting YAML for RDS and IAM policies. That’s where Compositions come in.

Composition: Blueprints for Lego

A composition is a blueprint that tells Crossplane what “one thing” really means. You define it once — -either as YAML or in Golang for more flexibility — -and every time a dev asks for that “thing,” Crossplane reliably assembles the right resources under the hood.

It’s the difference between asking for a car versus assembling all the bolts, axles, and gears yourself.

Our Solution

Armed with these tools, we built two key compositions for our client:

1. Secrets Management
2. Database Provisioning

Secrets Made Simple

The “secrets” composition allowed developers to declare a single YAML file:

apiVersion: secrets.example/v1
kind: SecretBundle
metadata:
  name: payment-service-secrets
spec:
  tag: team-payments```

That’s it.

Crossplane took it from there:

- Provisioned an AWS Secrets Manager vault.
- Linked it to the cluster via External Secrets Operator.
- Enforced access policies tied to the developer’s custom tag.

Now, developers with limited AWS permissions (the permissions were tied, restricted to the tags of their team, and the ones created in AWS) could create, update, or delete secrets safely — -without ever touching the cluster directly. The secrets flowed automatically into Kubernetes, ready to be mounted in deployments and pods.

### Databases On Demand

The database composition followed a similar path: developers requested a `DatabaseBundle` resource, and Crossplane spun up RDS instances with associated secrets, wired directly back into the cluster. No waiting for ops, no hunting through AWS consoles.

### Why This Worked

This approach solved three pain points at once:

- **Security**: Secrets stayed out of Git.
- **Self-service**: Developers had the power to provision without full AWS access.
- **Scalability**: Adding more apps was as simple as applying YAML.

And because compositions are just blueprints, the same logic can generalize to other providers and even other clouds. Want to try it on GCP? Swap the provider. On Azure? Same story.

### In case you wonder…

- **Scaling & Namespaces**: Be deliberate with naming. Within a namespace, you need conventions to avoid collisions.
- **Crossplace/Composition**: There is a small learning curve. Once mastered, it’s surprisingly elegant.
- **AWS Policies**: Spend the time upfront — -fine-grained IAM is critical to keep the model safe.
- **Secret Rotation**: Currently manual, but ripe for extension using AWS’s native rotation features.
- **Language Choice**: YAML is fine to start, Golang unlocks more customisation.
- **Multi-cloud**: This pattern is portable. The trick is picking the right providers.

### Wrapping Up

Is this the *only* way to solve secrets and database provisioning at scale? No. But it’s one of the **coolest** ways we’ve found — -balancing security, developer autonomy, and operational sanity.

If tomorrow you wake up with a similar headache in your dev department, you’ll want a team that can wield Crossplane and Compositions with confidence.