Configuration and secrets management

Configuration management controls the settings a program runs with, where they come from and who can change them. Secrets management controls how confidential credentials are stored, retrieved, replaced and revoked. In a trading system, a configuration change can alter how many orders a worker submits. A leaked broker credential can let someone act with that worker’s permissions. The two need separate controls.

A fictional order worker has a default limit of 100 orders per batch. Its deployment’s environment-specific file sets the limit to 80, and an authorised operator applies a runtime override of 60. The application applies defaults first, then the environment file, then approved overrides. The effective limit is 60 orders per batch because the approved override has the highest precedence. Without it, the environment file’s 80 applies.

Precedence resolves competing values after the system checks who may supply them. In this example, arbitrary job requests have no authority to override the limit. The worker validates the effective value against the permitted bounds before accepting orders. In its restricted audit record it can capture the effective limit, the winning source and the change approval. Those details help explain why two runs of the same code behaved differently, even in a reproducible environment.

The configuration contains a reference to the broker credential, such as the fictional name demo-order-worker/broker-access. The worker authenticates to the secret store with its own workload identity and asks for the credential behind that reference. The store checks the worker’s access to that specific secret. The worker uses the returned value to authenticate to the broker and keeps it out of Git, run records and logs. If a required credential cannot be retrieved, this worker stops before submitting orders. The SelfMadeQuant lesson Data Ingestion and APIs shows how a Python client rejects a missing market-data credential. Full lesson requires membership. AWS Secrets Manager best practices

The delivery mechanism also determines when changes take effect. For example, Kubernetes ConfigMaps hold non-confidential settings. A ConfigMap update leaves a running container’s environment variables at their old values until the pod restarts. A store update can likewise leave a worker using a cached credential until it refreshes. Kubernetes ConfigMaps, AWS Secrets Manager best practices

Rotation replaces a credential at the service that accepts it and updates the secret store so clients can obtain the replacement. The procedure verifies that the replacement works against the service. If the service allows both credentials to coexist, workers can switch to the replacement before the old credential is revoked. A service that invalidates the old credential immediately may reject requests from workers that still hold it. AWS Secrets Manager Lambda rotation functions

The worker needs permission to retrieve the credential. The rotation process needs a separate identity with the authority to update the stored secret and change the credential at the broker. In AWS Secrets Manager, for example, GetSecretValue retrieves the secret. Moving the current-version label is a separate operation authorised by UpdateSecretVersionStage. A rotation function’s execution role can have both actions plus permission to store a new version. Scope any operator’s rollback permission to the secrets they are authorised to restore. AWS GetSecretValue, AWS UpdateSecretVersionStage, AWS rotation execution role permissions

Removing the order-limit override restores 80 orders per batch under the same precedence rules. Rolling back a secret-store version changes what the store returns to clients that fetch the current version. If the broker has already revoked the earlier credential, moving a label in the store still leaves the worker unable to authenticate with it. Recovery then needs a credential the broker accepts, and workers need to pick it up before normal order submission resumes. AWS rollback guide