How should configuration differ across development, staging, and production?

An ML application often runs in three environments: development, staging, and production. The application code should remain as consistent as possible, but some settings must differ because each environment has a different job.

• Development (dev): A safe place to build features, explore ideas, and troubleshoot.
• Staging: A realistic rehearsal environment for checking a proposed release.
• Production (prod): The live system serving real users or business workloads.

The goal is controlled differences, not three independently maintained applications.

A practical example

Imagine a food-delivery service whose model predicts delivery times. The team needs to improve how the application handles restaurant preparation-time data.

In development, an engineer runs the prediction service against a small, synthetic dataset. Detailed debug logs are useful, and the service can use a low-cost local or test endpoint. It must not accidentally modify live order data.

In staging, the team deploys the *same candidate application and model artifacts* that it intends to release. The service uses an isolated test endpoint, representative appropriately sanitized or authorized data, and dependencies or capacity suitable for meaningful compatibility and performance tests. Staging should resemble production in the behavior being tested, even if its traffic volume or cost is lower.

In production, the approved release uses live endpoints, stricter access controls, appropriate logging and alerts, and capacity sized for real demand. Changes go through the team's release and rollback process.

What should differ between environments?

1. Data sources and destinations: Dev can use synthetic samples; staging uses isolated representative test data; production connects only to approved live data sources. Avoid letting tests write to production systems.
2. Service endpoints: Use separate URLs, databases, queues, storage locations, and service identities where isolation is needed. For example, staging must not send an experimental delivery estimate to a real customer.
3. Resource limits and cost: Dev may use smaller instances. Staging must have enough fidelity to test meaningful performance assumptions. Production capacity reflects availability and workload needs.
4. Logging and diagnostics: Dev may allow verbose debugging. Production needs structured monitoring and privacy-safe logs; staging should check that those production monitoring paths work. Avoid logging personal data or secrets in any environment.
5. Access and change controls: Developers may have broad access to their own sandbox, but production permissions should be limited and audited. Production configuration changes usually need stricter approval.

What should stay consistent?

Keep these as consistent as practicable:

• Application code, dependency versions, and container image: Promote a tested immutable artifact rather than rebuilding different code for each environment.
• Feature definitions and model input/output schemas: Different feature calculations in staging and production can silently change predictions.
• Critical validation rules: Schema checks, input handling, and other correctness rules should not disappear in production.
• Configuration structure: Use the same names, expected types, and validation across environments, even when the values differ.

There can be legitimate environment differences, but document them and test anything that might change model behavior.

How can you implement this simply?

A common approach is to keep shared, non-secret defaults in version control and override only environment-specific settings. For example:

YAML
# config/base.yaml
prediction_timeout_seconds: 2
model_input_schema: delivery-v2

# config/dev.yaml
service_url: "http://localhost:8000"
log_level: "DEBUG"

# config/staging.yaml
service_url: "https://staging.example.internal"
log_level: "INFO"

# config/prod.yaml
service_url: "https://predict.example.internal"
log_level: "INFO"

These files illustrate the idea; your application must explicitly load the base configuration and the selected environment file, and validate the result. The URLs are examples, not deployed services. Inject credentials using a secrets manager or another approved secure mechanism rather than committing them to YAML.

The release pipeline should record the resolved configuration version alongside the application image, model version, and deployment environment. That makes it easier to answer, “What exactly was running when this prediction changed?”

What is the most common mistake?

Treating staging as an unrelated test system. If its software, feature definitions, or dependencies differ too much from production, passing staging tests may create false confidence. Aim for production-like behavior with isolated resources, not necessarily an identical bill for compute.

In short: Make dev easy to experiment in, staging realistic and isolated, and production controlled and observable—while promoting the same tested code and model artifacts through each environment.