Customizing Dagster+ agent settings in dagster.yaml
This guide is applicable to Dagster+.
The Dagster+ Agent is a special variant of the Dagster instance used in Dagster Open Source and is configured through the same dagster.yaml
file. You can customize your agent with these settings.
For Kubernetes agents deployed with the Dagster+ Helm chart, you'll need to refer to the Helm chart's config map for customizing the agent.
Enabling user code server TTL
User code servers support a configurable time-to-live (TTL). The agent will spin down any user code servers that haven't served requests recently and will spin them back up the next time they're needed. Configuring TTL can save compute cost because user code servers will spend less time sitting idle.
TTL is disabled by default for full deployments, and can be configured separately for full and branch deployments. TTL defaults to 24 hours for both full and branch deployments.
To configure TTL:
# dagster.yaml
instance_class:
module: dagster_cloud.instance
class: DagsterCloudAgentInstance
dagster_cloud_api:
agent_token:
env: DAGSTER_CLOUD_AGENT_TOKEN
deployment: prod
user_code_launcher:
module: dagster_cloud.workspace.docker
class: DockerUserCodeLauncher
config:
server_ttl:
full_deployments:
enabled: true # Disabled by default for full deployments
ttl_seconds: 7200 # 2 hours
branch_deployments:
ttl_seconds: 3600 # 1 hour
Streaming compute logs
You can set up streaming compute logs by configuring the log upload interval (in seconds).
# dagster.yaml
instance_class:
module: dagster_cloud.instance
class: DagsterCloudAgentInstance
dagster_cloud_api:
agent_token:
env: DAGSTER_CLOUD_AGENT_TOKEN
deployment: prod
user_code_launcher:
module: dagster_cloud.workspace.docker
class: DockerUserCodeLauncher
compute_logs:
module: dagster_cloud
class: CloudComputeLogManager
config:
upload_interval: 60