By default, Dagster comes with a built-in logger that tracks all execution events. Built-in loggers are defined internally using the LoggerDefinition class. The @logger decorator exposes a simpler API for the common logging use case, which is typically what should be used to define your own loggers.
The decorated function should take a single argument, the init_context available during logger initialization, and return a logging.Logger. Refer to the Customizing loggers guide for an example.
The context object passed to every op execution includes the built-in log manager, context.log. It exposes the usual debug, info, warning, error, and critical methods you would expect anywhere else in Python.
Structured logs are enriched and categorized with metadata. For example, a label of which asset a log is about, links to an asset’s metadata, and what type of event it is available. This structuring also enables easier filtering and searching in the logs.
Logs stream back to the UI in real time:
Filtering log messages based on execution steps and log levels:
Errors in user code are caught by Dagster machinery to ensure jobs gracefully halt or continue to execute, but messages including the original stack trace get logged both to the console and back to the UI.
For example, if an error is introduced into an op's logic:
# demo_logger_error.pyfrom dagster import job, op, OpExecutionContext
@opdefhello_logs_error(context: OpExecutionContext):raise Exception("Somebody set up us the bomb")@jobdefdemo_job_error():
Messages at level ERROR or above are highlighted both in the UI and in the console logs, so they can be easily identified even without filtering:
In many cases, especially for local development, this log viewer, coupled with op reexecution, is sufficient to enable a fast debug cycle for job implementation.
Logging is environment-specific: for example, you don't want messages generated by data scientists' local development loops to be aggregated with production messages. On the other hand, you may find that console logging is irrelevant or even counterproductive in production.
Dagster recognizes this by attaching loggers to jobs so that you can seamlessly switch from environment to environment without changing any code. For example, let's say you want to switch from Cloudwatch logging in production to console logging in development and test: