Using resources in dg projects

info

dg and Dagster Components are under active development. You may encounter feature gaps, and the APIs may change. To report issues or give feedback, please join the #dg-components channel in the Dagster Community Slack.

Prerequisites

Before following this guide, you will need to create a project with the create-dagster CLI.

Assets, asset checks, and sensors in Dagster frequently require resources that are instantiated elsewhere in the project.

For example, if you have a resource defined at the root of your project in src/resources.py:

src/resources.py

import dagster as dg


class AResource(dg.ConfigurableResource): ...

You can make that resource available anywhere else in your project by creating a Definitions object that returns that resource:

src/resources.py

from my_project.resources import AResource

import dagster as dg


@dg.definitions
def defs() -> dg.Definitions:
    return dg.Definitions(
        resources={"a_resource": AResource(name="foo")},
    )

You can now use the resource elsewhere in your project - for example, in an asset defined in defs/asset_one.py:

defs/asset_one.py

from my_project.resources import AResource

import dagster as dg


@dg.asset
def asset_one(a_resource: AResource): ...

Resource binding can happen at any level of the defs hierarchy. If you moved asset_one in this example to a subdirectory, you could leave the existing resources.py file at src/defs/resources.py:

src
└── resource_docs
    ├── definitions.py
    ├── defs
    │   ├── assets
    │   │   └── asset_one.py # contains def asset_one():
    │   └── resources.py # contains AResource()
    └── resources.py # contains class AResource

Scaffolding resources

To create a resource dictionary like the above, you can run the following:

dg scaffold defs dagster.resources path/to/resources.py

which will create

import dagster as dg


@dg.definitions
def resources() -> dg.Definitions:
    return dg.Definitions(resources={})

and you can fill out the resource dictionary as needed.