dlt (dagster-dlt)

This library provides a Dagster integration with dlt.

For more information on getting started, see the Dagster & dlt documentation.

Component

class dagster_dlt.DltLoadCollectionComponent [source]: Expose one or more dlt loads to Dagster as assets.

To use the dlt component, see the dlt component integration guide.

YAML configuration

When you scaffold a dlt component definition, the following defs.yaml configuration file will be created:

type: dagster_dlt.DltLoadCollectionComponent

attributes:
  loads:
    - source: .loads.my_load_source
      pipeline: .loads.my_load_pipeline

Assets

@dagster_dlt.dlt_assets [source]

Asset Factory for using data load tool (dlt).

Parameters:

dlt_source (DltSource) – The DltSource to be ingested.
dlt_pipeline (Pipeline) – The dlt Pipeline defining the destination parameters.
name (Optional[str], optional) – The name of the op.
group_name (Optional[str], optional) – The name of the asset group.
dagster_dlt_translator (DagsterDltTranslator, optional) – Customization object for defining asset parameters from dlt resources.
partitions_def (Optional[PartitionsDefinition]) – Optional partitions definition.
backfill_policy (Optional[BackfillPolicy]) – If a partitions_def is defined, this determines how to execute backfills that target multiple partitions. If a time window partition definition is used, this parameter defaults to a single-run policy.
op_tags (Optional[Mapping[str, Any]]) – The tags for the underlying op.
pool (Optional[str]) – A string that identifies the concurrency pool that governs the dlt assets’ execution.

Examples:

Loading Hubspot data to Snowflake with an auto materialize policy using the dlt verified source:

from dagster_dlt import DagsterDltResource, DagsterDltTranslator, dlt_assets


class HubspotDagsterDltTranslator(DagsterDltTranslator):
    @public
    def get_auto_materialize_policy(self, resource: DltResource) -> Optional[AutoMaterializePolicy]:
        return AutoMaterializePolicy.eager().with_rules(
            AutoMaterializeRule.materialize_on_cron("0 0 * * *")
        )


@dlt_assets(
    dlt_source=hubspot(include_history=True),
    dlt_pipeline=pipeline(
        pipeline_name="hubspot",
        dataset_name="hubspot",
        destination="snowflake",
        progress="log",
    ),
    name="hubspot",
    group_name="hubspot",
    dagster_dlt_translator=HubspotDagsterDltTranslator(),
)
def hubspot_assets(context: AssetExecutionContext, dlt: DagsterDltResource):
    yield from dlt.run(context=context)

Loading Github issues to snowflake:

from dagster_dlt import DagsterDltResource, dlt_assets


@dlt_assets(
    dlt_source=github_reactions(
        "dagster-io", "dagster", items_per_page=100, max_items=250
    ),
    dlt_pipeline=pipeline(
        pipeline_name="github_issues",
        dataset_name="github",
        destination="snowflake",
        progress="log",
    ),
    name="github",
    group_name="github",
)
def github_reactions_dagster_assets(context: AssetExecutionContext, dlt: DagsterDltResource):
    yield from dlt.run(context=context)

dagster_dlt.build_dlt_asset_specs [source]

Build a list of asset specs from a dlt source and pipeline.

Parameters:

dlt_source (DltSource) – dlt source object
dlt_pipeline (Pipeline) – dlt pipeline object
dagster_dlt_translator (Optional[DagsterDltTranslator]) – Allows customizing how to map dlt project to asset keys and asset metadata.

Returns: List[AssetSpec] list of asset specs from dlt source and pipeline

class dagster_dlt.DagsterDltTranslator [source]

get_asset_key [source]

superseded

This API has been superseded. Use DagsterDltTranslator.get_asset_spec(...).key instead..

Defines asset key for a given dlt resource key and dataset name.

This method can be overridden to provide custom asset key for a dlt resource.

Parameters: resource (DltResource) – dlt resourceReturns: AssetKey of Dagster asset derived from dlt resource

get_auto_materialize_policy [source]

superseded

This API has been superseded. Use DagsterDltTranslator.get_asset_spec(...).auto_materialize_policy instead..

Defines resource specific auto materialize policy.

This method can be overridden to provide custom auto materialize policy for a dlt resource.

Parameters: resource (DltResource) – dlt resourceReturns: The auto-materialize policy for a resourceReturn type: Optional[AutoMaterializePolicy]

get_automation_condition [source]

superseded

This API has been superseded. Use DagsterDltTranslator.get_asset_spec(...).automation_condition instead..

Defines resource specific automation condition.

This method can be overridden to provide custom automation condition for a dlt resource.

Parameters: resource (DltResource) – dlt resourceReturns: The automation condition for a resourceReturn type: Optional[AutomationCondition]

get_deps_asset_keys [source]

superseded

This API has been superseded. Iterate over DagsterDltTranslator.get_asset_spec(...).deps to access AssetDep.asset_key instead..

Defines upstream asset dependencies given a dlt resource.

Defaults to a concatenation of resource.source_name and resource.name.

Parameters: resource (DltResource) – dlt resourceReturns: The Dagster asset keys upstream of dlt_resource_key.Return type: Iterable[AssetKey]

get_description [source]

superseded

This API has been superseded. Use DagsterDltTranslator.get_asset_spec(...).description instead..

A method that takes in a dlt resource returns the Dagster description of the resource.

This method can be overridden to provide a custom description for a dlt resource.

Parameters: resource (DltResource) – dlt resourceReturns: The Dagster description for the dlt resource.Return type: Optional[str]

get_group_name [source]

superseded

This API has been superseded. Use DagsterDltTranslator.get_asset_spec(...).group_name instead..

A method that takes in a dlt resource and returns the Dagster group name of the resource.

This method can be overridden to provide a custom group name for a dlt resource.

Parameters: resource (DltResource) – dlt resourceReturns: A Dagster group name for the dlt resource.Return type: Optional[str]

get_kinds [source]

superseded

This API has been superseded. Use DagsterDltTranslator.get_asset_spec(...).kinds instead..

A method that takes in a dlt resource and returns the kinds which should be attached. Defaults to the destination type and “dlt”.

This method can be overridden to provide custom kinds for a dlt resource.

Parameters:

resource (DltResource) – dlt resource
destination (Destination) – dlt destination

Returns: The kinds of the asset.Return type: Set[str]

get_metadata [source]

superseded

This API has been superseded. Use DagsterDltTranslator.get_asset_spec(...).metadata instead..

Defines resource specific metadata.

Parameters: resource (DltResource) – dlt resourceReturns: The custom metadata entries for this resource.Return type: Mapping[str, Any]

get_owners [source]

superseded

This API has been superseded. Use DagsterDltTranslator.get_asset_spec(...).owners instead..

A method that takes in a dlt resource and returns the Dagster owners of the resource.

This method can be overridden to provide custom owners for a dlt resource.

Parameters: resource (DltResource) – dlt resourceReturns: A sequence of Dagster owners for the dlt resource.Return type: Optional[Sequence[str]]

get_tags [source]

superseded

This API has been superseded. Use DagsterDltTranslator.get_asset_spec(...).tags instead..

A method that takes in a dlt resource and returns the Dagster tags of the structure.

This method can be overridden to provide custom tags for a dlt resource.

Parameters: resource (DltResource) – dlt resourceReturns: A dictionary representing the Dagster tags for the dlt resource.

Return type: Optional[Mapping[str, str]]

Resources

class dagster_dlt.DagsterDltResource [source]

run [source]

Runs the dlt pipeline with subset support.

Parameters:

context (Union[OpExecutionContext, AssetExecutionContext]) – Asset or op execution context
dlt_source (Optional[DltSource]) – optional dlt source if resource is used from an @op
dlt_pipeline (Optional[Pipeline]) – optional dlt pipeline if resource is used from an @op
dagster_dlt_translator (Optional[DagsterDltTranslator]) – optional dlt translator if resource is used from an @op
**kwargs (dict[str, Any]) – Keyword args passed to pipeline run method

Returns: An iterator of MaterializeResult or AssetMaterializationReturn type: DltEventIterator[DltEventType]

Component​

YAML configuration​

Assets​

Resources​

Component

YAML configuration

Assets

Resources