Introducing the Hightouch Dagster Integration

Technically, this works–but it has a lot of lags and dependencies. It hinges on the assumption that everything upstream is completed successfully and on time. This could be so much more efficient.

This example is quite possibly the simplest workflow. Imagine doing this with a complex nested DAG with dozens of steps and upstream dependencies. It gets unwieldy very fast.

As Nick Schrock at Elementl put it:

It’s like having a production line with three different workstations handing off components to each other. But each workstation is siloed and can’t communicate with the next. As each station packages and pushes components to the next, they have no clue if what they’re sending is updated or correct. The next station gets what it gets, or sometimes gets nothing at all. If the production line stops suddenly, it takes a while to figure out where the problem lies.

The Hightouch Dagster integration alleviates this problem by orchestrating the flow of data from Fivetran into your warehouse, refreshing the models via dbt Cloud, and syncing your data to your CRM using Hightouch.

What’s more, with the Dagster interface, you gain deep visibility into each step of the process: from the time it takes to refresh your models, to the number of rows that were successfully sent to your CRM. Equipped with those insights, you can prioritize and tackle engineering work to continuously improve your data pipelines.

Dagster Processes

This approach has many benefits for both engineering and business teams:

• Data freshness: The total time it takes to move customer data through the end-to-end pipeline shrinks from hours to as little as 15 minutes.
• Workflow automation: By centrally managing robust orchestration pipelines, your data teams can focus on more proactive work
• Operational efficiency: Your teams avoid using compute credits on redundant transformations or calculations.

As the modern data stack rapidly evolves, it is essential that leading data tools integrate seamlessly. With the great work done by both our teams, Dagster can now make Hightouch operate seamlessly and robustly, ensuring that the right data goes into the right app at exactly the right time. - Nick Schrock, Founder and CEO, Elementl

Setting up the Hightouch Integration

Configuring the Hightouch integration in Dagster is a streamlined process:

• Install the Hightouch Dagster Library
• Configure the Library to use your Hightouch API Key
• Write the code that defines your syncs and their dependencies

As you can see from the code sample below, there’s very little code required.

Code Dependencies

And that’s it! On top of integrating natively with all of the other orchestration steps in your modern stack, you can:

• See and schedule all of your Hightouch syncs
• View sync execution trends over time
• Get a birds-eye-view of execution status and recent runs.

You can run this pipeline as often as you want to ensure your data is constantly refreshed. You can also quickly view all of the destinations that Hightouch writes to as Dagster Assets, including the objects we write to. In addition, you can see a snapshot of your latest run (how long it took, rows added/changed/removed), and track these trends over time.

Dagster Assets

Next steps

The Hightouch Dagster integration is available now! If you’re not a Hightouch customer yet, you can sign up for a free account.

If you’re new to Dagster, it’s available as an open-source solution today and Dagster Cloud will be GA in August. Contact the Dagster team for details, or check out the growing Dagster community.