[CT-2864] Enhance Data Lineage Visualization with airflow-like graph for Better Organization and Grouping

[devmessias]

This can close #344, #330, #26

Describe the feature

I would like the data lineage feature in dbt to have more organization options, including grouping features similar to how tasks can be organized in Airflow. One way to achieve this in a relatively simple manner is by using d3.js and using the models schemas to pass the parameters to organize and render the data lineage.

Example

Describe alternatives you've considered

I've been facing a lot of difficulties making data lineage useful, without getting lost, due to flows with a lot of models. One alternative was to organize Airflow tasks, which execute dbt models, through parameters passed to the dbt schema. However, it would be beneficial to have this natively in dbt.

Additional context

This feature is not specific to any database. It is rather a general feature request that can improve the user interface and usability of dbt's data lineage visualization.

Who will this benefit?

I believe this would benefit any project or company that needs to deal with a reasonable number of models. As the number of models increases, the cognitive load to understand the data lineage significantly increases as well. Additionally, several issues could be automatically resolved thanks to the added features and customizations in d3, which I will list below.

#344, #330, #26

Are you interested in contributing this feature?

I am willing to work on this feature and can assist in its development, as I have a good knowledge of D3 and experience with JavaScript and data visualization.

To add more context to my request. Usually, I put this inside of a schema.yml, like the following:

dag:
   group: something
   expanded: true
   subgroup: ...

[graciegoheen]

Hello! Thanks for opening this issue.

We recently adding functionality in 1.5 for defining groups for models:

• groups docs
• add groups to your dag guide

In the docs site, you can view lineage by selection syntax - for groups this is just --select group:my_group_name. I hope this helps getting you part of the way there!

Candidly, our team isn't able to prioritize new feature development in dbt-docs, so this isn't something we'd be picking up ourselves.

[mattyb]

Just to link the PR: #363 might solve for this

[devmessias]

Hi @mattyb. Really good PR, congrats! With that, can we pass the group as parameters in the dbt configuration (schema.yaml) for each model?

[mattyb]

thanks! That PR currently supports database, type, path, and any tags. I think it's extensible enough to easily support group as long as that's output in the manifest.

[devmessias]

Nice, I'll close this issue, I think you're in the right path for that.