Best Practices for Storing Superset Dashboards in Git

[ampil]

Hi all,

I'm exploring the best practices for storing Superset dashboards in Git for version control purposes.

I haven’t come across many articles or discussions on this topic, so I wanted to reach out to the community for guidance.

Currently, my team and I follow this approach:

• Export newly developed dashboards as a zip archive.
• We have a 'Superset' folder in our Git repo with subfolders for charts, dashboards, databases, and datasets.
• We then place the contents of the zip (YAML files) into the corresponding directories and remove the ID numbers from the filenames. (for example dashboard Legal_Information_40.yaml becomes Legal_Information.yaml)
• Finally, we commit and push the changes.

As a result, each folder has a bunch of YAML-files. We don't store zip-archives.

Does this workflow align with best practices, or is there a more efficient way to handle version control for Superset dashboards? Any feedback, suggestions, or resources would be greatly appreciated!

Thanks in advance!

[dosubot[bot]]

Hey @ampil! 👋 I'm here to help you with any questions or issues you have. I can assist with bugs, answer your questions, and even guide you on becoming a contributor. Let's tackle this together!

The best practices for storing Superset dashboards in Git for version control include:

1. Exporting Dashboards as YAML Files: Use the ExportDashboardsCommand to export dashboards to YAML files, which are more suitable for version control compared to binary formats like zip archives.

2. Organizing Files into Subfolders: Organize the exported YAML files into logical subfolders (e.g., dashboards, charts, datasets, databases) for better manageability.

3. Removing ID Numbers from Filenames: Customize the filenames to exclude ID numbers if desired, using a consistent naming convention.

Here is a more efficient way to handle this process:

1. Automate the Export Process: Use the ExportDashboardsCommand to automate the export of dashboards to YAML files.
2. Use a Consistent Naming Convention: Customize the get_filename function to exclude ID numbers from filenames if desired.
3. Organize Files in a Logical Directory Structure: Ensure that the exported YAML files are organized into appropriate subfolders.

Example code snippet to export dashboards:

from superset.commands.dashboard.export import ExportDashboardsCommand

# Export a specific dashboard by its ID
dashboard_id = 1  # Replace with the actual dashboard ID
export_command = ExportDashboardsCommand([dashboard_id])
for file_name, file_content_callable in export_command.run():
    file_content = file_content_callable()
    with open(file_name, 'w') as file:
        file.write(file_content)

This approach leverages the existing ExportDashboardsCommand to efficiently export dashboards as YAML files, which can then be version-controlled in Git ^[1].

To continue talking to Dosu, mention @dosu.

[ampil]

Our team explored two competing approaches for organizing dashboard configuration files.

Approach 1: Centralized Directory Structure

In this approach, all dashboard YAML files were stored in a common folder hierarchy, as follows:

superset/
│
├── charts/
│   ├── chart_name_1.yaml
│   ├── chart_name_2.yaml
│   ├── chart_name_3.yaml
│   └── etc.
│
├── dashboards/
│   ├── dashboard_name_1.yaml
│   ├── dashboard_name_2.yaml
│   ├── dashboard_name_3.yaml
│   └── etc.
│
├── databases/
│   ├── Oracle_db.yaml
│   ├── Postgre_db.yaml
│   ├── Postgre_Test_db.yaml
│   └── etc.
│
└── datasets/
    ├── Oracle_db/
    │   ├── dataset_1.yaml
    │   ├── dataset_2.yaml
    │   ├── dataset_3.yaml
    │   └── etc.
    ├── Postgre_db/
    │   ├── dataset_4.yaml
    │   ├── dataset_5.yaml
    │   ├── dataset_6.yaml
    │   └── etc.
    └── etc.

Approach 2: Individual Dashboard Folders

This approach involved creating a separate folder for each dashboard, organizing data as follows:

superset/
├── dashboard_name_1/
│   ├── charts/
│   │   ├── chart_name_1.yaml
│   │   └── chart_name_2.yaml
│   ├── dashboards/
│   │   └── dashboard_name_1.yaml
│   ├── databases/
│   │   └── Oracle_db.yaml
│   └── datasets/
│       └── Oracle_db/
│           └── dataset_1.yaml
│
└── dashboard_name_2/
   ├── charts/
   │   └── chart_name_3.yaml
   ├── dashboards/
   │   └── dashboard_name_2.yaml
   ├── databases/
   │   └── Postgre_db.yaml
   └── datasets/
       └── Postgre_db/
           └── dataset_4.yaml

Decision

We opted for the second approach. It offers several significant advantages over the first:

• Enhanced Findability: Each dashboard's data is easily accessible.
• Simplified Import: We can quickly zip folder contents and import reports back into Superset. The first approach would require developing a script to look up UUIDs in each YAML file.
• Clearer Code Review: When reviewing code in Git, it's much simpler to understand file relationships to a specific dashboard.

Hope it helps other Superset fellows :)

[spens]

For your use case, is it possible to have dataset_4.yaml used in dashboard_name_3 and dashboard_name_4? How do you handle that? Do you just store a duplicate copy of the dataset in the repo under each dashboard?

[ampil]

Yes, that's possible. Actually, we have that a lot - our filtering datasets are shared between many dashboards. For example, one dataset containing dates is utilized in 10+ dashboards.

The tradeoff is between:
(a) keeping it simple for humans (the team).
Here we isolate each dashboard data while monitoring shared datasets. In case of changes in one dataset, all associated yaml files in all related dashboards must be changed.

(b) optimizing it for computers.
This means avoiding duplicates, optimizing storage space, and ensuring that our YAML files are always up to date. On the other hand, we need additional tools to create zip files for each dashboard and to pay closer attention to shared datasets during the review stage.

Since our inputs are:

• 'custom' datasets that are distinguished between each dashboard and
• 'simple' shared datasets that are not likely to change (like a column with dates, numbers, text like "Yes" and "No") ,

We decided to stick with option (a) until a better solution is found or the workload changes.