Apache Superset 4.0 Release Notes

Introduction

Today the Apache Superset community is exicted to announce Superset 4.0. This has been a huge undertaking of many organizations and individuals, and represents not just the software release itself, but also the culmination of much process change. The aim is to build a more stable and maintainable piece of software, and in doing so create a more reliable process for aligning on change management and release processes, to smooth out and speed up our release cadence. We hope that not only is Superset 4.0 the best release of Superset yet, but that this improved process and cadence will help you more seamlessly adapt to future versions as they arise.

Please note that much of this post is the "unabridged" version of release notes available on the GitHub repo. You can view the official release notes for 4.0 and all prior releases here.

About the Release

4.0.0 brings a plethora of exciting changes to Superset. We have introduced several breaking changes to improve the overall architecture and scalability of our codebase. These changes may require some code updates, but they are designed to enhance performance and maintainability in the long run. We have also upgraded various dependencies to their latest versions and deprecated certain features that are no longer aligned with our long-term roadmap. We encourage all developers to carefully review the CHANGELOG.md and UPDATING.md files and update their code accordingly. While our main focus was on code cleanup, this release also contains exciting new features and marks a significant milestone for the project.

Here are some of the highlights of this release.

Alerts and Reports modal redesign

The Alerts and Reports modal has been redesigned to improve the user experience and make it more intuitive. The new design has the following goals:

• Declutter the interface by providing a cleaner, more organized layout
• Create a linear setup process with the necessary options in a step-by-step manner to make alert/report setup more intuitive
• Prepare the interface for additional features that will be introduced in future releases, like the ability to pre-filter a dashboard being sent.

Tags

Tags are available using the TAGGING_SYSTEM feature flag. They address many of the requests made by the community and aim to make it easier to organize and curate charts, dashboards, and saved queries, allowing for effortless data discovery and collaboration within an organization. Users can create flexible and customizable tags for each piece of content, enabling different ways of organizing assets. Programmatic access to tag-related operations are supported via the RESTful API.

New CHANGELOG format

We changed the structure of the CHANGELOG.md file in #26800 to better organize the contents of each release and also to deal with GitHub size limitations when displaying the file. Now every release will have its own file at CHANGELOG/<version>.md. The main CHANGELOG.md file is now an index with links to all releases.

Improved drag and drop experience when editing a dashboard

When a component was being dragged towards the edge of the tab container or the row/column containers, multiple drop indicators were often displayed. This created confusion about the exact insertion point of the element. To fix this, we built in #26699 and #26313 a distinct, non-conflicting area for the drop zone.

As you can see here, the drop zone is highlighted during the dragging process to clearly indicate where the element will be placed.

We also improved the forbidden drop zones to prevent users from dropping elements in invalid locations.

Improved drag and drop experience when editing a chart

Now, during dragging, all droppable zones are highlighted, with distinct colors indicating available and unavailable drop locations. This enhancement clarifies potential drop points and helps avoid inadvertent placements in invalid areas. The update also aligns the drag-over feedback with the dashboard's drag-and-drop modifications, ensuring a uniform and enhanced user experience.

Feature flag changes

Following our 4.0 proposals, the following feature flags were removed, i.e., the feature was permanently enabled or removed.

• VERSIONED_EXPORT
• DASHBOARD_FILTERS_EXPERIMENTAL
• ENABLE_EXPLORE_JSON_CSRF_PROTECTION
• ENABLE_TEMPLATE_REMOVE_FILTERS
• REMOVE_SLICE_LEVEL_LABEL_COLORS
• CLIENT_CACHE
• DASHBOARD_CACHE
• DASHBOARD_NATIVE_FILTERS_SET
• ENABLE_EXPLORE_DRAG_AND_DROP
• DISABLE_DATASET_SOURCE_EDIT
• DASHBOARD_NATIVE_FILTERS
• GENERIC_CHART_AXES

The following feature flags were deprecated:

• DASHBOARD_CROSS_FILTERS
• ENABLE_JAVASCRIPT_CONTROLS
• KV_STORE

The following feature flags were enabled by default:

• DASHBOARD_VIRTUALIZATION
• DRILL_BY

Removed features

As part of the 4.0 approved initiatives, the following features were removed from Superset:

• Filter Box: #26328 removed the Filter Box code and its associated dependencies react-select and array-move. It also removed the DeprecatedSelect and AsyncSelect components that were exclusively used by filter boxes. Existing filter boxes will be automatically migrated to native dashboard filters.

• Filter Sets: #26369 removed the Filters Set feature including the deprecated DASHBOARD_NATIVE_FILTERS_SET feature flag and all related API endpoints. The feature is permanently removed as it was not being actively maintained, it was not widely used, and it was full of bugs. We also considered that if we were to provide a similar feature, it would be better to re-implement it from scratch given the amount of technical debt that the implementation had.

• Profile: #26462 removed the Profile feature given that it was not actively maintained nor widely used.

• Redirect API: #26377 removed the deprecated Redirect API that supported short URLs (/r) and the url metadata table used to store them that was used before the permalink feature. Users lost the ability to generate R links ~1.5 years ago which seems sufficient time to remove the API.

Business logic improvements

As part of [SIP-99] Proposal for correctly handling business logic (specifically SIP-99A and SIP-99B), this release contains many improvements to the handling of business logic in Superset, specifically related to SQLAlchemy sessions and transactions. The goal of these efforts is to simplify the code, improve code quality, ensure a consistent "unit of work" approach, and provide clear guidance and examples of accepted code standards. These changes aim to improve developer experience by making the code simpler, improving testing, and ensuring a more streamlined and reliable system. We still have a long way to go to fully implement the SIP-99 proposal, but we are making progress and we are excited about the improvements that have been made so far.

All country maps are now managed via Jupyter Notebook

In this release we made updates to the Jupyter Notebook to ensure reliable execution by removing deprecated methods, adding new countries, including missing maps, and fixing filename inconsistencies. This will make it easier to add more countries, dynamically add them to the country map plugin, and update map regions periodically. You can check #26300 for more details.

Sunburst chart migrated to ECharts

The ECharts version of the Sunburst chart was introduced by #22833 as part of our efforts to complete SIP-50. In 4.0, legacy Sunburst charts are automatically migrated to ECharts and the legacy version was removed.

Some cool stats

• ~15K lines of code were removed by PRs related to 4.0 proposals
• We reduced the number of NPM packages vulnerabilities by 72%
  • 3.1: 90 vulnerabilities (42 moderate, 34 high, 14 critical)
  • 4.0: 25 vulnerabilities (16 moderate, 8 high, 1 critical)
• 40+ dependency changes (upgrades, additions, and removals)

How to upgrade

As with any Superset version upgrade, the process is simple in the broadest strokes, as outlined in the documentation. However, as with any upgrade, we expect to see numerous speed bumps along that path depending on your configuration, your infrastructure, your databases in use, and other customizations/configurations. To make a safe leap to this version, we'd suggest the following steps:

• Back up your databases
• Carefully read CHANGELOG.md for all the incremental changes in this version (and any prior versions between your current installation and 4.0.0).
• Similarly, review UPDATING.md to keep an eye out for all changes that have been explicitly marked as breaking changes.
• Adjust your feature flags and configurations to meet your feature requirements and preferences.
• Execute the migrations
• If you have third-party apps interacting with Superset, check for relevant dependency updates or API endpoint changes that may affect compatibility.

Your mileage may vary depending on:

• How you install and deploy Superset (e.g. docker vs. pip vs. helm)
• How you’ve configured Superset
• What integrations, databases, etc. you're using

Reach out in #deploying-superset on Slack in case you find any problems, and if you find a reproducible bug, please file a new issue on GitHub.

What's new in the release process

Recently, the Superset community officially adopted a new release process that helped drive this release. For those unfamiliar, the Superset community functions under an Operational Model that includes a Release Strategy working group. Through many meetings of this group, and discussions in the Apache Superset Town Hall, we came up with many proposals that were brought up on the Apache Superset Dev Mailing List for official consensus. Long story short, the results of all this work have been published here on the Superset Wiki.

Among the highlights, this release was conducted on a GitHub Project kanban board, which is something we experimented with in 3.0, but the process was made official to help implement 4.0. This tracked all the proposals made for the release (breaking or otherwise) as they gathered consensus, underwent implementation, and were either completed, or punted to the next major release. The changes were eligible to be merged into the master branch only during an approved "breaking change window" as part of a "release train" model. Anything that missed the boat will be held over for reconsideration as part of the next major release (more on that shortly!)

If you happen to be a Preset customer, please note that you are already upgraded to Superset 4.0. Preset is now building upon official Apache Superset releases, adding and testing additional features and customizations upon the codebase, helping to ensure more comprehensive testing of the official release candidate for all open source Superset users, while ensuring that all Preset users experience a seamless and effortless transition to the latest release.

The roadmap ahead

With this release, the Superset community is targeting two different timelines in terms of versioning. The first is that of the next major and minor releases, and the second is in regards to legacy version support.

Future versions

In terms of upcoming versions, we have two goals. One is the next major version. We already have several standing proposals left over from the 4.0 process, and undoubtedly more to come. Since major versions may include breaking changes (feature flag changes, code/feature removal, etc.), those will, in time, be brought to the community for consensus to be merged during the next breaking change window. We intend to complete this major version process approximately every six months until we get through the backlog of known change proposals. In-between major releases, we intend to further stabilize each release with a minor release (including new features and fixes) approximately 3 months later, with the possibility of additional minor and patch releases in-between as necessitated by fixes or other needs that might arise.

Dropping support for 3.0.x and prior

The second timeline we're targeting (or evangelizing) that of version support. In accordance with our release process, we are dropping support for the 3.0.X versions. As a result, we will no longer be providing bug fixes for these versions.

The release managers and PMC intend to provide fixes/patches for the most recent two versions (major or minor versions). With this release, the supported versions will be 3.1.X and 4.0.X. Bug fixes will continue to be backported to 3.1.X until the next minor release. For more information, please refer to our release schedule.

While we may back-port security fixes further when we have the opportunity to do so, we strongly encourage users to update to these latest supported releases for the best experience, taking advantage of the newest features and bug fixes.

Opportunities to get involved

There are several things that you can do as a Superset (or Preset) user to get involved in this and future releases:

• Continue testing and filing issues. The sooner we're able to reproduce and diagnose any issues you encounter in 4.0, the better and more stable the patch and minor releases will become.
• Testing release candidates. If you aren't already on the Superset dev email list, you can subscribe by sending an email to dev-subscribe@superset.apache.org, and you'll get the opportunity to test and vote on release candidates as they arise.
• Join the Release Strategy group and/or attend Superset Town Hall sessions, which are all available on the public Community Calendar available on our website.
• Contribute! Whether it's fixing a small cosmetic bug, updating some documentation, helping review issues and PRs on the Github repo, or contributing a Superset Improvement Proposal, we're always looking for ways to get people more involved in making Superset better. Join us on Slack and reach out to any contributor to find some ways to help us continue steering the project forward.
• Tell the wold you use Superset. We want to spread the word far and wide. You can do so by starring the GitHub repo (we're almost the #1 ASF project!), adding your name/org to the In the Wild page, or posting any blogs/videos/projects involving Superset on the #content-share channel on Slack.

It's largely thanks to all of you that Superset has grown as popular as it has, and we look forward to continue working with you to make it bigger and better than it's ever been.