Building Custom Viz Plugins in Superset v1

Note: this blog post is now out of date. Please check out our updated post here

HELLO, WORLD!

Arguably, the most important part of any data visualization product is... well, the data visualization! Superset offers numerous charting/viz options out of the box, but inevitably one question continues to reverberate throughout the Superset community: "How do I make a new plugin?"

Historically, the answer has been that you simply don't, because:

1. The process was simply too complex for most developers.
2. The visualization architecture is due for an overhaul.
3. Superset's repos were officially closed to submissions of new plugins.

But wait, there's good news! We've been making a lot of improvements to the process (as the result of Superset Improvement Proposals, or "SIPs"). Now we're well underway to opening up the world of viz plugins to the community, and we want to demonstrate this progress, by showing you how easy it is to make a new visualization of your own (or at least the scaffolding of one).

Today, you'll be making a Hello, World plugin, which you can then modify to make any plugin you so desire. The process herein will be focused on adding a viz plugin to the superset-ui repository on github, then adding it to Superset, with only minor changes to the incubator-superset repository. If you'd rather forgo the tutorial and just see the code, you can check out the "Example Code" section below.

There's work underway to make it so that you can easily build these plugins in your own repositories, and make loading them into Superset fairly trivial. We'll touch on that more in the "Next Steps" section at the end of this post.

What is the Hello World Viz?

Hello World is a fully operational model of a viz plugin. The intent is to provide a basic scaffolding to build any sort of data visualization, using any viz libary you'd like (.e.g. ECharts, AntV, HighCharts, VX, D3, or... you name it!). To that end, the result of this Hello World is super simple, to act as a jumping off point, providing the basic ingredients of a viz component for you to work with. From there, the world is your oyster!

You can build the Hello World plugin by running a Yeoman generator, which takes a few simple options, and provides this plugin scaffolding with the following features:

• LOTS of inline comments (and a README file!) to act as documentation as you explore the file structure
• A working query using the newly updated chart query endpoint (a separate blog post on that is coming soon!)
• Various working examples of control inputs types you can use, and how to wire them up. You can use these to manipulate your data query and/or directly manipulate your visualization
• Basic unit tests to build on
• a TransformProps file, with which you can manipulate your data and/or utilize your control inputs
• A React component to build on, with a some options to help get you started:
  • Whether you want your React component to be a Class or a Function (with hooks). They each have their own merits. Just choose the one that you want to riff on, and the Yeoman generator will set up the files accordingly. Both let you render HTML, and give you access to the viz's DOM element to do as you please with (sky's the limit!)
  • Whether you want a timeseries or non-timeseries plugin. If you choose timeseries, your plugin will be set up with the relevant control inputs and query operations so it works out of the box
  • Whether you want badges in your README file (only useful for official Superset plugins)

How we got to, and where we're going

In order to re-engage the community with a pattern for creating new viz plugins, we wanted to overhaul the existing architecture. That included building out a new API for fetching data, breaking down monolithic control panels that were baked into the main Superset repository, and simplifying the interfaces and file structures used to create a plugin.

A viz plugin can now be made:

• With all plugin code in a single pull request - you no longer have to add code in two repositories. (Note, you still need to make an additional pull request to add your plugin to Superset, but we'll get to that later in this article...)
• By writing in a single language - you no longer need to write Python code (but you can if you need to... more on that later)
• With simpler means of adding controls
• With safer, simpler CSS styles, leveraging a global UI theme

The history of these changes are addressed in greater detail in SIP-38.

The work to bring all existing plugins up to current standards is ongoing (and open to contributions from the community).

Getting Set Up

Install Yeoman and the Superset Package Generator

The new Hello World plugin we'll be building is generated automatically with Yeoman. Let's first get that installed by opening up a terminal and installing both the yo module and the superset package generator (v0.14.7) to create the new plugin.

npm install -g yo @superset-ui/generator-superset

Install Superset

There are complete instructions available on the Superset Github repository. In a nutshell, the easiest way is to:

1. Have a Mac or linux-based machine
2. Install Docker
3. Clone the repository to your computer
4. Use your terminal to cd into the incubator-superset directory
5. Run docker-compose up
6. Open another terminal, and cd into incubator-superset/superset-frontend
7. Run npm install to load up all the npm packages.
8. Run npm run dev-server to spin up the Webpack hot-reloading server
9. Wait for it to build, and then open your browser to http://localhost:9000 and log in with admin/admin. You're off to the races! (Note: we'll be restarting this later)

Install Superset-UI

1. Clone the superset-ui repository to your computer. It can sit in the same parent directory as your incubator-superset repo
2. Use your terminal to cd into superset-ui
3. Run yarn install and wait for all the packages to get installed

Build Your "Hello, World"

Write generate some code!

1. Using your terminal, cd into your local superset-ui repo folder and then into the plugins subdirectory.

2. Make a new directory for your plugin, i.e. mkdir plugin-chart-hello-world.

   Note: we highly recommend following the plugin-chart-your-plugin-name pattern.

3. Now cd plugin-chart-hello-world

4. Finally, run yo @superset-ui/superset

5. Select Create superset-ui chart plugin package on the following screen:

6. Give it a name (in our case, go with the default, based on the folder name):

7. Give it a description (again, default is fine!)

8. Choose which type of React component you want to make (Class, or Function component).

9. Select whether you'd like your visualization to be timeseries-based or not

10. Select whether or not you want to include badges at the top of your README file (really only needed if you intend to contribute your plugin to the superset-ui repo).

11. Admire all the files the generator has created for you. Note that EACH of these is chock full of comments about what they're for, and how best to use them.

Add your Plugin to Superset (with NPM Link)

Now, we want to see this thing actually RUN! To do that, we'll add your package to Superset and embrace the magic power of npm link to see it in-situ, without needing to build the plugin, or open any PRs on Github.

1. Add your package to the package.json file in incubator-superset/superset-frontend.

Note: Do _not_ run `npm install`... explanation below.

2. Add your plugin to the MainPreset.js file (located in incubator-superset/superset-frontend/src/visualizations/presets/MainPreset.js) in two places, alongside the other plugins.

3. Open a terminal window to incubator-superset/superset-frontend. If you did the Install Superset steps above, you may still have webpack running there, and you can just stop it with ctrol-c. If not, just open a new window and or cd to that directory path.
4. Create a link using a relative path to your plugin , e.g. npm link ../../superset-ui/plugins/plugin-chart-hello-world.
5. Restart your webpack dev server with npm run dev-server. You'll know it worked if you see a line stating [Superset Plugin] Use symlink source for @superset-ui/plugin-chart-hello-world @ ^0.0.0.

NOTE: If/when you do an npm install that erases the symlink generated by npm link, so you'll have to redo those steps.

NOTE: Dynamic import is a work in progress. We hope you won't even need to DO this soon. We'll be blogging again when that day comes, we assure you. In short, we have a goal to make editing package.json and MainPreset.js unnecessary, so all the code changes are made in ONE repo.

See it with your own eyes!

You should now be able to go to the Explore view in your local Superset and add a new chart! You'll see your new plugin when you go to select your viz type.

Now you can load up some data, and you'll see it appear in the plugin!

The plugin also outputs three things to your browser's console:

• formData, a.k.a. everything sent into your viz from the controls
• props, as output from the transformProps file for your plugin's consumption
• The actual HTML element, which your plugin has hooks into for any necessary DOM maniupluation

Make it Your Own

Now you're free to run wild with your new plugin! Here are a few places to start digging in:

Read the comments and docs

Take a look through the full file tree of the plugin. The Readme gives details for the job of each file. EACH of these files has been annotated with extensive comments of what the file is for, and the basics of what you can do with it.

Take control!

The plugin includes a couple of example controls, but you can certainly continue to add as many as you need to. The comments/documentation within the controls file is a start, but we recommend looking at existing superset-ui plugins for more examples of how you can implement controls to enhance your queries, work with your data, and change your visualization's display.

Build the perfect query

The buildQuery file where your plugin actually fetches data from the Superset backend. This file builds he query "context" for your plugin. For a simple plugin, this file needn't do much. There are a couple changes that need to be made for a timeseries plugin, thus the option in the Yeoman generator.

This file also allows you to add various post-processing operations, to have the Superset backend process your data in various ways (pivoting, etc), but that's a whole other topic we'll cover separately in the near future.

Style with Emotion

Each of these methods lets you add custom CSS styles using Emotion 👩‍🎤(a CSS-in-JS approach) which has access to Superset's burgeoning set of theme variables, and also automatically scopes the styles to your plugin, so they don't "leak" to other areas of Superset.

In the Hello World plugin, we've included a few example Theme variables (colors, gridUnits, and typographic weights/sizes). We'll be continuing to add more variables to this theme file as we continue to push Superset (and the viz plugins) toward the standards of the Superset redesign (see SIP-34)

Give it a thumbnail

Because come on... that's the fun part, right?

Build it!

In this tutorial, you built your plugin in the superset-ui repo. This means you can use the built-in build scripts that the repo provides. With your terminal of choice, simply cd into the root directory of supeset-ui and run yarn build. This will kick off a build of ALL the Superset plugins and packages, including yours.

Test early, test often!

The Hello World plugin includes some basic Jest tests to act as a starting point to add unit tests to your plugin. These do a quick sanity check that the plugin actually loads correctly, and then run through the basics of making sure that your controls are properly respected by modifying the resulting data and/or props of the plugin. Running yarn test from the root directory of superset-ui will run all the tests for plugins/packages, including your Hello World.

Typescript

Superset makes extensive use of Typescript (see SIP-36), and thusly the Hello World plugin is written with Typescript accordingly. If you are not comfortable with Typescript, and do not intend to open a pull request with your new plugin, you have some options. You could either convert your .ts(x) files to .js(x) to opt out completely, or use the opt-out comments of // @ts-ignore to opt out of single lines and // @ts-nocheck at the top of a file to skip a file's type checking completely.

Add it to Storybook!

The superset-ui repo contains a Storybook to showcase existing viz plugins. If you ever open a PR to add a viz plugin to superset-ui please be sure to include a storybook entry here as well.

Share your plugin!

We'd love to see what you've built! Please share it with the Superset Slack community, on your GitHub, or just [send us an email](mailto:info@preset.io?cc=evan@preset.io&subject=Hello World) about it! We'd love to feature any cool work in the Superset newsletter.

If your plugin is something you think the entire Superset community can make use of, feel free to open a pull request to superset-ui! For plugins that are more experimental, or for specialized/rare use cases, we plan to provide a better means to publish such viz plugins in the near future (and let you know about it here!). If you do open a PR on superset-ui, please wait for that to be approved/merged before (optionally) opening a PR on incubator-superset to enable the plugin for everyone. Additional contributing guidelines are available for incubaor-superset and superset-ui

Next Steps

There are still several projects currently underway to make the visualization development experience even simpler. Here's a sneak preview of things coming soon:

• Dynamic import - Currently, if you want to actually build and deploy your plugin, you need to leverage the build scripts of superset-ui, and import the resulting built plugin as part of Superset's package.json file. We plan to make it so that single plugins can be built with their own Webpack configuration, hosted anywhere (a CDN for example), and added to (or removed from) Superset via its configuration file. We can't wait to tell you more in a future blog post.

• Migrating existing plugins - We've been working hard to make it easier to build new viz plugins, and will continue to do so. But now, we also have the work of updating the core Superset UI plugin packages to meet the new standards and patterns. If you'd like to help with this effort, please reach out! This work makes great starter tasks for anyone interested in joining the superset dev community. The Superset Slack workspace (e.g. the #visualization_plugins channel) is a great place to begin.

Example Code

If you want to see the results of the yeoman generator without running it, here are some links to pull requests with various permutations of the output:

Shout-outs

Many people have contributed extensively to making this possible, but thanks in particular to:

• Ville Brofeldt for his work on this Hello World project, the chart data API endpoint, and extensive python refactoring to make this all work
• Krist Wongsuphasawat for teaching us how these viz plugins were all put together in the first place, and helping us find a sensible path forward
• Jesse Yang for helping with countless imrpovements to build processes, plugin package migrations/refactoring, and a million PRs and reviews along the way

Note: this blog post is now out of date. Please check out our updated post here