You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Difference between revisions of "Analytics/Systems/Superset"

From Wikitech-static
Jump to navigation Jump to search
imported>Elukey
imported>Elukey
Line 5: Line 5:


If you have that access, you can log in at [https://superset.wikimedia.org superset.wikimedia.org] with your developer shell username and password.
If you have that access, you can log in at [https://superset.wikimedia.org superset.wikimedia.org] with your developer shell username and password.
== Druid datasources vs Druid tables ==
We should stop adding datasources to the [https://superset.wikimedia.org/druiddatasourcemodelview/list/ "Druid Datasources" list] if possible in favor of Druid tables (backed by Druid SQLAlchemy mappings). In order to add a Druid table, do the following:
* Go to Sources -> Tables
* Hit the + sign to add one
* Use <code>Druid Analytics SQL</code> as database, <code>druid</code> as schema and the Druid datasource name as table name (like <code>edits_hourly</code>, <code>wmf_netflow</code>, etc..)


== Usage notes ==
== Usage notes ==

Revision as of 09:49, 29 April 2020

Superset is an Apache incubator project, originally started at AirBnB. It enables visualizations and dashboards built from various analytics data sources. WMF's Superset instance can be found at https://superset.wikimedia.org. Like Turnilo, it provides access to various Druid tables.

Access

To access Superset, you need wmf or nda LDAP access. For more details, see Analytics/Data access#LDAP access.

If you have that access, you can log in at superset.wikimedia.org with your developer shell username and password.

Druid datasources vs Druid tables

We should stop adding datasources to the "Druid Datasources" list if possible in favor of Druid tables (backed by Druid SQLAlchemy mappings). In order to add a Druid table, do the following:

  • Go to Sources -> Tables
  • Hit the + sign to add one
  • Use Druid Analytics SQL as database, druid as schema and the Druid datasource name as table name (like edits_hourly, wmf_netflow, etc..)

Usage notes

  • The "Druid Datasources" list shows ingested tables that are available for querying. As of October 2018, this includes e.g. daily and hourly pageviews data (the daily version is only updates once a month, but goes further back), a sampled excerpt of webrequest data, unique devices, and a few select EventLogging schemas. If a recently created Druid datasource is not yet visible in the list, try clicking "Scan New Datasouces".
  • NULL values don't show up properly in the values selection dropdown list for filters (i.e. one can't use that dropdown to exclude NULL values from a chart or limit it to NULL values). But one can use the regex option instead: Type in ".+" (without the quotes), and accept the offer to create that as an option.
  • By default, always use predefined SUM metrics when available. When choosing a metric then picking the SUM aggregation function, the aggregation is managed by superset and uses the floatSum operator. This operator uses 32 bits floats instead of 64 bits longs or double, leading to inaccuracies. Usually predefined SUM(...) metrics are available and should be used, as they are manually defined using doubleSum or longSum 64 bits operators.

SQL Lab

Superset allows user to query data via SQL using a dedicated tool called SQL Lab: https://superset.wikimedia.org/superset/sqllab Multiple databases are available to query: - presto_analytics_hive to explore Hive's databases/tables via Presto. - Druid Analytics SQL to explore all Druid datasources via SQL (warning: not all the full power of SQL will be available) - etc..

Administration

Upgrading

To upgrade, first follow the instructions in the analytics/superset/deploy README to update the deploy repository. Once deployed, activate the superset virtualenv, add /etc/superset to PYTHONPATH (to allow superset to pick up configuration) and follow the Superset upgrade instructions (minus the pip install superset --upgrade part). This should be something like:

. /srv/deployment/analytics/superset/venv/bin/activate
export PYTHONPATH=/etc/superset
superset db upgrade
superset init

Deploy

This assumes that one has already filed the change for the superset deploy repository. The first thing to do is test the change on the staging instance, an-tool1005.eqiad.wmnet:

# ssh to deploy1001 and set the working directory
ssh deploy1001.eqiad.wmnet
cd /srv/deployment/analytics/superset/deploy

# create a new branch from the master one, name it as you prefer
git checkout -B testing_something_important

# cherry pick the change in the new branch created
git cherry-pick $change-from-gerrit

# deploy only to an-tool1005, without logging in the ops's sal
scap deploy --no-log-message -f -l an-tool1005.eqiad.wmnet "Test deployment for something important"

Then check if Superset works as expected:

# Create a ssh tunnel and then test it via localhost:8080 on the browser
ssh -L 8080:an-tool1005.eqiad.wmnet:80 an-tool1005.eqiad.wmnet

If you are happy with the Superset version, then merge and deploy to the production host:

# ssh to deploy1001 and set the working directory
ssh deploy1001.eqiad.wmnet
cd /srv/deployment/analytics/superset/deploy

scap deploy -l analytics-tool1004.eqiad.wmnet "Deployment for something important"

How to

See status

systemctl status superset.service

Bounce

systemctl restart superset

See also
{"cc_re__cc_re__cc_by_sa_3_0": ["le=\"Permanent link to this revision of the page\">Permanent link</a></li><li id=\"t-info\"><a href=\"/w/index.php?title=Analytics/Systems/Superset&amp;action=info\" title=\"More information about this page\">Page information</a></li></ul>\n\t\t\n\t</div>\n</nav>\n\n\t\n</div>\n\n</div>\n<footer id=\"footer\" class=\"mw-footer\" role=\"contentinfo\" >\n\t<ul id=\"footer-info\" >\n\t<li id=\"footer-info-lastmod\"> This page was last edited on 29 April 2020, at 09:49.</li>\n\t<li id=\"footer-info-copyright\">Content is available under <a class=\"external\" rel=\"nofollow\" href=\"http://creativecommons.org/licenses/by-sa/3.0/\">Creative Commons Attribution Share Alike</a> unless otherwise noted. By using Wikimedia Labs, you agree to the <a href=\"//wikitech.wikimedia.org/wiki/Wikitech:Labs_Terms_of_use\">Labs Terms of Use</a>.</li>\n</ul>\n\n\t<ul id=\"footer-places\" >\n\t<li id=\"footer-places-privacy\"><a href=\"/wiki/Wikitech-static:Privacy_policy\" title=\"Wikitech-static:Privacy policy\">Privacy policy</a></li>\n\t<li id=\"footer-places-about\"><a href=\"/wiki/Main_Page\" title=\"Main Page\">About Wikitech-static</a></li>\n\t<li id=\"foote", ">\n\t<li id=\"footer-places-privacy\"><a href=\"/wiki/Wikitech-static:Privacy_policy\" title=\"Wikitech-static:Privacy policy\">Privacy policy</a></li>\n\t<li id=\"footer-places-about\"><a href=\"/wiki/Main_Page\" title=\"Main Page\">About Wikitech-static</a></li>\n\t<li id=\"footer-places-disclaimer\"><a href=\"https://foundation.wikimedia.org/wiki/General_disclaimer\" class=\"extiw\" title=\"wikimedia:General disclaimer\">Disclaimers</a></li>\n</ul>\n\n\t<ul id=\"footer-icons\" class=\"noprint\">\n\t<li id=\"footer-copyrightico\"><a href=\"http://creativecommons.org/licenses/by-sa/3.0/\"><img src=\"/w/resources/assets/licenses/cc-by-sa.png\" alt=\"Creative Commons Attribution Share Alike\" width=\"88\" height=\"31\" loading=\"lazy\"/></a></li>\n\t<li id=\"footer-poweredbyico\"><a href=\"https://www.mediawiki.org/\"><img src=\"/w/resources/assets/poweredby_mediawiki_88x31.png\" alt=\"Powered by MediaWiki\" srcset=\"/w/resources/assets/poweredby_mediawiki_132x47.png 1.5x, /w/resources/assets/poweredby_mediawiki_176x62.png 2x\" width=\"88\" height=\"31\" loading=\"lazy\"/></a></li>\n</ul>\n\n</footer>\n\n<!-- Debu"], "copyright__copyright__copyright": ["t it via localhost:8080 on the browser\nssh -L 8080:an-tool1005.eqiad.wmnet:80 an-tool1005.eqiad.wmnet\n\nIf you are happy with the Superset version, then merge and deploy to the production host:\n\n# ssh to deploy1001 and set the working directory\nssh deploy1001.eqiad.wmnet\ncd /srv/deployment/analytics/superset/deploy\n\nscap deploy -l analytics-tool1004.eqiad.wmnet \"Deployment for something important\"\n\nHow to\n\nSee status\n\nsystemctl status superset.service\n\nBounce\n\nsystemctl restart superset\n\nSee also", "t it via localhost:8080 on the browser\nssh -L 8080:an-tool1005.eqiad.wmnet:80 an-tool1005.eqiad.wmnet\n\nIf you are happy with the Superset version, then merge and deploy to the production host:\n\n# ssh to deploy1001 and set the working directory\nssh deploy1001.eqiad.wmnet\ncd /srv/deployment/analytics/superset/deploy\n\nscap deploy -l analytics-tool1004.eqiad.wmnet \"Deployment for something important\"\n\nHow to\n\nSee status\n\nsystemctl status superset.service\n\nBounce\n\nsystemctl restart superset\n\nSee also"]}