<p dir="auto">I am interested in implementing a new connector for Superset to add MongoDB support, and would like to know if others have started similar work too so as to avoid duplicate effort.</p>
<h3 dir="auto">Inspiration</h3>
<p dir="auto"><a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/priyankajuyal/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/priyankajuyal">@priyankajuyal</a> I think there're 3 methods to support mongodb in superset:</p>
<ul dir="auto">
<li>Implement a mongodb in sqlalchemy, this requires a lot of work and may not be easy to do cause sqlalchemy is a object relational mapper library, while mongodb is not relational.</li>
<li>Implement a new connector for superset. Right now superset has two connectors, one is for sqlalchemy, one is for druid, and similar things can be implemented for mongodb, this should be the right choice.</li>
<li>Transform mongodb to some relational db, stripe has a deprecated project for this job: <a href="https://github.com/stripe/mosql">https://github.com/stripe/mosql</a>.</li>
</ul>
<p dir="auto"><em>Originally posted by <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/xiaohanyu/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/xiaohanyu">@xiaohanyu</a> in <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="289542969" data-permission-text="Title is private" data-url="https://github.com/apache/superset/issues/4231" data-hovercard-type="issue" data-hovercard-url="/apache/superset/issues/4231/hovercard?comment_id=358608747&amp;comment_type=issue_comment" href="https://github.com/apache/superset/issues/4231#issuecomment-358608747">#4231 (comment)</a></em></p>
<p dir="auto">Make sure these boxes are checked before submitting your issue - thank you!</p>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the superset logs for python stacktraces and included it here as text if there are any.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have reproduced the issue with at least the latest released version of superset.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the issue tracker for the same issue and I haven't found one similar.</li>
</ul>
<h3 dir="auto">Superset version</h3>
<p dir="auto">Lastest commit on master</p>
<h3 dir="auto">Expected results</h3>
<p dir="auto">Docker compose working</p>
<h3 dir="auto">Actual results</h3>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ docker-compose up
superset_postgres_1 is up-to-date
superset_redis_1 is up-to-date
Creating superset_superset_1 ... done
Attaching to superset_postgres_1, superset_redis_1, superset_superset_1
superset_1  | + '[' 0 -ne 0 ']'
superset_1  | + '[' development = development ']'
superset_1  | + celery worker --app=superset.sql_lab:celery_app --pool=gevent -Ofair
superset_1  | + cd superset/assets/
superset_1  | + npm ci
postgres_1  | 2019-02-07 15:41:55.069 UTC [1] LOG:  listening on IPv4 address &quot;0.0.0.0&quot;, port 5432
postgres_1  | 2019-02-07 15:41:55.069 UTC [1] LOG:  listening on IPv6 address &quot;::&quot;, port 5432
postgres_1  | 2019-02-07 15:41:55.109 UTC [1] LOG:  listening on Unix socket &quot;/var/run/postgresql/.s.PGSQL.5432&quot;
postgres_1  | 2019-02-07 15:41:55.235 UTC [22] LOG:  database system was shut down at 2019-02-07 15:31:58 UTC
postgres_1  | 2019-02-07 15:41:55.249 UTC [1] LOG:  database system is ready to accept connections
postgres_1  | 2019-02-07 15:42:51.678 UTC [30] ERROR:  duplicate key value violates unique constraint &quot;ab_user_username_key&quot;
postgres_1  | 2019-02-07 15:42:51.678 UTC [30] DETAIL:  Key (username)=(admin) already exists.
postgres_1  | 2019-02-07 15:42:51.678 UTC [30] STATEMENT:  INSERT INTO ab_user (id, first_name, last_name, username, password, active, email, last_login, login_count, fail_login_count, created_on, changed_on, created_by_fk, changed_by_fk) VALUES (nextval('ab_user_id_seq'), 'admin', 'user', 'admin', 'pbkdf2:sha256:50000$fgf6eYlb$bb1e6c1f420de4d894f7a917a72712c0b846dd969f51afb7cf386f0848508422', true, 'admin@fab.org', NULL, NULL, NULL, '2019-02-07T15:42:51.677213'::timestamp, '2019-02-07T15:42:51.677242'::timestamp, NULL, NULL) RETURNING ab_user.id
redis_1     | 1:C 07 Feb 15:41:55.444 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
redis_1     |                 _._
redis_1     |            _.-``__ ''-._
redis_1     |       _.-``    `.  `_.  ''-._           Redis 3.2.12 (00000000/0) 64 bit
redis_1     |   .-`` .-```.  ```\/    _.,_ ''-._
redis_1     |  (    '      ,       .-`  | `,    )     Running in standalone mode
redis_1     |  |`-._`-...-` __...-.``-._|'` _.-'|     Port: 6379
redis_1     |  |    `-._   `._    /     _.-'    |     PID: 1
redis_1     |   `-._    `-._  `-./  _.-'    _.-'
redis_1     |  |`-._`-._    `-.__.-'    _.-'_.-'|
redis_1     |  |    `-._`-._        _.-'_.-'    |           http://redis.io
redis_1     |   `-._    `-._`-.__.-'_.-'    _.-'
redis_1     |  |`-._`-._    `-.__.-'    _.-'_.-'|
redis_1     |  |    `-._`-._        _.-'_.-'    |
redis_1     |   `-._    `-._`-.__.-'_.-'    _.-'
redis_1     |       `-._    `-.__.-'    _.-'
redis_1     |           `-._        _.-'
redis_1     |               `-.__.-'
redis_1     |
redis_1     | 1:M 07 Feb 15:41:55.446 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
redis_1     | 1:M 07 Feb 15:41:55.446 # Server started, Redis version 3.2.12
redis_1     | 1:M 07 Feb 15:41:55.446 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never &gt; /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
redis_1     | 1:M 07 Feb 15:41:55.446 * DB loaded from disk: 0.000 seconds
redis_1     | 1:M 07 Feb 15:41:55.446 * The server is now ready to accept connections on port 6379
superset_1  | npm WARN prepare removing existing node_modules/ before installation
superset_1  | npm ERR! path /home/superset/superset/assets/node_modules/core-js/fn/array/virtual
superset_1  | npm ERR! code ENOTEMPTY
superset_1  | npm ERR! errno -39
superset_1  | npm ERR! syscall rmdir
superset_1  | npm ERR! ENOTEMPTY: directory not empty, rmdir '/home/superset/superset/assets/node_modules/core-js/fn/array/virtual'
superset_1  |
superset_1  | npm ERR! A complete log of this run can be found in:
superset_1  | npm ERR!     /home/superset/.npm/_logs/2019-02-07T15_47_06_436Z-debug.log
postgres_1  | 2019-02-07 15:47:06.572 UTC [36] LOG:  unexpected EOF on client connection with an open transaction
superset_1  | npm WARN prepare removing existing node_modules/ before installation
superset_1  | npm ERR! path /home/superset/superset/assets/node_modules/core-js/fn/array/virtual
superset_1  | npm ERR! code ENOTEMPTY
superset_1  | npm ERR! errno -39
superset_1  | npm ERR! syscall rmdir
superset_1  | npm ERR! ENOTEMPTY: directory not empty, rmdir '/home/superset/superset/assets/node_modules/core-js/fn/array/virtual'
superset_1  |
superset_1  | npm ERR! A complete log of this run can be found in:
superset_1  | npm ERR!     /home/superset/.npm/_logs/2019-02-07T15_50_51_169Z-debug.log
postgres_1  | 2019-02-07 15:50:51.297 UTC [44] LOG:  unexpected EOF on client connection with an open transaction
superset_superset_1 exited with code 217
superset_1  | npm WARN prepare removing existing node_modules/ before installation
superset_1  | npm ERR! path /home/superset/superset/assets/node_modules/core-js/fn/array/virtual
superset_1  | npm ERR! code ENOTEMPTY
superset_1  | npm ERR! errno -39
superset_1  | npm ERR! syscall rmdir
superset_1  | npm ERR! ENOTEMPTY: directory not empty, rmdir '/home/superset/superset/assets/node_modules/core-js/fn/array/virtual'
superset_1  |
superset_1  | npm ERR! A complete log of this run can be found in:
superset_1  | npm ERR!     /home/superset/.npm/_logs/2019-02-07T15_54_35_445Z-debug.log
postgres_1  | 2019-02-07 15:54:35.595 UTC [52] LOG:  unexpected EOF on client connection with an open transaction
superset_superset_1 exited with code 217
superset_1  | npm WARN prepare removing existing node_modules/ before installation
superset_1  | npm ERR! path /home/superset/superset/assets/node_modules/core-js/fn/array/virtual
superset_1  | npm ERR! code ENOTEMPTY
superset_1  | npm ERR! errno -39
superset_1  | npm ERR! syscall rmdir
superset_1  | npm ERR! ENOTEMPTY: directory not empty, rmdir '/home/superset/superset/assets/node_modules/core-js/fn/array/virtual'
superset_1  |
superset_1  | npm ERR! A complete log of this run can be found in:
superset_1  | npm ERR!     /home/superset/.npm/_logs/2019-02-07T15_58_19_852Z-debug.log
postgres_1  | 2019-02-07 15:58:19.963 UTC [61] LOG:  unexpected EOF on client connection with an open transaction
superset_superset_1 exited with code 217
superset_1  | npm WARN prepare removing existing node_modules/ before installation
superset_1  | npm ERR! path /home/superset/superset/assets/node_modules/core-js/fn/array/virtual
superset_1  | npm ERR! code ENOTEMPTY
superset_1  | npm ERR! errno -39
superset_1  | npm ERR! syscall rmdir
superset_1  | npm ERR! ENOTEMPTY: directory not empty, rmdir '/home/superset/superset/assets/node_modules/core-js/fn/array/virtual'
superset_1  |
superset_1  | npm ERR! A complete log of this run can be found in:
superset_1  | npm ERR!     /home/superset/.npm/_logs/2019-02-07T16_02_04_350Z-debug.log
postgres_1  | 2019-02-07 16:02:04.473 UTC [69] LOG:  unexpected EOF on client connection with an open transaction
superset_superset_1 exited with code 217
superset_1  | npm WARN prepare removing existing node_modules/ before installation
superset_1  | npm ERR! path /home/superset/superset/assets/node_modules/core-js/fn/array/virtual
superset_1  | npm ERR! code ENOTEMPTY
superset_1  | npm ERR! errno -39
superset_1  | npm ERR! syscall rmdir
superset_1  | npm ERR! ENOTEMPTY: directory not empty, rmdir '/home/superset/superset/assets/node_modules/core-js/fn/array/virtual'
superset_1  |
superset_1  | npm ERR! A complete log of this run can be found in:
superset_1  | npm ERR!     /home/superset/.npm/_logs/2019-02-07T16_05_48_096Z-debug.log
postgres_1  | 2019-02-07 16:05:48.222 UTC [78] LOG:  unexpected EOF on client connection with an open transaction
superset_superset_1 exited with code 217
superset_1  | npm WARN prepare removing existing node_modules/ before installation"><pre class="notranslate"><code class="notranslate">$ docker-compose up
superset_postgres_1 is up-to-date
superset_redis_1 is up-to-date
Creating superset_superset_1 ... done
Attaching to superset_postgres_1, superset_redis_1, superset_superset_1
superset_1  | + '[' 0 -ne 0 ']'
superset_1  | + '[' development = development ']'
superset_1  | + celery worker --app=superset.sql_lab:celery_app --pool=gevent -Ofair
superset_1  | + cd superset/assets/
superset_1  | + npm ci
postgres_1  | 2019-02-07 15:41:55.069 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
postgres_1  | 2019-02-07 15:41:55.069 UTC [1] LOG:  listening on IPv6 address "::", port 5432
postgres_1  | 2019-02-07 15:41:55.109 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
postgres_1  | 2019-02-07 15:41:55.235 UTC [22] LOG:  database system was shut down at 2019-02-07 15:31:58 UTC
postgres_1  | 2019-02-07 15:41:55.249 UTC [1] LOG:  database system is ready to accept connections
postgres_1  | 2019-02-07 15:42:51.678 UTC [30] ERROR:  duplicate key value violates unique constraint "ab_user_username_key"
postgres_1  | 2019-02-07 15:42:51.678 UTC [30] DETAIL:  Key (username)=(admin) already exists.
postgres_1  | 2019-02-07 15:42:51.678 UTC [30] STATEMENT:  INSERT INTO ab_user (id, first_name, last_name, username, password, active, email, last_login, login_count, fail_login_count, created_on, changed_on, created_by_fk, changed_by_fk) VALUES (nextval('ab_user_id_seq'), 'admin', 'user', 'admin', 'pbkdf2:sha256:50000$fgf6eYlb$bb1e6c1f420de4d894f7a917a72712c0b846dd969f51afb7cf386f0848508422', true, 'admin@fab.org', NULL, NULL, NULL, '2019-02-07T15:42:51.677213'::timestamp, '2019-02-07T15:42:51.677242'::timestamp, NULL, NULL) RETURNING ab_user.id
redis_1     | 1:C 07 Feb 15:41:55.444 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
redis_1     |                 _._
redis_1     |            _.-``__ ''-._
redis_1     |       _.-``    `.  `_.  ''-._           Redis 3.2.12 (00000000/0) 64 bit
redis_1     |   .-`` .-```.  ```\/    _.,_ ''-._
redis_1     |  (    '      ,       .-`  | `,    )     Running in standalone mode
redis_1     |  |`-._`-...-` __...-.``-._|'` _.-'|     Port: 6379
redis_1     |  |    `-._   `._    /     _.-'    |     PID: 1
redis_1     |   `-._    `-._  `-./  _.-'    _.-'
redis_1     |  |`-._`-._    `-.__.-'    _.-'_.-'|
redis_1     |  |    `-._`-._        _.-'_.-'    |           http://redis.io
redis_1     |   `-._    `-._`-.__.-'_.-'    _.-'
redis_1     |  |`-._`-._    `-.__.-'    _.-'_.-'|
redis_1     |  |    `-._`-._        _.-'_.-'    |
redis_1     |   `-._    `-._`-.__.-'_.-'    _.-'
redis_1     |       `-._    `-.__.-'    _.-'
redis_1     |           `-._        _.-'
redis_1     |               `-.__.-'
redis_1     |
redis_1     | 1:M 07 Feb 15:41:55.446 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
redis_1     | 1:M 07 Feb 15:41:55.446 # Server started, Redis version 3.2.12
redis_1     | 1:M 07 Feb 15:41:55.446 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never &gt; /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
redis_1     | 1:M 07 Feb 15:41:55.446 * DB loaded from disk: 0.000 seconds
redis_1     | 1:M 07 Feb 15:41:55.446 * The server is now ready to accept connections on port 6379
superset_1  | npm WARN prepare removing existing node_modules/ before installation
superset_1  | npm ERR! path /home/superset/superset/assets/node_modules/core-js/fn/array/virtual
superset_1  | npm ERR! code ENOTEMPTY
superset_1  | npm ERR! errno -39
superset_1  | npm ERR! syscall rmdir
superset_1  | npm ERR! ENOTEMPTY: directory not empty, rmdir '/home/superset/superset/assets/node_modules/core-js/fn/array/virtual'
superset_1  |
superset_1  | npm ERR! A complete log of this run can be found in:
superset_1  | npm ERR!     /home/superset/.npm/_logs/2019-02-07T15_47_06_436Z-debug.log
postgres_1  | 2019-02-07 15:47:06.572 UTC [36] LOG:  unexpected EOF on client connection with an open transaction
superset_1  | npm WARN prepare removing existing node_modules/ before installation
superset_1  | npm ERR! path /home/superset/superset/assets/node_modules/core-js/fn/array/virtual
superset_1  | npm ERR! code ENOTEMPTY
superset_1  | npm ERR! errno -39
superset_1  | npm ERR! syscall rmdir
superset_1  | npm ERR! ENOTEMPTY: directory not empty, rmdir '/home/superset/superset/assets/node_modules/core-js/fn/array/virtual'
superset_1  |
superset_1  | npm ERR! A complete log of this run can be found in:
superset_1  | npm ERR!     /home/superset/.npm/_logs/2019-02-07T15_50_51_169Z-debug.log
postgres_1  | 2019-02-07 15:50:51.297 UTC [44] LOG:  unexpected EOF on client connection with an open transaction
superset_superset_1 exited with code 217
superset_1  | npm WARN prepare removing existing node_modules/ before installation
superset_1  | npm ERR! path /home/superset/superset/assets/node_modules/core-js/fn/array/virtual
superset_1  | npm ERR! code ENOTEMPTY
superset_1  | npm ERR! errno -39
superset_1  | npm ERR! syscall rmdir
superset_1  | npm ERR! ENOTEMPTY: directory not empty, rmdir '/home/superset/superset/assets/node_modules/core-js/fn/array/virtual'
superset_1  |
superset_1  | npm ERR! A complete log of this run can be found in:
superset_1  | npm ERR!     /home/superset/.npm/_logs/2019-02-07T15_54_35_445Z-debug.log
postgres_1  | 2019-02-07 15:54:35.595 UTC [52] LOG:  unexpected EOF on client connection with an open transaction
superset_superset_1 exited with code 217
superset_1  | npm WARN prepare removing existing node_modules/ before installation
superset_1  | npm ERR! path /home/superset/superset/assets/node_modules/core-js/fn/array/virtual
superset_1  | npm ERR! code ENOTEMPTY
superset_1  | npm ERR! errno -39
superset_1  | npm ERR! syscall rmdir
superset_1  | npm ERR! ENOTEMPTY: directory not empty, rmdir '/home/superset/superset/assets/node_modules/core-js/fn/array/virtual'
superset_1  |
superset_1  | npm ERR! A complete log of this run can be found in:
superset_1  | npm ERR!     /home/superset/.npm/_logs/2019-02-07T15_58_19_852Z-debug.log
postgres_1  | 2019-02-07 15:58:19.963 UTC [61] LOG:  unexpected EOF on client connection with an open transaction
superset_superset_1 exited with code 217
superset_1  | npm WARN prepare removing existing node_modules/ before installation
superset_1  | npm ERR! path /home/superset/superset/assets/node_modules/core-js/fn/array/virtual
superset_1  | npm ERR! code ENOTEMPTY
superset_1  | npm ERR! errno -39
superset_1  | npm ERR! syscall rmdir
superset_1  | npm ERR! ENOTEMPTY: directory not empty, rmdir '/home/superset/superset/assets/node_modules/core-js/fn/array/virtual'
superset_1  |
superset_1  | npm ERR! A complete log of this run can be found in:
superset_1  | npm ERR!     /home/superset/.npm/_logs/2019-02-07T16_02_04_350Z-debug.log
postgres_1  | 2019-02-07 16:02:04.473 UTC [69] LOG:  unexpected EOF on client connection with an open transaction
superset_superset_1 exited with code 217
superset_1  | npm WARN prepare removing existing node_modules/ before installation
superset_1  | npm ERR! path /home/superset/superset/assets/node_modules/core-js/fn/array/virtual
superset_1  | npm ERR! code ENOTEMPTY
superset_1  | npm ERR! errno -39
superset_1  | npm ERR! syscall rmdir
superset_1  | npm ERR! ENOTEMPTY: directory not empty, rmdir '/home/superset/superset/assets/node_modules/core-js/fn/array/virtual'
superset_1  |
superset_1  | npm ERR! A complete log of this run can be found in:
superset_1  | npm ERR!     /home/superset/.npm/_logs/2019-02-07T16_05_48_096Z-debug.log
postgres_1  | 2019-02-07 16:05:48.222 UTC [78] LOG:  unexpected EOF on client connection with an open transaction
superset_superset_1 exited with code 217
superset_1  | npm WARN prepare removing existing node_modules/ before installation
</code></pre></div>
<p dir="auto">Note that the execution started on  2019-02-07 15:41:55.069 and the "screenshot" was taken on  2019-02-07 16:05:48.222 without any result. It seems that it is always repeating the:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="superset_1  | npm WARN prepare removing existing node_modules/ before installation
superset_1  | npm ERR! path /home/superset/superset/assets/node_modules/core-js/fn/array/virtual
superset_1  | npm ERR! code ENOTEMPTY
superset_1  | npm ERR! errno -39
superset_1  | npm ERR! syscall rmdir
superset_1  | npm ERR! ENOTEMPTY: directory not empty, rmdir '/home/superset/superset/assets/node_modules/core-js/fn/array/virtual'
superset_1  |
superset_1  | npm ERR! A complete log of this run can be found in:
superset_1  | npm ERR!     /home/superset/.npm/_logs/2019-02-07T16_05_48_096Z-debug.log
postgres_1  | 2019-02-07 16:05:48.222 UTC [78] LOG:  unexpected EOF on client connection with an open transaction
superset_superset_1 exited with code 217"><pre class="notranslate"><code class="notranslate">superset_1  | npm WARN prepare removing existing node_modules/ before installation
superset_1  | npm ERR! path /home/superset/superset/assets/node_modules/core-js/fn/array/virtual
superset_1  | npm ERR! code ENOTEMPTY
superset_1  | npm ERR! errno -39
superset_1  | npm ERR! syscall rmdir
superset_1  | npm ERR! ENOTEMPTY: directory not empty, rmdir '/home/superset/superset/assets/node_modules/core-js/fn/array/virtual'
superset_1  |
superset_1  | npm ERR! A complete log of this run can be found in:
superset_1  | npm ERR!     /home/superset/.npm/_logs/2019-02-07T16_05_48_096Z-debug.log
postgres_1  | 2019-02-07 16:05:48.222 UTC [78] LOG:  unexpected EOF on client connection with an open transaction
superset_superset_1 exited with code 217
</code></pre></div>
<h3 dir="auto">Steps to reproduce</h3>
<p dir="auto">I followed these steps: <a href="https://github.com/apache/incubator-superset/blob/master/docs/installation.rst">https://github.com/apache/incubator-superset/blob/master/docs/installation.rst</a><br>
based on that commit <a href="https://github.com/apache/incubator-superset/commit/823555e07db883c7d59ea4e867cca3efb784c721">823555e</a></p>