Answers.Body: <p>This is a bash script (#!/bin/bash) and you try to run python in it.
  You should save your python script (e.g. in &quot;myscript.py&quot;) and call it within your bash script.
  Maybe first try to see if you are able to run it in your terminal and then run it on your server with SLURM.</p>
  <p>EDIT1: Also I don't think you need 3 nodes to run such script and this will probably not be even possible here. I think you need some specific modules to parallelize a python script with MPI (see <a href="https://stackoverflow.com/questions/25792357/how-to-distribute-a-python-script-across-multiple-nodes">https://stackoverflow.com/questions/25792357/how-to-distribute-a-python-script-across-multiple-nodes</a>), but again I think this would be an overkill. Try first with <code>#SBATCH -c 1</code> instead.</p>
  <p>Edit2: Finally it might be possible to run jupyter-notebook on a node allocated by SLRUM instead of putting this into a python script by using <code>jupyter-notebook --no-browser --port=${port} --ip=${node}</code> command in your bash slurm script. Maybe you'll find this doc interesting: <a href="https://docs.ycrc.yale.edu/clusters-at-yale/guides/jupyter/" rel="nofollow noreferrer">https://docs.ycrc.yale.edu/clusters-at-yale/guides/jupyter/</a> .</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-24T13:01:42.673Z
Answers.Text: thanks I am running it as python script after loading jupyter and python module but still shows an error? I am going to update the script and the error.
Answers.UserDisplayName: bioinfonext
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-24T13:08:18.523Z
Answers.Text: `%matplotlib inline` is a jupyter notebook command. Try to comment this line.
Answers.UserDisplayName: jeremy
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-24T13:10:13.063Z
Answers.Text: Again error. it is updated now.
Answers.UserDisplayName: bioinfonext
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-24T13:19:06.577Z
Answers.Text: Those are again some **jupyter notebook** writings to use bash commands within a notebook. You should try with `subprocess` module instead or build a bash script that does those steps and then call your python script.
  I think you should think by yourself now and googling until you are really stuck, this is a good training to write real python script ;)
Answers.UserDisplayName: jeremy
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-24T13:24:10.537Z
Answers.Text: Hi @bioinfonext please don't forget to upvote this post and if you feel it answered your question to mark as 'resolved' if resolved.
Answers.UserDisplayName: M__
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-24T13:29:16.483Z
Answers.Text: You look to be running Jupyter notebook interactively and if run from PC/laptop/Mac via a jupyter-notebook command it should connect interactively. Pointing this onto your server I'm not certain however. Normally jupyter- notebook is run on one of the big cloud HPC platforms such as AWS or GCS.
Answers.UserDisplayName: M__
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-24T13:34:00.593Z
Answers.Text: I think now it is a bit confusing because the post has been edited a lot. But I think @bioinfonext wrote a jupyter notebook to plot his/here data and then wanted to do the same task but by running it on a cluster using SLURM job scheduling manager.
Answers.UserDisplayName: jeremy
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-24T14:44:12.347Z
Answers.Text: should I use below line of code instead "a = !unzip $qza" ;  import subprocess
  
  
  a = subprocess.Popen(["unzip'', qza])
Answers.UserDisplayName: bioinfonext
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-24T14:57:01.203Z
Answers.Text: unzip is your command, qza is a python variable, if you look for "give variable to subprocess call" you find: https://stackoverflow.com/questions/4795190/passing-variables-to-a-subprocess-call
Answers.UserDisplayName: jeremy
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-25T12:28:18.370Z
Answers.Text: Thank you so much. Now I am trying to run this on server terminal after loading the modules and with some changes, but still getting error? I have updated the code and error.
Answers.UserDisplayName: bioinfonext
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-24T12:47:05.750Z
Answers.LastActivityDate: 2020-09-24T14:45:43.693Z
Answers.LastEditDate: 2020-09-24T14:45:43.693Z
Answers.UserDisplayName: jeremy
Answers.Body: <p>I'll make a contribution and keep editing it rather than use 'comments', the error is about</p>
  <pre><code>import numpy as np
  </code></pre>
  <p>At the moment this isn't about Jupyter, its about getting Python working on your server. Is that a good assessment? With Jupyter notebook you write a little bit of code, check it, write a bit more. That's how it is designed to operate. The other way in is to use Python in your shebang statement otherwise it will default to <code>sh</code> or <code>bash</code>.</p>
  <p>The code you supplied doesn't explicitly use <code>numpy</code>. Pandas does use numpy and <code>np</code> is the standard shorthand, <code>Multiarray</code> is probably something pandas would use. Did you get this code working on your local machine?</p>
  <p>I am wondering whether the numpy module on your server is up to date? Did you creat a specific Python environment, use <code>pip</code> or <code>conda</code>? Checking the Python version on your server and whether <code>numpy</code> needs updating is perhaps where I might start.</p>
  <p>It is not pretty getting a HPC to run with your stuff and can take real work.</p>
  <hr />
  <p>I read your new post, I would simply reinstall Python from <code>Anaconda3</code> and run the script for now from shell pointing the shebang into Anaconda3 (I can't remember if you need to do this, I can check if needed). This a more professional way of using a HPC than using the resident Python. In good old days of Perl, when everything was nice and simple, <code>perlbrew</code> was essentially the Perl equivalent and no-one would use the system Perl.</p>
  <p>The reason why I'm saying  this is I get this feeling it will be an endless cycle of import errors. Anaconda3 will give you control of your Python system without admin privelidge.</p>
  <p>Just to explain if you are not calling <code>np</code> explicitly (and looking at your code I don't think you are) then explicitly stating <code>import numpy</code> isn't needed. The fact you wrote it and it then jumped onto a different error is a bit worrying. I would just use your own version of Python. In Jupyter I would guess, but don't know they use their own Python ... but from a data science point of view Jupyter is the front end to AWS and GCS because those guys want some control of what you are doing, not running riot. In DNN jupyter is big. If you want an absolute diagnosis then Stackoverflow might help, but I think you'll just another bug and another etc... One thing is that <code>pandas</code> changes alot and the version you are using might be ahead of the curve, thats my guess</p>
  <p>Final point is if you are just plotting the <code>ggplot2</code> is really good, but that is <code>R</code>. <code>seaborne</code> is really good, <code>matplotlib</code> but hmmmm, but nothing is as clear or explicit as <code>ggplot2</code>. Python is good at lots of things, pandas is fabulous, but plotting isn't on par with <code>ggplot2</code></p>
  <hr />
  <p>What is 100% clear is that you are loading the HPC version of Python and this is out of date (3.4) should be 3.7 (or even 3.8?). The thing I don't understand is why is it loads Python 2 modules?? This is a complete mystery to me, I thought Python 2 and Python 3 were incompatible, this is suggesting backwards compatibility or else the reason it is falling over. I have no idea, but I wouldn't use the resident Python.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-25T15:14:11.980Z
Answers.Text: script and error updated after importing numpy.
Answers.UserDisplayName: bioinfonext
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-25T15:42:24.743Z
Answers.Text: @bioinfonext commented below
Answers.UserDisplayName: M__
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-26T06:10:12.133Z
Answers.Text: Yes, thanks, this is all very confusing, R is much easier to adopt for potting.But I could not able to find any good R code for rarefaction plot that can plot it based on metadata column.
Answers.UserDisplayName: bioinfonext
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-26T12:09:20.237Z
Answers.Text: @bioinfonext answered above
Answers.UserDisplayName: M__
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-26T12:09:20.737Z
Answers.Text: @bioinfonext answered above
Answers.UserDisplayName: M__
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-25T14:36:15.357Z
Answers.LastActivityDate: 2020-09-26T12:13:51.617Z
Answers.LastEditDate: 2020-09-26T12:13:51.617Z
Answers.UserDisplayName: M__
Body: <p>Now jupyter installed on the server and I am using below code to plot the rarefaction plot but I am still getting some error. could you please suggest how I can get rid of it?</p>
  <pre><code>    #Run it in jupyter lab or notebook
  
  
  import seaborn as sns
  import matplotlib.pyplot as plt
  import pandas as pd
  import numpy as np
  import os
  
  #%matplotlib inline 
  
  qza = 'alpha-rarefaction-20000.qzv' # path to rarefaction file from qiime2
  os.system(&quot;unzip qza&quot;)
  
  digest = a[1].split('/')[0].replace('  inflating: ','')
  inf = digest + '/data/observed_otus.csv'
  otus = pd.read_csv(inf,index_col=0, sep=',')
  os.system('rm -r $digest')
  
  cols = [col for col in otus.columns if 'iter' not in col]
  mean,data = otus[cols],pd.DataFrame(columns=cols)
  depths = [col.split('_')[0] for col in otus.columns if 'depth' in col]
  otus = otus.drop(cols,axis=1)
  otus.columns = depths
  for depth in depths:
      mean['ASV'] = otus[depth].mean(axis=1)
      mean['depth']= depth.split('-')[-1]
      data = pd.concat([data,mean])
  # here provide colors for each item that will be plotted
  pal={'Irrigated':'brown','Rainfed':'red'} 
  fig,ax = plt.subplots(figsize=(2.5,2),dpi=600,tight_layout=True)
  sns.set(style='ticks',rc={&quot;lines.linewidth&quot;:0.7,&quot;axes.linewidth&quot;:0.5})
  # use your column name to plot here
  sns.lineplot(x='depth',y='ASV',data=data,palette=pal,hue='Season',sort=False,err_style='bars',\
               dashes=True,style='Season',ci=67)
  ax.set_xlabel('Sequencing depth',fontsize=8)
  ax.set_ylabel('Observed ASVs',fontsize=8)
  ax.tick_params(axis='x', labelrotation=90)
  ax.tick_params(axis='both',which='major',length=2,pad=0.5,labelsize=6)
  handles, labels = ax.get_legend_handles_labels()
  ax.legend(handles=handles[1:],labels=labels[1:],fontsize=5,frameon=False,numpoints=4,borderaxespad=0,handletextpad=0.2,loc=2,)
  plt.savefig('Infected_observed_ASVs.png', bbox_inches='tight')
  </code></pre>
  <p>Error;</p>
  <pre><code> $ python3 myscript.rarefaction.py
  Traceback (most recent call last):
    File &quot;myscript.rarefaction.py&quot;, line 4, in &lt;module&gt;
      import seaborn as sns
    File &quot;/opt/apps/seaborn/0.11.0/python3-3.7.4/lib/python3.7/site-packages/seaborn/__init__.py&quot;, line 2, in &lt;module&gt;
      from .rcmod import *  # noqa: F401,F403
    File &quot;/opt/apps/seaborn/0.11.0/python3-3.7.4/lib/python3.7/site-packages/seaborn/rcmod.py&quot;, line 7, in &lt;module&gt;
      from . import palettes
    File &quot;/opt/apps/seaborn/0.11.0/python3-3.7.4/lib/python3.7/site-packages/seaborn/palettes.py&quot;, line 9, in &lt;module&gt;
      from .utils import desaturate, get_color_cycle
    File &quot;/opt/apps/seaborn/0.11.0/python3-3.7.4/lib/python3.7/site-packages/seaborn/utils.py&quot;, line 11, in &lt;module&gt;
      import pandas as pd
    File &quot;/opt/apps/seaborn/0.11.0/python3-3.7.4/lib/python3.7/site-packages/pandas/__init__.py&quot;, line 11, in &lt;module&gt;
      __import__(dependency)
    File &quot;&lt;frozen importlib._bootstrap&gt;&quot;, line 983, in _find_and_load
    File &quot;&lt;frozen importlib._bootstrap&gt;&quot;, line 967, in _find_and_load_unlocked
    File &quot;&lt;frozen importlib._bootstrap&gt;&quot;, line 668, in _load_unlocked
    File &quot;&lt;frozen importlib._bootstrap&gt;&quot;, line 638, in _load_backward_compatible
    File &quot;/opt/gridware/depots/54e7fb3c/el7/pkg/libs/matplotlib_python34/1.5.1/gcc-4.8.5+python3-3.4.3+numpy_python34-1.11.3/python/lib/python3.4/site-packages/pytz-2019.1-py3.4.egg/pytz/__init__.py&quot;, line 1101, in &lt;module&gt;
    File &quot;/opt/gridware/depots/54e7fb3c/el7/pkg/libs/matplotlib_python34/1.5.1/gcc-4.8.5+python3-3.4.3+numpy_python34-1.11.3/python/lib/python3.4/site-packages/pytz-2019.1-py3.4.egg/pytz/lazy.py&quot;, line 101, in _lazy
    File &quot;/opt/gridware/depots/54e7fb3c/el7/pkg/libs/matplotlib_python34/1.5.1/gcc-4.8.5+python3-3.4.3+numpy_python34-1.11.3/python/lib/python3.4/site-packages/pytz-2019.1-py3.4.egg/pytz/__init__.py&quot;, line 1098, in &lt;genexpr&gt;
    File &quot;/opt/gridware/depots/54e7fb3c/el7/pkg/libs/matplotlib_python34/1.5.1/gcc-4.8.5+python3-3.4.3+numpy_python34-1.11.3/python/lib/python3.4/site-packages/pytz-2019.1-py3.4.egg/pytz/__init__.py&quot;, line 114, in resource_exists
    File &quot;/opt/gridware/depots/54e7fb3c/el7/pkg/libs/matplotlib_python34/1.5.1/gcc-4.8.5+python3-3.4.3+numpy_python34-1.11.3/python/lib/python3.4/site-packages/pytz-2019.1-py3.4.egg/pytz/__init__.py&quot;, line 102, in open_resource
    File &quot;&lt;frozen importlib._bootstrap&gt;&quot;, line 983, in _find_and_load
    File &quot;&lt;frozen importlib._bootstrap&gt;&quot;, line 967, in _find_and_load_unlocked
    File &quot;&lt;frozen importlib._bootstrap&gt;&quot;, line 668, in _load_unlocked
    File &quot;&lt;frozen importlib._bootstrap&gt;&quot;, line 638, in _load_backward_compatible
    File &quot;/opt/gridware/depots/54e7fb3c/el7/pkg/apps/setuptools/24.0.1/python-2.7.8/python/lib/python2.7/site-packages/setuptools-24.0.1-py2.7.egg/pkg_resources/__init__.py&quot;, line 2927, in &lt;module&gt;
    File &quot;/opt/gridware/depots/54e7fb3c/el7/pkg/apps/setuptools/24.0.1/python-2.7.8/python/lib/python2.7/site-packages/setuptools-24.0.1-py2.7.egg/pkg_resources/__init__.py&quot;, line 2913, in _call_aside
    File &quot;/opt/gridware/depots/54e7fb3c/el7/pkg/apps/setuptools/24.0.1/python-2.7.8/python/lib/python2.7/site-packages/setuptools-24.0.1-py2.7.egg/pkg_resources/__init__.py&quot;, line 2952, in _initialize_master_working_set
    File &quot;/opt/gridware/depots/54e7fb3c/el7/pkg/apps/setuptools/24.0.1/python-2.7.8/python/lib/python2.7/site-packages/setuptools-24.0.1-py2.7.egg/pkg_resources/__init__.py&quot;, line 956, in subscribe
    File &quot;/opt/gridware/depots/54e7fb3c/el7/pkg/apps/setuptools/24.0.1/python-2.7.8/python/lib/python2.7/site-packages/setuptools-24.0.1-py2.7.egg/pkg_resources/__init__.py&quot;, line 2952, in &lt;lambda&gt;
    File &quot;/opt/gridware/depots/54e7fb3c/el7/pkg/apps/setuptools/24.0.1/python-2.7.8/python/lib/python2.7/site-packages/setuptools-24.0.1-py2.7.egg/pkg_resources/__init__.py&quot;, line 2515, in activate
    File &quot;/opt/gridware/depots/54e7fb3c/el7/pkg/apps/setuptools/24.0.1/python-2.7.8/python/lib/python2.7/site-packages/setuptools-24.0.1-py2.7.egg/pkg_resources/__init__.py&quot;, line 2097, in declare_namespace
    File &quot;/opt/gridware/depots/54e7fb3c/el7/pkg/apps/setuptools/24.0.1/python-2.7.8/python/lib/python2.7/site-packages/setuptools-24.0.1-py2.7.egg/pkg_resources/__init__.py&quot;, line 2047, in _handle_ns
    File &quot;/opt/gridware/depots/54e7fb3c/el7/pkg/apps/setuptools/24.0.1/python-2.7.8/python/lib/python2.7/site-packages/setuptools-24.0.1-py2.7.egg/pkg_resources/__init__.py&quot;, line 2066, in _rebuild_mod_path
  AttributeError: '_NamespacePath' object has no attribute 'sort'
  
           
       
      
      metadata file ;
      #SampleID       Villages        Region  Treatment       Batch   Season
      PN0086D.1.S1    Kholabari       Mymensingh      T1      batch1  Irrigated
      PN0086D.2.S2    Bogajan Mymensingh      T1      batch1  Irrigated
      PN0086D.3.S3    Bonkua  Mymensingh      T1      batch1  Irrigated
  </code></pre>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-09-28T14:36:00.760Z
Comments.Text: Just ask your cluster admin / manager if it's possible to run notebooks via the queueing system / workload manager, if it is not, you won't get it running anyhow.
Comments.UserDisplayName: Pallie
Tags: <python><bash><hpc><jupyter>
Title: How to run Jupyter script on Slurm HPC
ThreadText: # How to run Jupyter script on Slurm HPC
  
  (Asked by: username_0 on 2020-09-24)
  
  Now jupyter installed on the server and I am using below code to plot the rarefaction plot but I am still getting some error. could you please suggest how I can get rid of it?
  
  ```
      #Run it in jupyter lab or notebook
  
  import seaborn as sns
  import matplotlib.pyplot as plt
  import pandas as pd
  import numpy as np
  import os
  
  #%matplotlib inline 
  
  qza = 'alpha-rarefaction-20000.qzv' # path to rarefaction file from qiime2
  os.system("unzip qza")
  
  digest = a[1].split('/')[0].replace('  inflating: ','')
  inf = digest + '/data/observed_otus.csv'
  otus = pd.read_csv(inf,index_col=0, sep=',')
  os.system('rm -r $digest')
  
  cols = [col for col in otus.columns if 'iter' not in col]
  mean,data = otus[cols],pd.DataFrame(columns=cols)
  depths = [col.split('_')[0] for col in otus.columns if 'depth' in col]
  otus = otus.drop(cols,axis=1)
  otus.columns = depths
  for depth in depths:
      mean['ASV'] = otus[depth].mean(axis=1)
      mean['depth']= depth.split('-')[-1]
      data = pd.concat([data,mean])
  # here provide colors for each item that will be plotted
  pal={'Irrigated':'brown','Rainfed':'red'} 
  fig,ax = plt.subplots(figsize=(2.5,2),dpi=600,tight_layout=True)
  sns.set(style='ticks',rc={"lines.linewidth":0.7,"axes.linewidth":0.5})
  # use your column name to plot here
  sns.lineplot(x='depth',y='ASV',data=data,palette=pal,hue='Season',sort=False,err_style='bars',\
               dashes=True,style='Season',ci=67)
  ax.set_xlabel('Sequencing depth',fontsize=8)
  ax.set_ylabel('Observed ASVs',fontsize=8)
  ax.tick_params(axis='x', labelrotation=90)
  ax.tick_params(axis='both',which='major',length=2,pad=0.5,labelsize=6)
  handles, labels = ax.get_legend_handles_labels()
  ax.legend(handles=handles[1:],labels=labels[1:],fontsize=5,frameon=False,numpoints=4,borderaxespad=0,handletextpad=0.2,loc=2,)
  plt.savefig('Infected_observed_ASVs.png', bbox_inches='tight')
  
  ```
  
  Error;
  
  ```
   $ python3 myscript.rarefaction.py
  Traceback (most recent call last):
    File "myscript.rarefaction.py", line 4, in <module>
      import seaborn as sns
    File "/opt/apps/seaborn/0.11.0/python3-3.7.4/lib/python3.7/site-packages/seaborn/__init__.py", line 2, in <module>
      from .rcmod import *  # noqa: F401,F403
    File "/opt/apps/seaborn/0.11.0/python3-3.7.4/lib/python3.7/site-packages/seaborn/rcmod.py", line 7, in <module>
      from . import palettes
    File "/opt/apps/seaborn/0.11.0/python3-3.7.4/lib/python3.7/site-packages/seaborn/palettes.py", line 9, in <module>
      from .utils import desaturate, get_color_cycle
    File "/opt/apps/seaborn/0.11.0/python3-3.7.4/lib/python3.7/site-packages/seaborn/utils.py", line 11, in <module>
      import pandas as pd
    File "/opt/apps/seaborn/0.11.0/python3-3.7.4/lib/python3.7/site-packages/pandas/__init__.py", line 11, in <module>
      __import__(dependency)
    File "<frozen importlib._bootstrap>", line 983, in _find_and_load
    File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
    File "<frozen importlib._bootstrap>", line 668, in _load_unlocked
    File "<frozen importlib._bootstrap>", line 638, in _load_backward_compatible
    File "/opt/gridware/depots/54e7fb3c/el7/pkg/libs/matplotlib_python34/1.5.1/gcc-4.8.5+python3-3.4.3+numpy_python34-1.11.3/python/lib/python3.4/site-packages/pytz-2019.1-py3.4.egg/pytz/__init__.py", line 1101, in <module>
    File "/opt/gridware/depots/54e7fb3c/el7/pkg/libs/matplotlib_python34/1.5.1/gcc-4.8.5+python3-3.4.3+numpy_python34-1.11.3/python/lib/python3.4/site-packages/pytz-2019.1-py3.4.egg/pytz/lazy.py", line 101, in _lazy
    File "/opt/gridware/depots/54e7fb3c/el7/pkg/libs/matplotlib_python34/1.5.1/gcc-4.8.5+python3-3.4.3+numpy_python34-1.11.3/python/lib/python3.4/site-packages/pytz-2019.1-py3.4.egg/pytz/__init__.py", line 1098, in <genexpr>
    File "/opt/gridware/depots/54e7fb3c/el7/pkg/libs/matplotlib_python34/1.5.1/gcc-4.8.5+python3-3.4.3+numpy_python34-1.11.3/python/lib/python3.4/site-packages/pytz-2019.1-py3.4.egg/pytz/__init__.py", line 114, in resource_exists
    File "/opt/gridware/depots/54e7fb3c/el7/pkg/libs/matplotlib_python34/1.5.1/gcc-4.8.5+python3-3.4.3+numpy_python34-1.11.3/python/lib/python3.4/site-packages/pytz-2019.1-py3.4.egg/pytz/__init__.py", line 102, in open_resource
    File "<frozen importlib._bootstrap>", line 983, in _find_and_load
    File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
    File "<frozen importlib._bootstrap>", line 668, in _load_unlocked
    File "<frozen importlib._bootstrap>", line 638, in _load_backward_compatible
    File "/opt/gridware/depots/54e7fb3c/el7/pkg/apps/setuptools/24.0.1/python-2.7.8/python/lib/python2.7/site-packages/setuptools-24.0.1-py2.7.egg/pkg_resources/__init__.py", line 2927, in <module>
    File "/opt/gridware/depots/54e7fb3c/el7/pkg/apps/setuptools/24.0.1/python-2.7.8/python/lib/python2.7/site-packages/setuptools-24.0.1-py2.7.egg/pkg_resources/__init__.py", line 2913, in _call_aside
    File "/opt/gridware/depots/54e7fb3c/el7/pkg/apps/setuptools/24.0.1/python-2.7.8/python/lib/python2.7/site-packages/setuptools-24.0.1-py2.7.egg/pkg_resources/__init__.py", line 2952, in _initialize_master_working_set
    File "/opt/gridware/depots/54e7fb3c/el7/pkg/apps/setuptools/24.0.1/python-2.7.8/python/lib/python2.7/site-packages/setuptools-24.0.1-py2.7.egg/pkg_resources/__init__.py", line 956, in subscribe
    File "/opt/gridware/depots/54e7fb3c/el7/pkg/apps/setuptools/24.0.1/python-2.7.8/python/lib/python2.7/site-packages/setuptools-24.0.1-py2.7.egg/pkg_resources/__init__.py", line 2952, in <lambda>
    File "/opt/gridware/depots/54e7fb3c/el7/pkg/apps/setuptools/24.0.1/python-2.7.8/python/lib/python2.7/site-packages/setuptools-24.0.1-py2.7.egg/pkg_resources/__init__.py", line 2515, in activate
    File "/opt/gridware/depots/54e7fb3c/el7/pkg/apps/setuptools/24.0.1/python-2.7.8/python/lib/python2.7/site-packages/setuptools-24.0.1-py2.7.egg/pkg_resources/__init__.py", line 2097, in declare_namespace
    File "/opt/gridware/depots/54e7fb3c/el7/pkg/apps/setuptools/24.0.1/python-2.7.8/python/lib/python2.7/site-packages/setuptools-24.0.1-py2.7.egg/pkg_resources/__init__.py", line 2047, in _handle_ns
    File "/opt/gridware/depots/54e7fb3c/el7/pkg/apps/setuptools/24.0.1/python-2.7.8/python/lib/python2.7/site-packages/setuptools-24.0.1-py2.7.egg/pkg_resources/__init__.py", line 2066, in _rebuild_mod_path
  AttributeError: '_NamespacePath' object has no attribute 'sort'
  
           
       
      
      metadata file ;
      #SampleID       Villages        Region  Treatment       Batch   Season
      PN0086D.1.S1    Kholabari       Mymensingh      T1      batch1  Irrigated
      PN0086D.2.S2    Bogajan Mymensingh      T1      batch1  Irrigated
      PN0086D.3.S3    Bonkua  Mymensingh      T1      batch1  Irrigated
  
  ```
  
  **Tags:** python, bash, hpc, jupyter
  
  > **Comments:**
  > **username_1**: Just ask your cluster admin / manager if it's possible to run notebooks via the queueing system / workload manager, if it is not, you won't get it running anyhow.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2020-09-24. Score: 2
  
  This is a bash script (#!/bin/bash) and you try to run python in it.
  You should save your python script (e.g. in "myscript.py") and call it within your bash script.
  Maybe first try to see if you are able to run it in your terminal and then run it on your server with SLURM.
  
  EDIT1: Also I don't think you need 3 nodes to run such script and this will probably not be even possible here. I think you need some specific modules to parallelize a python script with MPI (see <https://stackoverflow.com/questions/25792357/how-to-distribute-a-python-script-across-multiple-nodes>), but again I think this would be an overkill. Try first with `#SBATCH -c 1` instead.
  
  Edit2: Finally it might be possible to run jupyter-notebook on a node allocated by SLRUM instead of putting this into a python script by using `jupyter-notebook --no-browser --port=${port} --ip=${node}` command in your bash slurm script. Maybe you'll find this doc interesting: <https://docs.ycrc.yale.edu/clusters-at-yale/guides/jupyter/> .
  
  > **Comments:**
  > **username_0**: thanks I am running it as python script after loading jupyter and python module but still shows an error? I am going to update the script and the error.  
  > **username_2**: `%matplotlib inline` is a jupyter notebook command. Try to comment this line.  
  > **username_0**: Again error. it is updated now.  
  > **username_2**: Those are again some **jupyter notebook** writings to use bash commands within a notebook. You should try with `subprocess` module instead or build a bash script that does those steps and then call your python script. I think you should think by yourself now and googling until you are really stuck, this is a good training to write real python script ;)  
  > **username_3**: Hi @username_0 please don't forget to upvote this post and if you feel it answered your question to mark as 'resolved' if resolved.  
  > **username_3**: You look to be running Jupyter notebook interactively and if run from PC/laptop/Mac via a jupyter-notebook command it should connect interactively. Pointing this onto your server I'm not certain however. Normally jupyter- notebook is run on one of the big cloud HPC platforms such as AWS or GCS.  
  > **username_2**: I think now it is a bit confusing because the post has been edited a lot. But I think @username_0 wrote a jupyter notebook to plot his/here data and then wanted to do the same task but by running it on a cluster using SLURM job scheduling manager.  
  > **username_0**: should I use below line of code instead "a = !unzip $qza" ;  import subprocess   a = subprocess.Popen(["unzip'', qza])  
  > **username_2**: unzip is your command, qza is a python variable, if you look for "give variable to subprocess call" you find: https://stackoverflow.com/questions/4795190/passing-variables-to-a-subprocess-call  
  > **username_0**: Thank you so much. Now I am trying to run this on server terminal after loading the modules and with some changes, but still getting error? I have updated the code and error.  
  
  ---
  
  ### Answer by username_3 on 2020-09-25. Score: 0
  
  I'll make a contribution and keep editing it rather than use 'comments', the error is about
  
  ```
  import numpy as np
  
  ```
  
  At the moment this isn't about Jupyter, its about getting Python working on your server. Is that a good assessment? With Jupyter notebook you write a little bit of code, check it, write a bit more. That's how it is designed to operate. The other way in is to use Python in your shebang statement otherwise it will default to `sh` or `bash`.
  
  The code you supplied doesn't explicitly use `numpy`. Pandas does use numpy and `np` is the standard shorthand, `Multiarray` is probably something pandas would use. Did you get this code working on your local machine?
  
  I am wondering whether the numpy module on your server is up to date? Did you creat a specific Python environment, use `pip` or `conda`? Checking the Python version on your server and whether `numpy` needs updating is perhaps where I might start.
  
  It is not pretty getting a HPC to run with your stuff and can take real work.
  
  ---
  
  I read your new post, I would simply reinstall Python from `Anaconda3` and run the script for now from shell pointing the shebang into Anaconda3 (I can't remember if you need to do this, I can check if needed). This a more professional way of using a HPC than using the resident Python. In good old days of Perl, when everything was nice and simple, `perlbrew` was essentially the Perl equivalent and no-one would use the system Perl.
  
  The reason why I'm saying this is I get this feeling it will be an endless cycle of import errors. Anaconda3 will give you control of your Python system without admin privelidge.
  
  Just to explain if you are not calling `np` explicitly (and looking at your code I don't think you are) then explicitly stating `import numpy` isn't needed. The fact you wrote it and it then jumped onto a different error is a bit worrying. I would just use your own version of Python. In Jupyter I would guess, but don't know they use their own Python ... but from a data science point of view Jupyter is the front end to AWS and GCS because those guys want some control of what you are doing, not running riot. In DNN jupyter is big. If you want an absolute diagnosis then Stackoverflow might help, but I think you'll just another bug and another etc... One thing is that `pandas` changes alot and the version you are using might be ahead of the curve, thats my guess
  
  Final point is if you are just plotting the `ggplot2` is really good, but that is `R`. `seaborne` is really good, `matplotlib` but hmmmm, but nothing is as clear or explicit as `ggplot2`. Python is good at lots of things, pandas is fabulous, but plotting isn't on par with `ggplot2`
  
  ---
  
  What is 100% clear is that you are loading the HPC version of Python and this is out of date (3.4) should be 3.7 (or even 3.8?). The thing I don't understand is why is it loads Python 2 modules?? This is a complete mystery to me, I thought Python 2 and Python 3 were incompatible, this is suggesting backwards compatibility or else the reason it is falling over. I have no idea, but I wouldn't use the resident Python.
  
  > **Comments:**
  > **username_0**: script and error updated after importing numpy.  
  > **username_3**: @username_0 commented below  
  > **username_0**: Yes, thanks, this is all very confusing, R is much easier to adopt for potting.But I could not able to find any good R code for rarefaction plot that can plot it based on metadata column.  
  > **username_3**: @username_0 answered above  
  > **username_3**: @username_0 answered above  
  
  ---
  
  