Answers.Body: <p>I work a lot with large networks and I agree with what most other posters have said, in that it is hardly useful to explore a network like this visually and Cytoscape - while it is one of best GUIs in general - is pretty bad with large graphs. Here are some alternatives you could try <em>if you want to go down that road</em>:</p>
  <ul>
  <li><a href="https://gephi.org/" rel="nofollow noreferrer">Gephi</a> also probably won't handle that number of nodes/edges well, but it may not crash. Development on it has stagnated though.</li>
  <li><a href="https://graphia.app/" rel="nofollow noreferrer">Graphia</a> I haven't tried this, but it is supposed to be better at handling large graphs. This project is in its infancy, so don't expect too much</li>
  </ul>
  <p>If you care about the analysis of the graph itself, I'd go with <a href="https://networkit.github.io/" rel="nofollow noreferrer">NetworKit</a> or <a href="https://snap.stanford.edu" rel="nofollow noreferrer">SNAP</a> which are much better suited to large graphs.</p>
  <p>If you want to visualize the networks, something that tends to perform well here is:</p>
  <ol>
  <li>Graph embeddings with <a href="https://github.com/xgfs/verse" rel="nofollow noreferrer">VERSE</a></li>
  <li>Dimensionality reduction of the embeddings with <a href="https://umap-learn.readthedocs.io/en/latest/" rel="nofollow noreferrer">UMAP</a></li>
  <li>Plot with <a href="https://datashader.org" rel="nofollow noreferrer">Datashader</a></li>
  </ol>
  <p>This can handle absolutely massive graphs if you have the compute - which you seem to have. Here's one I am just working on (albeit small compared to yours with 30K nodes and 800K edges) - from that exact pipeline. Took ~10 minutes read to render on my desktop:
  <a href="https://i.sstatic.net/fITVg.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/fITVg.png" alt="graph example" /></a></p>
  <p>Finally, as others have mentioned, graph partitioning might be the way to go. I really like <a href="https://www.mapequation.org/code.html" rel="nofollow noreferrer">InfoMap</a>. It partitions the graph into a hierarchical structure that you can explore top down.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-20T20:14:45.613Z
Answers.LastActivityDate: 2020-10-20T20:14:45.613Z
Answers.UserDisplayName: Bastian Schiffthaler
Answers.Body: <p>The first thing you can try is to separate the network from the meta information. Assuming you have access to a *nix machine, you can run:</p>
  
  <pre><code>awk -F"," '{print <span class="math-container">$2,$</span>4}' file.csv &gt; network.csv
  </code></pre>
  
  <p>That will produce a file like this (based on your example data):</p>
  
  <pre><code>test1 test2
  test11 test12
  test12 test22
  </code></pre>
  
  <p>That should radically decrease the file size and will give you the best chance of loading it in Cytoscape. You can then similarly extract the attributes:</p>
  
  <pre><code>awk -F"," '{print <span class="math-container">$2,$</span>1}' file.csv &gt; source.column1.attributes
  awk -F"," '{print <span class="math-container">$2,$</span>3}' file.csv &gt; source.column3.attributes
  awk -F"," '{print <span class="math-container">$4,$</span>5}' file.csv &gt; target.column5.attributes
  awk -F"," '{print <span class="math-container">$4,$</span>6}' file.csv &gt; target.column6.attributes
  awk -F"," '{print <span class="math-container">$4,$</span>7}' file.csv &gt; target.column7.attributes
  awk -F',' '{print <span class="math-container">$2,$</span>4,<span class="math-container">$8}' file.csv &gt; edge.column8.attributes
  awk -F',' '{print $2,$4,$9}' file.csv &gt; edge.column9.attributes
  awk -F',' '{print $2,$4,$10}' file.csv &gt; edge.column10.attributes
  awk -F',' '{print $2,$4,$11}' file.csv &gt; edge.column11.attributes
  awk -F',' '{print $2,$4,$12}' file.csv &gt; edge.column12.attributes
  awk -F',' '{print $2,$4,$13}' file.csv &gt; edge.column13.attributes
  awk -F',' '{print $2,$4,$</span>14}' file.csv &gt; edge.column14.attributes
  </code></pre>
  
  <p>Now, try loading <code>network.csv</code> into Cytoscape and then load each of the node and edge attributes separately. </p>
  
  <hr>
  
  <p>However, a network with 200,000 nodes and 230,000 edges is just too large to be able to manipulate in any useful way in a GUI. It will just look like a horrible hairball. I really urge you to do some preprocessing and pare the network down to something more manageable. I used to work with a human PPI network of ~13000 nodes and ~15000 edges. While I <em>could</em> load that into Cytoscape, that was already far too big to be able to do anything useful with. I could use the data analysis aspects of Cytoscape, but not really the visualization. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-06T17:00:25.677Z
Answers.Text: I've done some testing and ~50,000 nodes with ~50,000 edges is the upper limit of what my 2015 macbook pro can handle. Even then it is painfully slow though. Anything you can do to reduce to graph will help.
Answers.UserDisplayName: conchoecia
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-06T17:03:33.093Z
Answers.Text: @conchoecia well, the OP is using a machine with 500GB, presumably more than your laptop :). So I am guessing their limit will be considerably higher. But even so, even if they do manage to load this behemoth of a network onto Cytoscape, you can't really do anything useful with a GUI on something so large.
Answers.UserDisplayName: terdon
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-07T07:56:15.897Z
Answers.Text: thanks all, that is right. if i can load such enormous network into cytoscape, GUI of it is not useful, but unfortunately, cytoscape displays data as soon as it is loaded.it is better that it load data without display and then we can filter on data and then display network.
Answers.UserDisplayName: user4219
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-06T11:40:32.550Z
Answers.LastActivityDate: 2019-03-06T11:40:32.550Z
Answers.UserDisplayName: terdon
Answers.Body: <p>That's certainly a lot of nodes and edges, however, there is a preference called viewThreshold that should prevent the creation of a view if the number of nodes+edges is larger than the set value.  I've just tested it on a large network I have (9000 nodes and 4,000,000 edges) and it doesn't appear to be working properly.  I'll mark this as a bug.  It does, however, render, but it's not really a useful network on Cytoscape 3.7.  It will actually be usable on Cytoscape 3.8 based on early testing, however.  If you want to make your network available somewhere, I'm happy to try to see why it's not loading.</p>
  
  <p>-- scooter</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-24T15:30:47.187Z
Answers.Text: Actually, after further testing, the viewThreshold property works fine.  I was testing with a session file, not an import.
Answers.UserDisplayName: Scooter Morris
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-26T14:49:14.217Z
Answers.LastActivityDate: 2019-09-26T14:49:14.217Z
Answers.UserDisplayName: Scooter Morris
Body: <p>I installed Cytoscape 3.7.1 with java 1.8.0_191 in windows server 2012. I have a 7.4 GB <code>csv</code> file (about 1,500,000 reords) and when I tried to load it into the Cytoscape it throws an error <code>java.lang.OutOfMemoryErroe:Java heap space.</code></p>
  
  <p>My system has 500GB ram and then I changed <code>Cytoscape.vmoptions</code> file according to <code>-Xms102400M</code> but when I load my file, in task manager, the memory increased until 50GB and then Cytoscape crashed and again it threw the same error: <code>java.lang.OutOfMemoryErroe:Java heap space.</code></p>
  
  <p>My data is like this sample:</p>
  
  <pre><code>45677888,test1,3453453453,test2,3534523453,235412352,3453452345,test3,235423452,test4,1980/09/02,14:13,23523525,test5
  45234288,test11,34234553453,test12,353434534553,23453452352,342422345,test13,23456543452,test14,1980/04/01,14:12,2323423425,test15
  4243424888,test12,3235253453,test22,3533456343453,233534532352,2343452345,test33,23345342,test44,1980/03/4,14:11,23674575,test55
  </code></pre>
  
  <p>My file is in CSV format. col2 is source node, col1 and col3 are attributes for source node,col 4 is destination node, col5,col6,col7 are attributes for destination node, col8 until end are attributes for edge. I have about 200,000 nodes and 230000 edges.</p>
  
  <p>How can I read this file in Cytoscape?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-03-06T09:00:34.540Z
Comments.Text: How much RAM does your computer have? And how big is this network? 7.4G csv? Can you show us a few lines? Does it have a lot of extra data or is it just a list of edges?
Comments.UserDisplayName: terdon
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-03-06T10:40:15.270Z
Comments.Text: thank you for your response. My answers to your questions add to my post.
Comments.UserDisplayName: user4219
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-03-06T10:45:02.287Z
Comments.Text: Please don't post images of text! Just copy the text and paste it directly into your question. Use the [formatting tools](https://bioinformatics.stackexchange.com/help/formatting) to format it as code. And could you explain what format this is? Where's the node? Where's the edge? How many edges and nodes do you have? And, more importantly, why do you want to load such an enormous network into a vizualization tool like cytoscape? You will probably not be able to do anything useful with such a large file anyway. Can't you limit the size somehow?
Comments.UserDisplayName: terdon
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-03-06T11:10:00.677Z
Comments.Text: this data is a connected network and its important for me that I can see it as much as possible on one page.
Comments.UserDisplayName: user4219
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-03-06T11:12:24.420Z
Comments.Text: You won't be able to see anything useful in a GUI with a network of this size. Even if you do manage to load it.
Comments.UserDisplayName: terdon
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-03-06T11:47:50.257Z
Comments.Text: if i first detect community in my network and assign a node for each community and then by click on each community, i can see all nodes of that community. for this, what solution do you recomment?
Comments.UserDisplayName: user4219
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-03-06T11:51:39.027Z
Comments.Text: On a network of 200,000 nodes? Even if cytoscape manages to somehow deal with something that huge, I really doubt you will be able to visualize it in any useful way. f you're looking for communities, why don't you first extract the communities into subgraphs and load those into cytoscape? Alternatively, can you isolate only the connected component of the network? That should also reduce the size. 23e4 edges for 20e4 nodes is a _very_ sparse network. You should be able to separate that into smaller, connected sub networks. There's no point in looking at the whole thing if it isn't connected.
Comments.UserDisplayName: terdon
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-03-06T12:37:54.080Z
Comments.Text: thanks for your time. My goal is to review a connected network with 200,000 nodes in cytoscape. To reach this goal and for the greate of the network, I decide first detect community and show each community as a node that edges between this nodes is edges between communities. Then by click on each node, I can see relation between nodes in a community. Can I do this solution by cytoscape?
Comments.UserDisplayName: user4219
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-03-06T12:40:08.610Z
Comments.Text: I really don't think you will be able to do anything useful in cytoscape with such a huge network. As you can see, it can't even _read_ it. Imagine how hard it will be for cytoscape to manipulate it! Why don't you just use the communities as nodes? Is that an option for you?
Comments.UserDisplayName: terdon
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-03-06T12:48:49.383Z
Comments.Text: what tool do you recommend? i can not just use the communities as nodes. relation between nodes in communities are important for me.
Comments.UserDisplayName: user4219
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-03-06T12:52:55.633Z
Comments.Text: I don't think _any_ graphical tool will be able to handle this amount of data. I would instead try two things: i) make a different network for each community and use that to analyze the relationships of the nodes within the community ii) make another network where the communities are the nodes and use that to analyze the relationship between the communities.
Comments.UserDisplayName: terdon
Tags: <networks>
Title: Error while loading a big file with Cytoscape
ThreadText: # Error while loading a big file with Cytoscape
  
  (Asked by: username_0 on 2019-03-06)
  
  I installed Cytoscape 3.7.1 with java 1.8.0\_191 in windows server 2012. I have a 7.4 GB `csv` file (about 1,500,000 reords) and when I tried to load it into the Cytoscape it throws an error `java.lang.OutOfMemoryErroe:Java heap space.`
  
  My system has 500GB ram and then I changed `Cytoscape.vmoptions` file according to `-Xms102400M` but when I load my file, in task manager, the memory increased until 50GB and then Cytoscape crashed and again it threw the same error: `java.lang.OutOfMemoryErroe:Java heap space.`
  
  My data is like this sample:
  
  ```
  45677888,test1,3453453453,test2,3534523453,235412352,3453452345,test3,235423452,test4,1980/09/02,14:13,23523525,test5
  45234288,test11,34234553453,test12,353434534553,23453452352,342422345,test13,23456543452,test14,1980/04/01,14:12,2323423425,test15
  4243424888,test12,3235253453,test22,3533456343453,233534532352,2343452345,test33,23345342,test44,1980/03/4,14:11,23674575,test55
  
  ```
  
  My file is in CSV format. col2 is source node, col1 and col3 are attributes for source node,col 4 is destination node, col5,col6,col7 are attributes for destination node, col8 until end are attributes for edge. I have about 200,000 nodes and 230000 edges.
  
  How can I read this file in Cytoscape?
  
  **Tags:** networks
  
  > **Comments:**
  > **username_1**: How much RAM does your computer have? And how big is this network? 7.4G csv? Can you show us a few lines? Does it have a lot of extra data or is it just a list of edges?  
  > **username_0**: thank you for your response. My answers to your questions add to my post.  
  > **username_1**: Please don't post images of text! Just copy the text and paste it directly into your question. Use the [formatting tools](https://bioinformatics.stackexchange.com/help/formatting) to format it as code. And could you explain what format this is? Where's the node? Where's the edge? How many edges and nodes do you have? And, more importantly, why do you want to load such an enormous network into a vizualization tool like cytoscape? You will probably not be able to do anything useful with such a large file anyway. Can't you limit the size somehow?  
  > **username_0**: this data is a connected network and its important for me that I can see it as much as possible on one page.  
  > **username_1**: You won't be able to see anything useful in a GUI with a network of this size. Even if you do manage to load it.  
  > **username_0**: if i first detect community in my network and assign a node for each community and then by click on each community, i can see all nodes of that community. for this, what solution do you recomment?  
  > **username_1**: On a network of 200,000 nodes? Even if cytoscape manages to somehow deal with something that huge, I really doubt you will be able to visualize it in any useful way. f you're looking for communities, why don't you first extract the communities into subgraphs and load those into cytoscape? Alternatively, can you isolate only the connected component of the network? That should also reduce the size. 23e4 edges for 20e4 nodes is a _very_ sparse network. You should be able to separate that into smaller, connected sub networks. There's no point in looking at the whole thing if it isn't connected.  
  > **username_0**: thanks for your time. My goal is to review a connected network with 200,000 nodes in cytoscape. To reach this goal and for the greate of the network, I decide first detect community and show each community as a node that edges between this nodes is edges between communities. Then by click on each node, I can see relation between nodes in a community. Can I do this solution by cytoscape?  
  > **username_1**: I really don't think you will be able to do anything useful in cytoscape with such a huge network. As you can see, it can't even _read_ it. Imagine how hard it will be for cytoscape to manipulate it! Why don't you just use the communities as nodes? Is that an option for you?  
  > **username_0**: what tool do you recommend? i can not just use the communities as nodes. relation between nodes in communities are important for me.  
  > **username_1**: I don't think _any_ graphical tool will be able to handle this amount of data. I would instead try two things: i) make a different network for each community and use that to analyze the relationships of the nodes within the community ii) make another network where the communities are the nodes and use that to analyze the relationship between the communities.  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_2 on 2020-10-20. Score: 1
  
  I work a lot with large networks and I agree with what most other posters have said, in that it is hardly useful to explore a network like this visually and Cytoscape - while it is one of best GUIs in general - is pretty bad with large graphs. Here are some alternatives you could try *if you want to go down that road*:
  
  * [Gephi](https://gephi.org/) also probably won't handle that number of nodes/edges well, but it may not crash. Development on it has stagnated though.
  * [Graphia](https://graphia.app/) I haven't tried this, but it is supposed to be better at handling large graphs. This project is in its infancy, so don't expect too much
  
  If you care about the analysis of the graph itself, I'd go with [NetworKit](https://networkit.github.io/) or [SNAP](https://snap.stanford.edu) which are much better suited to large graphs.
  
  If you want to visualize the networks, something that tends to perform well here is:
  
  1. Graph embeddings with [VERSE](https://github.com/xgfs/verse)
  2. Dimensionality reduction of the embeddings with [UMAP](https://umap-learn.readthedocs.io/en/latest/)
  3. Plot with [Datashader](https://datashader.org)
  
  This can handle absolutely massive graphs if you have the compute - which you seem to have. Here's one I am just working on (albeit small compared to yours with 30K nodes and 800K edges) - from that exact pipeline. Took ~10 minutes read to render on my desktop:
  [![graph example](https://i.sstatic.net/fITVg.png)](https://i.sstatic.net/fITVg.png)
  
  Finally, as others have mentioned, graph partitioning might be the way to go. I really like [InfoMap](https://www.mapequation.org/code.html). It partitions the graph into a hierarchical structure that you can explore top down.
  
  
  ---
  
  ### Answer by username_1 on 2019-03-06. Score: 1
  
  The first thing you can try is to separate the network from the meta information. Assuming you have access to a \*nix machine, you can run:
  
  ```
  awk -F"," '{print $2,$4}' file.csv > network.csv
  
  ```
  
  That will produce a file like this (based on your example data):
  
  ```
  test1 test2
  test11 test12
  test12 test22
  
  ```
  
  That should radically decrease the file size and will give you the best chance of loading it in Cytoscape. You can then similarly extract the attributes:
  
  ```
  awk -F"," '{print $2,$1}' file.csv > source.column1.attributes
  awk -F"," '{print $2,$3}' file.csv > source.column3.attributes
  awk -F"," '{print $4,$5}' file.csv > target.column5.attributes
  awk -F"," '{print $4,$6}' file.csv > target.column6.attributes
  awk -F"," '{print $4,$7}' file.csv > target.column7.attributes
  awk -F',' '{print $2,$4,$8}' file.csv > edge.column8.attributes
  awk -F',' '{print $2,$4,$9}' file.csv > edge.column9.attributes
  awk -F',' '{print $2,$4,$10}' file.csv > edge.column10.attributes
  awk -F',' '{print $2,$4,$11}' file.csv > edge.column11.attributes
  awk -F',' '{print $2,$4,$12}' file.csv > edge.column12.attributes
  awk -F',' '{print $2,$4,$13}' file.csv > edge.column13.attributes
  awk -F',' '{print $2,$4,$14}' file.csv > edge.column14.attributes
  
  ```
  
  Now, try loading `network.csv` into Cytoscape and then load each of the node and edge attributes separately.
  
  ---
  
  However, a network with 200,000 nodes and 230,000 edges is just too large to be able to manipulate in any useful way in a GUI. It will just look like a horrible hairball. I really urge you to do some preprocessing and pare the network down to something more manageable. I used to work with a human PPI network of ~13000 nodes and ~15000 edges. While I *could* load that into Cytoscape, that was already far too big to be able to do anything useful with. I could use the data analysis aspects of Cytoscape, but not really the visualization.
  
  > **Comments:**
  > **username_3**: I've done some testing and ~50,000 nodes with ~50,000 edges is the upper limit of what my 2015 macbook pro can handle. Even then it is painfully slow though. Anything you can do to reduce to graph will help.  
  > **username_1**: @username_3 well, the OP is using a machine with 500GB, presumably more than your laptop :). So I am guessing their limit will be considerably higher. But even so, even if they do manage to load this behemoth of a network onto Cytoscape, you can't really do anything useful with a GUI on something so large.  
  > **username_0**: thanks all, that is right. if i can load such enormous network into cytoscape, GUI of it is not useful, but unfortunately, cytoscape displays data as soon as it is loaded.it is better that it load data without display and then we can filter on data and then display network.  
  
  ---
  
  ### Answer by username_4 on 2019-09-26. Score: 0
  
  That's certainly a lot of nodes and edges, however, there is a preference called viewThreshold that should prevent the creation of a view if the number of nodes+edges is larger than the set value. I've just tested it on a large network I have (9000 nodes and 4,000,000 edges) and it doesn't appear to be working properly. I'll mark this as a bug. It does, however, render, but it's not really a useful network on Cytoscape 3.7. It will actually be usable on Cytoscape 3.8 based on early testing, however. If you want to make your network available somewhere, I'm happy to try to see why it's not loading.
  
  -- username_4
  
  > **Comments:**
  > **username_4**: Actually, after further testing, the viewThreshold property works fine.  I was testing with a session file, not an import.  
  
  ---
  
  