Answers.Body: <p>The bug in your loop appears be "<code>&gt; filename</code>" per column, this over-rides the last iteration resulting in a file with a single column, which has been over-written 85999 times. Thus the file output you observe should be the single column present in the last file of your array. If you replaced "<code>&gt; filename</code>" with "<code>&gt;&gt; filename</code>" within the loop you would then append to the file a new column with each iteration. If the loop fell over at some point you could then simply use one of the outputs of <code>wc filename</code> to assess how many columns you had (you might have to transponse the data for wc to work (lots of scripts for this in Perl), although there will almost certainly be an easy solution for counting columns in awk. </p>
  
  <p>Personally, I would use a dataframe for this operation (possibly R, definately Python pandas), because 86000 columns is a lot of columns in any other walk of life, even if it is just post-processing. Generally, if you just want bash, awk is a better solution in this case (I don't code with it, but some here certainly do). However, I do get the idea that <code>paste</code> will work if each column is an identical length.</p>
  
  <p>BTW <code>paste</code> must be lower case, you've got <code>Paste</code> in the first line of your code, but its okay in the second bit of code.</p>
  
  <p>BTW BTW, I did not know shell would take *.txt directly as an array, with Perl/Python you need to "glob" it and assign it as an array/list.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-10T13:50:32.017Z
Answers.LastActivityDate: 2019-07-10T14:06:53.557Z
Answers.LastEditDate: 2019-07-10T14:06:53.557Z
Answers.UserDisplayName: M__
Body: <p>I have multiple files (n=86000) with one column each and I want to combine them all into one file with 86000 columns.</p>
  <p>I tried the following command</p>
  <pre><code>Paste *.txt &gt; mean.txt
  </code></pre>
  <p>And as a for loop</p>
  <pre><code>for i in *.txt, do paste $i &gt; mean.txt; done 
  </code></pre>
  <p>This did not appear to work because there are 86000 to combine, I then divided my files into multiple folders with each folder not containing more than 15000 files but this did not working</p>
  <p>Format of my single file is as follows (It has 16000 lines)</p>
  <pre><code>0.047202943506056
  0.0383549703366668
  0.0159162908608477
  0.0238048058550539
  0.0694728928685859
  0.0389052658410832
  </code></pre>
  <p>What is the bug in the code?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-07-10T10:56:20.590Z
Comments.Text: Thanks. Do you want one long list? In this case the command is 'cat' not 'paste'. Just check you are not  exceeding your disk space.
Comments.UserDisplayName: M__
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-07-10T11:08:45.300Z
Comments.Text: What is the error message you get?
Comments.UserDisplayName: Fabio Marroni
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-07-10T11:10:36.860Z
Comments.Text: Equivalently answered here assuming OP wants to concatenate vertically, not horizontally, though the approach could be modified anyway: https://stackoverflow.com/questions/18695105/how-to-concatenate-huge-number-of-files
Comments.UserDisplayName: Joe Healey
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-07-10T12:08:25.190Z
Comments.Text: Re-reading the question: 16000 lines x 86000 files ... this is not alot. I think the error report is important, or else perhaps it just hangs?
Comments.UserDisplayName: M__
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-07-10T12:32:02.563Z
Comments.Text: @ Fabio Marroni There is no error. When I check the output file it has only one column.
Comments.UserDisplayName: Priya
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-07-10T12:33:25.603Z
Comments.Text: @Michael: No don't want one long list. I want an output file with 86000 columns.
Comments.UserDisplayName: Priya
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-07-10T13:09:42.167Z
Comments.Text: I'm voting to close this question as off-topic because it is not about bioinformatics but file manipulation. In any case it has been answered [here](https://unix.stackexchange.com/questions/205642/combining-large-amount-of-files).
Comments.UserDisplayName: Daniel Standage
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-07-10T13:12:35.877Z
Comments.Text: @Priya The fact that you wanted a file with 86000 columns is not clear from your original post. You simply said you wanted to combine them into a single file. When posting questions, please take time to clearly and precisely describe the issue.
Comments.UserDisplayName: Daniel Standage
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-07-10T13:22:34.017Z
Comments.Text: @DanielStandage Apologies for not being clear but I have not got my answer yet.
Comments.UserDisplayName: Priya
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-07-10T13:23:30.803Z
Comments.Text: @Priya Please explain what is not clear from the thread I linked in my first comment.
Comments.UserDisplayName: Daniel Standage
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-07-10T13:28:54.040Z
Comments.Text: @DanielStandage Sorry, I didn't click the link in your post. Thank you
Comments.UserDisplayName: Priya
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-07-10T16:23:20.723Z
Comments.Text: @DanielStandage such questions [are on topic](https://bioinformatics.meta.stackexchange.com/q/88/298) since manipulating and parsing text files is a very large part of a bioinformatician's work. This question is really not clear, mind you, but it is on topic.
Comments.UserDisplayName: terdon
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-07-11T12:48:45.063Z
Comments.Text: @Priya  tried to clarify the question a bit and I also suspected a typo in the code you tried. I hope the edits I have done are appropriate, if not please roll it back or [edit] it by your own :-).
Comments.UserDisplayName: Kamil S Jaron
Tags: <phylogenetics><phylogeny><bash>
Title: How to combine multiple files into one file?
ThreadText: # How to combine multiple files into one file?
  
  (Asked by: username_0 on 2019-07-10)
  
  I have multiple files (n=86000) with one column each and I want to combine them all into one file with 86000 columns.
  
  I tried the following command
  
  ```
  Paste *.txt > mean.txt
  
  ```
  
  And as a for loop
  
  ```
  for i in *.txt, do paste $i > mean.txt; done 
  
  ```
  
  This did not appear to work because there are 86000 to combine, I then divided my files into multiple folders with each folder not containing more than 15000 files but this did not working
  
  Format of my single file is as follows (It has 16000 lines)
  
  ```
  0.047202943506056
  0.0383549703366668
  0.0159162908608477
  0.0238048058550539
  0.0694728928685859
  0.0389052658410832
  
  ```
  
  What is the bug in the code?
  
  **Tags:** phylogenetics, phylogeny, bash
  
  > **Comments:**
  > **username_1**: Thanks. Do you want one long list? In this case the command is 'cat' not 'paste'. Just check you are not  exceeding your disk space.  
  > **username_2**: What is the error message you get?  
  > **username_3**: Equivalently answered here assuming OP wants to concatenate vertically, not horizontally, though the approach could be modified anyway: https://stackoverflow.com/questions/18695105/how-to-concatenate-huge-number-of-files  
  > **username_1**: Re-reading the question: 16000 lines x 86000 files ... this is not alot. I think the error report is important, or else perhaps it just hangs?  
  > **username_0**: @ username_2 username_2 There is no error. When I check the output file it has only one column.  
  > **username_0**: @Michael: No don't want one long list. I want an output file with 86000 columns.  
  > **username_4**: I'm voting to close this question as off-topic because it is not about bioinformatics but file manipulation. In any case it has been answered [here](https://unix.stackexchange.com/questions/205642/combining-large-amount-of-files).  
  > **username_4**: @username_0 The fact that you wanted a file with 86000 columns is not clear from your original post. You simply said you wanted to combine them into a single file. When posting questions, please take time to clearly and precisely describe the issue.  
  > **username_0**: @username_4 Apologies for not being clear but I have not got my answer yet.  
  > **username_4**: @username_0 Please explain what is not clear from the thread I linked in my first comment.  
  > **username_0**: @username_4 Sorry, I didn't click the link in your post. Thank you  
  > **username_5**: @username_4 such questions [are on topic](https://bioinformatics.meta.stackexchange.com/q/88/298) since manipulating and parsing text files is a very large part of a bioinformatician's work. This question is really not clear, mind you, but it is on topic.  
  > **username_6**: @username_0  tried to clarify the question a bit and I also suspected a typo in the code you tried. I hope the edits I have done are appropriate, if not please roll it back or [edit] it by your own :-).  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-07-10. Score: 4 | Accepted answer
  
  The bug in your loop appears be "`> filename`" per column, this over-rides the last iteration resulting in a file with a single column, which has been over-written 85999 times. Thus the file output you observe should be the single column present in the last file of your array. If you replaced "`> filename`" with "`>> filename`" within the loop you would then append to the file a new column with each iteration. If the loop fell over at some point you could then simply use one of the outputs of `wc filename` to assess how many columns you had (you might have to transponse the data for wc to work (lots of scripts for this in Perl), although there will almost certainly be an easy solution for counting columns in awk.
  
  Personally, I would use a dataframe for this operation (possibly R, definately Python pandas), because 86000 columns is a lot of columns in any other walk of life, even if it is just post-processing. Generally, if you just want bash, awk is a better solution in this case (I don't code with it, but some here certainly do). However, I do get the idea that `paste` will work if each column is an identical length.
  
  BTW `paste` must be lower case, you've got `Paste` in the first line of your code, but its okay in the second bit of code.
  
  BTW BTW, I did not know shell would take \*.txt directly as an array, with Perl/Python you need to "glob" it and assign it as an array/list.
  
  
  ---
  
  