Question: All the examples that I see for using spacy just read in a single text file (that is small in size).
How does one load a corpus of text files into spacy?

I can do this with textacy by pickling all the text in the corpus:

```
docs =  textacy.io.spacy.read_spacy_docs('E:/spacy/DICKENS/dick.pkl', lang='en')

for doc in docs:
    print(doc)

```

But I am not clear as to how to use this generator object (docs) for further analysis.

Also, I would rather use spacy, not textacy.

spacy also fails to read in a single file that is large (~ 2000000 characters).

Any help is appreciated...

Ravi

Answer: 
You would just read the files in one at a time. This is what I usually do with my corpora files:

```
import glob
import spacy
nlp = spacy.load("en_core_web_sm")
path = 'your path here\\*.txt'

for file in glob.glob(path):
    with open(file, encoding='utf-8', errors='ignore') as file_in:
        text = file_in.read()
        lines = text.split('\n')
        for line in lines:
            line = nlp(line)
            for token in line:
                print(token)

```
If you can convert your corpus into a data-frame in a way that each row corresponds to a document, you can basically write a function to do what you want to do and then:

```
df['new_column'] = df['document'].apply(lambda x: your_function(x))

```

Or, I am not sure if this is what you want, but you can try something like this:

```
import spacy
import os

nlp = spacy.load('en_core_web_lg')
docs ='path\\to\\the\\corpus_folder'

def get_filename(path):
    return [i.path for i in os.scandir(path) if i.is_file()]

files=get_filename(docs)
for filepath in files:
    with open(filepath, 'r') as file_to_read:
        some_text = file_to_read.read()
        print(os.path.basename(filepath))
        print(nlp(some_text))
        print([tok.text for tok in nlp.tokenizer(some_text) if not tok.is_stop])
        print('-'*40)

```

And this is the output:

```
text1.txt
Read multiple files.
['Read', 'multiple', 'files', '.']
----------------------------------------
text2.txt
Read it, man.
['Read', ',', 'man', '.']
----------------------------------------

```

However, it is not reading with spaCy.