Question: All the examples that I see for using spacy just read in a single text file (that is small in size).
How does one load a corpus of text files into spacy?

I can do this with textacy by pickling all the text in the corpus:

```
docs =  textacy.io.spacy.read_spacy_docs('E:/spacy/DICKENS/dick.pkl', lang='en')

for doc in docs:
    print(doc)

```

But I am not clear as to how to use this generator object (docs) for further analysis.

Also, I would rather use spacy, not textacy.

spacy also fails to read in a single file that is large (~ 2000000 characters).

Any help is appreciated...

Ravi

Answer: 
So I finally got this working, and it shall be preserved here for posterity.

Start with a generator, here named `iterator` because I'm currently too afraid to change anything for fear of it breaking again:

```
def path_iterator(paths):
    for p in paths:
        print("yielding")
        yield p.open("r").read(25)

```

Get an iterator, generator, or list of paths:

```
my_files = Path("/data/train").glob("*.txt")

```

This gets wrapped in our ... *function* from above, and passed to `nlp.pipe`. In goes a generator, out comes a generator. The `batch_size=5` is required here, or it will fall back into the bad habit of first reading all the files:

```
doc = nlp.pipe(path_iterator(my_paths), batch_size=5)

```

The important part, and reason why we're doing all this, is that *until now nothing has happened*. We're not waiting for a thousand files to be processed or anything. That happens only *on demand*, when you start reading from `docs`:

```
for d in doc:
    print("A document!")

```

You will see alternating blocks of five (our batch\_size, above) "Yielding" and "A document". It's an actual pipeline now, and data starts coming out very soon after starting it. 

And while I'm currently running a version one minor tick too old for this, the coup de grace is multi-processing:

```
# For those with these new AMD CPUs with hundreds of cores
doc = nlp.pipe(path_iterator(my_paths), batch_size=5, n_process=64) 

```
If you can convert your corpus into a data-frame in a way that each row corresponds to a document, you can basically write a function to do what you want to do and then:

```
df['new_column'] = df['document'].apply(lambda x: your_function(x))

```

Or, I am not sure if this is what you want, but you can try something like this:

```
import spacy
import os

nlp = spacy.load('en_core_web_lg')
docs ='path\\to\\the\\corpus_folder'

def get_filename(path):
    return [i.path for i in os.scandir(path) if i.is_file()]

files=get_filename(docs)
for filepath in files:
    with open(filepath, 'r') as file_to_read:
        some_text = file_to_read.read()
        print(os.path.basename(filepath))
        print(nlp(some_text))
        print([tok.text for tok in nlp.tokenizer(some_text) if not tok.is_stop])
        print('-'*40)

```

And this is the output:

```
text1.txt
Read multiple files.
['Read', 'multiple', 'files', '.']
----------------------------------------
text2.txt
Read it, man.
['Read', ',', 'man', '.']
----------------------------------------

```

However, it is not reading with spaCy.