I have a set of more than million records all of them in the `.txt` format. Each `file.txt` has just one line:

> 
> 'user\_name', 'user\_nickname', 24, 45
> 
> 
> 

I need to run a distribution check on the aggregated list of numeric features from the million files. Hence, I needed to aggregate these files into large data frame. The approach I have been following is as follows:

```py
import glob
import os
import pandas as pd

import sqlite3

connex = sqlite3.connect("data/processed/aggregated-records.db")

files_lst = glob.glob("data/raw/*.txt")

files_read_count = 1

for file_name in files_lst:
    data_df = pd.read_csv(file_name, 
                          header=None, 
                          names=['user_name', 'user_nickname',  
                                 'numeric_1', 'numeric_2'])

    data_df['date_time'] = os.path.basename(file_name).strip(".txt")

    data_df.to_sql(name=file_name, con=connex, if_exists="append", index=False)

    files_read_count += 1

    if (files_read_count % 10000) == 0:
        print(files_read_count, " files read")

```

The issue I have is that with this approach, I am able to write to the database at a very slow pace (about 10,000 files in an hour). Is there any way to run this faster?