Merge branch 'main' of https://huggingface.co/spaces/huggingface/data-measurements-tool-2 into main

c4 realnewslike train text

Loading per-widget. Various changes to streamlit interactions for efficiency.

More cache; this time adding length_df.feather

More cache

Merge branch 'main' of https://huggingface.co/spaces/huggingface/data-measurements-tool-2 into main

One more flag passing needed for setting live deployment

Adds flag for live deployment so that things will not be all recalculated when live.

Trying cache push without the largest files, as they throttle our pushes

wiki general stats

Finishing c4 en train text cache

c4 en train text cache

A variety of cache

text dset cache

glue cola train sentence cache

removing extraneous backup

Merge branch 'main' of https://huggingface.co/spaces/huggingface/data-measurements-tool-2 into main

Hate speech offensive

General stats cache

dset peek caches

c4 en noblocklist cache

c4 ennoblocklist cache

Starting imdb cache

Changing cache naming scheme to make consistent.

Hate speech cache

More flexibility in specifying cache directory.

Scripts to generate cache

Standardizing filenaming a bit.

More modularizing; npmi and labels

Some additional modularizing and caching of the text lengths widget

Modularization and caching of text length widget

Removes extraneous debugging print statements

Missing a dependency; adding to requirements.txt

Begins modularizing so that each widget can be independently loaded without having a requirement on the ordering of load_or_preparing in app.py. This means that each function corresponding to a widget will check if the variables it depends on have been calculated yet. If not, it will call back to calculate them. Because of the messiness this causes with passing the use_cache variable around, I've now set use_cache as a global variable, set when the DatasetStatisticsCacheClass is initialized, and removed the use_cache arguments appearing in nearly every function.

Removing need to keep around base dset for the header widget; now just saving what is shown -- the first n lines of the base dataset -- as a json, and loading if it's cached.

Removing any need for a dataframe in expander_general_stats; instead making sure to cache and load the small amount of details needed for this widget. Note I also moved around a couple functions -- same content, just moved -- so that it was easier for me to navigate through the code. I also pulled out a couple of sub-functions from larger functions, again to make the code easier to work with/understand, as well as helping to further modularize so we can limit what needs to be cached.

Splitting prepare_dataset into preparing the base dataset, and the tokenized dataset. This will help us to have further control over caching and loading data, eventually removing the storage of base dataset.

Updating NLTK requirements due to vulnerability in versions below 3.6.4: contained an inefficient Regular Expression and is vulnerable to regular expression denial of service attacks

Continuing cache minimizing in new repository. Please see https://github.com/huggingface/DataMeasurements for full history

:art: add line to file to bump ci

hate speech18 cache

hate speech 18 pmi file cache

Test to push up a simple cache file

:construction_worker: update CI to rebase

:bug: filter_vocab -> filter_words

:bug: really make sure log_files/ exists

:bug: add log_files dir if not exists

:rocket: add app

:rocket: add app and reqs

:construction_worker: add CI