run_squad with early stopping on a validation set
Hello. 
Is there a way to use run_squad with early stopping as a validation set? 

I have 3 files: train-v1.1.json, dev-v1.1.json, and test-v1.1.json.
I want to train on the train file, stop the training when the loss on the dev file starts to increase, and then do the final prediction and answers output on the test set. 

At Keras it's pretty straight forward: 
`history = model.fit(X_train, y_train, validation_data=(X_dev, y_dev), epochs=4000, verbose=0, callbacks=[EarlyStopping(monitor='dev_loss', mode='min'])`

How can I have a similar functionallity in huggingface's run_squad? I'm asking whether is there a built in way before implementing it from scratch. 

I thought of changing the code in the point of: 
`--evaluate_during_training` - it activates the evaluation during training - I can change the fie to the dev-v1.1.json file in this point, and stop the training if the validation loss increases. 

I would like to know if there is a better way. 

Thanks
Hi, this would be something you would need to build yourself.

Alternatively you could use our Layers as Keras Layers?