@smth I am sorry. I never realized that you had replied to it.  I have now enabled notifications.
For the PR, I have raised one here 19.
I have been trying some experiments with hogwild and have some questions. Let me describe my setup first.
I first divide my training data into k disjoint sets. Then I setup k processes which are sharing the model but each has a separate train dataset. Note that the test dataset is shared between all the processes.  Each process is run for n epochs, Each process performs the standard train-test iterations (training using only its split, testing using the entire dataset). Once all the processes are completed, I do one more round of testing, in the main process.
I observe once n epcohs have been performed, all processes report the same test performance, which is the same as the test performance reported by the main process (once every other process has terminated).
I could not figure out when the updates are shared between the different processes. Is it done per-epoch or after a fixed time interval or fixed number of training steps? This affects which strategy (of deciding when to test) is optimal.
There are several strategies for training/testing the model:
(i) Train the k processes for n epoch and test only once after all the processes have terminated. The upside is - testing is done just 1 time and not k*n times. It is also the simplest strategy.  The downside is, we can not do early stopping and need to know n in advance.
(ii) Train the k processes for 1 epoch, wait for all the process to terminate, do testing, then again train the k process for 1 epoch. the upside is now we can perform early stopping while we test just n times. The downside is, there is an additional wait period where k-1 processes are sitting idle. I suppose the additional overhead of triggering k processes per epoch would be negligible.
(iii) Train k process for n epochs. Each process implements early-stopping for itself, This is essentially what is happening in the example right now. One downside is the extra testing but in terms of time-taken I think this would be very similar to the second strategy.
(iv) Out of the k processes, only 1 process does the testing. All other processes just train on the data. the process which also tests would lag behind but atleast all other processes could move faster. This also allows for early stopping and is not too complex. Alternatively, we could have reserved a process just for testing but that slows down the training and performs “extra” testing.
Among all these options, I think (iv) is the best one provided its guaranteed that the updates are synchronized between the processes at regular intervals. But I am not sure when and how the updates are applied between the processes. What are your thoughts?