Thanks for your answer, you definitely helped me with part of the solution.
Now for the second which I think is the easier part.
If I want to exit the network in the middle what should I do?
For simplicity let’s say batch=1.  and my data just passed the second layer of the network, and now I want to check, if some condition is correct, if it is I want to stop the processing, else, continue processing.
Now the simplest thing I thought about is to break my network to pieces and do if condition between them, but that looks like a stupid Idea.
because I’m checking the condition only in the end of the network right now, means I’m running till the end, saving states, and then I seeing where I should have exited, But I’m not really exiting.
Am I clear enough? I have a hard time to explain this I know, actually, it’s a similar problem to the network “BranchyNet” if someone knows.