Yep, the thing is I tracked the mice with the same Model, now I am trying to find their snout and tail on a cropped image where they are usually placed somewhere around the center. All I did was to change the output to 4 instead of 2 nodes. Also my data size (due to manually labeling) is only 2000 images compared to 120.000 before.
Do I need to replace the sigmoid with some other kind of activation?
EDIT:
So I just got rid of the Sigmoid and this happened, does not look really promising  learning rate is 0.01
Epoch 1/1000
----------
train Loss: 0.007419419793224839, Acc: 0.0
val Loss: 0.0018860326872931587, Acc: 0.0

Epoch 2/1000
----------
train Loss: 0.0018890846087030633, Acc: 0.0
val Loss: 0.0018535365247064166, Acc: 0.0

Epoch 3/1000
----------
train Loss: 0.0018744729794364758, Acc: 0.0
val Loss: 0.0018500339653756883, Acc: 0.0

Epoch 4/1000
----------
train Loss: 0.0018520522586725377, Acc: 0.0
val Loss: 0.0018364218788014517, Acc: 0.0

EDIT2:
Using Adadelta() optimizer I get weird behaviours like this:
[[ 1.0000e+00,  4.1560e-39],
 [ 4.1560e-39,  1.0000e+00]] # <-- in one batch all outputs look similiar to this

[[ 0.4993,  0.5002],
 [ 0.4964,  0.5007]] # <-- in next batch similar to this


the outputs seem to jump from batch to batch between  around [[1, 0], [0, 1]] to around [[0.5, 0.5], [0.5, 0.5]]
This I find very weird…and I cannot explain why, maybe you have an idea. I tried messing around with the batch size which gave no improvements at all.