Dear albanD, thank you again for your feedback.
With the help of the following instructions I tested whether MKL is active.
https://gist.github.com/mingfeima/bdfb2db3928ca51b795622b29264ef11 1
Here are the results:
ldd libtorch.so
linux-vdso.so.1 (0x00007fff38544000)
libcudart-1b201d85.so.10.1 => /home/user/.local/lib/python3.7/site-packages/torch/lib/./libcudart-1b201d85.so.10.1 (0x000072a3e395a000)
libgomp-7c85b1e2.so.1 => /home/user/.local/lib/python3.7/site-packages/torch/lib/./libgomp-7c85b1e2.so.1 (0x000072a3e3730000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x000072a3e36f0000)
librt.so.1 => /lib64/librt.so.1 (0x000072a3e36e6000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x000072a3e36cc000)
libdl.so.2 => /lib64/libdl.so.2 (0x000072a3e36c6000)
libm.so.6 => /lib64/libm.so.6 (0x000072a3e357e000)
libc10_cuda.so => /home/user/.local/lib/python3.7/site-packages/torch/lib/./libc10_cuda.so (0x000072a3e3350000)
libnvToolsExt-3965bdd0.so.1 => /home/user/.local/lib/python3.7/site-packages/torch/lib/./libnvToolsExt-3965bdd0.so.1 (0x000072a3e3146000)
libc10.so => /home/user/.local/lib/python3.7/site-packages/torch/lib/./libc10.so (0x000072a3e2ef4000)
libstdc++.so.6 => /lib64/libstdc++.so.6 (0x000072a3e2cfa000)
libc.so.6 => /lib64/libc.so.6 (0x000072a3e2b34000)
/lib64/ld-linux-x86-64.so.2 (0x000072a42b45c000)
Note: all versions of PyTorch (with or without CUDA support) have Intel® MKL-DNN acceleration support enabled by default. https://software.intel.com/en-us/articles/getting-started-with-intel-optimization-of-pytorch
So it looks like PyTorch doesn’t use MKL in my distribution?
But this:
python3 -c 'import torch; a = torch.randn(10); print(a.to_mkldnn().layout)'
torch._mkldnn <<< result on my machine

does work?
I use Fedora 30 (Qubes OS) and have installed PyTorch via PIP. I will create a separate Conda Environment, install PyTorch with MKL and test it, thanks for the hint. I know that Theano uses openblas, maybe that explains the speed difference?
" .eval() every time"
The above loop is my bootstrap loop, “MAIN LOOP (everything else removed)” is wrong here. It should mean “BOOT LOOP (everything else removed)”.
I use it to fill the memory before i start training. So in this case you’re right, i don’t need it here. I already checked that, removing it does help a bit - but not very much.
BOOT LOOP (everything else removed):
while not done:
        #////////////////////////////////////////////////
        SYS.ENV['STEPS'] += 1
        #////////////////////////////////////////////////
        action = SYS.AGT['CALL'].get_action(SYS, state)
        #////////////////////////////////////////////////
        next_state, reward, done, info = SYS.ENV['CALL'].step(action)
        #////////////////////////////////////////////////
        SYS.AGT['CALL'].remember(state, action, reward, next_state, done) # add data to memory
        #////////////////////////////////////////////////
        state = deepcopy(next_state) # copy
        #////////////////////////////////////////////////
        tqdm_e.update(1)
        #////////////////////////////////////////////////
        if(done): return(True)

followed by the TRAIN LOOP:
while not done:
            #////////////////////////////////////////////////
            SYS.ENV['STEPS'] += 1
            #////////////////////////////////////////////////
            action = SYS.AGT['CALL'].get_action(SYS, state)
            #////////////////////////////////////////////////
            next_state, reward, done, info = SYS.ENV['CALL'].step(action)
            #////////////////////////////////////////////////
            SYS.AGT['CALL'].remember(state, action, reward, next_state, done)
            #////////////////////////////////////////////////
            SYS.ENV['SCORE']  += reward
            SYS.ENV['LOSSES'] += SYS.AGT['CALL'].train(SYS)
            #////////////////////////////////////////////////
            state = deepcopy(next_state)

Inside the TRAIN LOOP

in SYS.AGT[‘CALL’].get_action() i switch to eval()
in SYS.AGT[‘CALL’].train() i switch to train()

You can see that in DQNAgent.train() in the code of the first post. But as I can see I have done it twice, it will be corrected.
So my next steps are as follows:

move all data to tensors (TensorDataset) and use DataLoader
check the usage of float()
check the usage of eval()/train()
install PyTorch via Conda and compare the runtimes with/without MKL

It will take some time (days) but I will report the results here.
Since I have seen in other threads that you are very well informed about these topics:
Can you answer my question 2 (first posting)?
Thank you very much for your efforts to help me.