You don’t need to do anything special to perform bayesian optimization for your hyperparameter tuning when using pytorch. You could just setup a script with command line arguments like --learning_rate, --num_layers for the hyperparameters you want to tune and maybe have a second script that calls this script with the diff. hyperparameter values in your bayesian parameter optimization loop.
Conceptually, you can do sth like this
for round i in max_rounds:
     output = subprocess.call('model.py', '--learning_rate', 'some_val', '--num_layers',  'some_number')
     score = # parse score from output string (e.g., your loss output)
     #optimize p(score | hyperparam values)
     # get some_val, some_number for next round

But it is probably more efficient if you use a library like hyperopt (https://github.com/hyperopt/hyperopt 627)
Maybe sth like

def run_model(learning_rate):
    # your model init here
    # your training here
    return loss


import numpy as np
from hyperopt import hp, tpe, fmin

# Single line bayesian optimization of polynomial function
best = fmin(fn=lambda x: run_model(x),
            space=hp.normal('x', 0.00001, 0.9),
            algo=tpe.suggest, 
            max_evals=2000)

But given how expensive typical model training for DL is, it may take a long time …