I would start with a single nn.Conv2d, use your workflow to initialize the PyTorch module with the TF parameters, and make sure that the outputs as well as the gradients are identical (up to floating point precision) before checking larger blocks or the entire model, as it would be easier.