I guess the performance is bad, as the activations might be completely different using dilated convolutions.
Are you just checking the performance or are you also fine tuning the model?
The latter might help adapt the dilated convolutions to your use case.