You’d better concatenate them as they are; then do sampling according to the lower size, or randomly sample, or whatever; which is way better than augmenting probably.
The alternative is to use a weighted loss 18.
If provided, the optional argument weight should be a 1D Tensor assigning weight to each of the classes. This is particularly useful when you have an unbalanced training set.