Interpolation techniques, such as F.interpolate, work usually in the spatial dimensions, while it seems you would like to reduce the number of channels.
If that’s the case, you could use a conv layer with in_channels=640, out_channels=3 and kernel_size=1.
Alternatively you could also permute the input and use F.interpolate on the original channel dimension, which would be then in a spatial dim.