Assuming you are permuting the tensor to the channels-first memory layout and it contains a batch dimension, your code looks correct. A quick check would be:
i1 = np.zeros((24, 24, 3))
i2 = np.ones((24, 24, 4))
img = np.dstack((i1, i2))
print(img.shape)
> (24, 24, 7)

image = torch.from_numpy(img)
image = image.permute(2, 0, 1).unsqueeze(0)
thermal = image[:, 3:] 
rgb = image[:, :3]   
print((thermal == 1.).all())
> tensor(True)

print((rgb == 0.).all())
> tensor(True)