You are not doing anything wrong and the batches provided by train_loader and valid_loader should just use the provided indices from the corresponding sampler.
Since the underlying .dataset is not changed, the len will be the same as the original one.
However, during the creation of the batches, the samplers will use the provided indices to load the data from the internal Dataset.
Alternatively, you could also wrap your dataset in a Subset, which will then change the length.
Besides that both approaches should yield the same result.