Hi Adam,
This is a great question. Indeed, torchvision has cumulated a lot of tech debt over time, which is hard to change due to BC considerations, but which we would gladly change otherwise.
We have initiated some discussions around things we would love to change in torchvision if we didn’t have any BC considerations, we will be posting the summary of the discussion on GitHub once it’s fleshed out, but here are some specific points to your questions:

Datasets
We are in the process of re-designing all of our datasets so that they are built using DataPipes 12. This will allow them to be used to stream data from both disk but also the internet, like what WebDataset does.
In this process, we will also be making more consistent the return types of datasets – it will probably be a dict everywhere, which contains fields for the image / label / etc (but this is still being fleshed out and @pmeier is actively working on the design)
In this context, a transform will not be part of the dataset anymore, but will just be a .map transform on top of the DataPipe.

Transforms

PIL and Tensors
We started with PIL because it was widely available in Python and was kind of the standard for image processing. With the push for sharing the same transform implementation between Python and C++, we enabled Tensor Transforms, which also give other benefits such as GPU and autograd.
In my view, the PIL-based transforms are mostly legacy now, but we won’t be deprecating those until we have improved all the gaps that there are still compared to the Tensor Transforms (speed and antialising for resize are two things we are improving in the upcoming release)…
About making the transforms be nn.Module vs plain object, this was to facilitate torchscript support.
That being said, I’m not super happy with the current class API for transforms, as it doesn’t allow for a nice way of enabling joint transforms for boxes / images / etc, so a class-based API should be rethink. For now only the functional API for the transforms is powerful enough.
I think it would be interesting to consider that the concept of a “model” can involve everything from data decoding up to the final model output, which means that transforms could be part of the model (and thus are analogous to nn.Module such as nn.Conv2d). The current difficulty about this is how to make it efficient, without also making the dataloader part of the model.

For a few more specific questions:

should all datasets return (and all transforms accept) dicts with keys for possible components

Yes, I think that’s what we would be doing for the new dataset implementations. For the transforms, I don’t yet know, but a redesign would be good.
Happy to chat over a video call, ping me on slack and we can continue the discussion.