TL;DR
>>> from functools import reduce
>>> from torchvision.models import resnet34
>>>
>>> def get_module_by_name(module, access_string):
...     names = access_string.split(sep='.')
...     return reduce(getattr, names, module)
>>>
>>> model = resnet34()
>>> get_module_by_name(model, 'layer1.0.relu')
ReLU(inplace=True)


The most confusing point is nn.Sequential defaulting to naming its modules with numbers, that are not valid python identifiers, thus not being reachable by a dot notation like layer1.0.relu, if I take torchvision's resnet.
But you can work around it.


Accessing direct child by name
If you have your hands on the network implementation, you can choose valid names for the Sequential by using an OrderedDict like shown in the documentation, and access the layers like you would for anything other than Sequential:
>>> model = nn.Sequential(OrderedDict([
...     ('conv1', nn.Conv2d(1,20,5)),
...     ('relu1', nn.ReLU()),
...     ('conv2', nn.Conv2d(20,64,5)),
...     ('relu2', nn.ReLU())]))
>>> model.relu1
ReLU()

As @Inder  pointed out, an alternative to the dot notation in order to access attributes is the built-in getattr() function.
But unlike the dot notation, it accepts any string, and it happens to work on Sequential's default naming when using numerical strings (Sequential is actually built not on a list but on a dict with string keys).

To get back to my first example, doing getattr(model.layer1, '0').relu works.



Accessing nested child by name
Something nice of getattr() is that you can go multiple levels at a time, for example getattr(foo, 'bar.baz') to get to foo.bar.baz.
Unfortunately, it only works if the whole dot chain is made of valid identifiers.
The solution is to use getattr() recursively yourself, and that’s easy enough with reduce():
from functools import reduce
from typing import Union

import torch
from torch import nn


def get_module_by_name(module: Union[torch.Tensor, nn.Module],
                       access_string: str):
    """Retrieve a module nested in another by its access string.

    Works even when there is a Sequential in the module.
    """
    names = access_string.split(sep='.')
    return reduce(getattr, names, module)


if __name__ == '__main__':
    from torchvision.models import resnet34
    
    model = resnet34()
    get_module_by_name(model, 'layer1.0.relu')

Output:
ReLU(inplace=True)