Hi Kavi!



 kavi14:

[ [ [image], [robot_path], [robot_action] ] , reward_value ]
image is a list of 3 binary matrices = torch.Size[(5, 5, 3)]
robot_path is a single binary matrix = torch.Size[(5, 5)]
robot_actions is a list of 3 binary matrices = torch.Size[(5, 5, 3)]
reward_value is an integer.
…
I did realize that the conv2d takes in images of 3 channels and I think mine has about 7 channels, but this still did not work.


I don’t know whether your specific use case makes sense, but to
answer your immediate question:
Let image be a three-channel image (say RGB) of shape
[height = 5, width = 5, nChannel = 3];
let robot_path be a single-channel image (with no explicit nChannel
dimension) of shape [height = 5, width = 5];
and let robot_actions be a three-channel image of shape
[height = 5, width = 5, nChannel = 3].
We will unsqueeze() robot_path to give it an explicit nChannel
dimension of size nChannel = 1, and then cat() the three images
together to produce a single seven-channel image, input, of shape
[height = 5, width = 5, nChannel = 7].
We then permute input's dimensions so that nChannel comes
before height and width and unsqueeze() it so that it has a leading
nBatch dimension of size nBatch = 1.  We do this because Conv2d
requires an nBatch dimension and also requires that the nChannel
dimension precede the height and width dimensions.
You would now want your convolution neural network (CNN) to start
with a Conv2d layer that has in_channels = 7.
Here is an example that shows how to cat() your input images
together and pass them to an appropriate Conv2d.  (The rest of
the CNN is left as an exercise for the reader.)
>>> import torch
>>> torch.__version__
'1.9.0'
>>> image = torch.randn (5, 5, 3)
>>> robot_path = torch.randn (5, 5)
>>> robot_actions = torch.randn (5, 5, 3)
>>> image.shape
torch.Size([5, 5, 3])
>>> robot_path.shape
torch.Size([5, 5])
>>> robot_actions.shape
torch.Size([5, 5, 3])
>>> input = torch.cat ([image, robot_path.unsqueeze (-1), robot_actions], dim = -1)
>>> input.shape
torch.Size([5, 5, 7])
>>> input = input.permute (2, 0, 1).unsqueeze (0)
>>> input.shape
torch.Size([1, 7, 5, 5])
>>> conv = torch.nn.Conv2d (in_channels = 7, out_channels = 2, kernel_size = 3)
>>> conv (input).shape
torch.Size([1, 2, 3, 3])

Best.
K. Frank