Your kernels currently use all 4 input channels, which is fine, but also have a kernel size of (4, 1), which is probably not what you want.
The kernel will not only use all channels (different colors), but also use 4 values in one spatial dimension (e.g. values 0, 1, 2, 3).
In your use case, you could use nn.Conv1d with 4 input channels and a kernel size of 1 or alternatively just a linear layer.
Here is a small example:
batch_size, channel = 1, 8
input_vec1 = torch.randint(1, 100, (batch_size,channel)) # [1, 8]
input_vec2 = torch.randint(1, 100, (batch_size,channel)) # [1, 8]
input_vec3 = torch.randint(1, 100, (batch_size,channel)) # [1, 8]
input_vec4 = torch.randint(1, 100, (batch_size,channel)) # [1, 8]
# Extend vector from BxC --> BxC1xDxHxW where C=D
input_vec1 = input_vec1.view(batch_size, 1, channel)
input_vec2 = input_vec2.view(batch_size, 1, channel)
input_vec3 = input_vec3.view(batch_size, 1, channel)
input_vec4 = input_vec4.view(batch_size, 1, channel)
# Concat these vectors
input_concat = torch.cat([input_vec1, input_vec2, input_vec3, input_vec4], 1) #torch.Size([1, 4, 8, 1])
input_concat = input_concat.float()
# Feed to the network
conv = nn.Conv1d(4, 1, 1, 1, 0)
out = conv(input_concat)
print (out.size())

lin = nn.Linear(4, 1)
with torch.no_grad():
    lin.weight = nn.Parameter(conv.weight.squeeze(2))
    lin.bias = nn.Parameter(conv.bias)
out_lin = lin(input_concat.permute(0, 2, 1))

torch.allclose(out_lin.squeeze(), out.squeeze())