Yes, its (H,H) and x_i would be (H,1) so multiplication should work somehow. Just to add a bit of context I am trying to implement time convolution (https://arxiv.org/pdf/1803.05563.pdf 1, section 3.1). the goal is to take a set of vectors (seq_length,H), for each of this vectors extract context (C,H) dimensionality and apply W matrix in a way to combine this context tensor into a vector (H,). In other words, W represents weights combining context vector and is convolved (or maybe proper to say correlated) through the whole sequence.