In a similar situation, as my module started with a fully-connected layer, I just used matmul distributivity to do: linear([x_1 | context]; w0) + bias = linear(x_1; w1) + linear(context; w2) + bias. You can actually do inplace summation, and output from “linear(context)”  is a smaller non-expanded tensor.
If your scenario is different, and you can’t process vector parts independently, I don’t this you can do anything about memory, as you have to intersperse data from two sources to pass to your module.