Thanks for the code.
weight_fb needs to be on the device, as it’s not an nn.Parameter:
self.weight_fb =torch.Tensor(out_features, in_features).to(device)

However, the error message shouldn’t be an illegal memory access, so thanks for reporting it.