@omarfoq, thank you for your response! I tried adding to(torch.cfloat) like you mentioned and my code looks as below.
import torch
from torch import nn, optim


class ComplexTest(nn.Module):
    def __init__(self):
        super(ComplexTest, self).__init__()
        self.fc1 = nn.Linear(10, 20).to(torch.cfloat)
        self.fc2 = nn.Linear(20, 10).to(torch.cfloat)
        self.relu = nn.ReLU()

    def forward(self, inputs):
        return self.fc2(self.relu(self.fc1(inputs)))


device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
complex_test = ComplexTest().to(device)
complex_test.train()

opt = optim.Adam(complex_test.parameters())

mse_loss = nn.MSELoss()

for _ in range(100):
    inp = torch.randn((1000, 10), dtype=torch.cfloat)
    op = complex_test(inp)

    loss = mse_loss(op, inp)
    loss.backward()
    opt.step()
    print(loss.item())

Now the layers do not have problem with the complex valued input. But I could not figure out how to do this for activation function. The following error thereby pops up.

“threshold_cpu” not implemented for ‘ComplexFloat’

How can I proceed?