I’m not familiar with the mentioned Lesson in FastAI, however from a code perspective you are detaching the computation graph from vert and hori by using numpy.dot and rewrapping the result in a tensor with requires_grad=True.
Since vert and hori are already nn.Parameters, you could stick to PyTorch methods, so that Autograd will automatically create the computation graph (and can thus backpropagate).
I’m not sure, if this will yield the desired result, but this core updates vert and hori:
def hypothesis(vert,hori):
    return torch.matmul(vert,hori)

def mse(y_hat,y):
    return ((y_hat-y)**2).mean()

def update(y_hat):
    y_hat = hypothesis(vert,hori)
    # perform gradient descent
    loss = mse(y_hat,blockData)
    loss.backward()
    if t%10 == 0:
        print(t,'-------------',loss)
    with torch.no_grad():
        vert.sub_(lr * vert.grad)
        hori.sub_(lr * hori.grad)
        vert.grad.zero_()
        hori.grad.zero_()

vecSize = 10
shape = (20,14)
# random large block of data
blockData = torch.randn(shape)

hori = nn.Parameter(torch.randn(vecSize,shape[1]))
vert = nn.Parameter(torch.randn(shape[0],vecSize))

lr = 1e-1
for t in range(101):
    update(y_hat)