Hey everybody,
I reply here since I managed to solve my issue: the error was so stupid that I’m really embarrassed to write about it 
But still, I hope it may be useful for somebody, at least as a reminder that you need to take some breaks and rest from your code in order to clear your mind and see where the bugs are!
Turns out that the problem was how I wrote my_collate function: I spend a lot of time computing the sequences’ lengths on CPU, while the amount of computations executed on GPU is so low that I could not see it on any performance profiler.
Therefore, I simply solved my issue by doing the most obvious thing: make my custom Dataset class give me the sequence’s length along with the sequence itself, and using my_collate function to just create the PackedSequence object for the input sequence. Here is the code for both of them in case it may be helpful for somebody.
My custom Dataset class:
import torch
import numpy as np


class CoOccurrencesDatasetRnnTime(torch.utils.data.Dataset):

    """
        Support class for the loading and batching of the co-occurrences of video frames extracted offline.
        The class returns directly the sequence along with its length

        Args:
            root_dir (string): file path of the .npy file containing the co-occurrences
            sequence_length (int): length of the analyzed sequence by the RNN
            transforms (object torchvision.transform): Pytorch's transforms used to process the co-occurrences
    """

    ##  Constructor
    def __init__(self, root_dir, sequence_length=1, transforms=None):
        self.root_dir = root_dir
        self.seq_length = sequence_length
        self.transforms = transforms
        self.co_occurrences = torch.from_numpy(np.load(root_dir)).type(torch.FloatTensor)
        self.co_occurrences = self.co_occurrences.view(int(self.co_occurrences.size()[0]/10875), -1,
                                                       self.co_occurrences.size()[1])

    ##  Override total dataset's length getter
    def __len__(self):
        return int(self.co_occurrences.size()[0]*10875)
        #10875 is the number of features vector for each video frame

    ##  Override single items' getter
    def __getitem__(self, idx):
        f_idx = int(np.floor(idx / 10875)) #frame index
        p_idx = int(np.floor(idx % 10875)) #patch index inside the frame from which the features have been extracted
        if self.transforms is not None:
            if f_idx-self.seq_length < 0:
                seq = torch.zeros(self.seq_length, self.co_occurrences.size()[2])
                seq[0:f_idx+1, :] = self.transforms(self.co_occurrences[0:f_idx+1, p_idx, :])
                seq_len = f_idx + 1
                return [seq, seq_len], self.transforms(self.co_occurrences[f_idx, p_idx, :])
                #only need the last element of the sequence as target value for the loss
            else:
                return [self.transforms(self.co_occurrences[f_idx-self.seq_length:f_idx, p_idx, :]), self.seq_length], \
                       self.transforms(self.co_occurrences[f_idx, p_idx, :])
        else:
            if f_idx-self.seq_length < 0:
                seq = torch.zeros(self.seq_length, self.co_occurrences.size()[2])
                seq[0:f_idx+1, :] = self.co_occurrences[0:f_idx+1, p_idx, :]
                seq_len = f_idx + 1
                return [seq, seq_len], self.co_occurrences[f_idx, p_idx, :]
            else:
                return [self.co_occurrences[f_idx-self.seq_length:f_idx, p_idx, :], self.seq_length], \
                       self.co_occurrences[f_idx, p_idx, :]



and my custom collate function for the Dataloader:
def my_collate(batch):

    # Preparing input sequences
    data = [item[0][0] for item in batch]
    x = torch.stack(data)
    seqs_length = torch.ByteTensor([item[0][1] for item in batch])

    # Both padded sequences and lengths should be ordered descendingly wrt to the sequence length
    lengths, indices = torch.sort(seqs_length, descending=True)
    x = x[indices, :, :]

    # Packing the data
    x = torch.nn.utils.rnn.pack_padded_sequence(x, lengths, batch_first=True)

    # Preparing target values
    y = [item[1] for item in batch]
    y = torch.stack(y)

    return [x, y]


Now the whole procedure takes place on GPU!