The only tip here would be to take the time to vectorize it.
Note that one important step here is to have the proper representation of your data: you can do very expensive reprocessing of your data once to put it in the shape you want if that makes the forward faster.
For example, you could represent your graph with an adjacency matrix so that accumulating the features from all the neighbors can be done using a single matrix matrix multiplications.
Or store indices in a linearized way  to be able to gather all the neighbors in a single gather.
But this will depend a lot on the constraints given by your aggregation function.