That’s definitely a lot closer than I’ve gotten! To give some background, the reason I’m trying to do this is to “shrink” a larger matrix by cutting it up into windows and then taking the mode of these windows. Similar to how you would interpolate an image, but by taking the mode instead of some mean or nearest neighbor.
What I meant by 4x1 window was just to have the windows flattened so torch.mode would work. A 2x2 matrix would work too  but I haven’t figured out how to do it. Hopefully the example below makes it clearer:
# Each last dimension is made up of the 4 elements in a 2x2 window with no overlap
# between the windows. 
#There should be 9 of these windows for this particular example.

# Shape: 1x 9 (num_windows) x 2 (windows_size_x) x 2 (window_size_y)
the_dream = torch.tensor([[[1, 1], [1, 2]] ,
                       [ [3, 1],[ 0 , 0]], ...
                        [[5, 0], [3, 0]],
                        [[0, 0], [0, 0]]])