N is referring to the number of boxes, so if you are dealing with 10 boxes the shape would be [10, 4].
The RuntimeError you get during unsqueeze is a bit strange, as the single bounding box should contain 4 coordinates only. Based on the error message it seems you are trying to cat/stack tensors where one has a shape of 380 in dim0 while the other has 3.