Your approach is known as model sharding and would work.
However, since only a single GPU would be used during this approach, there are more advanced techniques such as pipeline parallelism, which would try to overlap the workloads more.