text: Title: Gaussian splatting: assertion error (coeffs.shape[-2] == num_sh_bases(degree))
  Body: **Describe the bug**
  Running ns-train gaussian-splatting crashed with assertion failure
  
  **To Reproduce**
  I built a docker image using the gaussian-splatting branch.
  
  1. `git clone https://github.com/nerfstudio-project/nerfstudio.git -b gaussian-splatting --recurse-submodules`
  2. `docker build --build-arg CUDA_VERSION=11.8.0 --build-arg CUDA_ARCHITECTURES=86 --build-arg OS_VERSION=22.04 --tag nerfstudio-gs --file Dockerfile .`
  3. `docker run --gpus all --privileged --network host  --rm -it -v /home/user/workspace/:/workspace -v /mnt/data:/data --shm-size=32G --name nerfstudio-gs nerfstudio-gs`
  4. Inside container: `ns-train gaussian-splatting --data data/posters_v3/`
  
  **Expected behavior**
  The training should not fail.
  
  **Screenshots**
  
  Logs here
  ```
  user@uscnsl-exxact-server:/workspace/nerfstudio/nerfstudio_ws$ ns-train gaussian-splatting --data data/posters_v3/
  [22:49:38] Using --data alias for --data.pipeline.datamanager.data                                          train.py:230
  ──────────────────────────────────────────────────────── Config ────────────────────────────────────────────────────────
  TrainerConfig(
      _target=<class 'nerfstudio.engine.trainer.Trainer'>,
      output_dir=PosixPath('outputs'),
      method_name='gaussian-splatting',
      experiment_name=None,
      project_name='nerfstudio-project',
      timestamp='2023-12-08_224938',
      machine=MachineConfig(seed=42, num_devices=1, num_machines=1, machine_rank=0, dist_url='auto', device_type='cuda'),
      logging=LoggingConfig(
          relative_log_dir=PosixPath('.'),
          steps_per_log=10,
          max_buffer_size=20,
          local_writer=LocalWriterConfig(
              _target=<class 'nerfstudio.utils.writer.LocalWriter'>,
              enable=True,
              stats_to_track=(
                  <EventName.ITER_TRAIN_TIME: 'Train Iter (time)'>,
                  <EventName.TRAIN_RAYS_PER_SEC: 'Train Rays / Sec'>,
                  <EventName.CURR_TEST_PSNR: 'Test PSNR'>,
                  <EventName.VIS_RAYS_PER_SEC: 'Vis Rays / Sec'>,
                  <EventName.TEST_RAYS_PER_SEC: 'Test Rays / Sec'>,
                  <EventName.ETA: 'ETA (time)'>,
                  <EventName.GAUSSIAN_NUM: 'Number of Gaussians'>
              ),
              max_log_size=10
          ),
          profiler='basic'
      ),
      viewer=ViewerConfig(
          relative_log_filename='viewer_log_filename.txt',
          websocket_port=None,
          websocket_port_default=7007,
          websocket_host='0.0.0.0',
          num_rays_per_chunk=32768,
          max_num_display_images=512,
          quit_on_train_completion=False,
          image_format='jpeg',
          jpeg_quality=70,
          make_share_url=False
      ),
      pipeline=VanillaPipelineConfig(
          _target=<class 'nerfstudio.pipelines.base_pipeline.VanillaPipeline'>,
          datamanager=FullImageDatamanagerConfig(
              _target=<class 'nerfstudio.data.datamanagers.full_images_datamanager.FullImageDatamanager'>,
              data=PosixPath('data/posters_v3'),
              masks_on_gpu=False,
              images_on_gpu=False,
              dataparser=ColmapDataParserConfig(
                  _target=<class 'nerfstudio.data.dataparsers.colmap_dataparser.ColmapDataParser'>,
                  data=PosixPath('.'),
                  scale_factor=1.0,
                  downscale_factor=None,
                  scene_scale=1.0,
                  orientation_method='up',
                  center_method='poses',
                  auto_scale_poses=True,
                  train_split_fraction=0.9,
                  depth_unit_scale_factor=0.001,
                  images_path=PosixPath('images'),
                  masks_path=None,
                  depths_path=None,
                  colmap_path=PosixPath('colmap/sparse/0'),
                  load_3D_points=True,
                  max_2D_matches_per_3D_point=-1
              ),
              camera_res_scale_factor=1.0,
              eval_num_images_to_sample_from=-1,
              eval_num_times_to_repeat_images=-1,
              eval_image_indices=(0,),
              cache_images='cpu'
          ),
          model=GaussianSplattingModelConfig(
              _target=<class 'nerfstudio.models.gaussian_splatting.GaussianSplattingModel'>,
              enable_collider=True,
              collider_params={'near_plane': 2.0, 'far_plane': 6.0},
              loss_coefficients={'rgb_loss_coarse': 1.0, 'rgb_loss_fine': 1.0},
              eval_num_rays_per_chunk=4096,
              prompt=None,
              warmup_length=500,
              refine_every=100,
              resolution_schedule=250,
              num_downscales=2,
              cull_alpha_thresh=0.1,
              cull_scale_thresh=0.5,
              reset_alpha_every=30,
              densify_grad_thresh=0.0002,
              densify_size_thresh=0.01,
              n_split_samples=2,
              sh_degree_interval=1000,
              cull_screen_size=0.15,
              split_screen_size=0.05,
              stop_screen_size_at=4000,
              random_init=False,
              extra_points=0,
              ssim_lambda=0.2,
              stop_split_at=15000,
              sh_degree=4,
              camera_optimizer=CameraOptimizerConfig(
                  _target=<class 'nerfstudio.cameras.camera_optimizers.CameraOptimizer'>,
                  mode='off',
                  trans_l2_penalty=0.0001,
                  rot_l2_penalty=0.0001,
                  optimizer=None,
                  scheduler=None
              )
          )
      ),
      optimizers={
          'xyz': {
              'optimizer': AdamOptimizerConfig(
                  _target=<class 'torch.optim.adam.Adam'>,
                  lr=0.00016,
                  eps=1e-15,
                  max_norm=None,
                  weight_decay=0
              ),
              'scheduler': ExponentialDecaySchedulerConfig(
                  _target=<class 'nerfstudio.engine.schedulers.ExponentialDecayScheduler'>,
                  lr_pre_warmup=1e-08,
                  lr_final=1.6e-06,
                  warmup_steps=0,
                  max_steps=30000,
                  ramp='cosine'
              )
          },
          'color': {
              'optimizer': AdamOptimizerConfig(
                  _target=<class 'torch.optim.adam.Adam'>,
                  lr=0.0005,
                  eps=1e-15,
                  max_norm=None,
                  weight_decay=0
              ),
              'scheduler': ExponentialDecaySchedulerConfig(
                  _target=<class 'nerfstudio.engine.schedulers.ExponentialDecayScheduler'>,
                  lr_pre_warmup=1e-08,
                  lr_final=0.0001,
                  warmup_steps=0,
                  max_steps=30000,
                  ramp='cosine'
              )
          },
          'opacity': {
              'optimizer': AdamOptimizerConfig(
                  _target=<class 'torch.optim.adam.Adam'>,
                  lr=0.05,
                  eps=1e-15,
                  max_norm=None,
                  weight_decay=0
              ),
              'scheduler': None
          },
          'scaling': {
              'optimizer': AdamOptimizerConfig(
                  _target=<class 'torch.optim.adam.Adam'>,
                  lr=0.005,
                  eps=1e-15,
                  max_norm=None,
                  weight_decay=0
              ),
              'scheduler': ExponentialDecaySchedulerConfig(
                  _target=<class 'nerfstudio.engine.schedulers.ExponentialDecayScheduler'>,
                  lr_pre_warmup=1e-08,
                  lr_final=0.001,
                  warmup_steps=0,
                  max_steps=30000,
                  ramp='cosine'
              )
          },
          'rotation': {
              'optimizer': AdamOptimizerConfig(
                  _target=<class 'torch.optim.adam.Adam'>,
                  lr=0.001,
                  eps=1e-15,
                  max_norm=None,
                  weight_decay=0
              ),
              'scheduler': None
          },
          'camera_opt': {
              'optimizer': AdamOptimizerConfig(
                  _target=<class 'torch.optim.adam.Adam'>,
                  lr=0.001,
                  eps=1e-15,
                  max_norm=None,
                  weight_decay=0
              ),
              'scheduler': ExponentialDecaySchedulerConfig(
                  _target=<class 'nerfstudio.engine.schedulers.ExponentialDecayScheduler'>,
                  lr_pre_warmup=1e-08,
                  lr_final=5e-05,
                  warmup_steps=0,
                  max_steps=30000,
                  ramp='cosine'
              )
          }
      },
      vis='viewer_beta',
      data=PosixPath('data/posters_v3'),
      prompt=None,
      relative_model_dir=PosixPath('nerfstudio_models'),
      load_scheduler=True,
      steps_per_save=2000,
      steps_per_eval_batch=100,
      steps_per_eval_image=100,
      steps_per_eval_all_images=100000,
      max_num_iterations=30000,
      mixed_precision=False,
      use_grad_scaler=False,
      save_only_latest_checkpoint=True,
      load_dir=None,
      load_step=None,
      load_config=None,
      load_checkpoint=None,
      log_gradients=False,
      gradient_accumulation_steps={'camera_opt': 100, 'color': 10, 'shs': 10}
  )
  ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
             Saving config to: outputs/posters_v3/gaussian-splatting/2023-12-08_224938/config.yml experiment_config.py:141
             Saving checkpoints to:                                                                         trainer.py:135
             outputs/posters_v3/gaussian-splatting/2023-12-08_224938/nerfstudio_models
             Using image downscale factor of 2                                                    colmap_dataparser.py:471
  [22:49:40] Caching / undistorting train images                                            full_images_datamanager.py:128
  [22:49:45] Caching / undistorting eval images                                             full_images_datamanager.py:199
  ╭─────────────── viser ───────────────╮
  │             ╷                       │
  │   HTTP      │ http://0.0.0.0:7007   │
  │   Websocket │ ws://0.0.0.0:7007     │
  │             ╵                       │
  ╰─────────────────────────────────────╯
  [NOTE] Not running eval iterations since only viewer is enabled.
  Use --vis {wandb, tensorboard, viewer+wandb, viewer+tensorboard} to run with eval.
  No Nerfstudio checkpoint to load, so training from scratch.
  Disabled comet/tensorboard/wandb event writers
  /home/user/.local/lib/python3.10/site-packages/torchvision/transforms/functional.py:1603: UserWarning: The default value of the antialias parameter of all the resizing transforms (Resize(), RandomResizedCrop(), etc.) will change from None to True in v0.17, in order to be consistent across the PIL and Tensor backends. To suppress this warning, directly pass antialias=True (recommended, future default), antialias=None (current default, which means False for Tensors and True for PIL), or antialias=False (only works on Tensors - PIL will still use antialiasing). This also applies if you are using the inference transforms from the models weights: update the call to weights.transforms(antialias=True).
    warnings.warn(
  Printing profiling stats, from longest to shortest duration in seconds
  Trainer.train_iteration: 0.9369
  VanillaPipeline.get_train_loss_dict: 0.9365
  Traceback (most recent call last):
    File "/home/user/.local/bin/ns-train", line 8, in <module>
      sys.exit(entrypoint())
    File "/home/user/nerfstudio/nerfstudio/scripts/train.py", line 262, in entrypoint
      main(
    File "/home/user/nerfstudio/nerfstudio/scripts/train.py", line 247, in main
      launch(
    File "/home/user/nerfstudio/nerfstudio/scripts/train.py", line 189, in launch
      main_func(local_rank=0, world_size=world_size, config=config)
    File "/home/user/nerfstudio/nerfstudio/scripts/train.py", line 100, in train_loop
      trainer.train()
    File "/home/user/nerfstudio/nerfstudio/engine/trainer.py", line 253, in train
      loss, loss_dict, metrics_dict = self.train_iteration(step)
    File "/home/user/nerfstudio/nerfstudio/utils/profiler.py", line 127, in inner
      out = func(*args, **kwargs)
    File "/home/user/nerfstudio/nerfstudio/engine/trainer.py", line 471, in train_iteration
      _, loss_dict, metrics_dict = self.pipeline.get_train_loss_dict(step=step)
    File "/home/user/nerfstudio/nerfstudio/utils/profiler.py", line 127, in inner
      out = func(*args, **kwargs)
    File "/home/user/nerfstudio/nerfstudio/pipelines/base_pipeline.py", line 306, in get_train_loss_dict
      model_outputs = self._model(ray_bundle)  # train distributed data parallel model if world_size > 1
    File "/home/user/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
      return forward_call(*args, **kwargs)
    File "/home/user/nerfstudio/nerfstudio/models/base_model.py", line 143, in forward
      return self.get_outputs(ray_bundle)
    File "/home/user/nerfstudio/nerfstudio/models/gaussian_splatting.py", line 588, in get_outputs
      rgbs = SphericalHarmonics.apply(n, viewdirs, colors_crop)
    File "/home/user/.local/lib/python3.10/site-packages/torch/autograd/function.py", line 506, in apply
      return super().apply(*args, **kwargs)  # type: ignore[misc]
    File "/home/user/.local/lib/python3.10/site-packages/gsplat/sh.py", line 39, in forward
      assert coeffs.shape[-2] == num_sh_bases(degree)
  AssertionError
  ```
  
  Any help would be much appreciated.