Colab TPU restarting due to protobuf incompatibility
I am trying to train a TemporalFusionTransformer model from pytorch-forecasting on the free colab TPU's.
Dependecies: !pip install pytorch-forecasting==0.9.0 !pip install --upgrade pandas==1.2.5 !pip install cloud-tpu-client==0.10 torch==1.12.0 https://storage.googleapis.com/tpu-pytorch/wheels/colab/torch_xla-1.12-cp37-cp37m-linux_x86_64.whl When I install the last one, I get this pip error message:
Installing collected packages: protobuf Attempting uninstall: protobuf Found existing installation: protobuf 3.9.2 Uninstalling protobuf-3.9.2: Successfully uninstalled protobuf-3.9.2 ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. tensorflow 2.8.2+zzzcolab20220719082949 requires protobuf<3.20,>=3.9.2, but you have protobuf 3.20.2 which is incompatible. 695
tensorflow 2.8.2+zzzcolab20220719082949 requires tensorboard<2.9,>=2.8, but you have tensorboard 2.10.0 which is incompatible. tensorboard 2.10.0 requires protobuf<3.20,>=3.9.2, but you have protobuf 3.20.2 which is incompatible. Successfully installed protobuf-3.20.2
My TFT code is as follows:
``` max_prediction_length = len(test) max_encoder_length = 4*max_prediction_length
training_cutoff = df_19344_tmp["time_idx"].max() - max_prediction_length
training = TimeSeriesDataSet( train, time_idx='time_idx', target='occupancy', group_ids=['property_id'], min_encoder_length=1, max_encoder_length=max_encoder_length, min_prediction_length=1, max_prediction_length=max_prediction_length, static_categoricals=['property_id'], static_reals=[], time_varying_known_categoricals=[], time_varying_known_reals=['time_idx', 'sin_day', 'cos_day', 'sin_month', 'cos_month', 'sin_year', 'cos_year'], time_varying_unknown_categoricals=[], time_varying_unknown_reals=['occupancy'], target_normalizer=GroupNormalizer( groups=['property_id'], transformation="softplus" ), # lags=['7-bookings', '14-bookings'], add_relative_time_idx=True, add_target_scales=True, add_encoder_length=True, allow_missing_timesteps=True )
validation = TimeSeriesDataSet.from_dataset(training, train, predict=True, stop_randomization=True)
batch_size = 32 # set this between 32 to 128 train_dataloader = training.to_dataloader(train=True, batch_size=batch_size, num_workers=0) val_dataloader = validation.to_dataloader(train=False, batch_size=batch_size * 10, num_workers=0)
trainer = pl.Trainer( max_epochs=10, # accelerator='cpu', # strategy='single_tpu', accelerator='tpu', # device=device, devices=1, enable_model_summary=True, auto_lr_find=False, # clipping gradients is a hyperparameter and important to prevent divergance # of the gradient for recurrent neural networks gradient_clip_val=0.1, check_val_every_n_epoch=1, # device=device
)
tft = TemporalFusionTransformer.from_dataset( training, # not meaningful for finding the learning rate but otherwise very important learning_rate=0. 422