ZeroDivisionError on `trainer.evaluate` if model and dataset are tiny
### System Info

- `transformers` version: 4.29.2
- Platform: Windows-10-10.0.22621-SP0
- Python version: 3.9.16
- Huggingface_hub version: 0.15.1
- Safetensors version: not installed
- PyTorch version (GPU?): 2.0.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help?

@sgugger

cc: @younesbelkada 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Consider the following snippet:
```python
from torch import nn
from transformers import Trainer
from datasets import Dataset

model = nn.Identity()
eval_dataset = Dataset.from_dict({"tokens": [1]})

trainer = Trainer(
    model,
    eval_dataset=eval_dataset,
)
metrics = trainer.evaluate()
print(metrics)
```
(Sometimes) results in 
```
Traceback (most recent call last):
  File "[sic]\demo.py", line 13, in <module>
    metrics = trainer.evaluate()
  File "[sic]\transformers\trainer.py", line 3043, in evaluate
    speed_metrics(
  File "[sic]\transformers\trainer_utils.py", line 354, in speed_metrics
    samples_per_second = num_samples / runtime
ZeroDivisionError: float division by zero
```
This is rarely an issue - only when models and datasets are tiny. The reason I am invested in resolving this is testing purposes. See for example this [Action](https://github.com/lvwerra/trl/actions/runs/5179991753/jobs/9351434458) on TRL. To keep the testing efficient, the TRL maintainers chose a small model and dataset - which sometimes caused this flaky test.

### Expected behavior

I would expect any of these:
```
1. {'eval_runtime': 0.0, 'eval_samples_per_second': 0.0, 'eval_steps_per_second': 0.0}
2. {'eval_runtime': 0.0, 'eval_samples_per_second': None, 'eval_steps_per_second': None}
3. {'eval_runtime': 0.0, 'eval_samples_per_second': torch.inf, 'eval_steps_per_second': torch.inf}
4. {'eval_runtime': 0.0}
```
Note that these cases would essentially never occur other than during tests. With other words, I think all are fine as long as there's no exception. However, I prefer option 4 personally, but I am open to suggestions. For simplicity, I'll push a simple PR to implement 4.

- Tom Aarsen