cms/v2.2.0/pyg-cms_20241212_101648_120237/README.md

# Model Card for mlpf-cms-v2.2.0

This model reconstructs particles in a detector, based on the tracks and calorimeter clusters recorded by the detector.
The primary difference with respect to v2.2.0 is the inclusion of the sqrt(pt) weight term in the pT and energy regression loss.
Additionally, the model has been scaled down to ~5M parameters (previously ~100M) for more efficient inference.

## Model Details

The performance is measured with respect to generator-level jets and MET computed from Pythia particles, i.e. the truth-level jets and MET.

<details>
  <summary>Jet performance</summary>
  
  <img src="plots_checkpoint-05-3.498507/cms_pf_qcd/jet_response_iqr_over_med_pt.png" alt="ttbar jet resolution" width="300"/>
  <img src="plots_checkpoint-05-3.498507/cms_pf_ttbar/jet_response_iqr_over_med_pt.png" alt="qq jet resolution" width="300"/>
  <img src="plots_checkpoint-05-3.498507/cms_pf_ztt/jet_response_iqr_over_med_pt.png" alt="ttbar jet resolution" width="300"/>

</details>

<details>
  <summary>MET performance</summary>
  
  <img src="plots_checkpoint-05-3.498507/cms_pf_qcd/met_response_iqr_over_med.png" alt="ttbar MET resolution" width="300"/>
  <img src="plots_checkpoint-05-3.498507/cms_pf_ttbar/met_response_iqr_over_med.png" alt="qq MET resolution" width="300"/>
  <img src="plots_checkpoint-05-3.498507/cms_pf_ztt/met_response_iqr_over_med.png" alt="ttbar MET resolution" width="300"/>

</details>

### Model Description

- **Developed by:** CMS MLPF Team 
- **Model type:** transformer
- **License:** Apache License

### Model Sources

- **Repository:** https://github.com/jpata/particleflow/releases/tag/v2.2.0

## Uses
### Direct Use

This model may be used to study the physics and computational performance on ML-based reconstruction in simulation within the CMS collaboration.

### Out-of-Scope Use

This model is not intended for physics measurements on real data or for use outside the CMS collaboration.

## Bias, Risks, and Limitations

The model has only been trained on simulation data and has not been validated against real data.
The model has not been peer reviewed or published in a peer-reviewed journal.

## How to Get Started with the Model

Use the code below to get started with the model.

```
#get the code
git clone https://github.com/jpata/particleflow
cd particleflow
git checkout v2.2.0

#get the models
git clone https://huggingface.co/jpata/particleflow models
```

## Training Details
Trained on 1x A100 for 5 epochs over ~2 days.

### Training Data
The following datasets were used:
```
18G     /scratch/persistent/joosep/tensorflow_datasets/cms_pf_qcd/1/2.5.0
18G     /scratch/persistent/joosep/tensorflow_datasets/cms_pf_qcd/2/2.5.0
18G     /scratch/persistent/joosep/tensorflow_datasets/cms_pf_qcd/3/2.5.0
18G     /scratch/persistent/joosep/tensorflow_datasets/cms_pf_qcd/4/2.5.0
18G     /scratch/persistent/joosep/tensorflow_datasets/cms_pf_qcd/5/2.5.0
18G     /scratch/persistent/joosep/tensorflow_datasets/cms_pf_qcd/6/2.5.0
18G     /scratch/persistent/joosep/tensorflow_datasets/cms_pf_qcd/7/2.5.0
18G     /scratch/persistent/joosep/tensorflow_datasets/cms_pf_qcd/8/2.5.0
18G     /scratch/persistent/joosep/tensorflow_datasets/cms_pf_qcd/9/2.5.0
18G     /scratch/persistent/joosep/tensorflow_datasets/cms_pf_qcd/10/2.5.0
8.4G    /scratch/persistent/joosep/tensorflow_datasets/cms_pf_qcd_nopu/1/2.5.0
8.4G    /scratch/persistent/joosep/tensorflow_datasets/cms_pf_qcd_nopu/2/2.5.0
8.4G    /scratch/persistent/joosep/tensorflow_datasets/cms_pf_qcd_nopu/3/2.5.0
8.4G    /scratch/persistent/joosep/tensorflow_datasets/cms_pf_qcd_nopu/4/2.5.0
8.4G    /scratch/persistent/joosep/tensorflow_datasets/cms_pf_qcd_nopu/5/2.5.0
8.4G    /scratch/persistent/joosep/tensorflow_datasets/cms_pf_qcd_nopu/6/2.5.0
8.4G    /scratch/persistent/joosep/tensorflow_datasets/cms_pf_qcd_nopu/7/2.5.0
8.4G    /scratch/persistent/joosep/tensorflow_datasets/cms_pf_qcd_nopu/8/2.5.0
8.4G    /scratch/persistent/joosep/tensorflow_datasets/cms_pf_qcd_nopu/9/2.5.0
8.4G    /scratch/persistent/joosep/tensorflow_datasets/cms_pf_qcd_nopu/10/2.5.0
18G     /scratch/persistent/joosep/tensorflow_datasets/cms_pf_ttbar/1/2.5.0
18G     /scratch/persistent/joosep/tensorflow_datasets/cms_pf_ttbar/2/2.5.0
18G     /scratch/persistent/joosep/tensorflow_datasets/cms_pf_ttbar/3/2.5.0
18G     /scratch/persistent/joosep/tensorflow_datasets/cms_pf_ttbar/4/2.5.0
18G     /scratch/persistent/joosep/tensorflow_datasets/cms_pf_ttbar/5/2.5.0
18G     /scratch/persistent/joosep/tensorflow_datasets/cms_pf_ttbar/6/2.5.0
18G     /scratch/persistent/joosep/tensorflow_datasets/cms_pf_ttbar/7/2.5.0
18G     /scratch/persistent/joosep/tensorflow_datasets/cms_pf_ttbar/8/2.5.0
18G     /scratch/persistent/joosep/tensorflow_datasets/cms_pf_ttbar/9/2.5.0
19G     /scratch/persistent/joosep/tensorflow_datasets/cms_pf_ttbar/10/2.5.0
8.6G    /scratch/persistent/joosep/tensorflow_datasets/cms_pf_ttbar_nopu/1/2.5.0
8.6G    /scratch/persistent/joosep/tensorflow_datasets/cms_pf_ttbar_nopu/2/2.5.0
8.6G    /scratch/persistent/joosep/tensorflow_datasets/cms_pf_ttbar_nopu/3/2.5.0
8.6G    /scratch/persistent/joosep/tensorflow_datasets/cms_pf_ttbar_nopu/4/2.5.0
8.6G    /scratch/persistent/joosep/tensorflow_datasets/cms_pf_ttbar_nopu/5/2.5.0
8.6G    /scratch/persistent/joosep/tensorflow_datasets/cms_pf_ttbar_nopu/6/2.5.0
8.6G    /scratch/persistent/joosep/tensorflow_datasets/cms_pf_ttbar_nopu/7/2.5.0
8.6G    /scratch/persistent/joosep/tensorflow_datasets/cms_pf_ttbar_nopu/8/2.5.0
8.6G    /scratch/persistent/joosep/tensorflow_datasets/cms_pf_ttbar_nopu/9/2.5.0
8.6G    /scratch/persistent/joosep/tensorflow_datasets/cms_pf_ttbar_nopu/10/2.5.0
18G     /scratch/persistent/joosep/tensorflow_datasets/cms_pf_ztt/1/2.5.0
18G     /scratch/persistent/joosep/tensorflow_datasets/cms_pf_ztt/2/2.5.0
18G     /scratch/persistent/joosep/tensorflow_datasets/cms_pf_ztt/3/2.5.0
18G     /scratch/persistent/joosep/tensorflow_datasets/cms_pf_ztt/4/2.5.0
18G     /scratch/persistent/joosep/tensorflow_datasets/cms_pf_ztt/5/2.5.0
18G     /scratch/persistent/joosep/tensorflow_datasets/cms_pf_ztt/6/2.5.0
18G     /scratch/persistent/joosep/tensorflow_datasets/cms_pf_ztt/7/2.5.0
18G     /scratch/persistent/joosep/tensorflow_datasets/cms_pf_ztt/8/2.5.0
18G     /scratch/persistent/joosep/tensorflow_datasets/cms_pf_ztt/9/2.5.0
18G     /scratch/persistent/joosep/tensorflow_datasets/cms_pf_ztt/10/2.5.0
5.8G    /scratch/persistent/joosep/tensorflow_datasets/cms_pf_ztt_nopu/1/2.5.0
5.8G    /scratch/persistent/joosep/tensorflow_datasets/cms_pf_ztt_nopu/2/2.5.0
5.7G    /scratch/persistent/joosep/tensorflow_datasets/cms_pf_ztt_nopu/3/2.5.0
5.8G    /scratch/persistent/joosep/tensorflow_datasets/cms_pf_ztt_nopu/4/2.5.0
5.7G    /scratch/persistent/joosep/tensorflow_datasets/cms_pf_ztt_nopu/5/2.5.0
5.7G    /scratch/persistent/joosep/tensorflow_datasets/cms_pf_ztt_nopu/6/2.5.0
5.7G    /scratch/persistent/joosep/tensorflow_datasets/cms_pf_ztt_nopu/7/2.5.0
5.7G    /scratch/persistent/joosep/tensorflow_datasets/cms_pf_ztt_nopu/8/2.5.0
5.7G    /scratch/persistent/joosep/tensorflow_datasets/cms_pf_ztt_nopu/9/2.5.0
5.8G    /scratch/persistent/joosep/tensorflow_datasets/cms_pf_ztt_nopu/10/2.5.0
```

## Training Procedure 

```bash
#!/bin/bash
#SBATCH --partition gpu
#SBATCH --gres gpu:a100:1
#SBATCH --mem-per-gpu 300G
#SBATCH -o logs/slurm-%x-%j-%N.out

IMG=/home/software/singularity/pytorch.simg:2024-12-03
cd ~/particleflow

ulimit -n 100000
singularity exec -B /scratch/persistent --nv \
    --env PYTHONPATH=`pwd` \
    --env KERAS_BACKEND=torch \
    $IMG python3 mlpf/pipeline.py --gpus 1 \
    --data-dir /scratch/persistent/joosep/tensorflow_datasets --config parameters/pytorch/pyg-cms.yaml \
    --train --conv-type attention \
    --gpu-batch-multiplier 5 --checkpoint-freq 1 --num-workers 8 --prefetch-factor 50 --comet --ntest 1000 --test-datasets cms_pf_qcd_nopu
```

## Evaluation
```bash
#!/bin/bash
#SBATCH --partition gpu
#SBATCH --gres gpu:mig:1
#SBATCH --mem-per-gpu 100G
#SBATCH -o logs/slurm-%x-%j-%N.out

IMG=/home/software/singularity/pytorch.simg:2024-08-18
cd ~/particleflow

WEIGHTS=experiments/pyg-cms_20241212_101648_120237/checkpoints/checkpoint-05-3.498507.pth
DATASET=$1
env
singularity exec -B /local -B /scratch/persistent --nv \
     --env PYTHONPATH=`pwd` \
     --env KERAS_BACKEND=torch \
     $IMG python mlpf/pipeline.py --gpus 1 \
     --data-dir /scratch/persistent/joosep/tensorflow_datasets --config parameters/pytorch/pyg-cms.yaml \
     --test --make-plots --gpu-batch-multiplier 2 --load $WEIGHTS --ntest 10000 --dtype bfloat16 --num-workers 1 --prefetch-factor 10 --test-datasets $DATASET
```
## Citation

## Glossary

- PF: particle flow reconstruction
- MLPF: machine learning for particle flow
- CMS: Compact Muon Solenoid

## Model Card Contact

Joosep Pata, [email protected]