maxwoe commited on
Commit
fca8b86
·
verified ·
1 Parent(s): 8ede82c

Local average decoding, benchmark numbers and comparison with other models

Browse files
Files changed (3) hide show
  1. README.md +73 -25
  2. config.json +3 -2
  3. model_cgd.py +29 -2
README.md CHANGED
@@ -25,7 +25,7 @@ model-index:
25
  name: COCO test photos (1,030 images), rotated and saved as JPEG
26
  metrics:
27
  - type: mae
28
- value: 3.77
29
  name: Mean absolute error (degrees)
30
  ---
31
 
@@ -34,12 +34,13 @@ model-index:
34
  **[Try the interactive demo](https://huggingface.co/spaces/maxwoe/image-rotation-angle-estimation)** | **[GitHub](https://github.com/maxwoe/image-rotation-angle-estimation)** | **[Paper](https://arxiv.org/abs/2603.25351)**
35
 
36
  Estimates by how many degrees a photo is rotated away from upright, anywhere in the full 360° range, so that it can be turned back.
37
- On COCO test photos the mean absolute error is 3.77° and 97% of the predictions are within 10°.
 
38
 
39
  ![Five rotated photos and the same photos turned back by the predicted angle](example.jpg)
40
 
41
  The model is a **ConvNeXt V2 Base** backbone with a **Circular Gaussian Distribution (CGD)** head.
42
- It outputs a probability distribution over 360 one-degree bins and returns the most probable angle, so there is no discontinuity at 0°/360°.
43
 
44
  ## Usage
45
 
@@ -77,21 +78,63 @@ angle, confidence = model.predict_angle(image, return_confidence=True)
77
  1,030 COCO test photos, each rotated by a random angle and cropped to the largest rectangle that contains no border; means over five sets of random angles.
78
  In the first column the rotated image is saved once as JPEG (quality 90) before the model sees it, which is how rotated photos exist in practice.
79
  In the second column it is passed to the model without ever being saved.
 
80
 
81
  | Metric | Saved as JPEG after rotation | Never saved after rotation |
82
  |---|---|---|
83
- | Mean absolute error | 3.77° | 4.45° |
84
- | Median error | 1.18° | 1.40° |
85
- | Within 2° | 71.4% | 64.5% |
86
- | Within 5° | 93.7% | 89.7% |
87
- | Within 10° | 97.4% | 95.8% |
88
- | 90th percentile error | 3.78° | 5.06° |
89
- | 95th percentile error | 5.76° | 8.43° |
90
- | RMSE | 17.31° | 18.24° |
91
-
92
- The error is nearly the same across JPEG quality settings (3.87° at quality 95, 3.75° at quality 75).
93
- The checkpoint is one of three training runs of this configuration, which reach 3.70°, 3.77° and 3.81° (mean 3.76°).
94
- The strongest earlier model, OAD-360, reaches 6.09° on the same images.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
95
 
96
  ## Confidence
97
 
@@ -106,13 +149,13 @@ The table shows the effect on the test views that were saved as JPEG after rotat
106
 
107
  | Confidence at least | Views answered | Within 10° among those | Mean absolute error among those |
108
  |---|---|---|---|
109
- | 0.50 | 99.0% | 97.9% | 3.30° |
110
- | 0.90 | 97.3% | 98.5% | 2.91° |
111
- | 0.95 | 95.9% | 98.7% | 2.63° |
112
- | 0.98 | 92.4% | 99.3% | 2.13° |
113
- | 0.99 | 84.1% | 99.6% | 1.76° |
114
 
115
- On views that were never saved after rotation, the same thresholds answer fewer views: 0.98 answers 86.2% of them, with 98.9% within 10°.
116
  On noise images that have no orientation at all, the median confidence is 0.09 and at most 3% of them reach 0.9.
117
 
118
  What the confidence cannot do:
@@ -135,20 +178,23 @@ What the confidence cannot do:
135
  A model for this task can score well without looking at the image content: when a decoded JPEG photo is rotated, the 8x8 block grid of the JPEG codec turns with it and reveals the angle.
136
  This model was trained under the trace-free protocol of the paper, in which every training image is saved as JPEG after rotation, and it was then tested for reliance on such traces:
137
 
138
- - **Content-free probe.** Noise images without any orientation, with and without a JPEG grid, are rotated and shown to the model, which should then be at chance. It meets both criteria of the probe in six of seven conditions. In the seventh, 14.0% of its predictions land within 5° of the grid angle (chance: 11.1%); the upper confidence bound of this excess is 5.1 points, against a limit of 5 points fixed in advance, so it misses that criterion by 0.1 point.
139
  - **Cue conflict.** When a grid is stamped at an angle that contradicts the content, the model follows the content; the grid attracts at most 6% of the predictions.
140
  - **Test photos.** A rotated source grid changes the share of predictions within 2° by -0.4 points, and the interpolation pattern of the rotation by 1.5 points; both are inside the limit of 5 points.
141
 
 
142
  The scripts for these checks are in the GitHub repository.
143
 
144
  ## Checkpoints
145
 
146
  | Checkpoint | Training data | Error, saved as JPEG after rotation | Error, never saved after rotation |
147
  |---|---|---|---|
148
- | `cgd_convnextv2_base_coco2017.ckpt` (default) | COCO 2017 | 3.77° | 4.45° |
149
  | `cgd_mambaout_base_coco2017.ckpt` (legacy) | COCO 2017 | 24.45° | 2.84° |
150
  | `cgd_mambaout_base_coco2014.ckpt` (legacy) | COCO 2014 | 21.05° | 3.71° |
151
 
 
 
152
  **Do not use the legacy checkpoints for real photos.**
153
  They were trained on images that were rotated but never saved again, and they learned to read the rotated JPEG grid described above.
154
  That makes them very precise on images that were never saved after rotation and unreliable on everything else.
@@ -156,20 +202,22 @@ They remain available so that earlier results stay reproducible, and can be load
156
 
157
  ## Model Details
158
 
159
- - **Method:** Circular Gaussian Distribution (CGD), 360 bins, sigma = 6.0°, KL divergence against soft Gaussian labels, angle decoded by argmax
 
160
  - **Backbone:** ConvNeXt V2 Base (`convnextv2_base.fcmae_ft_in22k_in1k`), pretrained on ImageNet, fine-tuned end to end
161
  - **Size:** 88 million parameters, 352 MB (weights only)
162
  - **Input:** 224 x 224 pixels, ImageNet normalization
163
  - **Training data:** COCO 2017 train split without the test images (117,372 images), 10% held out for validation
164
  - **Training images:** rotated by a random angle in [0°, 360°), cropped to the largest rectangle without border, saved as JPEG at a random quality between 40 and 90; the interpolation method of the rotation is drawn at random
165
  - **Optimizer:** AdamW, learning rate reduced on plateau, early stopping
166
- - **Confidence:** probability mass within 10° of the predicted angle, mapped by a logistic curve on its logit (slope 3.010, intercept -1.385) that was fitted on the validation split
167
 
168
  ## Paper and Citation
169
 
170
  The method comparison behind this model is described in [Image Rotation Angle Estimation: Five Circular-Aware Methods Across Sixteen Architectures](https://arxiv.org/abs/2603.25351).
171
  The first arXiv version (v1) predates the trace-free protocol: its COCO results (for example 2.84°) are those of the legacy checkpoints.
172
  Version 2 (17 November 2026) describes the protocol and the model published here.
 
173
 
174
  ```bibtex
175
  @misc{woehrer2026irae,
 
25
  name: COCO test photos (1,030 images), rotated and saved as JPEG
26
  metrics:
27
  - type: mae
28
+ value: 3.71
29
  name: Mean absolute error (degrees)
30
  ---
31
 
 
34
  **[Try the interactive demo](https://huggingface.co/spaces/maxwoe/image-rotation-angle-estimation)** | **[GitHub](https://github.com/maxwoe/image-rotation-angle-estimation)** | **[Paper](https://arxiv.org/abs/2603.25351)**
35
 
36
  Estimates by how many degrees a photo is rotated away from upright, anywhere in the full 360° range, so that it can be turned back.
37
+ On COCO test photos the mean absolute error is 3.71° and 97% of the predictions are within 10°.
38
+ With the simpler decoding of the paper, the same checkpoint reaches 3.77° (see [Evaluation](#evaluation)).
39
 
40
  ![Five rotated photos and the same photos turned back by the predicted angle](example.jpg)
41
 
42
  The model is a **ConvNeXt V2 Base** backbone with a **Circular Gaussian Distribution (CGD)** head.
43
+ It outputs a probability distribution over 360 one-degree bins and reads the angle from the bins around the most probable one, so there is no discontinuity at 0°/360°.
44
 
45
  ## Usage
46
 
 
78
  1,030 COCO test photos, each rotated by a random angle and cropped to the largest rectangle that contains no border; means over five sets of random angles.
79
  In the first column the rotated image is saved once as JPEG (quality 90) before the model sees it, which is how rotated photos exist in practice.
80
  In the second column it is passed to the model without ever being saved.
81
+ The test follows the [benchmark protocol](https://github.com/maxwoe/image-rotation-angle-estimation/blob/main/BENCHMARK.md) of the repository, which any model can be scored on.
82
 
83
  | Metric | Saved as JPEG after rotation | Never saved after rotation |
84
  |---|---|---|
85
+ | Mean absolute error | 3.71° | 4.38° |
86
+ | Median error | 1.11° | 1.33° |
87
+ | Within 2° | 73.8% | 66.4% |
88
+ | Within 5° | 93.9% | 90.1% |
89
+ | Within 10° | 97.3% | 95.9% |
90
+ | 90th percentile error | 3.74° | 4.99° |
91
+ | 95th percentile error | 5.60° | 8.24° |
92
+ | RMSE | 17.32° | 18.24° |
93
+
94
+ The error is nearly the same across JPEG quality settings (3.81° at quality 95, 3.69° at quality 75).
95
+
96
+ The paper uses a simpler decoding, so its numbers are slightly higher than the ones above.
97
+ It takes the most probable bin as the angle (argmax), which limits the answer to whole degrees.
98
+ The released code takes the circular mean of the bins within 10° of that bin.
99
+ The network and its weights are the same in both cases:
100
+
101
+ | Decoding | Saved as JPEG after rotation | Never saved after rotation |
102
+ |---|---|---|
103
+ | Most probable bin, as in the paper | 3.77° | 4.45° |
104
+ | Mean of the bins within 10°, as in the released code and on this card | 3.71° | 4.38° |
105
+
106
+ The paper gives the configuration as the mean of three training runs (3.76° and 4.45°), of which this checkpoint is one.
107
+ Its appendix on decoding reports the same effect for a soft decoding over all bins (3.76° against 3.68°).
108
+
109
+ ## Comparison with other models
110
+
111
+ Two other open models estimate the rotation angle over the full circle.
112
+ [OAD-360](https://github.com/pidahbus/deep-image-orientation-angle-detection) (Maji and Bose, 2020) is the strongest earlier model in the paper, and it was scored for the paper on the test views of this card.
113
+ [RightWayUp](https://github.com/ortusaitech/rightwayup) (ORTUS AI, September 2026) is a family of models whose output head follows the CGD formulation used here, and ORTUS AI published results of its models on the same test photos.
114
+ The table shows means or totals over five sets of random angles (5,150 views).
115
+
116
+ | | This model | RightWayUp Max | RightWayUp Fast | OAD-360 |
117
+ |---|---|---|---|---|
118
+ | Parameters | 88M | 306M | 23M | 86M |
119
+ | Training images | 117,372 | about 1.24 million | about 1.24 million | most of COCO |
120
+ | Median error, saved as JPEG | 1.11° | 1.21° | not published | 2.08° |
121
+ | Within 2°, saved as JPEG | 73.8% | 69.2% | not published | 48.7% |
122
+ | Within 10°, saved as JPEG after rotation | 97.3% | 98.4% | 96.4% | 93.0% |
123
+ | Within 10°, never saved after rotation | 95.9% | 98.8% | 96.5% | 93.1% |
124
+ | More than 10° off, saved as JPEG | 140 | 83 | not published | 361 |
125
+ | Upside down (150° or more off), saved as JPEG | 35 | 5 | not published | not recorded |
126
+ | Mean error, saved as JPEG | 3.71° | 2.13° | not published | 6.09° |
127
+ | Trace check, conditions passed | 7 of 7 | not run | not run | not run |
128
+
129
+ On a typical photo saved as JPEG this model is the most precise of the three models with published numbers: it has the lowest median error and the largest share of predictions within 2°.
130
+ It reaches this with less than a third of the parameters of RightWayUp Max and a tenth of its training images.
131
+ RightWayUp Max makes the fewest large errors, which is why its mean error is the lowest.
132
+ The trace check is the one of the [benchmark protocol](https://github.com/maxwoe/image-rotation-angle-estimation/blob/main/BENCHMARK.md).
133
+ The RightWayUp models pass a check of ORTUS AI's own on images without content.
134
+
135
+ The RightWayUp numbers are taken from its [benchmark notes](https://github.com/ortusaitech/rightwayup/blob/main/benchmarks/woehrer-2026/BENCHMARK-NOTES.md).
136
+ ORTUS AI builds the views with the same code but pairs the photos with the angles in another order, so the totals over the five sets are comparable and single views are not.
137
+ The paper itself compares five methods on three backbones under the same protocol.
138
 
139
  ## Confidence
140
 
 
149
 
150
  | Confidence at least | Views answered | Within 10° among those | Mean absolute error among those |
151
  |---|---|---|---|
152
+ | 0.50 | 99.0% | 97.8% | 3.25° |
153
+ | 0.90 | 97.3% | 98.4% | 2.85° |
154
+ | 0.95 | 95.9% | 98.7% | 2.57° |
155
+ | 0.98 | 92.4% | 99.3% | 2.07° |
156
+ | 0.99 | 84.4% | 99.6% | 1.74° |
157
 
158
+ On views that were never saved after rotation, the same thresholds answer fewer views: 0.98 answers 86.5% of them, with 98.9% within 10°.
159
  On noise images that have no orientation at all, the median confidence is 0.09 and at most 3% of them reach 0.9.
160
 
161
  What the confidence cannot do:
 
178
  A model for this task can score well without looking at the image content: when a decoded JPEG photo is rotated, the 8x8 block grid of the JPEG codec turns with it and reveals the angle.
179
  This model was trained under the trace-free protocol of the paper, in which every training image is saved as JPEG after rotation, and it was then tested for reliance on such traces:
180
 
181
+ - **Content-free probe.** Noise images without any orientation, with and without a JPEG grid, are rotated and shown to the model, which should then be at chance. It meets both criteria of the probe in all seven conditions. In the closest one, 13.5% of its predictions land within 5° of the grid angle (chance: 11.1%), and the upper confidence bound of this excess is 4.5 points, against a limit of 5 points fixed in advance. With argmax decoding the bound in that condition is 5.1 points, so the model then misses it by 0.1 point.
182
  - **Cue conflict.** When a grid is stamped at an angle that contradicts the content, the model follows the content; the grid attracts at most 6% of the predictions.
183
  - **Test photos.** A rotated source grid changes the share of predictions within 2° by -0.4 points, and the interpolation pattern of the rotation by 1.5 points; both are inside the limit of 5 points.
184
 
185
+ The cue conflict and the test on photos were measured with argmax decoding.
186
  The scripts for these checks are in the GitHub repository.
187
 
188
  ## Checkpoints
189
 
190
  | Checkpoint | Training data | Error, saved as JPEG after rotation | Error, never saved after rotation |
191
  |---|---|---|---|
192
+ | `cgd_convnextv2_base_coco2017.ckpt` (default) | COCO 2017 | 3.71° | 4.38° |
193
  | `cgd_mambaout_base_coco2017.ckpt` (legacy) | COCO 2017 | 24.45° | 2.84° |
194
  | `cgd_mambaout_base_coco2014.ckpt` (legacy) | COCO 2014 | 21.05° | 3.71° |
195
 
196
+ The legacy checkpoints are decoded with the most probable bin, as in the paper.
197
+
198
  **Do not use the legacy checkpoints for real photos.**
199
  They were trained on images that were rotated but never saved again, and they learned to read the rotated JPEG grid described above.
200
  That makes them very precise on images that were never saved after rotation and unreliable on everything else.
 
202
 
203
  ## Model Details
204
 
205
+ - **Method:** Circular Gaussian Distribution (CGD), 360 bins, sigma = 6.0°, KL divergence against soft Gaussian labels
206
+ - **Decoding:** circular mean of the bins within 10° of the most probable bin (the legacy checkpoints and the paper use argmax)
207
  - **Backbone:** ConvNeXt V2 Base (`convnextv2_base.fcmae_ft_in22k_in1k`), pretrained on ImageNet, fine-tuned end to end
208
  - **Size:** 88 million parameters, 352 MB (weights only)
209
  - **Input:** 224 x 224 pixels, ImageNet normalization
210
  - **Training data:** COCO 2017 train split without the test images (117,372 images), 10% held out for validation
211
  - **Training images:** rotated by a random angle in [0°, 360°), cropped to the largest rectangle without border, saved as JPEG at a random quality between 40 and 90; the interpolation method of the rotation is drawn at random
212
  - **Optimizer:** AdamW, learning rate reduced on plateau, early stopping
213
+ - **Confidence:** probability mass within 10° of the predicted angle, mapped by a logistic curve on its logit (slope 2.998, intercept -1.344) that was fitted on the validation split
214
 
215
  ## Paper and Citation
216
 
217
  The method comparison behind this model is described in [Image Rotation Angle Estimation: Five Circular-Aware Methods Across Sixteen Architectures](https://arxiv.org/abs/2603.25351).
218
  The first arXiv version (v1) predates the trace-free protocol: its COCO results (for example 2.84°) are those of the legacy checkpoints.
219
  Version 2 (17 November 2026) describes the protocol and the model published here.
220
+ Its numbers use argmax decoding and are therefore slightly higher than the ones on this card (see [Evaluation](#evaluation)).
221
 
222
  ```bibtex
223
  @misc{woehrer2026irae,
config.json CHANGED
@@ -4,10 +4,11 @@
4
  "filename": "cgd_convnextv2_base_coco2017.ckpt",
5
  "architecture": "convnextv2_base.fcmae_ft_in22k_in1k",
6
  "input_size": 224,
 
7
  "confidence": {
8
  "window_degrees": 10,
9
- "slope": 3.0100,
10
- "intercept": -1.3845
11
  }
12
  },
13
  "Legacy: CGD + MambaOut Base (COCO 2017, conventional protocol)": {
 
4
  "filename": "cgd_convnextv2_base_coco2017.ckpt",
5
  "architecture": "convnextv2_base.fcmae_ft_in22k_in1k",
6
  "input_size": 224,
7
+ "decoding": "local_weighted_average",
8
  "confidence": {
9
  "window_degrees": 10,
10
+ "slope": 2.9976,
11
+ "intercept": -1.3445
12
  }
13
  },
14
  "Legacy: CGD + MambaOut Base (COCO 2017, conventional protocol)": {
model_cgd.py CHANGED
@@ -23,6 +23,9 @@ from loguru import logger
23
  class CircularGaussianDistribution(nn.Module):
24
  """Circular Gaussian Distribution module for 360 degree image orientation."""
25
 
 
 
 
26
  def __init__(self, num_bins: int = 360, sigma: float = 6.0):
27
  super().__init__()
28
  self.num_bins = num_bins
@@ -39,7 +42,8 @@ class CircularGaussianDistribution(nn.Module):
39
 
40
  Args:
41
  distributions: Probability distributions [B, num_bins]
42
- method: 'argmax', 'weighted_average', or 'peak_fitting'
 
43
 
44
  Returns:
45
  angles: Extracted angles in degrees [B] in [0, 360)
@@ -77,6 +81,19 @@ class CircularGaussianDistribution(nn.Module):
77
  angles[i] = self.bin_centers[peak_idx] + offset * self.bin_size
78
  else:
79
  angles[i] = self.bin_centers[peak_idx]
 
 
 
 
 
 
 
 
 
 
 
 
 
80
  else:
81
  raise ValueError(f"Unknown extraction method: {method}")
82
 
@@ -190,10 +207,20 @@ class CGDAngleEstimation(pl.LightningModule):
190
 
191
  ckpt_path = hf_hub_download(repo_id=repo_id, filename=model_info["filename"])
192
  model = cls.try_load(checkpoint_path=ckpt_path, image_size=model_info["input_size"])
193
- model.confidence_calibration = model_info.get("confidence")
194
  model.eval()
195
  return model
196
 
 
 
 
 
 
 
 
 
 
 
197
  def forward(self, x: torch.Tensor, return_logits: bool = False) -> torch.Tensor:
198
  """Forward pass returning probability distribution over angles."""
199
  logits = self.model(x)
 
23
  class CircularGaussianDistribution(nn.Module):
24
  """Circular Gaussian Distribution module for 360 degree image orientation."""
25
 
26
+ # half-width of the window of the 'local_weighted_average' decoding, in degrees
27
+ LOCAL_WINDOW_DEGREES = 10.0
28
+
29
  def __init__(self, num_bins: int = 360, sigma: float = 6.0):
30
  super().__init__()
31
  self.num_bins = num_bins
 
42
 
43
  Args:
44
  distributions: Probability distributions [B, num_bins]
45
+ method: 'argmax' (most probable bin), 'local_weighted_average' (circular mean of the
46
+ bins within 10 degrees of it), 'weighted_average' or 'peak_fitting'
47
 
48
  Returns:
49
  angles: Extracted angles in degrees [B] in [0, 360)
 
81
  angles[i] = self.bin_centers[peak_idx] + offset * self.bin_size
82
  else:
83
  angles[i] = self.bin_centers[peak_idx]
84
+ elif method == 'local_weighted_average':
85
+ # Circular mean of the bins within LOCAL_WINDOW_DEGREES of the most probable bin:
86
+ # sub-bin precision without the pull of a second peak elsewhere on the circle
87
+ half_width = int(round(self.LOCAL_WINDOW_DEGREES / self.bin_size))
88
+ peak_indices = torch.argmax(distributions, dim=1, keepdim=True)
89
+ offsets = torch.arange(-half_width, half_width + 1, device=distributions.device)
90
+ index = (peak_indices + offsets) % self.num_bins
91
+ weights = distributions.gather(1, index)
92
+ bin_angles_rad = self.bin_centers[index] * torch.pi / 180.0
93
+ avg_cos = torch.sum(weights * torch.cos(bin_angles_rad), dim=1)
94
+ avg_sin = torch.sum(weights * torch.sin(bin_angles_rad), dim=1)
95
+ angles = torch.atan2(avg_sin, avg_cos) * 180.0 / torch.pi
96
+
97
  else:
98
  raise ValueError(f"Unknown extraction method: {method}")
99
 
 
207
 
208
  ckpt_path = hf_hub_download(repo_id=repo_id, filename=model_info["filename"])
209
  model = cls.try_load(checkpoint_path=ckpt_path, image_size=model_info["input_size"])
210
+ model.apply_config(model_info)
211
  model.eval()
212
  return model
213
 
214
+ def apply_config(self, model_info):
215
+ """Take the settings of a checkpoint from its entry in config.json.
216
+
217
+ "decoding" names the method that turns the output distribution into an angle
218
+ (default: argmax, as in the paper), "confidence" holds the calibration of the
219
+ confidence value (default: none).
220
+ """
221
+ self.inference_method = model_info.get("decoding", "argmax")
222
+ self.confidence_calibration = model_info.get("confidence")
223
+
224
  def forward(self, x: torch.Tensor, return_logits: bool = False) -> torch.Tensor:
225
  """Forward pass returning probability distribution over angles."""
226
  logits = self.model(x)