OK, it seems like a bug in 1.6 only as upgrading to 1.7 fixed this issue!
here are the results in 1.7:
Graph mode static quantization :
Quantization is done!
Default Model:  took 895.521 ms [min/max: 895.5/895.5] ms
Quantized Model:  took 337.002 ms [min/max: 337.0/337.0] ms
default model size: 22.936544 MB
quantized model size: 5.780453 MB

Eager mode static quantization :
Default Model:  took 1211.988 ms [min/max: 1212.0/1212.0] ms
Size (MB): 22.853838
Quantized Model:  took 336.531 ms [min/max: 336.5/336.5] ms
Size (MB): 5.798671

Although the timing for eager mode static quantization is a bit off, I guess this is normal