Packets per Second Limitations in EC2
A more plausible explanation is that the xen networking path is simply expensive, the intel VFs are limited by queue count and silicon (i40e isn't a great ASIC), and the Annapurna part is really an ARM64 NPU.  NPUs have been abandoned by most silicon vendors and have a tragic history.  It's simply hard to make NPUs work right at attractive price/power/performance and at high speed versus fixed function scatter/gather I/O units coupled with general purpose CPUs running software network stacks.  The only benefit Annapurna gives EC2 over a software device model is a hard security boundary of effectively another computer inside the computer for Nitro metal as a service.  I think this is one reason why EC2 is limited to 25G while 100G has been commodity for a long time.Here is a demonstration of a software stack that can scale toward hardware limits without relying on a particular vendor https://www.slideshare.net/SeanChittenden/freebsd-vpc-introd....  This approaches 100G line rate for large packets which is what it was optimized for.  I don't know PPS at low packet size but do know what would be required to optimize that use case and it could be done pretty quickly.