Attempt to do some optimizations for BERT models
- Use functional as much as possible instead of creating a class instance everytime
- Precompute and store the attention scaling factor to avoid `1/sqrt(...)` every forward
- Refactor Self-Attention to group QKV weights and and increase hardware density
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
