Transformer Architecture for NetsDB
<|reference_start|>Transformer Architecture for NetsDB: Transformers models have become the backbone of the current state-of-the-art models in language, vision, and multimodal domains. These models, at their core, utilize multi-head self-attention to selectively aggregate context, generating dynamic contextual embeddings and modeling long-range dependencies for a clear contextual understanding. Lixi et al. \cite{zhou2022serving} proposed a method to use relational databases for deploying large-scale deep learning models and created an open-source implementation called NetsDB for the same. We build upon the previous work of these authors by creating an end-to-end implementation of the Encoder part of the transformer for model serving in NetsDB. Specifically, we construct a two-block encoder that includes Multi-Head Attention and its accompanying self-attention mechanism, Layer-Norm, Dropout, FeedForward Layers, and the necessary residual connections. We load out weights from our model for distributed processing, deployment, and efficient inferencing. To prove the efficacy of our implementation, we conduct a comprehensive performance analysis by comparing it with existing implementations in PyTorch, Tensorflow, Flax, and MxNet across key metrics such as inference time and model size.<|reference_end|>
@article{kamble2024transformer,
    title={Transformer Architecture for NetsDB},
    author={Subodh Kamble, Kunal Sunil Kasodekar},
    journal={arXiv preprint arXiv:2405.04807},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.04807},
    primaryClass={cs.CV}
}