Revisiting this @lewtun You’re comments proved very helpful in putting me on the right path. I’ve moved to FastAPI and have some questions about concurrency.
I’m testing locally and running within a docker container using the command:
gunicorn app:app --worker-class uvicorn.workers.UvicornWorker --workers 2 --bind 0.0.0.0:8080
When I run 10 requests (each containing 50 pieces of text to run inference on) from two separate PHP workers (run by supervisor) that call http//0.0.0.0:8080/predict/batch response times are:
app_1  | 0:00:37.766941
app_1  | 0:00:38.065883
app_1  | 0:00:35.876927
app_1  | 0:00:38.612610
app_1  | 0:00:38.268158
app_1  | 0:00:37.676483
app_1  | 0:00:36.451956
app_1  | 0:00:39.346421
app_1  | 0:00:32.831906
app_1  | 0:00:33.972420

Total time for queue of 10 requests to clear ~3’
When I run 10 requests from 10 PHP workers I get response times of:
app_1  | 0:02:57.640630
app_1  | 0:03:09.019433
app_1  | 0:03:13.338791
app_1  | 0:03:16.411248
app_1  | 0:03:16.476156
app_1  | 0:03:20.084982
app_1  | 0:03:21.052187
app_1  | 0:03:23.064294
app_1  | 0:03:23.681471
app_1  | 0:03:23.975776

So you can see first request returned at ~2’57", whilst #10 returned after ~3’23"
So, really no different. Or, rather, when using 10 workers it appears they are not really running concurrently?
My code is essentially now:
import pathlib
import uvicorn
from fastapi import FastAPI
from pydantic import BaseModel
from transformers import AutoTokenizer, AutoModelWithLMHead
from typing import List
from datetime import datetime

title_model = None
title_model_tokenizer = None

app = FastAPI()

class TitleIn(BaseModel):
    id: int
    text: str

class TitleOut(BaseModel):
    id: int
    text: str

def infer(title: str) -> str:
    tokenized_text = title_model_tokenizer.encode(title, return_tensors="pt")
    title_ids = title_model.generate(
        tokenized_text,
        num_beams=1,
        repetition_penalty=1.0,
        length_penalty=1.0,
        early_stopping=False,
        no_repeat_ngram_size=1
    )

    return title_model_tokenizer.decode(title_ids[0], skip_special_tokens=True)

@app.on_event("startup")
async def startup_event():
    global title_model, title_model_tokenizer

    model_path = "/src/model_files"
    title_model = AutoModelWithLMHead.from_pretrained(pathlib.Path(model_path).absolute()).to("cpu")
    title_model_tokenizer = AutoTokenizer.from_pretrained(pathlib.Path(model_path).absolute(), use_fast=False)

@app.post('/predict/batch', response_model=List[TitleOut])
def predict_batch(title_list: List[TitleIn]):
    batch_predictions = []

    startTime = datetime.now()

    for title in title_list:
        batch_predictions.append(TitleOut(id=title.id, text=infer(title.text)))

    print(datetime.now() - startTime)
    return batch_predictions

I feel like I’m missing something? Any further tips? Thank you.