text: Title: [Usage]: how to cache the lora adapter in memory
  Body: ### Your current environment
  
  I want to build a multi-lora service, but the following code seems to reload the Lora adapter every time
  ```python
  class LoraModule(BaseModel):
      name: str
      path: str
  
  
  class UserRequest(BaseModel):
      lora_module: list[LoraModule]
      question: str
  
  
  @app.post("/")
  async def multi_loras(req: UserRequest):
      params = SamplingParams(max_tokens=512)
      tokenizer = await engine.get_tokenizer()
      messages = tokenizer.apply_chat_template(
          [{"role": "user", "content": req.question}],
          tokenize=False,
          add_generation_prompt=True,
      )
      output = []
      for i, lora in enumerate(req.lora_module):
          generator = engine.generate(
              messages,
              sampling_params=params,
              lora_request=LoRARequest(
                  lora_name=lora.name,
                  lora_path=lora.path,
                  lora_int_id=i,
              ),
              request_id=str(uuid4().hex),
          )
          final_output = None
          async for res in generator:
              final_output = res
          output.append(final_output)
      print(output)
  ```
  
  ### How would you like to use vllm
  
  I noticed in the documentation that the service started via CLI seems to cache the lora adapter in memory, but I didn't find the code to implement it. Can you tell me where to implement it?
  ```shell 
  vllm serve meta-llama/Llama-2-7b-hf \
      --enable-lora \
      --lora-modules sql-lora=$HOME/.cache/huggingface/hub/models--yard1--llama-2-7b-sql-lora-test/snapshots/0dfa347e8877a4d4ed19ee56c140fa518470028c/
  ```
  
  ### Before submitting a new issue...
  
  - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.