the takeoff server:pip install titan-irisChoose a Model‚ÄãTakeoff supports many of the most powerful generative text models, such as Falcon, MPT, and Llama. See the supported models for more information. For information about using your own models, see the custom models.Going forward in this demo we will be using the falcon 7B instruct model. This is a good open-source model that is trained to follow instructions, and is small enough to easily inference even on CPUs.Taking off‚ÄãModels are referred to by their model id on HuggingFace. Takeoff uses port 8000 by default, but can be configured to use another port. There is also support to use a Nvidia GPU by specifying cuda for the device flag.To start the takeoff server, run:iris takeoff --model tiiuae/falcon-7b-instruct --device cpuiris takeoff --model tiiuae/falcon-7b-instruct --device cuda # Nvidia GPU requirediris takeoff --model tiiuae/falcon-7b-instruct --device cpu --port 5000 # run on port 5000 (default: 8000)You will then be directed to a login page, where you will need to create an account to proceed.
TitanML helps businesses build and deploy better, smaller, cheaper, and faster NLP models through our training, compression, and inference optimization platform.
TitanML helps businesses build and deploy better, smaller, cheaper, and faster NLP models through our training, compression, and inference optimization platform. ->: the takeoff server:pip install titan-irisChoose a Model‚ÄãTakeoff supports many of the most powerful generative text models, such as Falcon, MPT, and Llama. See the supported models for more information. For information about using your own models, see the custom models.Going forward in this demo we will be using the falcon 7B instruct model. This is a good open-source model that is trained to follow instructions, and is small enough to easily inference even on CPUs.Taking off‚ÄãModels are referred to by their model id on HuggingFace. Takeoff uses port 8000 by default, but can be configured to use another port. There is also support to use a Nvidia GPU by specifying cuda for the device flag.To start the takeoff server, run:iris takeoff --model tiiuae/falcon-7b-instruct --device cpuiris takeoff --model tiiuae/falcon-7b-instruct --device cuda # Nvidia GPU requirediris takeoff --model tiiuae/falcon-7b-instruct --device cpu --port 5000 # run on port 5000 (default: 8000)You will then be directed to a login page, where you will need to create an account to proceed.