Quantization, Llm, Large Language Models, Langchain, Text Generation.
decimals without losing much precision. FP32, FP16, BFloat16 data types To be clear, while training occurs, we need all the precision we can take. But prioritizing speed and size over the 6th decimal point makes sense for inference. Can we decrease the memory usage from bfloat16 to, say, int8?