If you lose accuracy during conversion w/o quantization/pruning, that would be a bug. File an issue on TFLite repo w/ detailed steps to reproduce.
If you apply quantization/pruning, you should expect accuracy loss. You are basically trading accuracy for faster speed and smaller model size.