How to know if an iOS app does not access the internet via Wi-Fi or Cellular? Is there a clear indicator that an app does not access the internet via Wi-Fi or cellular? 

For instance in the Settings App there is an entry for every app installed on iOS 13. There you find an app specific toggle for „cellular data“. But some apps (eg KeePassium) do not have this toggle.

Would an app connect to the internet if its “cellular data” setting is missing?   Because every other app that can use Wi-Fi and cellular has that toggle. I am not worried about bluetooth tethering as I can turn off Bluetooth on individual app basis and cable tethering is not an issue.
iOS 12. 
There's Wi-Fi, Bluetooth and USB-Tethering that you cannot directly toggle if you're using any of those interfaces for any purpose. Only way to stop them is to turn off the connection path itself. For Wi-Fi and Bluetooth, they should be turned off globally in Settings. For USB, detaching the cable is the way to go.