Apparently I’m calling random numbers but there's nothing in the call log For the last few weeks I’ve had random calls from people saying that I called them, but there is nothing in my call log. I'm running the latest iOS.

Any suggestions as to what is happening here, and should I be alarmed enough to restore my phone?
These calls are not likely coming from your phone.  Rather, some robo caller is likely spoofing the caller ID using your number.  The FCC website says 


  What can you do if your number is being spoofed?
  
  If you get calls from people saying your number is showing up on their
  caller ID, it's likely that your number has been spoofed. We suggest
  first that you do not answer any calls from unknown numbers, but if
  you do, explain that your telephone number is being spoofed and that
  you did not actually make any calls. You can also place a message on
  your voicemail letting callers know that your number is being spoofed.
  Usually scammers switch numbers frequently. It is likely that within
  hours they will no longer be using your number.