Is there a security concern when adding a user to sudoers? I want to add an user who is not in admin or wheel group into the sudoers file, but I don't want to risk some kind of privilege escalation. 

Are there any additional privileges for users listed in sudoers other than running sudo commands with password prompt?
Since sudo -s makes you root, don't let anyone escalate their privileges with sudo until you've got a solid grasp of why you would let a non-admin user have limited or blanket ability to become an admin or root user.

There's really nor much else you could grant besides the ability to sudo so use that sparingly or come up with a better way to control data (encryption) or not placing it on the computer in question at all if you choose to let untrusted accounts have sudo privileges.