Do you have information on the camera calibration intrinsics? If so, you can project the points in the image space into 3D space relative to the camera and calculate the viewing angle there using trigonometry. The OpenCV library should have most of what you need to do this.