What do the Software Update options do? In Yosemite the App Store System Preferences pane has automatic updates for a few separate items:

"Install App updates"
"Install OS X updates"
"Install system data files and security updates" 

App updates would include 3rd party software as well as Pages, Numbers, iMovie, etc.

I'm not clear on the other two. My guess is OS X updates includes things like 10.10.3, iTunes, Safari, Remote Desktop Client Update, Camera RAW, etc. My guess for system data files and security updates may include things like XProtect definitions, Security Update 2015-001, etc.

Are those assumptions correct?

Would 10.11.0 when it released be considered an "OS X update" and be automatically installed?

I've found these articles (one two) with conflicting info, but an haven't located an official knowledge base article from Apple with definitive answers.
You're correct.

I have "Install App updates" and "Install system data files and security updates" checked in System Preferences and "Install OS X updates" deselected, and exactly what you describe is what I observe.

OS X Updates are user-facing updates that directly affect the user, regardless of whether the user actually uses the respective app (e.g. RAW updates).
The system/security updates are the Security Updates and File Quarantine (XProtect) definitions.

In the last 30 days according to my App Store Updates tab, the following was not auto-installed for me with "Install OS X updates" disabled:


Command Line Tools (OS X 10.10) 6.3
Digital Camera RAW Compatibility Update 6.04
Remote Desktop Client Update 3.8.4


OS X 10.11.0 would be considered a separate app on the App Store, so it wouldn't be automatically updated from a version prior to 10.11.0 regardless of the settings.