If OS X doesn't use the X Server, then what does it use? What is the window server on OS X?
It uses Quartz. But that's a bit of a marketing term in so much as there is not one single thing called Quartz, but a group of things that work together to make the system called Quartz. Specifially, Quartz encompasses the set of systems needed to do rendering (Quartz 2D), font management, graphics card interaction (Quartz Compositor), OpenGL support, 3D acceleration and what not.

The Quartz Compositor is the piece of the Quartz bundle that provides the server and window compositing and management features normally performed by the XServer and XClient pair.

An addendum to this question might be: why didn't Apple just use X11 and the X Server?

And there's a great answer to that addendum question on Slashdot from one of the primary Quartz developers:


  What Apple is providing is an Apple-original window system that is
  graphics model agnostic, as well as a vector drawing system that maps
  very well to PDF, which is a sort of PostScript without the
  non-graphical operators. This is packaged under the name 'Quartz' for
  easy reference by Marketing types.
  
  The window system is designed to support both buffered (like an
  offscreen PixMap) and unbuffered windows, and is graphics model
  agnostic, working equally well with QuickDraw, OpenGL, the Quartz
  drawing engine, X11, and third party solutions, and managing window
  geometry for the Classic, Carbon, and Cocoa environments. The server
  portion is a hybridization of screen arbiter and compositor models
  (and if that's all Geek to you, don't worry about it).
  
  The Quartz drawing engine supports drawing primitives similar to the
  graphics primitives that might be found in the DPSClient
  single-operator primitives library for X and NeXTSTEP. There are no
  math and flow control primitives, as these can be done more
  efficiently in the native compiled code. There are no DPS or PS
  wrappers, as this optimization for server-side graphics is not needed
  in the Quartz client-side graphics model.
  
  The operations provide imaging and path construction and filling
  operations as well as some interesting other bits that map well into
  the direction that 2D drawing is headed. (See Longhorn, or the X
  raster projects.) The drawing engine can output to rasters (like a
  window!), as well as PS and PDF streams to feed printers. The Mac OS X
  printing system takes advantage of the capabilities of Quartz to
  support all sorts of printers, and make the life of printer driver
  developers much, much easier.
  
  Things we'd need to add/extend in X Window software
  (protocol+server+manager+fonts+...):
  
  
  Extend font server and services to vend outlines and antialiased
  masks, support more font types, handle font subsetting.
  Extend
  drawing primitives to include PS-like path operations.
  Add
  dithering and phase controls.
  Add ColorSync support for drawing and
  imaging operations, display calibration 
  Add broad alpha channel
  support and Porter-Duff compositing, both for drawing in a window and
  for interactions between windows. 
  Add support for general affine
  transforms of windows 
  Add support for mesh-warps of windows
  Make
  sure that OpenGL and special video playback hardware support is
  integrated, and behaves well with all above changes.
  We find that
  we typically stream 200 Mb/sec of commands and textures for
  interactive OpenGL use, so transport efficiency could be an issue.
  
  
  So, yes, it looks like we can use X for Quartz. All we need do is
  define extensions for and upgrade the font server, add dithering with
  phase controls to the X marking engine, add a transparency model to X
  imaging with Porter-Duff compositing support, make sure GLX gets in,
  upgrade the window buffering to include transparency, mesh warps, and
  really good resampling, and maybe augment the transport layer a bit.
  
  Ummm... There doesn't appear to be much code left from the original X
  server in the drawing path or windowing machinery, and it doesn't
  appear that apps relying on these extensions can work with any other X
  server. Just what did we gain from this?
  
  Oh, yeah. My mom can run an xterm session on her desktop now without
  downloading the Apple X11 package, a shareware X server or buying a
  software package.
  
  Been there, evaluated that.