What is OS X's equivalent of Linux's NSS? Anyone familiar with the internals of Linux knows about NSS; it's the subsystem that services, like gethostname() and such, delegate to. The code that actually makes DNS queries or reads /etc/passwd, etc., is in the form of NSS modules. Furthermore, additional NSS modules can be compiled and linked to, allowing developers to provide different ways to lookup hosts, users, etc.

I'm trying to find the equivalent subsystem in OS X, but it seems to be something that has changed frequently over the years. Apparently an old system called NetInfo was used since the NeXtStep days, but that hasn't been around since 10.4. This was apparently superseded by something called lookupd, but I don't think it's present in Lion anymore either, since there's no lookupd process running on my system, and man lookupd redirects to dscacheutil.

What is the current system Lion uses to delegate these kinds of lookups? Does Apple provide any documentation for writing new plugins for whatever this subsystem is, to allow for different backends to provide these services?

Another way to think of this question is: Which subsystem in OSX is responsible for reading /etc/hosts and resolving lookups based on its contents?
I'm not very familiar with Linux, but it sounds like you're looking for Open Directory.  Warning: Apple uses this term to refer to all of the various parts of their directory service architecture, so if you google "Open Directory" you'll find both Apple's LDAP-based directory server, and the daemon that runs on OS X and does the function you're looking for.

Let me try to give you a historical overview of its development (and hopefully I'll remember which changes happened in which version):

NextSTEP (the OS that OS X was based on) had a daemon called lookupd that was, as I understand it, very similar to NSS.  Most of the actual data was stored in NetInfo databases (which could be either local on the computer, or accessed over the net from a server).

Early versions of OS X added another deamon, named DirectoryService, which did similar things (but a little differently); since it didn't do everything lookupd did, any query it couldn't find an answer to would get handed off to lookupd.  Result: a certain amount of redundancy and confusion.  IIRC, there were actually some queries that got directed to lookupd first, and them it might have to hand them off to DirectoryService.  There were also two different sets of plug-ins (equiv to NSS modules): those for DirectoryService and those for lookupd. Local users &amp; groups were still stored in NetInfo, but it was deprecated for network accounts in favor of LDAP.

I don't remember the details anymore, but over the versions responsibilities gradually got shifted from lookupd to DirectoryService.

In 10.5, NetInfo was removed; local users, groups, etc were stored in XML property list files under /var/db/dslocal/.  The plug-in (excuse me, they're called "connectors" now) to access them has the not-at-all-generic name "Local" (it's sometimes referred to as "dslocal" to disambiguate it a bit).  At this point, the only thing lookupd is still responsible for is DNS lookups (although it hands off multicast lookups to mDNSResponder).

In 10.6, lookupd was removed; mDNSResponder now handles all DNS lookups, both standard (unicast) and multicast.

In 10.7, the Open Directory daemon was renamed from DirectoryService to opendirectoryd.