How is the TCAS negotation information encoded? As I understand, according to this FAA booklet on TCAS, transponders will engage in cycles of interrogations and responses to establish if a collision may occur.

The next step is for both TCAS systems to negotiate respective resolution advisories. My question is how the negotiation takes place in the TCAS system internally, and in particular what communication protocol takes place for the negotiation.

To elaborate, as an example, a browser can make an HTTP request to a server (like this site) which follows protocols as to what information should be provided and how it should be encoded.

Along the same vein, how is information encoded for TCAS systems to negotiate? Since I don't know of any open source TCAS used in commercial aviation, I can't inspect code.
TCAS air-to-air messages use the same format and frequencies as "Mode S" surveillance radar. They're defined (in great and overwhelming standardesque detail) by a document called "ICAO Annex 10".

ICAO wants serious money for a copy of it -- and of course if you're producing actual avionics nothing less than the official source will do -- but for satisfying idle curiosity it's often possible to find a draft or bootleg by persistent googling.

The Mode S message format is also used for ADS-B broadcasts, so equipment to receive and decode them is relatively easily available to hobbyists. The lower layers of open-source ADS-B software will also be generally relevant.