Our server cluster consists of 20 machines, each with 10 pids of 5 threads. We'd like some way to prevent any two threads, in any pid, on any machine, from modifying the same object at the same time.
Our code's written in Python and runs on Linux, if that helps narrow things down.
Also, it's a pretty rare case that two such threads want to do this, so we'd prefer something that optimizes the "only one thread needs this object" case to be really fast, even if it means that the "one thread has locked this object and another one needs it" case isn't great.
What are some of the best practices?
Write code using immutable objects. Write objects that implement the Singleton Pattern.
Use a stable Distributed messaging technology such as IPC, webservices, or XML-RPC.
I would take a look at Twisted. They got plenty of solutions for such task. 
I wouldn't use threads in Python esp with regards to the GIL, I would look at using Processes as working applications and use a comms technology as described above for intercommunications.
Your singleton class could then appear in one of these applications and interfaced via comms technology of choice.
Not a fast solution with all the interfacing, but if done correctly should be stable.
What are some good ways to do intermachine locking?