Our server cluster consists of 20 machines, each with 10 pids of 5 threads. We'd like some way to prevent any two threads, in any pid, on any machine, from modifying the same object at the same time.
Our code's written in Python and runs on Linux, if that helps narrow things down.
Also, it's a pretty rare case that two such threads want to do this, so we'd prefer something that optimizes the "only one thread needs this object" case to be really fast, even if it means that the "one thread has locked this object and another one needs it" case isn't great.
What are some of the best practices?
if you can get the complete infrastructure for a distributed lock manager then go ahead and use that. But that infrastructure is not easy to setup! But here is a practical solution:
-designate the node with the lowest ip address as the the master node 
(that means if the node with lowest ip address hangs, a new node with lowest ip address will become new master)
-let all nodes contact the master node to get the lock on the object. 
-let the master node use native lock semantics to get the lock.
this will simplify things unless you need complete clustering infrastructure and DLM to do the job.
What are some good ways to do intermachine locking?