diff --git a/changes/256.feature.rst b/changes/256.feature.rst
new file mode 100644
index 00000000..5430b4fc
--- /dev/null
+++ b/changes/256.feature.rst
@@ -0,0 +1,5 @@
+Retain Objective-C objects when creating Python wrappers and release them when the
+Python wrapped is garbage collected. This means that manual ``retain`` calls and
+subsequent ``release`` or ``autorelease`` calls from Python are no longer needed with
+very few exceptions, for example when writing implementations of ``copy`` that return an
+existing object.
diff --git a/changes/256.removal.rst b/changes/256.removal.rst
new file mode 100644
index 00000000..92d3ecce
--- /dev/null
+++ b/changes/256.removal.rst
@@ -0,0 +1,5 @@
+Manual calls to ``release`` or ``autorelease`` no longer cause Rubicon
+to skip releasing an Objective-C object when its Python wrapper is
+garbage collected. This means that fewer ``retain`` than ``release`` calls will cause
+segfaults on garbage collection. Review your code carefully for unbalanced ``retain``
+and ``release`` calls before updating.
diff --git a/docs/how-to/memory-management.rst b/docs/how-to/memory-management.rst
index 37b897d5..12f916d7 100644
--- a/docs/how-to/memory-management.rst
+++ b/docs/how-to/memory-management.rst
@@ -2,6 +2,9 @@
 Memory management for Objective-C instances
 ===========================================

+Reference counting in Objective-C
+=================================
+
 Reference counting works differently in Objective-C compared to Python. Python
 will automatically track where variables are referenced and free memory when
 the reference count drops to zero whereas Objective-C uses explicit reference
@@ -13,28 +16,36 @@ When enabling automatic reference counting (ARC), the appropriate calls for
 memory management will be inserted for you at compile-time. However, since
 Rubicon Objective-C operates at runtime, it cannot make use of ARC.

-Reference counting in Rubicon Objective-C
------------------------------------------
+Reference management in Rubicon
+===============================
+
+In most cases, you won't have to manage reference counts in Python, Rubicon
+Objective-C will do that work for you. It does so by calling ``retain`` on an
+object when Rubicon creates a ``ObjCInstance`` for it on the Python side, and calling
+``autorelease`` when the ``ObjCInstance`` is garbage collected in Python. Retaining
+the object ensures it is not deallocated while it is still referenced from Python
+and releasing it again on ``__del__`` ensures that we do not leak memory.
+
+The only exception to this is when you create an object -- which is always done
+through methods starting with "alloc", "new", "copy", or "mutableCopy". Rubicon does
+not explicitly retain such objects because we own objects created by us, but Rubicon
+does autorelease them when the Python wrapper is garbage collected.

-You won't have to manage reference counts in Python, Rubicon Objective-C will do
-that work for you. It does so by tracking when you gain ownership of an object.
-This is the case when you create an Objective-C instance using a method whose
-name begins with ``alloc``, ``new``, ``copy``, or ``mutableCopy``. Rubicon
-Objective-C will then insert a ``release`` call when the Python variable that
-corresponds to the Objective-C instance is deallocated.
+Rubicon Objective-C will not keep track if you additionally manually ``retain`` an
+object. You will be responsible to insert appropriate ``release`` or ``autorelease``
+calls yourself to prevent leaking memory.

-An exception to this is when you manually ``retain`` an object. Rubicon
-Objective-C will not keep track of such retain calls and you will be
-responsible to insert appropriate ``release`` calls yourself.
+Weak references in Objective-C
+------------------------------

-You will also need to pay attention to reference counting in case of **weak
-references**. In Objective-C, creating a **weak reference** means that the
-reference count of the object is not incremented and the object will still be
+You will need to pay attention to reference counting in case of **weak
+references**. In Objective-C, as in Python, creating a weak reference means that
+the reference count of the object is not incremented and the object will be
 deallocated when no strong references remain. Any weak references to the object
 are then set to ``nil``.

-Some objects will store references to other objects as a weak reference. Such
-properties will be declared in the Apple developer documentation as
+Some Objective-C objects store references to other objects as a weak reference.
+Such properties will be declared in the Apple developer documentation as
 "@property(weak)" or "@property(assign)". This is commonly the case for
 delegates. For example, in the code below, the ``NSOutlineView`` only stores a
 weak reference to the object which is assigned to its delegate property:
diff --git a/docs/spelling_wordlist b/docs/spelling_wordlist
index b5ed95f3..18050a76 100644
--- a/docs/spelling_wordlist
+++ b/docs/spelling_wordlist
@@ -1,6 +1,9 @@
 Alea
+alloc
 Autorelease
+autorelease
 autoreleased
+autoreleases
 Bugfixes
 callables
 CPython
@@ -22,6 +25,7 @@ lookups
 macOS
 metaclass
 metaclasses
+mutableCopy
 namespace
 namespaces
 ObjC
diff --git a/src/rubicon/objc/api.py b/src/rubicon/objc/api.py
index 96104a29..a4349cfa 100644
--- a/src/rubicon/objc/api.py
+++ b/src/rubicon/objc/api.py
@@ -91,6 +91,49 @@
 # the Python objects are not destroyed if they are otherwise no Python references left.
 _keep_alive_objects = {}

+# Methods that return an object which is implicitly retained by the caller.
+# See https://clang.llvm.org/docs/AutomaticReferenceCounting.html#semantics-of-method-families.
+_RETURNS_RETAINED_FAMILIES = {"init", "alloc", "new", "copy", "mutableCopy"}
+
+
+def get_method_family(method_name: str) -> str:
+    """Returns the method family from the method name. See
+    https://clang.llvm.org/docs/AutomaticReferenceCounting.html#method-families for
+    documentation on method families and corresponding selector names."""
+    first_component = method_name.lstrip("_").split(":")[0]
+    for family in _RETURNS_RETAINED_FAMILIES:
+        if first_component.startswith(family):
+            remainder = first_component.removeprefix(family)
+            if remainder == "" or not remainder[0].islower():
+                return family
+
+    return ""
+
+
+def method_name_to_tuple(name: str) -> (str, tuple[str, ...]):
+    """
+    Performs the following transformation:
+
+    "methodWithArg0:withArg1:withArg2:" -> "methodWithArg0", ("", "withArg1", "withArg2")
+    "methodWithArg0:" -> "methodWithArg0", ("", )
+    "method" -> "method", ()
+
+    The first element of the returned tuple is the "base name" of the method. The second
+    element is a tuple with its argument names.
+    """
+    # Selectors end with a colon if the method takes arguments.
+    if name.endswith(":"):
+        first, *rest, _ = name.split(":")
+        # Insert an empty string in order to indicate that the method
+        # takes a first argument as a positional argument.
+        rest.insert(0, "")
+        rest = tuple(rest)
+    else:
+        first = name
+        rest = ()
+
+    return first, rest
+

 def encoding_from_annotation(f, offset=1):
     argspec = inspect.getfullargspec(inspect.unwrap(f))
@@ -210,6 +253,17 @@ def __call__(self, receiver, *args, convert_args=True, convert_result=True):
         else:
             converted_args = args

+        # Init methods consume their `self` argument (the receiver), see
+        # https://clang.llvm.org/docs/AutomaticReferenceCounting.html#semantics-of-init.
+        # To ensure the receiver pointer remains valid if `init` does not return `self`
+        # but a different object or None, we issue an additional retain. This needs to
+        # be done before calling the method.
+        # Note that if `init` does return the same object, it will already be in our
+        # cache and balanced with a `release` on cache retrieval.
+        method_family = get_method_family(self.name.decode())
+        if method_family == "init":
+            send_message(receiver, "retain", restype=objc_id, argtypes=[])
+
         result = send_message(
             receiver,
             self.selector,
@@ -222,13 +276,11 @@ def __call__(self, receiver, *args, convert_args=True, convert_result=True):
             return result

         # Convert result to python type if it is an instance or class pointer.
+        # Explicitly retain the instance on first handover to Python unless we
+        # received it from a method that gives us ownership already.
         if self.restype is not None and issubclass(self.restype, objc_id):
-            result = ObjCInstance(result)
-
-        # Mark for release if we acquire ownership of an object. Do not autorelease here because
-        # we might retain a Python reference while the Obj-C reference goes out of scope.
-        if self.name.startswith((b"alloc", b"new", b"copy", b"mutableCopy")):
-            result._needs_release = True
+            implicitly_owned = method_family in _RETURNS_RETAINED_FAMILIES
+            result = ObjCInstance(result, _implicitly_owned=implicitly_owned)

         return result

@@ -240,7 +292,10 @@ def __init__(self, name_start):
         super().__init__()

         self.name_start = name_start
-        self.methods = {}  # Initialized in ObjCClass._load_methods
+
+        # A dictionary mapping from a tuple of argument names to the full method name.
+        # Initialized in ObjCClass._load_methods
+        self.methods: dict[tuple[str, ...], str] = {}

     def __repr__(self):
         return f"{type(self).__qualname__}({self.name_start!r})"
@@ -258,21 +313,31 @@ def __call__(self, receiver, first_arg=_sentinel, **kwargs):
             args.insert(0, first_arg)
             rest = ("",) + order

+        # Try to use cached ObjCBoundMethod
         try:
             name = self.methods[rest]
+            meth = receiver.objc_class._cache_method(name)
+            return meth(receiver, *args)
         except KeyError:
-            if first_arg is self._sentinel:
-                specified_sel = self.name_start
-            else:
-                specified_sel = f"{self.name_start}:{':'.join(kwargs.keys())}:"
-            raise ValueError(
-                f"Invalid selector {specified_sel}. Available selectors are: "
-                f"{', '.join(sel for sel in self.methods.values())}"
-            ) from None
+            pass
+
+        # Reconstruct the full method name from arguments and look up actual method.
+        if first_arg is self._sentinel:
+            name = self.name_start
+        else:
+            name = f"{self.name_start}:{':'.join(kwargs.keys())}:"

         meth = receiver.objc_class._cache_method(name)

-        return meth(receiver, *args)
+        if meth:
+            # Update methods cache and call method.
+            self.methods[rest] = name
+            return meth(receiver, *args)
+
+        raise ValueError(
+            f"Invalid selector {name}. Available selectors are: "
+            f"{', '.join(sel for sel in self.methods.values())}"
+        ) from None

 class ObjCBoundMethod:
@@ -783,29 +848,13 @@ def objc_class(self):
             return super(ObjCInstance, type(self)).__getattribute__(self, "_objc_class")
         except AttributeError:
             # This assumes that objects never change their class after they are
-            # seen by Rubicon. There are two reasons why this may not be true:
-            #
-            # 1. Objective-C runtime provides a function object_setClass that
-            #    can change an object's class after creation, and some code
-            #    manipulates objects' isa pointers directly (although the latter
-            #    is no longer officially supported by Apple). This is not
-            #    commonly done in practice, and even then it is usually only
-            #    done during object creation/initialization, so it's basically
-            #    safe to assume that an object's class will never change after
-            #    it's been wrapped in an ObjCInstance.
-            # 2. If a memory address is freed by the Objective-C runtime, and
-            #    then re-allocated by an object of a different type, but the
-            #    Python ObjCInstance wrapper persists, Python will assume the
-            #    object is still of the old type. If a new ObjCInstance wrapper
-            #    for the same pointer is re-created, a check is performed to
-            #    ensure the type hasn't changed; this problem only affects
-            #    pre-existing Python wrappers. If this occurs, it probably
-            #    indicates an issue with the retain count on the Python side (as
-            #    the Objective-C runtime shouldn't be able to dispose of an
-            #    object if Python still has a handle to it). If this *does*
-            #    happen, it will manifest as objects appearing to be the wrong
-            #    type, and/or objects having the wrong list of attributes
-            #    available. Refs #249.
+            # seen by Rubicon. This can occur because the Objective-C runtime provides a
+            # function object_setClass that can change an object's class after creation,
+            # and some code manipulates objects' isa pointers directly (although the
+            # latter is no longer officially supported by Apple). This is not commonly
+            # done in practice, and even then it is usually only done during object
+            # creation/initialization, so it's basically safe to assume that an object's
+            # class will never change after it's been wrapped in an ObjCInstance.
             super(ObjCInstance, type(self)).__setattr__(
                 self, "_objc_class", ObjCClass(libobjc.object_getClass(self))
             )
@@ -815,7 +864,9 @@ def objc_class(self):
     def _associated_attr_key_for_name(name):
         return SEL(f"rubicon.objc.py_attr.{name}")

-    def __new__(cls, object_ptr, _name=None, _bases=None, _ns=None):
+    def __new__(
+        cls, object_ptr, _name=None, _bases=None, _ns=None, _implicitly_owned=False
+    ):
         """The constructor accepts an :class:`~rubicon.objc.runtime.objc_id` or
         anything that can be cast to one, such as a :class:`~ctypes.c_void_p`,
         or an existing :class:`ObjCInstance`.
@@ -833,10 +884,17 @@ class or a metaclass, an instance of :class:`ObjCClass` or
         :func:`register_type_for_objcclass`. Creating an :class:`ObjCInstance`
         from a ``nil`` pointer returns ``None``.

-        Rubicon currently does not perform any automatic memory management on
-        the Objective-C object wrapped in an :class:`ObjCInstance`. It is the
-        user's responsibility to ``retain`` and ``release`` wrapped objects as
-        needed, like in Objective-C code without automatic reference counting.
+        Rubicon retains an Objective-C object when it is wrapped in an
+        :class:`ObjCInstance` and autoreleases it when the :class:`ObjCInstance` is
+        garbage collected.
+
+        The only exception to this are objects returned by methods which create an
+        object (starting with "alloc", "new", "copy", or "mutableCopy"). We do not
+        explicitly retain them because we already own objects created by us, but we do
+        autorelease them on garbage collection of the Python wrapper.
+
+        This ensures that the :class:`ObjCInstance` can always be used from Python
+        without segfaults while preventing Rubicon from leaking memory.
         """

         # Make sure that object_ptr is wrapped in an objc_id.
@@ -852,68 +910,31 @@ class or a metaclass, an instance of :class:`ObjCClass` or
                 # If an ObjCInstance already exists for the Objective-C object,
                 # reuse it instead of creating a second ObjCInstance for the
                 # same object.
-                cached = cls._cached_objects[object_ptr.value]
-
-                # In a high-churn environment, it is possible for an object to
-                # be deallocated, and the same memory address be re-used on the
-                # Objective-C side, but the Python wrapper object for the
-                # original instance has *not* been cleaned up. In that
-                # situation, an attempt to wrap the *new* Objective-C object
-                # instance will cause a false positive cache hit; returning a
-                # Python object that has a class that doesn't match the class of
-                # the new instance.
-                #
-                # To prevent this, when we get a cache hit on an ObjCInstance,
-                # use the raw Objective-C API on the pointer to get the current
-                # class of the object referred to by the pointer. If there's a
-                # discrepancy, purge the cache for the memory address, and
-                # re-create the object.
-                #
-                # We do this both when the type *is* ObjCInstance (the case when
-                # instantiating a literal ObjCInstance()), and when type is an
-                # ObjCClass instance (e.g., ObjClass("Example"), which is the
-                # type of a directly instantiated instance of Example.
-                #
-                # We *don't* do this when the type *is* ObjCClass,
-                # ObjCMetaClass, as there's a race condition on startup -
-                # retrieving `.objc_class` causes the creation of ObjCClass
-                # objects, which will cause cache hits trying to re-use existing
-                # ObjCClass objects. However, ObjCClass instances generally
-                # won't be recycled or reused, so that should be safe to exclude
-                # from the cache freshness check.
+                cached_obj = cls._cached_objects[object_ptr.value]
+
+                # We can get a cache hit for methods that return an implicitly retained
+                # object. This is typically the case when:
                 #
-                # One edge case with this approach: if the old and new
-                # Objective-C objects have the same class, they won't be
-                # identified as a stale object, and they'll re-use the same
-                # Python wrapper. This effectively means id(obj) isn't a
-                # reliable instance identifier... but (a) this won't be a common
-                # case; (b) this flaw exists in pure Python and Objective-C as
-                # well, because default object identity is tied to memory
-                # allocation; and (c) the stale wrapper will *work*, because
-                # it's the correct class.
+                # 1. A `copy` returns the original object if it is immutable. This is
+                #    typically done for optimization. See
+                #    https://developer.apple.com/documentation/foundation/nscopying.
+                # 2. An `init` call returns an object which we already own from a
+                #    previous `alloc` call. See `init` handling in ObjCMethod. __call__.
                 #
-                # Refs #249.
-                if cls == ObjCInstance or isinstance(cls, ObjCInstance):
-                    cached_class_name = cached.objc_class.name
-                    current_class_name = libobjc.class_getName(
-                        libobjc.object_getClass(object_ptr)
-                    ).decode("utf-8")
-                    if (
-                        current_class_name != cached_class_name
-                        and not current_class_name.endswith(f"_{cached_class_name}")
-                    ):
-                        # There has been a cache hit, but the object is a
-                        # different class, treat this as a cache miss. We don't
-                        # *just* look for an *exact* class name match, because
-                        # some Cocoa/UIKit classes undergo a class name change
-                        # between `alloc()` and `init()` (e.g., `NSWindow`
-                        # becomes `NSKVONotifying_NSWindow`). Refs #257.
-                        raise KeyError(object_ptr.value)
-
-                return cached
+                # If the object is already in our cache, we end up owning more than one
+                # refcount. We release this additional refcount to prevent memory leaks.
+                if _implicitly_owned:
+                    send_message(object_ptr, "release", restype=objc_id, argtypes=[])
+
+                return cached_obj
             except KeyError:
                 pass

+            # Explicitly retain the instance on first handover to Python unless we
+            # received it from a method that gives us ownership already.
+            if not _implicitly_owned:
+                send_message(object_ptr, "retain", restype=objc_id, argtypes=[])
+
             # If the given pointer points to a class, return an ObjCClass instead (if we're not already creating one).
             if not issubclass(cls, ObjCClass) and object_isClass(object_ptr):
                 return ObjCClass(object_ptr)
@@ -932,7 +953,6 @@ class or a metaclass, an instance of :class:`ObjCClass` or
             super(ObjCInstance, type(self)).__setattr__(
                 self, "_as_parameter_", object_ptr
             )
-            super(ObjCInstance, type(self)).__setattr__(self, "_needs_release", False)
             if isinstance(object_ptr, objc_block):
                 super(ObjCInstance, type(self)).__setattr__(
                     self, "block", ObjCBlock(object_ptr)
@@ -944,39 +964,16 @@ class or a metaclass, an instance of :class:`ObjCClass` or

         return self

-    def release(self):
-        """Manually decrement the reference count of the corresponding objc
-        object.
-
-        The objc object is sent a dealloc message when its reference
-        count reaches 0. Calling this method manually should not be
-        necessary, unless the object was explicitly ``retain``\\ed
-        before. Objects returned from ``.alloc().init...(...)`` and
-        similar calls are released automatically by Rubicon when the
-        corresponding Python object is deallocated.
-        """
-        self._needs_release = False
-        send_message(self, "release", restype=objc_id, argtypes=[])
-
-    def autorelease(self):
-        """Decrements the receiver’s reference count at the end of the current
-        autorelease pool block.
-
-        The objc object is sent a dealloc message when its reference
-        count reaches 0. If called, the object will not be released when
-        the Python object is deallocated.
-        """
-        self._needs_release = False
-        result = send_message(self, "autorelease", restype=objc_id, argtypes=[])
-        return ObjCInstance(result)
-
     def __del__(self):
-        """Release the corresponding objc instance if we own it, i.e., if it
-        was returned by a method starting with :meth:`alloc`, :meth:`new`,
-        :meth:`copy`, or :meth:`mutableCopy` and it wasn't already explicitly
-        released by calling :meth:`release` or :meth:`autorelease`."""
-        if self._needs_release:
-            send_message(self, "release", restype=objc_id, argtypes=[])
+        # Autorelease our reference on garbage collection of the Python wrapper. We use
+        # autorelease instead of release to allow ObjC to take ownership of an object when
+        # it is returned from a factory method.
+        try:
+            send_message(self, "autorelease", restype=objc_id, argtypes=[])
+        except (NameError, TypeError):
+            # Handle interpreter shutdown gracefully where send_message might be deleted
+            # (NameError) or set to None (TypeError).
+            pass

     def __str__(self):
         """Get a human-readable representation of ``self``.
@@ -1623,44 +1620,40 @@ def _load_methods(self):
         if self.methods_ptr is not None:
             raise RuntimeError(f"{self}._load_methods cannot be called more than once")

-        methods_ptr_count = c_uint(0)
-
-        methods_ptr = libobjc.class_copyMethodList(self, byref(methods_ptr_count))
+        # Traverse superclasses and load methods.
+        superclass = self.superclass

-        if self.superclass is not None:
-            if self.superclass.methods_ptr is None:
-                with self.superclass.cache_lock:
-                    self.superclass._load_methods()
+        while superclass is not None:
+            if superclass.methods_ptr is None:
+                with superclass.cache_lock:
+                    superclass._load_methods()

             # Prime this class' partials list with a list from the superclass.
-            for first, superpartial in self.superclass.partial_methods.items():
+            for first, superpartial in superclass.partial_methods.items():
                 partial = ObjCPartialMethod(first)
                 self.partial_methods[first] = partial
                 partial.methods.update(superpartial.methods)

+            superclass = superclass.superclass
+
+        # Load methods for this class.
+        methods_ptr_count = c_uint(0)
+        methods_ptr = libobjc.class_copyMethodList(self, byref(methods_ptr_count))
+
         for i in range(methods_ptr_count.value):
             method = methods_ptr[i]
             name = libobjc.method_getName(method).name.decode("utf-8")
             self.instance_method_ptrs[name] = method

-            # Selectors end with a colon if the method takes arguments.
-            if name.endswith(":"):
-                first, *rest, _ = name.split(":")
-                # Insert an empty string in order to indicate that the method
-                # takes a first argument as a positional argument.
-                rest.insert(0, "")
-                rest = tuple(rest)
-            else:
-                first = name
-                rest = ()
+            base_name, argument_names = method_name_to_tuple(name)

             try:
-                partial = self.partial_methods[first]
+                partial = self.partial_methods[base_name]
             except KeyError:
-                partial = ObjCPartialMethod(first)
-                self.partial_methods[first] = partial
+                partial = ObjCPartialMethod(base_name)
+                self.partial_methods[base_name] = partial

-            partial.methods[rest] = name
+            partial.methods[argument_names] = name

         # Set the list of methods for the class to the computed list.
         self.methods_ptr = methods_ptr

diff --git a/tests/test_core.py b/tests/test_core.py
index 4fdedebd..47454c5a 100644
--- a/tests/test_core.py
+++ b/tests/test_core.py
@@ -6,6 +6,7 @@
 import sys
 import threading
 import unittest
+import uuid
 import weakref
 from ctypes import (
     ArgumentError,
@@ -21,6 +22,7 @@
 )
 from decimal import Decimal
 from enum import Enum
+from typing import Callable

 from rubicon.objc import (
     SEL,
@@ -31,6 +33,7 @@
     NSEdgeInsets,
     NSEdgeInsetsMake,
     NSMakeRect,
+    NSMutableArray,
     NSObject,
     NSObjectProtocol,
     NSPoint,
@@ -54,11 +57,29 @@
     send_super,
     types,
 )
-from rubicon.objc.runtime import autoreleasepool, get_ivar, libobjc, objc_id, set_ivar
+from rubicon.objc.api import get_method_family
+from rubicon.objc.runtime import (
+    autoreleasepool,
+    get_ivar,
+    libobjc,
+    load_library,
+    objc_id,
+    set_ivar,
+)
 from rubicon.objc.types import __LP64__

 from . import OSX_VERSION, rubiconharness

+appkit = load_library("AppKit")
+
+NSArray = ObjCClass("NSArray")
+NSImage = ObjCClass("NSImage")
+NSString = ObjCClass("NSString")
+
+
+class ObjcWeakref(NSObject):
+    weak_property = objc_property(weak=True)
+

 class struct_int_sized(Structure):
     _fields_ = [("x", c_char * 4)]
@@ -72,6 +93,26 @@ class struct_large(Structure):
     _fields_ = [("x", c_char * 17)]

+def assert_lifecycle(
+    test: unittest.TestCase, object_constructor: Callable[[], ObjCInstance]
+) -> None:
+    obj = object_constructor()
+
+    wr = ObjcWeakref.alloc().init()
+    wr.weak_property = obj
+
+    with autoreleasepool():
+        del obj
+        gc.collect()
+
+        test.assertIsNotNone(
+            wr.weak_property,
+            "object was deallocated before end of autorelease pool",
+        )
+
+    test.assertIsNone(wr.weak_property, "object was not deallocated")
+
+
 class RubiconTest(unittest.TestCase):
     def test_sel_by_name(self):
         self.assertEqual(SEL(b"foobar").name, b"foobar")
@@ -272,10 +313,6 @@ def test_objcprotocol_protocols(self):

     def test_objcclass_instancecheck(self):
         """isinstance works with an ObjCClass as the second argument."""
-
-        NSArray = ObjCClass("NSArray")
-        NSString = ObjCClass("NSString")
-
         self.assertIsInstance(NSObject.new(), NSObject)
         self.assertIsInstance(at(""), NSString)
         self.assertIsInstance(at(""), NSObject)
@@ -288,10 +325,6 @@ def test_objcclass_instancecheck(self):

     def test_objcclass_subclasscheck(self):
         """issubclass works with an ObjCClass as the second argument."""
-
-        NSArray = ObjCClass("NSArray")
-        NSString = ObjCClass("NSString")
-
         self.assertTrue(issubclass(NSObject, NSObject))
         self.assertTrue(issubclass(NSString, NSObject))
         self.assertTrue(issubclass(NSObject.objc_class, NSObject))
@@ -323,8 +356,6 @@ def test_objcprotocol_instancecheck(self):

     def test_objcprotocol_subclasscheck(self):
         """issubclass works with an ObjCProtocol as the second argument."""
-
-        NSString = ObjCClass("NSString")
         NSCopying = ObjCProtocol("NSCopying")
         NSCoding = ObjCProtocol("NSCoding")
         NSSecureCoding = ObjCProtocol("NSSecureCoding")
@@ -442,8 +473,6 @@ def test_method_incorrect_argument_count_send(self):

     def test_method_varargs_send(self):
         """A variadic method can be called using send_message."""
-
-        NSString = ObjCClass("NSString")
         formatted = send_message(
             NSString,
             "stringWithFormat:",
@@ -1409,7 +1438,8 @@ class Ivars(NSObject):
         self.assertEqual(r.size.height, 78)

     def test_class_properties(self):
-        """A Python class can have ObjC properties with synthesized getters and setters."""
+        """A Python class can have ObjC properties with synthesized getters and setters
+        of ObjCInstance type."""

         NSURL = ObjCClass("NSURL")

@@ -1459,6 +1489,9 @@ def getSchemeIfPresent(self):
         self.assertIsNone(box.data)

     def test_class_python_properties(self):
+        """A Python class can have ObjC properties with synthesized getters and setters
+        of Python type."""
+
         class PythonObjectProperties(NSObject):
             object = objc_property(object)

@@ -1494,9 +1527,10 @@ class PythonObject:
         properties.object = o
         self.assertIs(properties.object, o)

-        del o
-        del properties
-        gc.collect()
+        with autoreleasepool():
+            del o
+            del properties
+            gc.collect()

         self.assertIsNone(wr())

@@ -1859,76 +1893,136 @@ class TestO:

         self.assertIsNone(wr_python_object())

-    def test_objcinstance_release_owned(self):
-        # Create an object which we own.
-        obj = NSObject.alloc().init()
+    def test_objcinstance_returned_lifecycle(self):
+        """An object is retained when creating an ObjCInstance for it without implicit
+        ownership. It is autoreleased when the ObjCInstance is garbage collected.
+        """

-        # Check that it is marked for release.
-        self.assertTrue(obj._needs_release)
+        def create_object():
+            with autoreleasepool():
+                return NSString.stringWithString(str(uuid.uuid4()))

-        # Explicitly release the object.
-        obj.release()
+        assert_lifecycle(self, create_object)

-        # Check that we no longer need to release it.
-        self.assertFalse(obj._needs_release)
+    def test_objcinstance_alloc_lifecycle(self):
+        """We properly retain and release objects that are allocated but never
+        initialized."""

-        # Delete it and make sure that we don't segfault on garbage collection.
-        del obj
-        gc.collect()
+        def create_object():
+            with autoreleasepool():
+                return NSObject.alloc()

-    def test_objcinstance_autorelease_owned(self):
-        # Create an object which we own.
-        obj = NSObject.alloc().init()
+        assert_lifecycle(self, create_object)

-        # Check that it is marked for release.
-        self.assertTrue(obj._needs_release)
+    def test_objcinstance_alloc_init_lifecycle(self):
+        """An object is not additionally retained when we create and initialize it
+        through an alloc().init() chain. It is autoreleased when the ObjCInstance is
+        garbage collected.
+        """

-        # Explicitly release the object.
-        res = obj.autorelease()
+        def create_object():
+            return NSObject.alloc().init()

-        # Check that autorelease call returned the object itself.
-        self.assertIs(obj, res)
+        assert_lifecycle(self, create_object)

-        # Check that we no longer need to release it.
-        self.assertFalse(obj._needs_release)
+    def test_objcinstance_new_lifecycle(self):
+        """An object is not additionally retained when we create and initialize it with
+        a new call. It is autoreleased when the ObjCInstance is garbage collected.
+        """

-        # Delete it and make sure that we don't segfault on garbage collection.
-        del obj
-        gc.collect()
+        def create_object():
+            return NSObject.new()

-    def test_objcinstance_retain_release(self):
-        NSString = ObjCClass("NSString")
+        assert_lifecycle(self, create_object)

-        # Create an object which we don't own.
-        string = NSString.stringWithString("test")
+    def test_objcinstance_copy_lifecycle(self):
+        """An object is not additionally retained when we create and initialize it with
+        a copy call. It is autoreleased when the ObjCInstance is garbage collected.
+        """

-        # Check that it is not marked for release.
-        self.assertFalse(string._needs_release)
+        def create_object():
+            obj = NSMutableArray.alloc().init()
+            copy = obj.copy()

-        # Explicitly retain the object.
-        res = string.retain()
+            # Check that the copy is a new object.
+            self.assertIsNot(obj, copy)
+            self.assertNotEqual(obj.ptr.value, copy.ptr.value)

-        # Check that autorelease call returned the object itself.
-        self.assertIs(string, res)
+            return copy

-        # Manually release the object.
-        string.release()
+        assert_lifecycle(self, create_object)

-        # Delete it and make sure that we don't segfault on garbage collection.
-        del string
-        gc.collect()
+    def test_objcinstance_mutable_copy_lifecycle(self):
+        """An object is not additionally retained when we create and initialize it with
+        a mutableCopy call. It is autoreleased when the ObjCInstance is garbage collected.
+        """
+
+        def create_object():
+            obj = NSMutableArray.alloc().init()
+            copy = obj.mutableCopy()
+
+            # Check that the copy is a new object.
+            self.assertIsNot(obj, copy)
+            self.assertNotEqual(obj.ptr.value, copy.ptr.value)
+
+            return copy
+
+        assert_lifecycle(self, create_object)
+
+    def test_objcinstance_immutable_copy_lifecycle(self):
+        """If the same object is returned from multiple creation methods, it is still
+        freed on Python garbage collection."""
+
+        def create_object():
+            with autoreleasepool():
+                obj = NSString.stringWithString(str(uuid.uuid4()))
+                copy = obj.copy()
+
+            # Check that the copy the same object as the original.
+            self.assertIs(obj, copy)
+            self.assertEqual(obj.ptr.value, copy.ptr.value)
+
+            return obj
+
+        assert_lifecycle(self, create_object)
+
+    def test_objcinstance_init_change_lifecycle(self):
+        """We do not leak memory if init returns a different object than it
+        received in alloc."""
+
+        def create_object():
+            with autoreleasepool():
+                obj_allocated = NSString.alloc()
+                obj_initialized = obj_allocated.initWithString(str(uuid.uuid4()))
+
+            # Check that the initialized object is a different one than the allocated.
+            self.assertIsNot(obj_allocated, obj_initialized)
+            self.assertNotEqual(obj_allocated.ptr.value, obj_initialized.ptr.value)
+
+            return obj_initialized
+
+        assert_lifecycle(self, create_object)
+
+    def test_objcinstance_init_none(self):
+        """We do not segfault if init returns nil."""
+        with autoreleasepool():
+            image = NSImage.alloc().initWithContentsOfFile("/no/file/here")
+
+        self.assertIsNone(image)

     def test_objcinstance_dealloc(self):
+
         class DeallocTester(NSObject):
+            did_dealloc = False
+
             attr0 = objc_property()
             attr1 = objc_property(weak=True)

             @objc_method
             def dealloc(self):
-                self._did_dealloc = True
+                DeallocTester.did_dealloc = True

         obj = DeallocTester.alloc().init()
-        obj.__dict__["_did_dealloc"] = False

         attr0 = NSObject.alloc().init()
         attr1 = NSObject.alloc().init()
@@ -1939,10 +2033,14 @@ def dealloc(self):
         self.assertEqual(attr0.retainCount(), 2)
         self.assertEqual(attr1.retainCount(), 1)

-        # ObjC object will be deallocated, can only access Python attributes afterwards.
-        obj.release()
+        # Delete the Python wrapper and ensure that the Objective-C object is
+        # deallocated after ``autorelease`` on garbage collection. This will also
+        # trigger a decrement in the retain count of attr0.
+        with autoreleasepool():
+            del obj
+            gc.collect()

-        self.assertTrue(obj._did_dealloc, "custom dealloc did not run")
+        self.assertTrue(DeallocTester.did_dealloc, "custom dealloc did not run")
         self.assertEqual(
             attr0.retainCount(), 1, "strong property value was not released"
         )
@@ -1962,70 +2060,6 @@ def test_partial_with_override(self):
         obj.method(2)
         self.assertEqual(obj.baseIntField, 2)

-    def test_stale_instance_cache(self):
-        """Instances returned by the ObjCInstance cache are checked for staleness (#249)"""
-        # Wrap 2 classes with different method lists
-        Example = ObjCClass("Example")
-        Thing = ObjCClass("Thing")
-
-        # Create objects of 2 different types.
-        old = Example.alloc().init()
-        thing = Thing.alloc().init()
-
-        # Deliberately poison the ObjCInstance Cache, making the memory address
-        # for thing point at the "old" example. This matches what happens when a
-        # memory address is re-allocated by the Objective-C runtime
-        ObjCInstance._cached_objects[thing.ptr.value] = old
-
-        # Obtain a fresh address-based wrapper for the same Thing instance.
-        new_thing = Thing(thing.ptr.value)
-        self.assertEqual(new_thing.objc_class, Thing)
-        self.assertIsInstance(new_thing, ObjCInstance)
-
-        try:
-            # Try to access an method known to exist on Thing
-            new_thing.computeSize(NSSize(37, 42))
-        except AttributeError:
-            # If a stale wrapper is returned, new_example will be of type Example,
-            # so the expected method won't exist, causing an AttributeError.
-            self.fail("Stale wrapper returned")
-
-    def test_stale_instance_cache_implicit(self):
-        """Implicit instances returned by the ObjCInstance cache are checked for staleness (#249)"""
-        # Wrap 2 classes with different method lists
-        Example = ObjCClass("Example")
-        Thing = ObjCClass("Thing")
-
-        # Create objects of 2 different types.
-        old = Example.alloc().init()
-        example = Example.alloc().init()
-        thing = Thing.alloc().init()
-
-        # Store the reference to Thing on example.
-        example.thing = thing
-
-        # Deliberately poison the ObjCInstance cache, making the memory address
-        # for thing point at the "old" example. This matches what happens when a memory
-        # address is re-allocated by the Objective-C runtime.
-        ObjCInstance._cached_objects[thing.ptr.value] = old
-
-        # When accessing a property that returns an ObjC id, we don't have
-        # type information, so the metaclass at time of construction is ObjCInstance,
-        # rather than an instance of ObjCClass.
-        #
-        # Access the thing property to return the generic instance wrapper
-        new_thing = example.thing
-        self.assertEqual(new_thing.objc_class, Thing)
-        self.assertIsInstance(new_thing, ObjCInstance)
-
-        try:
-            # Try to access an method known to exist on Thing
-            new_thing.computeSize(NSSize(37, 42))
-        except AttributeError:
-            # If a stale wrapper is returned, new_example will be of type Example,
-            # so the expected method won't exist, causing an AttributeError.
-            self.fail("Stale wrapper returned")
-
     def test_compatible_class_name_change(self):
         """If the class name changes in a compatible way, the wrapper isn't recreated (#257)"""
         Example = ObjCClass("Example")
@@ -2181,3 +2215,12 @@ def work():
             thread.start()
             work()
             thread.join()
+
+    def test_get_method_family(self):
+        self.assertEqual(get_method_family("mutableCopy"), "mutableCopy")
+        self.assertEqual(get_method_family("mutableCopy:"), "mutableCopy")
+        self.assertEqual(get_method_family("_mutableCopy:"), "mutableCopy")
+        self.assertEqual(get_method_family("_mutableCopy:with:"), "mutableCopy")
+        self.assertEqual(get_method_family("_mutableCopyWith:"), "mutableCopy")
+        self.assertEqual(get_method_family("_mutableCopy_with:"), "mutableCopy")
+        self.assertEqual(get_method_family("_mutableCopying:"), "")

Improve correspondence between ObjectiveC objects and Python wrappers
### What is the problem or limitation you are having?

In the process of building the Toga GUI testbed, we discovered #249. A fix for this problem was included in #246; however, that fix was really a safety catch for a specific set of circumstances. The bigger question: how did that set of circumstances happen in the first place?

At present, it is possible for an object to be disposed of by Objective C, but due to delays in propagating garbage collection, the `ObjCInstance` wrapping that object may not be immediately disposed of. This means the weak reference dictionary can contain stale objects.

#246 protected against this by protecting against the specific case where a memory address was *reused* - but if a Python object points at memory that has been disposed of and *not* reused, you'll generate a segfault. This is obviously undesirable.

### Describe the solution you'd like

Functionally - it should be impossible to generate a segfault from Python code.

An idea posited by @mhsmith in [the discussion about #249](https://github.com/beeware/rubicon-objc/issues/249#issuecomment-1399472658) was that we could maintain stronger reference to the ObjC object on the Python side. This would mean we add add a `retain` calls whenever a Python-side wrapper is created; `release`-ing only when *Python* has finished with the object. The idea here would be that it's better to have larger memory footprint (or even a small leak), than to have a segfault because a Python object has a stale memory reference.

### Describe alternatives you've considered

Do nothing. The status quo is stable(ish?), and while it, may be theoretically possible to get a stale ObjCInstance object, in practice we're not seeing that problem (after #246 landed, anyway).

### Additional context

Other than the general hesitation of fiddling with a complex area of code that *appears* to work at present - the complication to watch out for here is cyclic references between Python objects and Objective C objects that prevents Objective C from ever releasing memory. It's easy to avoid segfaults by never releasing... but eventually the memory will run out.
Foundation collection instances can be cached without their helper wrappers
### Describe the bug

When an instance of a Foundation collection class is created (e.g., NSMutableDictionary), Rubicon uses a collection class (e.g., ObjCMutableDictInstance) to ensure that Pythonic helpers such as `__setitem__` are available.

It is also possible to create an ObjCInstance directly from an objc_id. If this happens, it may be cached as a raw ObjCInstance, and *not* have the helper methods.

Rubicon caches object instances based on the object pointer, performing cache eviction only when the class name of the underlying object class doesn't match. However, an NSMutableDictionary created directly, and one created from an objc_id will have the same pointer, and the same underlying class name. As a result, it's possible for a recycled collection class instance to *not* have the helper methods as a result of a cache hit.

### Steps to reproduce

I haven't worked out a simple way to reproduce this programmatically.

beeware/toga-chart#191 contains an example that can reproduce the issue with reasonable reliability. Run the example app, and resize the window continuously. I can usually trigger the exception reported by that ticket within a couple of seconds.

### Expected behavior

Cached ObjC collection instances should always be returned with the appropriate wrapper, even if the memory address has previously been used by an object of the same type.

### Screenshots

_No response_

### Environment

- Operating System: macOS
- Python version: Any
- Software versions:
  - Rubicon 0.4.9+

### Logs

```

```

### Additional context

The workaround fix is to modify the conditions for a cache eviction so that we don't *just* look at the class name, but at the class instance as well.

Even better would be to ensure that a non-wrapped instance isn't ever cached in the first place... but I'm having difficulty working out how the non-wrapped instance is being created.

Even better still would be to fix the issue with memory-based cache hits (#256).
Improve memory management documentation
Looking at old SO discussion (e.g. https://stackoverflow.com/a/6610/705086 ) it looks like when `SomeOBJCClass.alloc().init()` is called from Python code, it becomes Python's responsibility either to `.release()` the resulting object on the spot, or to mark it for GC at the bottom of event loop using `.autorelease()`.

Is that correct?

If that's correct, it must be documented...

A similar segfault occurred due to some missing `retain` calls which were added by [this commit](https://github.com/beeware/toga/pull/2075/commits/69e760b6d707871215b0ea2cf6cdc3dc383e7449).
I was revisiting this again due to https://github.com/beeware/toga/issues/2468, and wanted to jot down some options before I forget them again.

#### Background

The changes from #201 were introduced as convenience to rubicon-objc users. We cannot use [Automatic Reference Counting](https://clang.llvm.org/docs/AutomaticReferenceCounting.html) from Python, which means that objects created by `alloc`, `new`, `copy` or `mutableCopy` calls are never automatically released when no longer needed. Ensuring that memory is freed properly, we would need to explicitly `release` items when garbage collected, i.e., strictly adhere to https://developer.apple.com/library/archive/documentation/Cocoa/Conceptual/MemoryMgmt/Articles/mmRules.html. In practice, this was never done in the toga codebase, resulting in widespread memory leaks.

The affordance introduced in #201 was to always `release` objects on `__del__` calls if we took ownership of them earlier, by either one of the described methods ~or by an explicit `retain` call~. This solution still requires the user to:

1. Ensure that there are no reference cycles crossing this Objc-C to Python boundary that cannot be detected by garbage collection.
2. Manually retain and release objects which we need that are returned by any method other than the ones listed above. This was also not always done, leading to segfaults and other issues.

#### Possible options

1. As described in https://github.com/beeware/rubicon-objc/issues/249#issuecomment-1399472658, always retain objects referenced by Python and release them on garbage collection. For objects returned by `alloc`, etc., we would then have a refcount of 2 and would need two release calls or more explicit releases by users of rubicon-objc. There may also be other corner cases to consider.
2. Completely give up on magical conveniences and require strict adherence to memory management rules in toga. This goes back to the state before #201. The required ref-count aware programming does not come naturally to Python programmers so would likely lead to a re-emergence of memory leaks.
3. Keep the status quo but systematically vet the toga-cocoa codebase to add retain calls where required.

FWIW - I think the option described by (1) (i.e., the #249) is worth exploring. (3) is working well enough, but is still prone to leaks (and occasional crashes); as you've noted (2) requires a set of skills that most Python programmers (and, to be brutally honest, most programmers *period*) either don't have, or can't exercise with 100% reliability. (1) has the benefit of being easy to explain conceptually, and should (?) be relatively straightforward to implement.

I have 2 core sources of concern around (1) as an approach:

1. The lifecycle of objects where ownership is transferred from something in the ObjC runtime to the Python runtime (or vice versa). This is really the broader class of the `alloc()` case you've described.
2. Cross-engine reference cycles. If the Python side retains a reference because it thinks the ObjC side is still using the object, and the ObjC side retains a reference because it thinks the Python side is retaining a reference, then we're going to end up with leaks. This isn't really any different to reference cycles in Python itself; we just need to be aware of how to detect and resolve them.
Agreed with your broad assessment that option (2) is very hard to do without enforcing particular design patterns and potential tooling support. There is a reason why Apple strongly recommends ARC.

Regarding your concerns around https://github.com/beeware/rubicon-objc/issues/249:

1. This does need some fleshing out. My initial idea is to just add an additional `release` call whenever such objects are deleted on the Python side, but this relies on always knowing when we are handed ownership. If this goes beyond a list of hard-coded method names / prefixes, this might not possible.
3. Cross-engine reference cycles: Making Pythons Garbage Collection aware of ObjC objects will be a tall order, if feasible at all. I would tend toward leaving this as open problem, and put the burden on the Rubicon users to avoid such cycles by using weak properties.
> My initial idea is to just add an additional `release` call whenever such objects are deleted on the Python side

Alternatively, how about this?

* Call `retain` when the Python wrapper is created, unless it was created via one of the special method names.
* Call `release` exactly once when the Python wrapper is destroyed in `__del__`.

That covers all the use cases I can think of, except for the pattern from https://github.com/beeware/toga/issues/2468, where a failed `init` releases the object and returns NULL. Could we make Rubicon aware of this by supporting a more Pythonic syntax as described in #26: `NSImage(contentsOfFile=path)`? This would call `alloc`, then `initWithContentsOfFile` , and if the latter returns NULL, then `NSImage(...)` would call `retain` to cancel out the `release` in `__del__`, and then throw an exception.

Then, as long as we can assume that the plain `init` method always takes zero arguments, that could be used by writing `NSImage()`.

This may clash with the way the Python constructor is currently being used to wrap an existing Objective-C object, but it should be possible to distinguish that mode by checking argument types, or by passing a keyword argument like `_rubicon_something`.

> Cross-engine reference cycles: Making Pythons Garbage Collection aware of ObjC objects will be a tall order, if feasible at all. I would tend toward leaving this as open problem, and put the burden on the Rubicon users to avoid such cycles by using weak properties.

Yes, I think that's the only reasonable option.
There are also some type methods that don't follow the naming convention, like `NSURL.URLWithString`. As I understand it, the Rubicon user must manually `release` the object returned by such methods. Under this proposal, they would still need to do that, because the object would be returned by the method with a reference count of 1, and Rubicon would then increment that to 2.

How does Objective-C's automatic reference counting handle this? Does it depend on metadata in header files, and is that information available at runtime? Unless we can fix this completely, I think the documentation is a bit overoptimistic to say "You won’t have to manage reference counts in Python, Rubicon Objective-C will do that work for you."
> There are also some type methods that don't follow the naming convention, like `NSURL.URLWithString`. As I understand it, the Rubicon user must manually `release` the object returned by such methods. Under this proposal, they would still need to do that, because the object would be returned by the method with a reference count of 1, and Rubicon would then increment that to 2.

The baseline [memory management rules](https://developer.apple.com/library/archive/documentation/Cocoa/Conceptual/MemoryMgmt/Articles/mmRules.html) described the conditions for API memory retention; as I understand it, `NSURL.URLWithString` falls under the "owning an object you create" clause - i.e., the "alloc, new, copy..." list is indicative, but isn't exhaustive; class-level "create and init" methods also fall under this umbrella.

As far as I can make out, we are currently leaking in the `NSURL.URLWithString`-style case unless you call `.release()`, `.autorelease()`, or you manually tweak `_needs_release=True` on an instance created with this method. A process that runs:
```
while True:
    gc.collect()
    NSURL.URLWithString("https://example.com")
```
seems to slowly leak memory (as measured by RSS allocated size); whereas:
```
while True:
    gc.collect()
    NSURL.alloc().initWithString("https://example.com")
```
seems much more stable over time. There's still a lot of growth in process size, but this seems to be recovered over time, whereas there's no (or, at least, much less) observable memory recovery in the former version.

In both cases, the number of cached objects (`NSURL._cached_objects`), and the number of objects held by the garbage collector is constant, which suggests the memory is being lost on the ObjC side.

So... it would appear that we definitely have work to do here.

The approach you've described of adding a `retain()` to every `ObjCInstance` doesn't *quite* work, because there's an implied `retain` already on any object you create - so, any `alloc`, `new`, etc method will either require an additional `release`, or *avoid* invoking `retain` on every instance. Of course, that will still fall into the `NSURL.URLWithString` hole, as there's no way to determine from the runtime whether a reference you've been given is owned.

The other issue I can see is with uses that currently require the use of `autorelease`. Idiomatically, `autorelease` is used when you have a method that acts as a factory for new objects, but doesn't retain ownership of that object; the receiving object is expected to then take ownership. We use this pattern in APIs like the macOS implementation of Table and Tree to populate tree nodes, and in DetailedList for popup menus. If create an ObjC object on the Python side, but release as part of the ObjCInstance wrapper, then the object will cease to have an ObjCInstance when it is returned, which should cause it to be released (or, at least, put it on a candidate list for release). I'm not sure if the best option here would be to `autorelease` rather than `release` on `__del__`; or if we need to retain an `autorelease` mechanism in the API for this case.

> there's an implied `retain` already on any object you create - so, any `alloc`, `new`, etc method will either require an additional `release`, or _avoid_ invoking `retain` on every instance.

Yes, that's why I said "Call `retain` when the Python wrapper is created, unless it was created via one of the special method names."

> Idiomatically, `autorelease` is used when you have a method that acts as a factory for new objects, but doesn't retain ownership of that object; the receiving object is expected to then take ownership. We use this pattern in APIs like the macOS implementation of Table and Tree to populate tree nodes, and in DetailedList for popup menus.

The current implementation of those methods calls `retain` and `autorelease` before returning the object. That would continue to work under this proposal:
* `alloc` would return an object with a reference count of 1, and ObjCInstance would leave it unchanged because of the special method name.
*  `retain` increases it to 2
* Destroying the ObjCInstance on leaving the function reduces it to 1.
* The calling code calls `retain`, increasing it to 2
* The main loop iteration completes the `autorelease` block, reducing it to 1

> I'm not sure if the best option here would be to `autorelease` rather than `release` on `__del__`

Maybe we can be more specific: what if Rubicon automatically does a retain/autorelease on any object which is `return`ed to Objective-C? If we are transferring ownership, that will keep it alive long enough, and if we're not transferring ownership, it should be harmless.

Either way, any existing code that uses retain/release/autorelease correctly would continue to work, but many of those uses would now be redundant and could be removed when convenient.
> > there's an implied `retain` already on any object you create - so, any `alloc`, `new`, etc method will either require an additional `release`, or _avoid_ invoking `retain` on every instance.
>
> Yes, that's why I said "Call `retain` when the Python wrapper is created, unless it was created via one of the special method names."

🤦 My apolgies - you did.

> > Idiomatically, `autorelease` is used when you have a method that acts as a factory for new objects, but doesn't retain ownership of that object; the receiving object is expected to then take ownership. We use this pattern in APIs like the macOS implementation of Table and Tree to populate tree nodes, and in DetailedList for popup menus.
>
> The current implementation of those methods calls `retain` and `autorelease` before returning the object. That would continue to work under this proposal:
>
> * `alloc` would return an object with a reference count of 1, and ObjCInstance would leave it unchanged because of the special method name.
> * `retain` increases it to 2
> * Destroying the ObjCInstance on leaving the function reduces it to 1.
> * The calling code calls `retain`, increasing it to 2
> * The main loop iteration completes the `autorelease` block, reducing it to 1

Looking closer at the Tree/Table/DetailedList code, I think the `retain()` calls aren't actually needed in those cases - they just need to be `autorelease()`. If you take the explicit `retain()` out, we end up with a final count of 0 - and a count of 0 when the ObjCInstance is garbage collected. This would be a closer match for the autorelease example given in the [Cocoa memory management guide](https://developer.apple.com/library/archive/documentation/Cocoa/Conceptual/MemoryMgmt/Articles/mmRules.html#//apple_ref/doc/uid/20000994-SW6).

Making that situation work would require Rubicon to `autorelease()` the controlled object, rather than a `release()` when the ObjCInstance is garbage collected.

Alternatively, the ObjCInstance reference could be considered unrelated to the ObjC memory handling, and require full retain/release calls... but that seems like a bit of a headache.

> > I'm not sure if the best option here would be to `autorelease` rather than `release` on `__del__`
>
> Maybe we can be more specific: what if Rubicon automatically does a retain/autorelease on any object which is `return`ed to Objective-C? If we are transferring ownership, that will keep it alive long enough, and if we're not transferring ownership, it should be harmless.

I'm not certain that's true. If you have a method that is an accessor, rather than a factory, the memory handling requirements of return values are different.

Factory methods only need a `retain()` if the creation method doesn't have a magic name. However. if we always do a `retain()` on an accessor method, then every variable access will increase the retain count by 1, which will guarantee a leak.

An automatic `.autorelease()` on factory methods is what we actually want. An automatic autorelease on an accessor... that's the one I'm not clear on. I know it's safe to call autorelease on an object more than once - but I'm not sure if marking an arbitrary object as autorelease could have consequences elsewhere.

> It is also possible to create an ObjCInstance directly from an objc_id. If this happens, it may be cached as a raw ObjCInstance, and _not_ have the helper methods.

How could this happen, when `ObjCInstance.__new__` already uses `type_for_objcclass` to decide which Python wrapper class to use?
> > It is also possible to create an ObjCInstance directly from an objc_id. If this happens, it may be cached as a raw ObjCInstance, and _not_ have the helper methods.
>
> How could this happen, when `ObjCInstance.__new__` already uses `type_for_objcclass` to decide which Python wrapper class to use?

After some more investigation, I've found the culprit... it's a niche side effect of #70.

The branch that is being triggered is [this one](https://github.com/beeware/rubicon-objc/blob/main/src/rubicon/objc/api.py#L922). It's being triggered on the very first use of the subscript notation, which triggers `load_methods` handling; this causes [a retrieval of `self.objc_class`](https://github.com/beeware/rubicon-objc/blob/main/src/rubicon/objc/api.py#L1065), which causes [the creation of an `ObjCClass` instance](https://github.com/beeware/rubicon-objc/blob/main/src/rubicon/objc/api.py#L810). `ObjCClass` is a subclass of `ObjCInstance`, so it is put in the instance cache.

As far as I can make out, caching the ObjCClass in this case is correct (and quick test of *not* caching instances created with this branch cause other problems). The bug being addressed here is caused if the memory address used for the ObjCClass instance is re-used with an *actual* dictionary. There might be some deeper cleanups if #70 or #256 are addressed, but at least in the short term, I'm increasingly convinced the cache type check is actually the right fix in this instance.

Summary of in-person discussion: it seems like this ought to be caught by the existing check of the cached object's class, since:

* class of an NSDictionary *instance* is the NSDictionary *class*
* class of the NSDIctionary *class* is its [meta-class](https://stackoverflow.com/a/11267971)

It looks like the existing class comparison is making both of these look the same, but there should be a way of distinguishing them, perhaps by comparing pointers rather than names. This would avoid adding more complexity to what is already a pretty confusing area of the code.
Edited my previous comment to link to information about meta-classes. If the class and its meta-class both return the same value for `class_getName`, that would explain what we're seeing.
Under which circumstances can the memory address of the ObjCClass instance be freed and reused? IIUC, you mean the memory address of NSMutableDictionary implementation (__NSDictionaryM). Shouldn't this is just always stay as is?
@samschott That's a really good question - your analysis was my initial "This can't possibly be happening" analysis, as well. However, if you reproduce the bug and set a breakpoint, an `ObjCClass` instance that is created to satisfy the request for `self.objc_class` when the method cache is populated. As far as I can make out, that's the memory address that is being re-used; and the issue is that because it's an ObjCClass that has the same class *name*, it's a cache hit.

What I can't work out is (a) why an ObjCClass instance would be disposed of; and (b) how to structure the cache check so that it catches this edge case without spoiling other cases.
> As far as I can make out, that's the memory address that is being re-used; and the issue is that because it's an ObjCClass that has the same class name, it's a cache hit.

I wasn't able to reproduce myself on a M1 MacBook Air with macOS 15.1, so can't actually see for myself what is going on. If indeed the memory address of the NSMutableDictionary class is reused, can you check if NSMutableDictionary lives at a different memory address later? This should be theoretically easy to validate. Recording the address of NSMutableDictionary before the exception is thrown should additionally also allow us to add debug statements to track if we ever see a different object at this address.
I did actually manage to reproduce this now, turns out a bit more dragging and resizing was needed than I thought :)

As expected, the memory address of the `__NSDictionaryM` class appears to remain the same throughout, even after the incorrect cache hit causes an exception. So unless there is another class with name `__NSDictionaryM` (which would at least go against ObjC conventions of having unique class names) I am skeptical about its memory being reused.

Assuming that this issue is indeed caused by the instance's class name and the class's class name being equal (which they are) _and_ the class object being deallocated and its memory address reused (which I am skeptical about), I still see one more issue with the explanation that you give in https://github.com/beeware/rubicon-objc/issues/539#issuecomment-2472003054:

> The branch that is being triggered is [this one](https://github.com/beeware/rubicon-objc/blob/main/src/rubicon/objc/api.py#L922). It's being triggered on the very first use of the subscript notation, which triggers load_methods handling; this causes [a retrieval of self.objc_class](https://github.com/beeware/rubicon-objc/blob/main/src/rubicon/objc/api.py#L1065), which causes the creation of an ObjCClass instance. ObjCClass is a subclass of ObjCInstance, so it is put in the instance cache.

If the ObjCClass instance is created by [a retrieval of self.objc_class](https://github.com/beeware/rubicon-objc/blob/main/src/rubicon/objc/api.py#L1065), survives in the cache and causes a false hit, its string representation should still be `<ObjCClass: __NSDictionaryM>`. However, the stack trace from https://github.com/beeware/rubicon-objc/issues/256 as well as my own print statements show that its `<ObjCInstance: __NSDictionaryM>`.
Could it be possible that _some_ code in Rubicon-ObjC creates a NSMutableDictionary instance before the code that actually registers the wrapper gets to run?

https://github.com/beeware/rubicon-objc/blob/9db16a7dca31a6f85184db1b33a50437aca7212a/src/rubicon/objc/collections.py#L394

This is just wild speculation, I'm not even sure if that cache entry would have a chance to survive sufficiently log for this bug manifest. _Edit:_ Maybe it could, given the unpredictability of Python garbage collection.
Actually reading up on ARC, I'm completely confused... are calls to `.alloc()` recorded to current autorelease pool by the compiler or runtime?

Depending on that, Python must or must not call `.[auto]release()`...
ARC is a compiler feature, it makes the compiler insert `retain`/`release` calls so the programmer doesn't have to. Because rubicon-objc operates at the runtime level, there is no way to use ARC. That means you have to insert `retain`/`release`/`autorelease` calls in the correct places, like you would without ARC.

Alternatively, it would be possible to make rubicon-objc call `retain` and `release` automatically. This is how Pythonista's `objc_util` module does it - it calls `retain` when an `ObjCInstance` is created, and `release` when it is garbage-collected (in `__del__`). The only issue with this is that it needs special cases for `alloc`/`copy`/`mutableCopy`/`new` methods, which return an already retained object ([docs](https://developer.apple.com/library/content/documentation/Cocoa/Conceptual/MemoryMgmt/Articles/mmRules.html#//apple_ref/doc/uid/20000994-SW1)).
> it would be possible to make rubicon-objc call `retain` and `release` automatically. This is how Pythonista's `objc_util` module does it - it calls `retain` when an `ObjCInstance` is created, and `release` when it is garbage-collected (in `__del__`

This was implemented in #201, but the documentation is still lacking:

* [rubicon-objc-api](https://rubicon-objc.readthedocs.io/en/latest/reference/rubicon-objc-api.html) says "Rubicon currently does not perform any automatic memory management on the Objective-C object wrapped in an [ObjCInstance](https://rubicon-objc.readthedocs.io/en/latest/reference/rubicon-objc-api.html#rubicon.objc.api.ObjCInstance)", which is no longer true.
* [memory-management](https://rubicon-objc.readthedocs.io/en/latest/how-to/memory-management.html) says "You won’t have to manage reference counts in Python, Rubicon Objective-C will do that work for you." This is over-optimistic, as is shown by the various manual retain/release calls in Toga, and the discussion in #256.
* The fact that calling `release` or `autorelease` disables the automatic `release` in `__del__` is mentioned in those methods' docstrings, but those strings aren't included in the HTML documentation.


diff --git a/openlibrary/core/lending.py b/openlibrary/core/lending.py
index 6162ed5b081..d7e2a1949cb 100644
--- a/openlibrary/core/lending.py
+++ b/openlibrary/core/lending.py
@@ -511,13 +511,53 @@ def _get_ia_loan(identifier, userid):

 def get_loans_of_user(user_key):
     """TODO: Remove inclusion of local data; should only come from IA"""
+    if 'env' not in web.ctx:
+        """For the get_cached_user_loans to call the API if no cache is present,
+        we have to fakeload the web.ctx
+        """
+        delegate.fakeload()
+
     account = OpenLibraryAccount.get(username=user_key.split('/')[-1])

     loandata = web.ctx.site.store.values(type='/type/loan', name='user', value=user_key)
     loans = [Loan(d) for d in loandata] + (_get_ia_loans_of_user(account.itemname))
+    # Set patron's loans in cache w/ now timestamp
+    get_cached_loans_of_user.memcache_set(
+        [user_key], {}, loans or [], time.time()
+    )  # rehydrate cache
     return loans

+get_cached_loans_of_user = cache.memcache_memoize(
+    get_loans_of_user,
+    key_prefix='lending.cached_loans',
+    timeout=5 * dateutil.MINUTE_SECS,  # time to live for cached loans = 5 minutes
+)
+
+
+def get_user_waiting_loans(user_key):
+    """Gets the waitingloans of the patron.
+
+    Returns [] if user has no waitingloans.
+    """
+    from .waitinglist import WaitingLoan
+
+    account = OpenLibraryAccount.get(key=user_key)
+    itemname = account.itemname
+    result = WaitingLoan.query(userid=itemname)
+    get_cached_user_waiting_loans.memcache_set(
+        [user_key], {}, result or {}, time.time()
+    )  # rehydrate cache
+    return result or []
+
+
+get_cached_user_waiting_loans = cache.memcache_memoize(
+    get_user_waiting_loans,
+    key_prefix='waitinglist.user_waiting_loans',
+    timeout=10 * dateutil.MINUTE_SECS,
+)
+
+
 def _get_ia_loans_of_user(userid):
     ia_loans = ia_lending_api.find_loans(userid=userid)
     return [Loan.from_ia_loan(d) for d in ia_loans]
diff --git a/openlibrary/core/models.py b/openlibrary/core/models.py
index d582db128c4..b4b3c1cd2a5 100644
--- a/openlibrary/core/models.py
+++ b/openlibrary/core/models.py
@@ -16,6 +16,7 @@

 # TODO: fix this. openlibrary.core should not import plugins.
 from openlibrary import accounts
+from openlibrary.core import lending
 from openlibrary.catalog import add_book
 from openlibrary.core.booknotes import Booknotes
 from openlibrary.core.bookshelves import Bookshelves
@@ -24,7 +25,7 @@
 from openlibrary.core.observations import Observations
 from openlibrary.core.ratings import Ratings
 from openlibrary.core.vendors import create_edition_from_amazon_metadata
-from openlibrary.utils import extract_numeric_id_from_olid
+from openlibrary.utils import extract_numeric_id_from_olid, dateutil
 from openlibrary.utils.isbn import to_isbn_13, isbn_13_to_isbn_10, canonical

 # Seed might look unused, but removing it causes an error :/
@@ -916,25 +917,51 @@ def has_borrowed(self, book):
         loan = self.get_loan_for(book.ocaid)
         return loan is not None

-    def get_loan_for(self, ocaid):
+    def get_loan_for(self, ocaid, use_cache=False):
         """Returns the loan object for given ocaid.

         Returns None if this user hasn't borrowed the given book.
         """
         from ..plugins.upstream import borrow

-        loans = borrow.get_loans(self)
+        loans = (
+            lending.get_cached_loans_of_user(self.key)
+            if use_cache
+            else lending.get_loans_of_user(self.key)
+        )
         for loan in loans:
             if ocaid == loan['ocaid']:
                 return loan

     def get_waiting_loan_for(self, ocaid):
         """
-        :param str or None ocaid:
+        :param str or None ocaid: edition ocaid
         :rtype: dict (e.g. {position: number})
         """
         return ocaid and WaitingLoan.find(self.key, ocaid)

+    def get_user_waiting_loans(self, ocaid=None, use_cache=False):
+        """
+        Similar to get_waiting_loan_for, but fetches and caches all of user's waiting loans
+        :param str or None ocaid: edition ocaid
+        :rtype: dict (e.g. {position: number})
+        """
+        all_user_waiting_loans = (
+            lending.get_cached_user_waiting_loans
+            if use_cache
+            else lending.get_user_waiting_loans
+        )(self.key)
+        if ocaid:
+            return next(
+                (
+                    loan
+                    for loan in all_user_waiting_loans
+                    if loan['identifier'] == ocaid
+                ),
+                None,
+            )
+        return all_user_waiting_loans
+
     def __repr__(self):
         return "<User: %s>" % repr(self.key)

diff --git a/openlibrary/core/waitinglist.py b/openlibrary/core/waitinglist.py
index 5381e4614b6..207bae4da44 100644
--- a/openlibrary/core/waitinglist.py
+++ b/openlibrary/core/waitinglist.py
@@ -16,6 +16,7 @@
 import logging
 import web
 from openlibrary.accounts.model import OpenLibraryAccount
+from openlibrary.core import cache
 from . import helpers as h
 from .sendmail import sendmail_with_template
 from . import db
diff --git a/openlibrary/macros/LoanStatus.html b/openlibrary/macros/LoanStatus.html
index b9649006622..7c6427a7378 100644
--- a/openlibrary/macros/LoanStatus.html
+++ b/openlibrary/macros/LoanStatus.html
@@ -17,7 +17,7 @@
 $ work_key = work_key or (doc.get('works') and doc.works[0].key)

 $ waiting_loan_start_time = time()
-$ waiting_loan = check_loan_status and ocaid and ctx.user and ctx.user.get_waiting_loan_for(ocaid)
+$ waiting_loan = check_loan_status and ocaid and ctx.user and ctx.user.get_user_waiting_loans(ocaid, use_cache=True)
 $ waiting_loan_total_time = time() - waiting_loan_start_time
 $ my_turn_to_borrow = waiting_loan and waiting_loan['status'] == 'available' and waiting_loan['position'] == 1

@@ -34,7 +34,7 @@

 $# Checks to see if patron has actively loan / waitlist for this book
 $ get_loan_for_start_time = time()
-$ user_loan = doc.get('loan') or (check_loan_status and ocaid and ctx.user and ctx.user.get_loan_for(ocaid))
+$ user_loan = doc.get('loan') or (check_loan_status and ocaid and ctx.user and ctx.user.get_loan_for(ocaid, use_cache=True))
 $ get_loan_for_total_time = time() - get_loan_for_start_time

 $ is_edition = doc.key.split('/')[1] == 'books'
diff --git a/openlibrary/plugins/upstream/borrow.py b/openlibrary/plugins/upstream/borrow.py
index f48c1656115..9e9827046c7 100644
--- a/openlibrary/plugins/upstream/borrow.py
+++ b/openlibrary/plugins/upstream/borrow.py
@@ -149,6 +149,9 @@ def POST(self, key):
             account = OpenLibraryAccount.get_by_email(user.email)
             ia_itemname = account.itemname if account else None
             s3_keys = web.ctx.site.store.get(account._key).get('s3_keys')
+            lending.get_cached_loans_of_user.memcache_delete(
+                user.key, {}
+            )  # invalidate cache for user loans
         if not user or not ia_itemname or not s3_keys:
             web.setcookie(config.login_cookie_name, "", expires=-1)
             redirect_url = (
@@ -165,10 +168,16 @@ def POST(self, key):
             user.update_loan_status()
             raise web.seeother(edition_redirect)
         elif action == 'join-waitinglist':
+            lending.get_cached_user_waiting_loans.memcache_delete(
+                user.key, {}
+            )  # invalidate cache for user waiting loans
             lending.s3_loan_api(s3_keys, ocaid=edition.ocaid, action='join_waitlist')
             stats.increment('ol.loans.joinWaitlist')
             raise web.redirect(edition_redirect)
         elif action == 'leave-waitinglist':
+            lending.get_cached_user_waiting_loans.memcache_delete(
+                user.key, {}
+            )  # invalidate cache for user waiting loans
             lending.s3_loan_api(s3_keys, ocaid=edition.ocaid, action='leave_waitlist')
             stats.increment('ol.loans.leaveWaitlist')
             raise web.redirect(edition_redirect)

Cache Patron's Active Loans
<!-- A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] -->

On several pages (e.g. LoanStatus) we fetch the patron's active loans (which can be expensive) to see if they've borrowed a book (e.g. on the book page).

Ideally, we'd cache this every 5 minutes and invalidate within the /borrow endpoint. The one "gotcha" is if a patron borrows a book from archive.org, Open Library doesn't know how to bust the cache, but that should be okay because presumably this will be fixed within the 5 minute window.

Exploration required into whether cache actually works for this and how performance is impacted.


diff --git a/sisyphus/aws_batch_engine.py b/sisyphus/aws_batch_engine.py
index 4b0173f..80f454e 100644
--- a/sisyphus/aws_batch_engine.py
+++ b/sisyphus/aws_batch_engine.py
@@ -1,4 +1,4 @@
-""" This is an experimental implementation for the aws batch engine.
+"""This is an experimental implementation for the aws batch engine.

 WARNING: After running some setups I can currently not recommend using aws batch with Sisyphus.
 AWS parallelcluster (https://aws.amazon.com/blogs/opensource/aws-parallelcluster/) looks like a easy way how
@@ -88,10 +88,13 @@ def system_call(self, command, send_to_stdin=None):
         system_command = command

         logging.debug("shell_cmd: %s" % " ".join(system_command))
-        p = subprocess.Popen(system_command, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
         if send_to_stdin:
             send_to_stdin = send_to_stdin.encode()
-        out, err = p.communicate(input=send_to_stdin, timeout=30)
+        try:
+            p = subprocess.run(system_command, input=send_to_stdin, capture_output=True, timeout=30)
+        except subprocess.TimeoutExpired:
+            logging.warning("Timeout expired for command: %s" % " ".join(system_command))
+            return [], ["TimeoutExpired"], -1

         def fix_output(o):
             """
@@ -105,9 +108,9 @@ def fix_output(o):
                 assert False
             return o[:-1]

-        out = fix_output(out)
-        err = fix_output(err)
-        retval = p.wait(timeout=30)
+        out = fix_output(p.stdout)
+        err = fix_output(p.stderr)
+        retval = p.returncode

         return out, err, retval

diff --git a/sisyphus/load_sharing_facility_engine.py b/sisyphus/load_sharing_facility_engine.py
index 2b63318..5a24ad1 100644
--- a/sisyphus/load_sharing_facility_engine.py
+++ b/sisyphus/load_sharing_facility_engine.py
@@ -56,10 +56,13 @@ def system_call(self, command, send_to_stdin=None):
             system_command = command

         logging.debug("shell_cmd: %s" % " ".join(system_command))
-        p = subprocess.Popen(system_command, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
         if send_to_stdin:
             send_to_stdin = send_to_stdin.encode()
-        out, err = p.communicate(input=send_to_stdin, timeout=30)
+        try:
+            p = subprocess.run(system_command, input=send_to_stdin, capture_output=True, timeout=30)
+        except subprocess.TimeoutExpired:
+            logging.warning("Timeout expired for command: %s" % " ".join(system_command))
+            return [], ["TimeoutExpired"], -1

         def fix_output(o):
             # split output and drop last empty line
@@ -69,9 +72,9 @@ def fix_output(o):
                 assert False
             return o[:-1]

-        out = fix_output(out)
-        err = fix_output(err)
-        retval = p.wait(timeout=30)
+        out = fix_output(p.stdout)
+        err = fix_output(p.stderr)
+        retval = p.returncode

         # Check for ssh error
         err_ = []
diff --git a/sisyphus/simple_linux_utility_for_resource_management_engine.py b/sisyphus/simple_linux_utility_for_resource_management_engine.py
index a918928..48ad8db 100644
--- a/sisyphus/simple_linux_utility_for_resource_management_engine.py
+++ b/sisyphus/simple_linux_utility_for_resource_management_engine.py
@@ -86,16 +86,21 @@ def system_call(self, command, send_to_stdin=None):
         """
         if self.gateway:
             escaped_command = [shlex.quote(s) for s in command]  # parameters need to be shell safe when sending via ssh
-            system_command = ["ssh", "-x", self.gateway] + [" ".join(["cd", os.getcwd(), "&&"] + escaped_command)]
+            system_command = ["ssh", "-x", self.gateway, "-o", "BatchMode=yes"] + [
+                " ".join(["cd", os.getcwd(), "&&"] + escaped_command)
+            ]
         else:
             # no gateway given, skip ssh local
             system_command = command

         logging.debug("shell_cmd: %s" % " ".join(system_command))
-        p = subprocess.Popen(system_command, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
         if send_to_stdin:
             send_to_stdin = send_to_stdin.encode()
-        out, err = p.communicate(input=send_to_stdin, timeout=30)
+        try:
+            p = subprocess.run(system_command, input=send_to_stdin, capture_output=True, timeout=30)
+        except subprocess.TimeoutExpired:
+            logging.warning("Timeout expired for command: %s" % " ".join(system_command))
+            return [], ["TimeoutExpired"], -1

         def fix_output(o):
             """
@@ -109,9 +114,9 @@ def fix_output(o):
                 assert False
             return o[:-1]

-        out = fix_output(out)
-        err = fix_output(err)
-        retval = p.wait(timeout=30)
+        out = fix_output(p.stdout)
+        err = fix_output(p.stderr)
+        retval = p.returncode

         # Check for ssh error
         err_ = []
diff --git a/sisyphus/son_of_grid_engine.py b/sisyphus/son_of_grid_engine.py
index acda4f0..6682aba 100644
--- a/sisyphus/son_of_grid_engine.py
+++ b/sisyphus/son_of_grid_engine.py
@@ -87,16 +87,21 @@ def system_call(self, command, send_to_stdin=None):
         :rtype: list[bytes], list[bytes], int
         """
         if self.gateway:
-            system_command = ["ssh", "-x", self.gateway] + [" ".join(["cd", os.getcwd(), "&&"] + command)]
+            system_command = ["ssh", "-x", self.gateway, "-o", "BatchMode=yes"] + [
+                " ".join(["cd", os.getcwd(), "&&"] + command)
+            ]
         else:
             # no gateway given, skip ssh local
             system_command = command

         logging.debug("shell_cmd: %s" % " ".join(system_command))
-        p = subprocess.Popen(system_command, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
         if send_to_stdin:
             send_to_stdin = send_to_stdin.encode()
-        out, err = p.communicate(input=send_to_stdin, timeout=30)
+        try:
+            p = subprocess.run(system_command, input=send_to_stdin, capture_output=True, timeout=30)
+        except subprocess.TimeoutExpired:
+            logging.warning("Timeout expired for command: %s" % " ".join(system_command))
+            return [], ["TimeoutExpired"], -1

         def fix_output(o):
             """
@@ -110,9 +115,9 @@ def fix_output(o):
                 assert False
             return o[:-1]

-        out = fix_output(out)
-        err = fix_output(err)
-        retval = p.wait(timeout=30)
+        out = fix_output(p.stdout)
+        err = fix_output(p.stderr)
+        retval = p.returncode

         # Check for ssh error
         err_ = []

Too many open file descriptors
Hi, I was using sisyphus today for a big recipe and I got an error in my worker which claimed `too many open files`:

```
OSError: [Errno 24] Unable to synchronously open file (unable to open file: name = <filename>, errno = 24, error message = 'Too many open files', flags = 0, o_flags = 0)
```

However, not only had the worker crashed, but also the manager crashed with this error. Moreover, the terminal (tmux pane) entered a state where every character I typed was converted into a newline character. As a result, I couldn't type any instruction, so I ended up killing the tmux pane and recreating it.

I got into an investigation of what was happening and I developed a really small test:

```
# test.py

def py():
    pass
```

I found out that running the sisyphus manager on the test (bare; without any `settings.py`) opened ~3k files, from my baseline of 524 opened files to 3254 opened files after running `sis m test.py`, according to `lsof | grep <user> | wc`.

Besides that, I had the issue that every job triggered by the manager was adding exactly 105 opened files to the list of open file descriptors. However, I can't reproduce this starting from scratch, which leads me to think that it might be a problem about how our code base interacts with sisyphus (or just from our code base). I'll keep investigating and keep you tuned.

Is this issue about opening too many files by sisyphus intended because of some sisyphus caching strategy or related work? Was this ever addressed?

If you need more details, I'll be glad to provide them. Thank you in advance.

I had this happen again. It was with a relatively big setup, but I'm not sure what causes the issue yet since my manager shouldn't be opening many files, if any. Please find attached the corresponding stack trace from the manager
[here](https://github.com/user-attachments/files/15509563/manager_too_many_open_files_github.txt).

Note that the last newlines from the stack trace are relevant, since these represent me trying to write anything at all, and any character becoming a newline:

> Moreover, the terminal (tmux pane) entered a state where every character I typed was converted into a newline character. As a result, I couldn't type any instruction, so I ended up killing the tmux pane and recreating it.

I think this could be an interaction with the manager prompting me for my SSH password many times because I had left the tmux session, and then crashing, thus leaving the prompt in an unstable state (i.e. whatever you write when writing your SSH password key is transparent).
Analyzing the stack trace I found that both issues (too many open files, and ssh key prompt) could be related. What sisyphus seems to be trying to do after each password prompt is running a subprocess with the `squeue` command (I'm running in SLURM, but this also used to happen in SGE as well, so it should be cluster-independent). Right now I'm running my setup with the `gateway="..."` option in `settings.py`, but I recall it could have happened without such an option.

This happens for each 30 seconds, which is the time my sisyphus is configured to scan the queue. With an open file cap of 1024 in the manager (assuming sisyphus doesn't open any other files and any are opened to begin with), the time needed to reach the cap would be 1024 * 30 = 30k seconds = 8.5 hours. Even though the time cap is practically lower because there are more files opened in the manager, it makes sense given the lengths of duration in which I abandoned the ssh/tmux session (evening/night).

I'll try to solve it on my end, but I think it could also make sense to try to fix it in sisyphus. How can we tell sisyphus to wait before running a new queue scan job? Maybe storing the last queue command issued and setting it to `None` after completion?

Edit: a pretty easy solution would probably be setting the timeout of the corresponding SSH command queue scan to the number of seconds it takes for sisyphus to run a new queue command.