How can I safely remove a network connection that is no longer visible in `Control Panel\All Control Panel Items\Network Connections` on Windows 10?

I want to reuse the name of a previously existing connection, "USB Ethernet", so that I can programmatically find this connection on multiple PCs if it exists.

When I try to rename one of the connections I get the following error message: "Cannot rename this connection. A connection with the name that you specified already exists. Specify a different name."

[![Example error](https://i.stack.imgur.com/fV7Kk.gif)](https://i.stack.imgur.com/fV7Kk.gif)

There are no hidden adaptors in Device Manager: <https://support.microsoft.com/en-gb/help/315539/device-manager-does-not-display-devices-that-are-not-connected>

Other posts mention editing the registry and removing the GUID that represents that specific connection under:
`HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Network`

However, this connection may be referenced in other places within the registry. How can I **safely** remove the connection and all its references?

There are many unused/hidden connections on my machine according to the registry: `Local Area Connection* 11`, `Local Area Connection* 2`, `Local Area Connection* 7` etc.