I added `ipv6` to Raspbian (debian) on my Raspberry Pi using `modprobe`

I do not use it, and want to remove.

I tried `sudo modprobe -r ipv6` but get error `FATAL: Module ipv6 is in use.`

The only process I can find running is `ipv6_addrconf` which I can't seem to kill.