All that is happening is robots are “following” links in emails. That might happen automatically, say when Yahoo receives an email for one of its members, or anti-malware software checks a received email. There is nothing special about the unsubscribe link.
I guess that you have enabled one-click unsubscribe, which means the automatic link click really does unsubscribe. You can change that to require the real subscriber to confirm.
define('UNSUBSCRIBE_JUMPOFF', false);
Adding a robots.txt file should reduce the problem. I have a robots file and don’t have a problem with rogue unsubscriptions.