back to

Users reporting unauthorized unsubscribes


I’m having a number of users reporting unauthorized unsubscribes. I tracked down the IP addresses of some the offending unsubscribe request and it always has the following user agent:

72.xx.xxxx - - [12/Sep/2018:22:14:18 -0400] “GET /lists/?p=unsubscribe&uid=ae84cfbd8c30bxxxxxxx9a0xxxc49 HTTP/1.1” 200 2474 “-” “Mozilla/5.0 (compatible; Yahoo! Slurp; kafe/1536804858-0”

or - - [13/Sep/2018:22:07:26 -0400] “GET /lists/?p=unsubscribe&uid=9368xxxxxxxxxxxx05660xxxx637e HTTP/1.1” 200 2474 “-” “Mozilla/5.0 (compatible; Yahoo! Slurp;”

or with other IP addresses and with HTTP_USER_AGENT = Mozilla/5.0 (compatible; Yahoo! Slurp;

Indeed the IP addresses always reverse to Yahoo, so they are valid yahoo ips.

Anybody else getting these in their logs? Is there anyway to prevent this? How is someone/thing getting their UUID?

Pretty sure robots.txt entries would not be honored.
Thanks in advance.


If this was just a matter of Yahoo Slurp somehow having access to the headers of emails, or otherwise trolling links, then this would occur on more than a subset of users with Yahoo accounts - it’s happening on more than just Yahoo email accounts. . .


@pancakehollow I believe that Yahoo mail actively encourages users to unsubscribe from mailing lists, adding buttons and pop-ups in some cases to promote that, e.g. if messages haven’t been read within a few weeks. Could it be that the subscribers in question have clicked away a Yahoo message without understanding that this would result in them being unsubscribed?


I had an exchange with Yahoo and (I believe that is the company handling their crawler) and the response was less than satisfactory. Here is the response:

"The crawler is simply attempting to index URLs, it does not understand what the URL does to the backend. I believe the unsubscribe requests are being called by the crawler because the request URLs are discoverable by the crawler, which means they may have been called and cached someone at a certain point in time. If the robots.txt is correctly placed with the correct content, our crawler will no longer attempt to index those URLs. "

It’s pretty impossible for the unsubscribe URL’s to be discoverable - that would be a HUGE security risk. It is also odd that those who have been unsubscribed by their crawler are vocal spokespeople for a politically-positioned issue.

I have checked with those being unsubscribed and none of the parties every clicked any unsubscribe links.

This is curious. The technicians recommendation was to use a robots.txt file to disallow their crawler and unsubscribes but that too is very odd given if unsubscribes from Yahoo are done via a crawler, and if someone actually does unsubscribe, then the logic of the technician - that the crawler would honor the robots.txt file - would stand counter to the recipients interest in unsubscribing.


Examine the following snippet of log file: Yahoo first checks for a robots.txt file, then seeing it’s not present, decides to just go ahead and unsubscribe someone: - - [09/Sep/2018:12:09:07 -0400] “GET /robots.txt HTTP/1.1” 404 - “-” “Mozilla/5.0 (compatible; Yahoo! Slurp;” - - [09/Sep/2018:12:09:07 -0400] “GET /lists/?p=unsubscribe&uid=cc60aa198xxf80axxxxxxxxxx7exxd4 HTTP/1.1” 200 2474 “-” “Mozilla/5.0 (compatible; Yahoo! Slurp; kafe/1536509347-0”


@pancakehollow Very interesting; nice research.

Have you checked for a default robots file - is one currently shipped by default with phpList 3? If not, could you create one and open a GitHub pull request?


I didn’t see one in my initial install but I have since created the robots.txt file.

Thing is, I’m disturbed that Yahoo, Inktomi (one of their associated enablers), and (another enabler), are just unsubscribing users. This seems to me like malicious activity.

And since I have enabled the robots.txt file and someone actually does unsubscribe, what happens? I will ask yahoo. . .