phpList.org

PHPlist sometimes sending several duplicates / smtp pipelining / speed

Support
#1

Hello,

I’m running version 3.5.3 on a shared hosting besides WordPress using cron. Since the beginning I’m getting hourly (and sometimes 2 hours) report mails during the campaign, stating that the process had been terminated by another process. Processing continues and as all mails get sent nevertheless, so I did not consider this to be a massive problem.

As all else seemed to be running ok, I continued tweaking the sending speed (now batch processing is disabled) and switched on messagequeue_prepare. While in the beginning it took about 8 hours for about 4.000 mails, I got it down to 5 to 6 hours.

Today PHPlist is running for 17 hours and hasn’t finished yet, producing a lot of duplicates which results in recipients cancelling their subscription.

During the first campaigns, already some recipients reported getting the message several times, but that did stop after after the first tweaks in batch size and duration, so I suspected adjusting those settings might have been fixing that, too. But no. Today and weeks later about 50% of the recipients received 2-5 times the same message (according to the hosting provider).

There’s been one recipient it kept trying and trying. I suspended the campaign and marked it as sent. Now it does not show up in the list of sent messages. The campaign has been sent 9.628 times to 4.156 recipients.

Any idea what could be causing troubles like this?

Hosting also informed me, that there doesn’t seem to be smtp pipelining at work. This causes the script to wait for the recipient server to answer before sending the next command, which slows everything down, too. Is this a misconfiguration, a bug or by design?

And is it possible to speed up the sending by removing all variables from the message and tell the script to send batches of mails bcc to several recipients at once ? How so?

#2

@auto-type When using shared hosting there is usually a restriction on the number of emails that can be sent, such as an hourly rate. You should use the batch settings so that phplist works within that restriction.

Using a small batch period, say 10 minutes, and a corresponding small batch size, should probably avoid your other problems but check the Event Log page for any errors that have been reported by phplist.

#3

Hello duncanc,

unfortunately the error log stops after a small amount of entries. There’s not much to be found but retries.

PHPlist behaved the same right after Installation with a batch size below my hoster’s limitations and without using messagequeue_prepare. And then it worked. And then it worked when I overrode them. And then it worked when I used messaqueue_prepare. And now it didn’t and again sent out one message up to 7 times to the same recipient. So I think, assuming it has something to do with the batch size might be leading in the wrong direction. Somehow the system doesn’t seem to recognize which recipients it already sent the message to and maybe starts checking the database from the beginning, finding the same users over and over again until it finally ticks the mark off.

How can those limitations in usage of an smtp server (in fact it is a mail queue piling up the server works on in its own pace) lead to the system not being able to detect which mails it has already sent?

Nevertheless I’ll try out your hint. But as this happened or didn’t happen however I configured the batch size in the past, how can I really be sure it works?

If possible, could you also elaborate whether PHPlist is capable of sending mails without variables using BCC and if so, where I could find information on how to configure that?

Is it true, that PHPlist does not use smtp pipelining or ist this a flaw in my setup?

#4

If the event log shows phplist retrying then the smtp server is returning errors, or at least phpmailer thinks that the sending was unsuccessful. Sending with a small batch size and period, and a throttle gap between sending emails should make it clearer to see what might be happening. You can try enabling VERBOSE in the config.php file to get more information added to the event log.

No phplist doesn’t use BCC.

phplist uses the PHPMailer package which I think does not use pipelining. But sending emails more quickly does not sound to be the problem you are trying to solve.

#5

I found out the following: as phpList is not using pipelining, the output of messages largely depends on the capacity of the smtp server during a campaign. If the mail server is busy and a batch has not been finished when the next cronjob kicks in, the process of that batch will be killed (‘This process has been cancelled by another process.’).

My guess is that even recipients in that batch that have already been sent the message will therefore not be marked off as having already been mailed to, which results in some recipients being flooded by copies of the same message.

As there is no way to predict the capacities of the server, the only solution I found was to set batch size to a value far below than would be possible to send under average conditions.