Multi Message Plugins

ℹ️ Support
1

My mailing list has grown to over 70,000 subscribers. I am using Kerio.Cloud to send out the messages via SMTP (this is the Mac Version of Exchange). However, I can’t seem to send more that 5,000 messages per hour. Is this a limitation of SMTP? I have set the server to allow UNLIMITED messages per hour from this IP/Domain but still not working. I have set the blast to as many messages as possible in 29 minutes with the CRON job kicking every thirty minutes. Still can’t get it to go faster.

Is there a tool like the plugin used with Amazon SES that would allow me to multi-thread the messages out faster?

Any help would be much appreciated.

Doug

2

What are the batch and throttle settings in config.php ?

1 Like
3

Sorry this has taken so long. I will past here the queue processing section for your review:

=========================================================================

Queue and Load management

=========================================================================

*/

# If you set up your system to send the message automatically (from commandline),

# you can set this value to 0, so "Process Queue" will disappear from the site

# this will also stop users from loading the page on the web frontend, so you will

# have to make sure that you run the queue from the commandline

# check README.commandline how to do this

define ("MANUALLY_PROCESS_QUEUE",0);

# batch processing

# if you are on a shared host, it will probably be appreciated if you don't send

# out loads of emails in one go. To do this, you can configure batch processing.

# Please note, the following two values can be overridden by your ISP by using

# a server wide configuration. So if you notice these values to be different

# in reality, that may be the case

# max messages to process

# if there are multiple messages in the queue, set a maximum to work on

define('MAX_PROCESS_MESSAGE',5);

# define the amount of emails you want to send per period. If 0, batch processing

# is disabled and messages are sent out as fast as possible

define("MAILQUEUE_BATCH_SIZE",0);

# define the length of one batch processing period, in seconds (3600 is an hour)

define("MAILQUEUE_BATCH_PERIOD",0);

# to avoid overloading the server that sends your email, you can add a little delay

# between messages that will spread the load of sending

# you will need to find a good value for your own server

# value is in seconds, and you can use fractions, eg "0.5" is half a second

# (or you can play with the autothrottle below)

define('MAILQUEUE_THROTTLE',0);

# Mailqueue autothrottle. This will try to automatically change the delay

# between messages to make sure that the MAILQUEUE_BATCH_SIZE (above) is spread evently over

# MAILQUEUE_BATCH_PERIOD, instead of firing the Batch in the first few minutes of the period

# and then waiting for the next period. This only works with mailqueue_throttle off

# and MAILQUEUE_BATCH_PERIOD being a positive value

# it still needs tweaking, so send your feedback to mantis.phplist.com if you find

# any issues with it

define('MAILQUEUE_AUTOTHROTTLE',0);

# Domain Throttling

# You can activate domain throttling, by setting USE_DOMAIN_THROTTLE to 1

# define the maximum amount of emails you want to allow sending to any domain and the number

# of seconds for that amount. This will make sure you don't send too many emails to one domain

# which may cause blacklisting. Particularly the big ones are tricky about this.

# it may cause a dramatic increase in the amount of time to send a message, depending on how

# many users you have that have the same domain (eg hotmail.com)

# if too many failures for throttling occur, the send process will automatically add an extra

# delay to try to improve that. The example sends 1 message every 2 minutes.

define('USE_DOMAIN_THROTTLE',0);

define('DOMAIN_BATCH_SIZE',0);

define('DOMAIN_BATCH_PERIOD',0);

# if you have very large numbers of users on the same domains, this may result in the need

# to run processqueue many times, when you use domain throttling. You can also tell phplist

# to simply delay a bit between messages to increase the number of messages sent per queue run

# if you want to use that set this to 1, otherwise simply run the queue many times. A cron

# process every 10 or 15 minutes is recommended.

define('DOMAIN_AUTO_THROTTLE',0);

# MAX_PROCESSQUEUE_TIME

# to limit the time, regardless of batch processing or other throttling of a single run of "processqueue"

# you can set the MAX_PROCESSQUEUE_TIME in seconds

# if a single queue run exceeds this amount, it will stop, just to pick up from where it left off next time

# this allows multiple installations each to run the queue, but slow installations (eg with large emails)

# set to 0 to disable this feature.

define('MAX_PROCESSQUEUE_TIME',1750);

### DSD - January 2019 - that failed so setting Batch Period to Zero and using QueueTime

### DSD - January 2019 - set to 0 to disable this and use ONLY Mail Batch Period.

### DSD - January 2019 - processing the queue every 30 minutes.

### DSD - May 2018 - changed it to hourly with a 59 minute run time.

(I will await your input on how to make it work correctly. I included the change management lines so you can see the testing I am doing as well.)

4

Please use the “preformatted” button </> when pasting code to make it readable.

phplist should be sending as fast as it can with your configuration, so I don’t see that there is anything that you can do to increase the speed using the same SMTP server.

5

@Dougster If you enable VERBOSE in config.php then phplist will add event log entries for every email that is sent. From that you should be able to see the time that is taken to build and send each individual email.

1 Like
6

Thanks … Since I run the script using a CRON job and pour the results into a text file, I have enabled that and we will see what results show after the next large run.

Thanks again for the help,
Doug

7

@duncanc Thanks to this nudge I documented verbose mode in the phpList manual troubleshooting page: https://www.phplist.org/manual/ch046_troubleshooting-techniques.xhtml

8

After enabling the VERBOSE setting, I see that the system takes .55 seconds per email. That seems rather slow to me and it is the same (.53 to .62) for every single email. So that is probably not a problem with the destination. Is there a way to see what is taking up all that time? Would turning on a debug level also provide details? Would the debug info be added to the text output from the CRON job?

Thanks again for the assistance,
Doug

9

@Dougster The time per email includes the actual sending using SMTP.

I guess that it is the SMTP interactions that are taking the time because on my local machine phplist takes 0.05s per email but that is sending to a local SMTP server so there is almost negligible SMTP time

image

I don’t understand your setup,exactly what Kerio Cloud is doing, but have you raised the problem with them?

10

phpList - Sending 269 to ##blocked##@aol.com [0.0073590000] (96867)

phpList - It took 0.5485520000 seconds to send [0.5493930000] (96904)

phpList - Sending 269 to ##blocked##@carpetweavers.net [0.0066720000] (96917)

phpList - It took 0.5548980000 seconds to send [0.5556000000] (96954)

phpList - Sending 269 to ##Blocked##@gmail.com [0.0075820000] (96967)

phpList - It took 0.6164860000 seconds to send [0.6173840000] (97004)

phpList - Sending 269 to ##blocked##@yahoo.com [0.0075300000] (97017)

phpList - It took 0.5551180000 seconds to send [0.5559730000] (97054)

phpList - Sending 269 to ##blocked##@lafilm.edu [0.0067060000] (97067)

phpList - It took 0.6319390000 seconds to send [0.6327380000] (97104)

phpList - Sending 269 to ##blocked##@missouri.edu [0.0076780000] (97117)

phpList - It took 0.5519350000 seconds to send [0.5527360000] (97154)

phpList - Sending 269 to ##blocked##@gmail.com [0.0078210000] (97167)

phpList - It took 0.6339220000 seconds to send [0.6347490000] (97204)

phpList - Sending 269 to ##blocked##@gmail.com [0.0074260000] (97217)

phpList - It took 0.6222040000 seconds to send [0.6230710000] (97254)

phpList - Processed 1994 out of 1994 subscribers [0.0030540000] (97262)

phpList - It took 12 hours 22 minutes 33 seconds to send this message [0.5836940000] (97275)

phpList - Script stage: 5 [0.5370310000] (97278)

phpList - 1994 messages sent in 1358.67 seconds (5283 msgs/hr) [0.0016810000] (97281)

phpList - Finished this run [0.0032310000] (97287)

X-Powered-By: PHP/7.1.26

Set-Cookie: PHPSESSID=33f2bd436acac65e2bdfd06b6cb7a182; path=/

Expires: Thu, 19 Nov 1981 08:52:00 GMT

Cache-Control: no-store, no-cache, must-revalidate

Pragma: no-cache

X-UA-Compatible: IE=Edge

X-Robots-Tag: noindex

Content-type: text/html; charset=UTF-8Preformatted text

I’ve posted some lines from the Log File generated by phpList. I’ve redacted the addresses but show that it is all the emails being sent.

Is there a way to garner more detail showing the steps taken and time for each step? I know this will slow it more but should show where the backup is occuring.

Thanks,
Doug Davis

11

@Dougster Unless you have some unusual setting that might cause a delay, such as fetching remote content for the message, the time is going to be taken up by the SMTP interactions. The time taken by phplist to build each email is pretty small, as I showed.

If you enable SMTP debug then send a test email the debug output should include timings for each step.

There are a couple of code changes needed to display the time in microseconds

file admin/sendemaillib.php
$mail->Debugoutput = 'html';
to
$mail->Debugoutput = 'echo';

file admin/PHPMailer/class.smtp.php

change

echo gmdate('Y-m-d H:i:s')."\t".str_replace(

to

        $now = DateTime::createFromFormat('U.u', number_format(microtime(true), 6, '.', ''));
        echo $now->format("m-d-Y H:i:s.u")."\t".str_replace(

02-19-2019 15:31:29.787053 CLIENT -> SERVER: X-phpList-version: 3.3.9

1 Like
12

I turned off the Verbose and tried Debug Level 4 … it seems to need to try connecting with CRAM-MD5 a couple of times before it gets through. This doesn’t seem like something it should do. Is there a better security setting to use that will speed the process?

Thanks,
Doug

13

Is there some way to store and send the SMTP password in SHA format? I’ve received this request from our cloud email provider.

14

These are the authentication methods supported by phpmailer

foreach (array('CRAM-MD5', 'LOGIN', 'PLAIN', 'NTLM', 'XOAUTH2') as $method) {

1 Like
15

Is there a switch to set in the Config file to force it to CRAM-MD5 only?

16

CRAM-MD5 is the first in the list so will be used if the server supports it.

1 Like
17

I just want to thank everyone who offered assistance with this issue. I was finally put in touch with the admins at our cloud email provider who admitted that they were throttling our email. Yeah, the same ones who said, no, we don’t do that.

I immediately switched the system to sending from my cloud host provider and I send the whole 75,000 emails in two hours. So, when your email provider says they don’t do that … don’t believe them!!

1 Like