Sending simultaious campaigns and list splitting

Hi duncanc,

We are upgrading from PHPList 2.10.15 to 3.2.4. In 2.10.15 we are able to have 5 campaigns running at the same time to the same lists of email addresses. Also, PHPList knew not to send more than one message per minute to the same domain. We had to start the campaigns at different times, we have them scheduled (embargoed) an hour apart. If we had two ready to be sent at the same time, say it was 10:00 and we had one embargoed for 9:00 (that we forgot to process) and one for 10:00 and we clicked process queue, the system would do as duncanc states above, it would literally split the list of emails between the two campaigns and eventually the campaigns would show negative numbers in the sent column. So, we were very careful that we never had more than one queued up to start at the same time. As long as they were processed at different times, we could have 5 campaigns running at the same time to the same full lists of recipients, the same email addresses were not receiving the 5 campaigns at the same time, but eventually throughout the day each email address on the list would receive the 5 campaigns. It sounds like you are saying PHPList is no longer capable of doing this. What would be the use in having the list split up between the campaigns so each campaign is only sent to part of the list?

I didn’t really understand your point. In my earlier posts I explained about the PROCESSCAMPAIGNS_PARALLEL setting. Please open a new topic instead of adding to someone else’s old (three weeks) topic.

moved to new thread. @jben you my wish to break it down a bit more, that’s a lot of text and it’s not very clear to me either

Kind Regards

Anna
phpList community manager

Hi Anna,

Sorry for the confusion. The reason I posted on the other thread is the original poster was having the same problem I am having. We turned on Parallel processing, but we are still only able to have one campaign running at a time. We start campaign “A” to 30k recipients, it is running fine the xxx still to process number is going down, emails are going out. One hour later we process the queue to start campaign “B” to the same list of 30k recipients (a different advertisement for a different client to the same 30k recipients) and it just sits there showing submitted 30k still to process, the still to process number is not going down. In PHPList 2.10.15 we would process a campaign every hour for 5 hours. There would be 5 campaigns showing submitted xxx still to process and as you refreshed the view, the still to process number for all 5 would be going down. The old system did not wait until the first was done before sending the next. duncanc made it sound like PHPList would split the email list up between the campaigns, so campaign A would only go to 15k of the 30k and campaign B would be sent to the other 15k instead of each campaign being sent to all 30k (each of the 30k recipients receiving both campaigns); not sure this was what he intended to convey.

Can you describe how you did this? Which file etc etc.

You might have the wrong idea about what “parallel processing” actually does. I did explain that in the other topic but will repeat it here. From the config file

# process parallel
# if there are multiple messages in the queue, divide the max batch across them
# instead of sending them one by one.
# this only works if you use batch processing. It will divide the batch between the
# campaigns that need sending.
#define('PROCESSCAMPAIGNS_PARALLEL',true);

phplist will divide the batch across each available message. It does not support running more than one process at the same time.

In your case, you need to use batch processing, which isn’t clear whether you are, and you then want to have a reasonably small batch size and related period so that phplist is switching between the available messages quite often.

So for example, if you are able to send 2000 emails each hour, have a cron job that runs every 6 minutes (10 times each hour) with a batch size of 200 (= 2000 / 10). phplist should then split the batch size into two and send 100 emails from the first available message, and another 100 from the second available.

Of course, juggle these numbers to match your actual limits, but the key thing is to have phplist process a batch quite frequently.

When there is only one available message then phplist will use the whole batch size for that.

As I was waiting for your reply, my IT programmer had figured this out. He is adjusting the numbers now to get the processing rate that works best for us.

Our old PHPList must have been set up for this and it just wasn’t obvious what was happening in the background. Our old PHPList was on a Linux server while our new PHPList is on an IBM iseries server, so it is a bit different programming wise.

duncanc - I did misinterpret your previous post regarding the “splitting” of the batch. Good to know the campaign will eventually be sent to all recipients. :blush:

Thank you both [duncanc & anna] for your help. Have a good day! :sunny:

1 Like

Trying to understand why this is not working for us. Relevant settings are:

define(‘MAX_PROCESS_MESSAGE’, 999);
define(‘PROCESSCAMPAIGNS_PARALLEL’,true);
define(“MAILQUEUE_BATCH_SIZE”,700);
define(“MAILQUEUE_BATCH_PERIOD”,600);
define(“MAILQUEUE_THROTTLE”,0.7);
define(‘MAILQUEUE_AUTOTHROTTLE’, 0);

define(‘USE_DOMAIN_THROTTLE’,1);
define(‘DOMAIN_BATCH_SIZE’,4);
define(‘DOMAIN_BATCH_PERIOD’,120);

cron setup every 10 minutes

We are estimating a max of 4200 messages per hour with this setup.

When we have various messages running at the same time only the first one seems to be processed.
Also the active messages web interface shows:

“processing 240 msgs/hr”

Been trying to work this out for years without luck! Any assistance on how to debug would be greatly appreciated.

@luison Don’t use domain throttle because it will slow down the sending rate in an unpredictable way.

If you run processqueue only as a cron job, not through the admin interface, then you can reduce the batch period to a small period, say 10s. phplist limits the size of a new batch to BATCH_SIZE less the number of emails actually sent in the past BATCH_PERIOD, which will include the previous batch when the value is 600.

Thanks @duncanc. I see your point and will test but I would love to understand it. With the setting expressed (I also used an online excel calculator i found on an old forum) I must be wrong in the figure of 4200 per hour as over the last 18 hours I’ve only managed to send approx 9000 messages which is far off my estimate of 4200 per hour.

You suggest shorter batch periods but I have the impression when I had longer periods with more messages the delivery was better. The reason for moving to shorter ones is the parallel processing but not at the cost of having slower delivery.

I final question. Does the ‘MAILQUEUE_AUTOTHROTTLE’ still uses the MAILQUEUE_THROTTLE value between messages or not?
Thanks again.

@luison The sending rate will be reduced because of the domain throttle, which is why I recommended not to use that.
Keep the cron job frequency of 10 minutes, but reduce the value for MAILQUEUE_BATCH_PERIOD to try to stop phplist reducing the number sent in each batch.
The throttle value might be too high to send 700 emails in 600s, because it is in addition to the time taken to construct and send each email.

Thanks, already trying that but a prove of my lack of understanding… I was convinced that the MAILQUEUE_BATCH_PERIOD was related to the cron frequency and that your suggested config would make phplist not send anything during 9 minutes.

trying a MAILQUEUE_BATCH_PERIOD of 60 seconds with a batch of 600 and a 0.8 delay.