Parallel drip campaigns, do not welcome more campaigns when added

1

Good day,

I would, as an example, have six campaigns with 1 000 unique recipients each.

I would start a parallel drip campaign. I would start the first four campaigns, with a cron job that triggers once a day during the week, it would trigger 200 emails to be sent, 50 from each campaign.

This would work, and every day 200 emails would be sent, 50 from each campaign.

define(‘PROCESSCAMPAIGNS_PARALLEL’,true);
define(‘MAILQUEUE_BATCH_SIZE’, 200);
define(‘MAILQUEUE_BATCH_PERIOD’, 600);
define(‘MAILQUEUE_THROTTLE’, 0.5);
define(‘MAILQUEUE_AUTOTHROTTLE’,0);

Then when I would want to add another two campaigns, and up the batch total to 300 (50 from each of the six campaigns), only 200 emails would be sent from the original campaigns, and the new campaigns would be ignored.

define(‘PROCESSCAMPAIGNS_PARALLEL’,true);
define(‘MAILQUEUE_BATCH_SIZE’, 300);
define(‘MAILQUEUE_BATCH_PERIOD’, 900);
define(‘MAILQUEUE_THROTTLE’, 0.5);
define(‘MAILQUEUE_AUTOTHROTTLE’,0);

It is as if the parallel processing makes the two events, original four and two additional campaigns, to be out of sync.

If I pause the four old events, I can send the new campaigns with a cron job.
When I pause all of them, and then requeue all of them, only the original four campaigns will send their percentage (50 each).

Am I doing something wrong, or how could I possibly sync all the campaigns?

Thank you, to everyone in advance, for your help.
Phplist v3.6.15 PHP 8.1

2

@Mailman99 You probably need to define MAX_PROCESS_MESSAGE in config.php as the default value is 5. Also, enable VERBOSE to get more detailed output to the event log page which might help understand what is happening.

3

Dear @duncanc

I really appreciate your reply, and helping me to figure this out.

The MAX_PROCESS_MESSAGE is set to 999, so it cannot be that ( I am sending maximum six campaigns at a time). Verbose was enabled, so I looked at that, and deleted all the old logs, so I knew where to start.

As an experiment, I set four campaigns (of which two are giving me trouble—> not starting to send with the others), to send only 2 emails each of a total batch of 8, and it worked. All four campaigns took part.

When I reverted to my old settings this morning (hoping that the campaigns were now “in sync”), again only two campaigns took part to send their share of emails. Only half the emails were sent, and two campaigns did not participate.

I looked in the log and I can see the system started with good intentions:
9 May 2024 10:55:02 4 campaigns to process
9 May 2024 10:55:02 Processing campaign 20…
9 May 2024 10:55:44 Processing campaign 21…
and the very last processqueue entry is:
9 May 2024 10:56:18 Sending 21 to john@mail.com
After this only processbounces entries

This exact system is working on quite a few other domains, so I am wondering if it does not work here perhaps, because I added some campaigns later?
Could it be that the system needs a second trigger (second cron job a few minutes later?), to send the rest? It worked when I was sending very few emails with this system.

My actual settings for the four campaigns are.

define(‘PROCESSCAMPAIGNS_PARALLEL’, true);
define(‘MAX_PROCESS_MESSAGE’, 999);
define(‘MAILQUEUE_BATCH_SIZE’, 632);
define(‘MAILQUEUE_BATCH_PERIOD’, 316);
define(‘MAILQUEUE_THROTTLE’, 0.2);
define(‘MAILQUEUE_AUTOTHROTTLE’,0);

The Mailqueue_Throttle made no difference when it was 0.5

Any help will be appreciated.
Thank you.

4

There should be entries in the event log for completing the batch processing, ending with “Finished this run”.
If those are not present then phplist might be failing. Look for a php error log file, sometimes one is created in the admin directory named error_log.

5

Hi

Thank you @duncanc for your feedback. I tried everything, but I could not produce any error logs. I thought I would send the few non-starter campaigns, manually, and just start off correctly this time (send all the campaigns at exactly the same time), but I have the same situation.

I am using Amazon SES to send the emails with the plugin, and I am allowing enough time between campaigns ( sending at 0.2 seconds per email, but allowing 1 second per email for the total batch time). Despite this, the last campaign in the queue (or more than one), are not sent, or partially sent for the day, depending on the domain (where most domains are on the same server host).

The next day, those campaigns are again ignored, or partially sent, even if I allowed more time in the config file for sending.

The event logs show correctly, for instance 5 campaigns to process, but it would seem to stay on the path of the first day that emails were sent, as if the batches was determined on the first day of sending, then no changes to the config file would allow amending the situation, by allowing more time for instance. I would change the sending time from 0.2 to 0.7 seconds between emails, and still the same campaigns would not be fully sent. If I were to add another campaign to be sent drip wise, that would be ignored.

Is there a way to resync the campaigns, without resending to the email addresses that already received the campaign? I would pause the campaigns, go to edit, and then just put them in the queue again, without any changes to the situation. If I pause the campaigns that were sent, and restarted the cron job, some of the ones that were not sent would begin to be sent (and only their amount per campaign), but not always all of them.

It is as if the system determined the batch information on the first day, and would just stick to that, without changes to the config helping to change anything.

I love this software, and I have been using it for many years, across many domains, and I still discover new ways to use it, or new ways to use it more efficiently.

Any help would be appreciated, even a workaround.

6

@Mailman99 There is too much to take in here. Can you give me access to your phplist so that I can look at the event log page? Send me a private message with the details.

7

@Mailman99 There does appear to be something making phplist fail.

Looking at the event log page the processqueue entries just stop, e.g. at 13:39:55 entry 37281"Sending 15 to laura@oleimports.com" but the expected “It took xx seconds to send” does not appear.

Can you enable error reporting in file admin/init.php
line 9 should be

error_reporting(0);

change that to

error_reporting(-1);

and also capture the output of the cron job to a file, if not already doing that, e.g.

php /path/to/phplist/admin/index.php -p processqueue >processqueue.log 2>&1

If an error is occuring then it might be reported in the log file.

Also, check with your hosting company whether there is a time limit on cron jobs in case the cron job is being killed.

8

Hi Duncan,

I will follow all your suggestions. I really appreciate your help, and I will get back to you.

Thank you very very much.

9

Hi

Thank you to @Duncanc for figuring out the answer. I was using the Amazon SES plugin, and the “Whether to use curl multi to send emails concurrently” should have been set to “No”. Once I set it to “No”, everything was working the way it should.

Thank you again to Duncan for taking the time to find the answer. I appreciate it greatly.

Thank you to everybody.

10

@Mailman99 I suspect it was a problem with running out of memory, but without seeing an error log file it is not possible to confirm that. You can disable VERBOSE in the config.php file now.

11

Dear @duncanc

I will disable VERBOSE.

Thank you again for your help.