PhpList *appears* to be sending to a huge list instead of the selected lists?

Hi,

I typically send out emails once every 6 months, after a release of my software. At that time I send out 2-3 emails to several lists using include and exclude. Last year, roughly around this time, I also started using Duncan’s great Segmentation plugin. Many tens of thousands of emails are sent over a period of a few weeks.

Anyway, so much for the intro, just to illustrate that I have been using my current setup successfully for quite a while.

Yesterday I wanted to send out a quick email to a very small subset of people (~4000 people), alerting them of a beta release, and to the list of my translators (30-40 people in the latter list)

This morning I woke up to very strange results and I stopped sending entirely until I can figure this out. Something ain’t right. That’s all I can say at this point from past experience with sending emails.

For starters both campaigns were still running ?
The email to the small list of 30-40 people had already been sent to 2109 people ?
The email to the ~4000 people list had already been sent to 23927 people ?

The latter still to process value keeps steady, as the total value increases ?

I’m nervous as to what is happening ? Is phplist truly sending to that many people ?
Is it counting something else ? This has not happened before (last time used in June/July)
How can I check who the email was actually sent to ?

My server went through a CentOS upgrade in September

I actually noticed it counting ‘funny’ last night already but thought the values would set to proper values once done.
I also changed the config.php file last night, to previous (successfully used) setting thinking maybe it was that ?

Set back from:

define(“MAILQUEUE_BATCH_SIZE”,0);
define(“MAILQUEUE_BATCH_PERIOD”,2);
define(‘MAILQUEUE_THROTTLE’,1);

to:

define(“MAILQUEUE_BATCH_SIZE”,250);
define(“MAILQUEUE_BATCH_PERIOD”,2);
define(‘MAILQUEUE_THROTTLE’,2);

My Setup:
phplist version: 3.0.12
Segmentation version 2.1.8+20160515 by Duncan Cameron
Common Plugin version 3.5.7+20160527 by Duncan Cameron

These cron jobs are running:

Every 5 minutes:

        /usr/local/bin/php -q 
        /home/smart/public_html/mlist/admin/index.php -c 
        /home/smart/public_html/mlist/config/config.php 
        -pprocessbounces

Every 10 minutes:

         /usr/local/bin/php -q 
         /home/smart/public_html/mlist/admin/index.php -c 
         /home/smart/public_html/mlist/config/config.php 
         -pprocessqueue

Any insights as to what is happening ?
Any idea if upgrading to the latest and greatest phplist version will fix things ?

just looking over that report, the system looks like it’s counting some other campaign for sending, and never sent an email to the lists you wanted.

It seems to me that the database might not have been fully converted, or the table with campaign numbers is somehow not right.

Since it’s an old version, I’d probably recommend an upgrade, and see what it tells you then. Please take a couple of minutes to make a backup of your phplist directory, and the database.

Then do the upgrade.

Dan

Thanks Dan,

From the ~4000+ people list I have not had a single email as feedback so I wonder if anybody got it.
From the translators I have had several emails already. So some definitely got it !

You think the upgrade might fix this weird behaviour ?

How do I make a backup of the database ? What type file is it and where should I look for it ?

Is there a text on how to upgrade ?
It’s been so long since I installed the latest that I forgot how to do it.

Hi @ibPeter,

To backup the database, you would need to go into the cpanel or plesk, to phpMyadmin, and use the ‘export’ button/command. It will create and send you the commands to recreate the database.

To ‘backup’ the phplist install, I usually rename the ‘lists’ folder to something like ‘lists.3.0.12’… you can then compress that folder, and download it to your computer. this would be the backup of the file system. This again is done in the cpanel of your host.

For upgrading, you would need to upload the 3.2.6 version of the software, and re-use the settings of the previous config.php file. It’s a good idea to go into the new version of the file, and copy the settings from the older version into the newer version.

Sorry for the late reply, only now did I manage to do the update.
All seems to have gone well.

When I logged on phplist said I had to update the database, and that it could take a long time on a huge database (which mine is). However this was done lightning fast. A second at best. I hope this is OK going from previous version to this version ?

I haven’t done any other testing yet.
Would you recommend trying a few of the ‘System’ options ?
I did try ‘Verify the DB structure’ and that seemed to indicate everything is Ok.

PS. Something I noticed with previous version and now again with the latest and greatest:

I edited a campaign’s content. Went to a different tab (Format). Sent test email. Clicked the Content tab and got this:

Could this be indicating what is wrong with my setup ?

[EDIT]
I just had it again going to the last tab.
A simple F5 (refresh) of the page worked.
Perhaps this is FF and not phplist at all ?

what version of php is installed on your server?

you should have 5.6 or greater.

I just checked: 5.5.38
What version did phplist version: 3.0.12 need ?
because all the symptoms are the same and phplist version: 3.0.12 worked just fine until I started noticing issues a few weeks ago.

PS. I just checked and the requirements are php 5.4 or higher !?
https://resources.phplist.com/system/start?s[]=requirements

Let’s go back to the initial findings and I have more information as well.
Hopefully somebody can get me started again, because I need to start sending emails on Monday.

I realize I cluttered my original findings by also starting the queue processing by clicking the links via the website.
I did not do that this time. I relied purely on the cron jobs (see the crons in my original question), which is what I used to do before as well.

A better title for this Q would be: “PhpList doesn’t send email and keeps cycling through selected lists again and again during sending of a campaign”

  1. I upgraded from phplist version: 3.0.12 to the latest: 3.6.2

  2. I’m still seeing the same problems (I believe) as what I saw with previous version

  3. I used to successfully send lots of emails with phplist version: 3.0.12 until it stopped about 2 weeks ago (but I had not tried sending emails for roughly 6 months either)

  4. Using the latest and greatest setup I created a campaign to be sent to a combination of lists, to roughly 4300 people

  5. I finished editing, made sure it was not embargoed, and placed the campaign in the queue (dark button in last tab)

  6. Now I waited for the crons to start the process. See crons every 5 and 10 minutes in the initial question.

  7. I did not get the email that says the process has started (I normally do - not anymore) ?

  8. In the list of Active campaigns, I could see that there was activity ! The ‘Still to process’ number was counting down, the ‘total’ field was increasing, and the total of the two was roughly 4300. So it did seem the cron jobs were doing what they had always done.

  9. I did not get the phplist-sent emails that I normally get during each cron, when a campaign is active, giving details on how many emails are being sent per batch, bounced emails being processed etc.
    Is this something that can be turned off, because I might have done that in the past ? If so where do I turn it on. It may however also show that there is a problem.

  10. Since I was not getting the phplist-sent emails that I was so used of, I told my CPanel to send an email on each cron job, providing the output.

.11. For the processqueue cron I always got:

phpList version 3.2.6 (c) 2000-2016 phpList Ltd, http://www.phplist.com
PHPlist - A process for this page is already running and it was still alive 1 seconds ago
PHPlist - Running commandline, quitting. We'll find out what to do in the next run.

Never the content that I was used of, detailing the amount of emails being processed in a batch etc.

.12. For the processbounces cron I always got:

phpList version 3.2.6 (c) 2000-2016 phpList Ltd, http://www.phplist.com
PHPlist - 0 bounces to fetch from the mailbox
PHPlist - Please do not interrupt this process
PHPlist - Closing mailbox, and purging messages
PHPlist - Download failed, exiting

There were never bounces ??
I honestly doubt emails were being sent out to start with ?
Even though graphically, via the website Active campaigns, +1500 emails per hour were being sent out.

.13. I was cheeky and sent an email to the email box reserved for bounces.
It was seen by the cron and processed but at this point my server also started sending me emails saying the load on the server was abnormally high.

I do not understand the output sent via the Cpanel cron email (big email) :

PHPlist - 1 bounces to fetch from the mailbox
PHPlist - Please do not interrupt this process
PHPlist - Closing mailbox, and purging messages
PHPlist - reprocessing
PHPlist - 5703 bounces to reprocess
PHPlist - 25 out of 5703 processed
...
PHPlist - 5703 out of 5703 processed
PHPlist - Identifying consecutive bounces
PHPlist - Processed 0 out of 14022 subscribers
...
PHPlist - Processed 14020 out of 14022 subscribers
PHPlist - total of 14022 subscribers processed

The bounce email inbox was empty when I sent the one email. Only one email was seen and deleted again.
Where does this number 5703 come from ?

What section of the database needs to be checked fully every 5 minutes (and if there are bounce emails) ?
That doesn’t strike me as usual either (I do not remember happening this before). Is this not something that needs to be done at the end of a campaign ? btw. 14022 is a number I don’t understand either. I first thought it was going through the full database, but it contains 10 times as much entries. So this doesn’t make sense either.

.14. I watched the Active Campaign page as it finished sending to the 4300 people, as
’still to process’ emptied and ‘total’ reached 4300 … and … then it started again ? It is my believe now that this process keeps on going, which is why I thought when I first formulated the question that the campaign was being sent to all, instead of the selected lists. The total counter just keeps increasing however and I stopped the campaign when it reached 4600 people with still 4000 or so people to go.

.15. The bounced counter says 0, the viewed counter says 1 (I wonder about this one). I’m pretty sure I have a few of my own email addresses in that list. I received no email.

.16. Test emails, when editing the campaign, always arrive immediately. Sending email in itself seems to work ok.

Does anybody have any idea what broke on my server that all this suddenly doesn’t work anymore ?
Of course the managed server software undergoes updates … I assume it’s linked … but any idea what this could be ?

I hope I did not forget to tell anything and that this level of detail with spark a eureka moment with somebody ?

@ibPeter There seems to be a number of problems here.
The cron job for processing bounces is being run far too frequently, every 5 minutes. It probably hasn’t finished before the next cron job runs. Change that to run every two hours.

The message

A process for this page is already running and it was still alive 1 seconds ago

also means that the cron job to process the queue is being run again before the previous processqueue has finished. So you need to run that less frequently.

Look at the event log page to see whether errors in sending are being reported there.

Thank you @duncanc

Change that to run every two hours.

So strange because this always ran OK ?
But I’ll change it now.

The email inbox for bounce emails is empty however !
Nothing is stuck there !

Hence there is nothing for processbounces to do to begin with.
Me manually sending in one email triggered it alright !
So I doubt this is the problem I’m chasing.

also means that the cron job to process the queue is being run again before the previous processqueue has finished

Same observation from my part.
But it feels it never even started to begin with ? Although I missed that message since I had not yet configured Cpanel to send me emails.
So what previous process would still be active ?

Or could it be that something ‘got stuck’ 6 months ago when I sent out the last big email (10s of thousands of emails) ?
and it’s still having issues with that ?
If memory serves me right I experimented then with sending them all fast, not in batches.

Still 6 months should have cleared whatever job it is still working on …

Look at the event log page

Where in phplist is that ?

I have to log off. Getting late in AU and have to get up at 5 tomorrow.
I hope to get to the bottom of this soon. I appreciate you looking into this with me.

FYI, the discussion moved to: