back to phpList.org

Bounces body is not loading, bounces are not processed


#4

In the pop3 mailbox the bodies are in place. And reasons for bounce are written in the body.
So, it is not because of the bounce itself, but because phplist’s reading script for some reason doesn’t write the body into the database.


#5

@leo2 Possibly something odd about the structure of the bounce that makes phplist unable to process it. Can you provide an example of a bounce from the pop3 mailbox, either copy it here or add it as an attachment?

Also if you look at the phplist_bounce table, what is shown for the data column? If it is a BLOB then phplist is processing the bounce body, so take a look at that to see what is present.


#6

Example of bounces
This message was created automatically by mail delivery software.

A message that you sent could not be delivered to one or more of its
recipients. This is a permanent error. The following address(es) failed:

  k_resheniya@mail.ru
    SMTP error from remote mail server after end of data:
    host mxs.mail.ru [94.100.180.150]: 550 Message was not accepted -- invalid mailbox.  Local mailbox k_resheniya@mail.ru is unavailable:
    account is disabled

------ This is a copy of the message, including all the headers. ------

Return-path: <bounce@solarhome.ru>
Received: from vh283613 by vh09.hostline.ru with local (Exim 4.84_2)
    (envelope-from <bounce@solarhome.ru>)
    id 1d5aFb-0019E7-Et
    for k_resheniya@mail.ru; Tue, 02 May 2017 19:02:07 +0300
To: k_resheniya@mail.ru
Subject: =?UTF-8?B?0KHQutC40LTQutC4INC90LAg0L3QsNGH0LDQu9C+INC80LDRjyAtINC+0YI=?=  =?UTF-8?B?0LzQtdGC0LjQvCDQv9GA0LDQt9C00L3QuNC60Lgg0YEg0LLRi9Cz0L7QtNC+?=  =?UTF-8?B?0Lkh?=
X-PHP-Originating-Script: 3216:class.phpmailer.php
Received: from ip-176-192-35-206.bb.netbynet.ru [176.192.35.206] by lists.solarhome.ru with HTTP; Tue, 02 May 2017 19:02:00 +0300
Date: Tue, 2 May 2017 19:02:07 +0300
From: "shop@solarhome.ru" <shop@solarhome.ru>
Message-ID: <759df84c8f1b89615ffa8aade4f7ecce@solarhome.ru>
X-Mailer: PHPMailer 5.2.22 (https://github.com/PHPMailer/PHPMailer)
X-phpList-version: 3.3.1
X-MessageID: 98
X-ListMember: k_resheniya@mail.ru
Precedence: bulk
Bounces-To: bounce@solarhome.ru
List-Help: <http://www.solarhome.ru/lists/?p=preferences&uid=edc0c857305d3b8c881a926d4537bb40>
List-Unsubscribe: <http://www.solarhome.ru/lists/?p=unsubscribe&uid=edc0c857305d3b8c881a926d4537bb40&jo=1>
List-Subscribe: <http://www.solarhome.ru/lists/?p=subscribe>
List-Owner: <mailto:info@solarhome.ru>
MIME-Version: 1.0
Content-Type: multipart/alternative;
    boundary="b1_759df84c8f1b89615ffa8aade4f7ecce"

This is a multi-part message in MIME format.

--b1_759df84c8f1b89615ffa8aade4f7ecce
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

**=D0=A1=D0=BA=D0=B8=D0=B4=D0=BA=D0=B8 =D0=B2 =D0=BC=D0=B0=D0=B5 2017**

PHPlist record for this bounce is in unrecognised bounces as follows:
ID 2920 Date 2017-05-02 19:02:00 Status unidentified bounce

Comment not processed

Header
CloseReturn-path: <>
Envelope-to: bounce@solarhome.ru
Delivery-date: Tue, 02 May 2017 19:02:08 +0300
Received: from exim by vh09.hostline.ru with local (Exim 4.84_2)
id 1d5aFb-0019EK-Vg
for bounce@solarhome.ru; Tue, 02 May 2017 19:02:08 +0300
X-Failed-Recipients: k_resheniya@mail.ru
Auto-Submitted: auto-replied
From: Mail Delivery System Mailer-Daemon@vh09.hostline.ru
To: bounce@solarhome.ru
Subject: Mail delivery failed: returning message to sender
Message-Id: E1d5aFb-0019EK-Vg@vh09.hostline.ru
Date: Tue, 02 May 2017 19:02:07 +0300

Body


#7

I can see that phplist writes the failed address to this field

X-Failed-Recipients:

I don’t see it in the original bounce message.

Another example:
This message was created automatically by mail delivery software.

A message that you sent could not be delivered to one or more of its
recipients. This is a permanent error. The following address(es) failed:

  aes1@mail.ustu.ru
    SMTP error from remote mail server after end of data:
    host ironport10.urfu.ru [212.193.87.26]: 550 5.7.1 This message is blocked due to security reason

------ This is a copy of the message, including all the headers. ------

Return-path: <bounce@solarhome.ru>
Received: from vh283613 by vh09.hostline.ru with local (Exim 4.84_2)
    (envelope-from <bounce@solarhome.ru>)
    id 1d5aZI-001EFn-Jg
    for aes1@mail.ustu.ru; Tue, 02 May 2017 19:22:28 +0300
To: aes1@mail.ustu.ru
Subject: =?UTF-8?B?0KHQutC40LTQutC4INC90LAg0L3QsNGH0LDQu9C+INC80LDRjyAtINC+0YI=?=  =?UTF-8?B?0LzQtdGC0LjQvCDQv9GA0LDQt9C00L3QuNC60Lgg0YEg0LLRi9Cz0L7QtNC+?=  =?UTF-8?B?0Lkh?=
X-PHP-Originating-Script: 3216:class.phpmailer.php
Received: from ip-176-192-35-206.bb.netbynet.ru [176.192.35.206] by lists.solarhome.ru with HTTP; Tue, 02 May 2017 19:22:18 +0300
Date: Tue, 2 May 2017 19:22:28 +0300
From: "shop@solarhome.ru" <shop@solarhome.ru>
Message-ID: <b50ec56f3601cabc2604cc9eacc68bb9@solarhome.ru>
X-Mailer: PHPMailer 5.2.22 (https://github.com/PHPMailer/PHPMailer)
X-phpList-version: 3.3.1
X-MessageID: 98
X-ListMember: aes1@mail.ustu.ru
Precedence: bulk
Bounces-To: bounce@solarhome.ru
List-Help: <http://www.solarhome.ru/lists/?p=preferences&uid=e68a0cd06b8e11e7a539014c5070b1b4>
List-Unsubscribe: <http://www.solarhome.ru/lists/?p=unsubscribe&uid=e68a0cd06b8e11e7a539014c5070b1b4&jo=1>
List-Subscribe: <http://www.solarhome.ru/lists/?p=subscribe>
List-Owner: <mailto:info@solarhome.ru>
MIME-Version: 1.0
Content-Type: multipart/alternative;
    boundary="b1_b50ec56f3601cabc2604cc9eacc68bb9"

This is a multi-part message in MIME format.

--b1_b50ec56f3601cabc2604cc9eacc68bb9
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

correspondent record in the phplist:

Bounce details
ID
2922
Date
2017-05-02 19:22:00
Status
unidentified bounce

Comment
not processed

Header
Close
Return-path: <>
Envelope-to: bounce@solarhome.ru
Delivery-date: Tue, 02 May 2017 19:22:30 +0300
Received: from exim by vh09.hostline.ru with local (Exim 4.84_2)
id 1d5aZK-001EGQ-Ia
for bounce@solarhome.ru; Tue, 02 May 2017 19:22:30 +0300
X-Failed-Recipients: aes1@mail.ustu.ru
Auto-Submitted: auto-replied
From: Mail Delivery System <Mailer-Daemon@vh09.hostline.ru>
To: bounce@solarhome.ru
Subject: Mail delivery failed: returning message to sender
Message-Id: <E1d5aZK-001EGQ-Ia@vh09.hostline.ru>
Date: Tue, 02 May 2017 19:22:30 +0300


Body

#8

I have no rules yet. How to create the rules for the bounces I have? I try to press ‘Create new rule from this bounce’ but just move to the rule creation fields. How to create the rules?

Why phplist put the bounces to the Undefined, in spite X-Failed-Recipients: are in the subscribers lists?


#9

Another thing - I have similar to following bounces from host rumbler.ru

This message was created automatically by mail delivery software.

A message that you sent could not be delivered to one or more of its
recipients. This is a permanent error. The following address(es) failed:

  voltmaster@rambler.ru
    SMTP error from remote mail server after MAIL FROM:<bounce@solarhome.ru> SIZE=61920:
    host inmx.rambler.ru [81.19.78.64]: 554 5.7.1 Client host blocked

Seems te delivery is blocked by remote mail server.
But in the phplist I have ‘standard’ record, from which there is no evident that the mailbox exists and I don’t have to remove it from my list (because I have such messages for all emals @rambler.ru)

Bounce details
ID 2939 Date 2017-05-03 08:07:00 Status unidentified bounce

Comment  not processed

Header
Close
Return-path: <>
Envelope-to: bounce@solarhome.ru
Delivery-date: Wed, 03 May 2017 08:07:22 +0300
Received: from exim by vh09.hostline.ru with local (Exim 4.84_2)
id 1d5mVV-003xdd-RU
for bounce@solarhome.ru; Wed, 03 May 2017 08:07:22 +0300
X-Failed-Recipients: timurb1972@rambler.ru
Auto-Submitted: auto-replied
From: Mail Delivery System <Mailer-Daemon@vh09.hostline.ru>
To: bounce@solarhome.ru
Subject: Mail delivery failed: returning message to sender
Message-Id: <E1d5mVV-003xdd-RU@vh09.hostline.ru>
Date: Wed, 03 May 2017 08:07:21 +0300

#10

@leo2 Thanks, but I’m not sure what you are showing. Please can you copy a complete email from the POP3 account, so that we can see it before it is processed by phplist.
I need the complete email including headers, so you will have to “view source” or something similar.


#11

I can not load attachments here. I can not copy full email body because it has links and I am not allowed to put >2 links here (I have 3 links in header only)

So you can download the example from here https://www.dropbox.com/s/4ncsvt687ji0vl7/bounce-example.txt?dl=0

PS. I opened the bounces in Roundcube, which I use very rarely. So, the header above were not from the bounce, they were from the message sent. I hope the txt in dropboxt reflects full bounce including its headers. There is that field X-Failed-Recipients:
But why phplist doesn’t recognize a user from that field?


#12

@leo2 In Roundcube you can view the source of an email using the More button

Also, I have compared your bounce with one generated in my phplist installation and there is one difference that might be the cause of the problem. The bounce does not have a Content-Type header. That might cause the php imap module, which does the processing of a bounce email, to not recognise it, because it appears that phplist is being given an empty bounce body.


#13

yes, I already found how to see the headers in roundcube. The file in dropbox has that headers.

How to fix that? I have actually 2 phplists instances, first is main and it sends emails. But it has some error in php imap and can not download bounces (no connection to pop mailbox), and it was same for v2.xx. So had to install phplist in another virtual hosting which can download the bounces to database. I wish I moved completely to that new instance, but they have limit for sending emails (100 per hour). Main hosting has restriction 10 per minute, so I use it to send emails. Both instances use same database.


#14

@leo2 I was mistaken about the cause of the problem. I now think that it is a problem with the line endings of the bounce email when phplist processes it. That might indicate a problem with the mail server.

You can try to work around the problem by making two changes to the phplist source code.
file admin/processbounces.php

line 121 is
if (preg_match('/(?:X-MessageId|X-Message): (.*)\r\n/iU', $text, $match)) {

remove the two characters \r

line 134 is
if (preg_match('/(?:X-ListMember|X-User): (.*)\r\n/iU', $text, $match)) {

remove the two characters \r


#15

I removed \r as you advised, and downloaded the bounces again, but with same result - they are all in ‘unrecognized’, now twice.

I have in my config

$bounce_mailbox_purge = 1;
$bounce_mailbox_purge_unprocessed = 0;

and all the bounce messages are still in the mailbox.

Another thing - ‘Process bounces’ reports as follows

100 done
Closing mailbox, and purging messages
Identifying consecutive bounces
total of 4 subscribers processed          

But nothing is in the ‘processed’. Where are these 4 processes subscribers?


#16

@leo2 Sorry but I don’t really have anything more to suggest.


#17

duncanc, thank you for you try.

Seems I would move to mailchimp for some of my mailing lists.

phplist 3 has GA feature and google shows that effectiveness of phplist mailings from my virtual hosting is not high. Only 20 links were clicked from more than 2000 emails sent to my subscribers. I am afraid that most of my mails go to spam because of lot of bounces I can not track.

I tried to manually remove the bounced emails from my list, but it takes days and is not effective.

I hoped that phplist3 statistics will work for me, but it doesn’t. Emails are not tracked.


#18

What version of php is on your phpList server. for phpList 3.3.1, php v5.4 does not process the bounces properly. When I upgraded to php v.5.6, it started working.


#19

I have php v5.6 and phplist v3.3.1


#20

i would debug this step by step…

  1. turn your manual bounce processing back on.
  2. the first thing is to make sure phpList can read the messages in the mailbox
  3. You should then see if it can recognize a bounce (this is done by reading the headers on the bounced email).

Once that is done, we can look into the next step.
If you have php V5.6, it should work, maybe you don’t have all of the php packages installed?


#21
  1. It is “manual” already
  2. As I wrote, it can read, but loads only headers, not message bodies. Examples are in first messages of this topic. See above. And another example below.
  3. No, all downloaded bounces are in ‘unrecognized’. Phplist can not identify a user from a bounce, though there are the field X-Failed-Recipients: with the user address in the downloaded bounce headers.

Return-path: <>
Envelope-to: bounce@pvpower.ru
Delivery-date: Sat, 24 Jun 2017 04:00:45 +0300
Received: from mail by ha001.justhost.ru with spam-scanned (Exim 4.87)
id 1dOZRJ-002rd7-GF
for bounce@pvpower.ru; Sat, 24 Jun 2017 04:00:44 +0300
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on ha001.justhost.ru
X-Spam-Level:
X-Spam-Status: No, score=0.0 required=8.0 tests=none autolearn=disabled
version=3.3.1
Received: from vh09.hostline.ru ([83.69.230.25])
by ha001.justhost.ru with esmtps (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256)
(Exim 4.87)
id 1dOZRF-002rVA-Id
for bounce@pvpower.ru; Sat, 24 Jun 2017 04:00:41 +0300
Received: from exim by vh09.hostline.ru with local (Exim 4.84_2)
id 1dOZRC-000qJg-QA
for bounce@pvpower.ru; Sat, 24 Jun 2017 04:00:35 +0300
X-Failed-Recipients: shsiv@mail.ru
Auto-Submitted: auto-replied
From: Mail Delivery System Mailer-Daemon@vh09.hostline.ru
To: bounce@pvpower.ru
Subject: Mail delivery failed: returning message to sender
Message-Id: E1dOZRC-000qJg-QA@vh09.hostline.ru
Date: Sat, 24 Jun 2017 04:00:34 +0300
Forward-Confirmed-ReverseDNS: Reverse and forward lookup success on 83.69.230.25, -10 Spam score
SpamTally: Final spam score: -10

the part below is not loaded into phplist

This message was created automatically by mail delivery software.

A message that you sent could not be delivered to one or more of its
recipients. This is a permanent error. The following address(es) failed:

shsiv@mail.ru
SMTP error from remote mail server after end of data:
host mxs.mail.ru [94.100.180.104]: 550 Message was not accepted – invalid mailbox. Local mailbox shsiv@mail.ru is unavailable:
account is disabled

------ This is a copy of the message, including all the headers. ------

Return-path: bounce@pvpower.ru
Received: from vh283613 by vh09.hostline.ru with local (Exim 4.84_2)
(envelope-from bounce@pvpower.ru)
id 1dOZRC-000qJX-Eh
for shsiv@mail.ru; Sat, 24 Jun 2017 04:00:34 +0300
To: shsiv@mail.ru
Subject: =?UTF-8?B?0J3QvtCy0L7RgdGC0Lgg0LjRjtC90Y8gMjAxNyDQuCDRgdC/0LXRhtC40LA=?= =?UTF-8?B?0LvRjNC90YvQtSDRhtC10L3RiyDQvtGCINCS0KHQlA==?=
X-PHP-Originating-Script: 3216:class.phpmailer.php
Date: Sat, 24 Jun 2017 04:00:34 +0300
From: "shop@solarhome.ru" shop@solarhome.ru
Message-ID: 61a94132e826fef0b5e441252082f9e0@lists.solarhome.ru
X-Mailer: PHPMailer 5.2.22 (https://github.com/PHPMailer/PHPMailer)
X-phpList-version: 3.3.1
X-MessageID: 100
X-ListMember: shsiv@mail.ru
Precedence: bulk
Bounces-To: bounce@pvpower.ru
List-Help: http://lists.solarhome.ru/?p=preferences&uid=0e4b9b3dcaca6daabcea1b7122e80323
List-Unsubscribe: http://lists.solarhome.ru/?p=unsubscribe&uid=0e4b9b3dcaca6daabcea1b7122e80323&jo=1
List-Subscribe: http://lists.solarhome.ru/?p=subscribe
List-Owner:
MIME-Version: 1.0
Content-Type: multipart/alternative;
boundary=“b1_61a94132e826fef0b5e441252082f9e0”

This is a multi-part message in MIME format.

–b1_61a94132e826fef0b5e441252082f9e0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable


#22

There are two things that need to happen for phpList to process a bounce:

  1. the bounce processing verifies that the bounce contains a valid subscriber
  2. if you are using advanced bounce processing, the system needs to run some regular expressions on the bounced email to determine if it can recognize the reason for a bounce, and process according to the rule for that reason.

On your bounce headers, I don’t see the headers that phplist puts on the email, so it can identify the contact.
Did you remove those or somehow modify the phpmailer software so that it does not add the headers?


#23

define(‘USE_ADVANCED_BOUNCEHANDLING’, 0);[quote=“danwaterloo, post:22, topic:2803”]
Did you remove those or somehow modify the phpmailer software so that it does not add the headers?
[/quote]
no, I use fresh phplist installation with database converted at upgrade from v2.19

What header are you looking for? Where should I search for them?
I copied in my previous message full header with part of the body for a bounce (without message itself). All technical info is there.

Moreover, for most bounces the info about why the message was rejected and what is concerned user email is in the first lines of the bounce body (not its header), but the body is not loading to phplist.