As all the email providers stopping support for pop/imap connectivity and require oAuth route to connect to their services, I am not able to find any references or a way to configure PHPList to connect to Outlook or Gmail using PHPMailer and through oAuth route.
I currently cannot connect to Outlook365 using imap configuration which uses username/password as Microsoft has discontinued the basic authentication support; therefore, I can no longer processes our bounced emails.
In short, processing bounce emails using pop or imap is no longer possible if email server is Outlook, Gmail, Yahoo, … and the only way it to use oAuth.
I hope I am not the only one facing this issue and there are solutions already implemented and available that I can get some help and direction for.
I did try the code and it works as outlined below:
I followed the steps to create an app in Outlook (Gmail has a similar thing) following the steps in the official website (as a reference) and the steps outlined in this resource Accessing Office 365 with IMAP and OAuth2 • Codewrecks
Either going through Outlook or Gmail, essentially one creates an app, assigns some permissions as needed, and captures 4 things: “TENANT ID”, “CLIENT ID”, “CLIENT SECRET VALUE”, and “REDIRECT URL”
Using above 4 data points (values), now one can call and get an “access_token” to continue the communication thereafter to read, send, … using IMAP or SMPT (as needed), as one would do the old way.
I used the sample code provided on gitbub, /javanile/php-imap-outlook/blob/main/public/index.php (sorry I had to make the URL relative since I couldn’t add more than 2 links) and tested the functionally, and it works.
Given all this info, essentially, if PHPList can be extended to incorporate a view to save the 4 data points as mentioned above and use it to call an inbox using IMAP by incorporating a package such as the one referenced above, then the issue for many who may need this will be resolved.
I hope this can be taken up and be developed.
Lastly, I have not raised a feature request as you have suggested as of yet. I wanted to see what your take might be on the points above as you are a contributor to this project and your insight and the direction you may outline is valuable.
@arbialexandri Thanks for looking into this. I’m not really familiar with this, so am finding it difficult to understand which steps need to be done within phplist and which can be done outside of it.
So the registration is outside phplist but then is getting authentication code and access code something that phplist has to manage?
Usually the bounce processing is done through a cron job, so again it is not clear to me how an automated process has to handle oauth2.
@duncanc, essentially what you have outlined – high level – is the flow. Given my prior comment where I have listed 4 points/steps, here is what I can re-list – from my perspective and understanding — as PHPList or outside of PHPList requirements:
Registering an App and give proper permissions needs to be done within Gmail, Outlook, Yahoo (outside of PHPList). The App is essentially needed for the said providers to give few data points that are needed for oAuth, namely “TENANT ID”, “CLIENT ID”, “CLIENT SECRET VALUE”, and “REDIRECT URL”
PHPList should have a mechanism to save “TENANT ID”, “CLIENT ID”, “CLIENT SECRET VALUE”, and “REDIRECT URL” or configure in the config file and support a new email fetching mechanism besides pop and mbox. For example, it can be called “oauth”.
In PHPList, If oAuth email reading/processing mechanism is configured, then 4 data points “TENANT ID”, “CLIENT ID”, “CLIENT SECRET VALUE”, and “REDIRECT URL” are needed to be present.
Given points 3 and 4, PHPList should have new methods to call and open an inbox similar to the current imap_open method that is being used to open the stream for pop/mbox methods using a class/package such as the one mentioned earlier (see GitHub - javanile/php-imap2: PHP IMAP with OAUTH2 and this example code)
PHPList should have a page that is the “REDIRECT URL” where one needs to use in the App registration process or after. This REDIRECT URL is the page that Gmail, Outlook, … will send the user back upon requesting and logging in their platform to supply a set of data such as ACCESS TOKEN, REFRESH TOKEN, … This part will be a verification/login via UI/Web. Basically, PHPList will need a view/page to allow the user to log into the defined inbox, give the permissions to allow the access, … and upon successful login the ADMIN will be redirected to the REDIRECT URL page which can show a success confirmation or an error, whatever comes from the email provider.
In PHPList, the data from the REDIRECT URL will need to be stored, mainly REFRESH ROKEN as it will have a longer life cycle. On each call to open_imap2 (given above package as an example), the ACCESS TOKEN will be passed to read/process the emails as needed for the life of the ACCESS TOKEN.
Within the PHPList logic, if it is detected that ACCESS TOKEN is expired, it should call the same end point using REFRESH TOKEN which is essentially to request a new ACCESS TOKEN and the result will be a new ACCESS TOKEN where it can be used for the subsequent calls as needed.
Step 7 can essentially repeat itself internally to get new token as needed until REFRESH TOKEN is also expired. In this case, the logic will loop back into step 5 where ADMIN will need to once again interact from the UI/Web to allow access to the email configured and PHPList will restore the refresh token and continue the same as outlined. PHPList can show a warning on the UI that such a token is expired so the ADMIN can interact and allow access once again. This will only be needed depending on the lifespan of the REFRESH TOKEN which is defined and can be changed within the App settings on the platforms listed earlier.
I know above steps may not be the best workflow that one could come up with, but I thought I can outline what is in my mind as a reference for others to also brainstorm and see what can be done.
I am also aware, but not too sure, that the ADMIN interaction may also not be required if an App is registered in a way that eliminates the need of direct interaction. In such a case, the flow may be a bit different, but my info on that is a bit limited, but if I find more info, I will share back here.
Long story short, since Gmail. Outlook, … have already stopped supporting imap/pop protocols using simple username/password logins, anyone who uses PHPList that is configured to process the bounced emails from such inboxes will fail. It is inevitable thing that is happening to many software and the changes are a challenge to integrate with and support. In addition, PHP itself has not taken any action to extend or add support for oAuth in imap as of now although it is sort of a TODO on their end for a while. See PHP: todo:ext:imap:xoauth2.
@arbialexandri Thanks for your explanation. I have been looking into this for a few days and have created a plugin that uses the php-imap2 package with oauth2.
That works for getting an access token and then using that for accessing the mailbox. I have been testing against a microsoft test application with my own outlook.com mailbox. This is really just a prototype to try to see whether it can be made to work. If it can be made to work for processing bounces, and also for sending emails, then it could be incorporated into core phplist.
I’d be pleased if you could try the plugin. It is not yet published but I can do that if you are interested.
@duncanc, that is wonderful. I am grateful that you have looked into this. I would love to check out the plugin. Please do let me know how I can get a copy of it to test and any steps/directions I must take.
There are several things that I am not fully sure about, such as the URLs to authorise and get the token. I have simply used those that are in the sample code. Also, the scopes that you need might differ. If they do, then I can make them another configuration setting.
I also have no knowledge of the php-imap2 package and how closely it matches the php imap extension. There might be differences that affect the results of phplist processing.
Update
I have added integration with phpmailer, so now sending emails using SMTP will use the access token instead of password.
This highlights that there are two email addresses being used, one for sending and one for receiving bounces. Will you be using the same email address for both?
@duncanc, thank you for sharing the plugin info. I validated it and it seems to be functional. I also see that you have introduced changes/updates. I have the following requests/suggestions:
The REDIRECT URL is something that one must set when creating an “App” and must match when calling to API to get a token or refresh, but it would also be good for the redirect URL can be an actual page within PHPList so the user can see what the result/response was. I was thinking, the default REDIRECT URL can be the same for “Authenticate using OAuth2” page. Basically, this page/path can have a dual role and show the response success/fail as needed. I propose this since I am not sure the sample page “/?pi=OAuth2&p=authcallback” exists or how to create one.
I think it would be great to add another setting to specify what folder to process the bounces from. For example, /INBOX or /INBOX/Bounces … and that will be the path for imap to use.
Can SMPT be optional? It is not always the best to use SMTP of Gmal or Outlook due to rate controls they have. Basically, I would like to still send the emails using the local server SMTP mail sever – as it is not in PHPList – and only use oAuth to process the bounces. I hope this can be done.
I am not sure if this is already applied, but it seems like to refresh a token, ADMIN needs to manually interact with the view/page. It would be great if the refresh can be done automatically when the token is known to be expired or when there is a failure response sent so the process that may outlast the life of a token can re-fresh itself and continue.
Question: How would I set the bounce processing as part of a cron/automated call? Currently we have a call that goes to “/?page=processqueue&secret=XXXXXXXX” (the current/old way) and I assume with this new feature we have to call “/lists/admin/?page=processbouncesoauth2&pi=OAuth2&secret=XXXXXXXX” path? Is this correct?
Observation: the image/logo that is used in the button on “Authenticate using OAuth2” page is broken (non-existent).
I have also noticed that the new bounce processing page remains blank on the website.
I would appreciate your feedback on the points above. I must also state that I am absolutely grateful for your work here.
The page /?pi=OAuth2&p=authcallback is a public page within the plugin, you don’t need to do anything, just enter that URL when creating the app.
The problem I had with trying to use the “Authenticate with OAuth2” page was with cookies not being persisted when coming back to phplist from the Microsoft pages. The page is an admin page so you need to remain logged-in.
phplist created its cookie with “samesite strict” which seems make the browser discard the phplist cookie, so losing the state.
This might have been affected by testing on my local machine, so it is something that I can look at that again once this is working.
phplist has hard-coded the mailbox, see file processbounces.php
Ideally that would be a configuration setting for phplist itself. But I can look at the plugin to see whether it can easily override that.
Yes, I have made it optional. There is a field on the Settings page for that.
Yes, I can look at automatic refresh once this is working. I just didn’t want to try to do too much.
You can run the processbounces page in a cron job. It appears that you currently run it as a remote page, so I can change the plugin to support that. The URL should be as you showed.
Do you mean that it is not working? Please look in the php error log, and also look at the html source for the page as sometimes it is not displayed when phplist fails.
Thank you for the insight. Ideally it would be to have a page that shows the response (whatever it might be)
You are correct, I propose either adding a new config setting to define the folder and use it afterwards, or only override for the plugin. To access a specific folder, if I use the PHP imap call example, you can call like this: $link = imap_open('{'.$server.':'.$port.'}INBOX/Bounces', $user, $password);
Or to access any other folder that is not a sub-folder of inbox, you can use the folder name (with spaces and as it is). For example, if the folder name is “Junk Email” (this is a folder in Outlook), you can access it using: $link = imap_open('{'.$server.':'.$port.'}Junk Email', $user, $password);
Perfect
That would be amazing. In reality, without that auto-refresh mechanism, if bounce processing is an automated call from a cron on the server may not work.
That would be great, but of course it will be bound and dependent on point #5 to auto-refresh the access token.
Thank you.
If the token expired, I see this page (I have cutoff the reset of the menus and such)
But if token is valid and I access the page again, the page remains blank while it seems like that it is processing something (the page spinner in the browser keeps on turning).
It seems like, when the entire process is finished, then the page loads and it shows the results as shown below. Maybe the page is not built to be asynchronous to show the progress, but I could be wrong.
@arbialexandri There is a new version of the plugin that should resolve the earlier points. Now you should be redirected back to the phplist admin page when creating an access token. Also, the processbounces page will refresh the token if it has expired. So you should need only to create a token, then leave it alone.
You can upgrade on the Manage Plugins page but see the documentation at https://resources.phplist.com/plugin/oauth2 for a change that needs to be made to the config.php file and new fields on the Settings page.
@duncanc, thank you for your reply and all the changes. I took the latest changes following your instructions and added the changes to config.php, but now when accessing “OAuth2 access token” view, the page is completely blank, and I do not know what the issue might be. The same is when accessing “Process bounces using OAuth2”.
Do you experience the same?
Your previous version was working and when I revert back, everything comes back on. There might be either a logic flaw or I am missing a PHP module that might be needed. I would appreciate your help on this to resolve the matter.
@duncanc, I originally had 3.6.8, after your comment above, I upgraded to 3.6.10 and I made sure both Common and OAuth plugins are up to date by updating them using the links provided for each.
Now, I can access the page /lists/admin/?page=token&pi=OAuth2, but once I click on the button to “Sing in with Microsoft to create a new access token”, the next page/view is blank, and nothing happens.
@arbialexandri Can you use the browser’s developer tools to look at the network traffic? That should show http request is causing the problem.
Also, view the source of the “blank” page because there can be html that is not being displayed.
Check that the redirect url in phplist and in the registration of the app are exactly the same. If not that should be displayed as a clear error message but maybe that is not happening.
Also, the blank page content, although some HTML is there, doesn’t have much of an info that I can make any conclusion. Here is the HTML content I have saved as a text file > blank_page_content.zip (3.7 KB)
Note that if I revert to version 1.1.3 (the one before the latest), everything works within the framework of that version.
@duncanc, local environment that I am running the tests before I deploy is a Windows environment and PHP Version – which I can switch in between – is currently PHP 8, but the same is the case with PHP 7.4
The deployed environment which I will only push the updates when I pass them locally is Ubuntu 16.04 with PHP 7.4
