Plugin / API Questions (+ willing to contribute)

đź’» Development
1

Quick introduction: I implemented phplist into my forum (smf forum) 7 years ago. At that time, there was no API and I had to do some reverse engineering and had to implement quite some code to add / remove a subscriber and handle it’s subscription list.

Fast forward now, I upgraded it to the latest version and ran into quite a few issues and pain points and want to better integrate everything.

The documentation for the API isn’t ideal and leads to dead links as well, so my first question is where can I find the documentation for the endpoints?
I found this github(dot)com/phpList/rest-api/blob/phplist3/docs/Api/RestApi.apib in main branch there is just an empty doc folder.

My goal is actually simple: Add / remove subscriber (by email), modify it’s subscribed list, modify an attribute (group association).

By the looks of it, in those 7 years this is all not possible yet? In main I get a few more endpoints but they are all marked with “Status: Beta – This method is under development. Avoid using in production.” :cry:

Then I have a few special use cases which I originally had to modify in phplist code directly, but upgrading it every time is just such a waste of time:

  1. Use an html list (ul and li tags) instead of the “* %listname%\n” text.
    I couldn’t find any way to modify the [LISTS] placeholder

My attempt today to use a custom plugin failed, because I can’t seem to get any entry point for the system mails:

parseFinalMessage -> nope
parseOutgoingTextMessage -> nope
parseOutgoingHTMLMessage -> nope

Is there another way to catch the text before it will be sent out?

  1. I introduced a [FROMEMAIL] placeholder, which replaces the text with the formmail.
    This works fine for campaign emails with the above hooks (parseOutgoingXYZ) but again, doesn’t work for system emails

  2. When converting the HTML to text version (HTML2Text) I would like to remove some html blocks or do custom parsing. For example the “ul li” list won’t be parsed correctly and it’s just one line instead of a nice list :frowning:
    My other use case ist to remove html header formatting:
    I use MJML to generate nice HTML emails and they also do have a header and a preview block.
    But the conversion to text is “not nice” and I would like to remove the head html and also change the preview text.

I also had to modify the source code here to make it fit.

Time is precocious, but I’m willing to contribute some time and create PRs to improve the situation if there is interest and if these PRs will make it quickly into a new release if the provided PR is solid.

My suggestion would be:

  • Add new hooks for system messages (like parseOutgoingSystemTextMessage & parseOutgoingSystemHtmlMessage) or rewrite the code so that system also uses the existing ones (but this could lead to unwanted effects with previous code / hooks)
  • Or maybe add a new hook for ALL outgoing texts / before it gets sent?
  • My quick testing revealed that parsePlaceHolders is called by both and this could also be a great hook point to add custom placeholders
  • add a new hook to customize the [LIST] placeholder - or introduce a new function which all placeholders use and add there a hook so that we can modify all placeholders
  • add a pre and post hook for HTML2Text function

I know many ideas, some are quicker to implement and some require more discussion.
Feedback welcome, would love to contribute something to make it official and to avoid a custom solution where no body will benefit from and is harder for me to maintain.

2

@Delta You may get more joy by using the REST API plugin plugin:restapi [phpList Resources] instead of the phplist 4 REST API. You should be able to add and delete subscribers and add and remove them from a list.

The plugin method messageHeaders() gives access to the mailer object and so to the message content. You might be able to use that to modify the content of system messages in the way that you want.

That method is invoked at phplist3/public_html/lists/admin/class.phplistmailer.php at 998fde417f3f14bb1f251084d37cff7308dddf03 · phpList/phplist3 · GitHub

1 Like
3

@duncanc Thanks for the hint.
I haven’t looked at the REST API yet, thats on my todo list shortly.
A quick look at the example looks more promising, at least I can add/remove users there from a list, user attributes doesn’t seem to be supported yet?
Just a bit confused, I thought the latest phplist versions come with the REST API shipped already (phplist(dot)org/manual/books/phplist-manual/page/api-and-integrations).
But it looks the plugin you linked has more features - got it. :+1:

messageHeaders() doesn’t seem to work well, because also this just catches campaign emails not not transaction / system emails.

Spent now the last two days to go through everything and also do some fixes on my email generation and I think I have now a great system to generate top notch HTML emails, which have good support with phplist3 and auto text version through phplist works too.

It was quite a chaotic journey through the code (no offense, I’m so glad that this project exists :sparkling_heart:) and I found a couple of issues (besides UI/UX issues) and created now a fork with some changes.

To make it easier to follow me, here is the source code:

  • Plugin: github(dot)com/UGX-Mods/phplist-plugin-ugxmods/blob/main/plugins/ugxmods.php
  • phplist3 changes: github(dot)com/phpList/phplist3/compare/main…UGX-Mods:phplist3:feature/ugx-mods?expand=1

So lets break the changes down:

  • I created a function to unify how text lists are generated, because they sometimes differ
  • Added new hooks:
    • parseOutgoingSystemTextMessage & parseOutgoingSystemHTMLMessage
    • identical to parseOutgoingTextMessage & parseOutgoingHTMLMessage, just for system ones
    • processHtml2Text
      • used to hook into Html2Text at the beginning (pre) and at the end (post, before word wrap)
      • the reason for this is to do some preparations at the top (one could for example use the DOM parser and modify valid HTML, etc.) or do some post processing when the default phplist logic is done
    • parsePlaceHolders
      • this is actually unused and was just (one of many) ideas, to allow plugins to process placeholder data, etc.
  • then I noticed that replaceChars called from HTML2Text doesn’t convert the copyright sign correctly for text. It turns into some odd character. Is / was there a reason to do these specific characters? PHP’s html_entity_decode does handle this quite well, including taking care of encoding (UTF-8)
  • There is no option for me atm to add custom meta data to the campaign, but I need one for a subtilte in my HTML emails. I added a new entry in send_core.php and to my surprise this works quite well → a custom hook would allow here to add new fields and extend the meta
  • I noticed that the forwarded footer is not ran through HTML2Text, which is a bummer because I use everyhere HTML and the normal footer works well with HTML. I had to add this fix as well to make it work.

So besides these fixes / improvements, I noticed a few other things which were quite annoying:

  • I tried to update for my testing Delta@test.local email the preferences on the dedicated page
  • On save, I simply get the error “Please enter your email address” in a tiny font above other red text and it took me a while to realize that the validation fails - would be good idea to have a better error message. Surprisingly this is way better handled on the forward page, where it says clearly states “Delta@test.local is not a valid email address”
  • The footer parts and transaction emails are stored in the settings. Unfortunatly the table field doesn’t support emojis. This was fixed for campaign emails, but the settings don’t have utf8mb4 - I converted it myself and so far nothing broke - is there a reason against using utf8mb4 there?
  • Oh and the [EMAIL] placeholder is not replaced for the forward footer part (had to do this in my plugin as well)

My contribution question would be now, is there an interest to fix / improve this in core?
I’m open to write separate PRs, etc. and would also have a look at tests.

Going to mess around now with the API (I really thought I could be done by now already, hehe)

4

New day, new update.

The shipped API doesn’t work with php 8.2, it throws deprecations and errors.
composer.json for base also has lots of outdated packages and is set to php7 :frowning:

The linked plugin has some confusing readme statement:

"#phpList REST API v1 REST API as a plugin to phpList

This plugin has been depreciated by the phpList REST API v2 (a second plugin within this repository)."

but there is no second plugin - I assume v2 is the one which is present.
That plugin works, but it took a bit for me.
Very thankful that this exists, but oh boy does it break standards :cry:

  • Login with wrong credentials return a HTTP 200 with HTML :confused:
  • subscriberGetByEmail returns always HTTP 200, also for those who don’t exist (need to check if data is empty).

I also compared with my integration (which is basically the same idea to interact on database level - I was really hoping the plugin would interact with the phplist functions) and it seems that userstats is completely missing in the API. So subscribe, unsubscribe and total user count is not updated properly. Also the history is not updated (at least there is an API endpoint for this).

Also the endpoint listsSubscriber doesn’t return the correct entered and modified of the subscriber and accidentally uses the creation date (unfortunately also called entered) and modified date of the list instead.

So looks like I will have to use a hybrid of my solution in combination with the API, not a big fan of it but I spent time on this already and don’t want to throw it away, lol

I also would consider creating PRs here and improve the situation, but seeing PRs and issues open for years, I feel like I waste my time on this mission.
It looks abandoned which is really sad.

5

So I think I finally have (almost) the same functionality back as with my forum integration.
I ditched the user attributes atm, will do this when I have some time - didn’t want to spent almost a whole week on this to be honest, but once you get started, …

These are my changes github(dot)com/phpList/phplist-plugin-restapi/compare/master…UGX-Mods:phplist-plugin-restapi:feature/ugx-mods (why is github domain blocked?)

REST API is okayish if you ignore the fact that it’s not truly restful but it gets the job done with my changes now (especially the email sending was important to me).

As usual, feedback welcome and I’m still open to contribute and get this into stable somehow.

Sidenote: I couldn’t get the tests to work, spent an hour but gave up due to time limits. I had a docker container already up and running with the tests, but wasn’t perfect yet. But converting to php8.2 was actually quicker than I thought^^