phpList 3.3.7 Release Candidate ready for testing

[Suela]

A Release Candidate for phpList 3.3.7 is available for testing

This release includes a new Automatic Updater module that provides an alternative way of upgrading phpList from a Web browser. Eight fixes to other functionality is included, and one new feature for plugin developers.

Automatic updater

This is probably the most requested phpList feature of all time. It can be accessed via a new entry in the main menu.

The variety of servers, control panels, permissions, and software versions used with phpList means that your feedback is required on how the updater performs with your system.

How to test

To test the Automatic Updater follow the usual update procedure and click the new "Update" menu item. A new update will be available: this is a dummy update with an identical codebase and different version number, provided solely to make it easy for you to test upgrading.

The Automatic Updater can Upgrade from this Release Candidate to the final version of phpList 3.3.7, due next week, providing an additional upgrade opportunity. If additional RC versions are released, you can use the Automatic Updater to upgrade to those too.

Please test extensively in order to be able and identify as many potential fixes and improvements as possible for phpList 3.3.8, due in January.

The Database upgrade mechanism is unaffected by the new Automatic Updater and continues to work separately as in the previous versions. Your config file (in lists/config/config.php) won’t be overwritten; nevertheless: please backup your data to a safe location.

For more information on how the updater works please see the documentation in the updater repository, including a full list of features.

Fixes

Thank you to @duncanc who has eight pull requests included in this release, including:

• Support for importing subscriber attributes (import2.php) which include field separator characters, such as tab or comma, when the values are stored within quotes

• Fix for URLs which are stored incorrectly (truncated) with click tracking enabled

• Move the options provided by plugins on the ‘Edit subscribe page’ page to the end of the form

Check the roadmap for the full list of changes.

Download the Release Candidate here

Guidelines

• Upgrade from older releases following the usual upgrade process

• Use your phpList as normal, and report any new problems you find

Reporting issues

• Report any issues you find to the phpList Bugtracker, selecting “3.3.7-RC1” as the Product Version

• Use the usual bug fixing process if you know how to fix it.

Happy testing!

[duncanc]

@Suela

For more information on how the updater works please see the documentation in the updater repository

I had a look at the README but it is a bit unclear

• the code seems to assume that config.php is in the lists/config directory. Using the ConfigFile environment variable the config file can be anywhere.
• what happens to files and directories beneath /lists that are not part of phplist?
• what happens to additional plugins that have been installed in the plugins directory?

[samtuke]

@duncanc Thanks for your quick feedback.

• If the config file is outside of the phpList directories then it cannot be overwritten, so should be fine. But if the updater expects it to be there, it may fail. That should be tested.
• what kind of other files might be in the phpList directories? Yes it would be good to check for other files and warn the user during the upgrade to back them up separately, or perhaps give the option to include them in the automatic backup
• I think that these need to be installed again, but @xheni can confirm

[duncanc]

@samtuke

what kind of other files might be in the phpList directories

There are several files and directories that, while not part of the released phplist, are used by it. Some examples

• the image upload directory
• the attachment directory
• a custom ckeditor configuration file
• a custom build of ckeditor

These might be placed within the phplist directory in order “to keep everything together”.
Further, if phplist is installed in the web root directory, then any other files and directories will have to be in the phplist directory.

[xheni]

@duncanc

the code seems to assume that config.php is in the lists/config directory. Using the ConfigFile environment variable the config file can be anywhere.

Yes, we expect the config folder to be there and the config file is not overwritten during the update. The config folder needs to be writable because during the update we create an action.txt file inside it, needed to save the current step of the updater.

As long as your config file is outside lists directory, it should be fine.

There is actually one place (for the pdo connection) where the default config file location was required and I created a PR to improve that.

@duncanc

what happens to files and directories beneath /lists that are not part of phplist?

Every release has a certain directory structure, and we check if the expected files and directories are there. Your custom scripts and builds inside /lists would be overwritten. For that reason, we offer a back-up option of your code.

@duncanc

what happens to additional plugins that have been installed in the plugins directory?

With the current code, you might need to reinstall them again. It was actually a doubt because we have added some plugins by default in the past and maybe we want to add other ones in the future too. Currently, it is the same as it would be with a manual upgrade.

[Suela]

Hi everyone,

Following the valuable feedback from @duncanc and more testing during these days, a new RC for version 3.3.7 will be released tomorrow before the final release later this week.

If you have been testing please let us know how that worked for you.

[duncanc]

@Suela I did some more testing of the new updated yesterday and have a few more comments

• there needs to be more explanation that this is updating the phplist code, and when it is successful that the phplist database then needs to be updated

• the input field for the backup file should be wider to allow a longer path to be visible
  

  

• I think that the updater doesn’t validate that the entered backup file has a .zip extension

• I explained why there might be files and directories in the ‘lists’ directory. When I tested that I got a message
  
  and the updater process appeared to have stopped but that was not clear.
  I suggest that the updater should simply ignore any unrecognised files and directories in the top level. Also, it can ignore that an expected file is not present.

• clearer guidance on what the admin should do when the update process fails. I had to manually remove actions.txt to be able to try it again.

• twice I tried the backup step and both times the process appeared to stop in some way.
  After entering the backup file and clicking Next, the Next button was disabled but nothing then happened
  

  

• I suggest putting the Updater within the System menu, not an item on its own. The menu caption and that for the database upgrade would then need to made clearer.

• currently it is not necessary to have the php curl or zip functions enabled. Elsewhere in phplist the code checks whether they are available before trying to use them, and I think that the same should be done here.

[xheni]

@duncanc thanks a lot for testing and the feedback.

@duncanc

the input field for the backup file should be wider to allow a longer path to be visible

I have created a PR to increase the input field size for visibility.

Validate input for zip extension, increase input size for visibility

@duncanc

I think that the updater doesn’t validate that the entered backup file has a .zip extension

That’s right. It’s one of the issues that is open on GitHub project and I fixed that in the PR too.

@duncanc

• I explained why there might be files and directories in the ‘lists’ directory. When I tested that I got a message
  
  and the updater process appeared to have stopped but that was not clear.
  I suggest that the updater should simply ignore any unrecognised files and directories in the top level. Also, it can ignore that an expected file is not present.

That’s on purpose. I forgot to mention in my other reply that the updater stops when it finds unexpected files and tells the user what files.
That’s because the changes of a user could be overridden, things break, etc.
But what I would suggest for this case, would be maybe asking a verification from the user if they are ok to proceed.

@duncanc

twice I tried the backup step and both times the process appeared to stop in some way.
After entering the backup file and clicking Next, the Next button was disabled but nothing then happened

That has been identified and addressed in this PR:

Better error message for back up and retry option if it fails

@duncanc

currently it is not necessary to have the php curl or zip functions enabled. Elsewhere in phplist the code checks whether they are available before trying to use them, and I think that the same should be done here.

I agree. I think there is an open issue for that too.

@duncanc

I suggest putting the Updater within the System menu, not an item on its own. The menu caption and that for the database upgrade would then need to made clearer.

@Suela thoughts?

[duncanc]

@xheni Thanks for your response.

I should also have said that for the simple cases of phplist installations, which is probably the majority, the upgrade should work without any problems. This looks to be a very useful addition to phplist.

[Suela]

A new RC is now available for Download and that should include the following:

1. Increased HTML input element size for specifying backup file location, to allow longer paths to be visible
2. Validation of the submitted backup file path to check for a .zip extension
3. A clearer error message is displayed when the backup and retry option fails
4. Added check for ConfigFile environment variable required for pdo connection

Here are some points for clarification which will shortly be added to the Updater README.md documentation:

1. The updater stops when it finds unexpected files (not from phpList default installation) and lists them
2. Any plugins that are not included in releases are removed and need to reinstalled following update (settings for those plugins in the database are not affected; reinstalling the plugins should make them work as before)
3. It is possible to override the backup checks by reloading the page when the backup check fails. Do not reload the page unless you wish to proceed without a backup in this case
4. When the update process fails you should manually remove actions.txt file inside the config folder in order to reset the process and be able to try again

Reporting issues

• Report any issues you find to the phpList Bugtracker, selecting “Automatic Updater” as the project and “3.3.7-RC2” as the Product Version