Automatic import of csv file with curl

ℹ️ Support
1

Hello!

I want to automatically import users via a csv-file.

I found this curl command on the forum for the simple import:

> curl -v -F login=admin -F password='xxxx' -F "importcontent=<email.csv" -F "importlists[0]=2" -F "checkvalidity=1" -F "doimport=Import emails" 'http://www.xxx.com/lists/admin/?page=importsimple&list=2' -o curlout.html

It works very well. However, is it also possible to import a csv file with curl?

It seems like the upload of a csv file (import2) consists of several steps, so is it even possible with curl?

I’ve tried variations of

curl -v -F login=admin -F password='xxxx' -F import_file=@sampleusers.csv -F lists[0]=1 -F notify=no -F htmlemail=1 -F import=import 'http://localhost/lists/admin/?page=import2'

but no luck so far.

I use PHP 5.5.30 (cli), Phplist 3.2.4 and Google Chrome v48.

1 Like
2

Hi Lisa,

It’s a bit more complicated than one command. I believe you need to get to the import page, record the session id and token, and then use those with your curl command. This can be done via a script file.

Dan

3

@lisa @danwaterloo A few pages need to be run sequentially - login, the import page, then submit the import page.
This shell script seems to work for me importing the file import.csv into list number 9 using comma as the delimiter.

curl -v --cookie-jar cookie.txt  -F login=xxx -F password=yyy -F process=continue 'http://strontian/lists/admin/?page=home' >import1.log
TK=$(grep -o 'tk=[[:alnum:]]*' import1.log | head -1)
echo $TK
curl -v --cookie-jar cookie.txt -b cookie.txt "http://strontian/lists/admin/?page=import2&$TK" > import2.log
curl -v --cookie-jar cookie.txt -b cookie.txt -F formtoken=`awk '/formtoken/ {match($0,/name="formtoken" value="(.{32})"/,arr); print arr[1]}' <import2.log` -F lists[9]=9 -F import_file=@import.csv -F import_field_delimiter=',' -F import=import "http://strontian/lists/admin/?page=import2&$TK" > import3.log
curl -v --cookie-jar cookie.txt -b cookie.txt "http://strontian/lists/admin/?page=pageaction&action=import2&$TK" > import4.log
1 Like
4

@danwaterloo @duncanc You’re awesome! Thank you both for your answers!

In the meantime I started updating the database directly but this shell script works perfectly and is the better option I think. I added -F overwrite=‘yes’ to update existing entries.

curl -v --cookie-jar cookie.txt  -F login=xxx -F password=yyy -F process=continue 'http://localhost/lists/admin/?page=home' >import1.log
TK=$(grep -o 'tk=[[:alnum:]]*' import1.log | head -1)
echo $TK
curl -v --cookie-jar cookie.txt -b cookie.txt "http://localhost/lists/admin/?page=import2&$TK" > import2.log
curl -v --cookie-jar cookie.txt -b cookie.txt -F formtoken=`awk '/formtoken/ {match($0,/name="formtoken" value="(.{32})"/,arr); print arr[1]}' <import2.log` -F lists[9]=9 -F import_file=@import.csv -F import_field_delimiter=',' -F overwrite='yes' -F import=import "http://localhost/lists/admin/?page=import2&$TK" > import3.log
curl -v --cookie-jar cookie.txt -b cookie.txt "http://localhost/lists/admin/?page=pageaction&action=import2&$TK" > import4.log
6

Hm, locally everything works.

On the server, however, I get an invalid security token. I can login successfully via curl but in the 3rd curl command I get an error with awk. It fails to create a formpost data. In the shell it says

scripts# curl -v --cookie-jar cookie.txt -b cookie.txt -F formtoken=`awk '/formtoken/ {match($0,/name="formtoken" value="(.{32})"/,arr); print arr[1]}' <import2.log` -F lists[2]=2 -F import_file=@11_development.csv -F import_field_delimiter=',' -F overwrite='yes' -F import=import "http://mydomain.com/lists/admin/?page=import2&$TK" > import3.log
    awk: line 1: syntax error at or near ,
    * About to connect() to mydomain.com port 80 (#0)
    *   Trying 127.0.0.1...   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                     Dload  Upload   Total   Spent    Left  Speed
      0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0connected
    * Connected to mydomain..com (127.0.0.1) port 80 (#0)
    * failed creating formpost data
      0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Connection #0 to host mydomain.com left intact
    
    curl: (26) failed creating formpost data
    * Closing connection #0

Any ideas why the awk command is wrong on the server?

7

@lisa I had the same problem and seems to be caused by the cron job using a limited shell. Try placing this before the third curl line to use perl to extract the formtoken value

TOKEN=$(perl -ne 'print "$1" if /name="formtoken" value="(.{32})"/' import2.log)
echo $TOKEN

and then replace the awk command in the third curl command with

-F formtoken="$TOKEN"
1 Like
8

@duncanc Thanks for your answer! You’re right, it works with perl.

This varied awk command works as well (see stackoverflow answer)

awk ‘sub(/.name=“formtoken” value=“/,”“){ sub(/”./,“”); print}’ import2.log

9

Hi

I have managed to get Lisa’s first script to work from a dos bat file - and it works fine.

Does anyone know how to extend this curl script to clear down the subscriber list first prior to importing the new records?

Would the same approach work for the second case where the csv with attributes is being imported?

Update: I assume the screen to process would be something like
…/admin/?page=members&id=1

Regards,
Geoff

10

For anyone who is interested, I have found the solution for this…

Using Chrome, and using the Developer Tools, ie. under Customise > More tools > …or use ctrl & shift & I
Go through the step that you need to do to clear a list. Having done that

By right clicking on the ?page > Copy > Copy as cUrl (cmd) you can paste the command to notepad.
PS - The same approach can be used for any web page!

With this, and Duncan’s script details given earlier, we can derive the following…

curl -v --cookie-jar cookie.txt -F login=xxx -F password=xxx -F process=continue ‘http://…/admin/?page=home’ >import1.log
TK=$(grep -o ‘tk=[[:alnum:]]*’ import1.log | head -1)
echo $TK >tk.log

curl -v --cookie-jar cookie.txt -b cookie.txt “http://…/admin/?page=members&id=1&$TK” > import2.log
TOKEN=$(perl -ne ‘print “$1” if /name=“formtoken” value=“(.{32})”/’ import2.log)
echo $TOKEN >token.log

curl -v --cookie-jar cookie.txt -b cookie.txt -F formtoken=“$TOKEN” -F ovedestination=3 -F copydestination=3 -F tagaction=nothing -F tagaction_all=delete -F movedestination_all=3 -F copydestination_all=3 -F processtags=“do it” ‘http://…/admin/?page=members&id=1’ >import3.log

…which works perfectly as I needed to clear all member subscribers on list 1

The list can then can be re-populated as required with the cUrl script given before.