@ffranco phplist does not synchronise the data in the way that you describe. You will have to manually remove the subscribers that you do not want.
One way is to have a temporary list. First move all subscribers from the main list to the temporary list, then import into the main list. The main list then contains only people who are in the CRM and the temporary list contains everyone who has ever been in the CRM.
Alternatively remove all subscribers from the main list, then after importing, the people who are not in the CRM will not belong to any list.