I am not sure if this is the best way to handle this, but adding a content security policy header to my Apache configuration seems to have done the trick.
In my case, this is in a <VirtualHost> configuration, but it could also be in an .htaccess file.
Header set Content-Security-Policy "upgrade-insecure-requests; default-src https:"
This server header tells Chrome and other compliant browsers to fetch all http referenced content via https, which allows the stylesheets and other files to load over https.