Hyperlink Your Heart - indieweb

Image Optimisation

2021-05-30T00:27:00+02:00

In the last instalment of my epic blogging saga I recounted my discovery that the index page of this site had grown to over 1.7MB of content when loaded fresh, largely due to the images. One of my goals for this site was that it be lean - fast to load and energy efficient - and it was not meeting that goal at all. Clearly avoiding Javascript and CSS frameworks was not enough!

I immediately started thinking about how to improve the situation. Though I had previously ruled out the approach used by Low-tech magazine’s solar-powered website because I didn’t think it would fit my aesthetic, I decided to see if dithering coloured, rather than monochrome, PNGs would work better for me, and improve on the size and quality of an appropriately-sized 75% quality JPEG.

PNG Thunderdome

The one to beat - baseline 75% JPEG, Size: 16.1kb

Using Pillow and the same hitherdither library that Low-tech magazine used for their site, I iterated on a script that output hundreds of compressed variations of a given image using different dithering algorithms and parameters.

The best results I found were with the Bayesian algorithm with a 32 colour palette, a 2x2 matrix, and an image size half the expected display size. This produced a result that was relatively readable, and reminiscent of pixel art. The size savings varied by image - sometimes up to 10kB, but often only 2-3kB as for this example. These savings are modest compared to the loss in detail, and I think this approach could only be considered because of the unique aesthetic it produces.

Palette: 32, Dither: bayer, Threshold: 256/8-256-256/8, Order: 2, Size: 13.9kb

The three “threshold” parameters expected by hitherdither were a bit of a mystery to me. Through trial and error I found that some produced much smaller images, but unfortunately not at a level of quality that I found acceptable. Lower palette sizes also resulted in savings, but below 32 colours they started to look too abstract and unreadable to me.

Palette: 32, Dither: bayer, Threshold: 256/2-256-256/2, Order: 2, Size: 9.6kb

A Challenger Appears

I was about ready to commit to this approach and start converting all the images when my wife reminded me that WEBP exists! After converting a few of my test images to WEBP it was clear that it had my dithered PNGs beat - half the size of the JPEG without any loss of quality.

80% quality WEBP, Size: 9.9kb

Apparently support for WEBP is pretty good these days, but there are a couple of annoying outliers - Safari only supports it on Big Sur, and IE11 still exists, as I’m sure it always will.

As such, I decided I should probably try and fallback gracefully to a JPEG or PNG where WEBP isn’t supported. This can be achieved using <picture> and <source> elements to allow the browser to choose the format it likes best.

<picture>
    <source type="image/webp" srcset="{optimal image url}"/>
    <source type="image/jpeg" srcset="{compatible image url}"/>
    <img src="{compatible image url}"/>
</picture>

Let’s Automate

The above HTML snippet presents a problem - my posts are not written in HTML but in Markdown, and processed by Pelican into HTML, and that process just results in an <img> tag by default.

I threw together a quick Pelican plugin to post-process the generated HTML and replace any <img> tags with <picture> tags, if the referenced images could be replaced with WEBPs. It also processes the referenced images to create scaled JPEG/PNG versions as well as the WEBP version, so I don’t have to do any of that manually either.

Results

The index of the blog is now 778kB at the time of writing - so reduced by over half! I did also replace a particularly large and troublesome GIF with a static image as well, and converted some PNGs to JPEGs. This results in greater savings because the plugin converts PNGs to lossless WEBPs and JPEGs to lossy ones.

The plugin is actually not even working fully yet - for some reason it is missing some <img> tags on the index pages, leaving them serving their original unprocessed files.

I also haven’t done anything about the pixel art images, which if served at their original resolution could be a significant saving.

So in short, huge progress, but still much scope for improvement!

I am almost sorry I’m not going to end up using these dithered PNGs though…

Update 01/06/2021

I fixed the problem with the plugin and all images are now optimised except the few in this post that are flagged to be skipped. I also swapped out all the pixel art images and gifs for 1x resolution versions. These are just being scaled up by CSS, with reasonable results.

The index page is now 539kB, the whole site is just over 1MB, and it is in the 80th percentile for energy usage according to websitecarbon.com (whatever that’s worth). I think the link above showed 75th or 76th percentile or thereabouts at the time of posting, but it will show 80th now.

Energy Usage Update

2021-05-27T13:20:00+02:00

When I started this blog, one of my goals was for it to be lean and efficient, both to improve the reader’s experience and to reduce the power usage involved in serving it.

I’ve been planning recently to write a post on a related topic - the energy efficiency of the Python language and how using it as the backend or API of a more dynamic website would fit with those same goals. In preparation for that, I thought to check in on this blog and whether it is still as lean as it was back when I was figuring out how to embed the SVG icons in it.

Unfortunately, it seems that it is not. Unlike one of my inspirations, low-tech magazine’s solar powered website, I never found a solution for reducing the size of image files. For my posts about TV shows and movies I use fairly low quality JPEGs. But for my art and game dev posts I have been unwilling to compromise, and, frankly, a bit lazy, and have just been throwing large PNGs and GIFs into my posts without thinking about their size.

As a result, loading the index of this site now transfers 1.7MB if nothing is already cached. The text content and CSS are in the tens of kilobytes, with the rest being images. This is comparable to the initial load of many corporate news sites and large blogs with all their tracking javascript and ads blocked - an improvement over the norm perhaps, since I’m not doing any of the tracking or advertising that have become such a burden on the web, but not particularly notable. This site is now only in the 57th percentile according to websitecarbon.com.

For pixel art, one solution does present itself immediately. For my portfolio site, all of the artwork is sent to the browser at 1x resolution and resized there by an integer factor by a small amount of javascript, and perhaps the same thing could be achieved with css. The index of that site is under 400kB despite including all of my artwork, and does not require specially resized image thumbnails or anything like that. I could do the same thing here instead of throwing manually resized art into posts.

I’m less sure what to do about screenshots and screencaps. The solution used by low-tech magazine doesn’t really work for me, I think it compromises the readability of the images too much and doesn’t fit this site’s aesthetic. I tried a number of different techniques for vectorising, posterising and dithering screencaps from TV shows and movies to reduce the number of colours, but they didn’t result in significant savings. On the other hand, they did actually look kind of cool and stylised!

Something I should probably try is scaling such images down significantly and then back up to the required display size in CSS. Obviously this will degrade the quality, but it might be an acceptable trade-off.

Screenshots of things like a text-editor or IDE are even more problematic - if the quality is degraded there they are probably not useful. I will probably just have to try to keep that kind of thing to a minimum, and use tighter crops where necessary.

It’s easy to become complacent about efficiency when computers are so powerful that a few tens or hundreds of kilobytes don’t seem to cause them any strain, but when you have that mindset they can quickly add up to quite a significant waste of time and power. I think this is well demonstrated by USA Today’s pissy response to the GDPR, where instead of the usual 5.2MB bloated garbage site, readers from the EU were served a lean, content focused site that loaded almost instantly and didn’t track them.

Nothing I do is going to have the kind of impact that a site like USA Today improving their efficiency would have, purely because of the scale of the traffic involved, but I still want to try to do better.

Remember Blogs?

2020-01-02T20:05:00+01:00

I’ve read a lot of articles recently (here’s one) lamenting the state of the web. Once distributed, egalitarian, ungovernable, and fast, now centralised, intentionally manipulative, and bloated both technically and conceptually. Even when you manage to fight your way through the popups demanding your attention or personal information, often what is underneath is not worth the effort - more likely a vehicle for advertising than for insight.

It’s also incredibly power-hungry. It’s hard to tie down an exact figure for exactly how power-hungry, but the internet as a whole could account for up to 10% of global energy use. A good chunk of that is streaming video and music, which is a topic for another day, but of the power consumed in serving the web, some of it is related to actual valuable content that people want to see, and some of it is related to the trends described above. The latter is waste. At least bloated JavaScript and CSS frameworks can be cached, but advertising has to be constantly served anew.

So, anyway, all this to say… I’ve decided to start a blog.

The Tech

My technical goals for this website are for it to be…

Lightweight & fast to load - I set up a WordPress site recently, on the best hosting I can afford. It is not lightweight or fast to load.
Content focused - Read one thing or read them all, but I’m sure you can only read one article at a time.
Nice to look at - Apparently it doesn’t take much. Also going for consistent branding between all my sites and profiles.
Responsive - Readable on phones as well as desktops!
Easy to deploy - I don’t have time to configure and maintain a teetering stack of back-end technology, and if I have to move to different hosting at some point, I want it to be a simple task.
Easy to update - If writing posts is a chore, I won’t ever do it.
Hackable - Created using technologies that I’m somewhat familiar with, so that it is feasible for me to modify or extend if I want/need to.

I decided almost immediately that a statically-generated site was going to be the best way to achieve most of those goals. I’m a big fan of Python, so although hackability could be achieved by a JavaScript or C# based generator, I checked out the Python ones first, and found plenty of viable options. I settled on Pelican because it’s…

Popular - It seems to be one of the more popular Python generators.
Blog-oriented - Some generators are geared towards documentation or are intended as replacements for content management systems, but that’s not what I’m doing.
Supports Markdown - I’m sure reST is fine, but I already have to use Markdown elsewhere so I’d rather stick with that.
Easy to update - Just create a new Markdown file and run a command to rebuild.
Extensible - It includes a plugin system to modify the output.

I also decided to hand-craft my own theme, and to avoid a CSS framework. I love the look of Bootstrap, and how quick it is to get started with, but it’s over 200kb and a lot of that is undoubtedly unnecessary for my needs. The spirit of the exercise is bare-bones and DIY!

The Theme

The first step in hand-crafting a theme was… to find an existing theme to copy! Atilla was the closest to the style I was after, so I took a copy of that and gutted it of CSS and JavaScript and other elements that didn’t meet my needs. Then I started building the CSS back up while trying to keep it as minimal as possible. It may not implement every feature supported by Pelican, but you can find it on my Github if it seems like something you could adapt for your own needs.

One departure that I made from the standard Pelican configuration was to have the social media links be taken from a collection of tuples with three elements, so that I could specify both an icon and a title to use.

# Custom social list that includes icons
SOCIAL_ICONS = (('Twitter', 'twitter.svg', 'https://twitter.com/http_your_heart'),
                ('Mastodon', 'mastodon.svg', 'https://mastodon.art/@hyperlinkyourheart'),
                ('Instagram', 'instagram.svg', 'https://www.instagram.com/hyperlinkyourheart/'),
                ('YouTube', 'youtube.svg', 'https://www.youtube.com/channel/UCc_O9Hp5UfQ-IHswi1H54Zg'),
                ('Twitch', 'twitch.svg', 'https://www.twitch.tv/hyperlinkyourheart'),
                ('Itch', 'itchio.svg', 'https://hyperlinkyourheart.itch.io/'),
                ('GitHub', 'github.svg', 'https://github.com/khoulihan'),
                ('Atom Feed', 'rss.svg', '/feeds/all.atom.xml'),)

I like that I can just throw custom configuration into the config file and then make use of it in the templates. However, it probably makes the theme less generally useful.

As it stands currently, loading this post requires less than 30kb to be transferred.

Plugins

Currently, the only plugin I’m using is the css-html-js-minify plugin that is available in the pelican-plugins repository. I haven’t found anything I need to write my own plugin to handle yet, but I’m sure I will get to it.

One problem that needs solving is that the SVG icons are a big nuisance, because it doesn’t seem to be possible to change their colour without using the CSS filter property, which is not nearly as convenient as just setting the colour directly. In order to do that, using the fill property, I would have to embed the SVGs, or reference them as symbols in a <use> tag within an <svg> tag. The individual icon files (from FontAwesome) aren’t set up like that, and I didn’t want to use their spritesheet because it is rather large.

What I might do in the future is write a plugin to compile the individual files into a single spritesheet of symbols, then find and replace any references to them with appropriate <svg> tags. Essentially this will be doing the job that the FontAwesome toolkit usually does in the browser.

The Content

Uuuh… I’ll get back to you on that. Things I like, things I do, that sort of thing.

Feedback

There’s a couple of different strategies for allowing comments on a static site - I’m not going to attempt them for now, and perhaps never will! If you have any feedback or thoughts there are many ways to reach me, such as Mastodon or Twitter, and I think that’s just fine.