Launch of MakeStaticSite.sh Website

November 2022: A website dedicated to MakeStaticSite is now available at https://makestaticsite.sh/. It provides the latest version of the software, released under AGPL, along with various documentation to help getting started.

Motivation

The process of developing this website has evolved considerably. On setting it up a few years ago in the conventional manner using WordPress, I duly familiarised myself with routines for maintaining security and improving performance on a hosted platform. However, there were several aspects that weren’t ideal: the live content was hosted remotely, albeit with backups; updating the site in situ, particularly for major redesigns, was inconvenient; and performance wasn’t always as good as expected. Observing that most of the content was not dynamic in nature, I resolved to develop the site locally and then, as a separate process, deliver it as a largely static site.

The rationale for generating static sites has been around a long-time; I first tackled this more than 20 years ago, when I needed to generate a CD ROM from a CGI/Perl multimedia website. Nowadays, the field is mature; a Netlify blog post (Netlify is a cloud-based service where many static outputs end up) gives many valid reasons, rooted in the need for more responsive work processes. Typically underpinned by a simple language called markdown, there has been a surge in solutions; hence, options for Jamstack abound. However, the processes tend to be heavily oriented to developers at the expense of content authors, issues I know first-hand from enabling web access to museum collections.

I wanted instead to support existing content management processes, i.e. to continue with creating and managing content in WordPress, and then ‘at the press of a button’ generate and publish a static copy online. There are already some solutions, including dedicated plugins such as Simply Static. Tim Nash, having observed increasingly complexity with the Jamstack options, has described how he used WP2Static via WP-CLI, the WordPress command-line interface, for a (relatively) simple and compact solution. But I hesitate over the use of a WordPress plugin to generate the static site. Apart from the ongoing issue of maintenance and compatibility with WP Core, plugins are not really necessary since a public-facing website should be de facto crawlable by a web spider and not require internal system information.

And so I turned to general-purpose tools I could run from the command line. Enter Wget, a ubiquitous utility for retrieving and packaging any number of web pages for subsequent viewing. I needed to configure it, add a few additional inputs, and then choose where to store the outputs. This led naturally to delving into (Bash) shell scripting to bring it all together and I continued the process, adding deployment options and invoking more commands, particularly WP-CLI to prep the WordPress output (and to include a version of the plugin, WP Static Search, revised to work offline). If you examine the source code of this page, you will see the result.

Pipeline

Shell scripting became a means to orchestrate the overall process, which is readily seen as a pipeline comprising a sequence of phases. For example, one phases runs wget for the initial capture, whilst another re-runs wget to more completely capture a site. In fact, many core web development processes can be incorporated, such as replacing certain internal-facing pages or sections with alternative content; and invoking command-line tools for validation and streamlining output. As of version 0.21, there are ten phases altogether; and command-line options allow you to control the start phase and end phase.

The flowchart below illustrates an emerging pipeline (for the latest perspective, please refer to pages on workflow and phases).

Setup

Configuration dialogue

The first step is to run the setup.sh script, which guides the user in an interactive dialogue to specify a series of options.

Initialisation

Configure

The system reads and processes configuration options

• read runtime options
• read configuration file options
• prepare output layout

Prep the CMS

CMS-specific streamlining

Optionally use wp-cli to

• ensure nice permalinks
• remove query strings
• remote shortlinks
• replace with static search

(Currently WordPress-only, but modules could be developed for other CMS.)

Generate static site

Generate wget mirror (first run)

Run wget with the supplied configuration options

• optionally incorporate further input files
• optionally set up a zip files
• store the mirror in its own directory with timestamp

Augment static site

Further processing with wget

Run wget with the supplied configuration options to fill in gaps

• optionally crawl static files for further URLs
• Filter the URLs to retrieve only assets (multimedia files, etc.)
• Re-run wget on the additional URLs

Optimise static site

Further post-processing for deployment

Further post-wget processing

• convert feeds to be XML files and modify references accordingly
• Apply HTML Tidy to make more standards-compliant and use pretty-print (dependent on options)

An accessibility validator could be added here.

Create an offline zip archive

Create a custom zip based on the mirror

The mirror acts as the basis for a zip archive, ready for uploading with the site or other distribution channels.

Augment

Further additions

Optionally augment the static mirror with snippets, substituting chunks of HTML in particular places according to the use of snippet tags. These will augment the zip file also.
Additionally, allow the inclusion of further files that might not necessarily be static files.

Deploy

Finalise and deploy locally or remote

Optionally copy over additional

• Optionally copy over additional files, which might be scripts to provide missing interactivity
• Finally deploy on local or remote server using rsync

Extending the usefulness

MakeStaticSite is a prototype that was initially developed for individuals who are creating and maintaining websites on a personal computer. However, the question soon arose: how might a team of content authors and editors be supported? (The answer is: yes!)

For example, what if this site is to have multiple contributors? In this case, the most straightforward approach is to set up or continue hosting with a third-party hosting provider, in the usual way. This means those who are creating and managing content carry on as normal through the dashboard. The main change is to separate the editing environment from the public site, which can be achieved by restricting access by some means of web authentication. Then, run MakeStaticSite to deploy the static site to the public, whether on the server or remotely. This approach ensures continuity; the only additional consideration is how to initiate MakeStaticSite once changes have been made — from within the CMS or external to it?

An initial test has shown that once set up, the process of maintenance and deployment can be carried out entirely from within WordPress. This can be achieved by running the script under the terminal emulator WPTerm (MakeStaticSite has an option to run ‘unattended’, which makes default selections). This depends on appropriate support for MakeStaticSite on the server; for shared hosting, the versions of required components, particularly Wget, may be somewhat old and, of course, for any serious use with a third party, access to WPTerm should be properly secured as instructed. It’s preferable to keep the running of MakeStaticSite on a personal computer, an arrangement that has been used for fuengsin.org.

However, hosting with a third party means that the data is no longer immediately under our control; although a mixture of database replication and file synchronization can compensate, there’s some loss of independence. Perhaps a better solution lies in p2p networks, but that’s an area I have yet to explore …

For now, this tool serves my needs well and I hope it is helpful to others. Please refer to the MakeStaticSite project site for further information.

The generous ethos is admirable. Indeed, whereas the spotlight has recently been shone on software developers to become more aware of certain ethical issues, along with pressure to change their practices, there is in fact much to be learnt from that community. I have tried to intimate this in Open Sources: A Higher Consciousness in Software Development, a paper I gave at an unusual conference on the theme of wisdom.

As a developer myself, I am largely self-taught since the 1980s. It may explain why, when solving programming problems, I can be somewhat unconventional and occasionally original.

Open Source contributions

I’m not a prolific or fluent coder, but have contributed to a few open source projects and am currently working on two.

• MakeStaticSite, a static site generator and deployer, is a Bash script that I already use to generate static versions of locally-developed WordPress sites (like this one and research.siga.la). It is release as free/open source software under AGPL v.3 and is available for download.
• WP Offline Search is a further refinement of my fork of WP Static Search by Gergely Szerovay, who developed a WordPress plugin based on Lunr.js. As I’ve not received any response to my pull requests and have reworked quite a lot of the code, I changed the name and intend to set up a new repository.
• epubcheck-web (contributor) – enabling standards-based checking of ePub documents to help self-publishing, motivated by my experiences of self-publishing a Kindle book (see especially the EPUB conversion report slides).

I hope to share the Sigala code likewise, once it has reached a certain level of maturity.

For my work using other people’s software, take a look at the portfolio.

Legacy software, but enduring paradigms

At the turn of the millennium, there was no universal means for authors to edit and maintain their content through their web browser. The technical solutions that existed tended to work only on specific platforms, carried a fee, or else require a Java plugin.

In response, I devised one, NetWedit, possibly the world’s first cross-platform JavaScript-based solution, released under GNU LGPL, in the days when I still thought that the first ‘L’ stood for ‘library’ rather than ‘lesser’. It was experimental and used a crafty method, but it worked well enough for members of Oxford University’s Theology Faculty to maintain their profiles for a teach and learning resources database.

As browser DOM support improved, I subsequently enhanced the functionality, but kept the same basic user interface, to produce RTWedit, which was incorporated into early versions of WebLearn, Oxford University’s centrally-hosted virtual learning environment.

Standards have moved on and browsers only maintain limited backwards compatibility, so my tools don’t work in a standard desktop today, but the approach of working from first principles is ever-relevant. I’ve recorded some notes in a retrospective piece, Adventures in the Read/Write Web.

Nowadays, we take for granted the fact that we can write content in systems such as WordPress and Drupal using a rich text editor with an experience similar to that in a word processor. Such editors, based on packages such as TinyMCE and CK Editor are still written in JavaScript.

This website has been created in WordPress and continues to be maintained in this content management system. However, it is published as a set of static web pages using MakeStaticSite, a custom system comprising a set of Bash shell scripts, to be released as open source software.

WordPress plugins

The following plugins have been used for building the site.

• Collapse-O-Matic
• Enable Media Replacehttps://wordpress.org/plugins/teachpress/
• Flip Boxes
• Hello Dolly
• Kadence Blocks – Gutenberg Blocks for Page Builder Features
• Pages with category and tag
• PDFjs Viewer – Embed PDFs
• Perform – Optimize Performance
• Really Simple SSL
• Shortlink Removal
• Simple Page Ordering
• teachPress
• Timeline Block
• Visual Portfolio, Posts & Image Gallery
• WP Last Modified Info
• WP Rollback
• WP show more
• WP Sitemap Page
• WP Static Search (fork)
• WPCode – Insert Headers, Footers, and Code Snippets
  

2009-2012, 2014-2020

Role: Systems developer (as a staff member of the Museum of the History of Science, University of Oxford)

Brief: Develop and maintain the main Website for the Museum (but not the social media channels) and integrate the collections management system (KE EMu).

Solution: I oversaw the redevelopment of the WordPress site in 2010, mostly into one multisite instance, including previously disparate elements such as the collections search, and then its gradual refinement through to 2012. I overhauled the server provision, standardising on Debian Linux, in University data centres.

Security was enhanced at various levels, through content caches, script hardening, Apache mod Security with a fine-tuned ruleset, and integration with the University’s single sign-on system, first with Stanford Webauth and then with Shibboleth (to better support federation), registering and configuring a Service Provider (Apache module) and then configuring a WordPress plugin to invoke Shibboleth for authentication.

Subsequent work on the site focused on the design and instantiation of content, such as special exhibition sites (see example below). From 2017 onwards, it mainly concerned the digital preservation of legacy online exhibits during the migration to a centrally managed service, to which I also contributed substantially.

URLs
https://www.mhs.ox.ac.uk/ (main site until 2018), home page now redirecting to:
https://hsm.ox.ac.uk/ (main site from 2019 onwards).
https://blogs.mhs.ox.ac.uk/
Slides: integration of the collections database using the KE EMu PHP5 API.

–

MHS Special Exhibition: Back from the Dead – Demystifying Antibiotics

2016

Brief: To provide a website to accompany the Special Exhibition at the Museum of the History of Science (since renamed, History of Science Museum). Content needed to be easy to maintain by staff in various teams.

Solution: Using the existing WP multisite infrastructure I was able to create a number of special exhibition sites relatively easily, using responsive off-the-shelf themes. The basic structure was up and running very quickly and could be refined on an ongoing basis. Care was taken to ensure that educational aspects, especially, were made accessible.

URL: https://www.mhs.ox.ac.uk/backfromthedead/

Following a successful tender process, I was contracted to overhaul the website of the International Association for Religious Freedom (IARF), a UN NGO having general consultative status to ECOSOC with HQ in the UK.

The brief: To amalgamate relevant content from three existing sites, each developed separately during different periods and in a generally unwieldy state, into a single site that was current, covered the organisation’s activities well, and which would be easy to maintain. The new site would provide comprehensive coverage of the activities, styled with a look and feel similar to that of the United Nations.

Roles: After initial general consultancy to assess the current status, I undertook all the technical aspects of the project, from systems architecture and development, through to user training.

Solution: Working closely with the Secretariat Administrator, I implemented a new site to replace all legacy sites. I used WordPress (Multisite with subdomains), creating a custom theme derived from an existing one from the IARF US Chapter site. I also delivered experimental support for mobile devices using a theme switcher (before responsive designs became the norm).

I also oversaw the migration to a new VPS hosting platform with considerable cost savings. Detailed documentation was provided for administration, development and content authoring.

URL: https://iarf.net/ (via archive.org)

Please click on a thumbnail to access project details and slideshows.

.