software – ptworld.net

MakeStaticSite on GitHub

Paul — Wed, 08 Mar 2023 15:03:44 +0000

Work has continued on MakeStaticSite, the Bash utility to create static snapshots of websites, refine and deploy them — just through a single command.

Now at version 0.22.2 and available for download, recent updates include:

improved support for search engines through the generation of a sitemap based on actual file output, matched in each web page by the canonical URL specified in the tag. This sitemap is also cited by the site’s robots.txt file.
more robust handling of workflow, so you can use MakeStaticSite in conjunction with other scripts at various stages, to undertake other management tasks, such as reviewing accessibility or link checking.
more systematic treatment of options to clean WordPress ahead of being crawled by Wget. These small steps also point the way to broader support for fine-tuning CMS output.
Support for deployment on Netlify Content Distribution Network, which is where the site is now hosted. It’s expected that other CDN could be handled similarly.

We’ve even updated the animation on the Home page, extending the SimTerm plugin in the process.

Now that MakeStaticSite has been used successfully over a period of several months, it has been placed under Git version control. I am pleased to announce that I’ve created a repository on GitHub
https://github.com/paultraf/makestaticsite

In surveying approaches to generating static sites, I have not been able to find anything similar, and there is a myriad of ways in which it could be enhanced. Compared with some Jamstack approaches, the performance is slow, but building this site, which is quite substantial, only takes a few minutes whereas the MakeStaticSite project site takes less than a minute. For most purposes, this should be fine.

The use of the command line is an obstacle for some, but there are various ways in which a GUI could be provided, even Web CGI, which might be security concern on a public-facing server, but far less so on a personal computer. The beauty of using shell scripts is that you can quickly achieve desired results without being tied to any framework. So, please give it a go!

Introducing MakeStaticSite for website deployment

Paul — Wed, 21 Sep 2022 14:53:20 +0000

(from a technical perspective)

Launch of MakeStaticSite.sh Website

November 2022: A website dedicated to MakeStaticSite is now available at https://makestaticsite.sh/. It provides the latest version of the software, released under AGPL, along with various documentation to help getting started.

Motivation

The process of developing this website has evolved considerably. On setting it up a few years ago in the conventional manner using WordPress, I duly familiarised myself with routines for maintaining security and improving performance on a hosted platform. However, there were several aspects that weren’t ideal: the live content was hosted remotely, albeit with backups; updating the site in situ, particularly for major redesigns, was inconvenient; and performance wasn’t always as good as expected. Observing that most of the content was not dynamic in nature, I resolved to develop the site locally and then, as a separate process, deliver it as a largely static site.

The rationale for generating static sites has been around a long-time; I first tackled this more than 20 years ago, when I needed to generate a CD ROM from a CGI/Perl multimedia website. Nowadays, the field is mature; a Netlify blog post (Netlify is a cloud-based service where many static outputs end up) gives many valid reasons, rooted in the need for more responsive work processes. Typically underpinned by a simple language called markdown, there has been a surge in solutions; hence, options for Jamstack abound. However, the processes tend to be heavily oriented to developers at the expense of content authors, issues I know first-hand from enabling web access to museum collections.

I wanted instead to support existing content management processes, i.e. to continue with creating and managing content in WordPress, and then ‘at the press of a button’ generate and publish a static copy online. There are already some solutions, including dedicated plugins such as Simply Static. Tim Nash, having observed increasingly complexity with the Jamstack options, has described how he used WP2Static via WP-CLI, the WordPress command-line interface, for a (relatively) simple and compact solution. But I hesitate over the use of a WordPress plugin to generate the static site. Apart from the ongoing issue of maintenance and compatibility with WP Core, plugins are not really necessary since a public-facing website should be de facto crawlable by a web spider and not require internal system information.

And so I turned to general-purpose tools I could run from the command line. Enter Wget, a ubiquitous utility for retrieving and packaging any number of web pages for subsequent viewing. I needed to configure it, add a few additional inputs, and then choose where to store the outputs. This led naturally to delving into (Bash) shell scripting to bring it all together and I continued the process, adding deployment options and invoking more commands, particularly WP-CLI to prep the WordPress output (and to include a version of the plugin, WP Static Search, revised to work offline). If you examine the source code of this page, you will see the result.

Pipeline

Shell scripting became a means to orchestrate the overall process, which is readily seen as a pipeline comprising a sequence of phases. For example, one phases runs wget for the initial capture, whilst another re-runs wget to more completely capture a site. In fact, many core web development processes can be incorporated, such as replacing certain internal-facing pages or sections with alternative content; and invoking command-line tools for validation and streamlining output. As of version 0.21, there are ten phases altogether; and command-line options allow you to control the start phase and end phase.

The flowchart below illustrates an emerging pipeline (for the latest perspective, please refer to pages on workflow and phases).

Setup

Configuration dialogue

The first step is to run the setup.sh script, which guides the user in an interactive dialogue to specify a series of options.

Initialisation

Configure

The system reads and processes configuration options

read runtime options
read configuration file options
prepare output layout

Prep the CMS

CMS-specific streamlining

Optionally use wp-cli to

ensure nice permalinks
remove query strings
remote shortlinks
replace with static search

(Currently WordPress-only, but modules could be developed for other CMS.)

Generate static site

Generate wget mirror (first run)

Run wget with the supplied configuration options

optionally incorporate further input files
optionally set up a zip files
store the mirror in its own directory with timestamp

Augment static site

Further processing with wget

Run wget with the supplied configuration options to fill in gaps

optionally crawl static files for further URLs
Filter the URLs to retrieve only assets (multimedia files, etc.)
Re-run wget on the additional URLs

Optimise static site

Further post-processing for deployment

Further post-wget processing

convert feeds to be XML files and modify references accordingly
Apply HTML Tidy to make more standards-compliant and use pretty-print (dependent on options)

An accessibility validator could be added here.

Create an offline zip archive

Create a custom zip based on the mirror

The mirror acts as the basis for a zip archive, ready for uploading with the site or other distribution channels.

Augment

Further additions

Optionally augment the static mirror with snippets, substituting chunks of HTML in particular places according to the use of snippet tags. These will augment the zip file also.
Additionally, allow the inclusion of further files that might not necessarily be static files.

Deploy

Finalise and deploy locally or remote

Optionally copy over additional

Optionally copy over additional files, which might be scripts to provide missing interactivity
Finally deploy on local or remote server using rsync

Extending the usefulness

MakeStaticSite is a prototype that was initially developed for individuals who are creating and maintaining websites on a personal computer. However, the question soon arose: how might a team of content authors and editors be supported? (The answer is: yes!)

For example, what if this site is to have multiple contributors? In this case, the most straightforward approach is to set up or continue hosting with a third-party hosting provider, in the usual way. This means those who are creating and managing content carry on as normal through the dashboard. The main change is to separate the editing environment from the public site, which can be achieved by restricting access by some means of web authentication. Then, run MakeStaticSite to deploy the static site to the public, whether on the server or remotely. This approach ensures continuity; the only additional consideration is how to initiate MakeStaticSite once changes have been made — from within the CMS or external to it?

An initial test has shown that once set up, the process of maintenance and deployment can be carried out entirely from within WordPress. This can be achieved by running the script under the terminal emulator WPTerm (MakeStaticSite has an option to run ‘unattended’, which makes default selections). This depends on appropriate support for MakeStaticSite on the server; for shared hosting, the versions of required components, particularly Wget, may be somewhat old and, of course, for any serious use with a third party, access to WPTerm should be properly secured as instructed. It’s preferable to keep the running of MakeStaticSite on a personal computer, an arrangement that has been used for fuengsin.org.

However, hosting with a third party means that the data is no longer immediately under our control; although a mixture of database replication and file synchronization can compensate, there’s some loss of independence. Perhaps a better solution lies in p2p networks, but that’s an area I have yet to explore …

For now, this tool serves my needs well and I hope it is helpful to others. Please refer to the MakeStaticSite project site for further information.

MakeStaticSite

Paul — Tue, 20 Sep 2022 14:26:59 +0000

Foreword

The issues of Website security and performance continue to loom large, with organisations large and small continuing to suffer severe loss of service following a compromise — whether a site gets defaced, Google results become hijacked, or, more seriously, personal data is breached.

As will be described below, MakeStaticSite already offers a freely available solution (which has been implemented for this site), but it requires a certain level of technical knowledge. However, further development is on the way to provide a Web-based service that will be simple and easy to use. A proof of concept has already been implemented and been successfully tested, which should pave the way for business investment.

What’s the problem?

Security issues arise largely from the way the content is generated and delivered to site visitors using server-side processing, which typically involves scripts and databases to select and deliver specific text, multimedia, and so on. For example, content management systems (such as WordPress and Drupal) are based on PHP and MySQL, which, if misconfigured, might yield not only the desired pages but be open to requests crafted to subvert the setup and yield unintended access. Thus, maintaining the security of such a setup is an ongoing process that requires continual vigilance, responding promptly to security notices with the installation of updates, but it’s a race against time and there’s no guarantee that all vulnerabilities will be spotted.

MakeStaticSite belongs to the class of software known as static site generators, which removes all the dynamic components from public exposure, replacing them with a set of web pages (plain HTML, CSS, JavaScript) that require no server-side processing. The pages are prepared beforehand in a safer environment, such as an individual’s laptop or a shared space behind a firewall, protected by strict access controls. This simplifies the delivery process as well as helping with security. Shell scripting is used, making available various tools to handle different aspects of the task. An introductory post provides an overview of the processes involved.

For developers and system administrators, MakeStaticSite provides a convenient means to set up and manage the automatic creation and deployment of static versions of websites. Such sites can be administered locally and then deployed remotely to some other hosting provider. Strictly speaking, this is not an archival tool as the output is not an exact mirror, but a version of the site that preserves content whilst aiming to remain current. For example, RSS feeds are saved and then renamed with .xml extensions, and further files may be added.

Anyone who has some familiarity with the command line should be able to use the tool to assist in maintaining their sites, though such an interface is off-putting to some. Further, a scripting-based approach has been chosen to make this tool widely accessible for developers to further fine-tune; a number of refinements are already included that augment the standard use of Wget, such as support for arbitrary attributes and, in the case of WordPress, the use of WP-CLI to prepare sites beforehand.

MakeStaticSite has been released as open source software, available under the GNU Affero General Public License version 3 . For further details, please refer to the project site, makestaticsite.sh. Many thanks to various developers for sharing their knowledge on shell scripting, particularly on blogs and Q&A websites such as Stack Exchange and to those who have tested, commented on and otherwise supported MakeStaticSite.

Hosted Web Service

In its present form, it is technical in nature, aimed at developers and system administrators. However, as a further development, a hosted Web service is being developed, with a simple user interface.

The user logs in to a central dashboard to maintain their projects. By following a link or clicking a button, they can launch their CMS editor, build a static version of the site and browse the output.

How you can help!

At this stage, we are looking for investment. Please get in touch to try out the demo and to arrange for a presentation.

QMA Collections Intranet

Paul — Tue, 14 Jun 2022 14:47:11 +0000

Web-based Intranet for Qatar Museums (2012-2014)

Problem

The Qatar Museums Authority (via archive.org) had responsibility for a dozen or so new museums that were emerging to make Qatar a cultural hub for the Gulf. A single EMu database instance was set up with core data documentation standards, but access to EMu was limited to those who had registered user accounts on and access to the EMu client. Other museums staff, such as those in education, needed convenient access to essential information.

Solution

Inspired by the WordPress architecture, particularly its use of redirects, I conceived, architected and developed an intranet for the Qatar Museums Authority. This enabled Web access with custom views for each staff in museum, with session management integrated with MS Exchange. The user interface was responsive and implemented i18n, with instant toggle between English (left-to-right) and Arabic (right-to-left) — supported by a translation project to map all the necessary cataloguing fields. The resulting system was deployed to all the Museums and also the office of the Chair of QMA.

( Details … )

Architecture:

Multi-site architecture where each site (museum) has its own configuration; URL rewrites provide friendly URLs.
Authentication: integration with Active Directory
Authorisation: Per-site custom role-based authorisation supports Apache htgroups, EMu Groups (Registry module), and Active Directory LDAP

Features:

Customisable fields for the search and display of results and object pages
Data: XML template system generates a PHP XML template from the KE EMu XML schema
Presentation: self-contained XSLT transforms XML to HTML, kept separate from program logic
Session handling for runtime preferences
i18n internationalization English-Arabic (using gettext) – language toggle for each page
Image display configurable with IMu query optimisation
Responsive web design with collapsible columns with drag and drop panels, optimised for iPad.

Development environment:
*Oracle VirtualBox running Debian Linux, NetBeans, Subversion;
*Apache, LDAP, PHP (with IMu API), HTML, CSS, JavaScript, jQuery, jQueryUI.

( hide details … )

Software

Paul — Tue, 07 Jun 2022 13:15:13 +0000

The digital world is made up of software and hardware and increasingly permeates our lives. Analogous to making material donations, many developers offer software freely for the wider community in the hope that others may benefit also. They typically do this by releasing software under an open source license.

The generous ethos is admirable. Indeed, whereas the spotlight has recently been shone on software developers to become more aware of certain ethical issues, along with pressure to change their practices, there is in fact much to be learnt from that community. I have tried to intimate this in Open Sources: A Higher Consciousness in Software Development, a paper I gave at an unusual conference on the theme of wisdom.

As a developer myself, I am largely self-taught since the 1980s. It may explain why, when solving programming problems, I can be somewhat unconventional and occasionally original.

Open Source contributions

I’m not a prolific or fluent coder, but have contributed to a few open source projects and am currently working on two.

MakeStaticSite, a static site generator and deployer, is a Bash script that I already use to generate static versions of locally-developed WordPress sites (like this one and research.siga.la). It is release as free/open source software under AGPL v.3 and is available for download.
WP Offline Search is a further refinement of my fork of WP Static Search by Gergely Szerovay, who developed a WordPress plugin based on Lunr.js. As I’ve not received any response to my pull requests and have reworked quite a lot of the code, I changed the name and intend to set up a new repository.
epubcheck-web (contributor) – enabling standards-based checking of ePub documents to help self-publishing, motivated by my experiences of self-publishing a Kindle book (see especially the EPUB conversion report slides).

I hope to share the Sigala code likewise, once it has reached a certain level of maturity.

For my work using other people’s software, take a look at the portfolio.

Legacy software, but enduring paradigms

At the turn of the millennium, there was no universal means for authors to edit and maintain their content through their web browser. The technical solutions that existed tended to work only on specific platforms, carried a fee, or else require a Java plugin.

In response, I devised one, NetWedit, possibly the world’s first cross-platform JavaScript-based solution, released under GNU LGPL, in the days when I still thought that the first ‘L’ stood for ‘library’ rather than ‘lesser’. It was experimental and used a crafty method, but it worked well enough for members of Oxford University’s Theology Faculty to maintain their profiles for a teach and learning resources database.

As browser DOM support improved, I subsequently enhanced the functionality, but kept the same basic user interface, to produce RTWedit, which was incorporated into early versions of WebLearn, Oxford University’s centrally-hosted virtual learning environment.

An early example (2005-7) of rich text editing in WebLearn, using RTWedit.
(Screenshot of a slide from OUCS IT Learning Programme training course led by Paul Trafford.)

Standards have moved on and browsers only maintain limited backwards compatibility, so my tools don’t work in a standard desktop today, but the approach of working from first principles is ever-relevant. I’ve recorded some notes in a retrospective piece, Adventures in the Read/Write Web.

Nowadays, we take for granted the fact that we can write content in systems such as WordPress and Drupal using a rich text editor with an experience similar to that in a word processor. Such editors, based on packages such as TinyMCE and CK Editor are still written in JavaScript.

Cognitive Interventions

Paul — Tue, 07 Jun 2022 09:49:35 +0000

How do we design systems that lead to generally wholesome states of mind? Having studied some of the psychological observations concerning social media, it is evident that cognitive interventions already feature prominently, generally in a way that fosters addiction, which is an unwholesome state.

However, cognitive interventions can also be incorporated in the design process so as to foster positive states. Furthermore, there are well-established methods, particularly Harvard University’s Project Zero thinking routines that suggest how this might be implemented. This was the subject of my paper: Cultivating sīla Online: the use of Cognitive Interventions in Systems Design, which I presented at the Ninth annual conference of the Jubilee Centre for Character and Virtues. A copy of the paper is available on the conference website, as well as on Academia.

We may furthermore specify such systems mathematically using a formalism such as process algebra and then specify safety properties in a temporal logic, which we apply to the model for validation. Whereas in the last millennium, formal methods were considered too rarefied for general use, nowadays they have become much more mainstream spear-headed by Petri Nets and the Pi-Calculus.

These are only early intimations, but I’m quite sure this research would be worthwhile.

Sigala

Paul — Tue, 31 May 2022 18:41:50 +0000

The Sigala project concerns the research, design and development of software systems to support sustainable social networking. It is the main focus of my endeavours for the foreseeable future.

The Problem

Why is there so much discontent with existing the social media provided by the ‘big tech’ companies? One trite answer is simply because social media fosters discontent!

How does it foster discontent? By promoting craving and addiction, which is why paradoxically people continue using these services despite their reservations about them. As confessed by Sean Parker, former President of Facebook:

“The thought process that went into building these applications, … was all about: ‘How do we consume as much of your time and conscious attention as possible?'” “And that means that we need to sort of give you a little dopamine hit every once in a while, because someone liked or commented on a photo or a post or whatever. And that’s going to get you to contribute more content, and that’s going to get you … more likes and comments.” “It’s a social-validation feedback loop … exactly the kind of thing that a hacker like myself would come up with, because you’re exploiting a vulnerability in human psychology.
Mike Allen, ‘Sean Parker Unloads on Facebook: “God Only Knows What it’s Doing to Our Children’s Brains”’, Axios, 9
November 2017

A Solution

To redesign social media from the ground up rooted in the principles of Buddhist ethics and the cultivation of wholesome relationships. This should improve the quality of awareness, enhance human agency, encourage independence of thought, and not foster addiction. This is what the Sigala project is all about.

Why ‘Sigala’?

The name is derived from the name of a follower to whom the Buddha gave the most definitive teachings on how lay people should conduct themselves for human flourishing. Actually the layman’s name was Sigāla (long ‘a’), but the diacritics can easily get omitted. Also, with a short ‘a’, it can be pronounced in the same lively way as ‘Figaro’!

Current status

Research has been ongoing since 2007 and explored various themes. From 2018 onwards, in response to Parker’s admission, special attention has been given to mind and cognition. For descriptions of these investigations, please consult the dedicated Sigala research site: https://research.siga.la/.

The next step is to code a proof of concept, for which I have established Symfony PHP framework as a suitable candidate.

RAMBLE

Paul — Sun, 22 May 2022 14:21:18 +0000

Date: 2004-2005

Brief: This project was to demonstrate how service-oriented architecture (SOA) could link personal learning spaces and institutional systems.

Roles: Principle Investigator (JISC-funded project, staff member of the Learning Technologies Group, Oxford University Computing Service), systems analyst and systems administrator. This was concurrent to my main role of running WebLearn, Oxford University’s centrally-hosted virtual learning environment.

Solution: I established a small team of about six, including colleagues in Chemistry and Medical Sciences. We devised a workflow that encompassed student blogging on a handheld device and WebLearn. Technically, the post content was shared through RSS and Atom feeds, restricted by server and application configurations; I oversaw the development of a new blogging component in the VLE. Whilst establishing blogging as a reflective learning activity, I also uncovered insights into the conditions that foster deeper learning, as intimated in an article for Ariadne. Furthermore, the project had a more general impact on mobile developments at the University.

URL: http://projects.oucs.ox.ac.uk/ramble/

RAMBLE Architecture

RAMBLE Project: iPaq PDA and Belkin keyboard

RAMBLE: Blogging setup 1/2

RAMBLE: Blogging setup 2/2

RAMBLE: Wireless PDA Keyboard Settings

RAMBLE – Danny Tucker’s blog

RAMBLE Project blog

RAMBLE Project blog: seminar

RAMBLE Project: reading private blogs

RAMBLE: VLE Blog Creation

RAMBLE: VLE Blog properties

RAMBLE Project blog – VLE view

HSM DAMS

Paul — Sat, 21 May 2022 15:32:11 +0000

Date: 2019-2020 (staff member); November 2021 (consultant)

Brief: A digital asset management system (ResourceSpace, fully-supported by Montala) was procured for the History of Science Museum, along with other departments in Oxford University’s Gardens, Libraries and Museums (GLAM). The DAMS was used as an intranet, primarily for its object photography. It needed broad stakeholder adoption and to be integrated with the existing collections management system (Axiell EMu).

Roles: As the coordinator of digital projects at HSM, I was involved in the selection of the DAMS and then was responsible for its implementation, including the ingestion of tranches, together with documentation and basic training. For the consultancy I provided further support for this process.

Solution: Considerable preparation was required before we could use the system. I undertook or oversaw a number of stages, including the development of policy comprising a digital preservation policy, retention schedules and data protection. These ran alongside a digital assets audit, metadata specification process (involving all potential stakeholders), system requirements, data export, cleansing, munging and ingestion. As it was messy, I developed a Digital Assets Toolkit, comprising a set of custom scripts for some of the heavy lifting.

URL: https://www.glam.ox.ac.uk/digital-asset-management-system-dams-project

HSM DAMS: main dashboard

HSM DAMS: Collection Categories

HSM DAMS: Featured Collection

HSM DAMS asset preview

HSM DAMS – Resource Tools

HSM DAMS – Related Resources

HSM EMu exports specification

HSM Digital Assets Toolkit

MHS/HSM Website

Paul — Tue, 17 May 2022 16:42:08 +0000

2009-2012, 2014-2020

Role: Systems developer (as a staff member of the Museum of the History of Science, University of Oxford)

Brief: Develop and maintain the main Website for the Museum (but not the social media channels) and integrate the collections management system (KE EMu).

Solution: I oversaw the redevelopment of the WordPress site in 2010, mostly into one multisite instance, including previously disparate elements such as the collections search, and then its gradual refinement through to 2012. I overhauled the server provision, standardising on Debian Linux, in University data centres.

Security was enhanced at various levels, through content caches, script hardening, Apache mod Security with a fine-tuned ruleset, and integration with the University’s single sign-on system, first with Stanford Webauth and then with Shibboleth (to better support federation), registering and configuring a Service Provider (Apache module) and then configuring a WordPress plugin to invoke Shibboleth for authentication.

Subsequent work on the site focused on the design and instantiation of content, such as special exhibition sites (see example below). From 2017 onwards, it mainly concerned the digital preservation of legacy online exhibits during the migration to a centrally managed service, to which I also contributed substantially.

URLs
https://www.mhs.ox.ac.uk/ (main site until 2018), home page now redirecting to:
https://hsm.ox.ac.uk/ (main site from 2019 onwards).
https://blogs.mhs.ox.ac.uk/
Slides: integration of the collections database using the KE EMu PHP5 API.

MHS Home page 2012

MHS: History (with the ATBar)

MHS Collections Search

MHS Collections: Field Search

MHS Steampunk exhibition

MHS Steampunk exhibition – archive

MHS Steampunk Live Mannequin

The Renaissance in Astronomy

MHS Blogs

–

MHS Special Exhibition: Back from the Dead – Demystifying Antibiotics

2016

Brief: To provide a website to accompany the Special Exhibition at the Museum of the History of Science (since renamed, History of Science Museum). Content needed to be easy to maintain by staff in various teams.

Solution: Using the existing WP multisite infrastructure I was able to create a number of special exhibition sites relatively easily, using responsive off-the-shelf themes. The basic structure was up and running very quickly and could be refined on an ongoing basis. Care was taken to ensure that educational aspects, especially, were made accessible.

URL: https://www.mhs.ox.ac.uk/backfromthedead/