development – ptworld.net

MakeStaticSite on GitHub

Paul — Wed, 08 Mar 2023 15:03:44 +0000

Work has continued on MakeStaticSite, the Bash utility to create static snapshots of websites, refine and deploy them — just through a single command.

Now at version 0.22.2 and available for download, recent updates include:

improved support for search engines through the generation of a sitemap based on actual file output, matched in each web page by the canonical URL specified in the tag. This sitemap is also cited by the site’s robots.txt file.
more robust handling of workflow, so you can use MakeStaticSite in conjunction with other scripts at various stages, to undertake other management tasks, such as reviewing accessibility or link checking.
more systematic treatment of options to clean WordPress ahead of being crawled by Wget. These small steps also point the way to broader support for fine-tuning CMS output.
Support for deployment on Netlify Content Distribution Network, which is where the site is now hosted. It’s expected that other CDN could be handled similarly.

We’ve even updated the animation on the Home page, extending the SimTerm plugin in the process.

Now that MakeStaticSite has been used successfully over a period of several months, it has been placed under Git version control. I am pleased to announce that I’ve created a repository on GitHub
https://github.com/paultraf/makestaticsite

In surveying approaches to generating static sites, I have not been able to find anything similar, and there is a myriad of ways in which it could be enhanced. Compared with some Jamstack approaches, the performance is slow, but building this site, which is quite substantial, only takes a few minutes whereas the MakeStaticSite project site takes less than a minute. For most purposes, this should be fine.

The use of the command line is an obstacle for some, but there are various ways in which a GUI could be provided, even Web CGI, which might be security concern on a public-facing server, but far less so on a personal computer. The beauty of using shell scripts is that you can quickly achieve desired results without being tied to any framework. So, please give it a go!

MakeStaticSite

Paul — Tue, 20 Sep 2022 14:26:59 +0000

Foreword

The issues of Website security and performance continue to loom large, with organisations large and small continuing to suffer severe loss of service following a compromise — whether a site gets defaced, Google results become hijacked, or, more seriously, personal data is breached.

As will be described below, MakeStaticSite already offers a freely available solution (which has been implemented for this site), but it requires a certain level of technical knowledge. However, further development is on the way to provide a Web-based service that will be simple and easy to use. A proof of concept has already been implemented and been successfully tested, which should pave the way for business investment.

What’s the problem?

Security issues arise largely from the way the content is generated and delivered to site visitors using server-side processing, which typically involves scripts and databases to select and deliver specific text, multimedia, and so on. For example, content management systems (such as WordPress and Drupal) are based on PHP and MySQL, which, if misconfigured, might yield not only the desired pages but be open to requests crafted to subvert the setup and yield unintended access. Thus, maintaining the security of such a setup is an ongoing process that requires continual vigilance, responding promptly to security notices with the installation of updates, but it’s a race against time and there’s no guarantee that all vulnerabilities will be spotted.

MakeStaticSite belongs to the class of software known as static site generators, which removes all the dynamic components from public exposure, replacing them with a set of web pages (plain HTML, CSS, JavaScript) that require no server-side processing. The pages are prepared beforehand in a safer environment, such as an individual’s laptop or a shared space behind a firewall, protected by strict access controls. This simplifies the delivery process as well as helping with security. Shell scripting is used, making available various tools to handle different aspects of the task. An introductory post provides an overview of the processes involved.

For developers and system administrators, MakeStaticSite provides a convenient means to set up and manage the automatic creation and deployment of static versions of websites. Such sites can be administered locally and then deployed remotely to some other hosting provider. Strictly speaking, this is not an archival tool as the output is not an exact mirror, but a version of the site that preserves content whilst aiming to remain current. For example, RSS feeds are saved and then renamed with .xml extensions, and further files may be added.

Anyone who has some familiarity with the command line should be able to use the tool to assist in maintaining their sites, though such an interface is off-putting to some. Further, a scripting-based approach has been chosen to make this tool widely accessible for developers to further fine-tune; a number of refinements are already included that augment the standard use of Wget, such as support for arbitrary attributes and, in the case of WordPress, the use of WP-CLI to prepare sites beforehand.

MakeStaticSite has been released as open source software, available under the GNU Affero General Public License version 3 . For further details, please refer to the project site, makestaticsite.sh. Many thanks to various developers for sharing their knowledge on shell scripting, particularly on blogs and Q&A websites such as Stack Exchange and to those who have tested, commented on and otherwise supported MakeStaticSite.

Hosted Web Service

In its present form, it is technical in nature, aimed at developers and system administrators. However, as a further development, a hosted Web service is being developed, with a simple user interface.

The user logs in to a central dashboard to maintain their projects. By following a link or clicking a button, they can launch their CMS editor, build a static version of the site and browse the output.

How you can help!

At this stage, we are looking for investment. Please get in touch to try out the demo and to arrange for a presentation.

Business

Paul — Wed, 08 Jun 2022 16:28:15 +0000

In a very general sense, business means occupation (see the etymology). Here, the specific occupation is around software, which are the intended products, which may be partly free and partly commercial. Accordingly, in the early stages, the business is mainly concerned with the development of software prototypes, beginning with a proof of concept, i.e. a website (or, possibly app) that can be demonstrated and tested by selected parties.

The overall goal is the development of Sigala, but this is unconventional, so as a stepping stone attention is being given to more mainstream projects that can generate start-up capital.

MakeStaticSiteWeb

The first of these is to implement MakeStaticSite as an easy-to-use service hosted on the Web.

How you can help

One of the first tasks is to draft a business plan. At this early stage, each contributor needs to be self-financed, until we secure financial backing. The immediate focus is on MakeStaticSiteWeb, which is envisaged as a kind of financial bootstrap. For the longer term, I have expressed some thoughts on economic viability (on the Sigala research site).

We’ll need to form a team. There are many roles for an initiative like Sigala, especially once the project gains momentum and the software development is put on a more formal footing. Some which will be crucial are:

community liaison — I wish to target early versions at children, so as to make the design inherently safe. I would like the liaison to work with existing organisations such as the Jubilee Centre to develop in conjunction with schools, but also work through community elders, perhaps through bodies such as councils of faith.
researchers — the project’s research is ongoing and needs to be strengthened by relevant expertise in many fields. This can involve both formal and informal collaboration.
software development
- requirements analyst — this technical role, working closely with the community liaison, will translate the requirements from communities into software requirements. Particular attention should be given to safety requirements.
- systems architect — what software services do we need, how are the components related to each other? Particular attention is to be given to security, which is a role in itself.
- lead developer — in the early stages of prototyping, this role will be pivotal. Once the project starts progressing beyond that, they will need to build their own team of developers, including testers.
- usability specialist — a role intrinsic to the design and development process, with particular attention to accessibility.
legal — obligations and best practice covering accessibility, privacy, intellectual property rights, etc.
communications — to keep everyone, internally and externally, informed of project progress.

Software

Paul — Tue, 07 Jun 2022 13:15:13 +0000

The digital world is made up of software and hardware and increasingly permeates our lives. Analogous to making material donations, many developers offer software freely for the wider community in the hope that others may benefit also. They typically do this by releasing software under an open source license.

The generous ethos is admirable. Indeed, whereas the spotlight has recently been shone on software developers to become more aware of certain ethical issues, along with pressure to change their practices, there is in fact much to be learnt from that community. I have tried to intimate this in Open Sources: A Higher Consciousness in Software Development, a paper I gave at an unusual conference on the theme of wisdom.

As a developer myself, I am largely self-taught since the 1980s. It may explain why, when solving programming problems, I can be somewhat unconventional and occasionally original.

Open Source contributions

I’m not a prolific or fluent coder, but have contributed to a few open source projects and am currently working on two.

MakeStaticSite, a static site generator and deployer, is a Bash script that I already use to generate static versions of locally-developed WordPress sites (like this one and research.siga.la). It is release as free/open source software under AGPL v.3 and is available for download.
WP Offline Search is a further refinement of my fork of WP Static Search by Gergely Szerovay, who developed a WordPress plugin based on Lunr.js. As I’ve not received any response to my pull requests and have reworked quite a lot of the code, I changed the name and intend to set up a new repository.
epubcheck-web (contributor) – enabling standards-based checking of ePub documents to help self-publishing, motivated by my experiences of self-publishing a Kindle book (see especially the EPUB conversion report slides).

I hope to share the Sigala code likewise, once it has reached a certain level of maturity.

For my work using other people’s software, take a look at the portfolio.

Legacy software, but enduring paradigms

At the turn of the millennium, there was no universal means for authors to edit and maintain their content through their web browser. The technical solutions that existed tended to work only on specific platforms, carried a fee, or else require a Java plugin.

In response, I devised one, NetWedit, possibly the world’s first cross-platform JavaScript-based solution, released under GNU LGPL, in the days when I still thought that the first ‘L’ stood for ‘library’ rather than ‘lesser’. It was experimental and used a crafty method, but it worked well enough for members of Oxford University’s Theology Faculty to maintain their profiles for a teach and learning resources database.

As browser DOM support improved, I subsequently enhanced the functionality, but kept the same basic user interface, to produce RTWedit, which was incorporated into early versions of WebLearn, Oxford University’s centrally-hosted virtual learning environment.

An early example (2005-7) of rich text editing in WebLearn, using RTWedit.
(Screenshot of a slide from OUCS IT Learning Programme training course led by Paul Trafford.)

Standards have moved on and browsers only maintain limited backwards compatibility, so my tools don’t work in a standard desktop today, but the approach of working from first principles is ever-relevant. I’ve recorded some notes in a retrospective piece, Adventures in the Read/Write Web.

Nowadays, we take for granted the fact that we can write content in systems such as WordPress and Drupal using a rich text editor with an experience similar to that in a word processor. Such editors, based on packages such as TinyMCE and CK Editor are still written in JavaScript.

RAMBLE

Paul — Sun, 22 May 2022 14:21:18 +0000

Date: 2004-2005

Brief: This project was to demonstrate how service-oriented architecture (SOA) could link personal learning spaces and institutional systems.

Roles: Principle Investigator (JISC-funded project, staff member of the Learning Technologies Group, Oxford University Computing Service), systems analyst and systems administrator. This was concurrent to my main role of running WebLearn, Oxford University’s centrally-hosted virtual learning environment.

Solution: I established a small team of about six, including colleagues in Chemistry and Medical Sciences. We devised a workflow that encompassed student blogging on a handheld device and WebLearn. Technically, the post content was shared through RSS and Atom feeds, restricted by server and application configurations; I oversaw the development of a new blogging component in the VLE. Whilst establishing blogging as a reflective learning activity, I also uncovered insights into the conditions that foster deeper learning, as intimated in an article for Ariadne. Furthermore, the project had a more general impact on mobile developments at the University.

URL: http://projects.oucs.ox.ac.uk/ramble/

RAMBLE Architecture

RAMBLE Project: iPaq PDA and Belkin keyboard

RAMBLE: Blogging setup 1/2

RAMBLE: Blogging setup 2/2

RAMBLE: Wireless PDA Keyboard Settings

RAMBLE – Danny Tucker’s blog

RAMBLE Project blog

RAMBLE Project blog: seminar

RAMBLE Project: reading private blogs

RAMBLE: VLE Blog Creation

RAMBLE: VLE Blog properties

RAMBLE Project blog – VLE view

International Association for Religious Freedom

Paul — Mon, 16 May 2022 10:00:18 +0000

June 2011 – April 2012

Following a successful tender process, I was contracted to overhaul the website of the International Association for Religious Freedom (IARF), a UN NGO having general consultative status to ECOSOC with HQ in the UK.

The brief: To amalgamate relevant content from three existing sites, each developed separately during different periods and in a generally unwieldy state, into a single site that was current, covered the organisation’s activities well, and which would be easy to maintain. The new site would provide comprehensive coverage of the activities, styled with a look and feel similar to that of the United Nations.

Roles: After initial general consultancy to assess the current status, I undertook all the technical aspects of the project, from systems architecture and development, through to user training.

Solution: Working closely with the Secretariat Administrator, I implemented a new site to replace all legacy sites. I used WordPress (Multisite with subdomains), creating a custom theme derived from an existing one from the IARF US Chapter site. I also delivered experimental support for mobile devices using a theme switcher (before responsive designs became the norm).

I also oversaw the migration to a new VPS hosting platform with considerable cost savings. Detailed documentation was provided for administration, development and content authoring.

URL: https://iarf.net/ (via archive.org)

IARF homepage (2015)

About the International Association for Religious Freedom

IARF at the UN

IARF Member Groups

IARF projects in India

IARF Contact Page

IARF Religious Freedom Young Adult Network

IARF Events

IARF Congress 2014

Portfolio

Paul — Thu, 12 May 2022 12:37:01 +0000

This portfolio of work covers a number of projects across Web development and other forms of publication. It is not exhaustive.

Please click on a thumbnail to access project details and slideshows.

Theology Faculty Teaching and Learning Resources Web

CTCFL Elementary-Intermediate Chinese CD-ROM

Interactive Audio Comprehension Materials (Phase II)

JISC RAMBLE Project

Educause 2006: From Personalized Learning to Open Courseware: Learning Management Systems Can Be Flexible

MHS Website (WordPress developments)

IARF Website redevelopment

Jones’ Icones Online: Home

OUMNH Collections Online beta

Thursday’s Lotus (paperback)

Thursday’s Lotus – e-book production

MHS Special Exhibition: Back from the Dead

HSM Digital Asset Management

Projects

Paul — Sat, 09 Apr 2016 17:16:53 +0000

The projects I have been undertaking in the past 25 years or so span theoretical research, software development, policy, implementation and practice. Much is experimental and might not be realized as a product, but in seeking practical benefit, I try to retain what I’ve learnt as it may yet have some application later on. Many problems are recurrent and the approaches (and insights) may be similar at each iteration, albeit with different tools.

My main role has been as a problem solver / developer for/of web applications, mainly for the public sector, in both staff and consultancy roles. At a university, ‘off the shelf’ products are seldom sufficient out of the box: there is usually a considerable amount of customisation required, which motivates in-house solutions. Thus, I have worked individually and in small teams, particularly the (now closed) Academic Computing Development Team, allowing me to undertake bespoke full-stack development; wherever possible this is in accordance with open standards and using open source components (in the early days, CGI/Perl and then PHP, connecting to SQL databases; then XML and XSLT). These standards underpin the software I am currently developing. Nowadays, apart from Sigala, my development is oriented to customisations of popular systems such as WordPress.

Whilst most of my experience of development has been small-scale, I have also been heavily involved in an institutional project – WebLearn, Oxford University’s first virtual learning environment, for which I was involved in the procurement and subsequently was responsible for the operational management when it became a centrally-hosted production service. After a while, this involved crunching through tens of millions of event log database rows to generate summary usage figures.

I have also coordinated and managed a few projects, including RAMBLE and, more recently, digital asset management at the History of Science Museum.

For details of past projects, please consult my portfolio. Here, I only highlight the start of current work.

Sigala Research Knowledge Basis (2020-)

Problem

The Sigala project concerns the research, design and development of software systems to support sustainable social networking. There is a substantial body of theory underpinning the software development, but it’s been built up sporadically, initially documented in MS Word, with a few blog posts and the occasional presentation.

To put the research on a more solid footing is a project in itself; it needs to be maintained and disseminated more sustainably, whilst making it easier to manage. The form in which the research is shared needs to be convenient.

Solution

The references are being managed in Zotero, which not only supports bibliographic standards for records management and import/export, but also has a web-mediated infrastructure that supports shared working.

For dissemination, a WordPress site has been created locally and content migrated from the Word documents. A workflow enables convenient deployment of a static site on a remote server along with a zip archive of almost identical content. For added convenience, a WordPress plugin, WP Static Search (based on Lunr.js), has been extended to enable offline searches. Both of these software components will be released as open source software.

URL: https://research.siga.la/