ptworld.net

MakeStaticSite on GitHub

Paul — Wed, 08 Mar 2023 15:03:44 +0000

Work has continued on MakeStaticSite, the Bash utility to create static snapshots of websites, refine and deploy them — just through a single command.

Now at version 0.22.2 and available for download, recent updates include:

improved support for search engines through the generation of a sitemap based on actual file output, matched in each web page by the canonical URL specified in the tag. This sitemap is also cited by the site’s robots.txt file.
more robust handling of workflow, so you can use MakeStaticSite in conjunction with other scripts at various stages, to undertake other management tasks, such as reviewing accessibility or link checking.
more systematic treatment of options to clean WordPress ahead of being crawled by Wget. These small steps also point the way to broader support for fine-tuning CMS output.
Support for deployment on Netlify Content Distribution Network, which is where the site is now hosted. It’s expected that other CDN could be handled similarly.

We’ve even updated the animation on the Home page, extending the SimTerm plugin in the process.

Now that MakeStaticSite has been used successfully over a period of several months, it has been placed under Git version control. I am pleased to announce that I’ve created a repository on GitHub
https://github.com/paultraf/makestaticsite

In surveying approaches to generating static sites, I have not been able to find anything similar, and there is a myriad of ways in which it could be enhanced. Compared with some Jamstack approaches, the performance is slow, but building this site, which is quite substantial, only takes a few minutes whereas the MakeStaticSite project site takes less than a minute. For most purposes, this should be fine.

The use of the command line is an obstacle for some, but there are various ways in which a GUI could be provided, even Web CGI, which might be security concern on a public-facing server, but far less so on a personal computer. The beauty of using shell scripts is that you can quickly achieve desired results without being tied to any framework. So, please give it a go!

AI – Imagining in the morning, Engaging in the afternoon

Paul — Tue, 27 Sep 2022 14:47:31 +0000

Exhibition at the Weston Library

On Saturday morning I met up with a friend at the Bodleian’s Weston Library in Oxford. On entering the building, we paid a quick visit to a small exhibition, Imagining AI, curated by Professor Ursula Martin, who has considerable knowledge of the history of computing, and has written and presented about Ada Lovelace’s significant contribution.

This exhibition had a strong literary theme as it drew mainly on the Bodleian’s special collections, putting on display rare books, manuscripts and other documents. The introductory panel invited us to “explore the boundary between humans and machines”. I found this an interesting, even provocative pitch, setting humans and machines on opposite sides, like adversaries. This was then extrapolated. To paraphrase: will machines foster some fantastic utopia or nightmarish dystopia and take over the world? Lest we dwell in happy reverie, the selections, just to the right of that panel included extracts from Mary Shelley’s Frankenstein.

With the (surely, generally accepted) observation that machines are more than mere “number crunchers”, the exhibition developed the theme of computation, highlighting the work of Charles Babbage and Ada Lovelace on the Difference Engine (constructed) and Analytical Engine (conceived, never built). An underlying theme, supported by a quote from Lovelace, was human mimicry, especially of the brain’s thought processes. W. Stanley Jevons aimed to demonstrate rationality by his “logical piano”, on loan from the History of Science Museum. More subtle forms of human mental activity, considered by some to be among the first AI programs, were expressed in the simplistic love poetry generator of Christopher Strackey (as re-implemented (or re-enacted?) in modern code) – more than a decade before the more well-known Eliza. These developments were explored and brought up to date in an associated event (which I didn’t attend), evidencing how the complexity has increased exponentially and has become prominent in the arts, where there’s the ‘Wow!’ factor – picked up by a local paper, which invited readers to meet Ai-Da, a superstar robot artist.

Thus, with all these bells and whistles, there is a sense of awe, even reference around these “transformative technologies”. But who or what is being transformed and in what ways? Is our well-being transformed?

The exhibition begs so many unanswered questions. I join the many who have claimed the existential threat is real and urgent, but what is less said and which I feel is just as imperative, if not moreso, is that in order to address and properly evaluate AI, we need to balance the discussions of machine ability with human ability; we need to remind ourselves and explore further what makes human life so remarkable in terms of originality and creativity. Furthermore, we need to reflect on the qualities of human sentience – to be and know – and how the presence of machines impacts this.

Without such a perspective, it’s too easy to fall into the trap of making (belated) comparisons under a narrow instrumentalist framework. Especially when millions of pounds are invested to demonstrate measurable benefits. As the sophistication increases, the accompanying language sometimes becomes hyperbolic and even idolatrous. Whilst computation elicits a myriad of possibilities to excite, we should not lose focus on the nature of the human mind, of its often-sublime qualities such as autonomy, reflection, intuition, originality, purpose and perseverance. All of these contribute to human agency and hence human flourishing and happiness.

Ada Lovelace was quoted expressing the power of these machines, but I think she herself might have given greater prominence – particularly given the exhibition’s title – to what she herself had expressed about imagination:

Imagination is the Discovering Faculty, pre-eminently … It is that which feels & discovers what is, the REAL which we see not, which exists not for our senses… Mathematical science shows what is. It is the language of unseen relations between things… Imagination too shows what is … Hence she is or should be especially cultivated by the truly Scientific, those who wish to enter into the worlds around us!
Journal entry for 5 Jan 1841, Lovelace Papers, Bodleian Library, Oxford University, 175, folio 199

It is the REAL, not the computational that Lovelace emphasizes that should be strived for.

A chatbot reactivates an expired SIM card – or does it?

On leaving the Weston, I popped next door to Blackwell’s to buy a couple of books. Only one was in stock so when I returned home I went to make a purchase online, using Paypal. Using the payment facility required two-factor authentication, which meant a text message to the SIM card in my trusty mini phone. But, on switching it on, it greeted me with the message that the card was non-operational. It turned out it had expired because I had not used any chargeable services during the past 3 months.

I logged onto the Vodafone site and proceeded to enter into ‘dialogue’ with its chatbot, an experience I had managed to avoid previously. Somewhat sceptical, I commenced the mission to reactivate the card with the help of TOBi, Vodafone’s ‘friendly digital assistant’, which has already been around for 5 years. It is mentioned that the service is powered by IBM Watson and LivePerson, which are well-established systems using advanced machine-learning.

Pedigree is all very well, but what about the user experience? My conversation proceeded, or sometimes stumbled along, as follows (note that I lost some formatting in my copy and paste, so there may be some errors and/or omissions):

[14:50]
TOBi: I’m TOBi – Vodafone’s chatbot
Ask me anything, I’m here to help.

[14:51]
Paul: my number has expired, but I would like it back

TOBi:
Let’s see if we can get your number reactivated. First, I need to know what kind of customer you are?
Pay monthly
Pay as you go

Paul: Pay as you go

TOBi: To keep your Pay as you go number active, you need to use it for at least one chargeable activity – like sending a text or topping up – every 180 days.

If you don’t top up or use any credit for 90 days, we’ll send you a text explaining that your account will expire in the next 90 days.

To avoid this, simply use your phone for a chargeable activity after you receive the 90-day reminder text. This will let you carry on as normal with your current number.

However, if you don’t use your phone within that 90-day period, your mobile number will be disconnected and will no longer work.

If your Pay as you go number has been deactivated due to inactivity and you’d like to reactivate it, please speak to an adviser.

Read more in our Pay as you go terms and conditions

speak to an adviser

TOBi: Could you tell me your full name please?

Paul:

[14:52]
TOBi: To help save you some time when speaking to an adviser, I’ll complete some quick security checks with you first.

For this security check, I’ll need to take your phone number to send you a security code by text. I’ll launch a secure pop-up for you to enter your number now.

TOBi: Thanks, that’s the security part completed.

TOBi: Could you please tell me your phone number?

Paul:

[14:53]
TOBi: hello

TOBi: I hope you are doing well today

TOBi: How can I help you today ?

Paul: i wish to reactivate my expired number

[14:54]
TOBi: I will try my best for you but it is difficult but still I will try my best

TOBi: Please help me with the sim serial number on the back side of the sim card

Paul:

[14:55]
TOBi: Thank you let me check and try

[15:00]
TOBi: We have not heard from you in quite some time. Seems like we have lost connection. I’ll give it a minute more before I end this chat session.

[15:01]
Paul: Hello, have you reactivated the sim card?

TOBi: Just doing the process for you and it is almost 90% done

[15:02]
Paul: OK, thanks. I shall wait.

[15:05]
TOBi: You will be very very good to know tat I have raised the order and your number will activated with in 2 to 24 hours

TOBi: I hope it is sorted for you all

Is there any thing else I can do for you please ?

[15:06]
TOBi: I am glad that have been resolved

Paul: Thank you for your help. Much appreciated. Goodbye.

TOBi: I would request you to please end this chat by clicking on the X button on the top of the chat

Conversational Quirks

TOBi maintained a chirpy and seemingly cheerful disposition. The issue was correctly identified at the outset and after 15 minutes, it seemed it had been properly dealt with; we got there in the end. However, I found the process somewhat precarious and it was a quite disorientating experience. I felt uncertainty in several places, but whereas with a human service assistant I’d be confident in checking the status, here I was worried that I’d be misunderstood and then be back to square one. Altogether, there were several hiatuses which made the conversation feel like an obstacle course:

At 14:52, TOBi mentioned speaking to an advisor, but I wasn’t sure of the implication – whether it meant in the event of the bot not being able to handle the request at some stage down the line or whether I would be transferred imminently. There was no preparatory statement along the lines of, “a (human) advisor will deal with the query … “
The absence of the ‘secure’ pop-up and then the validation confirmation without my consciously doing anything was a bit unsettling and surely something that needs to be checked by technical staff. And only after passing the security check, was I asked for my phone number, which I would have thought would have been part of the check
Next (at 14:53), it was bizarre to be greeted afresh. At that point I had no idea about the status of my request, so I simply repeated it, echoing back the give terminology, “i wish to reactivate my expired number” (humans can learn too!)
From (14:54), the style of English became quite different, so perhaps at this stage I was dealing with a human? But the instructions were clear enough and so by 14:55 I had provided all the information needed.
But time was ticking by and these chat systems have quite strict timeouts, so at 15:00 I was threatened with a disconnection, with no reference to the status of my request. I wanted to check the status, but didn’t want to interrupt anything, so I just asked about the reactivation of the SIM and after a moment I got the response that it was nearly done. A few minutes later I got confirmation that it was done and I could conclude the chat.

And, as promised, the card was reactivated and working later that afternoon, which was a positive outcome (and a relief).

Observations

One of the disorientating aspects, possibly caused by my imagination, was my impression that in some places there was manual override that didn’t work well in concert with the basic chatbot workflow. Interleaving of processes requires some orchestration so that you don’t get disconnected whilst some useful process is happening … The chatbot seems to have been architected as essentially a sequential system, but this kind of troubleshooting is really a problem in parallel and distributed computing, especially if multiple agents (bot and humans) are involved.

I wonder to what extent the actions are being carried out by the bot and what by humans. I imagine that manual override is possible at any stage (as with a self-driving car). It probably occurred here as, surely, TOBi is not programmed to write, “You will be very very good to know tat I have raised the order and your number will activated with in 2 to 24 hours” (sic).
Or is that meant to be ultra-realistic human interaction, complete with typos?

As to the problem of SIM card expiring, the timescales have shortened – I originally bought the phone with the number more than 20 years ago and don’t recall any expiry at the time. Eventually, it was 12 months of inactivity. Now it has been reduced to 6 months, but I only received one reminder, in the middle, i.e. with 90 days remaining. This is a long notice period, easy to forget. It would be better to have received another reminder with 30 days to go.

Even with optimised machine-learning and having my problem solved, I somehow felt a lack of completion. Human communication is actually much more sophisticated and subtle in its interactions, with a remarkable amount being non-verbal and intuited. I am conscious that with bots, humans need to fit around the ‘thought processes’ of the machines – I found myself trying to work out what terminology to use, and wasn’t sure of it.

What about the staff that TOBi has replaced? Do they really have more interesting work to do? Do they still have jobs? Or is the situation like high street banks, which have reduced in number and where the role of many staff has become far narrower? Are human chatbot assistants like those who hover around to help people use ATMs and deposit points, answer a few general questions, before directing customers online? I.e. to adjust to the ways of the machines … ?

The subject of artificial intelligence is of ongoing interest in my research pages on mind.

Introducing MakeStaticSite for website deployment

Paul — Wed, 21 Sep 2022 14:53:20 +0000

(from a technical perspective)

Launch of MakeStaticSite.sh Website

November 2022: A website dedicated to MakeStaticSite is now available at https://makestaticsite.sh/. It provides the latest version of the software, released under AGPL, along with various documentation to help getting started.

Motivation

The process of developing this website has evolved considerably. On setting it up a few years ago in the conventional manner using WordPress, I duly familiarised myself with routines for maintaining security and improving performance on a hosted platform. However, there were several aspects that weren’t ideal: the live content was hosted remotely, albeit with backups; updating the site in situ, particularly for major redesigns, was inconvenient; and performance wasn’t always as good as expected. Observing that most of the content was not dynamic in nature, I resolved to develop the site locally and then, as a separate process, deliver it as a largely static site.

The rationale for generating static sites has been around a long-time; I first tackled this more than 20 years ago, when I needed to generate a CD ROM from a CGI/Perl multimedia website. Nowadays, the field is mature; a Netlify blog post (Netlify is a cloud-based service where many static outputs end up) gives many valid reasons, rooted in the need for more responsive work processes. Typically underpinned by a simple language called markdown, there has been a surge in solutions; hence, options for Jamstack abound. However, the processes tend to be heavily oriented to developers at the expense of content authors, issues I know first-hand from enabling web access to museum collections.

I wanted instead to support existing content management processes, i.e. to continue with creating and managing content in WordPress, and then ‘at the press of a button’ generate and publish a static copy online. There are already some solutions, including dedicated plugins such as Simply Static. Tim Nash, having observed increasingly complexity with the Jamstack options, has described how he used WP2Static via WP-CLI, the WordPress command-line interface, for a (relatively) simple and compact solution. But I hesitate over the use of a WordPress plugin to generate the static site. Apart from the ongoing issue of maintenance and compatibility with WP Core, plugins are not really necessary since a public-facing website should be de facto crawlable by a web spider and not require internal system information.

And so I turned to general-purpose tools I could run from the command line. Enter Wget, a ubiquitous utility for retrieving and packaging any number of web pages for subsequent viewing. I needed to configure it, add a few additional inputs, and then choose where to store the outputs. This led naturally to delving into (Bash) shell scripting to bring it all together and I continued the process, adding deployment options and invoking more commands, particularly WP-CLI to prep the WordPress output (and to include a version of the plugin, WP Static Search, revised to work offline). If you examine the source code of this page, you will see the result.

Pipeline

Shell scripting became a means to orchestrate the overall process, which is readily seen as a pipeline comprising a sequence of phases. For example, one phases runs wget for the initial capture, whilst another re-runs wget to more completely capture a site. In fact, many core web development processes can be incorporated, such as replacing certain internal-facing pages or sections with alternative content; and invoking command-line tools for validation and streamlining output. As of version 0.21, there are ten phases altogether; and command-line options allow you to control the start phase and end phase.

The flowchart below illustrates an emerging pipeline (for the latest perspective, please refer to pages on workflow and phases).

Setup

Configuration dialogue

The first step is to run the setup.sh script, which guides the user in an interactive dialogue to specify a series of options.

Initialisation

Configure

The system reads and processes configuration options

read runtime options
read configuration file options
prepare output layout

Prep the CMS

CMS-specific streamlining

Optionally use wp-cli to

ensure nice permalinks
remove query strings
remote shortlinks
replace with static search

(Currently WordPress-only, but modules could be developed for other CMS.)

Generate static site

Generate wget mirror (first run)

Run wget with the supplied configuration options

optionally incorporate further input files
optionally set up a zip files
store the mirror in its own directory with timestamp

Augment static site

Further processing with wget

Run wget with the supplied configuration options to fill in gaps

optionally crawl static files for further URLs
Filter the URLs to retrieve only assets (multimedia files, etc.)
Re-run wget on the additional URLs

Optimise static site

Further post-processing for deployment

Further post-wget processing

convert feeds to be XML files and modify references accordingly
Apply HTML Tidy to make more standards-compliant and use pretty-print (dependent on options)

An accessibility validator could be added here.

Create an offline zip archive

Create a custom zip based on the mirror

The mirror acts as the basis for a zip archive, ready for uploading with the site or other distribution channels.

Augment

Further additions

Optionally augment the static mirror with snippets, substituting chunks of HTML in particular places according to the use of snippet tags. These will augment the zip file also.
Additionally, allow the inclusion of further files that might not necessarily be static files.

Deploy

Finalise and deploy locally or remote

Optionally copy over additional

Optionally copy over additional files, which might be scripts to provide missing interactivity
Finally deploy on local or remote server using rsync

Extending the usefulness

MakeStaticSite is a prototype that was initially developed for individuals who are creating and maintaining websites on a personal computer. However, the question soon arose: how might a team of content authors and editors be supported? (The answer is: yes!)

For example, what if this site is to have multiple contributors? In this case, the most straightforward approach is to set up or continue hosting with a third-party hosting provider, in the usual way. This means those who are creating and managing content carry on as normal through the dashboard. The main change is to separate the editing environment from the public site, which can be achieved by restricting access by some means of web authentication. Then, run MakeStaticSite to deploy the static site to the public, whether on the server or remotely. This approach ensures continuity; the only additional consideration is how to initiate MakeStaticSite once changes have been made — from within the CMS or external to it?

An initial test has shown that once set up, the process of maintenance and deployment can be carried out entirely from within WordPress. This can be achieved by running the script under the terminal emulator WPTerm (MakeStaticSite has an option to run ‘unattended’, which makes default selections). This depends on appropriate support for MakeStaticSite on the server; for shared hosting, the versions of required components, particularly Wget, may be somewhat old and, of course, for any serious use with a third party, access to WPTerm should be properly secured as instructed. It’s preferable to keep the running of MakeStaticSite on a personal computer, an arrangement that has been used for fuengsin.org.

However, hosting with a third party means that the data is no longer immediately under our control; although a mixture of database replication and file synchronization can compensate, there’s some loss of independence. Perhaps a better solution lies in p2p networks, but that’s an area I have yet to explore …

For now, this tool serves my needs well and I hope it is helpful to others. Please refer to the MakeStaticSite project site for further information.

Welcome to the Relaunch of ptworld.net!

Paul — Tue, 20 Sep 2022 14:05:11 +0000

Hello. After months of preparation, I’m pleased to welcome you to a completely overhauled website, launched on 25 September 2022. The site covers various endeavours past and present, offering an unusual east-west approach to the intersection of society and technology. Faced with the increasing pervasiveness of digital systems encroaching ever deeper on our lives, I wish to apply my background in mathematics, computer science and religious studies to safeguarding and promoting human autonomy and independent living — in body, speech and mind.

Accordingly, this site, and these posts in particular, will be my primary vehicle for dissemination, to serve as a launchpad for projects, current ideas and news for reducing dis-ease and increasing ease. Much of it is interdisciplinary in nature, but the integration comes not from external connections, but wells up naturally as I seek to solve problems, often from first principles, drawing on existing knowledge.

The main goal is the creation of Sigala, a new kind of social networking service, based on the Buddha’s teachings on how to cultivate wholesome relationships. For this, a team will be needed. So, whilst this site is a solo effort at the outset, I hope it will gradually morph to a team site with multiple contributors.

Wytham Woods

The banner image, from a video still on a lush, rainy day, displays the following note:

In Wytham,
Exploring far and wide
inner and outer worlds,
for knowledge and vision
therein …

The view looks eastwards from Wytham Woods, a haven which I have frequently visited over many years for fresh air, exercise and reflection.

Today the woods are best-known as an ecological laboratory. However, it is said that many centuries ago, monastics on pilgrimage from Cirencester to Canterbury, when making their way along the ridge towards Oxford, would break into song on seeing the emergence of the city of Oxford and hence it came to be called the ‘Singing Way’. Not only Christian monastics have been in the woods, but more recently, in the 1970s, a Buddhist centre was established on the edge of the wood at Oakenholt, which witnessed historic gatherings of monastics from many traditions.

Many of the ideas expressed on this site arose in embryonic form whilst I was wandering through these woods. Some ideas, already existing, were distilled. Occasionally, I would stop mid-track to make some notes, and then move on.

Orientation

I’m hoping that the site’s structure and navigation makes it fairly straightforward to learn what it’s about, but here’s a quick overview.

About Background information covering the origins of this site and the domain name; my professional experience, mainly in higher education and non-profits; and, looking forward, my business intentions and how you can help.
Research Investigations usually start with a problem that we consider important for its impact in well-being, with quality of awareness a central concern. Drawing on many disciplines, one of the first tasks is to clarify the nature of the problem and, in particular, the definitions.
Projects The quest for a solution takes the theoretical analysis and extends it with experience of technological innovation gained from the ground up. For example, when we consider designs that address cognitive problems, we have the foundations of considerable hands-on knowledge of software and hardware used in the field of personalised learning. As evidenced in my portfolio, I’ve undertaken many bespoke projects, particularly at Oxford University, often addressing recurrent problems and requiring familiarity with relevant research themes.

Process

As a software developer tackling a well-known problem, I may re-invent the wheel and not very well! However, I sometimes come up with original code or provide a more complete solution. This is reflected in this site, where I have tried to embody the paradigm of autonomy and self-sufficiency in the creation of MakeStaticSite, a tool for generating and deploying static websites with the goal of improving performance and security.

Archival is a common task, epitomised in the WayBack machine, whilst some software developers already have a process for creating and deploying static sites, but I did not find a process that supported convenient authoring in WordPress together with other features I was looking for in static publication — such as the inclusion of a search facility, a more complete snapshot that took account of non-standard HTML attributes, support for HTML Tidy, and so on.

It seems a lot of effort just to get to the starting line. But now that process is in place, I hope blog content and project development will start to flow!