Sorry for the lack of news recently — I’ve been tinkering around with a spanner in the underbelly of the great steampunk machine which is the JC. The website’s performance — how quickly it serves pages — had fallen off a cliff. It had even started refusing to serve pages at all. The more I looked into it the more I realised I had to do something. The problem turned out to be twofold:

Bots

Firstly, bots. There are a lot of random webscraping devices roaming the dark netherworlds of the Grid that just incessantly hit unsuspecting websites. They have always done this, but it is getting worse. There are good webscrapers — Google does this so that its search engines work — sort of bad ones — all the AI engines just harvest text for training material, which is kind of annoying — IP fundamentalists get very wounded about it — but it is also nice to think my bloviations might form part of the forthcoming memeplex forging a contingent path to our future in the digital simulacrum — and baaaaad ones — these come from China, mostly, and Russia, and God knows what they want with the JC but, until recently, they were making up about 90% of all traffic and slowing down everything for everyone else.

No more. Good webhosts can help with this, and JC now has measures in place to throttle these bots. One is just blocking traffic from China altogether. Sorry Chinese ISDA fans!

This Substack is reader-supported. To receive new posts and support my work, consider becoming a free or paid subscriber.

Moi

Secondly, to put not too fine a point on it, me.

I have designed and built the JC in a decidedly evolutionary way. I got stuck with certain design decisions I made — or didn’t make — early on and that was that. It is like the famous XKCD cartoon:

This is a common feature of all big organisations of any longevity. At the heart of every investment bank, be assured there will be some Wang server kicking out ASCII code on an orange screen that is absolutely fundamental to the risk management systems, and which no-one dares remove for fear of unintended consequences. In smaller ways it is true of the JC. As I’ve gone along I have configured things in certain ways that seemed sensible at the time, but in hindsight not so much.

The nature of evolutionary adaptation is that ostensibly good decisions get sedimented into the architecture, overlaid with more and more stuff, and it gets harder and harder to go back and fix them because of all the knock-on consequences. Usually, it is not until much later that ostensibly good decisions reveal themselves to have been bad decisions.

Some were good decisions, once: whether a decision is good or bad is situational — we tend to forget this: hindsight conquers all. But unintended knock-on effects aren’t always obvious. They depend on how the future unfurls. (That figure of speech is wrong, by the way: it implies the future is already there waiting for is, like a rolled-up flag, and just needs to be revealed. But the future is not like that: it needs to be invented).

I have a piece upcoming about non-catastrophic failure modes in complex systems — “systems glitches” — that can lie latent in a system for a long time, and the longer they lie, the more they are normalised and start to look like normal functioning. The Post Office Horizon debacle is a good example. So, I think, is the Healthcare Serial Murder phenomenon. But there was one right under my nose.

This is also what happens when non-experts dabble. I am, first and foremost, a windbag: I am no web developer. And I made the choice a long time ago to use the MediaWiki platform, which is really good at being a knowhow wiki, but not very good at user rights management, founded as it is on the premise that information should be free, and the assumption that lots of people would be contributing and not just one.

I “solved” the problem of rights management by creating two wikis, and having them talk to each other. The premium wiki pulls all the free content from the main one, and then overlays it with extra juicy stuff. (This is mainly of interest to people involved in legal negotiation — especially ISDA — so if you are here for the Otto Büchstein’s cod Shakespearean melodrama fear not, you are not missing much. But if you want even more about Indemnifiable Taxes then, boy oh boy, is the Premium subscription for you!)

Anyway this two-wiki approach — which as far as I can tell, I invented by myself — has created some performance issues. Because requests to view one page creates all kinds of demands to load content from the other wiki. Because of the complex interplay of transcluded templates.

This is much more interesting than it sounds.

Templating

The thing that first sold me on Mediawiki is its really cool templating feature. Over my life parents, siblings, teachers, lecturers, bosses, children and most long-sufferingly, my spouse have railed and bemoaned my impulsiveness, lack of attention to detail, goldfish-like attention span, general slapdashery and a tendency to get easily carried away. (Also fecklessness and verbosity but they’re not important right now). I tend to be all over the place, as anyone who has tried to arrange a meeting with me more than ten minutes into the future will know.

Apparently, by the way, this is a classic trait of ENTPs. We also tend to build self-organising systems around ourselves and get into set routines, because force of habit is a great way of remembering to do things.

THERE I GO GETTING DISTRACTED AGAIN.

Mediawiki has this system of dynamic templating that is brilliant for building self-organising systems. I got really into them. They are what makes the site so apparently organised and so content-rich, but if you get too carried away with them they can become quite an albatross.

I got quite carried away.

Templates are essentially reusable gobbets of text. There is a different area on the wiki called the Template namespace where you can create little slugs of text, name them, save them down and then “call” them in the main pages of the wiki by writing their names inside {{double curly brackets}}.

At first blush this is just a time and labour saver: if I create a template called “QBF” with the text string, “the quick brown fox jumps over the lazy dog”, then I can “call” that text string on any page in the wiki by typing “{{QBF}}” and saving the page.

So if I type:

“my mother called me yesterday to tell me {{QBF}}.”

That would render as:

“my mother called me yesterday to tell me the quick brown fox jumps over the lazy dog.”

This is called “transcluding”. The term was coined by Ted Nelson in his 1982 book Literary Machines to describe the idea of a document including portions of another document by reference, rather than by copying the text.

Transcluding probably seems unremarkable. But it does two things: it saves keystrokes and it ensures consistency. Every time you get the same words, the same capitalisation, the same form: you can be confident it will always be “jumps” and not “jumped”. It saves you having to think about it. It compensates for a lack of attention to detail. It dramatically reduces faff. I have a template {{otto}}, for example, that returns the text “Otto Büchstein” because it is a pain in the arse remembering how to put the umlaut over the ü. And as all metal fans know, umlauts are a matter of, um, grave importance.

But there is more. Templates can have variables in them. Variables are designated by triple curly brackets. So I could set my {{QBF}} template up as follows:

“the quick brown {{{1}}} jumps over the lazy {{{2}}}”

Now if I “call” {{QBF|orang-utan|moon}} it will render as:

the quick brown orang-utan jumps over the lazy moon

I can quickly create a magical universe of all kinds of wonderful quick brown things jumping over lazy things. I can add more variables for the adjectives! There is a world of possibilities! I could rewrite the template as

The {{{1}}} {{{2}}} {{{3}}} {{{4}}} over the {{{5}}} {{{6}}}

But of course, at some point you might as well just not use a template. But as you can see, Mediawiki here is functioning as a symbol processor. It is a sort of Turing Machine.

Now, I dare say even that seems unremarkable. Get your kicks, JC!

But here is the thing: you can put templates within templates: you can have templates calling templates calling templates. So there is multilayered dimension here.

And you can put those variables inside template calls. This means you can use variables to tell the outer template which inner template to call.

So if my outer template calls a template (with double curly brackets) and I put a variable (triple curly brackets) in that inner template’s name then I can specify which inner template I call by the variable I put in the outer template call. This is conditional logic. We really are talking about Turing Machine style processing here.

Let’s say my template “{{clause}}” calls the text “{{isda {{{1}}}}}”.

If I type {{clause|1}}, the template will assemble {{isda 1}} — let’s say, that is the text of Clause 1 of the ISDA Master Agreement — and will return the text of that template. If I type {{clause|2}} it will assemble {{isda 2}} — Clause 2 of the ISDA — and will render that clause. If I have set up templates for all the clauses in the ISDA Master Agreement — and I have — then I have created a very easy way of pulling any clause I want on any page. The variables in the “outermost” template control which templates I want to render on the page.

This is very, very powerful. It means you can, basically, programme the wiki. There is essentially no limit to how complicated you could make this. This is what I did.

For example, the “owners manual” template, {{nman}} has only three variables (agreement, edition, clause) but calls twenty-two different inner templates, and some of those are dynamic and call their own inner templates. A single call, such as as {{nman|isda|2002|2(a)}}, may return as many as seventy or eighty subtemplates:

All of the components of this page are separate calls. The out template {{nman}} calls twenty-two different inner templates.

It all looks pretty good on the outsider — well, it does to me — and it makes creating new pages really easy: I just type {{nman|agreement type|edition|clause number}} and the whole page appears, with all the contextual material surrounding it, links to podcasts, further reading and so on — but to do all this the inner Turing Machine — Mediawiki as a symbol processing engine — is having to do an awful lot of work.

Now computers are good at accurately and quickly doing an awful lot of work, as we know. But there are good ways of doing a lot of work, and bad ways, and plainly my cack-handed system design over the generations was a stupefyingly bad way.

If that were not enough, when I set up the premium JC a couple of years ago, I created this barmy double wiki structure, so now there are a set of complicated “interwiki” calls between the main wiki and the premium wiki. Everything drives off a single page with a single call on it on the main wiki. When that call gets to the premium wiki, the underlying templates are subtly different and call on different templates, add different things in, thereby giving you lovely premium customers all that super extra content that you know and love.

But it also has to do something else: when it has imported wiki-linked text from the main site, it has to repoint those links to the equivalent pages in the premium wiki — otherwise, every time a user clicked on an imported link the user would be sent back to the main wiki.

This started causing problems

This was all working, but it was getting increasingly slow and unmanageable. For one thing, it was really hard to remember, or figure out, how the templates worked, and updating them was getting really hard — for example, when I started including links to podcast recordings on three different platforms.

And all of this was creating quite a lot of stress on the server. The two wikis were repeatedly pinging each other with API calls (whatever they are). All I could see was the bad performance — pages would hang or just take ages to load — and no obvious way to fix it.

It looked like, after 15 years of evolutionary decision-making, I had run out of road. Had I reached the limit of what I could achieve, given my decade-and-a-half breadcrumb trail of bad decisions.

There was also “key person risk”. If I got hit by a bus, it would be incredibly hard for anyone — even a Mediawiki specialist, and there aren’t many of those — to untangle what the hell I had done.

The JC had become a phantasmagoric, four-dimensional Heath Robinson machine. It was expiring under its own weight.

Enter the chatbot

And this is where Claude comes in. It could, I suppose, have been any large language model, but I have used Claude — especially Claude Code — a version that you install in a terminal environment which can directly see all the code you want to work on — and which you pay directly for in tokens.

I couldn’t ask a human mediawiki developer this stuff for fear of watching her pour gasoline on herself and set herself on fire. A virtual mediawiki developer might be less, er, volatile.

But are LLMs any good at developing Mediawiki sites? Well, it’s all relative, and Claude’s benchmark here is me, but my answer to this question is an unequivocal yes. Claude has been a game-changer.

Firstly I downloaded both sql databases, and the entire codebase for both wikis — this is gigabytes of information — and said, more or less, “help”.

Claude thought about it for a while then came up with nine suggestions, in order of bang-for-buck and complexity, that would help.

Simple things like caching webpages and automating links and audit between the the two sites. I am slowly working through them, but little by little the site’s performance has improved. You should notice that pages — especially on the premium site — load a lot quicker. Hope so!

The next phase is to do some actual content pruning: there are a number of offshoots, bright ideas, abandoned projects that I never got round to killing off. The total number of articles on the site should start to drop in the next couple of weeks.

Fear not, {{otto}}’s complete works are safe.

(See what I did there?)

The JC’s difference engine

Claude has helped with another component: “diffs”.

For a while there has been a feature comparing equivalent clauses in different editions of the same agreement — the idea being that it is useful to see how, say, Section 6(e) of the ISDA has evolved since the Children of the Woods thought of it all those untold generations ago. Ok: not the actual Children of the Woods — I didn’t go back as far as the 1985 Code — there are Burmese Junglers who would sooner give up than try to use that — but at least to the age of the First Men, those pioneers from the city of Salomon, who fired the One Agreement in the shadow of the Iron Mount.

I set up two-way comparisons between the three agreement types using Mediawiki’s inbuilt code comparison. Even with transcluded templates this was quite labour-intensive and very manual, for reasons I won’t bore you with, and they kept breaking and, every time the wiki updated, disappearing.

Enter Claude: he came up with some javascript that does it so much better and quicker — giving you a lightning quick, genuine redline, and, with the push of a button you can toggle between side-by side comparisons, and backwards and forwards comparisons.

I call it the JC’s “Difference Engine”. Cool, huh?

What is so surprising is how quickly you can create, essentially, a deltaview programme, that works better than an application you might pay hundreds of pounds a year for.

Doubtless there will be more improvements and enhancements to come as I think what else I can get Claude to do to help.

This all has cost me about £9 in LLM tokens so far. Seems to me like a good investment.

AI use case yes — but business case?

I expect this will be the enduring, and benign, use-case for AI. For now, we are in a new “wild frontier” where technology has escaped its garden and got into the hands of ordinary people, who can use it freely, creatively and really with minimal cost to improve their own lives, businesses (or make baby Trump videos — get your kicks). We normies don’t have the same outlook, much less agenda, as techbros, and it may well be (briefly) liberating for us to be able to commission technology programming to do what we think would be neat, rather than what a techbro thinks I think would be neat: they are quite different things.

History says this period of blissful frontier-like anarcho-syndicalism won’t last: rent-seekers, corporates, governments and systematisers will be along shortly to monopolise and monetise this opportunity, degrading the possibilities it offers an untethered imagination — so make the most of it while you have it.

Will it make the human race redundant? Hardly. Just as every technology before it, it extends and widens the range of things we can plausibly do. It will make possible things that would simply have been uneconomic, or infeasible, just too plain dreary to bother doing before. These new things will all create their own new opportunities, as they ever have done.

Playlist

Well, it had to be either seventies TV or — TRON. And with the magnificent news that not only is there a Tron sequel in the offing — here’s the JC’s review of the last one — but that Trent Reznor and the Nine Inch Nails have done the soundtrack and it is AWESOME. So without further ado here is - TRON ARES: ORIGINAL MOTION PICTURE SOUNDTRACK.

(For some reason this is not rendering in Chrome Browser. I hope you can see it. If you can’t , try clicking for the Spotify album here.)

Content should start to resume as I spend less time on on database reconfiguration. Happy autumn, folks!

JC

Thanks for reading! This post is public so feel free to share it.

This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit jollycontrarian.substack.com/subscribe