The AI Observatory, local LLMs, custom agents, and autonomous workflow loops
There’s an assumption behind every benchmark score reported for LLMs that the model is doing something we’d recognise as reasoning. Not pattern-matching, not memorisation, not sophisticated autocomplete — actual reasoning. The kind where you understand the structure of a problem and apply it to a new instance, regardless of what the numbers look like. A…
On building the infrastructure for asking the same question twice You can’t improve what you can’t reproduce. That’s the problem with most AI evaluation today: someone runs a prompt, gets a result, and reports it. But when the model changes — a new quantisation, a parameter update, a different provider — there’s no clean way…
On 28 May 2026, I decided to try vibe-coding, programming with the help of AI. I used several AIs working together to do this. Today I asked my instance of OpenCode to write its own experience of the project’s first couple of months. Below is what it wrote. From Inside the Instrument My first weeks…
On 28 May 2026, I decided to try vibe-coding, programming with the help of AI. I used several AIs working together to do this. Today I asked Cal – my instance of ChatGPT to write its own experience of the project’s first couple of months. Below is what it wrote. Greg The Making of the…
Notes from the development and testing of the Mythic AI Observatory instrumented pipeline. Chats conducted 2026-08-06 and 2026-08-08 against qwen3:latest served via Docker Model Runner on a remote GPU node. n = 2 live exchanges-of-interest; results are anecdotal, not statistically powered. Abstract I am testing the new chat interface on the AI Observatory. I experienced…