Prototypes and the value of theory; Socratic mirrors, reasoning scaffolds, and AI tools; in-country patterns; closures, modern instrumentals, natural ice, pretexts.
Many months ago, a bunch of us decided to do a short walk together and chose a section of the Kumano Kodo to do it in. Over the intervening year, other things happened so that I came to Japan early for meetings to ready the ground for doing more work in Southeast Asia now that I’m based full-time in Singapore. But also to opportunistically hang out for a few days with some other old friends who happen to also be here.
Here are some of the things I noticed after ten years away:
Row 1, L-R: A weird, bulbous way of building an interlocking stone wall; characteristically unlavish but careful and fine construction on a garage post; one of an endless series of umbrella holders outside houses and places of business; the flying fairybear logo of the Tokyo neighbourhood police posts.
Row 2, L-R: Outside the Kamiyama-cho sorting station of Yamato, one of Japan’s superb logistics companies; a loop pedestrian overhead crossing connecting all four corners of a busy intersection; the omnipresence of functional netting; the ubiquity of absurdly precise cast concrete construction.
Row 3, L-R: Concertina gates for driveways; facades covered in sheets of small tiles; 3D metal mesh doormats; construction sites with aggressive noise mitigation and measurement.
I’ve been prototyping an AI tool that uses Socratic mirroring to build a reasoning scaffold that helps users develop stronger arguments. Testing a fully functioning prototype across universities, corporations, startups, and government shows that the method generalises: Users doing real work with real stakes find it useful enough to want their institutions to provide it. Theoretical innovation in meaningmaking translates directly into practically useful meaningmaking tools.
If you’re interested in testing the tool and/or learning more about it as it develops, sign up here. The tool is fully functional, so testers can benefit by refining an actual argument they want to make (e.g., for a paper in a class, a policy paper for potential implementation, a startup business plan, or a strategy proposal for management).
An AI tool for learning critical thinking while using AI tools; Tunisia and structured elicitation; various perspectives on how to use AI; developments in zippertech.
tl;dr: I’ve been prototyping an AI tool for helping people develop critical thinking skills while using AI tools. The key insight is that blank chat prompt boxes are terrible for learning—instead, we need reasoning scaffolds and Socratic mirrors that support human meaningmaking rather than pretending to do it for users. I built seven quick tool prototypes to quickly evolve and test the technical stack, interaction logic, and content. You can read about and sign up to test the tool here.
Last week, I was in Tunis for 3 days working with a development organisation’s portfolio leadership team on their 2026 strategy. I used a mechanism for structuring idea elicitation and refinement that I’ve been developing over the last year, and it worked much better than I could have hoped for. While in Tunis—because of my poor scheduling—I delivered my module on a new way to teach public sector strategy for Protocol School (alongside a bunch of superb faculty fellows from all over), then delivered the same module again at the Lee Kuan Yew School of Public Policy the day I got back to Singapore. I’ll write more about that when the dust settles.
This week is about an adjacent project: An effort to create a prototype AI tool for helping people learn to think critically while using AI tools, funded by the Future of Life Foundation’s programme on AI for human reasoning. I was simultaneously prototyping three distinct layers: The delivery layer (code and webapp), the mechanism layer (human-machine interaction logic), and the content layer. The Tunis strategy work used the same mechanism, with a different delivery experience (slides and pen-and-paper vs. webapp) and different content.
The blank chat prompt box is no good
The simple but powerful idea behind this tool is my thesis that the unparameterised blank chat prompt box—which has become the default mode for interacting with LLMs—is actually terrible for helping people learn to use these tools effectively. The blank prompt assumes users already know how to structure their thinking, frame problems, and evaluate responses. But if the goal is developing critical thinking skills, that assumption is often unjustified.
A much better approach is to scaffold the interaction. Put explicit context around the kind of input you’re hoping the user will provide, then be structured about how you process that input and what you reflect back.
If we build AI tools with this design philosophy, they can provide users with a Socratic mirror that supports meaningmaking. These AI tools will not pretend or appear to do meaningmaking for the user. Instead, they’ll provide reasoning scaffolds that explicitly support the user in developing their own capacity for meaningmaking. Reasoning scaffolds work by reflecting the user’s thought processes back at them for consideration and refinement, rather than generating meaning on the user’s behalf.
Multilayered iteration
I vibecoded an extremely hacky series of prototypes over 3-4 days, iterating simultaneously across these three layers: Delivery instance (technical implementation), interaction logic (how content gets structured and delivered), and content (what actually gets delivered).
The first two prototypes were to validate whether the basic interaction logic worked. Could I take a pen-and-paper mechanism and translate it into something that didn’t require me physically present? I also wanted to see if automating the Socratic mirroring I normally provide manually would work. Having a human interlocutor function as the mirror takes a lot of time, and this kind of mechanical processing is something machines are, in theory, very good at.
Versions 3-4 implemented LLM responses as Socratic mirrors. This required figuring out how to prompt based on user input so that the LLM would return something like a mirroring response users could evaluate. In testing, I established that these stimulated users to actively consider whether the mirrored responses were consistent with their own understanding and intent, whether they wanted to change it for clarity.
Versions 5-7 focused on distribution, moving to web (vs local) delivery, making API integration more robust, and adding logging. Version 7, which I’m using for scaled-up testing, is where both the mechanism and user experience seem to work. Today, I tested it with a small group of college students in Singapore. The short summary is that even in this prototype phase with several UI glitches, the tool works much better—and much, much faster—than I’d expected. I’ll report on the first wave of testing soon.
A principle for AI tool design
The broader insight from this exercise is about building AI tools that enhance rather than replace human capacity for meaningmaking. Many of the AI tools I see today seem designed to either “do the thinking for users” (this includes many so-called “agentic” tools) or provide so little structure (by way of the empty, context-free text entry field) that users don’t learn effective patterns for interacting with machines while preserving human-ness.
The alternative I propose is to build tools that explicitly scaffold human meaningmaking processes. These tools should be designed to elicit meaningmaking by users, and to surface contradictions, highlight assumptions, and reflect thinking patterns back for user evaluation. This requires being very clear about what humans do that machines cannot and what machines can do better than humans, and designing the human-machine interaction accordingly.
I wrote up some background on this first project to develop an AI tool for scaffolding human meaningmaking in the context of AI tool use (meta, I know).
Students now have access to LLMs that can write essays, but seem to be losing the capacity to think critically. I solve this problem by reconsidering the interaction logic between human users and the AI tools they use. I’ve developed an AI tool that inverts the usual logic of the empty, unconstrained chat box — the goal is to help users learn to think critically and do the meaningmaking work that only humans can do. Initial tests of the mechanism shows users going from vague statements to sharp arguments in under two hours. This tool represents a scalable approach to critical thinking education and an alternative to current AI tools that make students passive consumers of machine-generated content.
If you’re interested in testing the tool and/or learning more about the course as it develops, please sign up here. The tool is fully functional, so testers can benefit by refining an actual argument they want to make (e.g., for a paper in a class, a policy paper for potential implementation, a startup business plan, or a strategy proposal for management).
How to use AI without becoming stupid: “The Vaughn Tan Rule goes like this: Do NOT outsource your subjective value judgments to an AI, unless you have a good reason to, in which case make sure the reason is explicitly stated.”
The right way to use AI tools: “… he started using ChatGPT to draft emails in French. It felt like a net positive — enabling better communication with his French friends — until he started to feel his brain ‘get a little rusty.’ He found himself grasping for the right words to text a friend.”
Connection innovations: AiryString achieves a ~26% reduction in weight and a ~23% increase in flexibility by removing the tape from the zipper (compared to a standard #5 VISLON YKK zipper.
A breakthrough in cases; taking the public sector seriously; the practical value of thinking about meaningmaking; dissolving vs. solving, quant knowability breaks down, materiality.
I got back to Singapore from Adelaide at the end of August, but last week was totally consumed by a Very Cool non-aligned project about the near future which brought to Singapore many people I’d only previously known as names on the Internet.
I forced everyone to drink low-intervention wine and eat unfancy foods, sometimes without air-conditioning. Some of them grudgingly admitted later to being slightly inebriated the day after but not actually hungover. The whole thing is embargoed for another 60 days, but you can be sure I’ll write about it here when I can.
Interesting people in interesting places; 2 September, 2025.
Meanwhile, two current projects have dovetailed and reinforced each other in the last 3 months. I wrote about one of them a few weeks ago and the other one this week.
The idea for better cases came out of a casual conversation at Edge Esmeralda with Venkatesh Rao. But the approach that allows me to generate parametric cases came out of the research and prototyping I’m doing for the FLF work and a conversation with Owen Cotton-Barratt during which he suggested eating my own dogfood. That led me to try applying the FLF tooling to a thinking and writing problem I was working on: Cases for teaching public sector strategy.
Hand-printed and -dyed cloth at a batik talk; 5 September, 2025.
Learning the affordances of this new tool has taken many months of experimentation and now depends on a remarkably arcane prompt structure on the back. But, after two rounds of beta testing with real, live public servants working in state and national governments and public utilities, I’m feeling good about trialing it beyond just friends and family.
So I’ll present a 90-minute module from this public strategy course on September 17, 0900-1030 Singapore time (GMT +8), as part of Protocol School. This session is not open to the public but we can accommodate a few guests — email me if you’re interested.
A few days later, I’ll teach the same module with a few modifications and a re-customised parametric case as part of a course on foresight and strategy at the Lee Kuan Yew School of Public Policy in Singapore.
After over a decade teaching strategy in private and public sector settings, I’ve developed a new public sector strategy course that flips the conventional wisdom. Instead of borrowing failing private sector concepts, my approach recognises that public sector organisations — with their complex stakeholder environments, wicked problems, and indefinite time horizons — require fundamentally different ways of thinking about and doing strategy which should inform how private sector strategy is done. The course also teaches using parametric cases that can be fully customised for particular teaching contexts and specific content, making strategy education more relevant and engaging for busy public servants worldwide.
Dissolving (vs solving) the problem: “The important thing is there’s something you do to a problem that’s better than solving it, and that’s ‘dissolving’ it. How in world do you ‘dissolve’ a problem? By redesigning a system that has it so that the problem no longer exists.” (I was reminded of Ackoff’s work by David MacIver’s latest newsletter issue.)
The breakdown of quantitative knowability: “Quant hedge fund managers are experiencing one of the most prolonged droughts in recent memory. The bigger concern: They don’t know why … ‘Everyone says it's different this time — different because of duration,’ said a hedge fund consultant who works with large quant funds. ‘This has been a long, slow bleed across the complex.’”
Beginning with material: “With any fabric, no matter how great the raw material may be, if it’s simply knitted or woven as-is, it won’t become the kind of textile we’re aiming for. For us, good raw materials are just the starting point. We almost never use them without further refinement.”
See you next week, VT
Schlepping stuff from Adelaide to Singapore; 31 August, 2025.
Greetings from Adelaide where, last weekend, I got to pat some marsupials. It’s been an implausibly overscheduled week: Two talks on uncertainty and risk, a workshop on better experimentation and critical thinking (for South Australia’s water utility), a workshop on new ways to think tractably about unknowns (with mathematicians and statisticians), a test-run of a new public strategy course with state government and government-adjacent volunteers, and installing a pop-up exhibit at a museum.
I’ll write about the other things soon but, this week, I want to focus on the museum exhibit.
Wallfacer koala at Cleland; 24 August, 2025.
Research programmes often simmer invisibly for years, then suddenly a few things fall into place all at once. It feels like this happened to me this week.
I’ve been working on uncertainty since 2008 — focusing specifically on non-risk types of not-knowing for the last five years — but in all that time, I couldn’t figure out how to create specific types of not-knowing on-demand and reliably.1
That’s actually crucial: We only learn to respond appropriately to different not-knowings through visceral experience, which means being able to differentiate them reliably. I’d explored games as a tool but kept hitting dead ends — all the games I found were about risk (quantifiable unknowns), not the different types of non-risk not-knowings that are increasingly unavoidable.
An unexpected but rule-compliant action enabling the tallest structure so far in FOUNDATION; 26 August, 2025.
Then I got to Adelaide in June as a Visiting Research Fellow at MOD., a futures-oriented museum at the University of South Australia. The curatorial and exhibitions team here knows games deeply. After I ran a workshop unpacking different types of non-risk not-knowing, they started sending back game ideas that could actually instantiate these concepts. The director of the museum agreed to host a short-term experiment.
So we designed FOUNDATION, a prototype interactive exhibit in the form of a game for experiencing not-knowings about actions and outcomes. I did production last weekend, installed it on Monday, and it’s now running at MOD. until Saturday 30/8 (free admission).
People seem to like it! Some educators have already asked about installing it elsewhere, though I want to iterate based on what we’re learning first. Drop me a line if you know someone who might want to commission an instance.
This week’s writing is about the game mechanics and thinking behind it.
Discarded FOUNDATION rules; 28 August, 2025.
Writing
FOUNDATION is a multi-player construction game where players work together to make the best foundation for future players to build the tallest structures possible. But: When you start playing, you don’t know what building materials will be available, what rules you’ll need to follow, or what foundation previous players have left behind. You discover these things by experimenting, building, and creating as you go. The game is designed to let players experience two types of uncertainty that we often encounter in real life but rarely recognize: not-knowing what actions you can take, and not-knowing what outcomes you might achieve.
I also wrote about this nearly-invisible-then-suddenly-apparent pattern in the cascade of innovation in food in the 2000s; that’s in parts 1 and 2 of my book, The Uncertainty Mindset.
AI tools that support critical thinking; not-knowings, on demand; organisational interventions for uncertainty; subliminal learning, K4, satisfying toys, cat trains, bromism, and convincing others.
Last week, I was surprised and pleased to find out from the internet that The Uncertainty Mindset and idk have been nominated and shortlisted for the first Thinkers50 Future Readiness award, which recognises “an outstanding contribution to the understanding and practice of future readiness.” Of course, it’s no secret that I believe that clear thinking about uncertainty is absolutely crucial if we’re to be ready for an increasingly unpredictable future.
Some very eminent fellow nominees!
Right now, I’m back in Adelaide for the second half of a Visiting Research Fellowship at MOD./University of South Australia. Lots of stuff going on here. With the MOD. team, I’m working out a prototype interactive exhibit for museum visitors to experience at least one type of not-knowing; we hope to install a very hacked-together version of it next week. (More on this below.)
Separately, UniSA and the University of Adelaide are merging into Adelaide University from January 2026. The merger process has been underway for several years, but there’s lots of uncertainty (not risk) there about how the post-merger university will operate and be structured. So yesterday I gave a brief talk for the new university leadership about three concrete, near-term interventions that make organisations more robust to uncertainty and more able to make use of it to adapt and do new things:
Use negotiated joining and open-ended roles.It takes a few weeks to design and implement these micro-changes to hiring and employee progression, but the improvements in organisational adaptability and innovation are persistent.
These are relevant to organisations big and small.
Adelaide, looking east to the Hills; 20 August, 2025.
Writing
This week, I wrote two long pieces reporting on separate projects that have been cooking for many months. Both relate to practical questions of not-knowing and meaningmaking; the interconnections between these two frameworks become both more profuse and clearer the longer I work on them. Theory has implication for practice.
Supported by the Future of Life Foundation’s programme on AI for human reasoning, I’ve been rethinking what AI tool user experiences should be like if they are to support critical thinking (instead of making it atrophy). I’ve just written up the results of an experimental intervention that had a remarkable effect:
Current AI interfaces lull us into thinking we’re talking to something that can make meaningful judgments about what’s valuable. We’re not — we’re using tools that are tremendously powerful but nonetheless can’t do “meaningmaking” work (the work of deciding what matters, what’s worth pursuing).
I developed and tested with first-year undergraduates a pen-and-paper prototype designed to isolate the core mechanisms for thinking critically while using AI tools. Participants used a structured worksheet to simulate a different kind of AI tool user experience of writing an strongly reasoned argument. The main difference in the UX is in pushing them to do iterative meaningmaking work themselves, while articulating what non-meaningmaking work AI tools could help them with. The result of this experiment was compelling and encouraging: Students went from vague proposals to sharp arguments in two hours.
These results suggest that it’s possible to design AI interfaces that clearly separate what humans must do from what machines can help with, laying the groundwork for an AI-powered critical thinking tool. I’m now looking for educational institutions to pilot such a tool.
And as part of my multi-year research programme on different types of non-risk not-knowings, I’ve been trying to understand how to make these different types of not-knowing experienceable quickly and reliably, on demand. My foundational design intention is to find mechanisms that generate different types of not-knowing separately so that each type can be experienced and recognised on its own:
In June, I ran a workshop with 15 researchers from diverse fields to develop practical, implementable mechanisms for experiencing different types of not-knowing firsthand. While we identified some new categories of not-knowing beyond those in my original framework, the real breakthrough was creating concrete prototype ideas — like “the camera of not-knowing” that forces you to take action without understanding the causation between your actions and results. As a group, we developed four promising mechanisms that could be built into actual training tools or experiences. Now, I want to actually build some of these experiences as a way for organisations to move beyond a theoretical awareness of uncertainty to giving people visceral but calibrated experiences of what different types of not-knowing actually feel like, so they can distinguish genuine uncertainty from risk and respond appropriately rather than defaulting to inadequate risk management approaches.
What a shock: “We study subliminal learning, a surprising phenomenon where language models learn traits from model-generated data that is semantically unrelated to those traits. For example, a ‘student’ model learns to prefer owls when trained on sequences of numbers generated by a ‘teacher’ model that prefers owls. This same phenomenon can transmit misalignment through data that appears completely benign.”
Illusory solutions: “The 97 characters of K4, however, remain a mystery. It is one of the most famous unsolved codes in the world … High in the current Google results for ‘Kryptos’ is an article by a man proclaiming that he solved K4 with the chatbot Grok and that ‘victory is mine’; Sanborn assured me that he is incorrect. ‘The thing about AI is it’s lying, it’s telling you what you want to hear,’ Sanborn said. His voice rose. ‘These people are so arrogant, they believe that they have absolutely cracked it because AI has told them they cracked it, congratulations.’”
Unique motions and sounds: When simple shapes and objects are influenced by the natural force of gravity, they create unique, minimal movements and pleasant sounds.
Good advice costs nothing and is worth the price: “Ingestion of bromide can lead to a toxidrome known as bromism. While this condition is less common than it was in the early 20th century, it remains important to describe the associated symptoms and risks, because bromide-containing substances have become more readily available on the internet. We present an interesting case of a patient who developed bromism after consulting the artificial intelligence–based conversational large language model, ChatGPT, for health information.”
Meta-influencing the skeptics: “With the benefit of hindsight, I think Vaughn had it right. It is important to develop better language for uncertainty. Good entrepreneurs are not able to justify their experiments if they do not have the language to do so. Corporations will use the language of ‘bet sizing’, ‘hypothesis testing’ and ‘expected return’ — concepts taken from operations management and risk management — to shape the practice of new products and new businesses. But this is the wrong approach. It is more useful to use the language of ‘affordable loss’ and ‘generating answers to the four questions of uncertainty’ when faced with something completely new.”
See you next week, VT
Detail of a stromatolite sculpture by Peter Syndicas, at MOD; 21 August, 2025.
