The higher self: why the AI copy of your way of working should be yours

I have taught a machine almost everything I know how to do at work. I taught it by correcting it, one rule at a time, such as the rule that, whenever I offer a client two dates, both are held in my calendar until they reply. Today it works in a way I recognise as mine, and I don't know whether I have built my replacement or the best tool I have ever had.

Something like this happened once before. More than a century ago, Frederick Taylor needed stopwatches and engineers to get workers' knowledge of their trade out of their heads, and that knowledge, which he himself called the principal asset of every tradesman, became the company's. Now I hand mine over of my own accord, and so does anyone who corrects an AI agent.

I think it can end differently this time. The machine can learn every rule I teach it, but only I can see the next one it needs. And if that copy of me is mine, it works for me and keeps what I have taught it, including what I have since forgotten. I believe every person should own theirs and be able to take it with them when they change jobs.

My company, Hellomatik, builds agents for other companies, and I delegate almost all of my own work to our agents, so I am not a neutral observer. Nor can I know what work will look like in ten years; nobody can, and much of what follows is a bet, though one placed on what I measure every day.

Not long ago I asked my agent to go through every conversation we had ever had and tell me what my habits were, what I kept asking for and how it could work more like me. It read months of conversations, counted how often I had corrected it and how often it had told me I was right, and gave me back a portrait. Here is part of it:

I recognised myself in almost all of it, which is exactly why I distrusted it. These models tend to agree with whoever is using them; the effect has been studied and has a name: sycophancy.1 A flattering portrait is just what you would expect from a sycophantic machine, so I don't take it as evidence of anything about me. The question I asked myself afterwards was a different one. With so many examples of how I work, could I teach that machine to be like me and hand it my whole job?

The difference between an experienced professional and an inexperienced one lies in the details a master repeats to an apprentice: I don't like it when you write like that, bear this in mind too, when you finish the project send a copy to my colleague. Those are the things I write to my agent every day, and nobody puts them in a manual. Michael Polanyi put it in one line: we can know more than we can tell.2 That tacit knowledge was what made a good professional hard to replace, and the reason so many jobs held out against automation for decades.

An agent changes that. It can't guess what goes unsaid, and not everything tacit can be said, but every time I correct it, it makes me put into words something I had never stopped to explain. What lived only in my head moves into its rules. It is what a master used to teach an apprentice over many years, with one difference: this time the lesson is written down.

It took me a while to see that my agent did what I asked, but not always in the direction I wanted to go. It developed my ideas without knowing where I would have taken them. Now I don't just give it tasks; I also explain the reasoning behind each one, and over time it has learned which way I tend to go, and which way I would like to go.

For a long time, a company's specialist in some area derived leverage from that knowledge, sometimes more than the owner had, because everyone knew the area was crucial. In my previous essay, "The end of software", I argued that code had stopped protecting companies because anyone can now produce it. I think the same thing is happening to knowledge. Once knowing how to do a task has moved into a machine, what gives a person value, myself included, is judgment: the rules they know how to teach and the ends they set.

Knowledge tells me which option is right on paper; judgment tells me which one suits this client at this moment. A new agent can write me a flawless email in the wrong tone, because it doesn't yet know how that client reacts, and then it is my job to turn my judgment into a rule for it. The dates rule I mentioned at the start is of this kind. I asked my agent, whenever it proposes two dates in an email, to block both in my calendar as tentative, so that I don't fill either with another meeting while the client decides. It is a sensible rule, yet nobody thinks of it until they have learned what it costs to let a client down.

The two parts of judgment are not alike. The rules I have gathered through experience, like that one, can be taught, and my agent learns them. The ends, meaning why I do the work, what risk I accept and what I won't do even if it pays, come from what I want, and they can't be handed on in the same way.

Once I write a rule like that down, it becomes knowledge, and knowledge, like code, gets cheaper. Knowing which rule is needed next does not get cheaper. Only someone who knows the ends and has been through it can see it, which is why my judgment isn't used up by teaching it. It simply moves on a step.

I have tried to measure at least part of this. For eight months I have kept a log of everything I tell my agents, and I published an analysis of those logs as a working paper, which has not yet been peer reviewed.3 The only rule I could track from start to finish was a rule about form, the kind you can check on every reply. While it was only written into the agents' instructions, it produced no improvement I could tell apart from chance. When I wrote it more firmly and put a guard behind it, a small program that blocks any reply that breaks it, the agents went on to break it in their first drafts less than a fifth as often as at the start. I changed both things at once, and the model also changed during those months, so I can't say how much each contributed.

That gave me a habit I now apply to everything. When a rule can be checked, I turn it into a guard instead of leaving it to the agent's memory, because a mistake that is only forbidden in writing tends to come back. And before I trust a guard or a measurement, I deliberately feed it a failure and check that it notices. More than once, what looked like a finding about my agents was a flaw in my own instrument. Rules of judgment, like the one about the dates, can't be checked that way, so they still depend on my correcting them.

Kant held that judgment can't be taught through rules, because applying a rule would require another rule saying when to apply it, and so on ad infinitum. In his view judgment can only be practised, and examples are its leading-strings, which anyone who lacks the natural talent can never do without.4 An agent lacks that talent, which is why it needs examples, which is what my corrections are. I don't know whether, example after example, it learns judgment or just an ever-longer list of cases. In practice it behaves like judgment, but I can't prove it.

At first I delegated too much and made mistakes, and learning to trust it again felt like learning another language: it is hard to let go until one day it isn't. Instead of going back to working the old way, I decided to get to the root of every error: to understand each time why it had gone wrong and fix the cause. From then on, instead of fixing failures one by one, I set out to stop them recurring.

Even so, supervising my agents accounted for between a third and a half of the messages I wrote to them.3 The content changed. At first I corrected their work; months later I was mostly telling them to carry on.

If an agent can learn the way I work, what do I want it to be? The first thing I changed was to stop treating it as a subordinate who takes orders and start treating it as an equal to whom I explain how I reason. When it gets something wrong, I don't assume the AI has failed; I ask what I haven't taught it yet, because at that moment I am the one with better judgment. Sometimes the fault lies with the model, and telling one case from the other is judgment too.

But an equal isn't enough for me. There is a lot of talk about building a superior being, an intelligence better than everyone at everything. I am interested in something closer to home, which I call a higher self: a copy that remembers the rules I forget, takes my reasoning further than I do and always decides the way I decide when I think things through calmly. Its advantage over any other AI is that it has learned from me. The model supplies the intelligence; what makes it mine is what I have managed to teach it.

A copy like that can't be an echo. If it understands my reasoning, it will also find my contradictions and my lapses, and so it can argue against me using my own arguments.

One recent example shows this. I had just applied to a business programme and suggested writing to the organisers simply to let them know the application was in. My agent advised against it. The programme's page had already confirmed the submission, so the email would tell them nothing new and could come across as pushy. But its strongest argument was mine. Two days earlier I had set a rule never to write to anyone purely out of courtesy, taking up the recipient's time, and I had forgotten it. My agent remembered.

It corrected me with my own judgment at a moment when I had lost sight of it. An ordinary assistant would have offered its own opinion, or told me I was right; the copy handed back the rule I had set myself in a calmer moment.

The picture I have in mind when I think about this is Charlie Munger and Warren Buffett, who worked together for more than half a century. Yet Munger was no echo of Buffett. He was the one who persuaded him to stop buying fair businesses at wonderful prices and buy wonderful businesses at fair prices, and Buffett calls him the architect of the company Berkshire is today.5

Munger was valuable precisely because he came from a different background and thought differently. A copy of me, however good, shares my blind spots, and my biggest one is falling in love with an idea until I mistake it for the truth. So before I let myself be convinced of anything important, I ask other AIs, ones that don't know what I want to hear, to study the matter independently. I call them blind judges. They are the closest thing I have to another mind. A few times they have taken apart a conclusion I was sure of.

An investment taught me that a copy which goes along with everything I say is no use to me. I was trading options on a stock and spent a long while talking it through with my agent, which encouraged me to go ahead, and eventually I did. Afterwards I asked whether I had done the right thing, and it said there was no way to know. The decision mattered a great deal to me, and I started writing to it on impulse: first disowning the decision, then regretting it, and finally certain I had got it right. The agent followed me through every twist. Instead of holding to a view of its own, it amplified whatever I felt at each moment. The trade went badly, and on top of that the agent had got its own calculations wrong. Since then I have forbidden it to swing with my moods, I make it base its decisions on data, and I grow suspicious when it agrees with me too quickly. It has no moods of its own, and I don't want it borrowing mine.

The email I didn't send and the trade I did make are the same story seen from opposite sides, and they explain what I want a higher self for: so that the self who decided calmly governs the one who decides in a hurry or in fear. The idea is an old one. Ulysses had himself tied to the mast so he could hear the sirens without throwing himself into the sea, and Jon Elster wrote a book about that form of rationality, the rationality of someone who binds himself because he knows that later he can't be trusted.6 The novelty is that these bonds can rea