The Best Way to Learn AI Is Not to Study It

On 4 February this year, at half past ten in the morning, I opened my laptop and started building a CAA practice site for my son.

By 3:43 that afternoon, the first version was running. I took a lunch break in between.

I've told this story a few times since, and the reaction is always the same: AI is incredible now. I agree.

There's a second half to it, though. I'm still working on that site. Six months and counting.

Ten years of enterprise systems, and feedback never moved this fast

I spent a decade doing enterprise IT, mostly rolling out SAP for large companies. The client states a requirement, we propose a solution, we build it, it goes live, someone signs the acceptance form. A project ran two or three months at the short end; the longest client engagement I had lasted six years. I had no desk at my own company — I was permanently stationed at the client's site. Essentially the old-school version of what people now excitedly call a forward-deployed engineer.

I was with those clients every day. But feedback in enterprise systems travels through layers. An end user finds something awkward and mentions it to their supervisor. The supervisor decides it's minor and absorbs it. Even when it does get escalated, by the time it reaches me it has become "users report the module is inconvenient to operate." Several people, several weeks, several layers of translation. Where the person actually got stuck, and what they actually said about it — gone.

In ten years I never once had the experience of someone using the thing I built, sitting right next to me, letting me know immediately whether it was any good.

It started with a calendar

The real starting point came two months before the website. On 17 December last year, I built a calendar for the family. Our lives were too scattered — who's dropping off whom and when, which day has an event, when something needs a follow-up appointment. Everyone tracked their own, and things kept slipping. I also wanted something with real depth: plan ahead, record as you go, review afterwards. I'd tried plenty of calendar apps. None of them fit.

I revised that thing more than seven hundred times over those two months. Only later did I realise that was practice — and that AI back then was nowhere near as capable as it is now.

By February my son had his CAA coming up. Beyond the official question bank and some scattered information, we knew almost nothing. Which is, I suppose, the particular kind of relaxed that New Zealand does so well.

Half a day to build, six months to finish

The first version was a title, a line of explanation, and a start button. Reading, Writing, Numeracy — three sections, click in and start practising.

He used it that first evening. His initial reaction was that it worked nicely. Under thirty minutes later, the problems started surfacing.

Buttons disappearing when switching questions. Pages that didn't respond to clicks. The kind of path you hit within a minute of real use and that I'd never reach testing it myself.

I asked people in a group chat to report bugs and suggestions, and kept a running record. First entry 5 February, last one 15 July. Font too small. Practice mode missing questions. Reading and Numeracy not responding. Images cut off. Reading passages not displaying in full. Requests for a practice history and a review of wrong answers.

Not one of them was a feature. Every one of them was "this part isn't right."

The barrier to building something demonstrable really has all but disappeared. But the road from demonstrable to usable-every-day is one AI can't help much with, because what that road requires is: someone actually using it, hitting a problem, you fixing it, them using it again.

The AI feature I built, he mostly ignores

One entry in that record is mine. 8 February: add AI tutoring and grading. If you're preparing for an exam, doing questions isn't enough — you need feedback, and that's exactly what AI is for. I put a fair amount of time into it.

He mostly ignores it. Do the question, check right or wrong, next one. The AI assessment I'd laboured over, he'd occasionally open and then move straight past.

Later I went and read those assessments myself, particularly on the writing section, and I understood. The feedback isn't accurate enough. In places it's actively misleading. A fourteen-year-old can't tell which parts to trust and which to discard — but he can tell that something about it doesn't ring true. So he doesn't read it.

I learned that from him, not from any tutorial. I'd read every article there is about the boundaries of AI's capability. Not one of them told me the specific thing: that on writing feedback, here is where it fails. My son found that out by using it.

If he weren't my son, if he weren't next to me every day, I almost certainly wouldn't know. From the data the feature gets used — the open rate is respectable — and I'd have assumed it was working and kept building on top of it.

I later did a proper study of this and found the shape of it: AI does considerably better assisting with things that have a definite answer, like maths. For writing — analysis and assessment — it's not that it can't, it's that the problem is a great deal harder.

I posted a link in a parents' group and got taken for a spammer

At first only he used it. Then a classmate asked, so I shared it. Then it seemed a waste to have only one person using it. The NZQA material is public anyway; all I'd done was arrange it into something you could practise and test against daily.

So I posted a link in his school's parent group, and in the Westlake groups nearby.

In one of those groups, I got taken for someone pushing an ad.

That stings a little. You build something, you give it away free, and you still have to think carefully about how to phrase it so it doesn't sound like a pitch. In an age drowning in information, trust is the genuinely scarce asset.

People started using it and started giving feedback, and that's when the record began filling up. From a handful of families to more than a hundred. I did no other promotion — those two groups, plus one post on social and one in my WeChat Moments.

Years in sales have made me considerably more fluent in how people work than I was as an IT guy. But when someone goes out of their way to thank you, pretending to be unmoved would be a performance. I hold those under a magnifying glass.

The best way to learn AI is not to study it

Friends often ask me where they should start learning AI. My answer is: don't.

The act of studying AI tends to come up empty. You work through a stack of courses, bookmark dozens of tools, watch one video after another. And then? The tools change every year. The prompting technique you learned last month may already be obsolete. Even Anthropic's own Claude Academy can't reliably keep pace with their own product releases. All of it just makes you more anxious.

More to the point: the AI in a tutorial is a specimen. What it claims to do is spelled out precisely. Where it will fail you, you have no way of knowing.

What actually taught me anything was those six months of being chased for bug fixes by my son and by the parents in that group. How large the font should be. Whether switching questions should preserve progress. When the AI assessment is useful and when it's noise. Not one of those came from a tutorial.

So if the thought has crossed your mind — don't start with what should I learn. And don't start with what could I build, either. That second question leads people into a ditch: either too grand, some world-changing product, or too hollow, something you won't open twice yourself.

Try a different question: is there someone near you, struggling with something, who is in front of you every single day?

Once you have that person, you have a user who won't spare your feelings. Their need is real — you don't have to guess at it. If it doesn't work, they'll be after you to fix it tomorrow.

As for time, here's my own number: the version I built in half a day looked about eighty percent done. The remaining twenty percent consumed every hour since. What I took from that is that in the age of AI you have to be far stingier about what you choose to spend yourself on.

Nothing in this world is truly difficult for someone who sets their mind to it. If you have a good question, or something that's been nagging at you, then congratulations — you already hold the key to AI, and it's the fastest and cheapest one available.