How I let AI quiz me

Two studies on AI and learning came to almost opposite conclusions last year. At MIT, researchers had 54 people write an essay, some of them with ChatGPT alongside. That group showed the weakest brain activity, and a day later about 17 percent of them could still quote their own piece correctly, against about 46 percent of the people who had written without help. At Harvard, 194 physics students were given the same material twice: once in an active workshop with an experienced teacher, once at home with an AI tutor the teachers had set up themselves. With that tutor they learned about twice as much, in less time.

That looks contradictory until you look at what the AI was doing in each case. In the first, it wrote your piece. In the second, it asked you questions about the material. The technology is almost the same, the role is not.

I only read both studies this summer. By then I had been using the method that follows from them for half a year, and not because I had read any research. Because an attempt at an explanation failed completely back in March.

It was not an explanation problem

I wanted to understand better how my own things worked. It all ran, but I could not always retell why. So I asked for an explanation. What I got back was correct and utterly useless: abstract examples about a shopping cart and a fictional user table. I understood it while reading and had lost it a day later.

My first diagnosis was that the explanation was too hard. That was wrong. The problem was the material. An AI that gets a general question answers with the average of everything it has ever seen, and that average is about nobody. It was not about my server, my settings, my outage last month.

The tool: Notebook

What solved that is Notebook by Google, which you may still know as NotebookLM. The difference with an ordinary chat is that you put sources in first and it only answers out of those. Documents, PDFs, websites, videos: it handles just about anything.

What makes it useful for learning sits in Studio, the part that makes things out of your sources. A spoken summary in which two voices discuss your material, and a quiz on exactly that material. By now it can also look for extra sources itself to make your subject more complete, which helps when your material has a gap you cannot see yet. And you can share a notebook, so what you build for yourself you can hand to a colleague.

Use it for subjects where general knowledge exists and where you want to get up to speed. The exact place where it falls short is where my March problem sat: it knows nothing about your situation as long as you do not put it in there.

Do not put code or data in it

The temptation is then to dump your whole project folder, an export or a database into it. Do not.

Two reasons. It is not built for that, so you get shallow answers about files instead of an explanation of how something works. And with data there is a second problem: everything you upload leaves your hands. For work files holding other people's data, you do not even want to have to ask that question.

The detour is shorter than it sounds. Ask your own AI assistant, the one that is allowed to see your code and your notes, to write study material out of it. One document, in plain language, with the pieces of code that matter quoted inside it. That document goes to Notebook. Your code stays where it belongs.

What that looks like in practice

Take my own little server. A handful of programs run on it side by side, with a front door in front of them that divides up the traffic and arranges the security certificates. I had built it, it worked, and I understood half of it.

I asked my assistant to turn that into a study document. What came out were six chapters about my own installation. With my own configuration file quoted inside it, and with the evening of 17 March as a separate chapter, when everything went offline at once over a certificate problem. Every chapter ends with a question.

That is the difference with a manual. A manual explains how something like that works in general. This explained why my own setup fell over that evening, and that evening I had not forgotten.

That document went into Notebook, together with a few official documentation pages as a counterweight. After that I could get to work with it.

I want to understand how [subject] works in my own project. Write one study document about it, five to six chapters, that I am going to load into Notebook as a source.

Conditions: use my real files and settings as the examples, and quote the relevant parts literally with the file name next to them. Do not use generic examples. Include outages or mistakes I have had myself as a separate chapter, because that is where most of the learnable material sits. Explain every concept through a comparison with something I already understand. End every chapter with one question that tests whether I get it, not a trivia question.

Write it as a running document, not as a summary. Repetition is fine.

The first version was too tidy

In Notebook I had a single spoken summary made from that document. Done, I thought. Listen while walking and it will stick.

It worked worst of everything I tried afterwards. One episode for an entire subject means everything gets compressed into the main line, and the main line I already knew. What I was missing sat precisely in the steps in between.

The fix was boring and effective: cut it up. Two to three episodes per subject, each covering no more than two chapters, with the instruction not to summarise. When I tried that again on the first subject the difference was immediately clear.

But the real change was not in the audio. It was in what I did afterwards.

The order that makes the difference

After every episode I go to the notebook's chat and let it quiz me. Not me asking the questions, but the AI asking them of me, one at a time, marking strictly. Within two minutes I can tell what I actually understand and what I only recognised while someone else was saying it.

The order that works for me: audio is the introduction, the quizzing round is the learning moment, the quiz from Studio is the test, and explaining it back is the proof. That last step is the heaviest. A quiz tests whether you recognise the right answer among four options. Explaining it back without looking tests whether you can produce it yourself, and that is a different level altogether.

Two things turned out to matter more than I expected.

The first: do not let the AI hand you the answer when you are wrong. That is its natural inclination, and it costs you exactly the moment where you could have learned something.

Quiz me on the check questions from chapters 1 and 2, one at a time, and mark strictly.

If I am wrong, do not give me the right answer straight away. Name which step in my reasoning went wrong and let me try again. If I still do not get there, explain it using the example from the source. At the end, come back to the questions I got wrong.

The second: never ask whether you understand it. You always say yes to that, even to yourself. Ask to explain it.

I am going to explain this subject to you now without looking back. Do not judge whether it is correct, but where my explanation has gaps, and which of those gaps would show up the moment somebody asks a follow-up question.

What I took from it

The gain is not in a better explanation. It sits in two things you arrange yourself before you ask anything: which material you put in, and which role you give the AI. Feed it something that is about your own situation and let it question you, and it stays. Let it take the work off your hands, and you get a good feeling and not much left a day later.

That lines up with what those two studies showed, but research is not how I got there. I got there because it did not work the first time.