Today I listened to a blog post: Creating a LLM-as-a-Judge That Drives Business Results. Yes, I did mean I listened!
It's a long post, and after a day of looking at the screen I felt more like listening than reading. I took the opportunity to try out NotebookLM. It's really easy, just paste the URL and click the Generate button.
NotebookLM generated a 23 minute podcast which was actually quite enjoyable and easy to listen to. I did pause a couple of times to read that particular section of the blog post, to understand it better.
What I particularly liked about the blog post was the simplicity of the suggested approach (pass/fail) along with the human understandable AI critique/reasoning. It also made a lot of sense to me that curating a high quality dataset forces you to really understand the domain and "codify" what are good and bad answers.
In practice, domain experts may not have fully internalized all the judgment criteria. By forcing them to make a pass/fail decision and explain their reasoning, they clarify their expectations and provide valuable guidance for refining the AI.
I still plan to revisit the original blog post, to make sure I get some of the details that the NotebookLM podcast might have skimmed over. Overall, I'm really impressed with the NotebookLM podcast feature, and I think it is a great way to get an overview about a new topic.