Our next topic addresses a common problem that’s encountered in RAG applications that rely on structured outputs produced by an LLM. When those outputs are to become inputs in the next processing steps of the application, their structure becomes very important.

A bit of background

Due to their non-deterministic nature, LLMs have the bad habit of sometimes producing responses in a format other than the requested one, adding unsolicited comments or descriptions – just like humans if you think about it. Simply relying on clever prompting techniques may not be enough to completely avoid this behavior.

Even models specifically trained to follow precise instructions occasionally deviate from the structure we’ve requested. In cases where that output is simply returned to the user, this doesn’t matter much – it might even create a more natural experience.

The problems arise when the structure of the response...