Watch this WaysConf session, then continue with related talks or explore the current programme.
In this WaysConf 2025 presentation, Jakub Serek, a Quantitative CX Researcher at Silky Coders, shares a case study on automating qualitative data analysis for customer feedback. By transitioning from manual analysis to an API-driven LLM pipeline, Serek's team achieved a 250x increase in analysis speed, unlocked a yearly value of 50,000 PLN by processing previously unanalyzed feedback, and maintained a 95% validity rate. Serek outlines his systematic approach to building trust in AI rather than relying on AI magic. He details how he established evaluation metrics, curated test datasets of 100 comments per language, and iteratively refined prompts from zero-shot to an extended few-shot structure using GPT-4o. The talk highlights the importance of choosing simple, repetitive problems for automation, leveraging APIs for greater control, and continuously evaluating model outputs against human-coded benchmarks to manage risks in business decision-making.
Automating the qualitative analysis of post-purchase surveys sped up the process 250 times and unlocked 50,000 PLN in annual value. This allowed the team to finally analyze a massive volume of negative feedback that was previously left unexamined due to resource constraints.
Watch from 1:12Interacting with LLMs via an API rather than a graphical user interface provides the precise control needed for production-level research. It allows developers to separate system and user prompts, tweak parameters like temperature, and run systematic evaluations.
Watch from 8:11To build trust in LLM outputs, researchers should evaluate prompts using a test dataset of around 100 diverse examples per language. This dataset must represent all defined categories and include complex edge cases that sit between multiple classifications.
Watch from 13:20A simple few-shot prompting technique with clear positive and negative examples yielded the highest validity score of 95% with GPT-4o. This outperformed both zero-shot prompts and complex chain-of-thought structures in final accuracy.
Watch from 20:33Relying on third-party LLM providers introduces risks when models are updated or deprecated. Maintaining a permanent, human-verified test dataset allows teams to quickly evaluate and transition to alternative models without losing quality.
Watch from 32:32These concise notes were edited from automatic captions and checked against the talk structure. They are not a verbatim transcript.
Jakub Serek defines two ways of working with LLMs. AI magic relies on ad-hoc, poorly defined prompts for exploratory tasks, which yields inconsistent results. In contrast, building trust involves setting clear goals, defining metrics, and systematically evaluating outputs. While building trust requires more upfront development time, it ensures long-term quality and reliability for automating repetitive tasks.
Watch from 2:23The research team faced a massive bottleneck with post-purchase email surveys. They collected around 20,000 negative comments annually across three fashion brands (Sinsay, Mohito, and Reserved) in 19 different languages. Manually translating and analyzing this volume of continuous feedback end-to-end would require a full-time researcher, prompting the team to seek LLM automation.
Watch from 4:57While graphical chat interfaces are highly accessible, they are insufficient for production-level research workflows. Serek chose to interact with OpenAI's models via an API because it offers a much higher level of control. The API allows developers to isolate system prompts, adjust parameters like temperature, and integrate the LLM with Python to clean and visualize data.
Watch from 7:54To run reliable evaluations, researchers must construct high-quality test datasets. Serek recommends using a rule of thumb of 100 examples per language to balance evaluation quality with API costs. These datasets must be representative of the actual feedback, reflecting the real-world frequency of different customer complaints and including diverse edge cases.
Watch from 12:58Serek's prompting experiments revealed that simplicity often outperforms complexity. A basic zero-shot prompt yielded only 70% to 72% validity. Introducing a simple few-shot prompt with basic category definitions boosted accuracy to 90%. The winning solution was an extended few-shot prompt that provided multiple positive and negative examples for every single category, allowing GPT-4o to reach 95% validity.
Watch from 18:31
