A method for generating synthetic user queries (e.g. for bootstrapping an eval set before real traffic exists) that avoids the blandness of asking an LLM for generic "plausible" questions.
Instead of prompting an LLM directly for sample questions, first define two or more orthogonal dimensions relevant to the product's personas or use cases — e.g. customer type × product tier — and enumerate the combinations those dimensions produce. Generate queries for each combination separately, rather than in one generic batch, to force diversity and coverage across the actual variation in real users.
Quote: "You don't want to just ask an LLM, hey, come up with plausible questions for a prospective tenant that might be looking to rent an apartment. The right way to do it is to come up with some categories... some dimensions."
This complements Error Analysis (Manual Trace Review): synthetic traces generated this way give a team something to read and code before real production traffic is available.
A single generic prompt such as 'come up with plausible customer questions' tends to produce homogeneous, low-diversity synthetic test data — many superficially different questions that actually cluster around the same few scenarios. The fix demonstrated is to explicitly cross concrete dimensions (e.g. customer type x product tier) so generation is forced to cover the cross-product of cases rather than repeatedly sampling the same mode. Even so, synthetic data is explicitly framed as a stopgap: 'nothing beats real data' once real traces are available for Error Analysis (Manual Trace Review).