Editing Openai/6910f96f-f278-8006-9996-5cf14ff7a27f (section)

==== Limitations & Caveats ====

The authors explicitly list several limitations:
* No real user study yet: The “human choice” is simulated by picking the prompt with the best F1 score. We don’t yet know how real users would interact with the system, what they prefer, or how much effort they are willing to spend.
* Computation time: Generating explanations for many examples is expensive and slow, and the current prototype doesn’t yet handle streaming results to users smoothly.
* Data splitting challenges: For user-provided datasets of different sizes, it’s non-trivial to choose good train/validation/test splits within the tool.
* Dependence on rephrasing quality: If the LLM rephrases prompts in a limited or repetitive way, later prompts may remain low-quality, and improvements can stall.
* Task and dataset sensitivity: Some datasets are not very sensitive to prompt wording, so even naive prompts can perform well.
* Ethical and privacy risks: User data stays local in their design, but underlying LLMs may still pose privacy risks; prompts found by the system are not guaranteed to be globally “best” or unbiased.