Editing Openai/693c0f4f-255c-8008-92e9-0cd44c6d6226 (section)

===== 1. δ is lossy - Same complaint as before: δ depends on a small probe set; two quite different LoRAs might share near-identical δ. - As a pure training target, δ under-constrains the generator. You still need weight-level loss or output-level distillation somewhere. =====
# Cost of computing δ during training - If you rely on the true Delta-operator (base+LoRA forward passes on probes) for every gradient step, this is heavy. - You’ll probably approximate it with a small network H(ΔW)H(\Delta W)H(ΔW), but then you’re stacking approximations.
# No direct effect on “no-training adaptation” - The hard part DnD solves is: from only prompts of a NEW dataset, output a useful LoRA with zero finetuning. - δ auxiliary loss helps learn a better mapping during training, but at test time, you still need the prompts → LoRA generator. You’re not replacing dataset-conditioning—just helping it.