Editing Openai/693c0f4f-255c-8008-92e9-0cd44c6d6226 (section)

==== : ====

I’d say:
* Yes, that’s a genuinely good idea as an architectural / training improvement: - δ is a natural, low-dim behavioral target. - Auxiliary “predict δ” or “match δ” losses can stabilize training and push the generator toward behaviorally relevant solutions instead of pure weight matching.
* But: - δ cannot be the only supervision if you want precise adapters. It’s too low-bandwidth and probe-dependent. - You still need some combination of: - weight-level MSE (even if downweighted), - and/or distillation on outputs for a subset of prompts.

If you want to turn this into a concrete research idea, an experiment that’s not insane in scale could be:
# Take a smaller LLM + a collection of LoRAs on multiple tasks.
# Compute δ for each LoRA using fixed probes.
# Train three LoRA generators (prompt-conditioned) with: - (a) weight-only MSE (DnD-style baseline), - (b) weight MSE + δ auxiliary loss, - (c) δ + output distillation but no weight MSE.
# Evaluate on: - In-domain tasks (reproducing known LoRAs), - Cross-task generalization (unseen tasks or unseen mixtures), - Robustness when reference LoRAs are noisy or partially corrupted.

My prediction:
* (b) will outperform (a) in stability and maybe generalization,
* (c) will be competitive on easy tasks but may underperform on hard ones → this will expose exactly how much information δ carries.

If you’d like, we can sketch that experimental setup more concretely (num tasks, probe design, what exact losses to use), or even think about borrowing the compositional-theory angle: δ as slot-space, generator as fff, and asking when “δ-compositionality” implies good LoRA compositionality.