Editing Openai/693c0f4f-255c-8008-92e9-0cd44c6d6226 (section)

==== Imagine we train a network GGG such that: ====

G(δ)≈LoRA weightsG(\delta) \approx \text{LoRA weights}G(δ)≈LoRA weights
using many (δ, LoRA) pairs from your LoRA zoo.

Then:
* For an existing LoRA, you can encode it → δ → decode back LoRA (autoencoder in behavior space).
* You can do ops like δcombo=δA+δB\delta_{\text{combo}} = \delta_A + \delta_Bδcombo=δA+δB then feed G(δcombo)G(\delta_{\text{combo}})G(δcombo) to get a new, “blended” LoRA.

That’s actually a cool idea:
* Delta Activations give you a nice, low-dimensional latent space over model behaviors.
* A generator GGG makes that latent space “actionable” by turning latents back into weights.

But notice: this is fundamentally about recombining / editing behaviors of LoRAs you already have.

If you don’t use datasets at all, then:
* You can sample random δ and generate random LoRAs…
* …but you don’t know what these LoRAs are good for. You’ve lost the link to any task semantics.

So with only δ as condition, you can:
* Explore a smooth manifold of “things like the LoRAs I trained before”
* Compose known domains (e.g., mix math & code LoRAs via δ-math + δ-code)
* Possibly regularize / denoise noisy LoRAs via autoencoding

…but you cannot say “give me a LoRA for this new dataset / capability”, because δ doesn’t know what that is unless you somehow produced δ from that task.