Editing Openai/694057b6-101c-8007-9a65-c40578c7252d (section)

=== User: As a general rule for this nuke & repave is the preferred strategy rather than trying delicate replay and similar trick… ===
As a general rule for this nuke & repave is the preferred strategy rather than trying delicate replay and similar tricks.

Multiple requests should be a designated number of "slots" which can be used. If all slots are full there is no reason to take a request it can't handle yet; that can wait or be directed to another instance.

For timeouts and other in flight behavior: this will among other things be used with large slow models on slow hardware and relatively large contexts. A practical concern is that time to first token can be significant for some requests.

Gathering streaming partial tokens is probably desirable even if not exposed to the outside, if only as part of the monitoring and recovery system.

Tools should probably be OpenAI format: something which models will easily recognize and be able to handle.

The internal tools here will be limited to lightweight calculator or lookup types. Larger calls would be routed to a higher level model.