We rolled out our own that does pretty much the same thing but perhaps more because our solution can also mount persistent storage that can be carried between multiple runners. It does take 1-5 seconds to boot the environment (firecracker vms). If this sandbox is faster I will instruct the team to consider for fast starup.
This is also very similar to Vercel's sandbox thing. The same technology?
What I don't like about this approach is the github repo bootstrap setup. Is it more convenient compared to docker images pushed to some registry? Perhaps. But docker benefits from having all the artefacts prebuilt in advance, which in our case is quite a bit.
When your agent performs 20 tasks saving seconds here and there becomes a very big deal. I cannot even begin to describe how much time we've spent on optimising code paths to make the overall execution fast.
Last week I was on a call with a customer. They where running OpenAI side-by-side with our solution. I was pleased that we managed to fulfil the request under a minute while OpenAI took 4.5 minutes.
The LLM is not the biggest contributor to latency in my opinion.
Thanks! While I agree with you on "saving seconds" and overall latency argument, according to my understanding, most agentic use cases are asynchronous and VM boot up time may just be a tiny fraction of overall task execution time (e.g., deep research and similar long running tasks in the background).
1-5 seconds seems high for Firecracker, depending on your requirements.
We boot VMs (using Firecracker) at ~20-50ms.
Obviously depending on the base image/overlay/etc., your system might need resources making it a network-bound boot, but based on what you've said it seems you should be able to make your system much faster!
I browsed through the documents but it does not seem to be possible to auto destroy a sandbox after certain amount of idle time. This forces who ever is implementing this to do their own cleanup. It is kind of missed opportunity if you ask me as this is a big pain. It is sold as fire and forget but it seems that more serious workflows will require also a lot of supporting infrastructure.
We rolled out our own that does pretty much the same thing but perhaps more because our solution can also mount persistent storage that can be carried between multiple runners. It does take 1-5 seconds to boot the environment (firecracker vms). If this sandbox is faster I will instruct the team to consider for fast starup.
This is also very similar to Vercel's sandbox thing. The same technology?
What I don't like about this approach is the github repo bootstrap setup. Is it more convenient compared to docker images pushed to some registry? Perhaps. But docker benefits from having all the artefacts prebuilt in advance, which in our case is quite a bit.