Can not host as vLLM ?
#17
by
tommywu052
- opened
We currently do not support vLLM but are actively working on integrating NVLM with vLLM. Our team is committed to delivering this support as soon as possible.
Thanks,
Boxin
Seems supported in this pr https://github.com/vllm-project/vllm/pull/9045, but not yet released.
Installing vllm from the latest code might work.
checking out from latest main and building docker image worked for me
Thanks guys. It work well after install the latest wheel -
pip install https://vllm-wheels.s3.us-west-2.amazonaws.com/nightly/vllm-1.0.0.dev-cp38-abi3-manylinux1_x86_64.whl
and serve as
vllm serve nvidia/NVLM-D-72B --tensor-parallel-size 4 --enforce-eager --max-num-seqs 16 --trust-remote-code