Multi-GPU Configuration

To use multiple GPUs, change the following settings in the model deployment directory:

1. Change the number of GPUs in .env:

   TENSOR_PARALLEL_SIZE=2
   

2. In docker-compose.yml, edit device_ids:

device_ids: [0, 1]

Service Management

Start service

docker-compose up -d

Stop service

docker-compose down

Restart service

docker-compose restart

View logs

docker-compose logs -f

View service status

docker-compose ps