Problem
A GPU-enabled instance does not seem to be able to use the device. The driver does not seem to be running. and when running "nvidia-smi" you get an error such as:
$> nvidia-smi NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.
This usually happens after an update of the Operating System kernel, and requires a rebuild of the NVIDIA driver to be compatible with the new kernel.
Solution
Using the morpheus web portal:
- Navigate to the instance showing the problems.
- Click on ACTIONS - Run Workflow.
- Pick "Nvidia driver refresh" and click EXECUTE.
- Morpheus will show the progress of this operation, and after a few moments, the GPUs should be available again.