POSTED BY
2024-05-13

apt upgrade, dist-upgradeのあとnvidia-smi

NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.

nvidiaのドライバはカーネルと密接しているので、カーネル関係がupgradeしたら入れ直しか。以下全部入れ直しで解決。

nvidia, cuda, cudnn関係パッケージの全削除

確認

dpkg -l | grep nvidia
dpkg -l | grep cuda
dpkg -l | grep cudnn

削除※自己責任注意

sudo -s
apt --purge remove *nvidia*
apt --purge remove *cuda*
apt --purge remove *cudnn*
apt --purge autoremove

再起動

reboot

nvidiaドライバーの再インストール

sudo -s
add-apt-repository ppa:graphics-drivers/ppa
apt update
apt install ubuntu-drivers-common
 
ubuntu-drivers devices
 
#recommendのものをインストール
apt install nvidia-driver-550

CUDA ToolKitとCUDNNの再インストール

https://developer.nvidia.com/cuda-downloads
https://developer.nvidia.com/rdp/cudnn-download

Linux→x86_64→Ubuntu→22.04→deb (network)

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.1-1_all.deb
sudo -s
dpkg -i cuda-keyring_1.1-1_all.deb
apt update
apt -y install cuda-toolkit-12-4
apt -y install cudnn

再起動

reboot

パスの設定

export CUDA_HOME=/usr/local/cuda-12.4
export PATH=$CUDA_HOME/bin:$PATH
export LD_LIBRARY_PATH=$CUDA_HOME/lib64:$LD_LIBRARY_PATH

稼働確認

 nvidia-smi
Mon May 13 20:12:05 2024       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.54.15              Driver Version: 550.54.15      CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 3090        Off |   00000000:17:00.0 Off |                  N/A |
| 47%   25C    P8             20W /  350W |    1342MiB /  24576MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA GeForce RTX 3090        Off |   00000000:65:00.0 Off |                  N/A |
| 46%   24C    P8             26W /  350W |       3MiB /  24576MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A      1434      C   python3                                      1336MiB |
+-----------------------------------------------------------------------------------------+

ps aux | grep nvidia
root         415  0.0  0.0      0     0 ?        S    18:13   0:00 [nvidia-modeset/]
root         416  0.0  0.0      0     0 ?        S    18:13   0:00 [nvidia-modeset/]
root         417  0.0  0.0      0     0 ?        S    18:13   0:00 [irq/76-nvidia]
root         418  0.0  0.0      0     0 ?        S    18:13   0:00 [nvidia]
root         441  0.0  0.0      0     0 ?        S    18:13   0:00 [irq/77-nvidia]
root         442  0.0  0.0      0     0 ?        S    18:13   0:00 [nvidia]
nvidia-+     959  0.0  0.0   5468  2000 ?        Ss   18:14   0:00 /usr/bin/nvidia-persistenced --user nvidia-persistenced --no-persistence-mode --verbose

nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2024 NVIDIA Corporation
Built on Thu_Mar_28_02:18:24_PDT_2024
Cuda compilation tools, release 12.4, V12.4.131
Build cuda_12.4.r12.4/compiler.34097967_0

お疲れ様でした。