-
Notifications
You must be signed in to change notification settings - Fork 47
Open
Description
With the recent google-batch provider, I am noticing that any stdout / err with % is broken. Maybe it is parsed as old style strings?
For example when I print nvidia-smi which shows % gpu usage, I see this:
Thu Jan 2 22:40:08 2025
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 560.35.03 Driver Version: 560.35.03 CUDA Version: 12.6 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 Tesla T4 Off | 00000000:00:04.0 Off | 0 |
| N/A 66C P8 14W / 70W | 3MiB / 15360MiB | 0%!D(MISSING)efault |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| No running processes found |
+-----------------------------------------------------------------------------------------+
0% is accurate but 0%!D(MISSING)efault is not
Metadata
Metadata
Assignees
Labels
No labels