Spaces:
Running
Running
File size: 3,025 Bytes
4e9ddf9 b5a071f 4e9ddf9 b5a071f 4e9ddf9 b5a071f 4e9ddf9 b5a071f 4e9ddf9 862fdcc 8c6c688 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 |
# Running benchmarks on multiple GPU nodes with Pegasus
[Pegasus](https://github.com/jaywonchung/pegasus) is an SSH-based multi-node command runner.
Different models have different verbosity, and benchmarking takes vastly different amounts of time.
Therefore, we want an automated piece of software that drains a queue of benchmarking jobs (one job per model) on a set of GPUs.
## Setup
### Install Pegasus
Pegasus needs to keep SSH connections with all the nodes in order to queue up and run jobs over SSH.
So you should install and run Pegasus on a computer that you can keep awake.
If you already have Rust set up:
```console
$ cargo install pegasus-ssh
```
Otherwise, you can set up Rust [here](https://www.rust-lang.org/tools/install), or just download Pegasus release binaries [here](https://github.com/jaywonchung/pegasus/releases/latest).
### Necessary setup for each node
Every node must have two things:
1. This repository cloned under `~/workspace/leaderboard`.
- If you want a different path, search and replace in `spawn-containers.yaml`.
2. Model weights under `/data/leaderboard/weights`.
- If you want a different path, search and replace in `setupspawn-containers.yaml` and `benchmark.yaml`.
### Specify node names for Pegasus
Modify `hosts.yaml` with nodes. See the file for an example.
- `hostname`: List the hostnames you would use in order to `ssh` into the node, e.g. `jaywonchung@gpunode01`.
- `gpu`: We want to create one Docker container for each GPU. List the indices of the GPUs you would like to use for the hosts.
### Set up Docker containers on your nodes with Pegasus
This spawns one container per GPU (named `leaderboard%d`), for every node.
```console
$ cd pegasus
$ cp spawn-containers.yaml queue.yaml
$ pegasus b
```
`b` stands for broadcast. Every command is run once on all (`hostname`, `gpu`) combinations.
## Benchmark
Now use Pegasus to run benchmarks for all the models across all nodes.
```console
$ cd pegasus
$ cp benchmark.yaml queue.yaml
$ pegasus q
```
`q` stands for queue. Each command is run once on the next available (`hostname`, `gpu`) combination.
## NLP-eval
Now use Pegasus to run benchmarks for all the models across all nodes.
```console
$ cd pegasus
$ cp nlp-eval.yaml queue.yaml
$ pegasus q
```
for some tasks, if the cuda memory of a single gpu is not enough, you can use more GPUs like follows —
1. create a larger docker with more gpus, e.g. 2 gpus:
```console
$ docker run -dit --name leaderboard_nlp_tasks --gpus '"device=0,1"' -v /data/leaderboard:/data/leaderboard -v $HOME/workspace/leaderboard:/workspace/leaderboard ml-energy:latest bash
```
2. then run the specific task with Pegasus or directly run with
```console
$ docker exec leaderboard_nlp_tasks python lm-evaluation-harness/main.py --device cuda --no_cache --model hf-causal-experimental --model_args pretrained={{model}},trust_remote_code=True,use_accelerate=True --tasks {{task}} --num_fewshot {{shot}}
```
change `model`, `task` and `shot` to specific tasks
|