Zebra is a shared resource, so please:
- Email data@reed.edu if you need more than the default 32 cores or 64 GB of memory. Tell us what you're running and roughly how long it will take.
- Run
nvidia-smi before starting a GPU job to make sure no one else is using the GPU, and let us know before starting any GPU job you expect to run for more than a day.
- Let us know if any job will run for more than a couple of days, so we know it's intentional.
- Test on a small piece of your data before launching a full run.
- Close tmux sessions you're no longer using, and delete files you no longer need.
If you have any questions, email data@reed.edu.
Run scripts inside a session
To keep things running if you disconnect from Zebra (by closing your laptop, losing Wi-Fi, or dropping off the VPN), you should run things inside a tmux session.
After logging into Zebra, start a session and give the session a name that is informative to you (here, myjob):
tmux new -s myjob
Your terminal will clear and a green bar will appear at the bottom. You're now inside the session.
Run your script and save its output to a log file
Run your script as usual, but send its printed output to a log file so you can check on it later:
Rscript myscript.R > myscript.log 2>&1
For Python, add -u so output is written to the log right away instead of in delayed chunks:
python -u myscript.py > myscript.log 2>&1
The > myscript.log 2>&1 part sends both regular output and error messages to myscript.log in your current folder.
Exit the session & reconnect later
Each tmux session is self-contained, like a separate terminal window that stays open on Zebra even after you log out. (You can have several running at once, which is useful for running different scripts side by side.) You can exit the session without stopping the process from running.
Press Ctrl+b, let go, then press d.
You'll be back at your normal prompt, and your script keeps running in the background. To log out of Zebra completely, type:
exit
To reconnect, log back into Zebra, then list your sessions:
tmux ls
To reconnect to a specific session type:
tmux attach -t myjob
When your script is finished and you no longer need the session, type exit while inside it to close it.
Check on your job
Open the system monitor:
htop
Press u and select your username to show only your processes. The CPU% and MEM% columns show how much your job is using. Press q to quit. (Leaving htop does not affect your job.)
If your job uses the GPU, check it with:
nvidia-smi
See your script's progress
To see the latest output in your log file as it's written:
tail -f myscript.log
Press Ctrl+c to stop watching. This only stops the display, not your script.
Estimate how long your job will take
Zebra can't tell you how long a script will take, but you can build progress messages into your code. In R, add a line inside your loop such as message("Finished ", i, " of ", n); in Python, print(f"Finished {i} of {n}") or the tqdm package works well. These messages will appear in your log file.
Before running your full analysis, try it on a small piece of your data first. This will give you an indication of how long it will take, and if there are errors in your code, you'll know sooner than if you ran everything.
Stop a job
If you're connected to the session, press Ctrl+c to stop the script, then close the session by typing:
exit
To end a session without reconnecting to it, use:
tmux kill-session -t myjob
If a process won't stop, kill it
Open htop, find your process, and note the number in the PID column. Then quit htop and run, replacing 12345 with that number:
kill 12345
If it's still running after a minute, use kill -9 12345 to force it to stop.
Running things in parallel
Each user can use up to 32 processor cores and 64 GB of memory by default. You can split your work across several of them (parallel computing) to make your code run much faster.
Set the number of cores
Set the number of cores explicitly. In R, for example:
parallel::mclapply(my_list, my_function, mc.cores = 8)
or, with the {future} package:
future::plan(future::multisession, workers = 8)
In Python:
multiprocessing.Pool(processes=8)
or, with joblib or scikit-learn, set n_jobs=8.
Start with around 8 cores and see how things go; more cores isn't always faster. Also, running things in parallel is only possible for things that actually can be run independently. So it works great for vectorized data, but any loops that rely on the previous iteration (i + 1) won't work in parallel.
Keep an eye on memory
Each parallel worker usually gets its own copy of your data, so memory use multiplies. If you have 8 cores and each needs 8 GB to run you're already at your limit and the job will stop. You can check htop after starting a parallel job to see memory usage.
Some packages (for example, ones that do heavy matrix math) use multiple cores on their own. If htop shows your job using many more cores than you wanted, set things manually with this before starting your script:
export OMP_NUM_THREADS=1