Online Help >
Computing and Processing Practices
Computing and Processing Practices
Running Jobs Remotely
Putting Jobs in the Background
The DECF computing labs and clusters are available for use by anyone with a DECF account for
running jobs and processes.
There are 2 computing labs, and 3 computing clusters:
Both the 1111 and 1171 Etchverry labs are reserved for class and in-lab uses. DECF users should always use the
Archipelagos, Reindeer, and Ice clusters for running jobs remotely.
However, during school holidays and off-hours, users may use the 1111/1171 machines for
short and light computing jobs, as long as the jobs do not interfere with the users in the lab.
Be adviced that since the 1111/1171 machines are accessible by lab users, it is beyond our control when
machines are somehow shut-down, which will kill any jobs that are running.
Always think twice and check the 1111/1171 class schedules before sending any jobs to the 1111/1171 machines.
If you are uncertain if you should or when you can run jobs on 1111/1171 machines,
Check Ganglia for computers with lower loads to run your jobs on. The red computers have HIGH loads and the blue computers have LOW or NO load. You can check the load on a computer by typing "top" at the command prompt. "top" displays the active processes, how much CPU and memory they are using, and the load.
Q: How do you interpret load?
A: Load can be thought of as a ratio of roughly the number of active processes to CPU's available.
A computer with 1 CPU and a load of 1.00 means there is one process using all the CPU's resources.
A computer with 2 CPUs and a load of 2.00 means there are 2 processes using all the CPU's resources.
Click on the machine name on Ganglia for the specifications of each machine.
Q: What is a "high" load?
A: A "high" load is when the load exceeds the number of CPU's available.
For example, a load of 4.00 on a computer with 1 CPU is considered a high load because
there are roughly 4 processes all trying to utilize the resources of 1 CPU.
CPUs running at a high load can become overloaded as the CPU tries to accomodate all the processes
and cause the computer to crash, rendering the computer useless.
If you are new to SSH and remote connection, please see the How to SSH
for software installation and general instructions.
To run jobs remotely, simply SSH to kepler.berkeley.edu and authenticate with your DECF account.
Once logged into kepler, ssh to one of the client machines, then start the program you want to run.
If you are running jobs that will take more than 10 minutes to complete, you should always run
it in the background. Connections that are idle (with no active user interactions)
for longer that 10 minutes are automatically disconnected by the server for secruity reasons.
You can only put command-line jobs in the background. If you are running a program with a graphical interface, you
will not be able to put the job in the background.
We will focus on 2 ways that will allow users to put command-line jobs in the background: the "&" and the screen utility.
Using the "&" (ampersand) is the easiest way to put a job in the background. Simply putting the ampersand at the end of your
command will put that command in the background.
For example, we want to run Matlab with inputfile and want the output to be written to outputfile.
Since we know our job is going to run for a while, we want to put it in the background so that we can leave it running on the
remote computer and simply come back for the output file once it's done. To accomplish all that, we can simply issue a one-line
/usr/bin/nohup matlab-nogui < inputfile > outputfile &
This will run Matlab in the background with the specified input and output files.
Each program has its own syntax on input and output files. You will need to research on your particular program to find out
how to write one-line commands like the one above.
The second way to put jobs in the background is to utilize the screen manager. Simply put, you will start the screen session
and run your job within the screen session. You can then detach the screen session while the job continues to run. You can
re-attach the screen session in a later time to see your job that is running.
See the "Unix Command: Screen" for details on how to use screen to run jobs in the background.
DO NOT RUN JOBS ON KEPLER
Kepler is a login server and is not designed to handle computations.
Any jobs running on kepler will be killed without prior notice.
RUN ONE JOB AT A TIME
DO NOT run multiple jobs simultaneously; running multiple jobs simultaneously does not make jobs run faster,
and it overloads the machines and the file server.
Following how "load" works, when you are running two jobs simultaneously, each job ends up getting half the CPU's processing
resources that one job by itself would, while increasing the load of the computer.
It would take just as much or even less time to run the jobs one after the other (instead of simultaneously)
and the load on the computers would be lower.
The file server houses EVERYBODY's files so it ABSOLUTELY CANNOT CRASH. The load on the file server increases when you
run two or more jobs at the same time because it has to handle twice as many or more data transfers,
which can overload the file server.
Users found running multiple jobs at the same time will have their jobs killed and their accounts locked.
Users running resource-intensive jobs on 1111/1171 machines during class time will also have their jobs killed.
This is a necessary step to keep from overloading our computers and servers, and to make sure that users in our labs can work during class.