JIP is another approach to implement a pipeline system that helps to manage a large number of jobs on a compute cluster. It simplifies the process of creating computational pipelines with dependency support, automatic expansions and simplified management of jobs and resources.
Even if you are not interested in pipeline and dependencies; JIP offers a set of management commands to simplify interactions with your compute cluster via allowing you to submit commands quickly, to restart and to move jobs, to edit commands interactively, to avoid duplicated job submission and more.
There is more documentation available and there are a few things you might need to understand before you create more sophisticated work-flows. But here is the quick-start to install and to configure JIP on your system. Take a look at the setup guide for more detailed description of the installation and configuration process.
JIP reads $HOME/.jip/jip.json to load your configuration. Create the file with the following content.
For a Slurm cluster:
{
"cluster": "jip.cluster.Slurm"
}
For a PBS/Torque cluster:
{
"cluster": "jip.cluster.PBS"
}
For a Gridengine/SGE/OGE cluster:
{
"cluster": "jip.cluster.SGE",
"sge" : {
"threads_pe": "threads"
}
}
Please note that in order to submit multi-threaded jobs for SGE, you have to specify the parallel environment that is configured for threaded jobs.
For a Platform LSF or Openlava cluster:
{
"cluster": "jip.cluster.LSF"
}
For the JIP local scheduler:
{
"cluster": "jip.grids.JIP"
}
In order to use the local scheduler that ships with JIP, please check the documention on how you can configure and start the JIP scheduler on you local machine.
You can run or submit a bash command using the jip bash <jip_bash> command. For example:
$> jip bash -c hostname
This will run hostname locally. Add the -s option to submit the command as a job to your cluster:
$> jip bash -s -c hostname
You can check the status of the job with jip jobs <jip_jobs>:
$> jip jobs
Take a look at the JIP command line wrapper <cli> that contains a list of all available JIP commands.
This is the famous hello world:
#!/usr/bin/env jip
# Greetings
# usage:
# hello.jip <name>
echo "Hello ${name}"
Make the file executable and you can use the JIP interpreter to run it or submit it:
$> chmod +x hello.jip
$> ./hello.jip Joe # runs the script
$> ./hello.jip Joe -- --dry --show # do not run but show dry run and command
#> ./hello.jip Joe -- submit # submit the script run
The combination of argument parsing and templates allows you to do much more. Please take a look at the Getting started guide and the templates documentation.