JIP Pipeline system

JIP is another approach to implement a pipeline system that helps to manage a large number of jobs on a compute cluster. It simplifies the process of creating computational pipelines with dependency support, automatic expansions and simplified management of jobs and resources.

Even if you are not interested in pipeline and dependencies; JIP offers a set of management commands to simplify interactions with your compute cluster via allowing you to submit commands quickly, to restart and to move jobs, to edit commands interactively, to avoid duplicated job submission and more.

Quick start

There is more documentation available and there are a few things you might need to understand before you create more sophisticated work-flows. But here is the quick-start to install and to configure JIP on your system. Take a look at the setup guide for more detailed description of the installation and configuration process.

Installation

You can install JIP quickly into your $HOME folder:

$~> pip install jip --user

Configuration

JIP reads $HOME/.jip/jip.json to load your configuration. Create the file with the following content.

For a Slurm cluster:

{
    "cluster": "jip.cluster.Slurm"
}

For a PBS/Torque cluster:

{
    "cluster": "jip.cluster.PBS"
}

For a Gridengine/SGE/OGE cluster:

{
    "cluster": "jip.cluster.SGE",
    "sge" : {
        "threads_pe": "threads"
    }
}

Please note that in order to submit multi-threaded jobs for SGE, you have to specify the parallel environment that is configured for threaded jobs.

For a Platform LSF or Openlava cluster:

{
    "cluster": "jip.cluster.LSF"
}

For the JIP local scheduler:

{
    "cluster": "jip.grids.JIP"
}

In order to use the local scheduler that ships with JIP, please check the documention on how you can configure and start the JIP scheduler on you local machine.

Run or submit commands

You can run or submit a bash command using the jip bash <jip_bash> command. For example:

$> jip bash -c hostname

This will run hostname locally. Add the -s option to submit the command as a job to your cluster:

$> jip bash -s -c hostname

You can check the status of the job with jip jobs <jip_jobs>:

$> jip jobs

Take a look at the JIP command line wrapper <cli> that contains a list of all available JIP commands.

Write a JIP Script

This is the famous hello world:

#!/usr/bin/env jip
# Greetings
# usage:
#     hello.jip <name>

echo "Hello ${name}"

Make the file executable and you can use the JIP interpreter to run it or submit it:

$> chmod +x hello.jip
$> ./hello.jip Joe  # runs the script
$> ./hello.jip Joe -- --dry --show # do not run but show dry run and command
#> ./hello.jip Joe -- submit  # submit the script run

The combination of argument parsing and templates allows you to do much more. Please take a look at the Getting started guide and the templates documentation.

Where to go next

The Getting started guide
goes through a couple of examples, explains basic tools and pipeline generation.
The Tools and Pipelines chapter
documents the script and template API and how the execution graph and be manipulated.
The Examples chapter
describes a set of real world examples.
The Setup and configuration chapter
explains the JIP installation and configuration options in more detail.
The JIP command line utilities
are bundles with the package and allow you to run, submit and manage your jobs.

Indices and tables

Table Of Contents

Links

This Page

Fork me on GitHub