jip.profiles

JIP module that handles job profiles.

A job profile contains all compute-cluster and execution related meta-data of a job, such as the number of threads reserved for the job or the time limit. Profiles can be named and stored in the user configuration.

In addition, hierarchical updates of profiles can be applied. For example, a default profile can be loaded from the configuration. This profile can then be refined by a pipeline script or command line options.

This enable you to start with a hard-coded profile in your tool implementation and then gradually modify and change the profile when the tool is embedded in another pipeline or from the command line at execution or submission time.

Note

Please note that the interpretation of some of the profiles properties depends on the cluster implementation.

The following properties are supported by a profile and can be maintained and updated.

General properties

The following properties are considered general and usually always used and interpreted, independent of where and how you execute the tool or pipeline:

name
You can assign an arbitrary name to your profiles. This name will be used either as a job name, if the profile is applied to a tool, or as a pipeline name if applied to a pipeline.
prefix
A name prefix that is applied to all embedded jobs. This can be useful if, in a pipeline context, you want to allow your tool to take their own name, but you want to prefix all tools that are part of a single pipeline.
threads
The number of threads or compute slots allocated by the execution. Although this property and its interpretation also depends on the cluster or grid implementation, this is considered a general property that is also considered when you execute a pipeline or tool outside of a compute grid.
working_dir or dir
The working directory for a job. This is initialized to the current working directory of the process that creates the profile.
temp
A boolean property that you can used to mark a job as temporary. Temporary jobs are treated specially in a pipeline execution. You can find more information about temporary jobs in the Pipeline documentation.
env
Dictionary that can be used to extend the jobs shell environment
description
Optional field that describes the profile and can be used to describe custom profiles in the user configuration

Cluster/Grid specific properties

The following properties can be set or modified, but their interpretation depends on the cluster implementation and the capabilities of the cluster:

tasks
Number of tasks assigned to a single job
tasks_per_node
If multiple nodes are reserved by a single job, this is the number of tasks assigned to each node.
nodes
Number of nodes requested by the job
queue
The queue the job is sent to
priority
A priority assigned to a job
environment
The name of the environment assigned to a job. This is not the shell environment, but an arbitrary name that is used, for example, in the Sun Grid Engine implementation to identify the parallel environment the job is submitted to.
account
Name of the account for this job
mem
The memory limit for the job. This is stored here as a string and passed on as is to the cluster implementation
time
The time limit for the job. Here, the time limit is specified as a string and passed on to the cluster implementation as is.
out
Path to the stdout log file for this job
log
path to the stderr log file for this job
err
path to the stderr log file for this job
extra
This is an array that takes additional options that are used when the submission command is constructed.

Note

Most of the

class jip.profiles.Profile(name=None, threads=None, nodes=None, tasks=None, tasks_per_node=None, environment=None, time=None, queue=None, priority=None, log=None, out=None, account=None, mem=0, extra=None, profile=None, prefix=None, temp=False, _load=True, env=None, tool_name=None, working_dir=None, description=None, specs=None, _name=None, **kwargs)

A Profile contains cluster and runtime specific information about a job.

apply(job, pipeline=False, overwrite=False)

Apply this profile to the given job.

apply_overwrite(job)

Apply the profile and overwrite all settings that are set in this profile

apply_to_pipeline(pipeline)

Apply this profile to the pipeline

Parameters:pipeline (jip.pipeline.Pipeline) – the pipeline
dir

Set the jobs working directory

Getter:access the jobs working directory
Setter:set the jobs working directory
Type:string
err

Set the jobs error log file

Getter:access the jobs name
Setter:set the jobs name
Type:string
classmethod from_dict(data)

Load a profile from a dictionary

classmethod from_file(file_name)

Load a profile from a json file

Parameters:file_name – the name of the input file
classmethod from_job(job)

Create a profile based on a given job. All properties are set according to the given job, except the jobs temp state, which will be kept unmodified.

Parameters:job – the job
Returns:new profile generated from the job
load(profile_name)

Set this profiles values to the values loaded from the profile stored under the given name. An exception is raised if no profile of that name could be found.

Parameters:profile_name (string) – the name of the profile that will be loaded
load_args(args)

Update this profile from the given dictionary of command line arguments. The argument names must match the profile attributes

merge(master)

Merge this profile with the given master profile.

Currently this merges the working directory of jobs

Parameters:master – the master profile
name

Set the jobs name

Getter:access the jobs name
Setter:set the jobs name
Type:string
update(profile, overwrite=True)

Update this profile from a given profile. All values that are not None in the other profile are applied to this profile

Parameters:
  • profile (Profile) – the other profile
  • overwrite – if True, value will be set regardless. Otherwise, the new value will only be applied if the old value is None
jip.profiles.get(name='default', tool=None)

Load a profile by name. If tool is specified, the specs are searched to the tool and if found, the specs are applied.

jip.profiles.get_specs(path=None)

Load specs form default locations and then update from specs in given path if specified.

Parameters:path – optional path to an additional spec file
jip.profiles.specs = None

global specs

Fork me on GitHub