JIP module that handles job profiles.
A job profile contains all compute-cluster and execution related meta-data of a job, such as the number of threads reserved for the job or the time limit. Profiles can be named and stored in the user configuration.
In addition, hierarchical updates of profiles can be applied. For example, a default profile can be loaded from the configuration. This profile can then be refined by a pipeline script or command line options.
This enable you to start with a hard-coded profile in your tool implementation and then gradually modify and change the profile when the tool is embedded in another pipeline or from the command line at execution or submission time.
Note
Please note that the interpretation of some of the profiles properties depends on the cluster implementation.
The following properties are supported by a profile and can be maintained and updated.
The following properties are considered general and usually always used and interpreted, independent of where and how you execute the tool or pipeline:
- name
- You can assign an arbitrary name to your profiles. This name will be used either as a job name, if the profile is applied to a tool, or as a pipeline name if applied to a pipeline.
- prefix
- A name prefix that is applied to all embedded jobs. This can be useful if, in a pipeline context, you want to allow your tool to take their own name, but you want to prefix all tools that are part of a single pipeline.
- threads
- The number of threads or compute slots allocated by the execution. Although this property and its interpretation also depends on the cluster or grid implementation, this is considered a general property that is also considered when you execute a pipeline or tool outside of a compute grid.
- working_dir or dir
- The working directory for a job. This is initialized to the current working directory of the process that creates the profile.
- temp
- A boolean property that you can used to mark a job as temporary. Temporary jobs are treated specially in a pipeline execution. You can find more information about temporary jobs in the Pipeline documentation.
- env
- Dictionary that can be used to extend the jobs shell environment
- description
- Optional field that describes the profile and can be used to describe custom profiles in the user configuration
The following properties can be set or modified, but their interpretation depends on the cluster implementation and the capabilities of the cluster:
- tasks
- Number of tasks assigned to a single job
- tasks_per_node
- If multiple nodes are reserved by a single job, this is the number of tasks assigned to each node.
- nodes
- Number of nodes requested by the job
- queue
- The queue the job is sent to
- priority
- A priority assigned to a job
- environment
- The name of the environment assigned to a job. This is not the shell environment, but an arbitrary name that is used, for example, in the Sun Grid Engine implementation to identify the parallel environment the job is submitted to.
- account
- Name of the account for this job
- mem
- The memory limit for the job. This is stored here as a string and passed on as is to the cluster implementation
- time
- The time limit for the job. Here, the time limit is specified as a string and passed on to the cluster implementation as is.
- out
- Path to the stdout log file for this job
- log
- path to the stderr log file for this job
- err
- path to the stderr log file for this job
- extra
- This is an array that takes additional options that are used when the submission command is constructed.
Note
Most of the
A Profile contains cluster and runtime specific information about a job.
Apply this profile to the given job.
Apply the profile and overwrite all settings that are set in this profile
Apply this profile to the pipeline
Parameters: | pipeline (jip.pipeline.Pipeline) – the pipeline |
---|
Set the jobs working directory
Getter: | access the jobs working directory |
---|---|
Setter: | set the jobs working directory |
Type: | string |
Set the jobs error log file
Getter: | access the jobs name |
---|---|
Setter: | set the jobs name |
Type: | string |
Load a profile from a dictionary
Load a profile from a json file
Parameters: | file_name – the name of the input file |
---|
Create a profile based on a given job. All properties are set according to the given job, except the jobs temp state, which will be kept unmodified.
Parameters: | job – the job |
---|---|
Returns: | new profile generated from the job |
Set this profiles values to the values loaded from the profile stored under the given name. An exception is raised if no profile of that name could be found.
Parameters: | profile_name (string) – the name of the profile that will be loaded |
---|
Update this profile from the given dictionary of command line arguments. The argument names must match the profile attributes
Merge this profile with the given master profile.
Currently this merges the working directory of jobs
Parameters: | master – the master profile |
---|
Set the jobs name
Getter: | access the jobs name |
---|---|
Setter: | set the jobs name |
Type: | string |
Update this profile from a given profile. All values that are not None in the other profile are applied to this profile
Parameters: |
|
---|
Load a profile by name. If tool is specified, the specs are searched to the tool and if found, the specs are applied.
Load specs form default locations and then update from specs in given path if specified.
Parameters: | path – optional path to an additional spec file |
---|
global specs