jip.executils

The module contains the classes and methods that are used to execute jobs and handle streams of data between jobs.

Note

Usually you do not have to interact with this module directly. The jip.jobs.run() method deals with the construction of the pipe graphs for you.

A job group is a set of jobs that have to be executed together because data is piped between the jobs. We call set of jobs and their dependencies a dispatch graph. These graphs are created using this module. With such a graph the following scenarios can be resolved.

Single jobs:

A dispatch graph can consist of a single node that wraps a single job without any dependencies. In such case no pipelining and no redirection will happen.

Direct pipes:

Given two jobs A and B, a direct pipe is used between the process for A and the process for B. In addition, if A writes an output file in addition to a direct pipe to B, this is handled by the dispatcher.

Fan out:

Given three jobs, A, B, and C, where A’s output piped to both B and C in parallel.

The pipes are resolved using a disaptcher graph, wich can be created using the create_dispatcher_graph() function. The functions returns a sorted list of jip.executils.DispatcherNode instances. The dispatcher nodes are executable units that can be started with their run methods. They will run asynchroniously and you have to use the nodes wait method to wait for termination.

class jip.executils.DispatcherNode(job=None)

Node element of a dispatcher graph that handles pipes between jobs. A dispatcher node wraps around a single job in a dispatcher graph and is able to execute the job and wait for its termination.

run(profiler=False)

Run the job wrapped but this node.

Parameters:profiler – enable job profiling

wait()

Blocks until this nodes process is terminated and returns True if the process terminated with 0.

Returns:True if the job finished successfully

jip.executils.create_dispatcher_graph(job, _nodes=None)

Create a dispatcher graph for a given job. If the job does not have any pipe targets, a list with a single dispatcher node is returned, otherwise the dispatching graph is created from all the pipe target job.

Parameters:job – the job Type:jip.db.Job Returns:list of dispatcher nodes Return type:list of jip.executils.DispatcherNode instances