Tools.pbs.BaseQstat

Usage

Source

Tools.pbs.BaseQstat(
    job_id, infile=None, outfile=None, home=None, cmd=None, capture_json=False
)

Methods

Name Description
__init__() The BaseQstat class processes the output from the pbs command ‘qstat’. It
configure_data_file() Configure the primary data file by appending extra data to it. The csv header
identify_jobs() Qstat Parsing - Stage 1
identify_qstat_keywords() Qstat Parsing - Stage 2
parse_resource_list() Qstat Parsing - Stage 6
parse_to_int() Qstat Parsing - Stage 7
parse_to_unordered() Qstat Parsing - Optional Stage 8
parse_variable_list() Qstat Parsing - Stage 5
qstat_output() A function that calls qdstat via subprocess. The data is the list returned from
remove_whitespace() Qstat Parsing - Stage 3
run_qstat() This method runs the qstat command, generates qstat data, parses it into various formats,
static_data() This function takes a qstat dictionary and returns a dictionary that contains
static_data_to_yaml() This method saves the static data to a YAML file.
target_data() Filter out all of the qstat jobs other than the target job. This sets up the
to_csv() Convert a qstat dictionary to a csv file that contains data, which can be
to_dataframe() Convert a qstat dictionary to a dataframe that contains data, which can be
to_dict() The to_dict parser takes qstat data and parses it in 7 to 8 stages.
update_qstat_keywords() Qstat Parsing - Stage 4

__init__()

The BaseQstat class processes the output from the pbs command ‘qstat’. It

Usage

Source

__init__(
    job_id, infile=None, outfile=None, home=None, cmd=None, capture_json=False
)

specifically parses output from ‘qstat -f’, which displays a full status report for all of the jobs in the queue. Because each line of output per job consists of attribute_names and values for those attributes, the qstat data is parsed into a dictionary. The qstat data is then converted to csv format and saved in a .csv file. The Base class only process one job, and it only gathers data on one point in time.

Parameters
job_id: str.

The name of the PBS job_id to analyze.

infile: str. = None

The input file and the output file are used in tandem to determine the data file that will be used. If only one of these values are given (infile/outfile), then it will be used as the data file. If neither of these values are given, then a default file (“job_data.csv”) will be used. If both are given, then the infile data is appended to the outfile, which is used as the data file.

outfile: str. = None

See infile.

home: str. = None

An absolute path to the director where qstat data will be stored.

cmd: str. = None
The qstat command used to produce the job status report.

configure_data_file()

Configure the primary data file by appending extra data to it. The csv header

Usage

Source

configure_data_file(file, extra_data)

is only appended when the primary data file does not exist.

Parameters
file: str.

The absolute path to a csv file that will be updated.

extra_data: str.
The abolute path to a csv file that contains extra qstat data.

identify_jobs()

Qstat Parsing - Stage 1

Usage

Source

identify_jobs(job_data)

This essential first step takes the qstat data and identifies jobs. It groups the data based on the PBS job id that it corresponds to.

Parameters
job_data: list
The qstat output data in a readlines format (e.g. - one list item per line).
Returns
OrderedDict.
A dictionary that uses PBS job ids as keys.

identify_qstat_keywords()

Qstat Parsing - Stage 2

Usage

Source

identify_qstat_keywords(job_data, extra_keywords=None)

Taking data from Stage 1, this function further parses the qstat data by nesting the data and using the PBS keywords as keys. It also differentiates between single and multi-line PBS keywords/values.

Parameters
job_data: Mapping[str, Sequence[str]]

The output data from the ’identify_jobs` method.

extra_keywords: Sequence[str] | None = None
A list of extra keywords potentially missing from the OrthoEvols yaml_config file.
Returns
OrderedDict.
A dictionary with keywords that have been identified and assigned to the appropriate data.

parse_resource_list()

Qstat Parsing - Stage 6

Usage

Source

parse_resource_list(job_data)

Taking data from Stage 5, this method takes all of the PBS variables that contain Resource_List information and parses them into a dictionary.

Parameters
job_data: dict.
The output data from the parse_variable_list method.
Returns
OrderedDict.
The job data now has each resource list parsed.

parse_to_int()

Qstat Parsing - Stage 7

Usage

Source

parse_to_int(job_data)

Taking data from Stage 6, this method converts any strings that ONLY contain numbers and converts them to integers.

Parameters
job_data: dict.
The output data from the parse_resource_list method.
Returns
OrderedDict.
The job data will now have some values as integers.

parse_to_unordered()

Qstat Parsing - Optional Stage 8

Usage

Source

parse_to_unordered(job_data)

Taking data from Stage 7, this function converts all of the OrderedDicts in the job_data to standard dict objects.

Parameters
job_data: dict.
The output data from the parse_to_int method.
Returns
dict.
The data will now be a dict instead of OrderedDict.

parse_variable_list()

Qstat Parsing - Stage 5

Usage

Source

parse_variable_list(job_data)

Taking data from Stage 4, this method parses the Variable_List, which contains special PBS environment variables.

Parameters
job_data: dict.
The output data from the update_qstat_keywords method.
Returns
OrderedDict.
The job data now has each variable list parsed.

qstat_output()

A function that calls qdstat via subprocess. The data is the list returned from

Usage

Source

qstat_output(cmd, log_file, print_flag=False, capture_json=False)

readlines().

Parameters
cmd: str.

The qstat command used to generate qstat data. This is usually ‘qstat -f’

log_file: str.

The log file is a file that is used to save the output data gererated by the qstat command.

print_flag: bool. = False
A flag used to print the output of the qstat command.
Returns
list.
Output generated and read from the qstat command.

remove_whitespace()

Qstat Parsing - Stage 3

Usage

Source

remove_whitespace(job_data)

Taking data from Stage 2, this method removes any whitespace from the data including the 4x spaces, newline characters ( ), and tab characters ( ).

Parameters
job_data: dict.
The output data from the identify_qstat_keywords method.
Returns
OrderedDict.
The data no longer has unneeded whitespace.

run_qstat()

This method runs the qstat command, generates qstat data, parses it into various formats,

Usage

Source

run_qstat(csv_flag=True, sqlite_flag=False, ordered=False, capture_json=False)

and saves the data if desired.

Parameters
csv_flag: bool. = True

A flag that determines if the data is saved in a csv file.

sqlite_flag: bool. = False

A flag that determines if the data is saved in a sqlite database.

ordered: bool. = False
A flag that controls whether or not the parsed qstat data will be returned as an ordered dictionary or a standard dict.

static_data()

This function takes a qstat dictionary and returns a dictionary that contains

Usage

Source

static_data(qstat_dict, target_job)

“static” data related to the job of interest.

Parameters
qstat_dict: dict.

Qstat data that has been parsed into a dictionary.

target_job: str.
The target job that’s being analyzed.
Returns
dict.
A dictionary that contains the static data of the target job.

static_data_to_yaml()

This method saves the static data to a YAML file.

Usage

Source

static_data_to_yaml(file, qstat_dict, target_job, overwrite=False)
Parameters
file: str.

The absolute path of the yaml file.

qstat_dict: dict.

Qstat data that has been parsed into a dictionary.

target_job: str.

The target job that’s being analyzed.

overwrite: bool. = False
A flag to determine weather the yaml file will be overwritten.

target_data()

Filter out all of the qstat jobs other than the target job. This sets up the

Usage

Source

target_data(qstat_dict, target_job)

class variables for the target dataframe and target dictionary.

    :param qstat_dict:  Qstat data that has been parsed into a dictionary.
Parameters
target_job: str.
The target job that’s being analyzed.
Returns
dict.
A dictionary that contains all of the target job’s static and dynamic data.

to_csv()

Convert a qstat dictionary to a csv file that contains data, which can be

Usage

Source

to_csv(file, qstat_dict, target_job, overwrite=False)

plotted.

Parameters
file: str.

The absolute path of the csv file.

qstat_dict: dict.

Qstat data that has been parsed into a dictionary.

target_job: str.

The target job that’s being analyzed.

overwrite: bool. = False
A flag to determine weather the csv file will be overwritten.

to_dataframe()

Convert a qstat dictionary to a dataframe that contains data, which can be

Usage

Source

to_dataframe(qstat_dict, target_job)

plotted.

Parameters
qstat_dict: dict.

Qstat data that has been parsed into a dictionary.

target_job: str.

The target job that’s being analyzed.

python_datetime: str.
The date and time that the qstat data was collected.
Returns
pd.DataFrame
A dataframe that contains dynamic data.

to_dict()

The to_dict parser takes qstat data and parses it in 7 to 8 stages.

Usage

Source

to_dict(qstat_data, ordered=True)

The final product is a nested dictionary that uses PBS job ids as keys.

Parameters
job_data: list

The qstat output data in a readlines format (e.g. - one list item per line).

ordered: bool. = True
A flag that controls whether or not the parsed qstat data will be returned as an ordered dictionary or a standard dict.
Returns
dict.
A dictionary of jobs.

update_qstat_keywords()

Qstat Parsing - Stage 4

Usage

Source

update_qstat_keywords(job_data)

Taking data from Stage 3, this method further parses the qstat data by removing any of the leading keyword data since the dictionary keys have already taken advantage of this information.

Parameters
job_data: dict.
The output data from the remove_whitespace method.
Returns
OrderedDict.
A cleaner version of the data that doesn’t contain any redundancies in the values.