## Tools.pbs.BaseQstat


Usage

``` python
Tools.pbs.BaseQstat(
    job_id, infile=None, outfile=None, home=None, cmd=None, capture_json=False
)
```


## Methods

| Name | Description |
|----|----|
| [__init__()](#__init__) | The BaseQstat class processes the output from the pbs command 'qstat'. It |
| [configure_data_file()](#configure_data_file) | Configure the primary data file by appending extra data to it. The csv header |
| [identify_jobs()](#identify_jobs) | Qstat Parsing - Stage 1 |
| [identify_qstat_keywords()](#identify_qstat_keywords) | Qstat Parsing - Stage 2 |
| [parse_resource_list()](#parse_resource_list) | Qstat Parsing - Stage 6 |
| [parse_to_int()](#parse_to_int) | Qstat Parsing - Stage 7 |
| [parse_to_unordered()](#parse_to_unordered) | Qstat Parsing - Optional Stage 8 |
| [parse_variable_list()](#parse_variable_list) | Qstat Parsing - Stage 5 |
| [qstat_output()](#qstat_output) | A function that calls qdstat via subprocess. The data is the list returned from |
| [remove_whitespace()](#remove_whitespace) | Qstat Parsing - Stage 3 |
| [run_qstat()](#run_qstat) | This method runs the qstat command, generates qstat data, parses it into various formats, |
| [static_data()](#static_data) | This function takes a qstat dictionary and returns a dictionary that contains |
| [static_data_to_yaml()](#static_data_to_yaml) | This method saves the static data to a YAML file. |
| [target_data()](#target_data) | Filter out all of the qstat jobs other than the target job. This sets up the |
| [to_csv()](#to_csv) | Convert a qstat dictionary to a csv file that contains data, which can be |
| [to_dataframe()](#to_dataframe) | Convert a qstat dictionary to a dataframe that contains data, which can be |
| [to_dict()](#to_dict) | The to_dict parser takes qstat data and parses it in 7 to 8 stages. |
| [update_qstat_keywords()](#update_qstat_keywords) | Qstat Parsing - Stage 4 |

------------------------------------------------------------------------


#### \_\_init\_\_()


The BaseQstat class processes the output from the pbs command 'qstat'. It


Usage

``` python
__init__(
    job_id, infile=None, outfile=None, home=None, cmd=None, capture_json=False
)
```


specifically parses output from 'qstat -f', which displays a full status report for all of the jobs in the queue. Because each line of output per job consists of attribute_names and values for those attributes, the qstat data is parsed into a dictionary. The qstat data is then converted to csv format and saved in a .csv file. The Base class only process one job, and it only gathers data on one point in time.


##### Parameters


`job_id: str.`  
The name of the PBS job_id to analyze.

`infile: str. = None`  
The input file and the output file are used in tandem to determine the data file that will be used. If only one of these values are given (infile/outfile), then it will be used as the data file. If neither of these values are given, then a default file ("job_data.csv") will be used. If both are given, then the infile data is appended to the outfile, which is used as the data file.

`outfile: str. = None`  
See infile.

`home: str. = None`  
An absolute path to the director where qstat data will be stored.

`cmd: str. = None`  
The qstat command used to produce the job status report.


------------------------------------------------------------------------


#### configure_data_file()


Configure the primary data file by appending extra data to it. The csv header


Usage

``` python
configure_data_file(file, extra_data)
```


is only appended when the primary data file does not exist.


##### Parameters


`file: str.`  
The absolute path to a csv file that will be updated.

`extra_data: str.`  
The abolute path to a csv file that contains extra qstat data.


------------------------------------------------------------------------


#### identify_jobs()


Qstat Parsing - Stage 1


Usage

``` python
identify_jobs(job_data)
```


This essential first step takes the qstat data and identifies jobs. It groups the data based on the PBS job id that it corresponds to.


##### Parameters


`job_data: list`  
The qstat output data in a `readlines` format (e.g. - one list item per line).


##### Returns


`OrderedDict.`  
A dictionary that uses PBS job ids as keys.


------------------------------------------------------------------------


#### identify_qstat_keywords()


Qstat Parsing - Stage 2


Usage

``` python
identify_qstat_keywords(job_data, extra_keywords=None)
```


Taking data from Stage 1, this function further parses the qstat data by nesting the data and using the PBS keywords as keys. It also differentiates between single and multi-line PBS keywords/values.


##### Parameters


`job_data: Mapping[str, Sequence[str]]`  
The output data from the 'identify_jobs\` method.

`extra_keywords: Sequence[str] | None = None`  
A list of extra keywords potentially missing from the OrthoEvols yaml_config file.


##### Returns


`OrderedDict.`  
A dictionary with keywords that have been identified and assigned to the appropriate data.


------------------------------------------------------------------------


#### parse_resource_list()


Qstat Parsing - Stage 6


Usage

``` python
parse_resource_list(job_data)
```


Taking data from Stage 5, this method takes all of the PBS variables that contain Resource_List information and parses them into a dictionary.


##### Parameters


`job_data: dict.`  
The output data from the parse_variable_list method.


##### Returns


`OrderedDict.`  
The job data now has each resource list parsed.


------------------------------------------------------------------------


#### parse_to_int()


Qstat Parsing - Stage 7


Usage

``` python
parse_to_int(job_data)
```


Taking data from Stage 6, this method converts any strings that ONLY contain numbers and converts them to integers.


##### Parameters


`job_data: dict.`  
The output data from the parse_resource_list method.


##### Returns


`OrderedDict.`  
The job data will now have some values as integers.


------------------------------------------------------------------------


#### parse_to_unordered()


Qstat Parsing - Optional Stage 8


Usage

``` python
parse_to_unordered(job_data)
```


Taking data from Stage 7, this function converts all of the OrderedDicts in the job_data to standard dict objects.


##### Parameters


`job_data: dict.`  
The output data from the parse_to_int method.


##### Returns


`dict.`  
The data will now be a dict instead of OrderedDict.


------------------------------------------------------------------------


#### parse_variable_list()


Qstat Parsing - Stage 5


Usage

``` python
parse_variable_list(job_data)
```


Taking data from Stage 4, this method parses the Variable_List, which contains special PBS environment variables.


##### Parameters


`job_data: dict.`  
The output data from the update_qstat_keywords method.


##### Returns


`OrderedDict.`  
The job data now has each variable list parsed.


------------------------------------------------------------------------


#### qstat_output()


A function that calls qdstat via subprocess. The data is the list returned from


Usage

``` python
qstat_output(cmd, log_file, print_flag=False, capture_json=False)
```


readlines().


##### Parameters


`cmd: str.`  
The qstat command used to generate qstat data. This is usually 'qstat -f'

`log_file: str.`  
The log file is a file that is used to save the output data gererated by the qstat command.

`print_flag: bool. = ``False`  
A flag used to print the output of the qstat command.


##### Returns


`list.`  
Output generated and read from the qstat command.


------------------------------------------------------------------------


#### remove_whitespace()


Qstat Parsing - Stage 3


Usage

``` python
remove_whitespace(job_data)
```


Taking data from Stage 2, this method removes any whitespace from the data including the 4x spaces, newline characters ( ), and tab characters ( ).


##### Parameters


`job_data: dict.`  
The output data from the identify_qstat_keywords method.


##### Returns


`OrderedDict.`  
The data no longer has unneeded whitespace.


------------------------------------------------------------------------


#### run_qstat()


This method runs the qstat command, generates qstat data, parses it into various formats,


Usage

``` python
run_qstat(csv_flag=True, sqlite_flag=False, ordered=False, capture_json=False)
```


and saves the data if desired.


##### Parameters


`csv_flag: bool. = ``True`  
A flag that determines if the data is saved in a csv file.

`sqlite_flag: bool. = ``False`  
A flag that determines if the data is saved in a sqlite database.

`ordered: bool. = ``False`  
A flag that controls whether or not the parsed qstat data will be returned as an ordered dictionary or a standard dict.


------------------------------------------------------------------------


#### static_data()


This function takes a qstat dictionary and returns a dictionary that contains


Usage

``` python
static_data(qstat_dict, target_job)
```


"static" data related to the job of interest.


##### Parameters


`qstat_dict: dict.`  
Qstat data that has been parsed into a dictionary.

`target_job: str.`  
The target job that's being analyzed.


##### Returns


`dict.`  
A dictionary that contains the static data of the target job.


------------------------------------------------------------------------


#### static_data_to_yaml()


This method saves the static data to a YAML file.


Usage

``` python
static_data_to_yaml(file, qstat_dict, target_job, overwrite=False)
```


##### Parameters


`file: str.`  
The absolute path of the yaml file.

`qstat_dict: dict.`  
Qstat data that has been parsed into a dictionary.

`target_job: str.`  
The target job that's being analyzed.

`overwrite: bool. = ``False`  
A flag to determine weather the yaml file will be overwritten.


------------------------------------------------------------------------


#### target_data()


Filter out all of the qstat jobs other than the target job. This sets up the


Usage

``` python
target_data(qstat_dict, target_job)
```


class variables for the target dataframe and target dictionary.

    :param qstat_dict:  Qstat data that has been parsed into a dictionary.


##### Parameters


`target_job: str.`  
The target job that's being analyzed.


##### Returns


`dict.`  
A dictionary that contains all of the target job's static and dynamic data.


------------------------------------------------------------------------


#### to_csv()


Convert a qstat dictionary to a csv file that contains data, which can be


Usage

``` python
to_csv(file, qstat_dict, target_job, overwrite=False)
```


plotted.


##### Parameters


`file: str.`  
The absolute path of the csv file.

`qstat_dict: dict.`  
Qstat data that has been parsed into a dictionary.

`target_job: str.`  
The target job that's being analyzed.

`overwrite: bool. = ``False`  
A flag to determine weather the csv file will be overwritten.


------------------------------------------------------------------------


#### to_dataframe()


Convert a qstat dictionary to a dataframe that contains data, which can be


Usage

``` python
to_dataframe(qstat_dict, target_job)
```


plotted.


##### Parameters


`qstat_dict: dict.`  
Qstat data that has been parsed into a dictionary.

`target_job: str.`  
The target job that's being analyzed.

`python_datetime: str.`  
The date and time that the qstat data was collected.


##### Returns


`pd.DataFrame`  
A dataframe that contains dynamic data.


------------------------------------------------------------------------


#### to_dict()


The to_dict parser takes qstat data and parses it in 7 to 8 stages.


Usage

``` python
to_dict(qstat_data, ordered=True)
```


The final product is a nested dictionary that uses PBS job ids as keys.


##### Parameters


`job_data: list`  
The qstat output data in a `readlines` format (e.g. - one list item per line).

`ordered: bool. = ``True`  
A flag that controls whether or not the parsed qstat data will be returned as an ordered dictionary or a standard dict.


##### Returns


`dict.`  
A dictionary of jobs.


------------------------------------------------------------------------


#### update_qstat_keywords()


Qstat Parsing - Stage 4


Usage

``` python
update_qstat_keywords(job_data)
```


Taking data from Stage 3, this method further parses the qstat data by removing any of the leading keyword data since the dictionary keys have already taken advantage of this information.


##### Parameters


`job_data: dict.`  
The output data from the remove_whitespace method.


##### Returns


`OrderedDict.`  
A cleaner version of the data that doesn't contain any redundancies in the values.