build_list(df, measure_list, weight_list, show_weightings=False, hide_rank_cols=True, return_df=False)¶
Construct a “hybrid” list ordering.
Note: first run the “prepare_master_list” function and use the output for the “df” input here.
Combine and sort various attributes according to variable multipliers to produce a list order. The list order output is based on a sliding scale of the priority assigned amoung the attributes.
The default output is a dataframe containing the new hybrid list order and employee numbers (empkey) only, and is written to disk as ‘dill/p_hybrid.pkl’.
The entire hybrid-sorted dataframe may be returned by setting the “return_df” input to True. This does not affect the hybrid list order dataframe - it is produced and stored regardless of the “return_df” option.
the prepared dataframe output of the prepare_master_list function
a list of attributes that form the basis of the final sorted list. The employee groups will be combined, sorted, and numbered according to these attributes one by one. Each time the current attribute numbered list is formed, a weighting is applied to that order column. The final result number will be the rank of the cummulative total of the weighted attribute columns.
a list of decimal weightings to apply to each corresponding measure within the measure_list. Normally the total of the weight_list should be 1, but any numbers may be used as weightings since the final result is a ranking of a cumulative total.
add columns to display the product of the weight/column mutiplcation
option to return the new sorted hybrid dataframe as output. Normally, the function produces a list ordering file which is written to disk and used as an input by the compute measures script.
remove the attrubute rank columns from the dataframe unless visual review is desired
compare_dataframes(base, compare, return_orphans=True, ignore_case=True, print_info=False, convert_np_timestamps=True)¶
Compare all common index and common column DataFrame values and report if any value is not equal in a returned dataframe.
Values are compared only by index and column label, not order. Therefore, the only values compared are within common index rows and common columns. The routine will report the common columns and any unique index rows when the print_info option is selected (True).
Inputs are pandas dataframes and/or pandas series.
This function works well when comparing initial data lists, such as those which may be received from opposing parties.
If return_orphans, returns tuple (diffs, base_loners, compare_loners), else returns diffs. diffs is a differential dataframe.
baseline dataframe or series
dataframe or series to compare against the baseline (base)
separately calculate and return the rows which are unique to base and compare
convert the column labels and column data to be compared to lowercase - this will avoid differences detected based on string case
option to print out to console verbose statistical information and the dataframe(s) instead of returning dataframe(s)
numpy returns datetime64 objects when the source is a datetime date-only object. this option will convert back to a date-only object for comparison.
Find the pandas dataframe index location of an array-like input of index labels.
Returns a list containing the index location(s).
dataframe - the index_values input is a subset of the dataframe index.
array-like collection of values which are a subset of the dataframe index
find_row_orphans(base_df, compare_df, col, ignore_case=True, print_output=False)¶
Given two columns (series) with the same column label in separate pandas dataframes, return values which are unique to one or the other column, not common to both series. Will also work with dataframe indexes.
Returns tuple (base_loners, compare_loners) if not print_output. These are dataframes with the series orphans.
Note: If there are orphans found that have identical values, they will both be reported. However, currently the routine will only find the first corresponding index location found and report that location for both orphans.
first dataframe to compare
second dataframe to compare
column label of the series to compare. routine will compare the dataframe indexes with the input of ‘index’.
convert col to lowercase prior to comparison
print results instead of returning results
find_series_locs(df, series_values, column_label)¶
Find the pandas dataframe index location of an array-like input of series values.
Returns a list containing the index location(s).
dataframe - the series_values input is a subset of one of the dataframe columns.
array-like collection of values which are a subset of one of the dataframe columns (the column_lable input)
the series within the pandas dataframe containing the series_values
names_to_integers(names, leading_precision=5, normalize_alpha=True)¶
convert a list or series of string names (i.e. last names) into integers for numerical sorting
Returns tuple (int_names, int_range, name_percentages)
List or pandas series containing strings for conversion to integers
Number of characters to use with full numeric precision, remainder of characters will be assigned a rounded single digit between 0 and 9
If True, insert ‘aaaaaaaaaa’ and ‘zzzzzzzzzz’ as bottom and top names. Otherwise, bottom and top names will be calculated from within the names input
an array of the name integers
the range of the name integers,
an array of corresponding percentages for each name integer relative to the range of name integers array
Note: This function demonstrates the possibility of constructing a list using any type or combination of attributes.
Add attribute columns to a master list. One or more of these columns will be used by the build_list function to construct a “hybrid” list ordering.
Employee groups must be listed in seniority order in relation to employees from the same group. Order between groups is uninmportant at this step.
New columns added: [‘age’, ‘s_lmonths’, ‘jnum’, ‘job_count’, ‘rank_in_job’, ‘jobp’, ‘eg_number’, ‘eg_spcnt’]
if True, lname strings are converted to an integer then a corresponding alpha-numeric percentage for constructing lists by last name. This is a demo only to show that any attribute may be used as a list weighting factor.
sort the master data dataframe doh and ldate columns prior to beginning any calculations. This sort has no effect on the other columns. The s_lmonths coulumn will be calculated on the sorted ldate data.
Job-related attributes are referenced to job counts from the settings dictionary.
sort_and_rank(df, col, tiebreaker1=None, tiebreaker2=None, reverse=False)¶
Sort a datframe by a specified attribute and insert a column indicating the resultant ranking. Tiebreaker inputs select columns to be used for secondary ordering in the event of value ties. Reverse ordering may be selected as an option.
- col (string)
dataframe column to sort
- tiebreaker1, tiebreaker2 (string(s))
second and third sort columns to break ties with primary col sort
- reverse (boolean)
If True, reverses sort (descending values)
sort_eg_attributes(df, attributes=['doh', 'ldate'], reverse_list=[0, 0], add_columns=False)¶
Sort master list attribute columns by employee group in preparation for list construction. The overall master list structure and order is unaffected, only the selected attribute columns are sorted (normally date-related columns such as doh or ldate)
The master data dataframe (does not need to be sorted)
columns to sort by eg (inplace)
If an attribute is to be sorted in reverse order (descending), use a ‘1’ in the list position corresponding to the position of the attribute within the attributes input
If True, an additional column for each sorted attribute will be added to the resultant dataframe, with the suffix ‘_sort’ added to it.
test_df_col_or_idx_equivalence(df1, df2, col=None)¶
check whether two dataframes contain the same elements (but not necessarily in the same order) in either the indexes or a selected column
- df1, df2
the dataframes to check
if not None, test this dataframe column for equivalency, otherwise test the dataframe indexes
Returns True or False