openpyxl-redmine workflow

Note

Below etest-workflow is used in the examples. To perform real work replace etest by the acronym of the Redmine project procedure to be worked with (e.g. pmo, erd, crd…).

Warning

Good practice is to start each session with your working Redmine project by using the workflow_in_redmine command to certify that it’s the right Redmine project.

Overview

The workflow module provides functional coupling between Table and Redmine representations of the workflow. The modification functions available appear to be symmetric and imply an equivalence of the representations, but this is not the case as not all information embedded in Redmine is contained in the Table. Most is, but metadata, e.g. description documentation, is not and one cannot extract a Table representation and inject it back without loosing this information. Additional programming effort could remedy this but it has not been thought necessary and one should look at Redmine as the master representation - which also reflects the expectation that developers will modify workflow state in Redmine and not in the Table. The Table’s role is therefore a management tool used for automating functional activities like workflow creation, batch changing multiple properties, etc.

Schematic showing Table-Redmine function coupling

Table-Redmine function coupling

The schematic shows the methods available for modifying Table and Redmine workflow representations. Management task aims determine which and the order of using the methods.

Create a new workflow requires creating a new Table and can be done by copying an existing table, changing it and running _validate until no ERRORs are reported.

Warning

Good practice is to _extract an example workflow from Redmine. All needed workflows should exist and be available in Redmine.

Creating a Redmine workflow instance once ready the workflow Table is injected to create the instance. Although manual _validate is advisable it will be performed automatically by _inject before starting to create tasks in Redmine.

Changing Redmine workflow content by _push aims at allowing managers to make multiple Redmine changes in a single operation.

Note

Use _preview to view pending _push Redmine changes without applying them.

Warning

Developers change Redmine content without updating Tables and it’s important that Table content is updated using _pull before making modifications.

Updating Table workflow content with _pull aligns the Table with Redmine changes made by developers.

Warning

Manual addition of tasks are not seen by _pull and it is necessary to perform _extract to include them.

Table schema change if Table definitions in code have to be changed, e.g. adding or removing a column, then _extract should be applied to all active workflows to align them to the new Table definition.

Workflow commands

Use the -h request to view module help text. Currently it shows:

$ etest-workflow -h

usage: etest-workflow [-h] [--username USERNAME]
                      password
                      {workflow_validate,workflow_compare,workflow_pull,
                      workflow_inject_with_dates ... workflow_in_redmine}
                      ...

positional arguments:
  password              if --username then password, else user's API key
  {workflow_validate,workflow_compare,workflow_pull,
  workflow_inject_with_dates ... workflow_in_redmine}

    workflow_validate   validate DATA workflow xlsx
    workflow_compare    compare workflows
    workflow_pull       update DATA workflow xlsx from Redmine
    workflow_inject_with_dates
                        inject workflow xlsx into Redmine with Task
                        start_date and due_date. Task Assignee and
                        TaskStatus column values are not touched inthe xlsx
    workflow_inject_bare
                        inject workflow xlsx into Redmine. Task Ticket,
                        StartDate, DueDate and TaskStatus column values are
                        cleared of their current values in the xlsx
    workflow_shift      shifts anchor workflow to new date
    workflow_push       push to DATA workflow xlsx
    workflow_preview    preview next push (w/o commit) to DATA workflow xlsx
    workflow_extract    create workflow xlsx
    workflow_delete     delete workflow in Redmine
    workflow_in_redmine
                        list workflows in Redmine

optional arguments:
  -h, --help            show this help message and exit
  --username USERNAME   account with access rights to EuXFEL Redmine

Workflow command parameters

Use the sub-command -h to view sub-command help text, e.g.

$ etest-workflow API_key  workflow_inject -h
usage: etest-workflow password workflow_inject [-h] filename anchorname date

positional arguments:
  filename    filename of xlsx, type <class 'str'>
  anchorname  name of project anchor Redmine task, type <class 'str'>
  date        2021-1-31 start date, type <class 'str'>

optional arguments:
  -h, --help  show this help message and exit

A cut-and-paste from source code listing for all sub-commands shows:

sub_commands = {
    'workflow_validate':
        {'help': 'validate DATA workflow xlsx',
         'params': [('filename', str, 'filename of xlsx')]},
    'workflow_compare':
        {'help': 'compare workflows',
         'params': [('id1', int, '1st anchor ticket ID'),
                    ('id2', int, '2nd anchor ticket ID')]},
    'workflow_pull':
        {'help': 'update DATA workflow xlsx from Redmine',
         'params': [('filename', str, 'filename of xlsx')]},
    'workflow_inject_with_dates':
        {'help': 'inject workflow xlsx into Redmine with Task '
                 'start_date and due_date. Task Assignee and'
                 'TaskStatus column values are not touched in'
                 'the xlsx',
         'params': [('filename', str, 'filename of xlsx'),
                    ('anchorname', str,
                     'name of project anchor Redmine task')]},
    'workflow_inject_bare':
        {'help': 'inject workflow xlsx into Redmine. '
                 'Task Ticket, StartDate, DueDate and TaskStatus '
                 'column values are cleared of their current values '
                 'in the xlsx',
         'params': [('filename', str, 'filename of xlsx'),
                    ('anchorname', str,
                     'name of project anchor Redmine task'),
                    ('date', str, '2021-1-31 start date')]},
    'workflow_shift':
        {'help': 'shifts anchor workflow to new date',
         'params': [('taskID', int, 'anchor (aka ticket) ID'),
                    ('date', str, '2021-1-31 target date')]},
    'workflow_push':
        {'help': 'push to DATA workflow xlsx',
         'params': [('filename', str, 'filename of xlsx')]},
    'workflow_preview':
        {'help': 'preview next push (w/o commit) to DATA workflow xlsx',
         'params': [('filename', str, 'filename of xlsx')]},
    'workflow_extract':
        {'help': 'create workflow xlsx',
         'params': [('filename', str, 'filename of xlsx'),
                    ('taskID', int, 'anchor (aka ticket) ID')]},
    'workflow_delete':
        {'help': 'delete workflow in Redmine',
         'params': [('id', int, 'id of anchor task')]},
    'workflow_in_redmine': {'help': 'list workflows in Redmine',
                            'params': []},
}

command execution authorization

Readers familiar with the Python argparse package will have seen above that two possibilities exist for authorization.

1. username and password authorization

$ etest-workflow --username youngman password workflow_in_redmine
Caching Redmine Tasks - Wait!
Redmine Caching Finished -  126 Tasks
87349 Add control support for "CTRL-EXAMPLE" devices
...

2. API key authorization

$ etest-workflow API_key workflow_in_redmine
Caching Redmine Tasks - Wait!
Redmine Caching Finished -  126 Tasks
87349 Add control support for "CTRL-EXAMPLE" devices
...

To set the API_key of a Redmine account navigate to https://in.xfel.eu/redmine/my/account and generated the API key, which is a string of ~30 alpha-numeric characters.

API key authorization is recommended and is used in the examples below.

Note

Define a bash shell variable helps avoid pasting in the API key manually: $ export API_key=”a1b2c3d4e5f….u6v7x8y9z” $ etest-workflow $API_key workflow_in_redmine Caching Redmine Tasks - Wait!

command execution exit status

Bash style exit status returns are used: 0 for SUCCESS and 1 for FAILURE.

Table column & Redmine task change on command matrix

Column _validate _inject_with_dates _inject_bare _extract _pull _push _preview
Nr       3      
Service   6 6 2   6  
Subject   1, 6 1, 6 2   6  
Type       2      
Issue       2      
Relatedissue(s)   7 7 2      
Relationship(s)   7 7 2      
HoldingTicket(s)   7 7 2      
HoldingRelationship(s)   9 7 2      
FTEdays   6   2      
Assignee 8 6?   2 4 6  
Actual FTEdays              
Ticket   5, 6 5, 6 2 4    
StartDate   5 10 2 4    
DueDate   5 10 2 4    
TaskStatus   5, 6   2 4 6  
Delay   7 7 2      
Category       2 4    
Other groups              
Comments              
Whatchers 8     2 4 6  

where:

  • 1 = sets command label into Table subjects double quotes replacing existing content
  • 2 = sets Table value to Redmine value
  • 3 = sets Table value to value derived for Redmine value
  • 4 = sets Table value to Redmine value if different
  • 5 = clears Table value
  • 6 = sets Redmine value to Table value if different
  • 7 = sets Redmine value to value derived from Table value
  • 8 = modify Table value, e.g. correct misspelt assignee name
  • 9 = ignore Table value and set blocked in Redmine
  • 10 = sets Redmine value to Table value for Tasks

workflow_in_redmine

Best practise on starting a work session is to list workflows using workflow_in_redmine and confirm that expected workflows are present.

$ etest-workflow API_key workflow_in_redmine
Caching Redmine Tasks - Wait!
Redmine Caching Finished -  126 Tasks
87349 Add control support for "CTRL-EXAMPLE" devices
87865 Add control support for "CTRL-EXAMPLE" devices
87911 Add control support for "ACME PRODUCT" devices
88089 Add control support for "CY_TED4015"
88176 Add control support for "xyz_groups"

workflow_extract

Constructs the Table from its Redmine representation.

etest-workflow API_key workflow_extract playground/filename.xlsx 86841

workflow_validate

Validates a spreadsheet’s content. It’s a good idea to validate the spreadsheet before using it with commands that interact with Redmine, but if you forget it will be called internally for those commands.

etest-workflow API_key workflow_validate playground/filename.xlsx

Identified problems are printed and need correcting!

workflow_preview

Previews actions to be applied to Redmine by workflow_push, but without committing the changes.

etest-workflow API_key workflow_preview playground/filename.xlsx

workflow_push

Pushes updates to Redmine tasks for column entries, see table below, which are different in the spreadsheet to the Redmine value.

etest-workflow API_key workflow_push playground/filename.xlsx

workflow_compare

Integration tests use the compare command to indicate that two workflows in Redmine are compatible. Compatibility means: same task task count, same sub-task structure, same start-due task duration lengths, and same relationship cound per task. This definition of compatibility does not guarantee a 100% match definition of compatibility but is sufficient for tests.

etest-workflow API_key workflow_compare 86786 86841

Warning

Manual addition or removal of Redmine tasks or their relationships in one workflow when compared to an initially identical workflow will result in comparison failure.

workflow_inject

Injecting a Table into Redmine is performed with either

  • workflow_inject_with_dates which uses dates in StartDate and DueDate Table columns to position the task in time. The with_dates variant is often used when creating a close-as-possible copy of a workflow which Table is created with workflow_extract. The 2nd parameter, below Label, is explained in the next variant.
  • workflow_inject_bare which sets all tasks to start with the start-date specified in the command line. The 2nd parameter, below Label, is inserted into task subject names by replacing the config between the insert characters (currently double quotes). The 3rd parameter is the anchor task’s start-date.
etest-workflow API_key workflow_inject_with_dates
    playground/final_FXE_PAM.xlsx Label

etest-workflow API_key workflow_inject_bare
    playground/filename.xlsx Label 2021-3-22
Original extracted workflow

Original extracted workflow

Injected bare workflow

Injected bare workflow

Injected with dates workflow

Injected with dates workflow

workflow_pull

Updates the Table for changes in Redmine and is driven by the Redmine ticket# in the Table.

etest-workflow
    API_key workflow_pull playground/filename.xlsx

Note

Adding, deleting or changing sub-task structure in Redmine will not be seen by _pull. To provide this functionality additional coding would be required.

workflow_shift

Shift moves all tasks in a Redmine workflow to a new date. The anchor task’s start-date will be the date specified in the command following the shift.

petest-workflow API_key workflow_shift 86786 2021-3-15

Note

Currently allowed shifts of the workflow are to the same day of the week. This ensures that tasks spanning a weekend retain their duration. In principle this is not required as workflow counts business days, but the current shift coding does not (as older code).

repair_redmine

Repair corrections are one-off realignments of Redmine tasks in place due to design changes of Table content definitions. Examples are: requiring two label markers in Anchor subjects, requiring ascii task subjects (they snook in with LO BOM), requiring NONE if the Responsible DATA team was specified as blank in the Table Service column, setting the duration time in hours to 0 if not set.

petest-workflow API_key repair_redmine

repair_table

Repair corrections are one-off realignments of Table rows due to design changes of Table content definitions.

petest-workflow API_key repair_table

workflow_delete

Not recommended, but sometimes it is necessary to delete a workflow. Redmine deletes all subtasks associated with the workflow task ticket# specified.

etest-workflow API_key workflow_delete 87160

Warning

Use with care!