PTC - Redmine tooling

Tooling aims

The PTC role requires a uniform interface to the various project procedures which, as detailed previously, are provided by Redmine. The extra tooling required aims at:

  • applying rules to workflow creation and operation, e.g. task names, task and sub-task structures
  • automating activity that is repetitive, e.g. creating workflow task structures
  • levering the information in Redmine to remove secretary work, e.g. produce status summaries and agendas for meetings.

The Rules, Automation, and Levering aim will be a key feature of the tooling provided and the RAL concept will accompany much of the work required by PTC. RAL is a good catch-phrase, reality may fall short.

The above are typical management requirements on project management software and presumably available in Redmine, but may require additional plugins or the Redmine Pro installation which are not available to us. Listening to comments made by project procedure contacts and ITDM experts, this seems to be the case. Therefore keeping to the fine tune rather than change direction the decision was made to investigate providing the tooling using external python tools, which had the advantage of seeing how to work with Redmine from a new viewpoint. The work model which crystallized out being:

  • use the Redmine browser interface for single task activities typical of those performed by assigned task workers (developers)
  • use the python tooling for assisting manager activities
  • PTC uses the tooling to simplify viewing of projects in multiple procedure to enable summary and schedule work

Tooling status

Usable but in development is a fitting description. Most of the features required to develop a script based interface toolset have been worked with. Next steps are focus on code cleaning, fine tuning development of the tooling and interface working with the PEPs, and aligning Redmine access to the PTC-PEP requirements. The final step is to see if the PTC role of coordinating project resources, that is summarizing and efficient scheduling, can be achieved.

Python-Redmine library

Redmine.org’s Restful-API. lists a number of candidate libraries.

PTC uses Python-Redmine as this is referred to as a library which supports 100% features of Redmine’s REST API on the Redmine.org site. It provides a simple but powerful Pythonic API inspired by a well-known Django ORM and is thoroughly tested.

Installing the library and needed extras

To get a working environment:

  • python-redmine, openPyxl and numpy libraries must be installed into with Python 3.5+
  • access to EuXFEL’s Gitlab PTC project must be established and Git-client s/w installed.

Linux

Most Unix users will be familiar with the pip installation of python packages, simply:

  • pip install python-redmine
  • pip install openpyxl (for Excel-Python)
  • pip install numpy (for extra functionality, e.g. workdays per week)

An attractive way of getting 2-for-1 is to install the Karabo GUI, and perform the above pip installs in it.

Git seems to come with Ubuntu, for me this client works:

$ git --version
git version 2.17.1

Windows 10

Finally a Windows 10 Python installation from scratch could also be used by perform the following steps.

  • To get a running installation of python installed on windows 10 install MiniConda from conda For my test the MiniConda location was C:\Users\account\Miniconda3 and Python version 3.8 was installed. For a user-friendly Python on Windows intro try here
  • Now hit the windows button, the one between Control and Alt on English keyboards, under ‘A’ the entry Anaconda3 (64 bit) use the pulldown menu to select Anaconda Prompt (miniconda3) and start a prompt terminal. The terminal knows where the python is which you’ve just installed.
  • To test but also see which packages are installed with miniconda enter ‘pip list’. Pip is there, but not python-redmine.
  • Install python-redmine and the extra pips listed above by performing the pip installs.

Note where the Miniconda installation is (e.g. C:UsersusernameMiniConda)

A nicely featured Git-client can be installed from git-client

Git for Windows fly sheet

Git for Windows fly sheet.

To install:

  • download the exe and follow the instructions taking “next” recommended answers all the way.
  • hit the windows button and start the “Git Bash” shell terminal. This is magnificent - it’s a Windows CLI!
  • now download the PTC project’s script and templates projects from Gitlab into a folder of your choice (this assumes you’ve been added as a member to the PTC project in EuXFEL’s Gitlab). In the example below the folder is created with mkdir, but you could have added the folder using the usual windows new folder method.
$ pwd
$ mkdir pep_stuff
$ cd pep_stuff
$ git clone xyz

Where xyz you need to find by looking in https://git.xfel.eu/gitlab/ptc/scripts for the clone https address of the scripts sub-project, currently (but may change) it’s “https://git.xfel.eu/gitlab/ptc/scripts.git”

To use the Miniconda Python and pip extras simply add the path to it at the front of the PATH and check that pep_tool.py runs with:

$ export PATH="/c/Users/username/MiniConda:"$PATH
$ python pep_tool.py your-username your-password -h

If successful you should see the pep_tool.py sub-commands listed, see below.

pep_tool.py application

pep_tool aims at providing RAL functionality to PTC. As a first interaction with Python-Redmine and its interface to Redmine’s REST API it acts as a trial-and-error testbed, with stable functionality exposed and provisional or dangerous functionality not.

A look at the code will show how unfinished the application is.

  • the config(uration) file is foreseen for ease customizing to the different procedures. Additional work is required.
  • the API used for Redmine access has no session context and one has to remember this when using thge application.
  • the trial-and-error testing of the API have left some half-baked ideas in the code, in particular the slim representation of Redmine issues which is foreseen as a way of handling missing property values and similar.

pep_tool configuration

Difference between procedures, e.g. the Redmine container project-id, are configured in a json formated configuration file.

pep_tool sub-command functionality groups

  • ERD and CRD management: workflow_validate, workflow_inject, workflow_update
  • PMO and XO cost-estimate management: cost_estimate_validate, cost_estimate_inject, cost_estimate_update.
  • task snapshot: diff_snapshot, save_snapshot
  • property query related: dump_custom_fields, dump_project, dump_sets, dump_tracker_statuses
  • issue related: delete_issue, force_tracker, peek, set_status, traverse_issue
  • slim related: slims_all, slims_ctrl, slims_without_parent, traverse_slim
  • general workflow handling: add_anchor, add_request, dump_signatures, request_to_anchor, run_test
  • summary and meeting related: meeting, summary

pep_tool sub-command options

Currently everything is in one python script - terrible but this will be broken down soon - but has the advantage of having only one place to look at!

pep_tool uses argparse to provide a user-friendly command-line interfaces which validates the arguments used, performs input string to type conversion, and provides help and usage messages and issues errors when give invalid arguments. Additionally its use allows only tested and non-dangerous functionality to be exposed.

$ python pep_tool.py youngman password -h
usage: pep_tool.py [-h]
               username password
               {add_anchor,add_request,delete_issue,diff_snapshot,
               dump_custom_fields,dump_project,dump_sets,dump_signatures,
               ...
               summary,traverse_issue,traverse_slim}
               ...

positional arguments:
    username              account with access rights to EuXFEL Redmine
    password              account password
    {add_anchor,add_request,delete_issue,diff_snapshot,dump_custom_fields,
    ...
    summary,traverse_issue,traverse_slim}

    add_anchor          add anchor task to project
    add_request         add request task to project
    delete_issue        delete Redmine task
    diff_snapshot       output differences current Slims to those recorded in
                        last save_snapshot
    dump_custom_fields  output redmine project custom fields
    dump_project        output the Redmine project content
    dump_sets           output task sets: all, top, with parents, anchor and
                        lone
    dump_signatures     output signature of top-level tasks
    dump_tracker_statuses
                        output tracker statuses
    dump_users          output known Redmine user names and ids
    workflow_validate   validate DATA workflow xlsx
    workflow_compare    compare workflows
    workflow_update     update DATA workflow xlsx from Redmine
    workflow_inject     inject DATA workflow xlsx into Redmine
    workflow_leafs      returns leaf (id, start_date, due_date) list of anchor
    workflow_shift      shifts anchor workflow to new date
    workflow_push       push to DATA workflow xlsx
    dump_transitions    test transition finding
    meeting             create minutes of next meeting
    peek                output Redmine issue and Slim content
    request_to_anchor   transforms request to anchor by adding anchor tree
    run_test            run test: delete all tasks in project, create new
    save_snapshot       save Slims to json file
    set_status          set Redmine task status
    set_start_date      set Redmine task start_date
    set_due_date        set Redmine task due_date
    set_estimated_days  set Redmine task estimated_days
    set_relationship    set Redmine task relationship
    delete_relationship
                        delete Redmine task relationship
    set_tracker         set Redmine task to config tracker
    force_tracker       force all Redmine tasks to config tracker
    slims_all           output Slims in project, 1 csv row per slim
    slims_ctrl          output KARABO work slim in ERD project, 1 csv row per
                        slim
    slims_without_parent
                        output Slims w/o parents in project, 1 csv row per
                        slim
    slims_anchor        output project anchors
    slims_request       output project requests
    summary             output summary of project
    traverse_issue      traverse Redmine task tree
    traverse_slim       traverse Slim task tree

optional arguments:
   -h, --help            show this help message and exit

pep_tool sub-command option qualifiers

To see if a sub-command requires a qualifier run the sub-command w/o and see if a missing qualifier error is flagged, e.g.

python pep_tool.py username password peek
usage: pep_tool.py username password peek [-h] taskID
pep_tool.py username password peek: error: the following arguments are
required: taskID

Or add a -h behind the sub-command which has the advantage of showing the applications type conversion applied to the qualifier value string provided, e.g.

python pep_tool.py username password peek -h
usage: pep_tool.py username password peek [-h] taskID

positional arguments:
  taskID      task (aka ticket) ID, type <class 'int'>

optional arguments:
  -h, --help  show this help message and exit

Project implementation workflow programming

tbd, but read the last paragraph in the next section and the cost-estimate section in the PMO procedure chapter.

The Group task entry allows grouping tasks with similar requirements. Specifying a PrevGroup as a diffrent Group allows a staged rollout of task groups - note that only one task in the group need have PrevGroup set and that the PrevGroup entry with the largest Nr is used. The PrevTask defines the relation of the task to the previous task, if blank the task starts at the same time as the Group task and if set follows it follows the previous task.

Example with exanded row information

DP021 cost-estimate’s row expanded implementation workflow

Example with exanded row and column information

DP021 cost-estimate’s row and column expanded implementation workflow

Example project workflow handling

Injecting the GENTEC_SLINK1 ERD project into Redmine is described.

The CTRL project workflow spreadsheet is extracted from Gitlab and renamed as required. CTRL because there are presumably only three ERD workflows associated with PLC only work required, PLC and CTRL joint work, and CTRL only.

To inject GENTEC_SLINK1 implementation workflow into Redmine using the tooling

python pep_tool.py youngman $chris workflow_inject \
../templates/ERD/CTRL_workflow.xlsx GENTEC_SLINK1 2021-1-11

creates the Redmine gantt representation of the project’s workflow

Example workflow post-update

GENTEC_SLINK1 Gantt chart view in Redmine.

Updating the spreadsheet later

python pep_tool.py youngman $chris workflow_update \
../templates/ERD/CTRL_workflow_GENTEC_SLINK1.xlsx

updates most information fields

Example workflow post-update

GENTEC_SLINK1 implementation workflow sheet (post injection update)

This was the first workflow injection into a live procedure and there were a number of shortfalls:

  • the ‘?’ in the spreadsheet subject has not been replaced by GENTEC_SLINK1
  • the assignment of Daniel Kane to his tasks failed silently
  • it transpired that the enquiery to Redmine using the assignee name to extract the user-id fails and the tooling workround is to use a lookup dictionary.
  • ActualDays spend on the task has not been filled in, we’re missing time spent handling
  • Comment is also empty. It should contain the task’s milestone custom-field string which is not available in the Task tracker currently used by the ERD Redmine procedure

Flattening the workflow’s tree graph structure into a spreadsheet table requires the the use of Issue, RelatedIssue(s), Relationship(s) and Type.

The Excel definitions used by PMO cost-estimates and the workflows of non-PMO procedures are currently identical.

Workflow Excel background information

Excel direct names are used to identify column usage. The following table lists the name and there content type (Number1 is a number with 1 decimal precision). Content alignment is always Horizontal: General, Vertical: Center, and Indent: 0.

Column title Column direct name Content
Nr Nr General
Service Service General
Subject Subject General
Type Type General
Issue LinkName General
RelatedIssue(s) TargetLinkname General
Relationship(s) Relationship General
HoldingTicket(s) HoldingTickets General
HoldingRelationship(s) HoldingRelationships General
FTEdays Days Number1
Assignee Assignee General
Actual FTEdays ActualDays Number1
Ticket Ticket Number0
StartDate StartDate General
DueDate DueDate General
TaskStatus TaskStatus General
Delay Delay Text
Category Category General
Other groups involved OtherGroups General
Comments Comments General

Note

Managers should never have to create workflow Excel spreadsheets, they should copy templates. If you’re creating a template you’re probably doing something wrong.

Where manager edit Excel spreadsheets (to add/modify assignees, etc.) is a matter of preference. I prefer to do this on Windows (Office 2019), but often modify on Ubuntu (LibreOffice 6.0.7.3 00m0(Build:3)) and save changes with Microsoft Excel 2007-2013 XML Format. Openpyxl also gets it right on both platforms.