PTC - Project Scheduling¶

Introduction¶

The alignment of short and long procedure Redmine project containers described in the proceeding chapters allows all project implementation workflows requiring Data Dept resources to be managed and operated on in the same way.

In this chapter the possibilities of scheduling implementation workflows are evaluated. Both long and short procedures have two scheduling stages for a project, a procedure review and approval stage followed by an implementation stage for approved projects.

The review stage is most clearly seen in the ERD process, albeit without being complete as request injection (by email) has to be included. Additionally the PMO review stage is not implemented as it is not a DATA dept responsibility and currently the DATA cost-estimate stage is not organised with Redmine.

Project scheduling constraints¶

Constraints arise due to missing resources, which can be:

a suitably experienced DATA FTE is not available. Specialization should be avoided but task completion can be accelerated by assigning a person who has dealt with the issue previously, a decision which amplifies specialization.
a time issue exists concerning when a given task must be performed. This can be the on-loan availability period of a piece of h/w, beamtime at an instrument, the availability of requester, etc.

Scheduling a project in a procedure¶

Solving a simplified version of a problem often leads to understanding the bigger problem and its solution. Therefore we start by understaning how to schedule a single project in a small set of projects of a single procedure.

For ease of reading project implementation is dropped and workflow used instead of project implementation workflow in the remainder of the chapter.

Scheduling strategies¶

Two extremes exists:

No scheduling, where work is allocated according to what can be done on the day. This strategy has light management requirements but is likely to delay project rollout due to a Dept resource being unavailable or occupied, unless workload is low or resource availability is high.
Perfect scheduling, where all tasks of all workflows to be scheduled are completely understood in terms of duration and resources required. This strategy is also not delay free as some tasks may require a specific assignee or an unexpected time issues arises.

Scheduling reality will presumably lie somewhere between the extremes, but minimizing delay summed over all workflows should provide a useful indicator for establishing whatever the stategy is.

Project injection into Redmine¶

A PMO workflow is chosen which simplifies the discussion further as no request stage handling is required and the cost-estimate FTE task definitions, agreed to with the PO, are injected into Redmine with:

an initial workflow start_date set,
all workflow tasks defined and grouped under sub-tasks when required,
task duration days set,
and assignees not assigned.

Injection does not require PO prerequisite or Financial activities, e.g providing purchased items required, to be addressed. Instead missing items are notified by developers setting the task (Redmine) status to Feedback and flagging the condition in the task (Redmine) Category when the deficit is reached.

Workflow injection is performed by:

# inject a comples workflow
python pep_tool.py youngman $chris workflow_inject ./complex_test.xlsx
Complex 2021-02-01
...
# get the anchor ID
python pep_tool.py youngman $chris slims_anchor | grep Complex
82371,,[82404,...],Elvis,Open,Normal,PTC-PMO test,PMO project Complex,
,139,-1,,10

Further tooling operations use the anchor task ID (here 82371) to identify the workflow. In the above example the ID was extracted by greping for Complex in the list of anchor tasks present, but the workflow ID can also be read from the xlsx Ticket column of the first row following injection.

Shifting an entire workflow¶

One aim of using Redmine is to minimize manager workload. A likely initial manager task is visually (lining up Gantt chart entries) checking (as yet no scheduling tooling is implemented to recommend and perform a correction) the injected workflow start_date for conflicts with previously injected workflows.

Shifting an entire workflow is performed by:

# move the workflow forward
python pep_tool.py youngman $chris workflow_shift 82371 '2021-3-8'
...
# move the workflow backward
python pep_tool.py youngman $chris workflow_shift 82371 '2021-2-1'
...

Shifts are performed by creating ordered task lists, where forward (backward) shifting requires the due_date (start_date) to be ordered with latest (earliest) due_date (start_date) listed first. The tasks in the list are then stepped through with forward (backward) shifts executed by updating the due_date then the start_date (start_date then due_date). The target date of a shift can be any date and the present (today) date has no significance.

The shift implementation described above ensures that a task start_date cannot be later than its due_date. This and the task chaining relationships used in procedure workflows shoul guarantee that shifts can be performed.

Changing a single task duration withing a workflow¶

Note

If the aim is to modify a single task duration, then do this via the browser interface. The single target task ID methods described in this section can fail to produce the desired result if preceding and follower task settings are ignored. The latter are not ignored when the Excel or browser interface is used for the change.

Changing a task duration, start_date to due_date, can resolve a resource conflict with another in-progress project. To change a task duration the difference between the task’s start_date and due_date must be changed whilst keeping start_date earlier than due_date and the start_date not before the end_date of the preceding task.

Observed Redmine start_date and due_date change behaviour (recheck needed):

setting due_date after the start_date of a follower task auto-aligns follower task(s) start_date(s) and end_date(s). Start_dates are aligned to Mon-Friday and due_dates to Mon-Sun.
setting due_date earlier auto-align start_date of a follower but leaves its due_date unchanged.
start_date can be set to Mon-Sun.
due_date can be set to Mon-Sun.
start_date of a task cannot be set before the due_date of the preceding task (exception.ValidationError).
setting a task’s due_date earlier auto-aligns follower task(s) start_date(s) but leaves due_date(s) unchanged.

An example of changing due_date is:

# find target's date settings
python pep_tool.py youngman $chris peek 82325
...
'start_date': '2021-04-01', 'due_date': '2021-04-03'
...
# set due date later
python pep_tool.py youngman $chris set_due_date 82325 '2021-04-05'
python pep_tool.py youngman $chris peek 82325
...
'start_date': '2021-04-01', 'due_date': '2021-04-05'
...

Similarly start_date can be changed with set_start_date. However, limitations exist: the start_date cannot be moved before a related task’s end_date, etc. and the Python-Redmine API will throw a informative error message if an illegal change attempt is made.

Changing many task durations within a workflow¶

Note

Try to avoid changing many task durations by estimating and setting realistic task durations on workflow injection. Later updates are more probably single tasks activities.

As a first offering task due_date moves to later dates are supported, which levers Redmine’s automatic alignment of following tasks. This should satisfy the use case where the workflow was initially injected with short placeholder task (Days) durations. A complete solution allowing start_date (earlier of later) and due_date (earlier) changes is left to later.

Note

Group (Type value) tasks are not operated on as their duration is set by the associated sub-tasks.

An example of changing task durations follows.

Always update the spreadsheet of the workflow:

python pep_tool.py youngman $chris workflow_update
../templates/ERD/CTRL_workflow_due_dates_later.xlsx

Increase the Days for those Tasks (Type columnn value) where an duration increase is required and validate to confirm what you want is there!

Warning

Validation is common and an important step in most Excel procedures as it checks actual task attribute setting in the Redmine project, and not only their Excel values. It is however limited as not all change requests can necessarily be identified as incorrect.

python pep_tool.py youngman $chris workflow_validate
../templates/ERD/CTRL_workflow_due_dates_later.xlsx
...
row 24 (due - start) work days  1 < 10 required days
... Subject CTRL-ICI implement KARABO device for DUE_DATES
device for DUE_DATES
...

Push the changes to Redmine. Note that the pushed use_dates are calculated using workdays which excludes weekends.

python pep_tool.py youngman $chris workflow_push
../templates/ERD/CTRL_workflow_due_dates_later.xlsx
...
82908 set due_date 2021-03-25
...

Pull the changes to Excel and validate the result

python pep_tool.py youngman $chris workflow_update
../templates/ERD/CTRL_workflow_due_dates_later.xlsx
...
python pep_tool.py youngman $chris workflow_validate
../templates/ERD/CTRL_workflow_due_dates_later.xlsx
...

The validation should print no due_date-start_date differences.

Changing many task statuses within a workflow¶

Why is setting task duration straightforward using the Excel interface? It’s because working days is a simple concept, there is no needed to work with dates, weekends and days in the year. Status changing requires knowing what states exist and the allowed transitions from one to another, the state transition diagram for the Redmine tracker used must be known. For Managers the Elvis’ allowed next state transitions are:

Current State	Next open States	Next closed States
Open	Under Review	Closed, Cancelled
Under Review	New, Open	Cancelled, Rejected
Cancelled	Open, Under Review
Rejected	Open, Under Review
New	In Progress	Closed, Cancelled
In Progress	New, Feedback, Blocked, Resolved	Cancelled
Feedback	Blocked, In Progress
Blocked	Feedback, In Progress
Resolved	In Progress	Closed
Closed	Resolved

The manager’s choice of final State should not be limited to next states only which requires that the path on the way to the target state is reached silently.

Warning

Excel state change updates are only allowed on Task (Type column value) rows. Group state changes may automatically update or need extra handling. The first offering is additionally limited to open to closed state transition paths for all rows as a trail.

An example of changing task states follows.

Always update the spreadsheet of the workflow:

python pep_tool.py youngman $chris workflow_update
../templates/ERD/CTRL_workflow_due_dates_later.xlsx

Change the TaskStatus for those Tasks (Type columnn value) where a state change is required and validate to confirm what you want is there!

python pep_tool.py youngman $chris workflow_validate
../templates/ERD/CTRL_workflow_due_dates_later.xlsx
...
row 8 Error no path In Progress -> Open
row 17 Error unknown status: BADname
row 19 Warning taking 1st path
Feedback -> In Progress -> New -> Closed
Feedback -> In Progress -> Resolved -> Closed
Feedback -> Blocked -> In Progress -> New -> Closed
Feedback -> Blocked -> In Progress -> Resolved -> Closed
...

There should be no TaskStatus associated warnings.

Push the changes to Redmine. Note that the pushed use_dates are calculated using workdays which excludes weekends.

python pep_tool.py youngman $chris workflow_push
../templates/ERD/CTRL_workflow_due_dates_later.xlsx
...
TBD
...

Pull the changes to Excel and, as in the Task duration section, validate the result.

Changing many task assignees within a workflow¶

You’re getting bored of the same procedure all the time. Setting a name in the assignee is handled as task Duration and Status:

upload,
change names,
validate,
and if no corrections push

To overcome the difficulty of extracting the Redmine user ID, needed in the push, by querying with the name which fails if Group access is not allowed for the manager. The set of known ID’s is stored in the config file and missing names, Elon below, need their IDs adding.

python pep_tool.py youngman $chris workflow_validate
../templates/ERD/CTRL_workflow_due_dates_later.xlsx
...
row 5 unknown Assignee name Elon Musk
...

Initial tooling scheduling requirements¶

The above discussion indicates an initial functionality set for schedule tooling.

to identify an unavailable FTE resource the procedure PEPs must provide a list of developer absences in a format (Excel, json, yaml…) which can be interfaced to the tooling.
to automate FTE resource checks and rescheduling an additional Redmine value is required to hold a csv list of candidate assignees for a task.
workflow inject (already implemented)
workflow modify into Redmine task attributes (start_date, due_date, assignee)
workflow update from redmine of task attributes, sub-tasking and relationships (attribute only update implemented)
workflow shift of its start date (entire workflow shifts implemented)
workflow multi task duration changes (end_date extension implemented)
workflow multi task assignee changes
workflow multi task status changees
workflow check for identifying resource conflicts with other projects
workflow reschedule to reschedule projects according to rules applied.
workflow compare for testing the above functionalities
workflow extract to generate the workflow spreadsheet from Redmine definitions, which eases workflow evolution, check pointing, etc. This functionality is most useful for workflows where all information content is in Redmine (e.g. ERD or CRDs), but not PMO where additional information is in the spreadsheet. However, some loss including Comments, Service, and OtherGroups will occur.
code a similation of Redmine, or is a Redmine differed change mode provided (rollback or check pointing), to allow complex multiple start_date and due_date changes to be evaluated as possible before updating Redmine.
import python config values into Excel for use with pulldown menues such as assignees

The functionality set should not include

workflow clear to reset Ticket and similar update characters is not needed as a copy of the template should be used. Resetting copies of working workflow spreadsheets is likely to produce the wrong result, e.g. the name insert parameter has no injection point.

Adding in other procedures¶

The above discussion and associated tooling tests identified the set of requirements for single procedure scheduling. It also provided sufficient insight to suggest the requirements for integrating workflow information from othe procedures. The suggestions are:

procedure precedence (PMO (or XO) > CRD > ERD) defines how procedure workflow information is to be included.
lower precedence workflow task information should be loosely coupled using the blocked by relationship rather than sub-tasking. This approach requires additional tooling to flag blocking situations.
workflow check functionality should allow identification of FTE conflicts in lower precedence workflows, but it may not be required to apply rescheduling to them