Perforce Defect Tracking Integration Project
replicate_job_pfunction
prepare_issuefunction
This manual is the Perforce Defect Tracking Integration Advanced Administrator's Guide. It describes various advanced methods of administering the P4DTI that are outside the scope of the Perforce Defect Tracking Integration Administrator's Guide because they require a higher level of expertise (see section 1.1, "Required experience"). should have the following experience:
The P4DTI has a general way to select which issues get replicated from the defect tracker: the
replicate_p function. To select which issues get replicated you need to: once an issue starts being replicated it remains replicated, even if it no longer matches the criteria.
The function is passed one argument, the issue, in the form of a Python dictionary mapping field name (string) to field value. The function must return 1 if the issue should be replicated, or 0 if it should not.
The
replicate_p function does not have to check the
start_date configuration parameter. The
start_date is always honored by the P4DTI, in addition to any test made by
replicate_p.
This example restricts replication to issues belong to the project whose ID is 6:
def replicate_p(issue): return issue["PROJECTID"] == 6
Suppose you want to replicate only those issues where the "Issue type" is
BUG. The issue type is stored in the
TS_ISSUETYPE field in the issue table, and the value of that field is the TS_ID field of the appropriate record in the
TS_SELECTIONS table [TeamShare 2001-06-19a]. You need to specify a
replicate_p function like this:
def replicate_p(issue): return issue["ISSUETYPE"] == 1
This example assumes the value 1 is the value for the issue type
BUG. Yours might be different, so you need to look in the
TS_SELECTIONS table, find the record where
TS_PREFIX is
BUG, and use the
TS_ID field from that record.
This example restricts replication to unresolved issues in the "horseshoe" product:
def replicate_p(issue): return (issue["product"] == "horseshoe" and issue["resolution"] == "")
Normally, the P4DTI requires users to create new issues using the defect tracker. It ignores new jobs created in Perforce. The P4DTI can be configured to transform new jobs created in Perforce into new issues in the defect tracker.
You need to add two functions to the end of
config.py:
replicate_job_p and
prepare_issue.
replicate_job_pfunction
The
replicate_job_p function selects which newly-created jobs in
Perforce should be translated and added as new issues in the defect tracker. It
takes the newly-created job as its argument, in the form of a Python
dictionary mapping field name (string) to field value (string). It must
return 1 if the job should be translated and added to the defect
tracker; 0 if not.
Note: once a job starts being replicated it remains replicated, even if it no longer matches the criteria.
If you want all newly-created jobs to be translated
and added to the defect tracker, you should write a
replicate_job_p function as follows:
def replicate_job_p(job): return 1
Suppose you want to replicate newly-created jobs for projects called "horseshoe" and "nail".
In TeamTrack, you add the
PROJECTID field to the
replicated_fields configuration parameter;
this gets replicated to the field "Project" in Perforce; you write a
replicate_job_p function as follows:
def replicate_job_p(job): return job["Project"] in ["horseshoe", "nail"]
Suppose you want to replicate newly-created jobs for products called "horseshoe" and "nail".
In Bugzilla, you add the
product field to the
replicated_fields configuration parameter;
this gets replicated to the field "Product" in Perforce; you write a
replicate_job_p function as follows:
def replicate_job_p(job): return job["Product"] in ["horseshoe", "nail"]
prepare_issuefunction
The P4DTI calls
prepare_issue to complete the
transformation of a new job in Perforce to an issue in the defect
tracker. The P4DTI does its best to build the new issue, but the
defect tracker may require fields that can't automatically be deduced from fields in
the job. This function must supply values for these fields.
The function takes two arguments (
issue,
job) as follows:
issueis the partly-constructed issue, in the form of a Python dictionary mapping field name (string) to field value.
jobis the newly-created job, in the form of a Python dictionary mapping field name (string) to field value (string).
The function must modify
issue so that
it is a valid issue for creation in the defect tracker.
Its return value is ignored.
In the TeamTrack integration, new issues must have a valid value for these fields:
ISSUETYPE
This must
be the
TS_ID of a record in the
TS_SELECTIONS table in the TeamTrack database
(look in the
TS_PREFIX field for the name of
the issue type). If the
ISSUETYPE field is
not one of the
replicated_fields, or if the value in the job
was not the name of an issue type, then the P4DTI picks
0
for
issue['ISSUETYPE']. This means that the
issue will have no type, just a number. If this is not appropriate, you
must supply a suitable value.
PROJECTID
This must
be the
TS_ID of a record in the
TS_PROJECTS table in the TeamTrack database (look
in the
TS_NAME field for the project name).
If the
PROJECTID field is not one of the
replicated_fields, or if the value in the job
was not the name of a project, then the P4DTI sets
issue['PROJECTID'] to
0, which is not a valid project, so the issue will
fail to be created. You must supply a suitable value.
You can use this function to supply default values for other fields in TeamTrack. See the TeamTrack database schema [TeamShare 2001-06-19] for the full details.
If
PROJECTID and
ISSUETYPE are both in the
replicated_fields, then you may be able to
omit this function altogether.
This these fields:
product. This must be the name of a product in the Bugzilla
productstable. If the
productfield is not one of the
replicated_fields, then the P4DTI picks the empty string for
issue["product"](except when there's only one product, in which case the P4DTI picks that product).
component. This must be the name of a component of the product to which the issue belongs. If the
componentfield is not one of the
replicated_fields, then the P4DTI picks the empty string for
issue["component"](except when there's only one component of the product, in which case the P4DTI picks that component).
version. This must be the name of a version of the product to which the issue belongs. If the
versionfield is not one of the
replicated_fields, then the P4DTI picks the empty string for
issue["version"](except when there's only one version of the product, in which case the P4DTI picks that version).
You can use this function to supply default values for other fields in Bugzilla. See the Bugzilla database schema [NB 2000-11-14a] for details.
This]
You must delete all jobs from your Perforce installation before running the P4DTI for the first time. The P4DTI takes over the jobs subsystem of Perforce and rewrites the Perforce jobspec.
Migration allows you to keep existing issues that are stored in Perforce jobs. Migration submits all the Perforce job data to the defect tracker, so that the issues can be replicated back to Perforce, and so appear in both systems.
You will need a plan for installing and testing the P4DTI and migrating your Perforce jobs. In particular, you will need to have configured the P4DTI replicator before you migrate, and to have a good understanding of what it does. This makes the migration process quite complicated.
This section describes a plan which will allow your users to continue to work with Perforce and your defect tracker until you're ready to go live.
The first stage is to configure and test both the P4DTI replicator and migration.
Create an empty Perforce server for testing.
Create a defect tracker for testing. If you have existing defect tracking data, this should be a copy that includes your workflow and at least a representative sample of your existing issues.
Set up and test the P4DTI replicator (without migration) on the test Perforce server and test defect tracker, following the instructions in the Perforce Defect Tracking Integration Administrator's Guide. Ask your users to try it out, and make sure you are happy with the workflow and the P4DTI configuration.
Make a note of the jobspec created by the P4DTI replicator. This will be used in configuring the translation step of the migrator. (See section 4.7, "Translating to the new jobspec".)
Replace the test Perforce server with one which has a copy of all your jobs and fixes. (You can do this by creating a checkpoint of your main Perforce server and restoring.2, "Going live with the P4DTI with migration".
Once.
Edit your
config.py file to point to your main Perforce and defect tracker servers, but don't start the P4DTI replicator yet.
Run the migration and check the results (4.10).
Start the P4DTI replicator.
You should now have a functioning integration with your old issue data intact.
Migration cannot be done automatically by the P4DTI, because:
Your existing issues must therefore be translated in order to migrate them from Perforce jobs to the defect tracker.
Here is an overview of preparing for migration. The details of each step are covered in the following sections.
Prepare the Perforce data for migration:
Ensure that users in Perforce match users in the defect tracker (4.3 and 4.4).
Make sure that all Perforce jobs are consistent with the current jobspec so that the P4DTI can transform them systematically (4.5).
Write Python functions to your
config.py to do the following:
Choose which jobs to migrate (4.6).
Translate these jobs so that they match jobs that would be replicated from the defect tracker (4.7).
Fix up the issues transformed from jobs so they satisfy any defect tracker constraints on submitted issues (4.8).
Run the migration to transform all the jobs and submit them to the defect tracker (4.10).
To migrate your jobs, work through these subsections.
Make sure that every Perforce user has a name and e-mail address.
If you're migrating to TeamTrack, this will allow the P4DTI to correctly match users in Perforce to users in TeamTrack (see section 3.5, "User accounts" of the Perforce Defect Tracking Integration Administrator's Guide).
If you're migrating to Bugzilla, this allows you to automatically make user accounts in Bugzilla for each Perforce user (see section 4.4, "Making defect tracker user accounts").
Make a defect tracker user account for each Perforce user who needs to do defect tracking.
For TeamTrack, you must create users one by one using the TeamTrack Administrator application. See the TeamTrack Administrator Manual for your version of TeamTrack.
For Bugzilla, you can automatically create a Bugzilla user for each Perforce user. Follow these steps:
Ensure that
config.py has
correct values for
dbms_host,
dbms_port,
dbms_user and
dbms_password.
Add values for the following configuration parameters to
config.py (unless you're happy with the
default values):
migrated_user_groups must be a list of
Bugzilla groups. Newly created users will belong to all the groups
in this list. For example, if you set
migrated_user_groups = ["editbugs",
"canconfirm"] then the newly created users will be able to
edit and confirm bugs. The default value is
[] (that is, newly created users will belong
to no groups).
To see the Bugzilla groups and their meaning, log into Bugzilla as an administrator and select "edit groups" from the options at the bottom right.
migrated_user_password must be a string.
Newly created users will have this password. For example, if you
set
migrated_user_password = "spong"
then the newly created users will have password "spong". The
default value is
Go to the P4DTI installation directory.
Run the command
python
migrate_users.py.
When you change the Perforce jobspec Perforce doesn't update its jobs to match, so if you've changed jobspecs in the past your jobs may be inconsistent with the current jobspec. For example, it may be that you have a "required" field which is missing from some jobs, or a "select" field for which some jobs have an illegal value. These problems will prevent the P4DTI from being able to read.
Choose which jobs to migrate by adding the function
migrate_p to the end of
config.py.
The
migrate_p(job) function determines whether to migrate a job to
the defect tracker. The
job argument is the job being
considered for migration (in the old jobspec), in the form of a
dictionary mapping field name to field value. The function must return
1 if the job should be migrated, 0 otherwise.
migrate_p
def migrate_p(job): return 1
def migrate_p(job): return job["Status"] == "open"
def migrate_p(job): return job["Project"] in ["horseshoe", "nail"]
Write a function to translate jobs from the old jobspec to the new jobspec. Add the function
translate_jobspec to the end of
config.py.
The
translate_jobspec function translates a job from the old jobspec
(before migration) to the new jobspec (after migration). The old job is
passed as the single argument, in the form of a dictionary mapping field
name to field value. The new job must be returned in the same form
(it's OK to update the argument and return that).
The P4DTI replicator takes over the jobs subsystem of Perforce and rewrites the Perforce jobspec according to the
replicated_fields configuration parameter. You should have a copy of the new jobspec, made after testing the P4DTI replicator as described in section 4.1.1, "Configuring and testing the P4DTI and migration".
You develop a workflow in TeamTrack and decide
to replicate the "Description", "Priority" and "Project" fields (in addition
to the system fields "Title", "State" and "Owner" which are always
replicated). So your
replicated_fields configuration parameter has
the value
["DESCRIPTION", "PRIORITY",
"FILENAME"] and your jobs look like this under the new jobspec:
Suppose that you need to make the following changes:
The old jobspec has a single "Description" field, but the new jobspec has both a "Title" (one line) and a "Description" (many lines). You want to split the old description so that its first line becomes the new title.
You need to change the "Status" field to a "State" field and the "User" field to "Owner".
In the old jobspec, status is "open", "closed" or "suspended". In the new jobspec, it is "_new", "assigned", "resolved" or "verified". You want to map "open" to "assigned", and both "closed" and "suspended" to "verified".
Here's how to write a
translate_jobspec
function that implements all the
You consider the Bugzilla fields and workflow and
decide to replicate the resolution, long description, priority and
product (in addition to the summary, status and assignee which are
always replicated). So your
replicated_fields configuration parameter has
the value
["resolution", "short_desc", "priority",
"product"] and your jobs look like this:
Suppose that you need to make the following changes:
The old jobspec has a single "Description" field, but the new jobspec has both a "Summary" (one line) and a "Description" (many lines). You want to split the old description so that its first line becomes the new summary.
You need to change the "User" field to "Assigned_To".
In the old jobspec, the status is "open", "closed" or "suspended". In the new jobspec, the status is represented by a combination of "Status" and "Resolution" fields. You want to translate status as shown in the table below.
Here's how to write a
translate_jobspec
function that implements all the above:
def translate_jobspec(job): # Description -> Summary + Description import string desc = job.get("Description", "") newline = string.find(desc, "\n") job["Summary"] = desc[:newline] job["Description"] = desc[newline+1:] # User -> Assigned_To job["Assigned_To"] = job.get("User", "") # Translate Status status_map = { "open": ("assigned", ""), "closed": ("closed", "FIXED"), "suspended": ("closed", "LATER"), } (status, resolution) = status_map[job.get("Status", "open")] job["Status"] = status job["Resolution"] = resolution return job
Jobs translated to the new jobspec may still not yet be suitable for submission to the
defect tracker. The defect tracker may have additional constraints, or
require fields that can't be translated from fields in the job. You must therefore write a
prepare_issue function as
documented in section 3.2, "The
prepare_issue function".
Figure 1 shows a summary of the translate, transform, and prepare steps of the migration process:
The job is translated from
the old jobspec to the new jobspec by the
translate_jobspec
function.
The P4DTI transforms the job to an
issue by converting each field in
replicated_fields
and any system fields.
The issue is prepared for
submission to the defect tracker by the
prepare_issue
function.
Figure 1. The translate, transform, and prepare steps. The example data in the figure is for TeamTrack.
Make sure that you have migrated users as described in section 4.4.
Make sure you have the functions
migrate_p,
translate_jobspec, and
prepare_issue in your
config.py, as described in the sections above.
Make a checkpoint of the Perforce server as described in chapter 2, "Supporting Perforce: Backup and Recovery", of the Perforce 2001.1 System Administrator's Guide.
Go to the P4DTI installation directory and run
the command
python migrate.py. If it
succeeds, you'll see.
If you're migrating to Bugzilla and have
specified a value for
bugzilla_directory, expect a long wait after
the last job is migrated and before the "Migration completed" message
appears. This is because Bugzilla's
processmail script is being run for each
migrated issue. (If you don't want notifications to be sent when
migrating, set
bugzilla_directory =
None.)
If migration fails, you'll typically get one of these kinds of error:
The replicator can't transform a job to an issue; for example:
(P4DTI-6288) No TeamTrack state in project 'Horseshoe' corresponding to Perforce state 'suspended'.
This means that the
translate_jobspec function needs to consider
more possibilities (in the example, it needs to translate the Perforce
status
suspended into a state in TeamTrack.
The replicator can't create an issue in the defect tracker; for example:
(P4DTI-522X) Can't create Bugzilla bug without component field.
This means that the
prepare_issue function has failed to meet all
the defect tracker constraints (in the example, it needs to provide a
value for the component field).
When you've fixed the problem, run
python
migrate.py again. Provide the command-line option
-s JOBNAME to start migrating at a
particular job. For example, if migration failed because of a problem
job000123, then you might continue migrating with
python migrate.py -s job000123.
Check the migrated issues. Are they as you expected? If not, fix the problem and run the migrator fails you may be able to fix up problems by hand. If not, you can restore your Perforce server from the checkpoint you made earlier.
Both the defect tracker and the Perforce server now contain transformed issues, and the system is prepared for starting the P4DTI replicator..
To do so:
Install a copy of the P4DTI for each Perforce server, as described in
readme.txt.
Configure each copy of the P4DTI. These configuration parameters must be different for each copy:
rid, so that the replicators can distinguish the issues they are responsible for.
sidand
p4_server_description, so that the defect tracker can report which Perforce server the issue is replicated to.
p4_port, so each replicate can find its Perforce server (depending on how you have the Perforce servers set up, you may need to change
p4_userand
p4_passwordas well).
replicate_p, so that the replicators don't try to replicate the same issues.
Start and manage each replicator as described in the Perforce Defect Tracking Integration Administrator's Guide.
Example. You have Perforce servers for your development groups in
France and Germany and a single TeamTrack server. Projects 4, 7 and 11
are worked on in France and projects 5, 6, 9 and 10 are worked on in
Germany. So you set up one P4DTI copy with the following in
config.py:
rid = "replicator_fr" sid = "france" p4_port = "perforce.company.fr:1666" p4_server_description = "French development group" def replicate_p(issue): return issue["PROJECTID"] in [4,7,11]
and the other with the following:
rid = "replicator_de" sid = "germany" p4_port = "perforce.company.de:1666" p4_server_description = "German development group" def replicate_p(issue): return issue["PROJECTID"] in [5,6,9,10]
Notes:
The P4DTI doesn't support integration between one Perforce server and multiple defect tracking servers.
There's no way to move issues from one Perforce server to
another. Once an issue is replicated to one Perforce server then it
will always be replicated to that server, even if it later matches the
replicate_p function for another replicator.
If it's going to be doubtful which development group is going to work on
an issue, then you need to.
When integrating with TeamTrack, in order to write
replicate_p
and
prepare_issue functions, you need to know the
key values for the tables in the TeamTrack database.
You can do this by looking directly at the TeamTrack database using
your database tool. Or you can use the
teamtrack_query.py script that comes with the
P4DTI.
Use it as follows:
Provide values in
config.py for
the
teamtrack_server,
teamtrack_user and
teamtrack_password configuration
parameters..
Use the TeamTrack database schema [TeamShare 2001-06-19a] to work out which table you need to query and how to understand the output.
Example 1. You want to replicate issues from projects called
"Horseshoe" and "Nail". The TeamTrack database schema]
Example 2. When a new job is created in Perforce, you want to
create an issue in TeamTrack with issue type "BUG". The TeamTrack
database schema.