We would like to create a service that deals with self-serve/buildapi scheduling on behalf of TH.
This service will listen to TH actions and trigger them through mozci.
The initial implementation will only deal with self-serve/buildapi until we support TC. NOTE: We will need to talk with the TC team once we aim to tackle such since they already have their own implementation.
At first we will design it to not trigger anything (--dry-run).
Once the treeherder team is satisfied we could look into not hitting self-serve directly from TH and trust that this listener will trigger the required actions.
adusca started today and will be working on this. She has experience with mozci as well as with Pulse.
We intend to make this tool as generic as possible so we can add more listeners and new actions depending on what we want to do.
and deploy it with Heroku.
In the future, this setup will also allow for more sophisticated trigger requests be made by TH (e.g. backfilling, trigger arbitrary builders).
I think we can start with a simpler setup and discuss in the future how far we want to take it.
Also notice that we would like to be able to test changes against treeherder.allizom.org (staging system) before we want to make them live for treeherder.mozilla.org (production system).
- worker.py listens to exchange/treeherder/v1/job-actions in an infinite loop. It calls make_retrigger_request when it sees a retrigger action and make_cancel_request when it sees a cancel action. Currently everything is ran with dry_run=True, so instead of doing requests the script just logs the requests it would have made.
- I'm dealing with credentials using a plaintext, manually-written file located at ~/.mozilla/mozci/pulse_credentials.
Question: Should I move make_cancel_request and make_retrigger_request to mozci/sources/buildapi ?
Proposed next step: deploy the script to heroku with dry_run=True(after possible revisions). Let it run for a couple days and see in the logs what would've happened.
Nobody cancelled a job while I was testing the script.
There are a lot of identical jobs. Maybe we could add a count parameter to exchange/treeherder/v1/job-actions so when a user retriggers a job N times it sends one message with 'count = N' instead of N messages? That way we could do just one retrigger request with count = N. The function make_retrigger_request already has this functionality.
I'm glad to see you made so much progress in such a shrot time!
The code is looking well organized and clean. I like it.
I'm going to give f+ instead of r+ since for me, review requests are about when the code is believed to be ready to land.
I would like to see that code be more modularized.
The treeherder functionality into its own and so.
I would only keep the worker logic in worker.py.
With regards to TH, there is a TH client that vaibhav and wlach have been working on. Would you be able to see if we can re-use that?
We can have a directory called actions or tasks where new tasks or actions to process pulse messages are defined.
This model will make it easy to add new stuff and be more generic.
> Question: Should I move make_cancel_request and make_retrigger_request to mozci/sources/buildapi ?
I would like to see mozci being called with the dry_run argument.
What is label for? What is duration for? Where can I read a bit more about it?
You can try a cancel job on one a try push or maybe on http://treeherder.allizom.org.
With regards to multiple retrigger requests, this is due to the way that treeherder's UI works. We could either queue up messages and then evaluate if we can trigger few in one shot.
What are we trying to optimize? Make less requests to buildapi?
Taking notes on how TC's setup work.
Alice: according to lightsofapollo as a long term solution we will need to create tasks on TC to call mozci for us.
We can start with *only* using Heroku, however, lightsofapollo warns that we will soon hit issues.
For now let's put these thoughts into the backburner and get our prototype working.
Once we have a POC, let's see what we change.
<armenzg> lightsofapollo, is there a blog post with basic steps to trigger tasks on TC?
- worker.py  now receives exchange and topic as command line arguments. It then uses HANDLERS_BY_EXCHANGE, a dictionary defined in config.py  to decide what function it will use to handle incoming messages. Currently only buildbot topics (e.g. buildbot.#.# or buildbot.try.retrigger) in exchange/treeherder/v1/job-actions are supported.
This is looking great. Thanks Alice!
Could you please have a blog post laying down what we're trying to achieve for Monday?
I would like it to be read by a-team members in advance of our Monday meeting and talk a bit about it with everyone so they're aware of what you're building.
In the future, could you have branches and do PR requests to your own repo?
By doing this, I can better see the changes you're making and easily make comments where needed.
You can then merge your changes and create new PRs for any feedback I give you.
In general, we should have some way of reading the config file and start up a pulse listener per entry on the config.
Instead of starting manually with command options.
That way when we push new code to Heroku it reloads the config and adds new listeners automatically.
We should probably create each listener into its own parallel process.
Could you also add some documentation on how to set this up? or info on how to add new stuf?
Even if very brief, it helps indirectly to review mentally what the project is trying to accomplish and if things are modularized properly.
In https://github.com/adusca/pulse_actions/blob/master/worker.py#L64 we use PulseConsumer which is generic, what do you think we could do to help people add their own consumers?
I think defining the class in the config OR in the same file as where the event handler lives. I really don't know.
Add some info on what the config is supposed to define.
Something I've learned in a video from Raymond about being explicit when calling python function to improve understanding.
Could you add each keyword for each parameter in here?
BuildConsumer(**pulse_args) is the same thing as PulseConsumer(exchange='exchange/build/', **pulse_args).
Almost all of them have the same format, the only thing that changes between then is the name and the exchange.
Since run_pulse() doesn't know beforehand what exchange it will listen to, I created a class that receives the exchange as a parameter.
I made the modifications you suggested. Most importantly, instead of command line arguments worker.py now reads exchange and topic from a file called run_time_config.json. There is still only support for running one pulse listener per time.
The project is now on pypi: https://pypi.python.org/pypi/pulse-actions/0.1 and it should be pip installable after the next mozci release.
Pulse Actions in now deployed to Heroku! Its logs are being tracked by papertrail .
Today I worked on improving the documentation . I even added a 'how to write a hello world service' step-by-step guide .
Logs for the first day running on Heroku.
Heroku cycles its dynos every day. That means that pulse_actions is not available for a couple of seconds.
Is this every day at the same time? I think that is less than 0.01% of the whole day; is that correct?
Is this a 0.9942% availability?
Can't we just make the queues durable?
We would like to test using pulse_actions for retrigger/cancel requests in the ash branch.
For this, treeherder would have to keep sending pulse messages to exchange/treeherder/v1/job-actions but stop doing the actual request to buildapi just in the ash branch. Do you think it's possible?
...which will stop just the buildapi retrigger, but not the treeherder API POST that then results in the pulse message.
Hi Ed, I see that treeherder shows a message when retriggering or cancelling goes well.
How can mozci let TH know that everything went well?
We're working on sending pulse messages upon taking action; would that work for TH?
> How can mozci let TH know that everything went well?
I've tested on Ash and we get a "successful retrigger" or "successful cancel" kind of message on TH.
I assume TH *only* reports that it managed to send a pulse message.
philor raised some really good points on #releng . Previously, I assumed TH only allowed retrigger/cancel requests from logged in users who had some sort of Mozilla link (a vouched mozillian profile, or something like that).
I tried retriggering on ash with a persona account unrelated to Mozilla and the request was sent. I turned pulse actions off after that. A total of 7 jobs were retriggered using pulse_actions, most of which by me, so no damage.
The set of people who should be allowed to retrigger and cancel is, well, exactly the set who can now: anyone with whatever the Level 1 commit access LDAP bit is called, scm_1 or something like that. Getting access to push to Try means you should also be able to retrigger your broken jobs, and to cancel the rest when it becomes clear your push is broken. So you can only depend on bug 1032163 if it's going to be fixed by having some sort of a regular scheduled LDAP import, and also if you can count on everyone using the same email address for LDAP and for Persona and good luck with that I'm afraid.
> sort of Mozilla link (a vouched mozillian profile, or something like that).
> jobs were retriggered using pulse_actions, most of which by me, so no damage.
This is currently expected. When taskcluster people added the submit to pulse feature to treeherder, they were fine with no additional auth. Coming up with a solution for this is mentioned in bug 1077053 comment 0.
2) Syncing with LDAP is presumably going to be a pain - both from a network flows/auth point of view (particularly if we move to Heroku) and a "people using different emails for ldap vs persona point of view.
While bug 1032163 gets solved we can on the short term put a limit per user (a higher one for mozilla.com accounts).
I'm thinking 100/hour per user and 2000/hour per mozilla.com address.
We can whitelist sheriffs if wanted.
Once bug 1032163 is fixed we can keep the limit on non-vouched users or block them completely but TH should tell them somehow that they can't or they reached a limit (We will have a pulse exchange for mozci/pulse_actions if need be).
No, that's not acceptable. You are not fixing a horrible break in productivity with this, you are solving a few seconds of annoyance once per browser session. It is not acceptable to cripple non-employees for that, nor is it acceptable to jeopardize the entire project by allowing anyone with handful of throwaway email addresses to tie up our entire infrastructure. There is no per-hour number you can choose which will be acceptable, because every number will be either larger than some entire slave pool or smaller than the number we want some contributor working on an intermittent to run.
When TH sends a job action message, it contains an email address that we can trust is controlled by the user who made the request, because of Persona authentication.
Pulse actions can use Mozillians' API  to check if that email belongs to a vouched user. That way we don't have to impose limits and throwaway accounts cannot be used to trigger anything.
- Users that have a LDAP account but don't have a vouched Mozillian profile will lose retriggering/cancel abilities on TH with this.
We can test this approach on ash to improve on the design.
> handful of throwaway email addresses to tie up our entire infrastructure.
> than the number we want some contributor working on an intermittent to run.
You're very right. The system as-is already allows capable users to abuse the system.
We're crippling those users due our inability to stop a new set of users.
It will be good to have adusca's experiment as a data point but as she points out we won't be able to support LDAP accounts that are not vouched. Hence, not making this as part of the solution.
I say we move on to tasks that only sheriffs could trigger at first like backfilling or run missing jobs on a revision.
I want to put stress to the system.
philor, does this make more sense to you?
We can file another bug to enable this after bug 1032163 is completed.
adusca, would this work for you? Am I missing anything?
I don't want to keep Ash booked longer than we need to and at least wrap the work in here so we don't feel this to be dragging.
> backfilling or run missing jobs on a revision.
Strictly speaking, the set of "sheriffs" who need to do that are "everyone with level 3 commit access" both for twigs and because tomorrow IT will have another of their misnamed "tree closing windows" which is the odd phrase they use to mean "tree left open but we will break as many jobs as we want during a time when we know there will be no sheriff and expect that developers who have been trained to push and never look will instead look and know how to tell their bustage from our bustage and retrigger," but, sure. There you're talking about things that nobody had before you came along, so you are free to restrict them however you want.
We could improve the current script abilities by pushing on the mach front.
Specifically cancel and re-triggers are dealt with by buildapi and mozilla-taskcluster.
We won't try to support this with pulse_actions/mozci.
A lot of the pieces necessary for this specific request have been accomplished over multiple PRs and bugs, however, re-triggering and cancelling won't be dealt with pulse_actions/mozci.
Thanks for all the help around this project.