text: Title: Custom Scheduler?
  Body: > <a href="https://github.com/cliffburdick"><img align="left" height="50" src="https://avatars1.githubusercontent.com/u/30670611?v=4"></a>        An issue by [cliffburdick](https://github.com/cliffburdick) at _2020-01-28 23:22:41+00:00_
  > Original URL: https://github.com/zalando-incubator/kopf/issues/301
  > &nbsp;
  
  Great project, and I enjoyed your talk at kubecon 2019!
  
  ## Problem
  
  Kubernetes 1.17 added a new scheduler framework, but at the same time, are slowly deprecating the ability to use Python as a scheduler language. Most of the new hooks added must be written in Go, and the old scheduler extension framework that was language agnostic is going away. 
  
  Since custom schedulers are similar to operators, but just watch for the scheduler-name field to appear in the spec, kopf might be able to fit that need.
  
  ## Proposal
  
  I don't know enough about the internals of kopf, but if the framework of callbacks when a pod using a custom scheduler appeared or disappeared could be reused, that would be ideal. Instead of dealing with a CRD (or in addition to), kopf could provide the scheduler framework the ability to bind to particular nodes. 
  
  ## Checklist
  
  - [X ] Many users can benefit from this feature, it is not a one-time case
  - [X ] The proposal is related to the K8s operator framework, not to the K8s client libraries
  
  
  Edit: Maybe this is already possible and I'm overthinking it. If kopf is registered to handle a CRD that always has the scheduler-name field set, the default scheduler won't touch it. Will kopf.on.create be enough to trigger the scheduler to know it's there?
  
  ---
  
  > <a href="https://github.com/nolar"><img align="left" height="30" src="https://avatars0.githubusercontent.com/u/544296?v=4"></a>            Commented by [nolar](https://github.com/nolar) at _2020-01-31 09:35:03+00:00_
  > &nbsp;
  
  Can you please give some links with a description of this new scheduler? — To better understand the change.
  
  ---
  
  > <a href="https://github.com/nolar"><img align="left" height="30" src="https://avatars0.githubusercontent.com/u/544296?v=4"></a>            Commented by [nolar](https://github.com/nolar) at _2020-01-31 10:51:59+00:00_
  > &nbsp;
  
  I only found https://kubernetes.io/docs/concepts/configuration/scheduling-framework/ — but it is about the pod scheduling, i.e. assigning them to the nodes. I'm not sure this was ever doable with operators (both Go- & Python-based).
  
  Technically, Kopf is able to handle built-in resources now, including pods. But this handling is limited to watching over them and patching their fields (either spec or status or metadata). If this is enough for scheduling, then it can be done now. 
  
  Can you provide some more detailed description of the idea? E.g. with some hypothetical code samples and a step-by-step flow of events explained?
  
  To the level of my low knowledge of Kubernetes internals, the plugins are only possible when embedded into the Kubernetes itself. We can probably write a Go-based "mediator" plugin that will communicate with the scheduler framework internally, but with no own logic implemented, just by getting the statuses/values from the pod's fields — and by putting them back. And then, a regular controller/operator can actually do the job and "talk" to the plugin via these fields, which implies "talking" to the scheduler. Is that what you mean?
  
  
  
  ---
  
  > <a href="https://github.com/cliffburdick"><img align="left" height="30" src="https://avatars1.githubusercontent.com/u/30670611?v=4"></a>            Commented by [cliffburdick](https://github.com/cliffburdick) at _2020-01-31 15:31:00+00:00_
  > &nbsp;
  
  Hi [nolar](https://github.com/nolar), thanks for the response. Let me give you a bit more background.
  
  We have a custom Python scheduler we are developing here: https://github.com/Viasat/nhd, that we presented at Kubecon last year. Right now the scheduler does its own reconciliation loop, where the only difference (I believe) from a normal operator is it looks for the "schedulerName" field in the pod spec, and is allowed to bind the pod to a node if that field is set for it. The main feature of the scheduler is it makes decisions based on available hardware resources on the node. As you can imagine, that involves watching for new pods to show up, taking resources from a node when they are scheduled, and freeing those resources when the pod dies. The piece where it watches for new pods to show up and be deleted seem to be the same as a kopf.on.create/delete. I had to write my own (poor) logic to do this, even though it's been done a million times before. It's no problem, and even preferable, to require a CRD for these pods, since they are very similar, but not quite the same, as a StatefulSet. You mentioned kopf is limited to watching over pods and patching, and I think other than calling the client API bind command, a scheduler is nothing more than that (at least a simple one). With a CRD, I don't think we have to worry about the "scheduler" part of it, because the CRD would be the first object to come in, and that being created gives kopf the ability to deploy the pods.
  
  So the work flow would be:
  
  1. Watch for CRD type ABC to show up
  2. on.create handler for ABC creates pods with the scheduler-name set to scheduler-ABC
  3. on.create pod handler with filter on scheduler-name or some other identifier sees pod come in and binds it to a node // This is the one I'm not sure is possible. Would on.create be triggered if the pod is in a pending state without a node bound to it? Does this handler only get triggered when the pod is successfully deployed?
  4. on.delete pod handlers for that same type, and schedulers frees appropriate internal resources.
  
  
  My idea is that the kopf framework is running an operator that looks for this CRD to show up, launches an appropriate number of pods as owned by that CRD, and sees when the pods (or CRD) are deleted. Because much of the difficulty is in writing the reconciliation loop, which you've already solved, I figured the scheduler could simply be a wrapper around on.create/delete. I saw a lot of issues related to watching pods under CRDs, and it wasn't entirely clear to me if that's supported/working yet, since it seemed you were still coming up with ideas on how to handle that. 
  
  Also, after the CRD is deployed, I wanted to have another kopf operator silently watching the child pods of these CRDs as well, since it's in charge of doing things like sending messages to these pods when they come up, providing them configuration, etc. 
  
  For more context on Kubernetes, what I really wanted to do was make our scheduler into a [scheduler extension](https://kubernetes.io/docs/concepts/extend-kubernetes/extend-cluster/#scheduler-extensions). In the past, this was simply making a webhook that was called after the main scheduler was run, and it acted as a pre-filter step for other schedulers by doing the normal things like removing dead nodes/unhealthy nodes, etc. Unfortunately, it seems scheduler extensions via a webhook is going to be deprecated at some point in the future in favor of the scheduler framework added recently. The reason is the scheduler framework allows more flexibility as to where you want to plug in to compared to the webhook. The scheduler framework, at least my understanding, requires your scheduler to be written in Go, so this effectively will make writing a scheduler extension in Python impossible in the future without hacking in a Go module. But I digress.
  
  I hope this helps describe the goal, and maybe you can say whether that's possible or not, since I really like the idea of kopf and think it would be a great application.
  
  
  
  
  ---
  
  > <a href="https://github.com/nolar"><img align="left" height="30" src="https://avatars0.githubusercontent.com/u/544296?v=4"></a>            Commented by [nolar](https://github.com/nolar) at _2020-01-31 21:13:57+00:00_
  > &nbsp;
  
  **For the first part,** as I understand, you want something like this:
  
  ```python
  import kopf
  import pykube
  
  @kopf.on.create('zalando.org', 'v1', 'kopfexamples')
  def spawn_kexy_pods(**_):
      pod_body = {'spec': {'scheduler-name': 'scheduler-ABC', ...}, ...}  # or parse a yaml template
      kopf.adapt(pod_body)  # for cascaded deletions
      # kopf.label(pod_body, {'mykexypod': 'yes'})  # perhaps, not needed
  
      api = pykube.HTTPClient()
      pod = pykube.Pod(api, pod_body)
      pod.create()
  
  
  def _designated_for_us(spec, **_):
      return spec.get('scheduler-name') == 'scheduler-ABC'
  
  
  @kopf.on.create('', 'v1', 'pods',
                  # labels={'mykexypod': 'yes'},  # perhaps, not needed
                  when=_designated_for_us)
  def bind_pod_to_node(namespace, name, patch, **_):
      node_name = call_api_to_bind_it(namespace, name)
      patch.setdefault('metadata', {}).setdefault('labels', {})['node'] = node_name
  
  
  # The code below is optional:
  def _assigned_to_a_node(old, new, **_):
      old_node = old.get('metadata', {}).get('labels', {}).get('node')
      new_node = new.get('metadata', {}).get('labels', {}).get('node')
      return new_node is not None and old_node != new_node
  
  
  @kopf.on.update('', 'v1', 'pods', when=_assigned_to_a_node)
  def notice_node_assigned(**_):
      pass  # congrats!
  ```
  
  Specifically:
  
  > Would on.create be triggered if the pod is in a pending state without a node bound to it? Does this handler only get triggered when the pod is successfully deployed?
  
  On-creation handlers are triggered when the pod is seen for the first time. I.e. when it is created. Usually, you can expect the handling to be done near instantly, much before the pod is actually started by any schedulers (but it already exists).
  
  You might want to take a look into the `@on.update` handlers, or `@on.event` low-level handlers — to track when the pod is assigned/bound to the nodes. If you know the field where this information is stored, you can also use `@kopf.on.field(..., field="metadata.labels.node")`.
  
  on-create, on-update, on-delete, on-field handlers will be retried until succeeded. on-event handler is fire-and-forget: if it fails, it will not be retried (or until the new event arrives some time later).
  
  There are no special events for the pod's conditions (yet), and there is no special treatment for the pods above other resource kinds (yet). But that can be expressed via the `when=` filters.
  
  Or do I miss something?
  
  ---
  
  > <a href="https://github.com/cliffburdick"><img align="left" height="30" src="https://avatars1.githubusercontent.com/u/30670611?v=4"></a>            Commented by [cliffburdick](https://github.com/cliffburdick) at _2020-02-01 19:47:35+00:00_
  > &nbsp;
  
  [nolar](https://github.com/nolar) thanks! I think this sounds very promising, and I'll do some prototyping over the coming weeks. I really appreciate your comments, and I'll let you know when I update our project to use it.
  
  By the way, I know nodes are not objects necessarily, but the ability to watch node status and be alerted is also really important for schedulers. This is likely outside the scope of kopf, though, and can still be done by separate code.
  
  ---
  
  > <a href="https://github.com/cliffburdick"><img align="left" height="30" src="https://avatars1.githubusercontent.com/u/30670611?v=4"></a>            Commented by [cliffburdick](https://github.com/cliffburdick) at _2020-08-19 14:59:21+00:00_
  > &nbsp;
  
  Marking this closed as this is now integrated into NHD:
  
  https://github.com/Viasat/nhd