Show HN: Hiring DevOps and SREs – testing troubleshooting live
I think this is fantastic. I can see your investment pitch being &quot;the HackerRank&#x2F;LeetCode of devops&quot;. I&#x27;ve only tried one scenario (DNS) so far but it seems well done, not too long, not too easy, not too hard. I am excited to try it out further.<p>First I&#x27;ll address your questions and then add some comments of my own.<p>1. I do a lot of technical interviewing. Many people can _sound_ like they know what they&#x27;re talking about, but it can be a struggle to get to the nitty gritty to really test their understanding. This product seems to have an angle of &quot;take-home tests as a service&quot;, which is cool. Over time I can see how you could come to have a large library of scenarios and use of them would be very helpful in filtering candidates for real, demonstrable competence.<p>2. I like it. I once tried a DIY approach involving having candidates talk to a Kafka server I provided, with client certificates. It was cumbersome. There was a 100% overlap of candidates who succeeded and candidates who could be bothered to take the test at all. From this I gathered that the mere existence of the test was a strong filter, but it&#x27;s also fair to say that judging test difficulty level well is extremely hard. (In other words: &quot;oh no, he wants me to actually understand how to use client certs? Bye.&quot;)<p>3. Pre-interview. Async hiring process with just-in-time candidate attention means the team does not lose too much productivity (as much as hiring is a valuable use of time, developer time in a small company is at a premium).<p>Some quick-fire minor comments: consider making it easier to share the test link (I generated it on my phone, wanted to run on my desktop); the password field is before the username; it wasn&#x27;t super obvious that different VMs were in play between the warmup and real scenario, nor that a scenario could involve <i>multiple</i> VMs itself. Also, you&#x27;ve no doubt thought of this, but have good defences against abuse of your VMs. By the way, have you considered this approach [1] to VMs? More controllable, but less realistic.<p>Some questions&#x2F;suggestions:<p>1. As the test admin I&#x27;d love to to provide &quot;verify.sh&quot; to grade the test using logic I specify. My script would test, for example, &quot;Can this VM now reach host X?&quot; &quot;Is command Y in the bash history?&quot; and dump out a report somewhere. -- What ideas do you have for auto-marking?<p>2. What revenue model do you have in mind?<p>3. How ready is this for real-life use in a hiring pipeline?<p>4. Do you think there is future scope for a customer to specify their own scenarios, or is the idea more that you build up a library of them?<p>5. Have an advanced mode where the candidate must supply a public key to you for SSH access, rather than password. Boom, you&#x27;ve filtered for a basic understanding of public key cryptography.<p>6. I am not sure the user flow currently has a clear distinction between &quot;recruiter view&quot; and &quot;candidate view&quot; but no doubt that is on the roadmap.<p>7. Scenario idea: certificate expiry blocking secure comms between hosts. This happens in K8S clusters often enough. Could be replicated here.<p>8. Just from curiosity, any details you can share about how the stack works? What am I talking to, a docker-compose script of Centos containers? You&#x27;ve done a nice job with the SSH user routing; am I being tunnelled through your jumpbox to the target machine&#x2F;container?<p>That&#x27;s a lot, sorry! I think this is an exciting idea and that you guys are on to something.<p>[1] <a href="https:&#x2F;&#x2F;blog.benjojo.co.uk&#x2F;post&#x2F;qemu-monitor-socket-rce-vnc" rel="nofollow">https:&#x2F;&#x2F;blog.benjojo.co.uk&#x2F;post&#x2F;qemu-monitor-socket-rce-vnc</a>