Answers: <p>I agree that relying on "developer education" is quite futile, at least when it's the sole course of action. Combine this, however, with automation and you can achieve far greater impact. Both sides, <a href="https://dzone.com/articles/dast-vs-sast-a-case-for-dynamic-application-securi" rel="nofollow noreferrer">DAST and SAST</a> can be covered with:</p>
  
  <ol>
  <li><p>Unit test authentication: Make sure that you have unit tests that check registration, login, password reset and that it is integrated with your development process (continuous integration). Implement processes that automatically block changes to your code when they fail.</p></li>
  <li><p>Testing against known vulnerabilities: These are automated DAST tests that test your application from outside. Programs that handle this task include <a href="https://www.continuumsecurity.net/bdd-security/" rel="nofollow noreferrer">BDD</a> and <a href="https://github.com/F-Secure/mittn" rel="nofollow noreferrer">mittn</a> which is inspired by <a href="https://github.com/GAUNTLT/GAUNTLT" rel="nofollow noreferrer">GAUNTLT</a>.</p></li>
  </ol>
  
  <p>Integrate everything <em>properly</em> with a CI server (Jenkins, Gitlab, Travis) and stop having to rely on your programmers' "best effort security"</p>
  
  <p>Note: my opinion of proper CI integration, from a language/platform agnostic view, covers the following</p>
  
  <p><strong>Tests/examples:</strong>
   - unit tests that cover at minimum 80%, better 90%
   - integration tests that ensure the modules work with one another
   - example programs that show how a user should use the API</p>
  
  <p><strong>Build automation:</strong>
   - Automatically build every pushed commit
   - Build library, test and example code
   - Build for each platform/configuration
   - Run the tests</p>
  
  <p><strong>Merge/pull requests:</strong>
   - Block merges into master branch if any thing fails
   - Require code review of at least one other colleague</p>
  
  <p><strong>Code analysis:</strong>
   - Run nightly static code analysis (style, anti-conventions, dangerous constructs)
   - Run nightly dynamic code analysis (memory leaks, race conditions, known security flaws)</p>
  
  <p><strong>Publish results:</strong>
   - Publish graphs/trends of test/analysis history
   - Talk about changes/targets in weekly meetings
   - Act on problem areas with sprints/work packages</p>
  
Answers: CC BY-SA 3.0
Answers: 2018-01-03T13:25:29.520Z
Answers: And how do you ensure there's enough unit tests for the whole application and none of the form are vulnerable to code/sql/javascript injection ? (That's everything this question is about)
Answers: Tensibai
Answers: CC BY-SA 3.0
Answers: 2018-01-03T14:52:11.353Z
Answers: Complement after thinking about it: the overall idea is OK, the main problem is ensuring the test coverage is really covering all the app, and that's far from straightforward to be summarized as 'Integrate properly with a CI server'
Answers: Tensibai
Answers: CC BY-SA 3.0
Answers: 2018-01-09T18:49:45.087Z
Answers: I agree that doing it properly has many facets and has to reflect the team’s values. That being said, there is so much support/many frameworks and the payoff so significant that spending time/money on it is well worth it.
Answers: Moritz
Answers: CC BY-SA 3.0
Answers: 2018-01-09T21:29:34.960Z
Answers: I fully agree it worth it :) It just sounds too summarized in your answer to fully answer the question in my personal opinion.
Answers: Tensibai
Answers: CC BY-SA 3.0
Answers: 2017-12-31T09:59:39.950Z
Answers: 2018-01-09T18:46:55.673Z
Answers: 2018-01-09T18:46:55.673Z
Answers: Moritz
Answers: <p>This is a little bit of a general question as you said, so my answer is a little general too.  We have always made security a part of the development process and we enforce policy to make sure our code is secure.  This would include forcing developers to only use languages that can work with your security tools.  This needs to be part of the review process.  Part of this could be in your automated testing process too.  </p>
  
  <p>In my experience many DevOps companies view security as an afterthought because of the fear of slowing down development.  This runs counter to the reason companies adopt DevOps in the first place.  As you said, security must shift left in the development process.  I believe this is in line with DevOps / agile principles.  Rather than waiting until a product is complete to think about security.  Trying to secure something once it is built or already breached will be far more costly in the long run.</p>
  
Answers: CC BY-SA 3.0
Answers: 2017-12-29T15:04:37.500Z
Answers: I appreciate your reply. As an Application ("software") Security Engineer, this makes me cry, I suppose. As an end user of some of the products, likely, anyway, of the software the teams doing as you describe also breaks my heart and makes me SMH. When will we learn? After how many breaches? Our end user's data and security and privacy are... an after thought. At least we, as a community, admit it. Can we now admit it to our user base as well? Especially when they sue us after we are compromised?
Answers: randomNerdboy
Answers: CC BY-SA 4.0
Answers: 2017-12-28T22:40:20.740Z
Answers: 2020-01-03T13:34:59.193Z
Answers: 2020-01-03T13:34:59.193Z
Answers: John Walsh
Answers: <p>While I completely agree with </p>
  
  <blockquote>
    <p>security must shift left in the development process</p>
  </blockquote>
  
  <p>though I can't completely agree with</p>
  
  <blockquote>
    <p>We all know "developer education" is a lost cause.</p>
  </blockquote>
  
  <p>Because I think of the answer to a Quora question I've seen recently, something like "<em>What is the hardest requirement to a software engineer?</em>".</p>
  
  <p>The answer there was like "*a feature which has highest priority unless it requires too much work". </p>
  
  <p>You probably don't want to do same to your security requirements!</p>
  
  <p>Because many software engineers are very eager to experiment with and learn new things if there is an <em>exactly defined desired outcome</em>.</p>
  
  <p>By this I mean to say that the following steps should be undertaken:</p>
  
  <ul>
  <li><strong>security design</strong>: what are exactly the security requirements by <em><a href="https://en.wikipedia.org/wiki/DevOps_toolchain" rel="nofollow noreferrer">DevOps lifecycle phases</a></em> coming from your customer/business/regulatory bodies?</li>
  <li><strong>security implementation</strong>: for every requirement, you will either find an applicable tool you could plugin to the CI/CD or you will need to build a custom solution (which is part of work requirement and task to your developers).</li>
  </ul>
  
  <p>A great pitfall is from my experience to ask for "state of the art security" either "what developers consider best to realized" because this produces a lot of opinionated discussion including in some cases questioning tool reliability either metrics applicability as such.</p>
  
  <p>To have a good implementation strategy, some research will be required. I will try what comes to my mind mostly from the open source world. In a large scale business critical industrial environment, investing many thousands to licensing of specialized tools is not much compared to financial and image damage numbers we know about from media. </p>
  
  <ul>
  <li><strong>Requirements</strong>: If written requirements exits, do people responsible for that aka <em>business analysts</em> iterate with stakeholders on specific security requirements if such exist? Left shift means also customer should be also aware of the information security and privacy protection as business topic. Which assets are to be protected, which risks are acceptable? </li>
  <li><strong>Static code analysis</strong>: Sonarqube and underlying tools provide metrics on potential vulnerabilities</li>
  <li><strong>Packaging</strong>: modern artifact repositories provide dependency management tools to highlight problems with package vulnerabilities and licenses (which is though not for free). Also information that some very old package version, either discontinued open source project could be already a source risk. In Java/Maven, you've got the dependency-check-maven based on the US NIST national vulnerability database. For Docker images, you have got security scanners as well. </li>
  <li><strong>Deployment</strong> You can grade TLS implementation with the open source Qualys SSL Labs' scanner, there are penetration automation tools from OWASP.</li>
  </ul>
  
  <p><em>Side note</em>. An additional assessment is also a potential conflict between advanced CI automation and existing organizational processes which have implemented secure processes bound to human interaction. Large organizations where software is business critical therefore should connect the lines between their digital transformation programs and the role of CI/CD systems in the value delivery chain.</p>
  
Answers: CC BY-SA 4.0
Answers: 2020-01-03T14:36:06.913Z
Answers: 2020-01-03T14:46:05.923Z
Answers: 2020-01-03T14:46:05.923Z
Answers: Ta Mu
Body: <p>This is a somewhat general question by design, intended for discussion of process. At my current (and all previous) companies we had issues with releasing insecure software into production. This cut across the divide from mobile, web, API's, internal, external, critical and non-critical applications and systems. If we built it, it had/has security issues... just like <em>your</em> company. In addition, we lack support stack (qualified application security engineers who "knew code" and not just glorified pen testers who didn't know JavaScript from Java) so we could not scale services as needed. Hence automated software security scanning in the SDLC and now in the DevSecOps CI/CD pipelines. We face the issues of:</p>
  
  <ol>
  <li><p>Service Level Agreements - our dev teams want/need results in minimal time, we all know that's not happening with either DAST or SAST tools without heavy configuration.</p></li>
  <li><p>Allowing teams to choose their own language and not even having static code support... leaving us with DAST if applicable or, ugh, pen tests, which in my view are essentially useless. </p>
  
  <p>We need to shift software security "left" - from point of inception, not wait until code is in production and hope we detect the vulnerabilities - I am sure someone will find them, most likely <em>not</em> the pen test team we are paying huge money for, however. </p>
  
  <p>Automation of static and dynamic tools, properly configured, driven off a threat model seem to be our only options to get ahead of the game. We all know "developer education" is a lost cause. </p>
  
  <p>How is your company/team solving this, assuming they even are or recognize it as a problem to address? </p></li>
  </ol>
  
Comments: CC BY-SA 3.0
Comments: 2018-01-02T21:41:04.970Z
Comments: Stack Exchange is best suited to questions that can have a definitive answer.  For a broad question like this, you are probably better off asking on Reddit, Quora, or similar forums.
Comments: Xiong Chiamiov
Comments: CC BY-SA 3.0
Comments: 2018-01-03T13:29:12.723Z
Comments: That's indeed too broad, there's some companies on this path (testing 'code security' like checkmarx for exemple, without any endorsing on the results, that's just informative) but 'how are you doing X' is unlikely to give satisfying answers and for this case very unlikely to give backed up answers with tangible results.
Comments: Tensibai
Tags: <process>
Title: Software Security Testing in a DevSecOps Pipeline
UserDisplayName: randomNerdboy
ThreadText: # Software Security Testing in a DevSecOps Pipeline
  
  (Asked by: username_0 on 2017-12-28)
  
  This is a somewhat general question by design, intended for discussion of process. At my current (and all previous) companies we had issues with releasing insecure software into production. This cut across the divide from mobile, web, API's, internal, external, critical and non-critical applications and systems. If we built it, it had/has security issues... just like *your* company. In addition, we lack support stack (qualified application security engineers who "knew code" and not just glorified pen testers who didn't know JavaScript from Java) so we could not scale services as needed. Hence automated software security scanning in the SDLC and now in the DevSecOps CI/CD pipelines. We face the issues of:
  
  1. Service Level Agreements - our dev teams want/need results in minimal time, we all know that's not happening with either DAST or SAST tools without heavy configuration.
  2. Allowing teams to choose their own language and not even having static code support... leaving us with DAST if applicable or, ugh, pen tests, which in my view are essentially useless.
  
     We need to shift software security "left" - from point of inception, not wait until code is in production and hope we detect the vulnerabilities - I am sure someone will find them, most likely *not* the pen test team we are paying huge money for, however.
  
     Automation of static and dynamic tools, properly configured, driven off a threat model seem to be our only options to get ahead of the game. We all know "developer education" is a lost cause.
  
     How is your company/team solving this, assuming they even are or recognize it as a problem to address?
  
  **Tags:** process
  
  > **Comments:**
  > **username_1**: Stack Exchange is best suited to questions that can have a definitive answer.  For a broad question like this, you are probably better off asking on Reddit, Quora, or similar forums.  
  > **username_2**: That's indeed too broad, there's some companies on this path (testing 'code security' like checkmarx for exemple, without any endorsing on the results, that's just informative) but 'how are you doing X' is unlikely to give satisfying answers and for this case very unlikely to give backed up answers with tangible results.  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_3 on 2017-12-31. Score: 2
  
  I agree that relying on "developer education" is quite futile, at least when it's the sole course of action. Combine this, however, with automation and you can achieve far greater impact. Both sides, [DAST and SAST](https://dzone.com/articles/dast-vs-sast-a-case-for-dynamic-application-securi) can be covered with:
  
  1. Unit test authentication: Make sure that you have unit tests that check registration, login, password reset and that it is integrated with your development process (continuous integration). Implement processes that automatically block changes to your code when they fail.
  2. Testing against known vulnerabilities: These are automated DAST tests that test your application from outside. Programs that handle this task include [BDD](https://www.continuumsecurity.net/bdd-security/) and [mittn](https://github.com/F-Secure/mittn) which is inspired by [GAUNTLT](https://github.com/GAUNTLT/GAUNTLT).
  
  Integrate everything *properly* with a CI server (Jenkins, Gitlab, Travis) and stop having to rely on your programmers' "best effort security"
  
  Note: my opinion of proper CI integration, from a language/platform agnostic view, covers the following
  
  **Tests/examples:**
  - unit tests that cover at minimum 80%, better 90%
  - integration tests that ensure the modules work with one another
  - example programs that show how a user should use the API
  
  **Build automation:**
  - Automatically build every pushed commit
  - Build library, test and example code
  - Build for each platform/configuration
  - Run the tests
  
  **Merge/pull requests:**
  - Block merges into master branch if any thing fails
  - Require code review of at least one other colleague
  
  **Code analysis:**
  - Run nightly static code analysis (style, anti-conventions, dangerous constructs)
  - Run nightly dynamic code analysis (memory leaks, race conditions, known security flaws)
  
  **Publish results:**
  - Publish graphs/trends of test/analysis history
  - Talk about changes/targets in weekly meetings
  - Act on problem areas with sprints/work packages
  
  > **Comments:**
  > **username_2**: And how do you ensure there's enough unit tests for the whole application and none of the form are vulnerable to code/sql/javascript injection ? (That's everything this question is about)  
  > **username_2**: Complement after thinking about it: the overall idea is OK, the main problem is ensuring the test coverage is really covering all the app, and that's far from straightforward to be summarized as 'Integrate properly with a CI server'  
  > **username_3**: I agree that doing it properly has many facets and has to reflect the team’s values. That being said, there is so much support/many frameworks and the payoff so significant that spending time/money on it is well worth it.  
  > **username_2**: I fully agree it worth it :) It just sounds too summarized in your answer to fully answer the question in my personal opinion.  
  
  ---
  
  ### Answer by username_4 on 2017-12-28. Score: 1
  
  This is a little bit of a general question as you said, so my answer is a little general too. We have always made security a part of the development process and we enforce policy to make sure our code is secure. This would include forcing developers to only use languages that can work with your security tools. This needs to be part of the review process. Part of this could be in your automated testing process too.
  
  In my experience many DevOps companies view security as an afterthought because of the fear of slowing down development. This runs counter to the reason companies adopt DevOps in the first place. As you said, security must shift left in the development process. I believe this is in line with DevOps / agile principles. Rather than waiting until a product is complete to think about security. Trying to secure something once it is built or already breached will be far more costly in the long run.
  
  > **Comments:**
  > **username_0**: I appreciate your reply. As an Application ("software") Security Engineer, this makes me cry, I suppose. As an end user of some of the products, likely, anyway, of the software the teams doing as you describe also breaks my heart and makes me SMH. When will we learn? After how many breaches? Our end user's data and security and privacy are... an after thought. At least we, as a community, admit it. Can we now admit it to our user base as well? Especially when they sue us after we are compromised?  
  
  ---
  
  ### Answer by username_5 on 2020-01-03. Score: 0
  
  While I completely agree with
  
  > security must shift left in the development process
  
  though I can't completely agree with
  
  > We all know "developer education" is a lost cause.
  
  Because I think of the answer to a Quora question I've seen recently, something like "*What is the hardest requirement to a software engineer?*".
  
  The answer there was like "\*a feature which has highest priority unless it requires too much work".
  
  You probably don't want to do same to your security requirements!
  
  Because many software engineers are very eager to experiment with and learn new things if there is an *exactly defined desired outcome*.
  
  By this I mean to say that the following steps should be undertaken:
  
  * **security design**: what are exactly the security requirements by *[DevOps lifecycle phases](https://en.wikipedia.org/wiki/DevOps_toolchain)* coming from your customer/business/regulatory bodies?
  * **security implementation**: for every requirement, you will either find an applicable tool you could plugin to the CI/CD or you will need to build a custom solution (which is part of work requirement and task to your developers).
  
  A great pitfall is from my experience to ask for "state of the art security" either "what developers consider best to realized" because this produces a lot of opinionated discussion including in some cases questioning tool reliability either metrics applicability as such.
  
  To have a good implementation strategy, some research will be required. I will try what comes to my mind mostly from the open source world. In a large scale business critical industrial environment, investing many thousands to licensing of specialized tools is not much compared to financial and image damage numbers we know about from media.
  
  * **Requirements**: If written requirements exits, do people responsible for that aka *business analysts* iterate with stakeholders on specific security requirements if such exist? Left shift means also customer should be also aware of the information security and privacy protection as business topic. Which assets are to be protected, which risks are acceptable?
  * **Static code analysis**: Sonarqube and underlying tools provide metrics on potential vulnerabilities
  * **Packaging**: modern artifact repositories provide dependency management tools to highlight problems with package vulnerabilities and licenses (which is though not for free). Also information that some very old package version, either discontinued open source project could be already a source risk. In Java/Maven, you've got the dependency-check-maven based on the US NIST national vulnerability database. For Docker images, you have got security scanners as well.
  * **Deployment** You can grade TLS implementation with the open source Qualys SSL Labs' scanner, there are penetration automation tools from OWASP.
  
  *Side note*. An additional assessment is also a potential conflict between advanced CI automation and existing organizational processes which have implemented secure processes bound to human interaction. Large organizations where software is business critical therefore should connect the lines between their digital transformation programs and the role of CI/CD systems in the value delivery chain.
  
  
  ---
  
  