Body: <p>We have a pre-receive hook script in a bash shell script that enforces commits with messages having the right ticket number in a pre-defined format. We also have a separate python script that can be called in the pre-receive hook bash script to call PMD to statically analyze source code in a branch that developers push to the target remote branch.</p>
  
  <p>Prior to now, we have been using the pre-receive bash script for validating our development process with respect to our Gitlab server issue tickets.</p>
  
  <p>I have recently attempted to call the python script in the bash script in such a way that our ticketing and milestone validations are done before calling up PMD to expose source code that does not meet the standard. This did not work in that order, so I decided to call the python script to run PMD before the ticket validations. This also resulted in only the python script running alone without calling the rest of the bash script code.</p>
  
  <p>I will appreciate any help. Please kindly find the source code of the pre-receive bash script for when I tried calling python/PMD first. Kindly point me to how I can call the python last after the ticket validations.</p>
  
  <pre class="lang-sh prettyprint-override"><code>#!/bin/bash
  #
  # pre-receive hook for Commit Check
  #
  COMPANY_EMAIL="mycorp.org"
  
  readonly PROGNAME=$(basename $0)
  readonly PROGDIR=$(readlink -m $(dirname $0))
  IS_MERGE=0
  
  check_single_commit()
  {
    COMMIT_CHECK_STATUS=1
      echo "Repo &gt;&gt; $REPOSITORY_BASENAME"
  
      if [[ "$COMMIT_MESSAGE" == "Merge branch"* ]] || [[ "$COMMIT_MESSAGE" == "WIP MI"* ]]; then
        COMMIT_CHECK_STATUS=0
        IS_MERGE=1
      else
      workFlowResult=`java -jar -Dspring.config.location=/home/gitlab/gitlab_custom_hooks/application.properties /home/gitlab/gitlab_custom_hooks/gitlab-tool.jar -prercv "$COMMIT_AUTHOR" "$COMMIT_MESSAGE" "$REPOSITORY_BASENAME"`
      echo "COMMIT_AUTHOR=$COMMIT_AUTHOR, COMMIT_MESSAGE=$COMMIT_MESSAGE, REPOSITORY_BASE=$REPOSITORY_BASENAME"
  
        echo " &gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt; $workFlowResult &gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;" &gt;&amp;2
      if [[ "$workFlowResult" == *"PRE_RECEIVE_OK"* ]]; then
        echo " &gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt; $workFlowResult &gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;" &gt;&amp;2
        COMMIT_CHECK_STATUS=0
      fi
  
      fi
  }
  
  check_all_commits()
  {
    REVISIONS=$(git rev-list $OLD_REVISION..$NEW_REVISION)
    IFS='\n' read -ra LIST_OF_REVISIONS &lt;&lt;&lt; "$REVISIONS"
  if [ $(git rev-parse --is-bare-repository) = true ]
  then
      REPOSITORY_BASENAME=$(basename "$PWD")
  else
      REPOSITORY_BASENAME=$(basename $(readlink -nf "$PWD"/..))
  fi
      echo REPOSITORY_BASENAME is $REPOSITORY_BASENAME
      REPOSITORY_BASENAME=$(basename "$PWD")
      REPOSITORY_BASENAME=${REPOSITORY_BASENAME%.git}
  
    for rid in "${!LIST_OF_REVISIONS[@]}"; do
      REVISION=${LIST_OF_REVISIONS[rid]}
      COMMIT_MESSAGE=$(git cat-file commit $REVISION | sed '1,/^$/d')
      COMMIT_AUTHOR=$(git cat-file commit $REVISION | grep committer | sed 's/^.* \([^@ ]\+@[^ ]\+\) \?.*$/\1/' | sed 's/&lt;//' | sed 's/&gt;//' | sed 's/@$COMPANY_EMAIL//')
      check_single_commit
  
      if [ "$COMMIT_CHECK_STATUS" != "0" ]; then
        echo "Commit validation failed for commit $REVISION" &gt;&amp;2
        exit 1
      fi
  
    done
  }
  
  
  
  echo "Checking the coding style, coding standards and look out for potential issues"
  
  if python /home/gitlab/gitlab_custom_hooks/pmd-policy.py; then
     echo "PMD_CHECK_OK"
     PMD_CHECK_STATUS=0
  
    # Get custom commit message format
      while read OLD_REVISION NEW_REVISION REFNAME ; do
          check_all_commits
      done
  
  
  
      exit 0
  
  else
  
    echo "PMD_CHECK_FAILED: Review your Code and fix the issues"
    exit 1
  
  fi
  </code></pre>
  
  <p>Below is the code for the python script that calls PMD last commit to a branch at the Gitlab server.</p>
  
  <pre class="lang-py prettyprint-override"><code>#!/usr/bin/env python
  
  import subprocess
  import sys
  import tempfile
  import shutil
  import os
  import errno
  
  
  pmd = '/home/gitlab/gitlab_custom_hooks/pmd-bin-5.5.4/bin/run.sh'
  
  # implementing check_output for python &lt; 2.7
  if not hasattr(subprocess, 'check_output'):
      def check_output(*popenargs, **kwargs):
          if 'stdout' in kwargs:
              raise ValueError('stdout argument not allowed, it will be overridden.')
          process = subprocess.Popen(stdout=subprocess.PIPE, *popenargs, **kwargs)
          output, unused_err = process.communicate()
          retcode = process.poll()
          if retcode:
              cmd = kwargs.get("args")
              if cmd is None:
                  cmd = popenargs[0]
              er = subprocess.CalledProcessError(retcode, cmd)
              er.output = output
              raise er
          return output
      subprocess.check_output = check_output
  
  
  # helper for calling executables
  def call(*args, **kwargs):
      return subprocess.check_output(*args, **kwargs).strip()
  
  
  # helper for calling git
  def call_git(cmd, *args, **kwargs):
      return call(['git'] + cmd, *args, **kwargs)
  
  
  # get all new commits from stdin
  def get_commits():
      commits = {}
      for line in sys.stdin:
          old, new, ref = line.strip().split(' ')
          if old == '0000000000000000000000000000000000000000':
              old = '4b825dc642cb6eb9a060e54bf8d69288fbee4904'
  
          if ref not in commits:
              commits[ref] = []
          commits[ref].append({
              'old': old,
              'new': new,
              'files': get_changed_files(old, new)
              })
  
      return commits
  
  
  # get a list of changed files between to commits
  def get_changed_files(old, new):
      return call_git(['diff', '--name-only', old, new]).split('\n')
  
  
  # get filemode, object type (blob,tree,commit), hash for the given file at the
  # given commit
  def get_change_type(commit, filename):
      return call_git(['ls-tree', commit, filename]).split('\t')[0].split(' ')
  
  
  commits = get_commits()
  
  # use the latest file commit only
  print "Cleaning up file list..."
  
  files = {}
  count = 0
  for ref, data in commits.iteritems():
      files[ref] = {}
      for commit in data:
          for filename in commit['files']:
              if not filename.lower().endswith('.java'): continue
              files[ref][filename] = get_change_type(commit['new'], filename)
      count += len(files[ref])
  
  print "%d Files to check in %d branches" % (count, len(files))
  
  # create temporary dir and save a copy of the new files
  tempdir = tempfile.mkdtemp('git_hook')
  for ref, files in files.iteritems():
      for filename, data in files.iteritems():
          dname = os.path.dirname(filename)
          bname = os.path.basename(filename)
          try:
              os.makedirs(os.path.join(tempdir, dname))
          except OSError, exc:
              if exc.errno == errno.EEXIST:  # directory exists already
                  pass
              else:
                  raise
  
          with open(os.path.join(tempdir, dname, bname), 'w') as fp:
              fp.write(call_git(['cat-file', data[1], data[2]]))
  
  try:
      # call checkstyle and/or pmd and print output
      print call([pmd, 'pmd', '-d', tempdir, '-f', 'text', '-R', 'rulesets/java/basic.xml,rulesets/java/unusedcode.xml,rulesets/java/imports.xml,rulesets/java/strings.xml,rulesets/java/braces.xml,rulesets/java/clone.xml,rulesets/java/design.xml,rulesets/java/clone.xml,rulesets/java/finalizers.xml,rulesets/java/junit.xml,rulesets/java/migrating.xml,rulesets/java/optimizations.xml,rulesets/java/strictexception.xml,rulesets/java/sunsecure.xml,rulesets/java/typeresolution.xml'])
      print "SUCCESS"
      sys.exit(0)
  except subprocess.CalledProcessError, ex:
      print ex.output  # print checkstyle and/or pmd messages
      exit(1)
  finally:
      # remove temporary directory
      shutil.rmtree(tempdir)
  </code></pre>
  
Comments: CC BY-SA 3.0
Comments: 2017-05-06T02:34:37.820Z
Comments: It's hard to correlate your description of the events with the code. Can you show some logs - how were the scripts invoked, what actually happened, what were you expecting?
Comments: Dan Cornilescu
Comments: CC BY-SA 3.0
Comments: 2017-05-06T21:34:01.973Z
Comments: @DanCornilescu, it is difficult posting the logs from PMD, as it is copious. The git hook pre-receive is called whenever the Git server (Gitlab in this case) receives a push from the client. It is a standard script that is provided for development teams to customize to suit their workflows. You can fail or allow a push depending on pre-defined conditions. The pre-receive hook, in turn, calls the python script.
Comments: sage
Comments: CC BY-SA 3.0
Comments: 2017-05-06T22:17:02.633Z
Comments: I mean the significant messages from the two scripts, showing how they relate to each other, like "PMD_CHECK_OK" or "PMD_CHECK_FAILED: Review your Code and fix the issues". Or how did the python script exit? Success/failure? etc. You need to be able to debug your scripts and understand what they do to understand how the end-to-end system works and why. Especially when they're not running on your own servers - post mortem logs is all you get - you need to figure out exactly what happened, you messages must tell you that.
Comments: Dan Cornilescu
Comments: CC BY-SA 3.0
Comments: 2017-05-09T07:54:09.357Z
Comments: Isn't what you're after just moving the while loop before the `if python` ?
Comments: Tensibai
Comments: CC BY-SA 3.0
Comments: 2017-05-10T09:28:48.010Z
Comments: @Tensibai, thank you very much for your reply. I have moved it before the if python statement and I have also moved it after. In each case, it seems that the git branch files are consumed by the first script that processes it. This might require a deeper understanding of how git works under the hood. The only option I think is viable in this instance is to move the Gitlab ticket validation code from the calling pre-receive bash script to the python script while the pre-receive script will only be a wrapper for invoking the python script. What do you think?
Comments: sage
Comments: CC BY-SA 3.0
Comments: 2017-05-10T09:44:27.510Z
Comments: I'm unsure of what your scripts does,  there's a bunch of things, including a java call within. both seems to be independent hooks, there's probably something to do to refactor all this into one script
Comments: Tensibai
Comments: CC BY-SA 3.0
Comments: 2017-05-10T09:44:49.187Z
Comments: Or to refactor them to handle only one commit and loop in a wrapper script, can't really tell.
Comments: Tensibai
Tags: <git><gitlab><git-hooks><shell-script><python>
Title: Problems with integrating Python script in a Bash script Git pre-receive hook
UserDisplayName: sage
ThreadText: # Problems with integrating Python script in a Bash script Git pre-receive hook
  
  (Asked by: username_0 on 2017-05-05)
  
  We have a pre-receive hook script in a bash shell script that enforces commits with messages having the right ticket number in a pre-defined format. We also have a separate python script that can be called in the pre-receive hook bash script to call PMD to statically analyze source code in a branch that developers push to the target remote branch.
  
  Prior to now, we have been using the pre-receive bash script for validating our development process with respect to our Gitlab server issue tickets.
  
  I have recently attempted to call the python script in the bash script in such a way that our ticketing and milestone validations are done before calling up PMD to expose source code that does not meet the standard. This did not work in that order, so I decided to call the python script to run PMD before the ticket validations. This also resulted in only the python script running alone without calling the rest of the bash script code.
  
  I will appreciate any help. Please kindly find the source code of the pre-receive bash script for when I tried calling python/PMD first. Kindly point me to how I can call the python last after the ticket validations.
  
  ```sh
  #!/bin/bash
  #
  # pre-receive hook for Commit Check
  #
  COMPANY_EMAIL="mycorp.org"
  
  readonly PROGNAME=$(basename $0)
  readonly PROGDIR=$(readlink -m $(dirname $0))
  IS_MERGE=0
  
  check_single_commit()
  {
    COMMIT_CHECK_STATUS=1
      echo "Repo >> $REPOSITORY_BASENAME"
  
      if [[ "$COMMIT_MESSAGE" == "Merge branch"* ]] || [[ "$COMMIT_MESSAGE" == "WIP MI"* ]]; then
        COMMIT_CHECK_STATUS=0
        IS_MERGE=1
      else
      workFlowResult=`java -jar -Dspring.config.location=/home/gitlab/gitlab_custom_hooks/application.properties /home/gitlab/gitlab_custom_hooks/gitlab-tool.jar -prercv "$COMMIT_AUTHOR" "$COMMIT_MESSAGE" "$REPOSITORY_BASENAME"`
      echo "COMMIT_AUTHOR=$COMMIT_AUTHOR, COMMIT_MESSAGE=$COMMIT_MESSAGE, REPOSITORY_BASE=$REPOSITORY_BASENAME"
  
        echo " >>>>>>>>>>>>>>>>> $workFlowResult >>>>>>>>>>>>>>>>>" >&2
      if [[ "$workFlowResult" == *"PRE_RECEIVE_OK"* ]]; then
        echo " >>>>>>>>>>>>>>>>> $workFlowResult >>>>>>>>>>>>>>>>>" >&2
        COMMIT_CHECK_STATUS=0
      fi
  
      fi
  }
  
  check_all_commits()
  {
    REVISIONS=$(git rev-list $OLD_REVISION..$NEW_REVISION)
    IFS='\n' read -ra LIST_OF_REVISIONS <<< "$REVISIONS"
  if [ $(git rev-parse --is-bare-repository) = true ]
  then
      REPOSITORY_BASENAME=$(basename "$PWD")
  else
      REPOSITORY_BASENAME=$(basename $(readlink -nf "$PWD"/..))
  fi
      echo REPOSITORY_BASENAME is $REPOSITORY_BASENAME
      REPOSITORY_BASENAME=$(basename "$PWD")
      REPOSITORY_BASENAME=${REPOSITORY_BASENAME%.git}
  
    for rid in "${!LIST_OF_REVISIONS[@]}"; do
      REVISION=${LIST_OF_REVISIONS[rid]}
      COMMIT_MESSAGE=$(git cat-file commit $REVISION | sed '1,/^$/d')
      COMMIT_AUTHOR=$(git cat-file commit $REVISION | grep committer | sed 's/^.* \([^@ ]\+@[^ ]\+\) \?.*$/\1/' | sed 's/<//' | sed 's/>//' | sed 's/@$COMPANY_EMAIL//')
      check_single_commit
  
      if [ "$COMMIT_CHECK_STATUS" != "0" ]; then
        echo "Commit validation failed for commit $REVISION" >&2
        exit 1
      fi
  
    done
  }
  
  echo "Checking the coding style, coding standards and look out for potential issues"
  
  if python /home/gitlab/gitlab_custom_hooks/pmd-policy.py; then
     echo "PMD_CHECK_OK"
     PMD_CHECK_STATUS=0
  
    # Get custom commit message format
      while read OLD_REVISION NEW_REVISION REFNAME ; do
          check_all_commits
      done
  
      exit 0
  
  else
  
    echo "PMD_CHECK_FAILED: Review your Code and fix the issues"
    exit 1
  
  fi
  
  ```
  
  Below is the code for the python script that calls PMD last commit to a branch at the Gitlab server.
  
  ```py
  #!/usr/bin/env python
  
  import subprocess
  import sys
  import tempfile
  import shutil
  import os
  import errno
  
  pmd = '/home/gitlab/gitlab_custom_hooks/pmd-bin-5.5.4/bin/run.sh'
  
  # implementing check_output for python < 2.7
  if not hasattr(subprocess, 'check_output'):
      def check_output(*popenargs, **kwargs):
          if 'stdout' in kwargs:
              raise ValueError('stdout argument not allowed, it will be overridden.')
          process = subprocess.Popen(stdout=subprocess.PIPE, *popenargs, **kwargs)
          output, unused_err = process.communicate()
          retcode = process.poll()
          if retcode:
              cmd = kwargs.get("args")
              if cmd is None:
                  cmd = popenargs[0]
              er = subprocess.CalledProcessError(retcode, cmd)
              er.output = output
              raise er
          return output
      subprocess.check_output = check_output
  
  # helper for calling executables
  def call(*args, **kwargs):
      return subprocess.check_output(*args, **kwargs).strip()
  
  # helper for calling git
  def call_git(cmd, *args, **kwargs):
      return call(['git'] + cmd, *args, **kwargs)
  
  # get all new commits from stdin
  def get_commits():
      commits = {}
      for line in sys.stdin:
          old, new, ref = line.strip().split(' ')
          if old == '0000000000000000000000000000000000000000':
              old = '4b825dc642cb6eb9a060e54bf8d69288fbee4904'
  
          if ref not in commits:
              commits[ref] = []
          commits[ref].append({
              'old': old,
              'new': new,
              'files': get_changed_files(old, new)
              })
  
      return commits
  
  # get a list of changed files between to commits
  def get_changed_files(old, new):
      return call_git(['diff', '--name-only', old, new]).split('\n')
  
  # get filemode, object type (blob,tree,commit), hash for the given file at the
  # given commit
  def get_change_type(commit, filename):
      return call_git(['ls-tree', commit, filename]).split('\t')[0].split(' ')
  
  commits = get_commits()
  
  # use the latest file commit only
  print "Cleaning up file list..."
  
  files = {}
  count = 0
  for ref, data in commits.iteritems():
      files[ref] = {}
      for commit in data:
          for filename in commit['files']:
              if not filename.lower().endswith('.java'): continue
              files[ref][filename] = get_change_type(commit['new'], filename)
      count += len(files[ref])
  
  print "%d Files to check in %d branches" % (count, len(files))
  
  # create temporary dir and save a copy of the new files
  tempdir = tempfile.mkdtemp('git_hook')
  for ref, files in files.iteritems():
      for filename, data in files.iteritems():
          dname = os.path.dirname(filename)
          bname = os.path.basename(filename)
          try:
              os.makedirs(os.path.join(tempdir, dname))
          except OSError, exc:
              if exc.errno == errno.EEXIST:  # directory exists already
                  pass
              else:
                  raise
  
          with open(os.path.join(tempdir, dname, bname), 'w') as fp:
              fp.write(call_git(['cat-file', data[1], data[2]]))
  
  try:
      # call checkstyle and/or pmd and print output
      print call([pmd, 'pmd', '-d', tempdir, '-f', 'text', '-R', 'rulesets/java/basic.xml,rulesets/java/unusedcode.xml,rulesets/java/imports.xml,rulesets/java/strings.xml,rulesets/java/braces.xml,rulesets/java/clone.xml,rulesets/java/design.xml,rulesets/java/clone.xml,rulesets/java/finalizers.xml,rulesets/java/junit.xml,rulesets/java/migrating.xml,rulesets/java/optimizations.xml,rulesets/java/strictexception.xml,rulesets/java/sunsecure.xml,rulesets/java/typeresolution.xml'])
      print "SUCCESS"
      sys.exit(0)
  except subprocess.CalledProcessError, ex:
      print ex.output  # print checkstyle and/or pmd messages
      exit(1)
  finally:
      # remove temporary directory
      shutil.rmtree(tempdir)
  
  ```
  
  **Tags:** git, gitlab, git-hooks, shell-script, python
  
  > **Comments:**
  > **username_1**: It's hard to correlate your description of the events with the code. Can you show some logs - how were the scripts invoked, what actually happened, what were you expecting?  
  > **username_0**: @username_1, it is difficult posting the logs from PMD, as it is copious. The git hook pre-receive is called whenever the Git server (Gitlab in this case) receives a push from the client. It is a standard script that is provided for development teams to customize to suit their workflows. You can fail or allow a push depending on pre-defined conditions. The pre-receive hook, in turn, calls the python script.  
  > **username_1**: I mean the significant messages from the two scripts, showing how they relate to each other, like "PMD_CHECK_OK" or "PMD_CHECK_FAILED: Review your Code and fix the issues". Or how did the python script exit? Success/failure? etc. You need to be able to debug your scripts and understand what they do to understand how the end-to-end system works and why. Especially when they're not running on your own servers - post mortem logs is all you get - you need to figure out exactly what happened, you messages must tell you that.  
  > **username_2**: Isn't what you're after just moving the while loop before the `if python` ?  
  > **username_0**: @username_2, thank you very much for your reply. I have moved it before the if python statement and I have also moved it after. In each case, it seems that the git branch files are consumed by the first script that processes it. This might require a deeper understanding of how git works under the hood. The only option I think is viable in this instance is to move the Gitlab ticket validation code from the calling pre-receive bash script to the python script while the pre-receive script will only be a wrapper for invoking the python script. What do you think?  
  > **username_2**: I'm unsure of what your scripts does,  there's a bunch of things, including a java call within. both seems to be independent hooks, there's probably something to do to refactor all this into one script  
  > **username_2**: Or to refactor them to handle only one commit and loop in a wrapper script, can't really tell.  
  
  ---
  
  ## 0 answers
  
  