Answers: <p>A not very devopsy, but quite simple and effective solution with no security implications (no credential exchange or authentication required) would be to establish a file-based "message" exchange scheme, in a "mailbox" - a well-known filesystem location (setup once and owned by <code>root</code>) with 2 directories:</p>
  
  <ul>
  <li>one owned by the jenkins user which creates <strong>request</strong> files containing deployment request information, one for each site</li>
  <li>one owned by the apache group where each site dedicated users create their own <strong>response</strong> files containing request handling information for deployment requests for their site</li>
  </ul>
  
  <p>When jenkins processing reaches a deployment stage for a particular site it creates a corresponding <strong>request</strong> file with the necessary information in it.</p>
  
  <p>Each site user periodically (cron-driven, for example) checks for <strong>request</strong> files pertaining to their site, handle the request as appropriate, following their own site policies and provide status updates in the respective <strong>response</strong> files, which the jenkins user checks periodically.</p>
  
  <p>When a request handling is completed the jenkins user removes the <strong>request</strong> file, signalling that it received the "message" and then the site user periodical job can remove the corresponding <strong>response</strong> file.</p>
  
  <p>The names of the <strong>request</strong> and <strong>response</strong> files can be used to encode the particular site and request identification, so that the periodical checks don't have to fumble through multiple files.</p>
  
  <p>The scheme can easily work across machines (if, for example, some of the sites are migrated to other servers) simply by placing the "mailbox" on a shared filesystem accessible from all those machines.</p>
  
  <p>OK, an example, as requested. Just a basic skeleton, in python, hopefully self-documenting.</p>
  
  <p>Prerequisites:</p>
  
  <pre><code>sudo mkdir -p /var/message_box/requests
  sudo chown jenkins /var/message_box/requests
  sudo chmod go-w /var/message_box/requests
  sudo mkdir /var/message_box/responses
  sudo chgrp apache /var/message_box/responses
  sudo chmod g+w /var/message_box/responses
  </code></pre>
  
  <p>The <code>mailbox.py</code> file:</p>
  
  <pre><code>#!/usr/bin/python2.7 -u
  
  import logging, os, re, getpass, sys, time, yaml
  
  class Mailbox(object):
  
      base_dir = '/var/message_box'
      request_filename_format = '%s.%s.yaml'  # username.id.yaml
  
      def __init__(self):
          pass
  
      @property
      def request_dir(self):
          return os.path.join(self.base_dir, 'requests')
  
      @property
      def response_dir(self):
          return os.path.join(self.base_dir, 'responses')
  
      def msg_filename(self, user, request_id):
          return self.request_filename_format % (user, request_id)
  
      def request_file(self, user, request_id):
          return os.path.join(self.request_dir, self.msg_filename(user, request_id))
  
      def response_file(self, user, request_id):
          return os.path.join(self.response_dir, self.msg_filename(user, request_id))
  
      def create_msg_file(self, user, request_id, data, is_response=False):
          assert user and request_id and data and isinstance(data, dict)
          msg_file = self.response_file(user, request_id) if is_response else \
                     self.request_file(user, request_id)
          with open(msg_file, 'w') as fd:
              fd.write(yaml.dump(data))
  
      def msg_file_data(self, user, request_id, is_response=False):
          msg_file = self.response_file(user, request_id) if is_response else \
                     self.request_file(user, request_id)
          if os.path.exists(msg_file):
              with open(msg_file) as fd:
                  data = yaml.load(fd.read())
              if data and isinstance(data, dict):  # expected data format
                  return data
          return None
  
      def create_request(self, user, request_id, data):
          self.create_msg_file(user, request_id, data)
          logging.info('created request %s for %s' % (request_id, user))
  
      def create_response(self, request_id, status, response_data=None):
          assert status
          user = getpass.getuser()
          self.create_msg_file(user, request_id, {'status': status, 'data': response_data}, is_response=True)
          logging.info('created response %s with status %s for %s' % (request_id, status, user))
  
  
      def handle_requests(self):
          user = getpass.getuser()
          while True:  # keep handling requests indefinitely
              time.sleep(1)  # new request polling rate, in seconds
              for filename in os.listdir(self.request_dir):
                  m = re.match('(.*)\.(.*)\.yaml', filename)
                  if not m:  # not a valid request filename
                      continue
                  [username, request_id] = m.groups()
                  if username != user:  # not a request for this user
                      continue
                  if os.path.exists(self.response_file(user, request_id)):
                      # request handling already started
                      # you may add here recovery code for request handling interrupted for whatever reason
                      continue
                  msg_data = self.msg_file_data(user, request_id)
                  if not msg_data:  # unexpected data format
                      continue
  
                  logging.info('received request %s: %s' % (request_id, msg_data))
  
                  # mark the request handling start
                  self.create_response(request_id, 'in_progress')
  
                  time.sleep(5)  # mock-up, replace with whatever request handling means
  
                  # mark the request handling done
                  self.create_response(request_id, 'done')  # you can add response data to the dict if needed
  
                  logging.info('handled request %s, waiting for confirmation' % request_id)
  
                  while True:  # wait for confirmation receipt before cleaning up
                      time.sleep(1)  # confirmation receipt polling rate, in seconds
                      if not os.path.exists(self.request_file(user, request_id)):
                          # the deletion of the request file is the confirmation receipt
                          logging.info('confirmation for request %s received, cleaning up' % request_id)
                          os.unlink(self.response_file(user, request_id))  # cleanup response file
                          break
  
  
      def execute_deployment(self, deployment_id, deployment_user, deployment_data):
  
          self.create_request(deployment_user, deployment_id, deployment_data)
          started = False
          while True:  # wait until it's done
              time.sleep(1)  # polling rate, in seconds
              msg_data = self.msg_file_data(deployment_user, deployment_id, is_response=True)
              if msg_data:
                  status = msg_data.get('status')
                  if status:
                      if not started:
                          logging.info('request %s handling started' % deployment_id)
                          started = True
                      if status == 'done':  # job completed
                          logging.info('request %s handling completed, cleaning up' % deployment_id)
                          # cleanup request file, which is the confirmation receipt
                          os.unlink(self.request_file(deployment_user, deployment_id))
                          break
  
  def usage(err_msg, option_parser):
      if err_msg:
          logging.error('%s\n\n%s\n' % (err_msg, option_parser.format_help()))
      sys.exit(-1)
  
  
  if __name__ == "__main__":
      import optparse
  
      logging.basicConfig(level=logging.DEBUG, format="%(levelname)5s  %(asctime)s %(filename)s:%(lineno)d] %(message)s")
      os.umask(022)
  
      p = optparse.OptionParser()
      p.add_option('-c', '--command', action='store', dest='command', choices=['deploy', 'handler'],
                   help='command/mode: &lt;deploy|handler&gt;, mandatory', default=None)
      p.add_option('-i', '--ID', action='store', dest='id',
                   help='deployment ID, mandatory for deploy command', default=None)
      p.add_option('-u', '--user', action='store', dest='user',
                   help='deployment user, mandatory for deploy command', default=None)
      p.add_option('-a', '--artifact', action='store', dest='artifact',
                   help='deployment artifact, mandatory for deploy command', default=None)
  
      opts, _ = p.parse_args()
  
      if not opts.command:
          usage('command is mandatory', p)
  
      mailbox = Mailbox()
  
      if opts.command == 'deploy':
          if not opts.id or not opts.user or not opts.artifact:
              usage('ID and user must be specified for deploy command', p)
          data = {'artifact': opts.artifact}
          mailbox.execute_deployment(opts.id, opts.user, data)
  
      elif opts.command == 'handler':
          mailbox.handle_requests()
  </code></pre>
  
  <p>The jenkins user driving the deployments, deployment info hacked to a string for this example - 'artifact' :</p>
  
  <pre><code>$ ./mailbox.py -c deploy -i 20 -u dancorn -a artifact
   INFO  2017-10-17 13:32:34,663 mailbox.py:49] created request 20 for dancorn
   INFO  2017-10-17 13:32:35,666 mailbox.py:109] request 20 handling started
   INFO  2017-10-17 13:32:40,678 mailbox.py:112] request 20 handling completed, cleaning up
  $ ./mailbox.py -c deploy -i 123 -u dancorn -a artifact
   INFO  2017-10-17 13:33:32,359 mailbox.py:49] created request 123 for dancorn
   INFO  2017-10-17 13:33:33,362 mailbox.py:109] request 123 handling started
   INFO  2017-10-17 13:33:38,375 mailbox.py:112] request 123 handling completed, cleaning up
  $
  </code></pre>
  
  <p>The apache users would launch the handler which in this example remains running (could be converted to a daemon, or a cron-driven approach):</p>
  
  <pre><code>$ ./mailbox.py -c handler
   INFO  2017-10-17 13:32:34,819 mailbox.py:77] received request 20: {'artifact': 'artifact'}
   INFO  2017-10-17 13:32:34,821 mailbox.py:55] created response 20 with status in_progress for dancorn
   INFO  2017-10-17 13:32:39,827 mailbox.py:55] created response 20 with status done for dancorn
   INFO  2017-10-17 13:32:39,827 mailbox.py:87] handled request 20, waiting for confirmation
   INFO  2017-10-17 13:32:40,828 mailbox.py:93] confirmation for request 20 received, cleaning up
   INFO  2017-10-17 13:33:32,888 mailbox.py:77] received request 123: {'artifact': 'artifact'}
   INFO  2017-10-17 13:33:32,889 mailbox.py:55] created response 123 with status in_progress for dancorn
   INFO  2017-10-17 13:33:37,891 mailbox.py:55] created response 123 with status done for dancorn
   INFO  2017-10-17 13:33:37,891 mailbox.py:87] handled request 123, waiting for confirmation
   INFO  2017-10-17 13:33:38,893 mailbox.py:93] confirmation for request 123 received, cleaning up
  </code></pre>
  
Answers: CC BY-SA 3.0
Answers: 2017-10-16T14:06:11.260Z
Answers: a. Wow, it isn't what I've expected..
  b. The periodically checks will make it less "real-time", don't mean to be rude, but there isn't a simpler solution?
  c. Do you have an example of using "mailbox" like you suggested?
Answers: Rotem
Answers: CC BY-SA 3.0
Answers: 2017-10-16T15:09:05.570Z
Answers: I've been using this scheme for a long time, but with full-custom CI setups, not jenkins. Exactly because it addresses the different user problem. Indeed, not real time, but with 1min crons << typical CI durations it's not a big deal, I chose simplicity.
Answers: Dan Cornilescu
Answers: CC BY-SA 3.0
Answers: 2017-10-16T19:21:31.890Z
Answers: Can you share an example of the technical implementation? Maybe a tutorial?
Answers: Rotem
Answers: CC BY-SA 3.0
Answers: 2017-10-17T18:00:54.807Z
Answers: Basic example added.
Answers: Dan Cornilescu
Answers: CC BY-SA 3.0
Answers: 2017-10-16T13:53:21.823Z
Answers: 2017-10-17T18:00:32.723Z
Answers: 2017-10-17T18:00:32.723Z
Answers: Dan Cornilescu
Body: <p>I have a shared web server with multiple sites and each site has a dedicated user used as the owner and with the group apache.</p>
  
  <p>I want to install Jenkins node on the server, but how will it be able to change files and run commands like <code>git pull</code>, I think that using <code>runuser</code> will miss the point.</p>
  
  <p>The script on the node will basically run <code>git pull</code>, <code>drush</code> commands (Drupal sites), rsync from remote server and more.
  I will also need to run chmod and chown at the end, but maybe for those commands I will just use the sudoers file (?).</p>
  
  <p>Thanks.</p>
  
Comments: CC BY-SA 3.0
Comments: 2017-10-16T13:50:16.197Z
Comments: Give jenkins user sudo power to execute as the site owner ?
Comments: Tensibai
Comments: CC BY-SA 3.0
Comments: 2017-10-16T14:16:52.250Z
Comments: @Tensibai but that means that I'll have to add one ssh key for the entire server for all our Git repos. and it is much less secured, isn't it?
Comments: Rotem
Comments: CC BY-SA 3.0
Comments: 2017-10-16T14:45:22.323Z
Comments: Git pull in production is already insecure by nature... I'd argue deploying though a git pull is shooting yourself on the foot, there's practices about building artifacts and deploying them.
Comments: Tensibai
Comments: CC BY-SA 3.0
Comments: 2017-10-16T19:20:14.143Z
Comments: It's not only for production. Do you think that it'll be better to clone it using jenkins and deploy the artifacts? Any articles about that practices?
Comments: Rotem
Comments: CC BY-SA 3.0
Comments: 2017-10-16T19:28:29.750Z
Comments: Did you try googling about artifact releases ? I can't look in a crystal keyboard and guess what you're hiding behind 'and more'. I do have methods for my company's use cases but I doubt they could be a magic wand for your own use case, OSS tools are frameworks, you will have to build upon them. People here will be happy to help you around road blocks but you have to be more precise and start some work before in my own opinion (mod status is unrelated to this comment)
Comments: Tensibai
Tags: <jenkins><continuous-integration>
Title: Jenkins node (agent) on shared web server - what are the right permissions
UserDisplayName: Rotem
ThreadText: # Jenkins node (agent) on shared web server - what are the right permissions
  
  (Asked by: username_0 on 2017-10-16)
  
  I have a shared web server with multiple sites and each site has a dedicated user used as the owner and with the group apache.
  
  I want to install Jenkins node on the server, but how will it be able to change files and run commands like `git pull`, I think that using `runuser` will miss the point.
  
  The script on the node will basically run `git pull`, `drush` commands (Drupal sites), rsync from remote server and more.
  I will also need to run chmod and chown at the end, but maybe for those commands I will just use the sudoers file (?).
  
  Thanks.
  
  **Tags:** jenkins, continuous-integration
  
  > **Comments:**
  > **username_1**: Give jenkins user sudo power to execute as the site owner ?  
  > **username_0**: @username_1 but that means that I'll have to add one ssh key for the entire server for all our Git repos. and it is much less secured, isn't it?  
  > **username_1**: Git pull in production is already insecure by nature... I'd argue deploying though a git pull is shooting yourself on the foot, there's practices about building artifacts and deploying them.  
  > **username_0**: It's not only for production. Do you think that it'll be better to clone it using jenkins and deploy the artifacts? Any articles about that practices?  
  > **username_1**: Did you try googling about artifact releases ? I can't look in a crystal keyboard and guess what you're hiding behind 'and more'. I do have methods for my company's use cases but I doubt they could be a magic wand for your own use case, OSS tools are frameworks, you will have to build upon them. People here will be happy to help you around road blocks but you have to be more precise and start some work before in my own opinion (mod status is unrelated to this comment)  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2017-10-16. Score: 2
  
  A not very devopsy, but quite simple and effective solution with no security implications (no credential exchange or authentication required) would be to establish a file-based "message" exchange scheme, in a "mailbox" - a well-known filesystem location (setup once and owned by `root`) with 2 directories:
  
  * one owned by the jenkins user which creates **request** files containing deployment request information, one for each site
  * one owned by the apache group where each site dedicated users create their own **response** files containing request handling information for deployment requests for their site
  
  When jenkins processing reaches a deployment stage for a particular site it creates a corresponding **request** file with the necessary information in it.
  
  Each site user periodically (cron-driven, for example) checks for **request** files pertaining to their site, handle the request as appropriate, following their own site policies and provide status updates in the respective **response** files, which the jenkins user checks periodically.
  
  When a request handling is completed the jenkins user removes the **request** file, signalling that it received the "message" and then the site user periodical job can remove the corresponding **response** file.
  
  The names of the **request** and **response** files can be used to encode the particular site and request identification, so that the periodical checks don't have to fumble through multiple files.
  
  The scheme can easily work across machines (if, for example, some of the sites are migrated to other servers) simply by placing the "mailbox" on a shared filesystem accessible from all those machines.
  
  OK, an example, as requested. Just a basic skeleton, in python, hopefully self-documenting.
  
  Prerequisites:
  
  ```
  sudo mkdir -p /var/message_box/requests
  sudo chown jenkins /var/message_box/requests
  sudo chmod go-w /var/message_box/requests
  sudo mkdir /var/message_box/responses
  sudo chgrp apache /var/message_box/responses
  sudo chmod g+w /var/message_box/responses
  
  ```
  
  The `mailbox.py` file:
  
  ```
  #!/usr/bin/python2.7 -u
  
  import logging, os, re, getpass, sys, time, yaml
  
  class Mailbox(object):
  
      base_dir = '/var/message_box'
      request_filename_format = '%s.%s.yaml'  # username.id.yaml
  
      def __init__(self):
          pass
  
      @property
      def request_dir(self):
          return os.path.join(self.base_dir, 'requests')
  
      @property
      def response_dir(self):
          return os.path.join(self.base_dir, 'responses')
  
      def msg_filename(self, user, request_id):
          return self.request_filename_format % (user, request_id)
  
      def request_file(self, user, request_id):
          return os.path.join(self.request_dir, self.msg_filename(user, request_id))
  
      def response_file(self, user, request_id):
          return os.path.join(self.response_dir, self.msg_filename(user, request_id))
  
      def create_msg_file(self, user, request_id, data, is_response=False):
          assert user and request_id and data and isinstance(data, dict)
          msg_file = self.response_file(user, request_id) if is_response else \
                     self.request_file(user, request_id)
          with open(msg_file, 'w') as fd:
              fd.write(yaml.dump(data))
  
      def msg_file_data(self, user, request_id, is_response=False):
          msg_file = self.response_file(user, request_id) if is_response else \
                     self.request_file(user, request_id)
          if os.path.exists(msg_file):
              with open(msg_file) as fd:
                  data = yaml.load(fd.read())
              if data and isinstance(data, dict):  # expected data format
                  return data
          return None
  
      def create_request(self, user, request_id, data):
          self.create_msg_file(user, request_id, data)
          logging.info('created request %s for %s' % (request_id, user))
  
      def create_response(self, request_id, status, response_data=None):
          assert status
          user = getpass.getuser()
          self.create_msg_file(user, request_id, {'status': status, 'data': response_data}, is_response=True)
          logging.info('created response %s with status %s for %s' % (request_id, status, user))
  
      def handle_requests(self):
          user = getpass.getuser()
          while True:  # keep handling requests indefinitely
              time.sleep(1)  # new request polling rate, in seconds
              for filename in os.listdir(self.request_dir):
                  m = re.match('(.*)\.(.*)\.yaml', filename)
                  if not m:  # not a valid request filename
                      continue
                  [username, request_id] = m.groups()
                  if username != user:  # not a request for this user
                      continue
                  if os.path.exists(self.response_file(user, request_id)):
                      # request handling already started
                      # you may add here recovery code for request handling interrupted for whatever reason
                      continue
                  msg_data = self.msg_file_data(user, request_id)
                  if not msg_data:  # unexpected data format
                      continue
  
                  logging.info('received request %s: %s' % (request_id, msg_data))
  
                  # mark the request handling start
                  self.create_response(request_id, 'in_progress')
  
                  time.sleep(5)  # mock-up, replace with whatever request handling means
  
                  # mark the request handling done
                  self.create_response(request_id, 'done')  # you can add response data to the dict if needed
  
                  logging.info('handled request %s, waiting for confirmation' % request_id)
  
                  while True:  # wait for confirmation receipt before cleaning up
                      time.sleep(1)  # confirmation receipt polling rate, in seconds
                      if not os.path.exists(self.request_file(user, request_id)):
                          # the deletion of the request file is the confirmation receipt
                          logging.info('confirmation for request %s received, cleaning up' % request_id)
                          os.unlink(self.response_file(user, request_id))  # cleanup response file
                          break
  
      def execute_deployment(self, deployment_id, deployment_user, deployment_data):
  
          self.create_request(deployment_user, deployment_id, deployment_data)
          started = False
          while True:  # wait until it's done
              time.sleep(1)  # polling rate, in seconds
              msg_data = self.msg_file_data(deployment_user, deployment_id, is_response=True)
              if msg_data:
                  status = msg_data.get('status')
                  if status:
                      if not started:
                          logging.info('request %s handling started' % deployment_id)
                          started = True
                      if status == 'done':  # job completed
                          logging.info('request %s handling completed, cleaning up' % deployment_id)
                          # cleanup request file, which is the confirmation receipt
                          os.unlink(self.request_file(deployment_user, deployment_id))
                          break
  
  def usage(err_msg, option_parser):
      if err_msg:
          logging.error('%s\n\n%s\n' % (err_msg, option_parser.format_help()))
      sys.exit(-1)
  
  if __name__ == "__main__":
      import optparse
  
      logging.basicConfig(level=logging.DEBUG, format="%(levelname)5s  %(asctime)s %(filename)s:%(lineno)d] %(message)s")
      os.umask(022)
  
      p = optparse.OptionParser()
      p.add_option('-c', '--command', action='store', dest='command', choices=['deploy', 'handler'],
                   help='command/mode: <deploy|handler>, mandatory', default=None)
      p.add_option('-i', '--ID', action='store', dest='id',
                   help='deployment ID, mandatory for deploy command', default=None)
      p.add_option('-u', '--user', action='store', dest='user',
                   help='deployment user, mandatory for deploy command', default=None)
      p.add_option('-a', '--artifact', action='store', dest='artifact',
                   help='deployment artifact, mandatory for deploy command', default=None)
  
      opts, _ = p.parse_args()
  
      if not opts.command:
          usage('command is mandatory', p)
  
      mailbox = Mailbox()
  
      if opts.command == 'deploy':
          if not opts.id or not opts.user or not opts.artifact:
              usage('ID and user must be specified for deploy command', p)
          data = {'artifact': opts.artifact}
          mailbox.execute_deployment(opts.id, opts.user, data)
  
      elif opts.command == 'handler':
          mailbox.handle_requests()
  
  ```
  
  The jenkins user driving the deployments, deployment info hacked to a string for this example - 'artifact' :
  
  ```
  $ ./mailbox.py -c deploy -i 20 -u dancorn -a artifact
   INFO  2017-10-17 13:32:34,663 mailbox.py:49] created request 20 for dancorn
   INFO  2017-10-17 13:32:35,666 mailbox.py:109] request 20 handling started
   INFO  2017-10-17 13:32:40,678 mailbox.py:112] request 20 handling completed, cleaning up
  $ ./mailbox.py -c deploy -i 123 -u dancorn -a artifact
   INFO  2017-10-17 13:33:32,359 mailbox.py:49] created request 123 for dancorn
   INFO  2017-10-17 13:33:33,362 mailbox.py:109] request 123 handling started
   INFO  2017-10-17 13:33:38,375 mailbox.py:112] request 123 handling completed, cleaning up
  $
  
  ```
  
  The apache users would launch the handler which in this example remains running (could be converted to a daemon, or a cron-driven approach):
  
  ```
  $ ./mailbox.py -c handler
   INFO  2017-10-17 13:32:34,819 mailbox.py:77] received request 20: {'artifact': 'artifact'}
   INFO  2017-10-17 13:32:34,821 mailbox.py:55] created response 20 with status in_progress for dancorn
   INFO  2017-10-17 13:32:39,827 mailbox.py:55] created response 20 with status done for dancorn
   INFO  2017-10-17 13:32:39,827 mailbox.py:87] handled request 20, waiting for confirmation
   INFO  2017-10-17 13:32:40,828 mailbox.py:93] confirmation for request 20 received, cleaning up
   INFO  2017-10-17 13:33:32,888 mailbox.py:77] received request 123: {'artifact': 'artifact'}
   INFO  2017-10-17 13:33:32,889 mailbox.py:55] created response 123 with status in_progress for dancorn
   INFO  2017-10-17 13:33:37,891 mailbox.py:55] created response 123 with status done for dancorn
   INFO  2017-10-17 13:33:37,891 mailbox.py:87] handled request 123, waiting for confirmation
   INFO  2017-10-17 13:33:38,893 mailbox.py:93] confirmation for request 123 received, cleaning up
  
  ```
  
  > **Comments:**
  > **username_0**: a. Wow, it isn't what I've expected.. b. The periodically checks will make it less "real-time", don't mean to be rude, but there isn't a simpler solution? c. Do you have an example of using "mailbox" like you suggested?  
  > **username_2**: I've been using this scheme for a long time, but with full-custom CI setups, not jenkins. Exactly because it addresses the different user problem. Indeed, not real time, but with 1min crons << typical CI durations it's not a big deal, I chose simplicity.  
  > **username_0**: Can you share an example of the technical implementation? Maybe a tutorial?  
  > **username_2**: Basic example added.  
  
  ---
  
  