Answers: <p>Checkout this solution with codepipeline.  <a href="https://aws.amazon.com/blogs/big-data/implement-continuous-integration-and-delivery-of-serverless-aws-glue-etl-applications-using-aws-developer-tools/" rel="nofollow noreferrer">https://aws.amazon.com/blogs/big-data/implement-continuous-integration-and-delivery-of-serverless-aws-glue-etl-applications-using-aws-developer-tools/</a></p>
  
Answers: CC BY-SA 4.0
Answers: 2019-12-20T08:39:19.997Z
Answers: The link looks good. Could you summarize the important elements in the answer to prevent that the information will be lost if the link is deprecated.
Answers: 030
Answers: CC BY-SA 4.0
Answers: 2019-09-19T19:26:35.810Z
Answers: 2019-09-19T19:26:35.810Z
Answers: user17183
Answers: <p>If you are unit testing, you should be mocking external dependencies like Spark and AWS Glue.  Look into Python mocking frameworks like <a href="https://docs.python.org/3/library/unittest.mock.html" rel="nofollow noreferrer">unittest.mock</a>, <a href="https://docs.pytest.org/en/2.9.2/monkeypatch.html" rel="nofollow noreferrer">monkeypatch</a> and <a href="https://pypi.org/project/pytest-mock/" rel="nofollow noreferrer">pytest-mock</a>.  Your unit tests should just test your code and how it interacts with Spark and AWS Glue.  Your unit tests should not test Spark and AWS Glue functionality.  Do that in your component and integration testing.</p>
  
Answers: CC BY-SA 4.0
Answers: 2020-09-22T19:46:50.680Z
Answers: 2020-09-22T19:46:50.680Z
Answers: David R. Willson
Body: <p>I have an AWS Glue job written in Python that I would like to perform pyunit tests on. However, from reading and experimenting on my own, there seem to be quite a few dependencies on the Glue service, making this task very impractical if not impossible. My only thought would be to separate the non glue dependent functions in the job into a separate module to perform unit testing on.</p>
  
  <p>Is unit testing with AWS Glue possible, and if so, what are the best practices for doing so? </p>
  
Comments: CC BY-SA 4.0
Comments: 2019-09-19T19:26:35.810Z
Comments: Checkout this solution with codepipeline. https://aws.amazon.com/blogs/big-data/implement-continuous-integration-and-delivery-of-serverless-aws-glue-etl-applications-using-aws-developer-tools/
Comments: user17183
Comments: CC BY-SA 4.0
Comments: 2019-12-19T13:57:42.503Z
Comments: @user17183 Please add this as an answer by explaining the details rather than only adding a link
Comments: 030
Tags: <amazon-web-services><python><aws-codepipeline><unit-testing>
Title: Unit Test AWS Glue job in CodePipeline
UserDisplayName: Preston Martin
ThreadText: # Unit Test AWS Glue job in CodePipeline
  
  (Asked by: username_0 on 2019-07-08)
  
  I have an AWS Glue job written in Python that I would like to perform pyunit tests on. However, from reading and experimenting on my own, there seem to be quite a few dependencies on the Glue service, making this task very impractical if not impossible. My only thought would be to separate the non glue dependent functions in the job into a separate module to perform unit testing on.
  
  Is unit testing with AWS Glue possible, and if so, what are the best practices for doing so?
  
  **Tags:** amazon-web-services, python, aws-codepipeline, unit-testing
  
  > **Comments:**
  > **username_1**: Checkout this solution with codepipeline. https://aws.amazon.com/blogs/big-data/implement-continuous-integration-and-delivery-of-serverless-aws-glue-etl-applications-using-aws-developer-tools/  
  > **username_2**: @username_1 Please add this as an answer by explaining the details rather than only adding a link  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2019-09-19. Score: 2
  
  Checkout this solution with codepipeline. <https://aws.amazon.com/blogs/big-data/implement-continuous-integration-and-delivery-of-serverless-aws-glue-etl-applications-using-aws-developer-tools/>
  
  > **Comments:**
  > **username_2**: The link looks good. Could you summarize the important elements in the answer to prevent that the information will be lost if the link is deprecated.  
  
  ---
  
  ### Answer by username_3 on 2020-09-22. Score: 1
  
  If you are unit testing, you should be mocking external dependencies like Spark and AWS Glue. Look into Python mocking frameworks like [unittest.mock](https://docs.python.org/3/library/unittest.mock.html), [monkeypatch](https://docs.pytest.org/en/2.9.2/monkeypatch.html) and [pytest-mock](https://pypi.org/project/pytest-mock/). Your unit tests should just test your code and how it interacts with Spark and AWS Glue. Your unit tests should not test Spark and AWS Glue functionality. Do that in your component and integration testing.
  
  
  ---
  
  