Use Your Private Python Libraries from Artifactory in Zepl
Zack Shainsky
December 11, 2020
Problem
Oftentimes data science teams build their own custom libraries and store them in common repositories such as Artifactory or Github. These libraries are the culmination of many hours of work and are critical for the data science process. The inevitable need to move from local notebook development environments to cloud-based resources is pushing teams to consider how they will continue to access their private python libraries. This blog will show you how you can access these libraries securely from the Zepl notebook.
Please note:
This blog does not focus on public libraries hosted on the public PyPI server. Zepl has an easy to download and build reusable images from a list of required libraries through our Custom Image builder.
Solution
We will use Zepl’s flexible notebook interface, to access the container shell to integrate our team’s Artifactory Python libraries in Zepl. This same approach can be used to access other types of repositories such as Maven (Java) and CRAN (R).
How to Guide
Step 1. Set up an Artifactory PyPI repo & Python package
Set up an Artifactory PyPI repository and added Python packages. If this is something you have never done before, you can learn more about doing so here:
- Python: Build Python Packages
- Artifactory: Create PyPI Repositories & Upload Packages
Step 2. Access the Container Shell
Each Zepl notebook runs on a linux container. A useful tool in Zepl is accessing the container’s shell to run Operating System commands. This is handled through the python interpreter (%python) and using the ‘!’ symbol.
%python
!<linux shell command here>
In Python, installing libraries is handled by a package manager called, pip. Pip searches for library packages based on a package name and index-url provided. The default index-url is, however we will update this value in step 3 to match your Artifactory repository index-url value.
Step 3. Install Custom Library
In one single line, here is how you can install a Package from your Artifactory repository:
%python
!pip install <LIBRARYNAME>==<LIBRARYVERSION> –index https://<USERNAME>:<PASSWORD>@<JFROG ACCOUNT URL>/artifactory/api/pypi/<YOUR REPO NAME>/simple
Replace these variables in your code:
- <USERNAME> : Your Artifactory username
- <PASSWORD> : Your Artifactory user password
- <JFROG ACCOUNT URL>: Your Artifactory account url
- <YOUR REPO NAME>: Repository with desired package to install
In order to take address security concerns of exposing your user password in clear text, take advantage Zepl’s new Secret Store:
We can verify that the package was installed by running this command:
%python
!pip show
Step 4. Import Custom Library
This is the final step! Make sure that the install statement worked by importing the library into your python code. My example installs a library called “capitalize” and here is what it looks like in Python:
%python
import capitalize
capitalize.capital_mod.capitalize_line("print all words in this sentence as capitals")
Open in Zepl
This Zepl notebook shows an extended example. Feel free to copy or import our sample notebook here.
Conclusion
Installing from Artifactory isn’t much harder than installing a public package. With one single line of code we are able to easily connect to a private Artifactory PyPI repository, install the library in a Zepl container!