Shipping Your Own Python to macOS

2020-09-20 1295 words 7 minutes

Contents

With the announcement at Catalina’s release that some third party run times will be removed, and the fact that Python 2 is end of life it is time to ship your own. Just like everything in tech, there are many ways to accomplish this. I have been using a tool for about over a year now called relocatable python.

The reasons I chose to use relocatable python were pretty good ones in my opinion. They are:

Easy to use
Builds full self-contained Python environment
Easily able to wrap it up in a standard installer PKG

Once you have it in an installer package, you can use whatever tools you want to distribute it. Every management tool and application deployment tool should be able to deploy an installer pkg.

quick start guide

Download the repo from the link above from the directory you wish to download it to

1

git clone https://github.com/gregneagle/relocatable-python.git

Look at the --help argument to see what we can do. Ensure you are in the repo folder.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


% ./make_relocatable_python_framework.py --help
Usage: make_relocatable_python_framework.py [options]

Options:
  -h, --help            show this help message and exit
  --destination=DESTINATION
                        Directory destination for the Python.framework
  --baseurl=BASEURL     Override the base URL used to download the framework.
  --os-version=OS_VERSION
                        Override the macOS version of the downloaded pkg.
                        Current supported versions are "10.6" and "10.9". Not
                        all Python version and macOS version combinations are
                        valid.
  --python-version=PYTHON_VERSION
                        Override the version of the Python framework to be
                        downloaded. See available versions at
                        https://www.python.org/downloads/mac-osx/
  --pip-requirements=PIP_REQUIREMENTS
                        Path to a pip freeze requirements.txt file that
                        describes extra Python modules to be installed. If not
                        provided, certain useful modules for macOS will be
                        installed.

Lets make the folders where we want to create our Python environment

1

% sudo mkdir -p /usr/local/acme/ /usr/local/acme/bin

Note: I made two directories there and will explain later why. You can put this anywhere you want. In this example I am using /usr/local but if you want it away from user space you can place it in say something like /opt

Now lets build our first Relocatable Python Package

1

sudo ./make_relocatable_python_framework.py --destination=/usr/local/acme --python-version=3.8.5

Note: you will see the tool output a bunch fo stuff in the terminal, let it finish

Next we will create our symbolic link to make this easier when we want to call this environment in code

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


# go to the bin folder we created
cd /usr/local/acme/bin
# create a symbolic link to our new framework
sudo ln -s /usr/local/acme/Python.framework/Versions/3.8/bin/python3 python3
# now test it
./python3
Python 3.8.5 (v3.8.5:580fbb018f, Jul 20 2020, 12:11:27) 
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
# exit when you are done
>>> exit()

Finally, you just need to use your preferred packaging tool to add this folder path, including the symbolic link in an Apple Installer Package.

Caveats, and things to consider

As XKCD has already pointed out, managing a Python Environment can be a chore

Installation Path

In the above example there are definitely some things to know and consider. To keep the quick start guide less esoteric I just used /usr/local as the directory to deploy my Python 3 environment into. In practice, I actually do not do this. I deploy to /opt/myorg instead to my fleet. I have worked with lots of clever developers over the years, and they need to often install tools to do their job. Those tools often go into /usr/local, thus I stay out of that area. I let user’s have that space to themselves.

Which Python?

I also have Xcode on this Mac, and I also installed the Python3 Xcode tools as well.

1
2


% which python3
/usr/bin/python3

When you install that, it does put python3 in the standard $PATH and if your code calls that path or if you just type python3 in the shell, without typing the full path, you will get that Python environment. Some planning will be needed as well as some decision-making by the IT Admin/Engineer that wants to deploy their own Python. There is no gold standard answer, so you will need to figure out what is the best answer for your team and Org.

Requirements.txt file

The author of Relocatable Python did something clever, but some may have missed it. They made an assumption that if you were to build a Python3 environment to deploy to your macOS fleet you might want the objc bridge, and tools like xattr So, those tools are just included in the default argument. You should note this just in case you plan on customizing your Python3 package to add more Python packages to it. You will need to add those back. One can view the documentation on how this works.

To see what packages you have you can use pip to do so:

1
2


% cd /usr/local/acme/Python.framework/Versions/3.8/bin 
% ./pip3 list

This will output a giant list of packages you have installed. If you want to add more packages you should also include the ones the author of Relocatable Python gave you. When using a requirements.txt file it strictly follows that file. Anything not listed will not get installed.

Refer to the --help output we used earlier to add the requirements.txt file

1

sudo ./make_relocatable_python_framework.py --destination=/usr/local/acme --python-version=3.8.5 --pip-requirements=/path/to/requirements.txt

Mac Admin Community Python

There also those in the Mac Admin community who are already maintaining a public Python3 package which anyone can just go download and/or contribute to. You can find the repo here. This method does make design choices for you. If these design choices are okay with you, then this would be the easiest and fastest way to just ship your own Python3 environment to your fleet.

Virtual Environments

This is also an option, but it seems most Orgs want to ship their own isolated Python and if they need to create a venv they can do so from the Python they have shipped. Personally, I hae never used a venv outside of my on personal development. I chose to ship the environments, so I can control it. If you want to control venv to a fleet I think shipping your own is the best to start and then build your venv off of that.

Tracking Versions, upgrades, and remediation

I think in the past ~3 years I have found 2-4 systems total that had bad Python environments I was shipping. I am not exactly sure why they broke, it could have just been a failed installation. So, I started tracking the status and version I was shipping in a simple Jamf EA. Here is a quick example I wrote in the shell:

1
2
3
4
5


if results=$(/usr/local/acme/bin/python3 -V | awk '{ print $2 }') 
if> then echo "<result>${results}</result>"
then> else echo "<result>false</result>"
else> fi
<result>3.8.5</result>

This script will return a false value for any broken Python environment. So, I have an ongoing policy that will reinstall my Python3 environment scoped to this EA with a value of false so this is intentional. I highly recommend you never use blank values in anything you use. Always be explicit with your data, you will never know what a null or blank string value will affect, down or up stream.

So, you can track it by the version and create policies to auto remediate any broken Python3 environment you encounter

Conclusion and acknowledgements

You have many options to choose from. It really is a lot easier than one might think, and if you have any questions or trouble, please join us in #python and #python-beginners on MacAdmins Slack and various community members can assist

Acknowledgements:

Greg Neagle
Mac Admins Slack
Mac Admin Community