Time for reading: 15 minutes
Who am I?
- A Pipeline TD trying to write, test and ship code that requires third party dependencies
For those who are not from the VFX industry, a Pipeline TD is a poor fellow that was too technical to be a VFX artist but didn’t feel pursuing the good, cozy warm life of a software developer. He’s basically a dev who likes movies more than startups, and should seriously reconsider his/her life choices.
Who am I not?
- A Pipeline TD that is fine with just vendorizing such third party dependencies manually for each package I need to ship
NB: I’m not saying vendorizing is evil, but I feel that as devs we should strive for organizing things in a way that’s more efficient and feels less like a “I need a solution that works now, I’m just gonna dump all of my dependencies here and call it a day”. There’s a few interesting insights here: http://bitprophet.org/blog/2012/06/07/on-vendorizing so I definitely recommend having a read. While Vendorizing can sometimes be the way to go, it can make mantaining your dependencies quite hard, because it all boils down to a manual process. It’s also inherently inefficient because each single software package bundles everything without sharing its dependencies.
What do I want?
- A way to be able to follow and build upon the vfx-reference-platform and distribute my python libraries/packages in a easy way. My Python code might rely on some third-party dependencies (like
numpy), because I might need to built more complex tools.
What the heck is Conda? And why do y'all use snakes names?
My surname is literally the name of a snake so I’m not the best person to advise here. But what’s not to like about Pythons and Anacondas?
Anyway, Conda is an open source package management system and environment management system that runs on Windows, macOS and Linux. Think of it as a more advanced
rpm that you can use to create and install software packages.
The main idea of a conda environment is that it creates an isolated space where you have precise control over the software dependencies installed. The control comes from declaring a recipe, a list of software ingredients required to bake a specific software package.
Quoting from conda’s website:
Users can create virtual environments using one of several tools such as Pipenv or Poetry, or a conda virtual environment. Pipenv and Poetry are based around Python’s built-in venv library, whereas conda has its own notion of virtual environments that is lower-level (Python itself is a dependency provided in conda environments).
The beauty of conda is that Python is considered a dependency itself. With pip, instead, is the other way around: pip can be seen as a dependency of Python.
Talk is cheap, show us the code
Ok ok, Linus. Here ya go.
Here’s is how a bare minimum recipe for a conda env looks like.
Since recipe is actually the name for other conda specific stuff, let’s just start calling this with its true name:
name: test-env channels: - conda-forge - defaults dependencies: - ca-certificates=2021.5.30=h033912b_0 - libcxx=11.1.0=habf9029_0 - libffi=3.3=h046ec9c_2 - ncurses=6.2=h2e338ed_4 - openssl=1.1.1k=h0d85af4_0 - python=3.9.4=h9133fd0_0_cpython - readline=8.1=h05e3726_0 - sqlite=3.35.5=h44b9ce1_0 - tk=8.6.10=h0419947_1 - tzdata=2021a=he74cb21_0 - xz=5.2.5=haf1e3a3_1 - zlib=1.2.11=h7795811_1010 prefix: /Users/vvzen/miniconda3/envs/test-env
Let’s analyze that:
name is just the arbitrary name I gave to this environment.
channels indicates from where the packages for this environment can be downloaded.
defaultsis the main and official conda channel, handled by Continuum Analytics.
conda-forgeis an awesome bunch of peeps creating packages and relying on a fully automated CI/CD (running mostly on Azure) to build these packages for a number of platforms. Basically, conda-forge is the home of all community contributed packages. If something is not on default, somebody probably packaged it on conda-forge. If not, join the cult and do it yourself! (I’ll make a blog post about that too, but it’s quite easy and doesn’t require a lot of time!).
You can also have a
local channel, and that’s where conda might come handy for Pipeline TDs. If your code lives in a content network where you don’t have access to the internet, having a local channel is the way to go. All it takes is to have a web server that serves a local directory, and refer to your packages with
file:///. I’ll try to write a bit more about that, too.
dependencies The real fun. A list of all dependencies names, versions and checksums. Right now, what you see is the result of just running a simple
conda create -name test-env python=3 to create a base env with python3. I haven’t specified the minor version of python3, so conda picked the latest available (3.9.4).
prefix Where this environment lives. All of the binaries and libraries installed via conda will live here under the respective
lib etc.. directories. When asked to (
conda init), conda will prepend this dir to your $PATH and other update relevant environment variables. There are many ways to tailor this in order to have as little impact as possible on your existing environment.
Now that we’ve got the hang of a simple environment, let’s spice up things a bit. I’m a Pipeline TD working on a python library to deal with editorial formats, and I know that there’s a nice library for parsing AAF files, so that I don’t reinvent the wheel myself. Let’s install it:
conda install pyaaf2 --channel conda-forge
Note how I specified the channel via the cli args. I know that
pyaaf2 is a niche thing, and it’s not on the
default channel, but somebody (in this case, myself!) has luckily made it available on conda-forge.
Conda will then tell me that
The following NEW packages will be INSTALLED:
You can see the name of the dependency, the channel, the version and something that resembles a checksum.
noarch prefix just means that it’s a pure python package, so it doesn’t need to be compiled against a specific architecture.
If you run
conda env export now (the command to generate that
environment.yml automatically, based on my current active env), we’ll just see a new line popped:
dependencies: [...] - pyaaf2=1.4.0=pyhd8ed1ab_0 - python=3.9.4=h9133fd0_0_cpython [...]
That makes sense. pyaaf2 is a pure python package, with no additional dependencies. Installing it only brought just
pyaaf2 itself. All good.
Let’s now say that I also need to do some data processing, and I want to use the full power of pandas because I need to be able to run complex joins in a simple way.
conda install pandas
will tell me that
The following packages will be downloaded:
package | build ---------------------------|----------------- libblas-3.9.0 | 9_openblas 11 KB conda-forge libcblas-3.9.0 | 9_openblas 11 KB conda-forge libgfortran-5.0.0 |9_3_0_h6c81a4c_22 19 KB conda-forge libgfortran5-9.3.0 | h6c81a4c_22 1.7 MB conda-forge liblapack-3.9.0 | 9_openblas 11 KB conda-forge libopenblas-0.3.15 |openmp_h5e1b9a4_1 8.7 MB conda-forge llvm-openmp-11.1.0 | hda6cdc1_1 268 KB conda-forge numpy-1.20.3 | py39h7eed0ac_1 5.5 MB conda-forge pandas-1.2.4 | py39h4d6be9b_0 10.8 MB conda-forge ------------------------------------------------------------ Total: 27.0 MB
The following NEW packages will be INSTALLED:
libblas conda-forge/osx-64::libblas-3.9.0-9_openblas libcblas conda-forge/osx-64::libcblas-3.9.0-9_openblas libgfortran conda-forge/osx-64::libgfortran-5.0.0-9_3_0_h6c81a4c_22 libgfortran5 conda-forge/osx-64::libgfortran5-9.3.0-h6c81a4c_22 liblapack conda-forge/osx-64::liblapack-3.9.0-9_openblas libopenblas conda-forge/osx-64::libopenblas-0.3.15-openmp_h5e1b9a4_1 llvm-openmp conda-forge/osx-64::llvm-openmp-11.1.0-hda6cdc1_1 numpy conda-forge/osx-64::numpy-1.20.3-py39h7eed0ac_1 pandas conda-forge/osx-64::pandas-1.2.4-py39h4d6be9b_0 python-dateutil conda-forge/noarch::python-dateutil-2.8.1-py_0 python_abi conda-forge/osx-64::python_abi-3.9-1_cp39 pytz conda-forge/noarch::pytz-2021.1-pyhd8ed1ab_0 six conda-forge/noarch::six-1.16.0-pyh6c4a22f_0
pandas uses a lot of dependencies, the most heavy and notable one being numpy. conda has fetched all of them and is promptly telling me how installing this new package is gonna affect my environment. Note that there’s some really low level stuff here, like libgfortran, llvm-openmp (for multiprocessing), et cetera. I don’t see any error so I’m happy to go on and install all of them.
Now my environment.yml contains all of these new dependencies.
But ouch.. wait. I’ve installed numpy 1.20.3, but the vfx-reference-platform for 2021 clearly mentions that numpy should stay at 1.19.x. I’m a good dev, and the main point of the vfx-reference-platform is to make things consistent, in order to avoid future headaches. So lemme just stay on par with that.
conda install numpy=1.19
Once again, conda will tell me that
The following packages will be downloaded:
package | build ---------------------------|----------------- numpy-1.19.5 | py39he588a01_1 5.1 MB conda-forge ------------------------------------------------------------ Total: 5.1 MB
The following packages will be DOWNGRADED:
numpy 1.20.3-py39h7eed0ac_1 –> 1.19.5-py39he588a01_1
Once again, it seems all good. Downgrading numpy didn’t cause any issues with other dependencies, otherwise conda would have warned me. That makes sense, since right now pandas is the only package in my env that requires numpy.
Coming soon: try to break the system by installing conflicting dependencies!
Where conda can shine!
I’m in the process of gathering a few interesting use cases of when conda can save us by avoid the installation of 2 dependencies that require a slightly different and incompatible version of another dependency!
Conda pack - making self-contained tarballs of environments
Amazing, now I’ve got a way to handle environments in a consistent way. So what - does that mean that I have to use conda everywhere? Not just at installation time, but also at runtime? What if I don’t want to, because my pipeline is old and I can’t just reboot it and use conda environments everywhere?
conda-pack is a neat way to package and distribute conda environments to be run on machines that don’t have conda installed.
NB: I personally see it as an alternative to building python wheels, since I still haven’t fully investigated wheels, but there’s another command,’conda build’, that allows you to build wheels too: I just didn’t have the time to play around with it long enough.
This means that you can use conda simply to handle the dependency resolution mechanism at “build time”. Then, you can leave the conda nest, pack your environment, deploy it, untar it and just
source the content of the tar in your shell without the need for conda or virtualenv to be installed. In my case, I can set up a nice GitLab CI/CD that revolves around a curated
environment.yml, and automatically run
conda-pack to generate the tarball whenever somebody makes a new PR.
You can see a really nice demo here: https://conda.github.io/conda-pack/
The gist is:
conda install conda-pack
conda install -c conda-forge conda-pack
conda-pack --name pipeline-third-party-packages -o pipeline-third-party-packages.tar.gz
pipeline-third-party-packages is the name of the environment you want to pack.
Speeding up things with mamba
Dependency resolution is not an easy task, and conda can sometimes take a long time when trying to resolve dependencies and find a compatible subset. Picking your dependencies wisely is crucial to help Conda do its job, and there’s a few tips you can learn to generally help the conda algorithm to do its thing, see https://www.anaconda.com/blog/understanding-and-improving-condas-performance and https://www.anaconda.com/blog/understanding-and-improving-condas-performance.
Another good strategy is to pick a slightly faster conda client, like mamba: https://github.com/mamba-org/mamba. While conda itself is written in python, mamba is written in C++, and exposes a few nice option in its CLI to query who depends on a certain package, and what a certain package depends on.