A simple workflow to create a clean python library based on Anaconda

The open-source Anaconda Distribution is an easy way to perform Python/R data science and machine learning on Linux, Windows, and Mac OS X. We could easily prepare an isolated and constant environment on anywhere based on a configure file.

Considering the various kind of strange errors due to the different versions of python and python libraries, this management tool makes our life much more comfortable in deploying and cooperating.

Prerequisite

To make a clean environment, we may operate it on a virtual machine, which may make each of the steps controllable. We will use a docker image to achieve it.

Suppose we have already got a docker environment, and we can call it using command docker.

If we are executing the commands on a Linux environment, please add a 'sudo' before each 'docker' command. "alias docker='sudo docker'" would work as well.

Prepare the configure file

We can launch a clean image using Docker. If we use ubuntu:18.04 as the base image, the command may seem as follow

$ docker run --rm -it -v$(pwd):/src/my-env ubuntu:18.04 /bin/bash
root@897bdfa59727:/#

We could work on /src/my-env, and all our modification in this folder will be saved, and all other edition will discarded.

If it is necessary, we can install some packages using command apt.

root@897bdfa59727:/# apt-get update && apt-get --yes install wget
Get:1 http://archive.ubuntu.com/ubuntu bionic InRelease [242 kB]
(a lot of logs ...)
Running hooks in /etc/ca-certificates/update.d...
done.

And then, we can install anaconda using wget.

root@897bdfa59727:/# wget --progress=dot:mega https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh -O miniconda.sh && \
    chmod +x miniconda.sh && ./miniconda.sh -b -p /usr/local/conda3 && \
    rm -f miniconda.sh
--2019-02-25 19:55:09--  
(a lot of logs ...)
2019-02-25 19:55:14 (12.6 MB/s) - 'miniconda.sh' saved [69826864/69826864]

PREFIX=/usr/local/conda3
installing: python-3.7.1-h0371630_7 ...
Python 3.7.1
(a lot of logs...)
installation finished.

Add /usr/local/conda3/bin in front of the PATH

root@897bdfa59727:/# export PATH=/usr/local/conda3/bin:$PATH
root@897bdfa59727:/# which conda
/usr/local/conda3/bin/conda
root@897bdfa59727:/# which python
/usr/local/conda3/bin/python
root@897bdfa59727:/# python --version
Python 3.7.1

If we wish to prepare our python library based on python 3.6, we can install python 3.6 instead

root@897bdfa59727:/# conda install --yes python=3.6
Solving environment: done
(a lot of logs...)
Executing transaction: done
root@897bdfa59727:/# python --version
Python 3.6.8 :: Anaconda, Inc.

And then, we can install our packages here based on conda or pip

root@897bdfa59727:/# conda install --yes pytorch torchvision cudatoolkit=9.0 -c pytorch && \
    pip --no-cache-dir install https://github.com/kpu/kenlm/archive/master.zip && \
    pip --no-cache-dir install python_speech_features webrtcvad ffmpeg-python redis && \
    conda install -c conda-forge --yes celery logzero pysoundfile resampy pydub ffmpeg nuitka
Collecting package metadata: done
Solving environment: done
(a lot of logs...)
Successfully installed...

Test the environment, we can also add/remove some libraries, and test it based on the project.

If everything is done, we can fix it using env export

root@897bdfa59727:/# conda env export -n base | grep -v "^prefix: " > /src/my-env/environment.yml
root@897bdfa59727:/# head /src/my-env/environment.yml
name: base
channels:
  - pytorch
  - conda-forge
  - defaults
dependencies:
  - amqp=2.4.1=py_0
  - asn1crypto=0.24.0=py36_0
  - billiard=3.5.0.4=py36h14c3975_1000
  - blas=1.0=mkl

If we quit the environment, the docker container will automatically be removed. However, we can find the generated environment.yml

If we prepare a new Dockerfile, we can prepare the base image as follow:

FROM ubuntu:18.04

ENV LANG=C.UTF-8 LC_ALL=C.UTF-8

# Install tools
RUN apt-get update \
        && apt-get --yes install \
        wget \
        && rm -rf /var/lib/apt/lists/*

# Install miniconda
RUN wget --progress=dot:mega https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh -O miniconda.sh \
        && chmod +x miniconda.sh && ./miniconda.sh -b -p /usr/local/conda3 \
        && rm -f miniconda.sh

# Set environment PATH
ENV PATH /usr/local/conda3/bin:$PATH

# Update configure files
RUN echo 'export PATH=/usr/local/conda3/bin:$PATH' >> /etc/profile.d/pynn.sh \
        && ln -sf /usr/local/conda3/etc/profile.d/conda.sh /etc/profile.d/

# Copy environment.yml
COPY environment.yml /

# Update conda base libraries
RUN conda env update

# Rest of the operations here

Note: All the operations introduced above are executed in a docker environment, this means we don't need to consider anything about multiple environments.

However, if we are running on a large server, we are highly recommended to use a customized environment.

For example, we can create a brand new environment using python 3.6

root@897bdfa59727:/# conda create -n myenv python=3.6
#
# To activate this environment, use:
# > conda activate myenv
#
# To deactivate an active environment, use:
# > conda deactivate
#

We can conda activate myenv to enter an isolated environment, and continue our work.

Please see here for more introduction.