The open-source Anaconda Distribution is an easy way to perform Python/R data science and machine learning on Linux, Windows, and Mac OS X. We could easily prepare an isolated and constant environment on anywhere based on a configure file.
Considering the various kind of strange errors due to the different versions of python and python libraries, this management tool makes our life much more comfortable in deploying and cooperating.
Prerequisite
To make a clean environment, we may operate it on a virtual machine, which may make each of the steps controllable. We will use a docker image to achieve it.
Suppose we have already got a docker environment, and we can call it using command docker.
If we are executing the commands on a Linux environment, please add a 'sudo' before each 'docker' command. "alias docker='sudo docker'" would work as well.
Prepare the configure file
We can launch a clean image using Docker. If we use ubuntu:18.04 as the base image, the command may seem as follow
$ docker run --rm -it -v$(pwd):/src/my-env ubuntu:18.04 /bin/bash root@897bdfa59727:/#
We could work on /src/my-env, and all our modification in this folder will be saved, and all other edition will discarded.
If it is necessary, we can install some packages using command apt.
root@897bdfa59727:/# apt-get update && apt-get --yes install wget Get:1 http://archive.ubuntu.com/ubuntu bionic InRelease [242 kB] (a lot of logs ...) Running hooks in /etc/ca-certificates/update.d... done.
And then, we can install anaconda using wget.
root@897bdfa59727:/# wget --progress=dot:mega https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh -O miniconda.sh && \ chmod +x miniconda.sh && ./miniconda.sh -b -p /usr/local/conda3 && \ rm -f miniconda.sh --2019-02-25 19:55:09-- (a lot of logs ...) 2019-02-25 19:55:14 (12.6 MB/s) - 'miniconda.sh' saved [69826864/69826864] PREFIX=/usr/local/conda3 installing: python-3.7.1-h0371630_7 ... Python 3.7.1 (a lot of logs...) installation finished.
Add /usr/local/conda3/bin in front of the PATH
root@897bdfa59727:/# export PATH=/usr/local/conda3/bin:$PATH root@897bdfa59727:/# which conda /usr/local/conda3/bin/conda root@897bdfa59727:/# which python /usr/local/conda3/bin/python root@897bdfa59727:/# python --version Python 3.7.1
If we wish to prepare our python library based on python 3.6, we can install python 3.6 instead
root@897bdfa59727:/# conda install --yes python=3.6 Solving environment: done (a lot of logs...) Executing transaction: done root@897bdfa59727:/# python --version Python 3.6.8 :: Anaconda, Inc.
And then, we can install our packages here based on conda or pip
root@897bdfa59727:/# conda install --yes pytorch torchvision cudatoolkit=9.0 -c pytorch && \ pip --no-cache-dir install https://github.com/kpu/kenlm/archive/master.zip && \ pip --no-cache-dir install python_speech_features webrtcvad ffmpeg-python redis && \ conda install -c conda-forge --yes celery logzero pysoundfile resampy pydub ffmpeg nuitka Collecting package metadata: done Solving environment: done (a lot of logs...) Successfully installed...
Test the environment, we can also add/remove some libraries, and test it based on the project.
If everything is done, we can fix it using env export
root@897bdfa59727:/# conda env export -n base | grep -v "^prefix: " > /src/my-env/environment.yml root@897bdfa59727:/# head /src/my-env/environment.yml name: base channels: - pytorch - conda-forge - defaults dependencies: - amqp=2.4.1=py_0 - asn1crypto=0.24.0=py36_0 - billiard=3.5.0.4=py36h14c3975_1000 - blas=1.0=mkl
If we quit the environment, the docker container will automatically be removed. However, we can find the generated environment.yml
If we prepare a new Dockerfile, we can prepare the base image as follow:
FROM ubuntu:18.04 ENV LANG=C.UTF-8 LC_ALL=C.UTF-8 # Install tools RUN apt-get update \ && apt-get --yes install \ wget \ && rm -rf /var/lib/apt/lists/* # Install miniconda RUN wget --progress=dot:mega https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh -O miniconda.sh \ && chmod +x miniconda.sh && ./miniconda.sh -b -p /usr/local/conda3 \ && rm -f miniconda.sh # Set environment PATH ENV PATH /usr/local/conda3/bin:$PATH # Update configure files RUN echo 'export PATH=/usr/local/conda3/bin:$PATH' >> /etc/profile.d/pynn.sh \ && ln -sf /usr/local/conda3/etc/profile.d/conda.sh /etc/profile.d/ # Copy environment.yml COPY environment.yml / # Update conda base libraries RUN conda env update # Rest of the operations here
Note: All the operations introduced above are executed in a docker environment, this means we don't need to consider anything about multiple environments.
However, if we are running on a large server, we are highly recommended to use a customized environment.
For example, we can create a brand new environment using python 3.6
root@897bdfa59727:/# conda create -n myenv python=3.6 # # To activate this environment, use: # > conda activate myenv # # To deactivate an active environment, use: # > conda deactivate #
We can conda activate myenv to enter an isolated environment, and continue our work.
Please see here for more introduction.