.. _installationInstructions: ############# Installation ############# **spam** works on **Python 3.6+** (older python versions are no longer maintained). There are now a number of ways to run spam: - **Installing it on "bare metal" on your computer** This works great on Linux and is possible on OSX with some work. The install is a three step process: 1. :ref:`sysDep` 2. :ref:`installVenv` 3. :ref:`spamPIPinstall` (currently only for linux) **or** :ref:`spamGITinstall`. If you want to contribute your code to spam, you need the git install (or the docker image) - :ref:`anacondaInstall` with our package on `conda-forge`_ Works for Linux, OSX and Windows. - Running a **docker image** that we provide Very easy to run and should work on any system that can host docker images. Graphical outputs (`spam-ereg`, `spam-mmr-graphical` or the 'imShowProgress' options for `spam.DIC.register()`) are not yet supported. Before we start, commands for you to execute .. code-block:: console $ look -like this | .. _sysDep: Installing system dependencies =============================== This step is required for both user and developer install. The installation of these required packages (for `spam` compilation and for some of our Python dependencies). The details of this step is very system-dependent, please see below for your system: * :ref:`debianDepends` * :ref:`ScientificLinuxl6.9Depends` * :ref:`otherLinuxDistros` * :ref:`OSXdepends` .. * :ref:`win10depends` Note: we use **R** and **rpy2** only for the generation of random fields. It's considered on optional dependency, however **test_excursions.py** will fail without it. .. _debianDepends: System dependencies for Debian/Ubuntu Linux -------------------------------------------- If you're on Debian/Ubuntu: .. code-block:: console $ sudo apt update $ sudo apt upgrade $ sudo apt install git python3-dev python3-venv python3-tk gcc g++ libeigen3-dev r-base r-cran-randomfields libicu-dev libgmp-dev libmpfr-dev libcgal-dev gmsh libfreetype6-dev libxml2-dev libxslt-dev Now you're ready to go to: :ref:`installVenv` | .. _ScientificLinuxl6.9Depends: System dependencies for Scientific Linux 6.9 --------------------------------------------- **Warning: This is probably out of date**. If you're on Scientific Linux 6.9 (and you don't want to update to a reasonably modern distribution, `e.g.,` it's installed on a cluster): .. code-block:: console $ sudo yum install epel-release $ sudo sh -c 'wget -qO- http://people.redhat.com/bkabrda/scl_python27.repo >> /etc/yum.repos.d/scl.repo' $ sudo yum install git swig python27 python27-python-libs python27-runtime python27-python-devel python-pip python-virtualenv gcc eigen3-devel R gmp-devel mpfr-devel Now you're ready to go to: :ref:`installVenv` | .. _otherLinuxDistros: System dependencies to install for other Linux distributions ------------------------------------------------------------- For other linux installations our dependencies are currently: Python components: - python3 and python3 development files - python3 virtual env (highly recommended to not install **spam** right into the system) - python3 tk libraries for matplotlib Compilation dependencies: - gcc and/or g++ - eigen3 development files - CGAL development files External programs we call: - R - gmsh Dependencies of the pip packages we use: - libfreetype6-dev (for matplotlib) - libxml2-dev - libxslt-dev - libicu-dev - libgmp-dev - libmpfr-dev Once these are install you're ready to go to: :ref:`installVenv` | .. _OSXdepends: System dependencies for OSX 10.14 ---------------------------------- * Install :code:`xcode` * Install the following packages from source *e.g.,* to **/usr/local**, or with `homebrew`_ (*e.g.,* :command:`brew install cmake`). Homwbrew is recommended for simplicity. * **cmake** * **eigen3** * **gmsh** 4.4.1 * **gmp** * **mpfr** * **cgal** * install **Python** 3.7.4 package from `python.org for OSX`_ * (optional, see above) install **R** 3.6.1 package from `R for OSX`_. The **rpy2** Python module requires **llvm** to compile correctly (since there is no openMP support in Apple's clang). * Install **llvm** with `homebrew`_ (*e.g.,* :command:`brew install llvm`) - or the instructions from llvm.org for installing llvm from source (with the clang component), and install openmp from llvm.org as well. * llvm may require that you install OSX SDK headers using the following command (which in turn require the Xcode command line tools): .. code-block:: console $ xcode-select --install $ open /Library/Developer/CommandLineTools/Packages/macOS_SDK_headers_for_macOS_10.14.pkg * add the following to **~/.profile**: .. code-block:: bash # general /usr/local variables , also needed by brew export PATH=$PATH:/usr/local/bin export LIBRARY_PATH=$LIBRARY_PATH:/usr/local/lib export DYLD_FALLBACK_LIBRARY_PATH=$DYLD_FALLBACK_LIBRARY_PATH:/usr/local/lib ########### # for LLVM: export LLVM_PREFIX=/usr/local/opt/llvm # or your llvm install location if not using brew export PATH=$PATH:$LLVM_PREFIX/bin export LIBRARY_PATH=$LIBRARY_PATH:$LLVM_PREFIX/lib export DYLD_FALLBACK_LIBRARY_PATH=$DYLD_FALLBACK_LIBRARY_PATH:$LLVM_PREFIX/lib export CC=$LLVM_PREFIX/bin/clang export CXX=$LLVM_PREFIX/bin/clang++ #the following line may not be needed on all installations export CPLUS_INCLUDE_PATH=$CPLUS_INCLUDE_PATH:/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolcha\ in/usr/include/c++/v1 #if LLVM came with brew you may need this too: export KMP_DUPLICATE_LIB_OK=TRUE ############# Thanks to Adrian Sheppard, Steve Hall, and Ryan Hurley for these OSX install notes. .. _python.org for OSX: https://www.python.org/downloads/mac-osx/ .. _R for OSX: https://cran.r-project.org/bin/macosx/ .. _the homebrew page: https://brew.sh .. _homebrew: https://brew.sh Now you're ready to go to: :ref:`installVenv` | .. .. _win10depends: .. .. System dependencies for Windows 10 .. ----------------------------------- .. .. If you're on Windows 10, please install `Ubuntu 18.04 WSL`_ (Windows Subsystem for Linux), and follow :ref:`debianDepends`. .. .. In order for graphs and images to appear with `matplotlib` you need to first install `VcXsrv`_ and have it running (otherwise `XMing`_ can also work). .. In your Ubuntu subsystem you must then run the following in the terminal: .. .. .. code-block:: bash .. .. echo "export DISPLAY=:0;" >> ~/.bashrc .. .. Now you're ready to go to: :ref:`installVenv`. .. .. Note: If you'd like to run VSCode in Windows 10 and run spam in WSL it is possible, please follow `these`_ `three`_ `links`_. .. .. .. .. _Ubuntu 18.04 WSL: https://www.microsoft.com/en-us/p/ubuntu-1804-lts/9n9tngvndl3q .. .. _XMing: https://sourceforge.net/projects/xming/ .. .. _VcXsrv: https://sourceforge.net/projects/vcxsrv/ .. .. _these: https://code.visualstudio.com/docs/remote/wsl-tutorial .. .. _three: https://code.visualstudio.com/docs/python/python-tutorial .. .. _links: https://code.visualstudio.com/docs/python/environments | .. _installVenv: Set up a Python virtual environment for **spam** ====================================================== If the dependencies above installed correctly, now you can go ahead and create a virtual environment for **spam**. If you're using Anaconda, please see this note: :ref:`anacondaInstall`. You can create the virtual environment any writable folder on your computer. The virtual environment is created, in the folder you want, using: .. code-block:: console $ python3 -m venv spam This will create a virtual environment called **spam** in the folder spam. You can then activate the virtual environment: .. code-block:: console $ source spam/bin/activate (spam) $ You should see :code:`(spam)` written at the beginning of your command line. You will need to activate this virtual environment in the terminal each time you want to use spam -- check for the (spam) at the command prompt! You should then upgrade your virtual environment to the latest version of **pip** and **setuptools**: .. code-block:: console (spam) $ pip install -U pip setuptools Congratulations, now that your virtual environment is set up, you can either: * :ref:`spamPIPinstall` * :ref:`spamGITinstall` | .. _spamPIPinstall: Install **spam** from **pip** ============================== Execute the following line... .. code-block:: console (spam) $ pip install spam **spam** should now be installed! Check whether: .. code-block:: console (spam) $ spam-ldic --help prints out the help for the function, if it does, you're done! | .. _spamGITinstall: Install **spam** from **git** ============================== .. For the moment you need to be added to our Gitlab project. Make an account `on gricad`_. .. .. _on gricad: https://gricad-gitlab.univ-grenoble-alpes.fr/users/sign_in .. ...and ask a core developer to add you to the **spam** project. .. First of all, make sure the :ref:`sysDep` is done. .. 1. Copy SSH public key .. ---------------------- .. .. If you don't have an SSH public/private key pair you can generate one like this: .. .. .. code-block:: console .. .. $ ssh-keygen .. .. Then print out the PUBLIC key .. .. .. code-block:: console .. .. $ cat ~/.ssh/id_rsa.pub .. .. and copy paste it in `here`_ .. .. .. _here: https://ttk.gricad-gitlab.univ-grenoble-alpes.fr/profile/keys 1. Clone **spam** --------------------- It can be cloned anywhere on the computer. For example in a folder :code:`~/bin`. Before cloning go to the directory: .. code-block:: console $ cd ~/bin/ Then clone the repository: .. code-block:: console $ git clone https://gricad-gitlab.univ-grenoble-alpes.fr/ttk/spam.git Now `cd` into the spam directory that has been downloaded: .. code-block:: console $ cd spam 2. Set up virtual environment ------------------------------ Activate the previously-created virtual environment: .. code-block:: console $ source path/to/virtualenv/spam/bin/activate (spam) $ Install pip requirements: .. code-block:: console (spam) $ pip install -r requirements.txt and (optionally) the development tools to calculate coverage, rpy2, and build documentation: .. code-block:: console (spam) $ pip install -r requirements-dev.txt 3. Install **spam** into the :code:`virtualenv` --------------------------------------------------- .. code-block:: console (spam) $ python setup.py install In the future, when you do .. code-block:: console (spam) $ git pull to download the latest version of the code, you'll have to repeat setup install above. | Installing Jupyter for use with spam (optional) ========================================================= You need to install Jupyter (with pip or you package manager). If you work in a virtualenv, you have to add `ipykernel` in it : .. code-block:: console (spam) $ pip install ipykernel and declare your virtualenv in Jupyter : .. code-block:: console (spam) $ python -m ipykernel install --user --name=spam Then you should be able to select this kernel from Jupyter and download the Jupyter Notebook files from the examples. | .. _anacondaInstall: Installing into Anaconda ========================= If you're using an Anaconda Python environment, and you want to install **spam**, there is now a `conda-forge`_ package. Install anaconda (or miniconda), then do (we're using mamba to solve the environment otherwise it si *very* slow, but this is optional): .. code-block:: console $ conda create -n spam python=3.7 $ conda activate spam (spam) $ conda config --add channels conda-forge (spam) $ conda install mamba -c conda-forge (spam) $ mamba install spam To have graphical things working, you might also have to do: .. code-block:: console (spam) $ pip install PyQt5 qimage2ndarray .. from **pip** please try the following: .. .. .. code-block:: console .. .. conda create -n spam python=3.7 .. conda activate spam .. pip install spam .. spam-ldic --help .. .. Thanks to `Jan-Willem Buurlage`_ for the tip. .. .. .. _Jan-Willem Buurlage: http://jwbuurlage.github.io/ .. _conda-forge: https://github.com/conda-forge/spam-feedstock | .. Problem with **tifffile** ? .. ------------------------------- .. .. You may find that **tifffile** complains that it needs a c file to compile :code:`_tifffile`. .. It is not a **spam** issue but a pip/tifffile one. .. It happens when you build **tifffile** for the first time in a **virtualenv** without **numpy**. .. After that, the packages without C extension is cached, and each time you :code:`pip install`, you will use the cached version... .. .. Quick solution, install **tifffile** with :code:`no-cache-dir` option : .. .. .. code-block:: console .. .. (spam) $ pip --no-cache-dir install tifffile .. .. Longer and cleaner solution : .. .. - get and rm the cached package: .. .. .. code-block:: console .. .. (spam) $ pip uninstall tifffile .. (spam) $ pip --verbose install tifffile|grep cached .. ... .. Using cached wheel link: file:///path/to/file .. (spam) $ rm /path/to/file .. .. - uninstall and reinstall in an env with numpy : .. .. .. code-block:: console .. .. (spam) $ pip install numpy .. (spam) $ pip --verbose install tifffile |grep tiffile.so .. ... .. adding 'tifffile/_tifffile.so' .. ... .. _runSpamWithDocker: Running **spam** with docker ================================== With this method you don't need to install **spam** to run it. You only need to install `docker`_. .. _Docker: https://docs.docker.com/get-docker/ We've created a docker image based on `stretch` and `python 3.7` with **spam** and all its dependencies already installed on it. This way you won't have to install anything to run your programs. 1. Pull the image --------------------- .. code-block:: console $ docker pull gricad-registry.univ-grenoble-alpes.fr/ttk/docker-ttk/spam 2. Input/output folder on the host ------------------------------------- On your computer (the `host`), have a folder with all your input data and scripts. The output files will be written in this folder too, as if you were running your script directly on the host. **WARNING**: avoid reading and writing outside of this directory. For example your folder can look like this: .. code-block:: console /home/user/myfolder/ im1.tif im2.tif myscript.py 3. Run your script -------------------- You can now run you script with the following command (change `/home/user/myfolder/`) with the actual path of your folder on the host. .. code-block:: console $ docker run --rm -v /home/user/myfolder:/r --workdir /r gricad-registry.univ-grenoble-alpes.fr/ttk/docker-ttk/spam python myscript.py You can also directly use **spam** commands .. code-block:: console $ docker run --rm -v /home/user/myfolder:/r --workdir /r gricad-registry.univ-grenoble-alpes.fr/ttk/docker-ttk/spam spam-ldic --help 4. Jupyter notebooks --------------------- Finally you can run the image as a jupyter notebook server accessible at `http://127.0.0.1:8888` with the command .. code-block:: console $ docker run --rm -p 8888:8888 -v /home/user/myfolder:/work/notebooks/myfolder gricad-registry.univ-grenoble-alpes.fr/ttk/docker-ttk/spam where `-v /home/user/myfolder:/work/notebooks/myfolder` binds your host folder with the container. It means that all inputs/outputs will be read/written in `/home/user/myfolder`. 5. Development --------------- It is possible to develop spam with the help of docker to avoid having to install all dependencies on your computer. You only need to clone the source on your host: .. code-block:: console [/home/user] $ git clone git@gricad-gitlab.univ-grenoble-alpes.fr:ttk/spam.git which will put the source code in .. code-block:: console /home/user/spam Then you can run a spam container (with all dependencies) and bind your source code folder with the container (only modify `/home/user/spam` to meet your actual path): .. code-block:: console $ docker run -it --rm -v /home/user/spam:/work --workdir /work gricad-registry.univ-grenoble-alpes.fr/ttk/docker-ttk/spam bash which will lead you to a bash terminal on the container in the spam source folder. The workflow can be as follow: * On the container: compile and execute spam * On the host: modify spam with your IDE, look at output files and handle git. __Note:__ there is no need to keep a container alive and you can use standalone command for each step on the container: .. code-block:: console $ # install $ docker run -it --rm -v /home/user/spam:/work --workdir /work gricad-registry.univ-grenoble-alpes.fr/ttk/docker-ttk/spam python setup.py install $ # test $ docker run -it --rm -v /home/user/spam:/work --workdir /work gricad-registry.univ-grenoble-alpes.fr/ttk/docker-ttk/spam pytest $ # doc $ docker run -it --rm -v /home/user/spam:/work --workdir /work gricad-registry.univ-grenoble-alpes.fr/ttk/docker-ttk/spam python setup build_sphinx Installing docker -------------------- We recommend following the `official instructions`_ to install docker. However on debian based systems you can follow these steps: .. code-block:: console $ sudo apt-get remove docker docker-engine docker.io containerd runc $ sudo apt-get update $ sudo apt-get install ca-certificates curl gnupg lsb-release $ curl -fsSL https://download.docker.com/linux/debian/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg $ echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/debian $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null $ sudo apt-get update $ sudo apt-get install docker-ce docker-ce-cli containerd.io $ sudo chmod 666 /var/run/docker.sock .. _official instructions: https://docs.docker.com/get-docker/