Installing PyAMS application

PyAMS default installation is based on Buildout utility.

Tip

Take a look at virtualenv. This tool provide isolated Python environments, which are more practical than installing packages systemwide. They also allow installing packages without administrator privileges.

Current PyAMS version is based and validated for Python 3.5; your Python environment must also include a C compiler as well as development headers for Python, libjpeg, libpng, libfreetype, libxml2, libxslt and eventually libldap, libpq, libffi, libgdal or libzmq. Cython compiler can also be useful to optimize several packages.

Note

PyAMS default components configuration also pre-suppose that the following external services are available.

  • Mandatory services:

    • a ZODB shared server (see ZODB server before). Several background processes needing a concurrent access to ZODB are started by PyAMS main process. Three ZODB storages are already provided through PyAMS: ZEO, RelStorage or Newt.db.
    • a Memcached or Redis* server, to store sessions and cache

  • Optional services:

    • an LDAP server for authentication
    • an ElasticSearch server for full text indexing (see PyAMS_content_es package)
    • a WebSockets server using AsyncIO. This is used to manage notifications (see PyAMS_notify and PyAMS_notify_ws packages). An out of the box environment can be built using pyams_notify scaffold.

Creating initial buildout

PyAMS provides a new Pyramid scaffold, called pyams, generated via a cookiecutter template.

A simple option to install PyAMS is to create a buildout environment including Pyramid and all PyAMS packages:

$ mkdir /var/local/
$ pip3 install virtualenv
$ virtualenv --python=python3.5 env
$ cd env
$ . bin/activate
(env) $ pip3.5 install cookiecutter
(env) $ cookiecutter hg+http://hg.ztfy.org/pyams/scaffolds/pyams

CookieCutter will ask you for a small set of input variables that you can change or not (parameter [default value]:)

  • pyams_release [latest] : version of PyAMS configuration file to use. “latest” will point to last release; you can also choose to point to a given release (“0.1.4” for example)
  • project_name [PyAMS]: current environment name in “human form”
  • project_slug: [pyams] “technical” package name, based on project name
  • virtual_hostname [pyams.mydomain.com]: Apache virtual-host name
  • webapp_name [webapps]: web application package name
  • webapp_port [6543]: TCP/IP port to use when running application outside Apache.
  • eggs_directory [eggs]: relative or absolute path to directory containing downloaded eggs; this directory can be shared with other projects.
  • logs_directory [/var/log]: absolute path to directory containing Apache’s log files
  • run_user [www-data]: user name under which Apache process will run
  • run_group [www-data]: group name under which Apache process will run
  • beaker_backend : name of Beaker backend to use to store sessions and cache data (“redis” as default)
  • beaker_server [127.0.0.1:6379]: IP address and port of Beaker backend server
  • db_type [zeo]: ZODB database storage; available options include ZEO, RelStorage and NewtDB
  • db_host [127.0.0.1]: IP address of database server (“127.0.0.1” as default); WARNING: database server installation is not part of application installation; another “zeo_server” cookiecutter recipe is available for ZEO
  • db_port [8100]: listening port of database server (“8100” is given as default for ZEO)
  • db_name [pyams]: database or ZEO storage name to use
  • db_username: database user name
  • db_password: database password
  • zeo_realm: ZEO authentication realm
  • blobs_dir [/var/db/blobs]: local directory to use to store cache of ZODB blobs; cache size is limited to 10GB as default
  • use_postgresql: specify if PostgreSQL access is required; if so, please check that PostgreSQL development files are available to compile PsycoPG2 extension
  • use_oracle: specify if Oracle access is required; if so, please check that Oracle development files are available to compile cx_Oracle extension, and that ORACLE_HOME environment variable is correctly defined (see below)
  • use_ldap: specify if LDAP access will be required for authentication
  • use_elasticsearch: specify if an ElasticSearch server will be used for indexation
  • elasticsearch_server: URL used to access Elasticsearch server (“http://127.0.0.1:9200” as default); this URL can include login and password (“http://login:password@127.0.0.1:9200”), if required…
  • elasticsearch_index: name of Elasticsearch index to use (“pyams” as default)
  • create_elasticsearch_index: specify if Elasticsearch index should be created after installation is complete
  • define_elasticsearch_mappings : specify if Elasticsearch mappings should be defined after installation is complete
  • smtp_server: DNS name of SMTP server (“localhost” as default)
  • smtp_server_name: “human” name given to SMTP server (“pyams” as default)
  • pyams_scheduler [127.0.0.1:5555]: TCP/IP address and port to use to access PyAMS tasks scheduler process (see pyams_scheduler package ⊞)
  • start_scheduler [True]: boolean value to indicate if scheduler process is started by this application instance
  • pyams_medias_converter [127.0.0.1:5556]: TCP/IP address and port to use to access PyAMS medias converter process (see pyams_media package ⊞)
  • start_medias_converter [True]: boolean value to indicate if medias converter process is started by this application instance
  • pyams_es_indexer [127.0.0.1:5557] : TCP/IP address and port to use to access PyAMS Elasticsearch indexer process (see pyams_content_es package ⊞)
  • start_es_indexer boolean value to indicate if Elasticsearch indexer process is started by this application instance
  • use_notifications: specify if PyAMS notifications services are to be used (see pyams_notify package ⊞)
  • pyams_ws_notify [127.0.0.1:8081]: TCP/IP address and port of PyAMS websockets server managing notifications service
  • lexicon_languages: NLTK lexicon languages to use (“en:english fr:french” as default)
  • extension_package: name of a PyAMS extension package to include in environment configuration
  • need_pyams_gis: specify if PyAMS GIS features are to be used by given extension package; if so, please check that libgdal development files are available; on Debian (and maybe others), you have to specify environment variables (see below).

Tip

  1. Setting any option to empty string will keep the default proposed value.
  2. Boolean values accept “true”, “yes”, “on” or “1” (in any case) as true values, and anything else as false;

You can then check, and eventually update, the proposed Buildout configuration file buildout.cfg, to add or remove packages or update settings to your needs.

This last operation can be quite long, as many packages have to downloaded, compiled and installed in the virtual environment. If you encounter any compile error, just install the required dependencies and restart the buildout.

Some dependencies can require the definition of custom environment variables before running buildout, like:

  • for libgdal, which is required by PyAMS_gis package, use:
(env) $ export C_INCLUDE_PATH=/usr/include/gdal
(env) $ export CPLUS_INCLUDE_PATH=/usr/include/gdal

Warning

You have to check also that your libgdal release is matching “GDAL” release given in PyAMS configuration file (actually 2.1.0).

  • for cx_Oracle, which is required if you use Oracle database connections, use:
(env) $ export ORACLE_HOME=/usr/lib/oracle/12.1/client64

These examples are given for Debian GNU/Linux. You may have to adapt configuration based on your own Linux distribution and packages versions.

Then finalize Bootstrap initialization:

(env) $ python3.5 bootstrap.py
(env) $ ./bin/buildout

See also

zc.buildout is a powerful tool for creating repeatable builds of a given software configuration and environment.

Environment settings

The project generated from pyams scaffold is based on default Pyramid’s zodb scaffold, but it adds:

  • a custom application factory, in the webapp directory (see Site Manager guide)
  • a set of directories to store runtime data, in the var directory; each directory contains a README.txt file which should be self-explanatory to indicate what this directory should contain, including a ZEO cache
  • a set of configuration files, in the etc directory; here are standard development.ini and production.ini configuration files, a ZODB configuration files (zodb-zeo.conf) for a ZEO client storage and two Apache configurations (for Apache 2.2 and 2.4) using mod_wsgi.

Once the project have been created from the scaffold, you are free to update all the configuration files.

If you need to add packages to the environment, you have to add them to the buildout.cfg file AND to the INI file (in the pyramid.includes section) before running the buildout another time; don’t forget to add the requested version at the end of buildout.cfg file, as Buildout is not configured by default to automatically download the last release of a given unknown package.

development.ini and production.ini files contain many commented directives related to PyAMS components. Read and update them carefully before initializing your application database!