PyAMS relies mainly on a ZODB (Zope Objects DataBase) to store it’s configuration and for exemple PyAMS_content, but other packages may rely on other database(s).
Concurrent accesses are required on the ZODB, to fully enjoy PyAMS. Several ZODB storages implementations providing a shared access are available and the installation are describe below:
When you start your application in “single process” mode, PyAMS packages start several processes and the “synchronization” is done via ØMQ.
1. ZEO server¶
ZEO (Zope Enterprise Objects) is the first available implementation available for concurrent access to a FileStorage, provided through the ZEO package.
a) Installation with initial buildout¶
PyAMS provides a ZEO server scaffold, called zeo_server, generated via a cookiecutter template.
A simple option to create a ZEO server is to create a buildout environment including ZEO and ZODB packages:
$ mkdir /var/local/ $ pip3 install virtualenv $ virtualenv --python=python3.5 env $ cd env $ . bin/activate (env) $ pip3.5 install cookiecutter (env) $ cookiecutter hg+http://hg.ztfy.org/pyams/scaffolds/zeo_server
CookieCutter will ask you for a small set of input variables that you can change or not:
- pyams_release: version of PyAMS configuration file to use. “latest” (default value) will point to last release; you can also choose to point to a given release (“0.1.4” for example)
- project_name: current environment name in “human form”
- project_slug: “technical” package name, based on project name
- eggs_directory: relative or absolute path to directory containing downloaded eggs; this directory can be shared with other projects (“eggs” as default)
- run_user: user name under which ZEO process will run (“zeo” as default)
- run_group: group name under which ZEO process will run (“zeo” as default)
- zeo_server_port: listening port of ZEO server (“8100” as default)
- zeo_storage: name of first ZEO storage (“pyams” as default)
- zeo_pack_report: email address to which pack reports should be sent
- logs_directory: absolute path to directory containing ZEO’s log files.
A message is displayed after initialization to finalize environment creation:
Your ZEO environment is initialized. To finalize it''s creation, just type: - cd zeo_server - python3.5 bootstrap.py - ./bin/buildout To initialize authentication database, please run following command after buildout: ./bin/zeopasswd -f etc/auth.db -p digest -r "ZEO_server" zeouser xxxx
b) ZEO server configuration¶
All ZEO configuration files are generated in “etc” subdirectory. These includes:
- etc/zeo_server-zdaemon.conf: ZDaemon configuration file
- etc/zeo_server-zeo.conf: ZEO server configuration file
- etc/auth.db: ZEO authentication file; WARNING: this file is not created automatically, you have to create it after buildout.
In these file names, always replace “zeo_server” with the value which was given to “project_slug” variable during CookieCutter template creation.
c) ZEO server tools¶
A set of system configuration files are produced to handle your ZEO environment. These includes:
- etc/init.d/zeo-zeo_server: ZEO server start/stop script in Init-D format. Create a link to this file in /etc/init.d and update Init.d scripts (update-rc.d zeo-zeo_server defaults) to include ZEO in server start/stop process. You can also use this script to start/stop ZEO by hand with start and stop arguments.
- etc/systemd/zeo-zeo_server.service: SystemD service configuration file for ZEO server. Create a link to this file in /etc/systemd/system and reload SystemD daemon (systemctl daemon-reload) before activating ZEO service (systemctl enable zeo-zeo_server.service and systemctl start zeo-zeo_server.service).
- etc/logrotate.d/zeo-zeo_server: LogRotate configuration file for ZEO log files. Create a link to this file in /etc/logrotate.d to activate log rotation for ZEO server.
- etc/cron.d/pack-zeo-zeo_server: Cron configuration file for ZEO database packing. Just create a link to this file in /etc/cron.d directory to enable ZODB packing on a weekly basis (by default).
In these file names, always replace “zeo_server” with the value which was given to “project_slug” variable during CookieCutter template creation. All directory names are those used on a Debian GNU/Linux distribution and may have to be changed on other distributions.
2. RelStorage server¶
RelStorage (http://relstorage.readthedocs.io/en/latest) is an alternate ZODB storage implementation, that stores Python pickles in a relational database; PostgreSQL (>= 9.0), MySQL (>= 5.0.32) and Oracle (> 10g) databases are supported.
To create a database compatible with RelStorage, you just have to install the database server and create a database dedicated to RelStorage; schema initialization is then completely done by RelStorage on application startup.
RelStorage is supposed to provide better performances than ZEO, notably under high load. RelStorage can also get benefit from many extensions (clustering, fail-over, hot-standby…) provided by these databases.
3. NewtDB server¶
NewtDB (http://www.newtdb.org/en/latest) is another ZODB storage implementation. It’s using RelStorage but is dedicated to PostgreSQL (>= 9.5).
NewtDB adds conversion of data from the native serialization used by ZODB to JSON, stored in a PostgreSQL JSONB column. The JSON data supplements the native data to support indexing, search, and access from non-Python application. Because the JSON format is lossy, compared to the native format, the native format is still used for loading objects from the database. For this reason, the JSON data are read-only.
Newt adds a search API for searching the Postgres JSON data and returning persistent objects. It also provides a convenient API for raw data searches.
Database creation is done as with RelStorage, but NewtDB add several schema objects. Migration scripts are available if you need to switch from a classic RelStorage database to a Newt database.
After installation, you can switch from a given storage to another one with the help of the zodbconvert command line script provided by RelStorage.
This Python script is using a configuration file containing directives of both source and target databases, which can be any storage described in the previous sections (or can even use the same storage).
Here is a sample configuration file to convert a ZODB from a ZEO to RelStorage:
%import relstorage <zeoclient source> server zeo-server.mydomain:8100 storage pyams blob-dir /var/local/env/zeo/var/zeoclient/blobs shared-blob-dir false </zeoclient> <relstorage destination> keep-history false blob-dir /var/local/env/pyams/var/relstorage/blobs shared-blob-dir false <postgresql> dsn host='postgresql-server.mydomain' dbname='pyams_rs' user='pyams' password='xxxxxxxx' </postgresql> </relstorage>