.. _zodb:


ZODB server
============


Installation
''''''''''''

PyAMS relies mainly on a ZODB (**\Z**\ope **\O**\bjects **\D**\ata\ **\B**\ase) to store it's configuration and for exemple *PyAMS_content*, but other packages may
rely on other database(s).

Concurrent accesses are **required** on the ZODB, to fully enjoy PyAMS.
Several ZODB storages implementations providing a shared access are available and the installation are describe below:

    * `ZEO Server`_
    * `RelStorage`_
    * `Newt.DB`_


.. note::

    When you start your application in "*single process*" mode, PyAMS packages start several processes
    and the "*synchronization*" is done via **ØMQ**.



.. _`ZEO Server`:

1. ZEO server
+++++++++++++

ZEO (**\Z**\ope **\E**\nterprise **\O**\bjects) is the first available implementation available for concurrent access to a
FileStorage, provided through the ZEO package.

.. seealso::

    ZEO package documentation and complete configuration settings are available on PyPI_ (cf: `ZOE 5.2.0`_).

.. _PyPI: https://pypi.python.org/pypi/ZEO
.. _`ZOE 5.2.0`: https://pypi.python.org/pypi/ZEO/5.2.0


a) Installation with initial buildout
-------------------------------------

PyAMS provides a ZEO server scaffold, called *zeo_server*, generated via a *cookiecutter* template.

A simple option to create a ZEO server is to create a buildout environment including *ZEO* and *ZODB* packages:

.. code-block:: bash

    $ mkdir /var/local/
    $ pip3 install virtualenv
    $ virtualenv --python=python3.5 env
    $ cd env
    $ . bin/activate
    (env) $ pip3.5 install cookiecutter
    (env) $ cookiecutter hg+http://hg.ztfy.org/pyams/scaffolds/zeo_server

*CookieCutter* will ask you for a small set of input variables that you can change or not:

- **pyams_release**: version of PyAMS configuration file to use. "latest" (default value) will point to last release;
  you can also choose to point to a given release ("0.1.4" for example)

- **project_name**: current environment name in "human form"

- **project_slug**: "technical" package name, based on project name

- **eggs_directory**: relative or absolute path to directory containing downloaded eggs; this directory can be
  shared with other projects ("eggs" as default)

- **run_user**: user name under which ZEO process will run ("zeo" as default)

- **run_group**: group name under which ZEO process will run ("zeo" as default)

- **zeo_server_port**: listening port of ZEO server ("8100" as default)

- **zeo_storage**: name of first ZEO storage ("pyams" as default)

- **zeo_pack_report**: email address to which pack reports should be sent

- **logs_directory**: absolute path to directory containing ZEO's log files.

A message is displayed after initialization to finalize environment creation:

.. code::

    Your ZEO environment is initialized.
    To finalize it''s creation, just type:
    - cd zeo_server
    - python3.5 bootstrap.py
    - ./bin/buildout

    To initialize authentication database, please run following command after buildout:
    ./bin/zeopasswd -f etc/auth.db -p digest -r "ZEO_server" zeouser xxxx

b) ZEO server configuration
---------------------------

All ZEO configuration files are generated in "etc" subdirectory. These includes:

- **etc/zeo_server-zdaemon.conf**: ZDaemon configuration file

- **etc/zeo_server-zeo.conf**: ZEO server configuration file

- **etc/auth.db**: ZEO authentication file; WARNING: this file is not created automatically, you have to create it
  after buildout.

In these file names, always replace "zeo_server" with the value which was given to "project_slug" variable during
*CookieCutter* template creation.


c) ZEO server tools
-------------------

A set of system configuration files are produced to handle your ZEO environment. These includes:

- **etc/init.d/zeo-zeo_server**: ZEO server start/stop script in Init-D format. Create a link to this file in
  */etc/init.d* and update Init.d scripts (*update-rc.d zeo-zeo_server defaults*) to include ZEO in server start/stop
  process. You can also use this script to start/stop ZEO by hand with *start* and *stop* arguments.

- **etc/systemd/zeo-zeo_server.service**: SystemD service configuration file for ZEO server. Create a link to this
  file in */etc/systemd/system* and reload SystemD daemon (*systemctl daemon-reload*) before activating ZEO service
  (*systemctl enable zeo-zeo_server.service* and *systemctl start zeo-zeo_server.service*).

- **etc/logrotate.d/zeo-zeo_server**: LogRotate configuration file for ZEO log files. Create a link to this file in
  */etc/logrotate.d* to activate log rotation for ZEO server.

- **etc/cron.d/pack-zeo-zeo_server**: Cron configuration file for ZEO database packing. Just create a link to this
  file in */etc/cron.d* directory to enable ZODB packing on a weekly basis (by default).

In these file names, always replace "zeo_server" with the value which was given to "project_slug" variable during
*CookieCutter* template creation. All directory names are those used on a Debian GNU/Linux distribution and may have
to be changed on other distributions.


.. _RelStorage:

2. RelStorage server
++++++++++++++++++++

RelStorage (http://relstorage.readthedocs.io/en/latest) is an alternate ZODB storage implementation, that stores
Python pickles in a relational database; PostgreSQL (>= 9.0), MySQL (>= 5.0.32) and Oracle (> 10g) databases are
supported.

To create a database compatible with RelStorage, you just have to install the database server and create a database
dedicated to RelStorage; schema initialization is then completely done by RelStorage on application startup.

RelStorage is supposed to provide better performances than ZEO, notably under high load. RelStorage can also get
benefit from many extensions (clustering, fail-over, hot-standby...) provided by these databases.


.. _Newt.DB:

3. NewtDB server
++++++++++++++++

NewtDB (http://www.newtdb.org/en/latest) is another ZODB storage implementation. It's using RelStorage but is
dedicated to PostgreSQL (>= 9.5).

NewtDB adds conversion of data from the native serialization used by ZODB to JSON, stored in a PostgreSQL JSONB
column. The JSON data supplements the native data to support indexing, search, and access from non-Python application.
Because the JSON format is lossy, compared to the native format, the native format is still used for loading
objects from the database. For this reason, the JSON data are read-only.

Newt adds a search API for searching the Postgres JSON data and returning persistent objects. It also provides a
convenient API for raw data searches.

Database creation is done as with RelStorage, but NewtDB add several schema objects. Migration scripts are available
if you need to switch from a classic RelStorage database to a Newt database.


.. _ZODB.migration:

Migration
'''''''''

After installation, you can switch from a given storage to another one with the help of the *zodbconvert* command
line script provided by RelStorage.

This Python script is using a configuration file containing directives of both source and target databases, which can
be any storage described in the previous sections (or can even use the same storage).

Here is a sample configuration file to convert a ZODB from a ZEO to RelStorage:


.. code-block:: html

    %import relstorage

    <zeoclient source>
        server zeo-server.mydomain:8100
        storage pyams
        blob-dir /var/local/env/zeo/var/zeoclient/blobs
        shared-blob-dir false
    </zeoclient>

    <relstorage destination>
        keep-history false
        blob-dir /var/local/env/pyams/var/relstorage/blobs
        shared-blob-dir false
        <postgresql>
            dsn host='postgresql-server.mydomain' dbname='pyams_rs' user='pyams' password='xxxxxxxx'
        </postgresql>
    </relstorage>