This document guides you through the basics of installing and configuring openBIS. Its purpose it to allow newcomers to install openBIS and play with it, so exhaustive details are left out on purpose. The complete openBIS administration manual is maintained here Installation and Administrator Guide of the openBIS Server.


Prerequisites

The successful installation of openBIS requires a machine with:

  • Linux or Mac OS X operating system
  • PostgreSQL 11. You are required to have database superuser privileges (by default, the postgres user has those).
  • Java 11.

Please, make sure all the prerequisites are met before continuing to the next chapters of this document. 


Recommended CPU and memory settings 


See Recommended CPU and memory settings for openBIS 20.10



Deployment overview

The basic openBIS deployment consists of two servers, the Application Server (AS) and the Data Store Server (DSS). Generally speaking, the AS manages the metadata and links to the data while the DSS manages the data itself. The AS employs a PostgreSQL server to persist users, authorization information, various entities and their metadata, as well as index information about all datasets. The DSS operates on a managed part of the file system (data store), where the data is kept. One AS can manage multiple DSS instances, which can make sense e.g. if the data are acquired in multiple, distributed laboratories each having a local DSS.

In this document we will describe an installation procedure with a single DSS, where all three servers (AS, DSS and PostgreSQL) run on the same computer. In general, each of them could be deployed on a dedicated machine, which would improve the overall performance of the system.

Note, that regardless of the particular deployment scenario, the servers must be started in the following order

  1. PostgreSQL
  2. openBIS Application Server
  3. Data Store Server

This is required since the AS needs a running PostgreSQL server to start and the DSS needs an AS to boot up. These dependencies are depicted on the deployment overview diagram with arrows pointing from the dependent towards the dependee.

To avoid inconsistent states of the system, shutdowns must be executed backwards - starting from DSS and proceeding with AS and PostgreSQL respectively.

Installation

It is considered a good practice to install openBIS under a dedicated operating system user, but it is not mandatory to do so. Throughout the document we assume that openBIS will be installed and maintained by a user named openbis. The default installation path for the system will be the home folder of openbis. We also refer to the installation path as OPENBIS_HOME.

PostgreSQL server configuration

Create a database user with the same name as the OS user (in our case openbis)

$ sudo -u postgres createuser openbis 
Shall the new role be a superuser? (y/n) n
Shall the new role be allowed to create databases? (y/n) n
Shall the new role be allowed to create more new roles? (y/n) n

The openBIS setup assumes that PostgreSQL is configured with the trust setting on localhost, which means that a user connecting from 127.0.0.1 can connect without providing a password. These settings are changed in the file pg_hba.conf, see http://www.postgresql.org/docs/9.3/static/auth-pg-hba-conf.html. If these settings are not correct, you will get an error of the form "Database role 'openbis' couldn't be created."

If the PostgreSQL server is configured correctly, then the following sequence of commands must return without errors.

> su - openbis
> psql -U postgres -l
...
> psql -l
...

Postgres on a remote server

When the database is on a remote server, the pg_hba.conf and postgresql.conf files need to be configured as follows:        

pg_hba.conf
# TYPE     DATABASE           USER            ADDRESS                 METHOD # application access privilege 
host       database_name      all             0.0.0.0/0                    md5 / trust
host       postgres           all             0.0.0.0/0                    md5 / trust

So, the databse postgres need to be added to the file. Alternatively, instead of specifying each database and the postgres database, the database can be set to "all" as shown below: 


pg_hba.conf
# TYPE  DATABASE        USER            ADDRESS                 METHOD # application access privilege 
  host  all             all             0.0.0.0/0                    md5 / trust
postgresql.conf
listen_addresses= '*'  (or alternatively one or more IP addresses can be listed)

Also have look here how to configure openBIS to connect to the remote Postgres DB.

Manual database creation

If openBIS cannot be allowed to create its own database (because of missing privileges, for example), you can use the script bin/init_database.sh (in your openBIS installation directory) that will create SQL statements to initialize the database. These SQL statememts can be given to be executed by a database administrator. Example:

$ bin/init_database.sh openbis_dev openbis

-- Execute the following SQL script as a database superuser
create user openbis;
create database openbis_dev with owner openbis encoding 'UTF8' template=template0 tablespace=pg_default;
alter database openbis_dev set default_with_oids = off; 
alter database openbis_dev set join_collapse_limit = '32'; 
alter database openbis_dev set from_collapse_limit = '32'; 

\connect openbis_dev

begin;
create table database_version_logs(db_version varchar(4) not null, module_name varchar(250), run_status varchar(10), run_status_timestamp timestamp, module_code bytea, run_exception bytea);
insert into database_version_logs(db_version, run_status, run_status_timestamp) values ('0', 'SUCCESS', now());
alter database openbis_dev owner to openbis;
alter table database_version_logs owner to openbis;
commit;


1st argument to the script is the name of the database to be created. 
2nd argument to the script is the name of the user openBIS will use to access the database we created.
Note: both the database name and user name need to be reflected in service.properties configuration files.

Download binaries

  • Go to openBIS Download Page.
  • Download the latest release of openBIS installer. It should be a TAR.GZ archive and its name should look like openBIS-installation-<version>-r<revision>.tar.gz, where <version> is the release number (e.g. S117).
$ ls -a 
openBIS-installation-XXX.tar.gz


Run the installation

NOTE: openBIS will ask you to provide the install directory. For a new installation this directory is created by the openBIS installer, so it should not already exist. If the directory already exists the installation will fail.

Extract the installation tarball

> tar xzvf openBIS-installation-S117-r23517.tar.gz
x openBIS-installation-S117-r23517/console.properties
x openBIS-installation-S117-r23517/jul.config
x openBIS-installation-S117-r23517/extract.sh
x openBIS-installation-S117-r23517/run-console.sh
x openBIS-installation-S117-r23517/run-ui.sh
x openBIS-installation-S117-r23517/openBIS-installer.jar

Headless installation

To install openBIS on a headless system you need to

  • Edit the file console.properties
  • Execute the script run-console.sh
> cd openBIS-installation-S117-r23517
> ./run-console.sh

Congratulations (thumbs up) ! If you have reached this far your openBIS system is installed and is ready to be started. Please do so by following the instructions in the next section.

Starting/Stopping

The starting scripts of openBIS assume that java is available on the system path, so you have to add it if this is not true.

Launch the openBIS AS and DSS servers by executing <OPENBIS_HOME>/bin/allup.sh.

$ ./bin/allup.sh

To verify AS and DSS function and there are no configurational issues, please check the log files of the two applications using the bislog.sh and dsslog.sh scripts. Additionally, the openBIS application should be accessible at https://localhost:8443/openbis and you should be able to login as the admin user with the password specified in the installation process.

You can shutdown both the servers by executing the <OPENBIS_HOME>/bin/alldown.sh script

$ ./bin/alldown.sh


Configure openBIS to run on one standard port 443

We would like all users who work in the network to use the standard https port (443) to access openbis and the DSS (datastore server).

We start openbis on port 8080 and DSS on port 8081. Those ports are not accessible from outside. Then we redirect all the requests to 443 port from outside to a right internal port depending on the URL suffix. We do this by switching on Apache forwarding (we use mod_proxy).

See the details below.

Apache configuration

  • we configure Apache instance (inside /etc/httpd/conf.d/openbis.conf file on Linux, see attached openbis.conf as an example) to:


<VirtualHost *:443>
 
  ErrorLog /var/log/apache2/ssl_error_log
  TransferLog /var/log/apache2/ssl_access_log
  LogLevel info
 
  ServerName localhost
  SSLEngine on
  SSLProtocol all -SSLv2
  SSLCipherSuite ALL:!ADH:!EXPORT:!SSLv2:RC4+RSA:+HIGH:+MEDIUM:+LOW
 
  SSLCertificateFile /etc/pki/tls/apache.crt
  SSLCertificateKeyFile /etc/pki/tls/apache.key
 
  SSLProxyEngine on
  ProxyRequests Off
  ProxyPreserveHost On
 
  RewriteEngine On
 
AllowEncodedSlashes On
#
#       Rewrite url if there is a missing / at the end
#
# Remove the following line if using cifex!
  RewriteRule ^/$ /openbis/ [R,L]
  RewriteRule ^/openbis$ /openbis/ [R,L]
  RewriteRule ^/datastore_server$ /datastore_server/ [R,L]
 
#Replace localhost with your host name
  ProxyPass /openbis/ http://localhost:8080/openbis/ timeout=600 Keepalive=Off
  ProxyPassReverse /openbis/ http://localhost:8080/openbis/
  ProxyPass /datastore_server/ http://localhost:8081/datastore_server/
  ProxyPassReverse /datastore_server/ http://localhost:8081/datastore_server/


# Used by Screening instances
  ProxyPass /datastore_server_screening http://localhost:8081/datastore_server_screening
  ProxyPassReverse /datastore_server_screening http://localhost:8081/datastore_server_screening

  ProxyPass /rmi-datastore-server-screening-api-v1 http://localhost:8081/rmi-datastore-server-screening-api-v1
  ProxyPassReverse /rmi-datastore-server-screening-api-v1 http://localhost:8081/rmi-datastore-server-screening-api-v1
</VirtualHost>


  • we configure openbis and DSS servers not to use SSL (because Apache uses it already)
  • install SSL module (mod_ssl) for Apache if it is not there: yum install mod_ssl.
  • after changing the configuration you have to restart Apache. On Linux you call:
  • /etc/init.d/httpd restart
    

make sure that Apache will be restarted automatically

[root@~]# chkconfig --list | grep httpd
httpd           0:off   1:off   2:on    3:on    4:on    5:on    6:off


openBIS configuration


Create a file called http.ini in /openBIS-server/jetty/start.d/:

# Module: http
--module=http
### HTTP Connector Configuration
## HTTP port to listen on
jetty.port=8080
## HTTP idle timeout in milliseconds
http.timeout=30000
## HTTP Socket.soLingerTime in seconds. (-1 to disable)
# http.soLingerTime=-1
## Parameters to control the number and priority of acceptors and selectors
# http.selectors=1
# http.acceptors=1
# http.selectorPriorityDelta=0
# http.acceptorPriorityDelta=0

Rename the files https.ini and ssl.ini. These files are then then no longer used by jetty to serve at port 8443.


Settings for the Data Store Server (DSS)

(warning) These changes are needed for all Jetty versions which are used in openBIS

  • do the same in DSS service.properties (sprint/datastore_server/etc) for 'server-url' and 'download-url' properties, additionally add use-ssl = false
  • Changes to DSS service.properties file:

    --- service.properties.ssl      Tue Mar  9 16:51:40 2010
    +++ service.properties  Sat Mar 13 07:54:28 2010
    @@ -8,3 +8,3 @@
     # Port
    -port = 8444
    +port = 8081
    
    @@ -43,4 +43,5 @@
     # The URL of the openBIS server
    -server-url = https://openbis-test.ethz.ch:8443
    +server-url = http://openbis-test.ethz.ch:8080
    
    +use-ssl = false
    
    +download-url = https://openbis-test.ethz.ch:443 

Upgrade server to newer version

Standard upgrade procedure

Prerequisites:

  • javapsqlpg_dump are present and accessible from the shell

Upgrade steps:

  • ssh to the server
  • copy openBIS installation tarball (a.k.a. "Installation and Upgrade Wizard") to <OPENBIS_HOME>/servers directory. You can download the latest version from here.
  • read instruction about necessary configuration changes here.
  • run <OPENBIS_HOME>/bin/upgrade.sh

Sometimes changes of the configuration are needed when a server is upgraded. You can do it after the upgrade. If you want to do this before, you can copy all the configuration of the servers to a temporary directory, modify it there and "apply" it to the upgraded server using backup-config.sh <dir> and restore-config-from-backup.sh <dir> scripts.

Upgrade installations with versions older than S118

The described upgrade steps are only valid for HCS installations prior S118 or for generic openBIS installations which have been created by the installation tarball. Generic openBIS installation assembled from two independent server archives (AS and DSS) are not be supported.

Prerequisites:

  • javapsqlpg_dump are present and accessible from the shell

Upgrade steps:

  • ssh to the server
  • copy openBIS installation tarball (to temporary directory (e.g. /tmp). You can download the latest version from here.
  • read instruction about necessary configuration changes here.
  • extract the installation tarball

    > tar xzvf openBIS-installation-XXX.tar.gz
    ....
    
  • edit the file console.properties and set the property INSTALL_PATHto <OPENBIS_HOME> e.g.

    #.... 
    INSTALL_PATH=/home/openbis/
    
  • execute ./run-console.sh. The installation procedure will then upgrade the openBIS installation at INSTALL_PATH.

Changing server configuration

The most important configuration files of openBIS AS and DSS can be found in openBIS-server/jetty/etc and datastore_server/etc folders.

Sometimes when many configuration files should be changed it can be convenient to copy all configuration files to one directory, modify them there and apply the changes. It can be done in the following way:

  • Execute the following command to copy all configuration files to a folder of your choice

    $ ./bin/backup-config.sh <directory-path>
    
  • Modify the files in <directory-path>
  • Apply the configuration changes by executing

    $ ./bin/restore-config-from-backup.sh <director-path>
    

Configure JAVA options

The files openbis.conf and datastore_server.conf contain the JVM options for the respective server. For recommendation on how to configure them see 
Recommended Java and Postgres memory settings.


By default, the AS and DSS configurations assume a 64bit JDK. If you are running 32bit JDK you must remove the -d64 parameter from openbis.conf and datastore_server.conf.

Configure authentication service

openBIS Application Server currently supports several authentication systems

  • file-authentication-service: a self-contained system based on a UNIX-like passwd file.
  • dummy-authentication-service: a "dummy" that allows everyone access to the system.
  • LDAP
  • Crowd system (see http://www.atlassian.com/software/crowd).

The latter two options will not be discussed in this document. For more information regarding LDAP or Crown integration check the Installation and Administrator Guide of the openBIS Server.

Dummy authentication service

If you only wish to play with openBIS and do not want to enforce security restrictions on the visiting users, you might want to choose the dummy-authentication-service.

To configure the dummy authentication scheme, edit <OPENBIS_HOME>/server/openBIS-server/jetty/etc/service.properties and set the value of the property authentication-service to dummy-authentication-service.

service.properties
authentication-service = dummy-authentication-service

File authentication service

The default installation procedure preconfigures openBIS with file-authentication-service. It also creates the user acounts for admin and etlserver. If you wish to add more user accounts, you can do so with the help of the command line tool <OPENBIS_HOME>/servers/openBIS-server/jetty/bin/passwd.sh by executing the following :

$ passwd.sh add [-f <first name>] [-l <last name>] [-e <email>] admin
Enter new password: *****

All passwords for file-authentication-service are stored in the file <OPENBIS_HOME>/servers/openBIS-server/jetty/etc/passwd.

Configure database options

By default openBIS assumes that the database in accessible on localhost as a 'postgres' user with an empty password. To change that one has to edit service.properties and configure the usernames and passwords for an PostgreSQL superuser (in sample below postgres) and the user, which will take ownership of the openBIS database instance. Some of you have created this low-privileged user in #Create a database user for openBIS (optional).

It is permissible, albeit not advisable to configure the same user (e.g. postgres) for database.owner and database.admin, providing the account is a PostgreSQL superuser.

service.properties
# Part of the name of the database. The full name of the database will be openbis_<kind>.
database.kind=demo

# ID of the user owning the data. In case of an empty string it is the same user who 
# started up openBIS Application Server.
database.owner = openbisdbuser
database.owner-password = *******

# ID of the user on database server with admin rights, like creation of tables. 
# Should be an empty string if default admin user should be used. In case of PostgreSQL 
# the default admin user is assumed to be postgres
database.admin-user = postgres
database.admin-password = *******

Additionally, ensure the configured database.kind property corresponds to the variable OPENBIS_DB declared in the file <OPENBIS_HOME>/bin/env. In our example database.kind is configured as "demo", so the value of OPENBIS_DB should be "openbis_demo".

Configure imaging database (HCS installations only)

For HCS installations you have to make a similar configuration for an additional database (imaging) in the DSS properties in servers/core-plugins/screening/NUM/dss/data-sources/imaging-db/plugin.properties file.

Please note that NUM is the highest number in the screening plugin. Currently this is 4.


plugin.properties
 
version-holder-class = ch.systemsx.cisd.openbis.dss.etl.ImagingDatabaseVersionHolder
databaseEngineCode = postgresql
basicDatabaseName = imaging
urlHostPart = ${imaging-database.url-host-part:localhost}
databaseKind = ${imaging-database.kind:prod}
scriptFolder = ${screening-sql-root-folder:}sql/imaging
owner = ${imaging-database.owner:}
password = ${imaging-database.password:}
 

This defines the username/passwords of an administrator user to perform migrations for the DSS database and an "owner" i.e. normal account to read/write data. In the example above the database will be called imaging_productive. Make sure this name corresponds to the IMAGING_DB variable in <OPENBIS_HOME>/bin/env.

You can create the "owner" database user as described in #Create a database user for openBIS (optional)

Additionally openBIS Application Server configuration has to be changed (usually 
in servers/core-plugins/screening/NUM/as/dss-data-sources/DSS1/plugin.properties file)


Please note that NUM corresponds to the highest number in the screening plugin.

database-driver = org.postgresql.Driver
database-url = jdbc:postgresql://${imaging-database.url-host-part:localhost}/imaging_${imaging-database.kind:prod}
database-username = ${imaging-database.owner:}
database-password = ${imaging-database.password:}
 

Appendix

List of administration scripts


Startup/shutdown scripts

Description

alldown.sh

stops AS and DSS

allup.sh

starts AS and DSS

bisup.sh

starts AS

bisdown.sh

stops AS

bislog.sh

shows AS log file

dssup.sh

starts DSS

dssdown.sh

stops DSS

dsslog.sh

shows DSS log file

Utility scripts

Description

backup-config.sh

Stores all configuration files in a specified directory

restore-config-from-backup.sh

Overrides servers' configurations with the configuration from a specified directory (which can be created using backup-config.sh and modified afterwards).

list-postgres-queries.sh

shows all current postgres queries (useful for debuging)

reset-display-settings.sh

resets all users display settings (edit the TEMPLATE_USER variable first)

Installation scripts

Description

env

Contains instance specific settings. This file is not updated from SVN.

upgrade.sh

upgrades openBIS to the new version

List of important configuration files


Configuration file

Description

datastore_server.conf

Contains properties configuring the DSS process launch (e.g. JAVA options, number of log files).

dss-log.xml

DSS log configuration.

dss-service.properties

DSS application properties. This is the most important DSS configuration file.

jetty.xml

Jetty server configuration (see http://docs.codehaus.org/display/JETTY/Syntax+Reference)

jetty.properties

Supplementary Jetty properties

log.xml

AS log configuration.

openbis.conf

Contains properties configuring the AS process launch (e.g. JAVA options, number of log files).

service.properties

AS application properties. This is the most important AS configuration file.

web-client.properties

openBIS web UI configuration properties.

welcomePageSimple.html

Altering file will change the login page of the openBIS web interface. Administrators can use this to add banners (e.g. with important announcements) visible to all users loggin in openBIS.