This document guides you through the basics of installing and configuring openBIS. Its purpose it to allow newcomers to install openBIS and play with it, so exhaustive details are left out on purpose. The complete openBIS administration manual is maintained here Installation and Administrator Guide of the openBIS Server.
Prerequisites
The successful installation of openBIS requires a machine with:
- Linux or Mac OS X operating system
- PostgreSQL 11. You are required to have database superuser privileges (by default, the
postgres
user has those). - Java 11.
Please, make sure all the prerequisites are met before continuing to the next chapters of this document.
Recommended CPU and memory settings
See Recommended CPU and memory settings for openBIS 20.10
Deployment overview
The basic openBIS deployment consists of two servers, the Application Server (AS) and the Data Store Server (DSS). Generally speaking, the AS manages the metadata and links to the data while the DSS manages the data itself. The AS employs a PostgreSQL server to persist users, authorization information, various entities and their metadata, as well as index information about all datasets. The DSS operates on a managed part of the file system (data store), where the data is kept. One AS can manage multiple DSS instances, which can make sense e.g. if the data are acquired in multiple, distributed laboratories each having a local DSS.
In this document we will describe an installation procedure with a single DSS, where all three servers (AS, DSS and PostgreSQL) run on the same computer. In general, each of them could be deployed on a dedicated machine, which would improve the overall performance of the system.
Note, that regardless of the particular deployment scenario, the servers must be started in the following order
- PostgreSQL
- openBIS Application Server
- Data Store Server
This is required since the AS needs a running PostgreSQL server to start and the DSS needs an AS to boot up. These dependencies are depicted on the deployment overview diagram with arrows pointing from the dependent towards the dependee.
To avoid inconsistent states of the system, shutdowns must be executed backwards - starting from DSS and proceeding with AS and PostgreSQL respectively.
Installation
It is considered a good practice to install openBIS under a dedicated operating system user, but it is not mandatory to do so. Throughout the document we assume that openBIS will be installed and maintained by a user named openbis
. The default installation path for the system will be the home folder of openbis
. We also refer to the installation path as OPENBIS_HOME
.
PostgreSQL server configuration
Create a database user with the same name as the OS user (in our case openbis
)
$ sudo -u postgres createuser openbis Shall the new role be a superuser? (y/n) n Shall the new role be allowed to create databases? (y/n) n Shall the new role be allowed to create more new roles? (y/n) n
The openBIS setup assumes that PostgreSQL is configured with the trust setting on localhost, which means that a user connecting from 127.0.0.1
can connect without providing a password. These settings are changed in the file pg_hba.conf
, see http://www.postgresql.org/docs/9.3/static/auth-pg-hba-conf.html. If these settings are not correct, you will get an error of the form "Database role 'openbis' couldn't be created."
If the PostgreSQL server is configured correctly, then the following sequence of commands must return without errors.
> su - openbis > psql -U postgres -l ... > psql -l ...
Postgres on a remote server
When the database is on a remote server, the pg_hba.conf and postgresql.conf files need to be configured as follows:
# TYPE DATABASE USER ADDRESS METHOD # application access privilege host database_name all 0.0.0.0/0 md5 / trust host postgres all 0.0.0.0/0 md5 / trust
So, the databse postgres need to be added to the file. Alternatively, instead of specifying each database and the postgres database, the database can be set to "all" as shown below:
# TYPE DATABASE USER ADDRESS METHOD # application access privilege host all all 0.0.0.0/0 md5 / trust
listen_addresses= '*' (or alternatively one or more IP addresses can be listed)
Also have look here how to configure openBIS to connect to the remote Postgres DB.
Manual database creation
If openBIS cannot be allowed to create its own database (because of missing privileges, for example), you can use the script bin/init_database.sh (in your openBIS installation directory) that will create SQL statements to initialize the database. These SQL statememts can be given to be executed by a database administrator. Example:
$ bin/init_database.sh openbis_dev openbis -- Execute the following SQL script as a database superuser create user openbis; create database openbis_dev with owner openbis encoding 'UTF8' template=template0 tablespace=pg_default; alter database openbis_dev set default_with_oids = off; alter database openbis_dev set join_collapse_limit = '32'; alter database openbis_dev set from_collapse_limit = '32'; \connect openbis_dev begin; create table database_version_logs(db_version varchar(4) not null, module_name varchar(250), run_status varchar(10), run_status_timestamp timestamp, module_code bytea, run_exception bytea); insert into database_version_logs(db_version, run_status, run_status_timestamp) values ('0', 'SUCCESS', now()); alter database openbis_dev owner to openbis; alter table database_version_logs owner to openbis; commit;
1st argument to the script is the name of the database to be created.
2nd argument to the script is the name of the user openBIS will use to access the database we created.
Note: both the database name and user name need to be reflected in service.properties configuration files.
Download binaries
- Go to openBIS Download Page.
- Download the latest release of openBIS installer. It should be a TAR.GZ archive and its name should look like
openBIS-installation-<version>-r<revision>.tar.gz
, where<version>
is the release number (e.g. S117).
$ ls -a openBIS-installation-XXX.tar.gz
Run the installation
NOTE: openBIS will ask you to provide the install directory. For a new installation this directory is created by the openBIS installer, so it should not already exist. If the directory already exists the installation will fail.
Extract the installation tarball
> tar xzvf openBIS-installation-S117-r23517.tar.gz x openBIS-installation-S117-r23517/console.properties x openBIS-installation-S117-r23517/jul.config x openBIS-installation-S117-r23517/extract.sh x openBIS-installation-S117-r23517/run-console.sh x openBIS-installation-S117-r23517/run-ui.sh x openBIS-installation-S117-r23517/openBIS-installer.jar
Headless installation
To install openBIS on a headless system you need to
- Edit the file
console.properties
- Execute the script
run-console.sh
> cd openBIS-installation-S117-r23517 > ./run-console.sh
Congratulations ! If you have reached this far your openBIS system is installed and is ready to be started. Please do so by following the instructions in the next section.
Starting/Stopping
The starting scripts of openBIS assume that java
is available on the system path, so you have to add it if this is not true.
Launch the openBIS AS and DSS servers by executing <OPENBIS_HOME>/bin/allup.sh
.
$ ./bin/allup.sh
To verify AS and DSS function and there are no configurational issues, please check the log files of the two applications using the bislog.sh
and dsslog.sh
scripts. Additionally, the openBIS application should be accessible at https://localhost:8443/openbis
and you should be able to login as the admin
user with the password specified in the installation process.
You can shutdown both the servers by executing the <OPENBIS_HOME>/bin/alldown.sh
script
$ ./bin/alldown.sh
Configure openBIS to run on one standard port 443
We would like all users who work in the network to use the standard https port (443) to access openbis and the DSS (datastore server).
We start openbis on port 8080 and DSS on port 8081. Those ports are not accessible from outside. Then we redirect all the requests to 443 port from outside to a right internal port depending on the URL suffix. We do this by switching on Apache forwarding (we use mod_proxy).
See the details below.
Apache configuration
- we configure Apache instance (inside
/etc/httpd/conf.d/openbis.conf
file on Linux, see attached openbis.conf as an example) to:
<VirtualHost *:443> ErrorLog /var/log/apache2/ssl_error_log TransferLog /var/log/apache2/ssl_access_log LogLevel info ServerName localhost SSLEngine on SSLProtocol all -SSLv2 SSLCipherSuite ALL:!ADH:!EXPORT:!SSLv2:RC4+RSA:+HIGH:+MEDIUM:+LOW SSLCertificateFile /etc/pki/tls/apache.crt SSLCertificateKeyFile /etc/pki/tls/apache.key SSLProxyEngine on ProxyRequests Off ProxyPreserveHost On RewriteEngine On AllowEncodedSlashes On # # Rewrite url if there is a missing / at the end # # Remove the following line if using cifex! RewriteRule ^/$ /openbis/ [R,L] RewriteRule ^/openbis$ /openbis/ [R,L] RewriteRule ^/datastore_server$ /datastore_server/ [R,L] #Replace localhost with your host name ProxyPass /openbis/ http://localhost:8080/openbis/ timeout=600 Keepalive=Off ProxyPassReverse /openbis/ http://localhost:8080/openbis/ ProxyPass /datastore_server/ http://localhost:8081/datastore_server/ ProxyPassReverse /datastore_server/ http://localhost:8081/datastore_server/ # Used by Screening instances ProxyPass /datastore_server_screening http://localhost:8081/datastore_server_screening ProxyPassReverse /datastore_server_screening http://localhost:8081/datastore_server_screening ProxyPass /rmi-datastore-server-screening-api-v1 http://localhost:8081/rmi-datastore-server-screening-api-v1 ProxyPassReverse /rmi-datastore-server-screening-api-v1 http://localhost:8081/rmi-datastore-server-screening-api-v1 </VirtualHost>
- we configure openbis and DSS servers not to use SSL (because Apache uses it already)
- install SSL module (mod_ssl) for Apache if it is not there:
yum install mod_ssl
. - after changing the configuration you have to restart Apache. On Linux you call:
/etc/init.d/httpd restart
make sure that Apache will be restarted automatically
[root@~]# chkconfig --list | grep httpd httpd 0:off 1:off 2:on 3:on 4:on 5:on 6:off
openBIS configuration
Create a file called http.ini
in /openBIS-server/jetty/start.d/
:
# Module: http --module=http ### HTTP Connector Configuration ## HTTP port to listen on jetty.port=8080 ## HTTP idle timeout in milliseconds http.timeout=30000 ## HTTP Socket.soLingerTime in seconds. (-1 to disable) # http.soLingerTime=-1 ## Parameters to control the number and priority of acceptors and selectors # http.selectors=1 # http.acceptors=1 # http.selectorPriorityDelta=0 # http.acceptorPriorityDelta=0
Rename the files https.ini
and ssl.ini
. These files are then then no longer used by jetty to serve at port 8443.
Settings for the Data Store Server (DSS)
These changes are needed for all Jetty versions which are used in openBIS
- do the same in DSS service.properties (sprint/datastore_server/etc) for 'server-url' and 'download-url' properties, additionally add
use-ssl = false
Changes to DSS service.properties file:
--- service.properties.ssl Tue Mar 9 16:51:40 2010 +++ service.properties Sat Mar 13 07:54:28 2010 @@ -8,3 +8,3 @@ # Port -port = 8444 +port = 8081 @@ -43,4 +43,5 @@ # The URL of the openBIS server -server-url = https://openbis-test.ethz.ch:8443 +server-url = http://openbis-test.ethz.ch:8080 +use-ssl = false +download-url = https://openbis-test.ethz.ch:443
Upgrade server to newer version
Standard upgrade procedure
Prerequisites:
java
,psql
,pg_dump
are present and accessible from the shell
Upgrade steps:
- ssh to the server
- copy openBIS installation tarball (a.k.a. "Installation and Upgrade Wizard") to
<OPENBIS_HOME>/servers
directory. You can download the latest version from here. - read instruction about necessary configuration changes here.
- run
<OPENBIS_HOME>/bin/upgrade.sh
Sometimes changes of the configuration are needed when a server is upgraded. You can do it after the upgrade. If you want to do this before, you can copy all the configuration of the servers to a temporary directory, modify it there and "apply" it to the upgraded server using backup-config.sh <dir>
and restore-config-from-backup.sh <dir>
scripts.
Upgrade installations with versions older than S118
The described upgrade steps are only valid for HCS installations prior S118 or for generic openBIS installations which have been created by the installation tarball. Generic openBIS installation assembled from two independent server archives (AS and DSS) are not be supported.
Prerequisites:
java
,psql
,pg_dump
are present and accessible from the shell
Upgrade steps:
- ssh to the server
- copy openBIS installation tarball (to temporary directory (e.g.
/tmp
). You can download the latest version from here. - read instruction about necessary configuration changes here.
extract the installation tarball
> tar xzvf openBIS-installation-XXX.tar.gz ....
edit the file
console.properties
and set the propertyINSTALL_PATH
to <OPENBIS_HOME> e.g.#.... INSTALL_PATH=/home/openbis/
- execute
./run-console.sh
. The installation procedure will then upgrade the openBIS installation atINSTALL_PATH
.
Changing server configuration
The most important configuration files of openBIS AS and DSS can be found in openBIS-server/jetty/etc
and datastore_server/etc
folders.
Sometimes when many configuration files should be changed it can be convenient to copy all configuration files to one directory, modify them there and apply the changes. It can be done in the following way:
Execute the following command to copy all configuration files to a folder of your choice
$ ./bin/backup-config.sh <directory-path>
- Modify the files in <directory-path>
Apply the configuration changes by executing
$ ./bin/restore-config-from-backup.sh <director-path>
Configure JAVA options
The files openbis.conf
and datastore_server.conf
contain the JVM options for the respective server. For recommendation on how to configure them see
Recommended Java and Postgres memory settings.
By default, the AS and DSS configurations assume a 64bit JDK. If you are running 32bit JDK you must remove the -d64
parameter from openbis.conf
and datastore_server.conf
.
Configure authentication service
openBIS Application Server currently supports several authentication systems
file-authentication-service
: a self-contained system based on a UNIX-like passwd file.dummy-authentication-service
: a "dummy" that allows everyone access to the system.- LDAP
- Crowd system (see http://www.atlassian.com/software/crowd).
The latter two options will not be discussed in this document. For more information regarding LDAP or Crown integration check the Installation and Administrator Guide of the openBIS Server.
Dummy authentication service
If you only wish to play with openBIS and do not want to enforce security restrictions on the visiting users, you might want to choose the dummy-authentication-service
.
To configure the dummy authentication scheme, edit <OPENBIS_HOME>/server/openBIS-server/jetty/etc/service.properties
and set the value of the property authentication-service
to dummy-authentication-service
.
authentication-service = dummy-authentication-service
File authentication service
The default installation procedure preconfigures openBIS with file-authentication-service
. It also creates the user acounts for admin
and etlserver
. If you wish to add more user accounts, you can do so with the help of the command line tool <OPENBIS_HOME>/servers/openBIS-server/jetty/bin/passwd.sh
by executing the following :
$ passwd.sh add [-f <first name>] [-l <last name>] [-e <email>] admin Enter new password: *****
All passwords for file-authentication-service
are stored in the file <OPENBIS_HOME>/servers/openBIS-server/jetty/etc/passwd
.
Configure database options
By default openBIS assumes that the database in accessible on localhost as a 'postgres' user with an empty password. To change that one has to edit service.properties
and configure the usernames and passwords for an PostgreSQL superuser (in sample below postgres
) and the user, which will take ownership of the openBIS database instance. Some of you have created this low-privileged user in #Create a database user for openBIS (optional).
It is permissible, albeit not advisable to configure the same user (e.g. postgres
) for database.owner
and database.admin
, providing the account is a PostgreSQL superuser.
# Part of the name of the database. The full name of the database will be openbis_<kind>. database.kind=demo # ID of the user owning the data. In case of an empty string it is the same user who # started up openBIS Application Server. database.owner = openbisdbuser database.owner-password = ******* # ID of the user on database server with admin rights, like creation of tables. # Should be an empty string if default admin user should be used. In case of PostgreSQL # the default admin user is assumed to be postgres database.admin-user = postgres database.admin-password = *******
Additionally, ensure the configured database.kind
property corresponds to the variable OPENBIS_DB
declared in the file <OPENBIS_HOME>/bin/env
. In our example database.kind
is configured as "demo", so the value of OPENBIS_DB
should be "openbis_demo".
Configure imaging database (HCS installations only)
For HCS installations you have to make a similar configuration for an additional database (imaging) in the DSS properties in servers/core-plugins/screening/NUM/dss/data-sources/imaging-db/plugin.properties file.
Please note that NUM is the highest number in the screening plugin. Currently this is 4.
version-holder-class = ch.systemsx.cisd.openbis.dss.etl.ImagingDatabaseVersionHolder databaseEngineCode = postgresql basicDatabaseName = imaging urlHostPart = ${imaging-database.url-host-part:localhost} databaseKind = ${imaging-database.kind:prod} scriptFolder = ${screening-sql-root-folder:}sql/imaging owner = ${imaging-database.owner:} password = ${imaging-database.password:}
This defines the username/passwords of an administrator user to perform migrations for the DSS database and an "owner" i.e. normal account to read/write data. In the example above the database will be called imaging_productive
. Make sure this name corresponds to the IMAGING_DB
variable in <OPENBIS_HOME>/bin/env
.
You can create the "owner" database user as described in #Create a database user for openBIS (optional)
Additionally openBIS Application Server configuration has to be changed (usually
in servers/core-plugins/screening/NUM/as/dss-data-sources/DSS1/plugin.properties file)
Please note that NUM corresponds to the highest number in the screening plugin.
database-driver = org.postgresql.Driver database-url = jdbc:postgresql://${imaging-database.url-host-part:localhost}/imaging_${imaging-database.kind:prod} database-username = ${imaging-database.owner:} database-password = ${imaging-database.password:}
Appendix
List of administration scripts
Startup/shutdown scripts | Description |
---|---|
alldown.sh | stops AS and DSS |
allup.sh | starts AS and DSS |
bisup.sh | starts AS |
bisdown.sh | stops AS |
bislog.sh | shows AS log file |
dssup.sh | starts DSS |
dssdown.sh | stops DSS |
dsslog.sh | shows DSS log file |
Utility scripts | Description |
backup-config.sh | Stores all configuration files in a specified directory |
restore-config-from-backup.sh | Overrides servers' configurations with the configuration from a specified directory (which can be created using |
list-postgres-queries.sh | shows all current postgres queries (useful for debuging) |
reset-display-settings.sh | resets all users display settings (edit the |
Installation scripts | Description |
env | Contains instance specific settings. This file is not updated from SVN. |
upgrade.sh | upgrades openBIS to the new version |
List of important configuration files
Configuration file | Description |
---|---|
datastore_server.conf | Contains properties configuring the DSS process launch (e.g. JAVA options, number of log files). |
dss-log.xml | DSS log configuration. |
dss-service.properties | DSS application properties. This is the most important DSS configuration file. |
jetty.xml | Jetty server configuration (see http://docs.codehaus.org/display/JETTY/Syntax+Reference) |
jetty.properties | Supplementary Jetty properties |
log.xml | AS log configuration. |
openbis.conf | Contains properties configuring the AS process launch (e.g. JAVA options, number of log files). |
service.properties | AS application properties. This is the most important AS configuration file. |
web-client.properties | openBIS web UI configuration properties. |
welcomePageSimple.html | Altering file will change the login page of the openBIS web interface. Administrators can use this to add banners (e.g. with important announcements) visible to all users loggin in openBIS. |