General

What is the difference between attachments and datasets?

Attachments offer much simpler functionality than datasets, but in some cases can be easier to use and upload.
Attachments should be used only to store small amounts of unstructured data (technical details: they are stored directly in the database in contrast to datasets, which are stored in the file system).
Datasets are "first class objects" in openBIS and are much more sophisticated than attachments - they can be annotated, archived, deleted in batches, visualized in special ways etc.
A nice feature of attachments is that they offer the concept of "versions" - an older version of attachment can be retrieved.
Only attachments can be linked to projects directly.

Attachments are not available in openBIS ELN-LIMS.

What is the difference between Master Data and Meta Data?

All experiment types, sample types, vocabularies, property types and their assignments and all scripts are master data. This needs to be defined by an Instance admin. 

All information entered in the system by users is metadata.  

It is easiest to describe with examples:
Let's assume that I have a sample type called 'Flowcell'. This is master data. If I enter an existing flowcell called '121127_SN792_0186_AD1F9VACXX' to openBIS I have added Meta Data.

Where can I get news on openBIS?

If you want to get regular information and news on openBIS, e.g. about new development, you should subscribe to the openBIS mailing list (mailing list archive). The list is in general "low-volume".

Administration

How much time do I need to plan for introducing openBIS in my lab?

This depends on the complexity of your lab and on your prior experience with databases and information systems. openBIS is a full-blown information system and will require you to learn some concepts before it becomes useful to you. If you have prior experience with databases and installing software on Unix systems, you should plan about a week to setup the system and understand the most important concepts. Then you can start integrating your lab's data sources. We recommend you start with a simple data source to get used to writing dropbox scripts. Your first dropbox script will likely take you something like two to three days to write. If you don't have prior experience with databases and basic Unix skills, you will have to learn basics about these topics on the way. Unfortunately we can't give you any estimate for that.

What hardware do I need to run openBIS?

This depends on the number of users and data sources you have and the size of your data. A regular openBIS server to be used as ELN in a regular experimental lab will usually have specs like 2 CPU cores, 10GB RAM, 100GB local harddisk and access to one or more NAS servers, whose size depends on the data produced by the lab. For high data volumes and a larger user community, mid-size servers may be required. In this range we cannot give general recommendations as the needs are too diverse. To give you an idea, we list the specs of a larger high-thoughput lab: 12 CPU cores Intel Xeon X5680 3.33GHz, 148GB RAM, 500GB SSD, 4TB RAID6 array (DAS), dynamically growing share on an EMC NAS system. Many admins prefer virtual machines, as theirs specs can be upgraded easily when needed.

Independent of the size of your server, do not forget that you need a data backup that covers the database and the data store!

What software prerequisites do I need to have installed for openBIS?

We recommend you use Linux to run an openBIS server. There is not much experience with running openBIS on Windows, even though it should work in principle. Note: For testing we use Redhat Enterprise Linux 7.

All you need to have installed other than the operating system itself is:

  • Oracle Java 8 (recommended)
  • PostgreSQL 9.6, 10 (recommended)

Handling SSL certificates

openBIS is shipped with a self-signed SSL certificate which is not accepted by web browsers without an explicit confirmation.
It causes that non advanced users can have problems with using the system.

If you access openBIS through the API you have to add "-Djavax.net.ssl.trustStore=/path_to_openbis/servers/datastore_server/etc/openBIS.keystore" to the JVM to accept openBIS self-signed SSL certificate.

Switching off SSL in openBIS versions newer than sprint S207 or 16.05.0

If you are running the instance in a test mode or do not intend to arrange a valid certificate, you can switch off SSL in a following way:

  • Change the AS jetty configuration:
    1. rename the https.ini and ssl.ini files in servers/openBIS-server/jetty/start.d/ to https_orig.ini and ssl_orig.ini
    2. create a new file called http.ini with this content

# ---------------------------------------
# Module: http
--module=http
### HTTP Connector Configuration
## HTTP port to listen on
jetty.port=8080
## HTTP idle timeout in milliseconds
http.timeout=30000
## HTTP Socket.soLingerTime in seconds. (-1 to disable)
# http.soLingerTime=-1
## Parameters to control the number and priority of acceptors and selectors
# http.selectors=1
# http.acceptors=1
# http.selectorPriorityDelta=0
# http.acceptorPriorityDelta=0


change DSS configuration file in servers/datastore_server/etc/service.properties:

  • in host-address replace https with http
  • set use-ssl to false
  • change the port in server-url to 8080
    Your file will look like this:

# host name of the machine on which the datastore server is running
host-address = http://my-openbis-domain.com
...

# Port
port = 8081
use-ssl = false
...
# The URL of the openBIS server
server-url = ${host-address}:8080
  • restart AS and DSS (e.g. by calling alldown.sh and allup.sh)

Switching off SSL in openBIS versions older than sprint S207 or 16.05.0

If you are running the instance in a test mode or do not intend to arrange a valid certificate, you can switch off SSL in a following way:

  • change AS jetty configuration file servers/openBIS-server/jetty/etc/jetty.xmlbe replacing

              <New class="org.eclipse.jetty.server.ssl.SslSelectChannelConnector">
    	             <Set name="Port">8443</Set>
    	                 <Set name="maxIdleTime">30000</Set>
                       <Set name="Acceptors">2</Set>
                       <Set name="AcceptQueueSize">100</Set>
    	                 <Set name="Keystore">
    	                     <SystemProperty name="jetty.home" default="." />/etc/openBIS.keystore
    	                 </Set>
                      <Set name="Password">changeit</Set>
                      <Set name="KeyPassword">changeit</Set>
              </New>
    

    with

              <New class="org.eclipse.jetty.server.nio.SelectChannelConnector">
                 <Set name="Port">8080</Set>
                 <Set name="maxIdleTime">30000</Set>
                 <Set name="Acceptors">2</Set>
                 <Set name="AcceptQueueSize">100</Set>
              </New>
    
  • change DSS configuration file in servers/datastore_server/etc/service.properties:
    • in host-address replace https with http
    • set use-ssl to false
    • change the port in server-url to 8080
      Your file will look like this:

      # host name of the machine on which the datastore server is running
      host-address = http://my-openbis-domain.com
      ...
      use-ssl = false
      ...
      # The URL of the openBIS server
      server-url = ${host-address}:8080
      
  • restart AS and DSS (e.g. by calling alldown.sh and allup.sh)

Note that now openBIS is running on port 8080 and because of that the URL with which you can access it in the web browser has changed.

Be careful, if you do not use SSL users passwords are transmitted as plain text over the wire.


Postgres performance 

On heavily used openBIS instances the performance of the Postgres DB can decrease. Even a full autovacuum does not help. Therefore we currently recommend to do a full dump/load cycle of the DB to get back the full performance.


Running openBIS locally for demonstration purposes or testing

If you want to run  openBIS locally use the general address for your computer, which is localhost or 127.0.0.1 as a configuration in the service.properties files of openBIS. This might prevent problems when the openBIS Application Server needs to communicate with the openBIS Datastore Server. Additionally you might run into problems of Cross-Site-Scripting Errors. If you still have problems also check the /etc/hosts file of your computer to spot any misconfiguration.

Tips and Tricks

How do I find out the version of openBIS in the Web Browser?

Login to openBIS. On the right hand-side, click on your username, and then click on the "About" button: this will open up a pop-up window with the openBIS version.

How do I find out the version of openBIS on the shell?

On version S141 or later, call:

$ <OPENBIS_HOME>/jetty/bin/version.sh

In order versions, have a look at <OPENBIS_HOME>/jetty/webapps/openbis/WEB-INF/classes/BUILD*:
General openBIS:

$ cat <OPENBIS_HOME>/jetty/webapps/openbis/WEB-INF/classes/BUILD-openbis.INFO

Screening openBIS:

$ cat <OPENBIS_HOME>/jetty/webapps/openbis/WEB-INF/classes/BUILD-openbis-screening.INFO

How to force openBIS to re-create its search index?

In rare cases it can occur that the full-text search index gets out-of-sync with the database. In this case the re-creation of the fulltext search indices can be forced by:

$ rm openBIS-server/jetty/indices/.MARKER_full_index
$ openBIS-server/jetty/bin/shutdown.sh
$ openBIS-server/jetty/bin/startup.sh

You can check in the file openBIS-server/jetty/logs/openbis_log.txt whether the indexing is successful: look for lines containing OPERATION.DefaultFullTextIndexer, and in particular for lines like:

OPERATION.DefaultFullTextIndexer - 'SamplePE' index complete. <n> entities have been indexed.
OPERATION.DefaultFullTextIndexer - 'MaterialPE' index complete. <m> entities have been indexed.
OPERATION.DefaultFullTextIndexer - 'ExperimentPE' index complete. <x> entities have been indexed.
OPERATION.DefaultFullTextIndexer - 'DataPE' index complete. <y> entities have been indexed.

After you've got the last of the four lines, you're set and your full-text search index is up-to-date again.

Apache 'Proxy Error' pops up

Currently we have not a full solution. Only an Apache setting which lowers the number of those errors:

Add to /etc/httpd/conf/httpd.conf:

Timeout 5400
ProxyTimeout 5400

And restart Apache: /sbin/service httpd restart

The problem is described here as well: http://stackoverflow.com/questions/169453/bad-gateway-502-error-with-apache-mod-proxy-and-tomcat


Using iptables instead of Apache

Instead of using Apache to serve openBIS on port 443 you can also use iptables for this. This might solve several issues:

# > cat /etc/sysconfig/iptables
# Firewall configuration written by system-config-securitylevel
# Manual customization of this file is not recommended.
*nat
#
#       Redirect ports 443 to 8443
#
-A PREROUTING -p tcp --dport 443 -j DNAT --to :8443
COMMIT
 
# > /etc/init.d/iptables restart
 
Check if rules are in place:
# > iptables -t nat -L
 
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination
DNAT       tcp  --  anywhere             anywhere            tcp dpt:https to::8443
Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination
Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination


Limit of open files

When putting a lot of files in a drop box you might run into the problem of  'too many open files error'. Please consider changing the ulimit value (for RHEL6 set nofile in /etc/security/limits.conf ) to a higher value.

org.springframework.remoting.RemoteAccessException: Could not access HTTP invoker remote service at [https://<Server-IP>:8444/datastore_server/service-conversation-server-manager]; nested exception is java.net.SocketException: Too many open files

The default value of RHEL, which is 1024, is known to be sufficient for most usage scenarios.


Relation between the Sharding Structure of the DSS store and the Data Set IDs:

Get the shard folder structure from a given Data Set ID:

# Mac: md5 -s <Data Set ID>
> md5 -s 20100127100637344-1114
MD5 ("20100127100637344-1114") = acf62afe5320b66b91ccf06fc0bfe0ca

# Linux: echo -n <Data Set ID> | md5sum
# you need the -n flag to avoid the \n char to be appended by default. Read more here:
# http://labs.sasslantis.ee/2011/04/md5sum-vs-md5-different-algorithms/

> echo -n 20100127100637344-1114 | md5sum
acf62afe5320b66b91ccf06fc0bfe0ca  -

Location in the DSS:
1/CD847E34-26E8-4396-965F-8EA210A203B4/ac/f6/2a/20100127100637344-1114


The corresponding calculator which uses the the MD5sum to generate the folder structure:

ch.systemsx.cisd.openbis.dss.generic.shared.utils.DatasetLocationUtil
    // In order not to overwhelm the file system implementation, we won't use a completely flat
    // hierarchy, but instead a structure called 'data sharding'. 'Data sharding' ensures that there
    // are no directories with an extremely large number of data sets.
    private static File createShardingDir(File parentDir, String dataSetCode)
    {
        String checksum = MD5ChecksumCalculator.calculate(dataSetCode);
        final File dirLevel1 = new File(parentDir, checksum.substring(0, 2));
        final File dirLevel2 = new File(dirLevel1, checksum.substring(2, 4));
        final File dirLevel3 = new File(dirLevel2, checksum.substring(4, 6));
        return dirLevel3;
    }
  • No labels