Core Plugins
Motivation
The service.properties
file of openBIS Application Server (AS) and Data Store Server (DSS) can be quite big because of all the configuration data for maintenance tasks, drop-boxes, reporting and processing plugins, etc. Making this configuration more modular will improve the structure. It would also allow to have core plugins shipped with distribution and customized plugins separately. This makes maintenance of these plugins more independent. For example, a new maintenance task plugin can be added in an update without any need for an admin to put the configuration data manually into the service.properties
file.
Core Plugins Folder Structure
All plugins whether they are a part of the distribution or added and maintained are stored in the folder usually called core-plugins
. Standard (i.e. core) plugins are part of the distribution. During installation the folder core-plugins
is unpacked as a sibling folder of openBIS-server
and datastore_server
.
The folder structure is organized as follows:
- The file
core-plugins.properties
containing the following properties:enabled-modules
: comma-separated list of regular expressions for all enabled modules.disabled-core-plugins
: comma-separated list of disabled plugins. All plugins are disabled for which the beginning of full plugin ID matches one of the terms of this list. To disable initialization of master data of a module - disable it's core plugin "initialize-master-data"
- The children of
core-plugins
are folders denoting modules like the standard technologies,proteomics
andscreening
. For customization, any module can be added. - Each module folder has children which are numbered folders. The number denotes the version of the plugins of that module. The version with the largest number will be used. Different modules can have different largest version numbers.
- Every version folder has the subfolder
as
and/ordss
which have subfolders for the various types of plugins. The types are different for AS and DSS:- AS:
maintenance-tasks
: Maintenance tasks triggered by some time schedule.
class
denotes fully-qualified class name of a class implementingch.systemsx.cisd.common.maintenance.IMaintenanceTask
.
For more details see Maintenance Tasks.dss-data-sources
: Definition of data sources with corresponding data source definitions for DSS. For more details see Installation and Administrator Guide of the openBIS Server.query-databases
: Databases for SQL queries. For more details see Custom Database Queries.custom-imports
: Custom file imports to DSS via Web interface. For more details see Custom Import.services
: Custom services. For more details see Custom Application Server Services.webapps
: HTML5 applications that use the openBIS API. For more details see openBIS webapps.miscellaneous
: Any additional properties.
DSS:
drop-boxes
: ETL server threads for registration of data sets.
reporting-plugins
: Reports visible in openBIS.
Propertyclass
denotes fully-qualified class name of a class implementingch.systemsx.cisd.openbis.dss.generic.server.plugins.tasks.IReportingPluginTask
.
For more details see Reporting Plugins.processing-plugins
: Processing tasks triggered by users.
class
denotes fully-qualified class name of a class implementingch.systemsx.cisd.openbis.dss.generic.server.plugins.tasks.IProcessingPluginTask
.
maintenance-tasks
: Maintenance tasks triggered by some time schedule.
class
denotes fully-qualified class name of a class implementingch.systemsx.cisd.common.maintenance.IMaintenanceTask
.
search-domain-services
: Services for variaous search domains (e.g. search on sequence databases using BLAST).
Propertyclass
denotes fully-qualified class name of a class implementingch.systemsx.cisd.openbis.dss.generic.shared.api.internal.v2.ISearchDomainService
.data-sources
: Internal or external database sources.services
: Services based on servlets.
Propertyclass
denotes fully-qualified class name of a class implementingjavax.servlet.Servlet
.imaging-overview-plugins
: Data set type specific provider of the overview image of a data set.
Propertyclass
denotes fully-qualified class name of a class implementingch.systemsx.cisd.openbis.dss.generic.server.IDatasetImageOverviewPlugin
.file-system-plugins
: Provider of a custom DSS file system (FTP/SFTP) view hierarchy.
Propertyclass
denotes fully-qualified class name of a class implementingch.systemsx.cisd.openbis.dss.generic.server.fs.IResolverPlugin
Property code denotes the name of the top-level directory under which the custom hierarchy will be visiblemiscellaneous
: Any additional properties.
- AS:
- Folders of each of these types can have an arbitrary number of subfolders. But if the type folder is present it should have at least one subfolder. Each defining one plugin. The name of these subfolders define the plugin ID. It has to be unique over all plugins independent of module and plugin type. It should not contain the characters space ' ', comma '
,
', and equal sign '=
'. - Each plugin folder should contain at least the file
plugin.properties
. There could be additional files (referred inplugin.properties
) but no subfolders.
Here is an example of a typical structure of a core plugins folder:
core-plugins core-plugins.properties proteomics 1 as initialize-master-data.py dss drop-boxes ms-injection plugin.properties maintenance-tasks data-set-clean-up plugin.properties screening 1 core-plugin.properties as initialize-master-data.py maintenance-tasks material-reporting mapping.txt plugin.properties custom-imports myCustomImport plugin.properties dss drop-boxes hcs-dropbox lib custom-lib.jar hcs-dropbox.py plugin.properties
You might noticed the file initialize-master-data.py
in AS core plugins sections in this example. It is a script to register master data in the openBIS core database. For more details see Installation and Administrator Guide of the openBIS Server.
Each plugin can refer to any number of files. These files are part of the plugin folder. In plugin.properties
they are referred relative to the plugin folder, that is by file name. Example:
incoming-dir = ${incoming-root-dir}/incoming-hcs incoming-data-completeness-condition = auto-detection top-level-data-set-handler = ch.systemsx.cisd.openbis.dss.etl.jython.JythonPlateDataSetHandler script-path = hcs-dropbox.py storage-processor = ch.systemsx.cisd.openbis.dss.etl.PlateStorageProcessor storage-processor.data-source = imaging-db storage-processor.define-channels-per-experiment = false
Merging Configuration Data
At start up of AS and DSS merges the content of service.properties
with the content of all plugin.properties
of the latest version per enabled module. Plugin properties can be deleted by adding <plugin ID>.<plugin property key> = __DELETED__
to service.properties. Example:
simple-dropbox.incoming-data-completeness-condition = __DELETED__
This leads to a deletion of the property incoming-data-completeness-condition
specified in plugins.properties
of the plugin simple-dropbox
.
Merging is done by injection the properties of plugin.properties
into service.properties
by adding the plugin ID as a prefix to the property key (not for miscellaneous).
For example, the property script-path
of plugin hcs-dropbox
becomes hcs-dropbox.script-path
. References to files inside the plugin are replaced by a path relative to the working directory. For the various plugin types (except miscellaneous
) the plugin ID is appended to the related property in service.properties
for this plugin type. For example, plugins of type drop-boxes
are added to the property inputs
.
Enabling Modules and Disabling Plugins
There are three methods to control which plugins are available and witch not:
- enabling by property
enabled-modules
incore-plugins.properties
: This enables all plugins of certain modules. - disabling by property
disabled-core-plugins
incore-plugins.properties
- disabling by marker file: Plugin developers should use this method when developing new plugins.
Enabling Modules
The property enabled-modules
in core-plugins.properties
is a comma-separated list of regular expressions denoting modules. All plugins in a module folder of core-plugins
folder are enabled if the module name matches one of these regular expressions. If this list is empty or the property hasn't been specified no core-plugin will be used. Note, that this property is manipulated by openBIS Installer for Standard Technologies. Example:
enabled-modules = screening, proteomics, dev-module-.*
Disabling Core Plugins by Property
The property disabled-core-plugins
in core-plugins.properties
allows to disable plugins selectively either by module name, module combined with plugin type or full plugin ID. Example:
disabled-core-plugins = screening, proteomics:reporting-plugins, proteomics:maintenance-tasks:data-set-clean-up
Disabling Core Plugins by Marker File
The empty marker file disabled
in a certain plugin folder disables the particular plugin.
Core Plugin Dependency
A core plugin can depend on another core plugin. The dependency is specified in <module>/<version>/core-plugin.properties
. It has a property named required-plugins
. Its value is a comma-separated list of core-plugins on which it depends. The dependency can be specified selectively either by module name, module combined with plugin type or full plugin ID. Example:
required-plugins = module-a, module-b:initialize-master-data, module-b:reporting-plugins, module-a:drop-boxes:generic
Rules for Plugin Writers
As a consequence of the way plugins are merged with service.properties
writers of plugins have to obey the following rules:
- Plugin IDs have to be unique among all plugins whether they are defined in
service.properties
or as core plugins. The only exceptions are plugins of typemiscellaneous
. - In
plugin.properties
other properties can be referred by the usual${<property key>
} notation. The referred property can be inservice.properties
or in anyplugin.properties
. - As convention use
${incoming-root-dir
} when defining the incoming folder for a drop box. - Refer files in
plugin.properties
only by names and add them as siblings ofplugin.properties
to the plugin folder. Note, that different plugins can refer files with the same name. There will be no ambiguity which file is meant. - In order to be completely independent from updates of the core plugins which are part of the distribution create your own module, like
my-plugins
, and put all your plugins there. Do not forget to add your module to the propertyenabled-modules
incore-plugins.properties
.
Using Java libraries in Core Plugins
OpenBIS allows you to include Java libraries in core plugin folders. The *.jar files have to be stored in "<code plugin folder>/lib" folder. For instance, in order to use "my-lib.jar" in "my-dropbox" a following file structure is needed:
my-technology 1 dss drop-boxes my-dropbox lib my-lib.jar dropbox.py plugin.properties
Having this structure, Java classes from "my-lib.jar" can be imported and used in "dropbox.py" script.
NOTICE: Currently this feature is only supported for DSS core plugins. Under the hood, a symbolic link to a jar file is created in "datastore_server/lib" folder during DSS startup.