JSON-RPC in a nutshell
JSON-RPC is a well documented standard for webservices. It is like XML-RPC but instead of XML, the data exchange format is JSON (Java Script Object Notation).
- all requests are HTTP POST requests
- all requests look like this:
{ "method": "methodName", "params": ["an", "array", "with", "some", "parameters"], "id": "my mandatory request reference id", "jsonrpc": "2.0" }
- without the id, the server gets notified but sends nothing back (only HTTP 200 OK but no data)
- all responses from the server return HTTP 200 OK - even when an error occured
- all responses return the same id as in the request
- because of this id, asynchronous communication is possible
- in case of a success, the response looks like this:
{ "result": { "message": "you sent some parameters" }, "id": "my mandatory request reference id", "jsonrpc": "2.0" }
- in case of an error, the response looks like this:
{ "error": { "code": 12345, "message": "you sent me some wrong parameters" }, "id": "my mandatory request reference id", "jsonrpc": "2.0" }
In our examples below we use "id" : "1"
but of course, you should use a better id (e.g. the permId or something else which is unique)
openBIS endpoints
We try to use openBIS v3 as often as possible. If no other information is given, all POST requests in our examples request the application server (AS) using the URL below. The typical port for the AS is 8443 (Change your hostname and your port number accordingly)
POST https://openbis-hostname:8443/openbis/openbis/rmi-application-server-v3.json
Not all methods have yet been implemented in the new v3 API. We sometimes need one of the following
POST https://openbis-hostname:8443/openbis/openbis/rmi-general-information-v1.json
The v1 JSCON API has many more endpoints which are described here
For requests regarding datasets on the datastore server (DSS), we have again different enpoints. The DSS usually runs on a different port, typically 8444. If we need datastore specific information (e.g. list of files) we need to use this endpoint:
POST https://openbis-dss-hostname:8444/datastore_server/rmi-dss-api-v1.json
You might have more than one DSS.
To play with the openBIS API (or any other webservice) I recommend using the Chrome plugin Postman. There is a (newer, shinier) Chrome App with the same name but I found its layout and readability not being as good.
Log in and out
Log into openBIS - get a token
Before you can request anythin from openBIS, you need to obtain a session token. Any subsequent request need this token.
POST https://openbis-hostname:8443/openbis/openbis/rmi-application-server-v3.json
{ "method": "login", "params": [ "my_username", "my_password" ], "id": "1", "jsonrpc": "2.0" }
You will get a response like this:
{ "result": "my_username-160601233044713x0F476AEA0FE875E4E87CB16C6D6902D6", "id": "1", "jsonrpc": "2.0" }
The string of the result is your session token. It contains your username, a dash '-' and an arbitary hexcode.
This token is always the first parameter value of any openBIS request.
Logout - destroy a session token
To end a session and destroy a session token, we use the logout
method:
POST https://openbis-hostname:8443/openbis/openbis/rmi-application-server-v3.json
{ "method": "logout", "params": [ "my_username-160601233044713x0F476AEA0FE875E4E87CB16C6D6902D6" ], "id": "1", "jsonrpc": "2.0" }
The response is as shown below. Because the result is null anyway, you could omit the id
in the request.
{ "result": null, "id": "1", "jsonrpc": "2.0" }
check if session token is valid
You can check whether your session is still valid by using the v1 API:
POST http://openbis-hostname:8443/openbis/openbis/rmi-general-information-v1.json
{ "method": "isSessionActive", "params": [ "my_username-160601233044713x0F476AEA0FE875E4E87CB16C6D6902D6" ], "id": "1", "jsonrpc": "2.0" }
The 'result' of the response should contain true
if your session is still valid, false
otherwise.
If your session token is invalid, any request will result in an error:
{ "error": { "code": 0, "message": "my_username-160601233044713x0F476AEA0FE875E4E87CB16C6D6902D6' is invalid: user is not logged in.", "data": { "@id": 1, "exceptionTypeName": "ch.systemsx.cisd.common.exceptions.InvalidSessionException", "message": "Session token 'my_username-160601233044713x0F476AEA0FE875E4E87CB16C6D6902D6' is invalid: user is not logged in." } }, "id": "1", "jsonrpc": "2.0" }
Fetching meta-information
There are a number of methods to get information about our openBIS server. All information can be fetched in a similar way, by just changing the method name:
kind of information | methodName |
---|---|
spaces | searchSpaces |
sample types | searchSampleTypes |
dataset types | searchDataSetTypes |
experiment types | searchExperimentTypes |
material types | searchMaterialTypes |
vocabulary | searchVocabularyTerms |
file types |
|
If we have no special search criteria (just list them all), the typical request then looks like this:
POST https://localhost:8443/openbis/openbis/rmi-application-server-v3.json
{ "method": "methodName", "params": [ "my_username-160601233044713x0F476AEA0FE875E4E87CB16C6D6902D6", { /* search criteria */ }, { /* fetch options */ } ], "id": "1", "jsonrpc": "2.0" }
working with Samples
The following methods requests data about a specific sample. Every sample is identified by an identifier and a permId.
get sample by identifier
To fetch a sample, we use the getSamples method and providing an identifier attribute together with a corresponding @type
POST https://openbis-hostname:8443/openbis/openbis/rmi-application-server-v3.json
{ "method": "getSamples", "params": [ "my_username-160601233044713x0F476AEA0FE875E4E87CB16C6D6902D6", [ { "identifier": "/TEST/TEST-SAMPLE-2", "@type": "as.dto.sample.id.SampleIdentifier" } ], { /* fetch options, see below */ } ], "id": "1", "jsonrpc": "2.0" }
get sample by permId
Fetching a sample by permId is done exactly the same way. Instead of an identifier we provide a permId attribute and a different @type:
POST https://openbis-hostname:8443/openbis/openbis/rmi-application-server-v3.json
{ "method": "getSamples", "params": [ "my_username-160601233044713x0F476AEA0FE875E4E87CB16C6D6902D6", [ { "permId": "20130415091923485-402", "@type": "as.dto.sample.id.SamplePermId" } ], { /* fetch options, see below */ } ], "id": "1", "jsonrpc": "2.0" }
create a sample
A sample needs a name (code), must be of a certain type (typeId), and must belong to a space (spaceID). It might contain specific properties as well as parents (parentIds), children (childIds) and many other connections.
POST https://openbis-hostname:8443/openbis/openbis/rmi-application-server-v3.json
{ "method":"createSamples", "params":[ "my_username-160601233044713x0F476AEA0FE875E4E87CB16C6D6902D6", [ { "properties":{}, "typeId":{ "permId":"SAMPLE_TYPE", "@type":"as.dto.entitytype.id.EntityTypePermId" }, "code":"THIS_SAMPLE_HAS_NO_NAME", "spaceId":{ "permId":"MY_USERNAME", "@type":"as.dto.space.id.SpacePermId" }, "tagIds":[ { "code":"THIS_IS_A_TAG_NAME", "@type":"as.dto.tag.id.TagCode" } ], "@type":"as.dto.sample.create.SampleCreation", "experimentId":null, "containerId":null, "componentIds":null, "parentIds":null, "childIds":null, "attachments":null, "creationId":null, "autoGeneratedCode":null } ] ], "id":"1", "jsonrpc":"2.0" }
If you want to specify one or more parents for that new sample, you need to specify it like this:
"parentIds": [{ "permId" : "20160706001644827-208", "@type":"as.dto.sample.id.SamplePermId" }],
Fetch options
When you get a sample or any other openBIS entity, you often want to get some additional information along with the result, such as:
- its properties
- its parents
- its datasets
These informations can be obtained with additional fetch options, which can be nested, too:
POST https://openbis-hostname:8443/openbis/openbis/rmi-application-server-v3.json
{ "type": { "@type": "as.dto.sample.fetchoptions.SampleTypeFetchOptions" }, "properties": { "@type": "as.dto.property.fetchoptions.PropertyFetchOptions" }, "parents": { "@type": "as.dto.sample.fetchoptions.SampleFetchOptions", "properties": { "@type": "as.dto.property.fetchoptions.PropertyFetchOptions" } }, "dataSets": { "@type": "as.dto.dataset.fetchoptions.DataSetFetchOptions", "properties": { "@type": "as.dto.property.fetchoptions.PropertyFetchOptions" } } }
The fetch options are the third (mandatory) parameter in your 'params' array. It may be an empty dictionary.
Here is an overview over all the possible fetch options and its type. Parents and childrens are usually of the same @type as the requested entity. This means, the parent of a sample is a parent too, the children of a
dataSet are dataSets too and so on.
name of entity | @type | Remarks |
---|---|---|
type | as.dto.sample.fetchoptions.SampleTypeFetchOptions |
|
properties | as.dto.property.fetchoptions.PropertyFetchOptions |
|
parents | as.dto.sample.fetchoptions.SampleFetchOptions | identical to the requested entity |
childrens | as.dto.sample.fetchoptions.SampleFetchOptions | identical to the requested entity |
container | as.dto.sample.fetchoptions.SampleFetchOptions | identical to the requested entity |
components | as.dto.sample.fetchoptions.SampleFetchOptions |
|
dataSets | a:qs.dto.dataset.fetchoptions.DataSetFetchOptions |
|
experiment | as.dto.experiment.fetchoptions.ExperimentFetchOptions |
|
space | as.dto.space.fetchoptions.SpaceFetchOptions |
|
project | as.dto.project.fetchoptions.ProjectFetchOptions |
|
materialProperties | as.dto.material.fetchoptions.MaterialFetchOptions |
|
history | as.dto.history.fetchoptions.HistoryEntryFetchOptions |
|
tags | as.dto.tag.fetchoptions.TagFetchOptions |
|
registrator | as.dto.person.fetchoptions.PersonFetchOptions |
|
modifier | as.dto.person.fetchoptions.PersonFetchOptions |
|
attachments | as.dto.attachment.fetchoptions.AttachmentFetchOptions |
|
content | as.dto.common.fetchoptions.EmptyFetchOptions |
|
get Datasets
Datasets are our links to the actual data, which is hosted on the datastore servers. Like samples,
datasets can have parents, children
Unlike samples, datasets only have a permId but not identifier. Its parents and containers
are datasets too, which means they are of the same @type.
POST https://openbis-hostname:8443/openbis/openbis/rmi-application-server-v3.json
{ "method": "getDataSets", "params": [ "my_username-160601233044713x0F476AEA0FE875E4E87CB16C6D6902D6", [ { "permId": "20130412142942295-198", "@type": "as.dto.dataset.id.DataSetPermId" } ], { "parents": { "@type": "as.dto.dataset.fetchoptions.DataSetFetchOptions" }, "containers": { "@type": "as.dto.dataset.fetchoptions.DataSetFetchOptions" }, "physicalData": { "@type": "as.dto.dataset.fetchoptions.PhysicalDataFetchOptions" }, "linkedData": { "@type": "as.dto.dataset.fetchoptions.LinkedDataFetchOptions" }, "dataStore": { "@type": "as.dto.datastore.fetchoptions.DataStoreFetchOptions" }, "sample": { "@type": "as.dto.sample.fetchoptions.SampleFetchOptions" }, "properties": { "@type": "as.dto.property.fetchoptions.PropertyFetchOptions" }, "@type": "as.dto.dataset.fetchoptions.DataSetFetchOptions" } ], "id": "1", "jsonrpc": "2.0" }
get Dataset files
Finally we want to fetch the original files themselves. Our application server only hosts the metadata, not the original files. But it knows whom to ask. In the above example, we get back a lot of metadata, including 'dataStore'. This metadata object contains information about on which dataStore server our data is actually stored:
POST https://openbis-hostname:8443/openbis/openbis/rmi-application-server-v3.json
"dataStore": { "@type": "as.dto.datastore.DataStore", "@id": 11, "fetchOptions": { "@type": "as.dto.datastore.fetchoptions.DataStoreFetchOptions", "@id": 12, "cacheMode": "NO_CACHE" }, "code": "DSS1", "downloadUrl": "https://openbis-hostname:8444", "remoteUrl": "https://openbis-hostname:8444", "registrationDate": 1365753969172, "modificationDate": 1464467813297 }
To actually fetch the list of files, we use the v1 API on the specified datastore server:
POST https://openbis-hostname:8444/datastore_server/rmi-dss-api-v1.json
{ "method": "listFilesForDataSet", "params": [ "my_username-160601233044713x0F476AEA0FE875E4E87CB16C6D6902D6", "20130412142942295-198", "/", true ], "id": "1", "jsonrpc": "2.0" }
The argument list of 'params' is as follows:
- our session token
- the **permId of our dataset
- the folder to start with, usually '/' which means the root directory, '/start/here' if your starting point is somewhere else
- true = recursively look for all files in all subfolders; false = just get the list of the specified folder
This is the result we get back from the above query:
{ "result": [ { "@type": "FileInfoDssDTO", "@id": 1, "pathInDataSet": "thumbnails_256x256.h5ar", "pathInListing": "thumbnails_256x256.h5ar", "isDirectory": true, "fileSize": "-1" }, { "@type": "FileInfoDssDTO", "@id": 2, "pathInDataSet": "thumbnails_256x256.h5ar/wA10_d1-1_cCy5.png", "pathInListing": "thumbnails_256x256.h5ar/wA10_d1-1_cCy5.png", "isDirectory": false, "crc32Checksum": -1396446681, "fileSize": "1505" }, ... (many more files)... ], "id": "1", "jsonrpc": "2.0" }
the 'isDirectory' flag indicates if whether the entry is a folder or not. Together with the earlier
information about our datastore server ('downloadUrl'), we are now able to actually download a file:
https://openbis-hostname:8444/datastore_server/thumbnails_256x256.h5ar/wA10_d1-1_cCy5.png
upload data
Uploading data is done in three steps:
- get the DSS URL
- upload the file(s) to the session workspace
- register the file(s) in openBIS
1. get available DSS
Before we can upload any data, we need to ask the application server (AS) which DSS are available. In many cases, we only have one DSS, but in some cases we can have more than one, so we have to choose. Fetching the list of available DSS done like this:
POST https://openbis-hostname:8443/openbis/openbis/rmi-general-information-v1.json
{ "method": "listDataStores", "params": [ "my_username-160601233044713x0F476AEA0FE875E4E87CB16C6D6902D6" ], "id": "1", "jsonrpc": "2.0" }
which gives you following result:
{ "result": [ { "@type": "DataStore", "@id": 1, "code": "DSS2", "downloadUrl": "https://openbis-hostname:8445/datastore_server", "hostUrl": "https://openbis-hostname:8445" }, { "@type": "DataStore", "@id": 2, "code": "DSS1", "downloadUrl": "https://openbis-hostname:8444/datastore_server", "hostUrl": "https://openbis-hostname:8444" } ], "id": "1", "jsonrpc": "2.0" }
In our example we get two different datastores, and the attributes of our interest are downloadUrl which we need for the upload process and code which we need to initiate the registration process.
2. upload a file to the DSS
Now we have a choice of a DSS, we can start uploading a file. Our endpoint for uploading files:
downloadUrl/session_workspace_file_upload
In the URL we place the additional attributes:
- filename
- id
- startByte
- endByte
- sessionID
all of the above attributes are mandatory. If you don't know the file size, you can just set startByte=0
and endByte=0
.
Example:
You need to include Content-Type: multipart/form-data
in your HTTP header in order to upload the file.
After successful upload, the response includes a status message, the filename and the size in bytes:
{ "id": 1, "status": "ok", "endByte": 0, "startByte": 0, "filename": "my_filename", "size": 1280 }
3. start data registration process
Once the file is uploaded to the user session workspace, we can initiate the data registration process. Every data file needs to be coupled to a dataset object which in turn needs to be connected to a sample. Without this important registration step, our file remains in the session workspace and gets deleted as soon as the session expires.
The registration process can be started by posting a request to our application server (AS):
{ "method": "createReportFromAggregationService", "params": [ sessionToken, "DSS2", "my_dropbox_plugin", additional_parameters ] }
This will start the dropbox my_dropbox_plugin (a Jython script which lives inside your DSS plugins folder) to look for files in our user session workspace and register them in openBIS. DSS2 is the code of the datastore server where we uploaded our file. The additional_parameters are sent to the process
method of our my_dropbox_plugin. These parameters typically include a sampleIdentifier, because all datasets need to be connected to a sample.
Example:
POST https://openbis-hostname:8443/openbis/openbis/rmi-query-v1.json
{ "method": "createReportFromAggregationService", "params": [ "my_username-160601233044713x0F476AEA0FE875E4E87CB16C6D6902D6", "DSS2", "jupyter-uploader-api", { "sample": { "identifier": "/FOO/BAR" }, "container": { "name": "Analysis name", "description": "This is a description of my analysis" }, "dataSets" : [ { "dataSetType" : "JUPYTER_RESULT", "sessionWorkspaceFolder" : "results/", "fileNames" : ["results/blublu"] } ] } ], "id": "1", "jsonrpc": "2.0" }