...

• withListener(): Adds a listener which will be notified when
  • the download session has been started/finished/failed,
  • the download of a file/folder has been started/finished and
  • a chunk has been downloaded.
    There can be several listeners. By default there are no listeners. Note, that listeners are notified in a separated thread associated with the download session.
• withLogger(): Sets a logger. By default nothing is logged.
• withRetryProviderFactory(): Sets the factory which creates a retry provider. A retry provider knows when and how often a failed action (e.g. sever request) should be retried. By default it is retried three times. The first retry is a second later. For each following retry the waiting time is increases by the factor two.

Register Data Sets

Data Store V3 API provides means to register data sets using both Javascript and Java versions of the API. In both cases the registration process consists of two steps:

1. upload of data set files via "/datastore_server/store_share_file_upload" servlet
2. creation of a data set metadata and registration of the uploaded files via DataStoreServerApi.createUploadedDataSet method

Ad.1

The data set upload servlet waits for files at "/datastore_server/store_share_file_upload" url. It expects to receive a "multipart/form-data" HTTP request with uploaded files sent in a body of the request and some additional parameters specified as part of the URL query string (not in the body).

The servlet expects at least one file to be uploaded. Still, it can handle multiple files in one request as well. The additional servlet parameters are:

• sessionID - (mandatory) openBIS session token.
• uploadID - (mandatory) a unique upload identifier which is used to match a given upload or multiple uploads (i.e. step 1 of a registration) with a creation of a data set (i.e. step 2 of a registration); a client of the API is responsible for generating such an identifier; in Java "java.util.UUID" class is recommended for this purpose; in Javascript "createDataSetUpload" function is provided (see code examples below).
• dataSetType - (mandatory) type of a data set to be registered; basing on the type and a configuration of a data store a dropbox that will be used for a registration is found; at this point the dropbox is not executed, only a share that this dropbox uses is identified; the uploaded files are stored at that share to eliminate unnecessary copying of files in step 2 of the registration (i.e. when a dropbox is run).
• folderPath - (optional) if a data set contains only a single file then this file is going to be stored directly under "original" folder; if a data set contains multiple files then these files are going to be stored in "original/upload" sub folder; in both cases, "folderPath" parameter can be used to define an additional path under "original" folder where uploaded files are going to be stored;
• ignoreFilePath - (optional, default: true) most browsers send only base file names for files which were uploaded via a regular html form. Still, there are some browsers (e.g. Opera) that are known to send file paths as well. To handle all browsers consistently by default we ignore file paths even if they are given. We only use the file paths if it has been explicitly requested. This may be handy in contexts where we have a full control over what gets sent to the servlet and we want the uploaded files to be created with a specific folder structure.

Ad.2

Once data set files are uploaded DataStoreServerApi.createUploadedDataSet method should be called to create data set metadata and register the uploaded files in the data store. For the createUploadedDataSet method to know which files to register with a given data set the uploadID that was used for uploading the files has to be set in the data set creation (i.e. UploadedDataSetCreation.uploadId). The actual registration of a data set is performed by a dropbox. Which dropbox is used depends on a data store configuration. Data sets of different types can be assigned a different dropbox if needed. Also, there is a possibility to define a default dropbox that will be used for types that do not have any specific configuration. All this can be done in data store service.properties file. The default dropbox is specified with "dss-rpc.put-default" property, while type specific configurations have to follow this syntax: "dss-rpc.put.MY_TYPE = my-dropbox".

A minimal dropbox code that is needed to properly register a data set is as follows:

def process(tr):
	dataSet = tr.createNewDataSet()
	tr.moveFile(tr.getIncoming().getAbsolutePath(), dataSet)

Please note that in the above code snippet the data set returned by "createNewDataSet" method call does already have all the metadata properly set (i.e. it contains all the values that were defined in UploadedDataSetCreation object passed to "createUploadedDataSet" method).To register datasets using the Java or JavaScript API use one of the following examples as a template.

Example (Java)

import java.util.UUID;
import org.eclipse.jetty.client.HttpClient;
import org.eclipse.jetty.client.api.Request;
import org.eclipse.jetty.client.util.MultiPartContentProvider;
import org.eclipse.jetty.client.util.StringContentProvider;
import org.eclipse.jetty.http.HttpMethod;

import ch.ethz.sis.openbis.generic.asapi.v3.IApplicationServerApi;
import ch.ethz.sis.openbis.generic.asapi.v3.dto.dataset.id.DataSetPermId;
import ch.ethz.sis.openbis.generic.asapi.v3.dto.entitytype.id.EntityTypePermId;
import ch.ethz.sis.openbis.generic.asapi.v3.dto.sample.id.SampleIdentifier;
import ch.ethz.sis.openbis.generic.dssapi.v3.IDataStoreServerApi;
import ch.ethz.sis.openbis.generic.dssapi.v3.dto.dataset.create.UploadedDataSetCreation;
import ch.systemsx.cisd.common.http.JettyHttpClientFactory;
import ch.systemsx.cisd.common.spring.HttpInvokerUtils;

public class RegisterDataSet
{
    public static void main(String[] args) throws Exception
     {
{         
        final String AS_URL = "http://localhost:8888/openbis/openbis";
        final String DSS_URL = "http://localhost:8889/datastore_server";

        final OpenBIS openbisV3 = new OpenBIS(AS_URL, DSS_URL);

        openbisV3.login("admin", "password");

        final StringPath AS_URLpath = Path.of("http://localhost:8888/openbis/openbis"/uploadPath");
        final String DSS_URLuploadId = "http://localhost:8889/datastore_server"openbisV3.uploadFileWorkspaceDSS(path);

        final IApplicationServerApiUploadedDataSetCreation ascreation = HttpInvokerUtils.createServiceStub(IApplicationServerApi.class,new UploadedDataSetCreation();
        creation.setUploadId(uploadId);
        AS_URL + IApplicationServerApi.SERVICE_URL, 10000);
creation.setExperimentId(new ExperimentIdentifier("/DEFAULT/DEFAULT/DEFAULT"));
        IDataStoreServerApi dss = HttpInvokerUtils.createStreamSupportingServiceStub(IDataStoreServerApi.class,
creation.setTypeId(new EntityTypePermId("ATTACHMENT", EntityKind.DATA_SET));

        try
        {
  DSS_URL + IDataStoreServerApi.SERVICE_URL, 10000);

       final StringDataSetPermId sessionTokendataSetPermId = asopenbisV3.login("admin", "password"createUploadedDataSet(creation);
        String uploadId = UUID.randomUUID().toString();
   // A data set assigned Stringto dataSetTypethe =experiment "UNKNOWN";

        HttpClient client = JettyHttpClientFactory.getHttpClient();
/DEFAULT/DEFAULT/DEFAULT" with the folder "uploadPath" is created
             MultiPartContentProvider multiPart = new MultiPartContentProvider(System.out.println("dataSetPermId=" + dataSetPermId);
        multiPart.addFilePart("file1", "/filePath1/file1.txt", new StringContentProvider("content1"), null);} catch (final Exception e)
        multiPart.addFilePart("file2", "/filePath2/file2.txt", new StringContentProvider("content2"), null);
{
            multiParte.closeprintStackTrace();

        }

   Request request = client.newRequest(DSS_URL + "/store_share_file_upload").method(HttpMethod.POST openbisV3.logout();
    }
}

Example (Javascript)

<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<title>Dataset  request.param("sessionID", sessionToken);
        request.param("uploadID", uploadId);
        request.param("folderPath", "/folderPath");
        request.param("ignoreFilePath", String.valueOf(false));upload</title>

<script type="text/javascript" src="/openbis-test/resources/api/v3/config.js"></script>
<script type="text/javascript" src="/openbis-test/resources/api/v3/require.js"></script>

</head>
<body>
    <label for="myfile">Select a file:</label>
    <input type="file" id="myFile"/>
    <script>         
        request.param("dataSetType", dataSetType);require(["openbis", "dss/dto/dataset/create/UploadedDataSetCreation", "as/dto/experiment/id/ExperimentIdentifier",
        request.content(multiPart);
    "as/dto/entitytype/id/EntityTypePermId", "as/dto/entitytype/EntityKind"],
   request.send();

     function(openbis,   UploadedDataSetCreation, creationExperimentIdentifier, =EntityTypePermId, new UploadedDataSetCreation();EntityKind) {
        creation.setUploadId(uploadId);
    var testProtocol =  creation.setTypeId(new EntityTypePermId(dataSetType))window.location.protocol;
        creation.setSampleId(new SampleIdentifier("/DEFAULT/DEFAULT"));

   var testHost = window.location.hostname;
  // A data set with the following files is created:
 var testPort      // - /original/folderPath/filePath1/file1
= window.location.port;

           // - /original/folderPath/filePath2/file2
 
        DataSetPermId permId = dss.createUploadedDataSet(sessionToken, creation);
 var testUrl = testProtocol + "//" + testHost + ":" + testPort;
        System.out.println(permId);
    }
}

...

<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<title>Register Data Set</title>

<script type="text/javascript" src="/openbis/resources/api/v3/config.js"></script>
<script type="text/javascript" src="/openbis/resources/api/v3/require.js"></script>

</head>
<body>

	<script>
		require([ "jquery", "openbis", "as/dto/entitytype/id/EntityTypePermId", "as/dto/experiment/id/ExperimentIdentifier", "as/dto/sample/id/SampleIdentifier", "dss/dto/dataset/create/UploadedDataSetCreation" ], function($, openbis, EntityTypePermId, ExperimentIdentifier, SampleIdentifier, UploadedDataSetCreation) {

			$(document).ready(function() {
				var facade = new openbis();

				facadevar testApiUrl = testUrl + "/openbis/openbis/rmi-application-server-v3.json";

            var openbisV3 = new openbis(testApiUrl);

            var fileInput = document.getElementById("myFile");
            fileInput.onchange = (e) => {
                var files = e.target.files;

                openbisV3.login("admin", "password").done(function(sessionToken) => {
					                    var dataStoreFacade = facadeopenbisV3.getDataStoreFacade();

					dataStoreFacade.createDataSetUpload("UNKNOWN"                    dataStoreFacade.uploadFilesWorkspaceDSS(files).done(function(upload) {
						var uploadFrame = $("#uploadFrame");
						uploadFrame.load(function() {
							alert("Upload finished")
						});

						// Below we use upload.getUrl(folderPath, ignoreFilePath) method to generate
						// an upload url that contains additional parameters for the upload servlet.
						// See a detailed description of these parameters in the documentation above.

						var uploadForm = $("#uploadForm");
						uploadForm.attr("action", upload.getUrl("/folderPath", false));

						var createForm = $("#createForm");
						createForm.submit(function(e) {
							e.preventDefault();

							var experimentIdentifier = createForm.find("input[name=experimentIdentifier]").val();
							var sampleIdentifier = createForm.find("input[name=sampleIdentifier]").val();

							var creation = new UploadedDataSetCreation();
							uploadId => {
                        var creation = new UploadedDataSetCreation();
                        creation.setUploadId(uploadId);
                        creation.setExperimentId(new ExperimentIdentifier("/DEFAULT/DEFAULT/DEFAULT"));
                        creation.setTypeId(new EntityTypePermId(upload.getDataSetType()"ATTACHMENT", EntityKind.DATA_SET));
							creation.setUploadId(upload.getId());

							if (experimentIdentifier) {
								creation.setExperimentId(new ExperimentIdentifier(experimentIdentifier))
							}

							if (sampleIdentifier) {
								creation.setSampleId(new SampleIdentifier(sampleIdentifier));
							}

							dataStoreFacade.createUploadedDataSet(creation).done(function(permId) {
								alert("Data set " + permId + " created")
							}).fail(function(error) {
								alert("Couldn't create a data set: " + error.message);
							});

							return false;
						});
					});
				});
			});
		});
	</script>

	<iframe id="uploadFrame" name="uploadFrame" style="display: none"></iframe>
 
	<h1>Step 1 : upload file(s)</h1>
	<form id="uploadForm" method="post" enctype="multipart/form-data" target="uploadFrame">
		<input type="file" name="file" multiple="multiple">
		<input type="submit">
	</form>

	<h1>Step 2 : create data set</h1>
	<form id="createForm">
		<label>Experiment</label>
		<input type="text" name="experimentIdentifier">
		<label>Sample</label>
		<input type="text" name="sampleIdentifier"> <input type="submit">
	</form>

</body>
</html>


                        dataStoreFacade.createUploadedDataSet(creation).done(dataSetPermId => {
                             // A data set assigned to the experiment "/DEFAULT/DEFAULT/DEFAULT" with the folder "uploadPath" is created
                            console.log("dataSetPermId=" + dataSetPermId);
                            openbisV3.logout();
                        }).fail(error => {
                            console.error(error);
                            openbisV3.logout();
                        });
                    });
                });
            }
        });     
    </script>
</body>
</html>

VI. Web application context

...