Abstract

The Allotrope Data Format (ADF) [ADF] consists of several APIs and taxonomies. The ADF Data Package API [ADF-DP] defines an interface for storing files and folder structures and thus provides one of most essential operations of the ADF. This document provides a Developer's Guide for the Allotrope Data Format Data Package API.

Disclaimer

THESE MATERIALS ARE PROVIDED "AS IS" AND ALLOTROPE EXPRESSLY DISCLAIMS ALL WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, INCLUDING, WITHOUT LIMITATION, THE WARRANTIES OF NON-INFRINGEMENT, TITLE, MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.

Status of This Document

This section describes the status of this document at the time of its publication. Other documents may supersede this document. A list of current Allotrope publications and the latest revision of this technical report can be found in the Allotrope technical reports index at http://purl.allotrope.org/TR/.

This document is part of a set of specifications on the Allotrope Data Format [ADF]

This document was published by the Allotrope Foundation as a First Public Working Draft. This document is intended to become an Allotrope Recommendation. If you wish to make comments regarding this document, please send them to more.info@allotrope.org. All comments are welcome.

Publication as a First Public Working Draft does not imply endorsement by the Allotrope Membership. This is a draft document and may be updated, replaced or obsoleted by other documents at any time. It is inappropriate to cite this document as other than work in progress.

Table of Contents

1. Introduction

The Allotrope Data Format (ADF) defines an interface for experimental data generated in analytical laboratory processes. It is intended for data exchange, long-term preservation and fast real-time data access. The ADF Data Package (ADF-DP) API defines an interface for storing files and folder structures and thus provides one of most essential operations of the ADF.

Next, the operations on files and folders are explained along examples. Towards the end of this document example code is listed.

1.1 Document Conventions

1.1.1 Namespaces

Within this specification, the following namespace prefix bindings are used:

Prefix Namespace
owl:http://www.w3.org/2002/07/owl#
rdf:http://www.w3.org/1999/02/22-rdf-syntax-ns#
rdfs:http://www.w3.org/2000/01/rdf-schema#
xsd:http://www.w3.org/2001/XMLSchema#
skos:http://www.w3.org/2004/02/skos/core#
dct:http://purl.org/dc/terms/
foaf:http://xmlns.com/foaf/0.1/
adf-dp:http://purl.allotrope.org/ontologies/datapackage#

1.1.2 Indication of Requirement Levels

Within this document the definitions of MUST, SHOULD and MAY are used as defined in [rfc2119].

1.1.3 Number Formatting

Within this document, decimal numbers will use a dot "." as the decimal mark.

2. ADF Data Package Operations

This section describes the operations that can be carried out with the Data Package API. First, the operations on folders are described, then the operations on files.

The terminology that is used below is defined in [ADF-DPO].

2.1 Accessing the Data Package

The Data Package API provides a method that offers access to the Data Package contained in a given ADF file. The method takes as argument the ADF file and returns a reference to the Data Package contained in the file.

Example 1
JAVA:
// Retrieves the "Data Package" of the opened ADF file.
DataPackage dataPackage = adfFile.getDataPackage();

C#:
// Retrieves the "Data Package" of the opened ADF file.
DataPackage dataPackage = adfFile.getDataPackage();

2.2 Folder Operations

The following sections describe the operations that can be carried out on folders with the Data Package API.

2.2.1 Creating a Folder

The Data Package API provides a method to create a folder in another folder of the Data Package. The method takes the name of the new folder as argument and returns a reference to the new folder. The method throws an Exception, if a folder or file with this name already exists in the target folder. Further, the name of the folder MUST fulfil the interoperability requirements (described in [ADF-DP]). Otherwise an exception is thrown.

The following example shows the creation of a folder:

Example 2
JAVA:
// Retrieves the Data Package of the opened file.
DataPackage dataPackage = adfFile.getDataPackage();

// Retrieves the root folder of the DataPackage.
DpFolder rootFolder = dataPackage.openRootFolder();
			
// Creates a new folder "myFolder"
DpFolder newFolder = rootFolder.createFolder("myFolder");

C#:
// Retrieves the Data Package of the opened file.
DataPackage dataPackage = adfFile.getDataPackage();

// Retrieves the root folder of the DataPackage.
DpFolder rootFolder = dataPackage.openRootFolder();

// Creates a new folder "myFolder"
DpFolder newFolder = rootFolder.createFolder("myFolder");

The meta-data created in the triple store look like this:

Example 3
<urn:uuid:6eca6e3e-0e88-4de1-a048-334e124b0f6d>
      a       af-dp:Folder , ldp:Container ;
      dc:title "myFolder"^^xsd:string ;
      af-dp:representedBy <hdf:/data-package> .

<hdf:/data-package>
      a       hdf:Group ;
      hdf:name "data-package"^^xsd:string .

2.2.2 Accessing a Folder

The ADF-DP API provides a method to access a given folder in a Data Package. The folder MAY be given by URI, by absolute path or by relative path. The method takes the name resp. the URI of the folder as argument and returns a reference to this folder, if it exists.

The method throws an exception, if the folder given by URI or by path does not exist. The folder returned by this method MUST fulfil the requirements for accessing a folder specified in [ADF-DP].

2.2.2.1 Accessing a Folder by URI
Example 4
JAVA:
DpFolder folder = dataPackage.getFolderByURI("urn:uuid:1123123-123123...");

C#:
DpFolder folder = dataPackage.getFolderByURI("urn:uuid:1123123-123123...");
2.2.2.2 Accessing a Folder by path
Example 5
JAVA:
DpFolder folder = dataPackage.getFolderByPath("/a/b/c");

C#:
DpFolder folder = dataPackage.getFolderByPath("/a/b/c");
2.2.2.3 Accessing a Folder by browsing
Example 6
JAVA:
DpFolder folder = dataPackage.openRootFolder().openFolder("a").openFolder("b").openFolder("c");
folder.getAbsolutePath(); -> /a/b/c

JAVA:
DpFolder folder = dataPackage.openRootFolder().openFolder("a").openFolder("b").openFolder("c");
folder.getAbsolutePath(); -> /a/b/c

2.2.3 Listing the Content of a Folder

The ADF-DP API provides a method to list the contents of a folder in a given Data Package. The method returns an empty list, if the folder does not have any contents.

Example 7
JAVA:
DpFolder rootFolder = dataPackage.openRootFolder();

List<DpNode> contents = rootFolder.contents();
for (DpNode content : contents)
{
	if (content instanceOf DpFolder) {
		// print out the inherent meta data of the folder, e.g. content.getName(), content.getCreationOn(), ...
	} else if (content instanceOf IFile) {
		// print out the inherent meta data of the file, e.g. content.getName(), content.getSize(), ...
	}
}

C#:
DpFolder rootFolder = dataPackage.openRootFolder();

java.util.List contents = rootFolder.contents();
foreach (DpNode content in contents)
{
	if (content is DpFolder) {
		// print out the inherent meta data of the folder, e.g. content.getName(), content.getCreationOn(), ...
	} else if (content is IFile) {
		// print out the inherent meta data of the file, e.g. content.getName(), content.getSize(), ...
	}
}

The meta data of files and folders to load is based on the requirements for accessing files and accessing folders specified in [ADF-DP].

2.2.4 Deleting a Folder

The ADF-DP API provides a method to delete a folder in a given Data Package:

Example 8
JAVA:
// Deletes the folder "toBeDeleted"
dataPackage.deleteFolder("/helloWorld/toBeDeleted");			

C#:
// Deletes the folder "toBeDeleted"
dataPackage.deleteFolder("/helloWorld/toBeDeleted");

2.3 File Operations

The following sections describe the operations that can be carried out on files of the Data Package with the Data Package API.

2.3.1 Creating a File

The following example demonstrates how a file is created:

Example 9
JAVA:
// Retrieves the folder "helloWorld"
DpFolder helloWorld = dataPackage.openFolder("/helloWorld");

// Creates the file "Hello World.txt" inside the "helloWorld" folder
DpFile file = helloWorld.createFile("Hello World.txt");

C#:
// Retrieves the folder "helloWorld"
DpFolder helloWorld = dataPackage.openFolder("/helloWorld");

// Creates the file "Hello World.txt" inside the "helloWorld" folder
DpFile file = helloWorld.createFile("Hello World.txt");

When creating a file, a UUID URN URI is generated (file-URI) and the class of the file stated (adf-dp:File and ldp:Resource) in the ADF Triple Store:

Example 10
$fileURI rdfs:type adf-dp:File, ldp:Resource.
Example 11
<urn:uuid:fde37fc2-296a-4bbf-8f7c-219d50de2dc1>
	a      		ldp:Resource, adf-dp:File ;
	dc:title 	"fileName"^^xsd:string ;
	adf-dp:representedBy <hdf:/data-package/fileName> .

<hdf:/data-package/fileName>
	a      		hdf:Dataset ;
	hdf:name	"fileName"^^xsd:string .

2.3.2 Accessing a File

The ADF-DP API provides a method to access a given file in a Data Package. The file MAY be given by URI, by absolute path or by relative path (as shown in the next example). The method takes the name resp. the URI of the file as argument and returns a reference to this file, if it exists.

The method throws an exception, if the file given by URI or by path does not exist.

The file returned by this method fulfils the requirements specified in [ADF-DP].

2.3.2.1 Accessing a File directly
Example 12
JAVA:
// Retrieves the file "Hello World.txt" directly. The name here has to be absolute!
DpFile file1 = dataPackage.openFile("/helloWorld/Hello World.txt");

C#:
// Retrieves the file "Hello World.txt" directly. The name here has to be absolute!
DpFile file1 = dataPackage.openFile("/helloWorld/Hello World.txt");
2.3.2.2 Accessing a File from parent folder
Example 13
JAVA:
// Retrieves the folder "helloWorld"
DpFolder helloWorld = dataPackage.openFolder("/helloWorld");
DpFile file1 = helloWorld.openFile("Hello World.txt");

C#:
// Retrieves the folder "helloWorld"
DpFolder helloWorld = dataPackage.openFolder("/helloWorld");
DpFile file1 = helloWorld.openFile("Hello World.txt");

2.3.3 Reading a File

The ADF-DP API provides methods to read a given file in a Data Package either as byte content or as an input stream.

2.3.3.1 Reading a File as byte content

The following example shows how to read the byte content directly from a file:

Example 14
JAVA:
// Retrieves the file "Hello World.txt" directly.
DpFile file = dataPackage.openFile("/helloWorld/Hello World.txt");

// Reads the content of the file.
byte[] bytes = file.read();
String text = new String(bytes);

C#:
// Retrieves the file "Hello World.txt" directly.
DpFile file = dataPackage.openFile("/helloWorld/Hello World.txt");

// Reads the content of the file.
sbyte[] bytes = file.read();
String text = esme.util.StringUtil.create(bytes);
2.3.3.2 Reading a File as input stream

The following example shows how to read from a file through an input stream:

Example 15
JAVA:
DpFile file = dataPackage.openFile("/helloWorld/streamedFile.bin");

// Creates an input stream to read from the file. 
try (InputStream is1 = new DpInputStream(file))
{
	// Streaming to file
	int int1;
	while ((int1 = is1.read()) != -1)
	{
		byte byte1 = (byte) int1;
		log.info("" + byte1);
	}
}

C#:
// Creates an input stream to read from the file.
DpInputStream is1 = new DpInputStream(null);
try
{
	int int1;
	while ((int1 = is1.read()) != -1)
	{
		sbyte byte1 = (sbyte) int1;
		Log.info("" + byte1);
	}
}
finally
{
	is1.close();
}									

2.3.4 Updating a File

The ADF-DP API provides a method to write bytes into a given File within a Data Package. When updating a file, the meta data of the class adf-dp:File MUST be fulfilled at modification time and updated in the ADF-TS as specified in [ADF-DP].

The detailed specification for writing into a file is given in [ADF-DP].

2.3.4.1 Overwriting a File
Example 16
JAVA:
// Retrieves the file "Hello World.txt" directly.
DpFile file = dataPackage.openFile("/helloWorld/Hello World.txt");

// Writes new content to the file. Default is overwriting.
file.write("2, 3, 5, 7 and 11".getBytes());

// -> "2, 3, 5, 7 and 11"

C#:
// Retrieves the file "Hello World.txt" directly.
DpFile file = dataPackage.openFile("/helloWorld/Hello World.txt");

// Writes new content to the file. Default is overwriting.
file.write(esme.util.StringUtil.getBytes("2, 3, 5, 7 and 11"));

// -> "2, 3, 5, 7 and 11"
2.3.4.2 Appending to a File
Example 17
JAVA:
// continued from previous example
file.write(" are prime numbers".getBytes(), OpenOption.APPEND);

// -> "2, 3, 5, 7 and 11 are prime numbers"

C#:
// continued from previous example
file.write(esme.util.StringUtil.getBytes(" are prime numbers"), OpenOption.APPEND);

// -> "2, 3, 5, 7 and 11 are prime numbers"
2.3.4.3 Writing to a File by stream
Example 18
JAVA:
// Retrieves the folder "helloWorld" directly.
DpFolder helloWorld = dataPackage.openFolder("/helloWorld");

// Creates a new file to stream to.
DpFile file = helloWorld.createFile("streamedFile.bin");

int numberToWrite = 0;

// Creates an output stream to write to the new file.
try (OutputStream os = new DpOutputStream(file))
{
	while (numberToWrite < 1000)
	{
		os.write(numberToWrite++);
	}
}

C#:
// Retrieves the folder "helloWorld" directly.
DpFolder helloWorld = dataPackage.openFolder("/helloWorld");

// Creates a new file to stream to.
DpFile file = helloWorld.createFile("streamedFile.bin");

int numberToWrite = 0;

// Creates an output stream to write to the new file.
java.io.OutputStream os = new DpOutputStream(file);
try
{
	while (numberToWrite < 1000)
	{
		os.write(numberToWrite++);
	}
}
finally
{
	os.close();
}
2.3.4.4 Deleting a File

The ADF-DP API provides a method to delete a file in a given Data Package:

Example 19
JAVA:
// Deletes the file "toBeDeleted.txt"
dataPackage.deleteFile("/helloWorld/toBeDeleted.txt");			

C#:
// Deletes the file "toBeDeleted.txt"
dataPackage.deleteFile("/helloWorld/toBeDeleted.txt");

2.4 Import and export files and folders

The ADF-DP API provides methods to import files and folders into ADF and export these files back to the system again.

2.4.1 Import

The import functionality imports the whole content of a folder or just a file into the ADF. It creates DpFolder and DpFile according to the hierarchy of the import content and preserves meta-data of the original files and folders. Symbolic links are automatically followed. If this behavior is not wanted, it can be turned off by org.allotrope.adf.enums.LinkOption.NOFOLLOW_LINKS.

Example 20
JAVA and C#:
// import contents of system folder into ADF folder "dpFolder"
dataPackage.importIntoDpFolder(importDirectoryPath, dpFolder);

// it is also possible to just import a single file into the ADF DpFolder
dataPackage.importIntoDpFolder(anyFilePath, dpFolder);		

// Symbolic links are automatically followed. If this behavior is not wanted, it can be turned off by specifying LinkOption.NOFOLLOW_LINKS
dataPackage.importIntoDpFolder(anyFilePath, dpFolder, LinkOption.NOFOLLOW_LINKS);	

2.4.2 Export

The export functionality exports the content of the ADF DpFolder or just a DpFile according to their hierarchy in the ADF to the specified system folder. Preserved meta-data is automatically restored.

Example 21
JAVA and C#:
// export contents of dpFolder into system folder "exportDirectory"
dataPackage.exportDpFolder(dpFolder, exportDirectory);

// it is also possible to just export a single DpFile
dataPackage.exportDpFile(dpFile, exportDirectory);

3. Complete Example

The ADF-DP First Steps Example Application illustrates the Java API of ADF-DP by one complete code example. It is contained in the file FirstSteps.java in the package org.allotrope.adf.dp.firststeps.

4. Change History

Version Release Date Remarks
0.4.0 2015-06-29 Initial Working Draft version
1.0.0 RC 2015-09-17
  • Renamed document from User Manual to Developer's Guide
  • Renamed section Example Code to Complete Example
  • Removed unnecessary sub section from section Complete Example
  • Added provenance information to the Complete Example
1.0.0 2015-09-29
  • Updated versions, dates and document status
  • Removed code from section Complete Example
1.1.0 RC 2016-03-11
  • updated versions, dates and document status
  • added section on number formatting to document conventions
  • added information and examples for C#/.NET
1.1.0 RF 2016-03-31
  • updated versions, dates and document status
1.1.5 2016-05-13
  • updated versions and dates
1.2.0 Preview 2016-09-23
  • updated versions and dates
  • Corrected the data type of the size of a DpFile from int to long
1.3.0 Preview 2017-03-31
  • updated versions and dates
1.3.0 RF 2017-06-30
  • updated versions and dates, authors
  • added section on Importing and exporting files and folders

A. References

A.1 Normative references

[ADF]
Allotrope. Allotrope Data Format Overview. URL: http://purl.allotrope.org/TR/adf/
[ADF-DP]
Allotrope. ADF Data Package API. URL: http://purl.allotrope.org/TR/adf-dp/
[ADF-DPO]
Allotrope. Allotrope Data Package Ontology. URL: http://purl.allotrope.org/TR/adf-dpo/
[rfc2119]
S. Bradner. IETF. Key words for use in RFCs to Indicate Requirement Levels. March 1997. Best Current Practice. URL: https://tools.ietf.org/html/rfc2119