Category: Hoopoe

The Hoopoe File System service reference can be found at: http://www.hoopoe-cloud.com/HoopoeFS.asmx

This post will present the File System interface to Hoopoe distributing engine. The File System (FS) interface can be used by users to transfer data files to be processed by Hoopoe with CUDA computing kernels. After processing completes, the same interface can be used to read back computed results.

Topics

• Features
• General terms
• API description
• API examples
  • Creating new instance
  • Authenticating
  • Creating a file
  • Creating a directory
  • Creating a file under a sub-directory
  • Deleting a file
  • Writing data into files
  • Reading data from files

1. Features

HoopoeFS exposes a simple interface for data and file management. In general, most features available by general OS file systems are given by HoopoeFS service, providing high flexibility for users.

Taking security into consideration, every users is provided with a completely isolated environment, so no special security functions should be used or exposed, as every users sees, and able to access only the files he generated or uploaded.

As a simplified manner, the API provided by HoopoeFS is generic, but there are few limitations to user operations and capabilities. For example, a user is allowed to place files in the root directory or under sub directories. A user is also allowed to create only one level of sub directories being able to contain additional files.

2. General terms

As previously mentioned, HoopoeFS provides all general constructs for working with files and directories.

A file, is simply a container for data, either raw or compressed and can be named using every supported character.

A directory, is a container for files, and provided is the root directory, and further sub directories that can be created by the user.

3. API description

For data constructs (File, Directory), are provided management functions as follows:

• CreateFile/CreateDirectory – allows to create a new file or directory, respectively. Calling these functions is required as a first step prior to accessing them.
• DeleteFile/DeleteDirectory – deletes a previously created file or directory.
• RenameFile/RenameDirectory – given existing files or directories, allows to modify their name.
• WriteData/ReadData – modifies the content of a file (write) or reads content from a specific file.
• GetFileSize – returns the number of bytes in a file.

For general operation, few more functions are given:

• Authenticate – returns a value indicating whether the user is registered and recognized by HoopoeFS
• IsUserOverQuota – returns a value indicating whether the user has exceeded the allowed storage space. In such case the user cannot create new files or directories, but can delete and read contents of files.

4. API examples

4.1 Creating new instance

In order to work with HoopoeFS, it is necessary to create a new instance of HoopoeFS class:
HoopoeFS hps = new HoopoeFS();

4.2 Authenticating

It is a good practice to check with HoopoeFS if we are authenticated, before performing futher operations. Every operation to be performed must use this authentication level:

Authentication a = new Authentication();
a.User = test@company_alias;
a.Password = "my_password"
hfs.AuthenticationValue = a;


4.3 Creating a file 

Creating a file is a simple task with HoopoeFS API:

hfs.CreateFile("test.dat");

4.4 Creating a directory

Following the previous example, a similar API can be used to create a new sub-directory (all directories are created under the root):

hfs.CreateDirectory("test_data");

4.5 Creating a file under a sub-directory

Once created a sub directory, any number of files can be created under it.

To do that, the following operations are necessary:

Directory d = new Directory();
d.Name = "test_data";
hfs.DirectoryValue = d;
hfs.CreateFile("child_temp.dat");

You may note, that once hfs.DirectoryValue is set, all file related operations correspond to the directory (creating new files, deleting, modifying etc.), so when working with the directory ends, hfs.DirectoryValue should be set to null.

4.6 Deleting a file

A very straight forward operation:

hfs.DeleteFile("test.dat");

4.7 Writing data into files

The concept of writing data to files within Hoopoe maps to the real world, with a simplified API.

byte[] data = new byte[512*1024];
// Load/generate data
....

// Write the data, starting at offset 0 of the file
long offset = 0;
hfs.WriteFile("temp.dat", data, offset);

// If willing to write more data, then consider a
// new offset
offset += data.Length;


4.8 Reading data from files

The same rules for writing data apply to reading it from files.

// Read the data, starting at offset 0 of the file
long offset = 0;
// Determines the amount of bytes to read
int length = 512*1024;
byte[] data = hfs.ReadFile("temp.dat", offset, length);

// Past this point, data will contain the bytes
// that were read.
// In case fewer bytes than requested were read,
// the size of data will be consistent with the actual
// bytes read.

// If willing to read more data, then consider a
// new offset
offset += data.Length;