A simple iRODS Micro-Service

Introduction

The goal I had for this task was to identify and understand the steps and configurations involved in writing a micro-service and seeing it in action - for details regarding iRODS please refer to documentation at https://www.iRODS.org/. The micro-service that I wrote is very simplistic (it writes a hello world message to the system log), however it serves its purpose by providing an overview of steps that will be involved in writing a useful micro-service. 

Before I document the configurations and codes involved in creating and registering the new micro-service let’s look at figure 1.




Figure 1 shows a high level view of  invocation of a micro-service by the iRODS rules engine. One way of looking at the micro-service and the iRODS rule engine is to think of it as an event based triggering system that can perform ‘operations’ on the data objects, and/or external resources. The micro-services are registered in iRODS rule definitions and the rule engine invokes them based on the condition specified for that rule. For a list of places in the iRODS workflow where a micro-service may be triggered please visit: https://www.irods.org/index.php/Default_iRODS_Rules.

Also you may refer to https://www.iRODS.org/index.php/Rule_Engine for a detailed diagram of a micro-service invocation.

Figure 2 above shows the communication between the iRODS rule engine and a micro-service. A simplistic view of the communication layers is that the rule engine calls a defined C procedure, which exposes its functionality through an interface (commonly prefixed with msi). The arguments to the procedure are passed through a structure named msParam_t that is defined below:

typedef struct MsParam {
  char *label;
  char *type;         /* this is the name of the packing instruction in
                       * rodsPackTable.h */
  void *inOutStruct;
  bytesBuf_t *inpOutBuf;
} msParam_t;

Writing the micro-service

Figure 3 shows the steps involved in creating a new micro-service:

Write the C procedure

The C code below (lets call it test.c) has a function writemessage that writes a message to the system log. There is an interface to the function named msiWritemessage which exposes the writemessage function. The msi function takes a list of arguments of type msParam_t and a last argument of type ruleExecInfo_t for the result of the operation.

#include <stdio.h>
#include <unistd.h>
#include <syslog.h>
#include <string.h>
#include "apiHeaderAll.h"


void writemessage(char arg1[], char arg2[]);
int msiWritemessage(msParam_t *mParg1, msParam_t *mParg2,  ruleExecInfo_t *rei);


void writemessage(char arg1[], char arg2[]) {
    openlog("slog", LOG_PID|LOG_CONS, LOG_USER);
    syslog(LOG_INFO, "%s %s from micro-service", arg1, arg2);
    closelog();
}

int msiWritemessage(msParam_t *mParg1, msParam_t *mParg2,  ruleExecInfo_t *rei)
{
 char *in1;
 int *in2;
 RE_TEST_MACRO ("    Calling Procedure");
 // the above line is needed for loop back testing using irule -i option
 if ( strcmp( mParg1->type, STR_MS_T ) == 0 )
 {
    in1 = (char*) mParg1->inOutStruct;
 }
 if ( strcmp( mParg2->type, INT_MS_T ) == 0 )
 {
    in2 = (int*) mParg2->inOutStruct;
 }
 writemessage(in1, in1);
 return rei->status;
}

Next I will make a folder structure in the module folder of iRODS home for placing this micro-service and copy a few files from an example properties module and modify them to fit the test.c micro-service

cd ~irods
mkdir modules/HCC
cd modules/HCC

mkdir microservices
mkdir rules
mkdir lib
mkdir clients
mkdir servers

mkdir microservices/src
mkdir microservices/include
mkdir microservices/obj
cp ../properties/Makefile .
cp ../properties/info.txt .

Listed below is my working copy of Makefile and the info.txt

#Makefile
ifndef buildDir
buildDir = $(CURDIR)/../..
endif

include $(buildDir)/config/config.mk
include $(buildDir)/config/platform.mk
include $(buildDir)/config/directories.mk
include $(buildDir)/config/common.mk

#
# Directories
#
MSObjDir =    $(modulesDir)/HCC/microservices/obj
MSSrcDir =    $(modulesDir)/HCC/microservices/src
MSIncDir =    $(modulesDir)/HCC/microservices/include

# Source files

OBJECTS =    $(MSObjDir)/test.o


# Compile and link flags
#
INCLUDES +=    $(INCLUDE_FLAGS) $(LIB_INCLUDES) $(SVR_INCLUDES)
CFLAGS_OPTIONS := $(CFLAGS) $(MY_CFLAG)
CFLAGS =    $(CFLAGS_OPTIONS) $(INCLUDES) $(MODULE_CFLAGS)

.PHONY: all server client microservices clean
.PHONY: server_ldflags client_ldflags server_cflags client_cflags
.PHONY: print_cflags

# Build everytying
all:    microservices
    @true

# List module's objects and needed libs for inclusion in clients
client_ldflags:
    @true

# List module's includes for inclusion in the clients
client_cflags:
    @true

# List module's objects and needed libs for inclusion in the server
server_ldflags:
    @echo $(OBJECTS) $(LIBS)

# List module's includes for inclusion in the server
server_cflags:
    @echo $(INCLUDE_FLAGS)

# Build microservices
microservices:    print_cflags $(OBJECTS)

# Build client additions
client:
    @true

# Build server additions
server:
    @true

# Build rules
rules:
    @true

# Clean
clean:
    @echo "Clean image module..."
    rm -rf $(MSObjDir)/*.o


# Show compile flags
print_cflags:
    @echo "Compile flags:"
    @echo "    $(CFLAGS_OPTIONS)"

# Compile targets
#
$(OBJECTS): $(MSObjDir)/%.o: $(MSSrcDir)/%.c $(DEPEND)
    @echo "Compile image module `basename $@`..."
    @$(CC) -c $(CFLAGS) -o $@ $<

info.txt

Name:        HCC
Brief:        HCC Test microservice
Description:    HCC Test microservice.
Dependencies:
Enabled:    yes
Creator:    Ashu Guru
Created:    December 2011
License:    BSD

In the next step I will define the micro-service header and micro-service table files so that the iRODS can be configured with the new micro-service. This is done in the folder microservices/include. In this example  there is no header for this code so I have left the header file blank;  in the micro-service table file I have the entry for the table definition.  The specifics to note below are that the first argument is the label of the micro-service, the second argument is the count of input arguments  (do not count the ruleExecInfo _t argument) of the msi interface and the third argument is the name of the msi interface function.

File microservices/include/microservices.table

{ "msiWritemessage",2,(funcPtr) msiWritemessage },   

Following is the directory tree structure for the HCC module that I have so far:

bash-4.1$ pwd 

/opt/iRODS/modules
bash-4.1$ tree HCC
HCC
├── clients
├── info.txt
├── lib
├── Makefile
├── microservices
│   ├── include
│   │   ├── microservices.header
│   │   ├── microservices.table
│   ├── obj
│   └── src
│       ├── test.c
├── rules
└── servers

Next I will make an entry for enabling the new module (this micro-service), this is done in the file ~irods/config/config.mk so that the iRODS Makefile can include the new micro-service for build. To do this simply add the module folder name (in my case HCC) to the variable MODULES.

Compile and test

cd ~irods/modules/<YOURMODULENAME>
make

The above commands should result in creation of an object file in the micro-service/obj folder. I am going to test the micro-service manually first, to accomplish this I will create a client side rule file in the folder ~irods/ clients/icommands/test/rules. I have named the file aguru.ir and following are the contents of the file:

aguruTest||msiWritemessage(*A,*B)|nop
*A=helloworld%*B=testing

 

The first line in file  is the rules definition and the second line are the input parameters. To test the micro-service I will  invoke the micro-service which will then write a message to the system log (see figure below).

Recompile iRODS

Before this step I must make the entries for the headers and the msi table in the iRODS main micro-service action table (i.e. file ~irods/server/re/include/reAction.h). This should be done using the following commands:

rm server/re/include/reAction.h
make reaction 

However, I had to manually add the code segment below to the file server/re/include/reAction.h file to accomplish that:

int msiWritemessage(msParam_t *mParg1, msParam_t *mParg2,  ruleExecInfo_t *rei);

Finally, recompile iRODS

cd ~irods
make test_flags
make modules
./irodsctl stop
make clean
make
./irodsctl start
./irodsctl status

Register Micro-service and Test

In this step we define a rule that will trigger the micro-service when a new data object is uploaded to iRODS. Open the file ~irods/server/config/reConfigs/core.re and add the following line  the Test Rules section.

acPostProcForPut {msiWritemessage("HelloWorld","String 2"); }

That is it… if now I put (iput) any file into iRODS a message is added to the /var/log/messages file on the iRODS server. Please note that the above rule is not filtering a particular occurrence but is a catchall rule that applies to all put events.

References:
https://www.irods.org/
http://www.wrg.york.ac.uk/iread/compiling-and-running-irods-with-micros-services
http://technical.bestgrid.org/index.php/IRODS_deployment_plan