Guide for Data Administrator

The Data Administrator represents in DITAS an actor (a single user, as well as, a representative of an entire organization) which will to expose some of the internal data using the facilities provided by DITAS. In fact, exposing the data to external users (which could be customers or even people interested on this data) requires to setup a proper infrastructure to make the user experience smooth and, at the same time, able to tailor the data accord to the level of privacy which must be preserved.

On this basis, the Data Administrator owns the data and it is responsible to create the DAL (Data Access Layer) which constitutes the middleware between the Data Administrator original data sources and the DITAS platform. The DAL hides the technicalities of the data sources which contain the data to be exposed and will communicate with the VDC (developed by the VDC Developer which has to be in contact and under the control of the Data Administrator). As a critical element of the DAL, the Data Administrator has also to configure the Privacy Enforcement Engine to make the data exposed compliant with the access rights of the application which will consume the data. Finally, the Data Administrator has to define the so-called CAF, a REST-API signature (which will be developed by the VDC Developer) that contains all the methods to access to the exposed data made available to the final user.

Once the DAL and the VDC components are ready, the Data Administrator is in charge of generating and publishing the related Abstract VDC Blueprint to the repository. To this aim, the BPGen tool is provided by DITAS to partially automate this process.

This part of the SDK contains all the guidelines in order to run the DAL Image with Privacy Enforcement Engine (if needed) using the deployment engine, as well as how to configure the abstract blueprint. At the end, there is Abstract Blueprint Schema Documentation.

Guidelines for DAL Configuration

Data administrator is responsible for defining the DAL and with its interface. The DAL exposes a protobuf API using gRPC.

Any DAL message, which is sent through this API and is defined as a protobuf, should include DalMessageProperties (https://github.com/DITAS-Project/ehealth-spark-vdc-with-dal/blob/master/VDC-DAL-grpc/src/main/protobuf/DalMessageProperties.proto) and DalPrivacyProperties (https://github.com/DITAS-Project/ehealth-spark-vdc-with-dal/blob/master/VDC-DAL-grpc/src/main/protobuf/DalPrivacyProperties.proto).

Dal Message Properties


syntax = "proto3";
package com.ditas.ehealth;
message DalMessageProperties {
string purpose = 1;
string requesterId = 2;
string authorization = 3;
}


Dal Privacy Properties


syntax = "proto3";
package com.ditas.ehealth;
message DalPrivacyProperties {
enum PrivacyZone {
PUBLIC = 0;
PRIVATE = 1;
}
PrivacyZone privacyZone = 1;
}


The DAL can be implemented in any language that has protobuf and gRPC implementations, e.g. Python, Java, Scala, Go. An example DAL definition as a service can be found in: https://github.com/DITAS-Project/ehealth-spark-vdc-with-dal/blob/master/VDC-DAL-grpc/src/main/protobuf/EHealthService.proto. The DAL uses the generated server implementation of the defined protobuf.

The DAL should be configured with access details of its corresponding data sources, with Policy Enforcement URL and IAM details. The Administrator is responsible of deploying the DAL with the Policy Enforcement, if it is needed. The policy enforcement engine is in https://github.com/DITAS-Project/policy-enforcement-engine. The configuration file inside conf/application.conf should be modified according to the enforcement engine implementation.

The DAL can be either deployed manually by the data administrator on his premises, or using the deployment engine on the DITAS infrastructure.

Abstract Blueprint Repository CRUD Operations

All the services, in order to help the Data Administrator to store, update and delete the abstract blueprint, are documented in the tables below.

{
           "type": "object",
           "properties": {
             "blueprint_id": {
               "type": "array",
               "items": {
                 "type": "string"
               }
             }
           },
           "required": [
             "blueprint_id"
           ]
         }

Services to Configure Abstract Blueprint

This section is dedicated to describing all the services that help the Data Administrator to configure the abstract VDC Blueprint

Semi-automated abstract blueprint generation and publication

DITAS provides a software component, named BPGen, to ease the data administrator in the creation of the abstract blueprint. In particular, BPGen automatically describes the API exposed by the VDC given an OpenAPI file, and populates the remaining fields of the abstract blueprint with default values. In this way, the data administrator is not required to remember the blueprint structure. Instead, (s)he only has to customize the default values and, if needed, remove unused attributes off the generated abstract blueprint. In addition, BPGen also helps the data administrator in publishing the abstract blueprint in the Repository Engine and keeping the published blueprint up-to-date. BPGen is available here

Prerequisites

In orded for BPGen to work properly, the following prerequisites are needed:

• Python 3.7
• GIT command-line client
• A GitHub account with read-write permissions on the desired project workspace
• A Repository Engine account with read-write permissions

Setup and configuration

Once downloaded from here, BPGen must be configured by executing the following command:

sh install.sh

In addition, GitHub access must be granted to BPGen. To this aim, the following steps must be followed:

1. A GitHub personal access token must be created (instructions can be found here)
2. File secrets.example inside BPGen must be renamed to secrets.json
3. File secrets.json must be opened inside an editor, and the following strings must be replaced with the following values:
   • GITHUB_TOKEN with the GitHub Token created in step 1
   • RE_USERNAME and RE_PASSWORD with the Repository Engine credentials provided

Repository initialization

To create a new repository hosting all the support files to create the abstract blueprint, the repo-init command must be passed to BPGen:

sh sdk-bp.sh repo-init VDC myrepo

Once executed, this command will create a new GitHub repository named myrepo inside the project workspace, and it will add the following structure:

• bp_gen_vdc.cfg – repository configuration file: determines which portions of the abstract blueprint will be generated from source files
• cookbook/cookbook.json – infrastructure definitions file: specifies the infrastructures made available by the application developer, which will be inserted in the COOKBOOK_APPENDIX section of the abstract blueprint.
• output/blueprint.json contains the abstract blueprint produced by BPGen.
• api – folder hosting the OpenAPI file describing the API to be exposed by the VDC, which will be inserted in the EXPOSED_API section of the abstract blueprint.
• api_data_samples – folder containing data utility attributes for each exposed method, which will be inserted in the DATA_MANAGEMENT section of the abstract blueprint.

Default data utility attributes generation

To automatically generate data utility attributes for each method exposed by the VDC, the OpenAPI file must be put inside the api folder. In addition, file bp_gen_vdc.cfg must be edited and the following properties must be set:

• api to api/swagger.yaml, where swagger.yaml is the OpenAPI file name
• data-management to api_data_samples, if not already set

Once done so, the std-metrics command must be passed to BPGen:

sh sdk-bp.sh std-metrics -l myrepo

where myrepo is the name of the repository. Once executed, this command will create inside the api_data_samples as many JSON files as the methods exposed by the VDC. The data administrator will then have to edit these files and either change the values for each data utility metric, or delete the unneeded metrics.

Abstract blueprint generation

To generate the abstract blueprint for the first time, all files in the repository must be firstly pushed to the GitHub project workspace. Once done so, the create command must be passed to BPGen:

sh sdk-bp.sh create -p repo_url

where repo_url is the HTTP URL of the repository (e.g., https://github.com/DITAS-Project/myvdc.git). Once executed, this command will automatically generate the abstract blueprint inside the output folder, starting from the source files created and added in the previous steps. Note that, for the generated blueprint to pass schema validation, the OpenAPI file must contain for each method the x-data-sources property. For further details on this property, please refer to the EXPOSED_API description.

Abstract blueprint update

To update a previously created abstract blueprint (e.g., when the contents of one of the source files are changed), the update command must be passed to BPGen:

sh sdk-bp.sh update -p repo_url

The behavior of this command is analoguous to the one of create. To prevent sections manually altered in the abstract blueprint blueprint.json file to be overwritten with auto-generated data, file bp_gen_vdc.cfg must be edited and the corresponding section removed (e.g., if the data administrator has edited the EXPOSED_API section of the generated blueprint, to prevent such modifications from being lost (s)he must remove the api attribute in bp_gen_vdc.cfg before updating the blueprint.

Abstract blueprint publication

To publish the abstract blueprint to the Repository Engine, the publish command must be passed to BPGen:

sh sdk-bp.sh publish path/to/blueprint.json -server re_url

where path/to/blueprint.json is the path to the generated blueprint file (typically tmp/myvdc.git/output/blueprint.json), and re_url is the URL of the Repository Engine.

Services for Internal Structure

Data Administrator is responsible to define the internal Structure section that describes VDC as a product. Look at OSR blueprint and IDEKO blueprint for examples of such blueprints.

• Filling the metadata information about the DAL image(s): In order to tell the Deployment Engine where to take and how to serve the DAL image, the data administrator has to fill the DAL_images section of the blueprint with the de. The template of this section is as follows:
  

  "DAL_Images": {	
    "[DAL-NAME]": {
      "images": {
        "default": {
          "internal_port":[INTERNAL_PORT_OF_CONTAINER],
          "external_port": [EXTERNAL_PORT_OF_CONTAINER],
          "image": "[IMAGE_NAME:TAG_NAME]”,
          “environment”: {
            [environment variables]
          }
        }
     }
  }

• Filling the metadata information about the data sources. The data sources have to be described as following:
  

                  "id": "[DATA-SOURCE-ID]",
                  "description": "[DESCRIPTION]",
                  "class": "relational database/object storage/streaming",
                  "type": "MySQL/Minio/InfluxDB",
                  "parameters": {
                      [Connection parameters of the specific data source]
                  },
                  "schema": {}
  

•  
  "policy-enforcement-engine": {
      "internal_port": [INTERNAL_PORT_OF_CONTAINER],
      "image": "[IMAGE_NAME:TAG_NAME]",
      "environment": {
            [environment variables]
       }
  }
  

Identity Management

For direct control of the Identity Management you can login to KeyCloakand largely follow this documentation at https://www.keycloak.org/docs/latest/server_admin/index.html. Alternatively the config client (https://github.com/DITAS-Project/KeycloakConfigClient) can be used. A password must be requested from the DITAS operator. The URI of the Identity Management services is part of the internal structure of the blueprint under the “Identity_Access_Management” section. Furthermore, predefined roles and users can be provided in this section after they have been defined in KeyCloak.

Services for Data Management

• Data Administrator is responsible to define the initial values of the Data Management section. Fill the QoS and QoD parameters of Data Management
  • Expose the DUE Service to get the initial values of the Data management section.

Services for Cookbook Appendix

KeyCloak Managment

he DITAs KeyCloak deployment provides a unique management API that can be used to create new users or groups for a VDC.

Abstract Blueprint Schema Documentation

In order to form the Blueprint that will reflect the structure and the function of a VDC, it is essential to create a format that will describe the body of this Blueprint, defining all the information and fields that it could possibly include. Based on this format, there are five core sections that a Blueprint contain: the Internal Structure, the Data Management, the Abstract Properties, the Cookbook Appendix and finally the Exposed API section. The complete Blueprint Schema can be found in Project Git Repository.

Internal Structure(Section 1)

The Internal Structure section records general information related to the VDC composition system, like a business description of the VDC, data sources that could be used, the category and scope of the VDC, etc.

Data Management(Section 2)

Data Management is a section dedicated to the VDC that plans on measuring the performance or monitoring the system for optimization and business purposes. Also, this section is responsible for enfolding information about data quality. Quality of service aspects along with data quality aspects constitutes the definition of data utility for the exposed data.

Abstract Properties(Section 3)

This section contains the goal model that specifies the non-functional application requirements that the blueprint is expected to fulfill once the concrete blueprint is instantiated. Such a goal model is used by the SLA Manager to detect violations, and by the DS4M to identify the best data and computation movement actions.This section remains empty in the abstract blueprint. Once VDC Blueprint resolution takes place, an intermediate blueprint is generated. In particular, the Data Utility Resolution Engine inserts in this section a subset of the goal model taken from the application designer requirements.

Cookbook Appendix(Section 4)

The Cookbook Appendix section contains all the necessary information for the orchestration and the deployment of each module/microservice (aka CookBook recipes). Also, it provides software configuration management, agnostic in nature, in order to support multiple tools such as Puppet and Chef.

Exposed API(Section 5)

This section of the abstract blueprint includes all the information about the methods, through which the administrator exposes totally or partially the data that are stored in the sources that he/she controls. The CAF RESTful API of the VDC is written according to the OpenAPI Specification (originally known as the Swagger specification) so that big vendors and also new providers are able to publish their services and components.