Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Skip to content

DICE Project

Jump to bottom
Giuliano Casale edited this page Jun 4, 2021 · 1 revision

Table of Contents

Introduction
DICE IDE
DICE Models
DICE Tools

Introduction

DICE is a DevOps platform for Big Data application development created by the DICE consortium through a collaborative effort sponsored by Horizon 2020. A list of project results such as deliverables, datasets and papers is available on CORDIS.

A high-level overview of the DICE framework is shown in the figure below. This framework includes the following components:

  • DICE IDE: an integrated development environment to accelerate coding, design and application prototyping based on the DICE Profile for UML and the DICE Methodology;

  • Quality analysis tools: a set of tools for quality analysis during the early-stage of application design via simulation, verification and optimization methods;

  • Feedback and Iterative Enhancement tools: a monitoring platform tailored to Big Data technologies and coupled with tools that continuously detect quality anomalies that affect the application design and explain how the data-intensive application utilizes resources to enhance the design;

  • Continuous Delivery and Testing tools: a set of tools and methods supporting delivery on private and public clouds via a TOSCA-compliant deployment tool, optimal application configuration, continuous integration, quality testing, and fault injection.

DICE Framework

The above architectural components are further broken down in several tools, each one belonging to one of two categories:

  • Development tools, which are primarily centered on the development stage of the data-intensive application. The IDE implements the DICE quality-driven methodology that guides the developer through the different stage of refinement of design models up to implementing the initial prototypes of its application. The IDE supports multiple quality analyses offered by the verification, simulation and optimization tools.

  • Runtime tools, which collect data during the application quality testing to characterize the efficiency, reliability, and correctness of the components. This data is consolidated in the DICE monitoring platform and used to semi-automatically detect anomalies, optimize the system configuration, and enhance the design.The purpose of the tools within each group of tools is explained in details in the following subsections.

Development tools

The development tools are as follows:

The central element of the DICE architecture is the Integrated Development Environment (IDE),where the developer specifies the data-intensive application using a model-driven engineering approach. To support this activity, the DICE Eclipse-based IDE embeds the DICE UML profile which provides the stereotypes and tagged values needed for the specification of data-intensive applications in UML.

Following an Model-Driven Architecture (MDA) approach, models are defined by the user in a top-down fashion, stemming from platform-independent specifications of components and architecture (DICE Platform Independent Models), through assignment of specific technologies to implement such specifications (DICE Technology Specific Models), and finally to the mapping of the application components into a concrete TOSCA-compliant deployment specification (DICE Deployment Specific Models) which can be realised as elaborate UML Diagrams or simple and deployment-specific DDSM images within the DICER stand-alone deployment modelling tool. Such models can be related by DICE model-to-model transformations that are automatically executed within the IDE, to reduce the amount of manual work required from the user.

Throughout the application design, the DICE IDE offers the possibility to automatically translate certain DICE models into formal models for assessment of quality properties (efficiency, costs,safety/correctness, etc.). Each analysis requires to run dedicated tools that reside outside the IDE environment, in order to obtain prediction metrics. The simulation, optimization and verification plugins take care of translating models in-between IDE and these external tools. They also collect via REST APIs the outputs of these tools that are shown to the user inside the IDE. The interface of these plugins assumes the user to be unskilled in the usage of the formal models. Furthermore, the quality properties are defined in terms of constraints using appropriate language constructs.

As the developer progressively refines the application model and the application code, s/he is going to periodically commit them to a repository, i.e., a version control system (vcs). In DICE we ensure that every commit increases the version number of the application, which is a unique identifier used across tools to keep synchronized models and code. The repository acts as a shared source of models and code across different versions for all the DICE tools that need to access them. The repository resides externally to the IDE and will be accessed through appropriate tools (e.g. SVN, GIT, etc.).

Runtime tools

The runtime tools are as follows:

After the initial prototyping of the application, the developer can request to deploy the current prototype. After an automatic commit of all models and code to the external repository, the continuous deployment tool retrieves a copy of both of them from the repository, build the application, and internally store the outputs and their associated artifacts. The delivery tool then initializes the deployment environment (if not already created), consisting of VMs and software stack, and deploy (or update the existing deployment of) the application. The delivery operation also connects the DICE monitoring platform to the deployed application. The monitoring platform is started and stopped by REST APIs and acquires a pre-defined set of metrics that are continuously stored in a performance data repository.

The anomaly detection and trace checking tools also feature an IDE plugin and are able to query the monitoring platform for relevant metrics and use them to generate analyses concerning anomaly in performance, reliability or operational behavior of the application at a given release version. The anomaly detection tool reasons on the base of black-box and machine-learning models constructed from the monitoring data. Conversely, the trace checking tools are going to analyze the correctness of traces. These analyses are manually invoked by the user from the IDE plugin. Similar to these, the enhancement tool is able to automatically annotate the DICE models stored in the repository with statistics on the inferred and recorded monitoring data, in order to help the user to inspect the root-causes of performance or reliability anomalies.

The quality testing tools supports the generation of test workloads to the application. Such workloads are those that are used to assess the quality of prototypes. Similarly, the fault injection tool generates faults and malicious interferences that can help verifying the application resilience.Both tools integrate a heterogeneous set of actuators and can be run manually by an operator through a command-line interface and configuration files. They are also exploited by the configuration optimization tool to generate an experimental plan automatically given a time budget. The output of this tool is to confirm the optimal configuration of the deployment for an application in its final stages before being pushed to production. Compared to the optimization plugin, configuration optimization also deals with fine-grained systems parameters (e.g. buffer size, block size, JVM configuration,etc.), which are difficult to model in design-time exploration. Moreover, configuration optimization is black-box and solely measurement-driven, whereas design space exploration is primarily model-driven.

DICE IDE

Items Contents
Short Description The DICE IDE is an integrated environment for MDE where a designer can create models to describe data-intensive applications and their underpinning technology stack. The IDE integrates the execution of the different DICE tools, in order to minimize learning curves and simplify adoption. It is based on Eclipse Neon 4.6 which is a de-facto standard for the creation of software engineering models based on the MDE approach.
Documentation
Tutorial https://github.com/dice-project/DICE-Platform/blob/master/README.md
Video https://tinyurl.com/ya7fbrpw
Source code https://github.com/dice-project/DICE-Platform
Licence EPL
Contact
  • Marc Gil (mgil[at]prodevelop.es)
  • Christophe Joubert (cjoubert[at]prodevelop.es)

DICE Models

Items Contents
Short Description The DICE Models project is a basic offer of models to explore the core representational means and knowledge manipulation and analysis we are adopting in DICE. DICE models have been used as a foundation for the design of the DICE profile.
Documentation
Source code https://github.com/dice-project/DICE-Models
Licence EPL
Contact
  • Damian A. Tamburri ((damianandrew.tamburri[at]polimi.it))
  • Michele Guerriero (michele.guerriero[at]polimi.it)

DICE Tools

UML Profile

Items Contents
Short Description The DICE Profile is a UML Profile based on MARTE (Modeling and Analysis of Real-time and Embedded Systems) and DAM (Dependability Modeling and Analysis) profiles. The DICE Profile introduces concepts needed for the modeling of DIA at the different abstraction levels of DICE, i.e., DPIM, DTSM and DDSM.
Documentation D2.2 - Design and quality abstractions - Final version at http://wp.doc.ic.ac.uk/dice-h2020/wp-content/uploads/sites/75/2017/02/D2.2_Design-and-quality-abstractions-Final-version.pdf
DICE IDE Execution Tutorial https://zaguan.unizar.es/record/60722?ln=es
Video You can learn how to introduce DICE-Profile annotations, in a UML model, just watching the video for the Simulation tool, http://tiny.cc/z1qzay
Source code https://github.com/dice-project/DICE-Profiles
Licence EPL
Contact
  • José Merseguer (jmerse[at]unizar.es)
  • José-Ignacio Requeno (nrequeno[at]unizar.es)
  • Diego Pérez (diegop[at]unizar.es)

Deployment Modelling Tool

Items Contents
Short Description DICER is a tool developed in the context of the DICE H2020 European Project enabling the model-driven deployment of data-intensive applications (DIAs) leveraging the Infrastructure-as-Code (IaC) paradigm. More specifically, DICER adopt the OASIS Topology and Orchestration Specification for Cloud Applications (TOSCA) standard and is able to automatically generate IaC for DIAs in the form of TOSCA blueprints from stereotypes UML models. To this aim, DICER exploits the DICE-Profiles applied on UML Deployment Diagrams as well as the DIA-specific TOSCA library, both defined and developed in the scope of the DICE project.
Documentation http://wp.doc.ic.ac.uk/dice-h2020/wp-content/uploads/sites/75/2017/07/D2.4_Deployment-abstractions-Final-version.pdf
Stand-Alone Tutorial https://github.com/dice-project/DICER/wiki
Video http://tinyurl.com/ls999cn
Source code https://github.com/dice-project/DICER
Licence Dual licensing: BSD-3 + commercial
Contact

Delivery Tool

Items Contents
Short Description DICE delivery tools enable a simple way to continuously deploy data-intensive applications in a testbed. Starting up Big Data clusters for running applications such as Storm is a matter of executing a TOSCA YAML document. The tools consist of a deployment service, which is complete with a web user interface and command line interface, and a DICE technology library. They can be used for experimenting with various setups of Big Data technology without the need to spend effort on manually installing and configuring the cluster. We have also designed the tools to work well in a Continuous Integration workflow, where they assist with the quality assurance of the data-intensive applications being developed.
Documentation https://github.com/dice-project/DICE-Deployment-Service/blob/master/README.md
Stand-Alone Tutorial https://github.com/dice-project/DICE-Deployment-Service/blob/master/doc/UserGuide.md
DICE IDE Execution Tutorial https://github.com/dice-project/DICE-Deployment-IDE-Plugin/wiki/Getting-Started
Video https://www.youtube.com/watch?v=xtZnTwN1l4o
Source code https://github.com/dice-project/DICE-Deployment-Service
Licence Apache 2
Contact Matej Artac (matej.artac[at]xlab.si)

Verification Tool

Items Contents
Short Description DICE Verification Tool allows designers to evaluate their design against user-defined properties, in particular safety ones, such as reachability of undesired configurations of the system which might be caused by the effect of node failures or by the incorrect design of timing constraints. The verification process allows the DIA designer to perform verification tasks using a lightweight approach. DICE Verification Tool fosters an approach whereby formal verification is launched through interfaces that hide the complexity of the underlying models and engines. These interfaces allow the user to easily produce the formal model to be verified and the properties to be checked without the need of high technical expertise.
Documentation D3.7 - DICE verification tools - Final version, available at www.dice-h2020.eu/deliverables/. The document provides an architectural and behavioral description of the tool, shows the model-to-model transformations required to translate a description of a DIA into a model which undergoes verification and the class of properties to verify. Finally, it provides a use case to exemplify a usage scenario for the tool.
DICE IDE Execution Tutorial https://github.com/dice-project/DICE-Verification/wiki/Getting-Started
Stand-Alone Tutorial Same as for the DICE IDE Execution tutorial. Further installation steps are required.
Video https://www.youtube.com/watch?v=HHbFqXroiaY
Source code https://github.com/dice-project/DICE-Verification
Licence Apache 2
Contact
  • Francesco Marconi (francesco.marconi[at]polimi.it)
  • Marcello Maria Bersani (marcellomaria.bersani[at]polimi.it)
  • Matteo Rossi (matteo.rossi[at]polimi.it)

Simulation Tool

Items Contents
Short Description The Simulation Tool is a key component of the DICE Framework. The tool is able to simulate the behavior of a DIA to assess its performance and reliability using a Petri net model. The DIA can be defined both at DPIM level or at DTSM level using a particular technology (e.g., Storm, Spark). The DIA needs to be modelled using the DICE profile.
Documentation D3.4 - DICE simulation tools - Final version at http://wp.doc.ic.ac.uk/dice-h2020/wp-content/uploads/sites/75/2017/08/D3.4_DICE-simulation-tools-Final-version.pdf
DICE IDE Execution Tutorial https://github.com/dice-project/DICE-Simulation/wiki/Getting-Started
Stand-Alone Tutorial The Simulation tool can also be installed as stand-alone software within Eclipse. For installation see https://github.com/dice-project/DICE-Simulation/wiki/Installation
Video http://tiny.cc/z1qzay
Source code https://github.com/dice-project/DICE-Simulation
Licence EPL
Contact
  • José Merseguer (jmerse[at]unizar.es)
  • José-Ignacio Requeno (nrequeno[at]unizar.es)
  • Diego Pérez (diegop[at]unizar.es)

Optimization Tool

Items Contents
Short Description The optimization tool allows the DICE architect to assess the performance and minimize the deployment cost of data-intensive applications against user-defined properties, in particular meeting of deadlines under different levels of cluster congestion. The input is a DICE DDSM model, enriched with the SLAs to be achieved and a description of the execution environment (e.g., list of candidate providers, list of virtual machine types, etc.). The optimization consists in finding the less expensive cluster configuration able to guarantee the application jobs to be executed before a user defined deadline. The architect can analyze the application behavior under different conditions. For example, she can study pros and cons of public clouds versus private cloud in terms of execution costs.
Documentation D3.9 DICE Optimization Tools – Final Version
Stand-Alone Tutorial https://github.com/dice-project/DICE-Optimisation-Front-End/wiki
Video
Source code
Licence Apache 2
Contact
  • Danilo Ardagna (danilo.ardagna[at]polimi.it)
  • Eugenio Gianniti (eugenio.gianniti[at]polimi.it)

Monitoring Tool

Items Contents
Short Description DICE monitoring platform (DMon) collects, stores, indexes and visualizes monitoring data in real-time from applications running on Big Data frameworks. It supports DevOps professionals with iterative quality enhancements of the source code and with the optimisation of the deployment environment. Leveraging on Elastic.co’s open source technology stack, DMon is a fully distributed, highly available and horizontally scalable platform. All the core components of the platform have been wrapped in microservices accessible through HTTP RESTful APIs for an easy control. DMon is able to monitor both the infrastructure (memory, CPU, disk, network etc.) and multiple Big Data frameworks, currently supported being Apache HDFS, YARN, Spark, Storm and MongoDB. The core components of the platform (Elasticsearch, Logstash, Kibana) and the node components running on the monitored cluster are easily controlled thanks to a Web-based user interface that backs up the DMon controller RESTful service. Visualization of collected data is fully customizable and can be structured in multiple dashboards based on your needs, or tailored to specific roles in your organization, such as an administrator, quality assurance engineer or software architect.
Documentation
Stand-Alone Tutorial https://github.com/dice-project/DICE-Monitoring/wiki/Getting-Started
DICE IDE Execution Tutorial https://github.com/dice-project/DICE-Monitoring/wiki/DICE-IDE-Tutorial
Video https://drive.google.com/open?id=0B5xg7vkGTDqmTDY5Z25ubnhsTWM
Source code https://github.com/dice-project/DICE-Monitoring
Licence Apache 2
Contact
  • Gabriel Iuhasz (iuhasz.gabriel[at]e-uvt.ro)
  • Daniel Pop (daniel.pop[at]e-uvt.ro)

Anomaly Detection Tool

Items Contents
Short Description The anomaly detection tool will reason on the base of black box and machine learning models constructed from the monitoring data. In order to models able to detect not only point anomalies but also contextual anomalies, the tool will select a subset of data features from the Monitoring Platform to train and validate a predictive model, which is later stored in Monitoring Platform itself. The predictive models are then used to detect contextual anomalies in real-time monitoring data streams. A second use case supported by the anomaly detection tool is the analysis of monitoring data based on two different versions of DICE application, thus detecting anomalies introduced by latest code changes.
Documentation https://github.com/dice-project/DICE-Anomaly-Detection-Tool/wiki
Video https://www.youtube.com/watch?v=RnRxc09vphg
Source code https://github.com/dice-project/DICE-Anomaly-Detection-Tool
Licence Apache 2
Contact Gabriel Iuhasz (iuhasz.gabriel[at]e-uvt.ro)

Trace Checking Tool

Items Contents
Short Description Trace checking is an approach for the analysis of system executions, that are recorded as sequences of timestamped events, to establish whether a certain property (usually expressed in temporal logic) is satisfied. Trace checking tool is employed to extract information that is Coming Soon from standard monitoring platforms whereas this information is specific of the modeling used for verification in DICE. The collected data is used to assess the validity of the results obtained at design time with the verification tool.
Documentation D4.3 - Quality anomaly detection and trace checking tools - Final version
Stand-Alone Tutorial https://github.com/dice-project/DICE-Trace-Checking/wiki/Getting-Started
Video https://drive.google.com/open?id=0B8Llc1jTWIA7RDgwejY1UGt3UG8
Source code https://github.com/dice-project/DICE-Trace-Checking
Licence Apache 2
Contact
  • Marcello M. Bersani (marcellomaria.bersani[at]polimi.it)
  • Matteo Rossi (matteo.rossi[at]polimi.it)
  • Francesco Marconi (francesco.marconi[at]polimi.it)

Enhancement Tool

Items Contents
Short Description The Enhancement Tools are the main tool to close the loop between runtime (monitoring data) and design time (models and tools). They feature the correlation between monitoring data to find the existence of different abstractions between design concepts and runtime measurements. They also provide the initial parameters for the Simulation tools and optimization tools from monitoring data and detect the anti-patterns in the application design.
Documentation
Stand-Alone Tutorial
Video https://www.youtube.com/watch?v=Ii3V4zaIwe4
Source code
Licence BSD 3-clause
Contact
  • Chen Li (chen.li1[at]imperial.ac.uk)
  • Giuliano Casale (g.casale[at]imperial.ac.uk)

Quality Testing Tool

Items Contents
Short Description This is a suite of tools to help stress testing data-intensive applications based on technologies such as Hadoop/MapReduce, Storm and Cassandra. The idea is to put the developer in condition to run basic load tests on the application throughout the development cycle, in order to support the activities of configuration optimization and anomaly detection across software versions. An initial release of this tool is set for February 2017.
Documentation https://github.com/dice-project/DICE-Quality-Testing/wiki/
Stand-Alone Tutorial https://github.com/dice-project/DICE-Quality-TestingG/wiki/Getting-Started
Video https://www.youtube.com/watch?v=SfYfAgfDde8&feature=youtu.be
Source code https://github.com/dice-project/DICE-Quality-Testing
Licence BSD 3-clause
Contact Giuliano Casale (g.casale[at]imperial.ac.uk)

Configuration Optimization Tool

Items Contents
Short Description The Configuration optimization tool automatically tune the configuration parameters of data-intensive applications (DIA). DIAs (e.g., Oryx2) are developed with several technologies (e.g., Apache Storm, Hadoop, Spark, Cassandra) each of which has typically dozens of configurable parameters that should be carefully tuned in order to perform optimally. CO tool enables end users of such application to auto-tune their application in order to get the best performance. CO is already integrated with DICE delivery tools (including deployment service and continuous integration) as well as DICE monitoring platform.
Documentation https://github.com/dice-project/DICE-Configuration-BO4CO
Stand-Alone Tutorial https://github.com/dice-project/DICE-Configuration-BO4CO/wiki
Video https://www.youtube.com/watch?v=QGC5MGMEfYo
Source code https://github.com/dice-project/DICE-Configuration-BO4CO
Licence BSD 3-clause
Contact Giuliano Casale (g.casale[at]imperial.ac.uk)

Fault Injection Tool

Items Contents
Short Description The DICE Fault Injection Tool has been developed to generate faults within Virtual Machines and at the Cloud Provider Level. The FIT provides the ability for a user to generate faults at the VM level and an administrator at a cloud level. The purpose of the FIT is to allow cloud platform owners /Application VM owners a means to test the resiliency of a cloud installation as an application target. With this approach, the designers can use robust testing, showing where to harden the application before it reaches a commercial environment and allows a user/application owner to test and understand their application design/ deployment in the event of a cloud failure or outage. Thus allowing for the mitigation of risk in advance of a cloud-based deployment. This tool will assist developers and cloud operators in offering their best services to all customers and simplify testing within the DevOps paradigm.
Documentation https://github.com/dice-project/DICE-Fault-Injection-Tool/blob/master/README.md
Stand-Alone Tutorial https://github.com/dice-project/DICE-Fault-Injection-Tool/wiki/Getting-Started
Video https://www.youtube.com/watch?v=gqaO42sk-Z4
Source code https://github.com/dice-project/DICE-Fault-Injection-Tool
Licence Apache 2
Contact Craig Sheridan (csheridan@flexiant.com)
Clone this wiki locally