A Dependable Massive Storage Service for Medical Imaging

Introduction

The amount of digital information that the world is accumulating, as well as the increasing rates under which information has been produced, have become the driving forces that move the scientific community to find new solutions, in order to face the so-called “deluge” of information [1, 2]. As storage has become a strategic asset, we are witnessing the evolution of storage technologies. Not only scientific computing but also the government, banking, and any IT-based organization will be the beneficiaries of this shift in massive storage [3–5]. In the short term, IT managers will be required to make an important decision concerning the storage capacities of their corresponding organizations: whether storage is supported based on their own resources, or as a service that is provided by outsourcing [6–9].

We understand that people need to store their private information and, for this, they can resort to either a physical device, or a virtual device. This is, in the first case, they choose a product and, in the second case, they choose a service. In either of the two situations, the user is the owner of the information and can independently choose the solution to save his or her information.

As for the storage of organizations, we have that the users depend on the access policies fixed by their organization and may be limited in the related decisions with the use, storage, and transport of information. IT managers have, among other responsibilities, to implement primary and secondary storage solutions, to mitigate risks, foreseeing contingencies and even disasters. In this context, they have to decide again whether, to build these capabilities, they buy a product, or they hire a service. However, the answer is not as simple as in the case of personal storage, because this time, the volume plays a key role to consider.

Whether it is the storage of a person or an organization, the solution chosen in both cases must take into account that access to information must be done under the control of its owner or the authorized persons. However, due to the volume, it is usually easier to guarantee the confidentiality of the number of documents of a person, if we compare them with the quantity of documents that an organization can classify as sensitive. If the managers of an organization are interested in buying a product to solve their storage needs, they would be assuming the cost of their management in exchange for maintaining control of access to their information. If, on the other hand, they were inclined towards the purchase of a service, they would be deciding to release the cost of the management, but relinquishing control of the storage details.

In the first instance, it would seem that an organization should not relinquish control of the details related to the storage of its information; however, there are two requirements to consider. Normally, the requirement of availability of information increases with the size of the organization. Besides, information grows even at exponential rates, and that means that the management of storage devices can become a task that absorbs more and more resources of the organization itself. The cost-benefit analysis must have a medium- and long-term horizon where the availability of information and the scalability of its storage are key elements in the decision-making process [10–14]. This scenario, related to the storage of organizations, is the context of our work.

The Babel Storage System (after “The Library of Babel,” a short story by Jorge Luis Borges [15]) is a dependable, scalable, and flexible software-defined storage system, developed by INFOTEC. Among its main features, it can be underlined the availability of different types of data redundancy, a careful decoupling between data and metadata, a middleware that enforces metadata consistency, and its own load balance and allocation procedure which adapts to the number and capacities of the supporting storage devices. It can be deployed over different hardware platforms, i.e., fully hardware-agnostic. Enterprises and Service Providers are challenged to build scale-out storage infrastructures that support multiple application workloads and provide the highest data resiliency with the current technical limitations and costs of current traditional vendors. To stay flexible, efficient, and cost-effective, Babel provides different mechanisms for protecting data including replication and the information dispersal algorithm (IDA) [16]. Babel supports objects (HTTP/WebDAV/REST) [17, 18], easily integrating into many standard and custom applications.

Health institutions produce a huge number of digital images from clinical studies that may reach more than 100 thousand radiology procedures per year, which implies the preservation (also known as retention) of dozens of terabytes of uncompressed historical data. The minimum retention time of medical information depends on national regulations (for example: a study of x-rays in the USA has a retention period of 5 years from its generation [19], while in Australia, it is 7 years [20]). In Mexico, the institutions certified to provide health services must guarantee a retention period of 5 years [21] for all information concerning the health status of patients, the corresponding files weigh between 3.5 and 1500 MB and, according to national regulations, must be kept under very rigorous requirements of integrity, availability, and privacy.

We have developed an interface that connects Babel with the PACS (picture archiving and communication system) at the INR (Instituto Nacional de Rehabilitación) [22] which is a system to transmit, store, retrieve, and display medical imaging information, in strict conformance to DICOM (Digital Imaging and Communications in Medicine) [23] which is the international standard on this field. In this work, we present the design principles followed in the construction of this communication interface, we consider that the value of our proposal stands on the feasibility of an alternative approach, to meet the most stringent requirements on medical image storage.

The new trends in health sciences (such as genomics) will benefit from massive storage technology and indeed, our storage architecture may be exported to these scenarios too. Nevertheless, medical imaging will always be a key tool supporting health services. In this sense, we can grant that the PACSINR [22], being a system that is deployed and working in a National Institute, is fully complying with all the regulations required to provide support on such critical environments. Every day, at its most demanding moment, the PACS supports 300 concurrent users, including surgery operations, this shows that this is a highly dependable system. Medical image management procedures are continuously evolving to benefit not only from new image technologies, but also from the latest achievements on information management. This is the context that frames our contribution, we present a storage model that is not tied to any particular infrastructure and, as we already mentioned, can be exported to any situation where information is required to remain available over long terms, which implies the challenge of volume and scalability. We considered that the PACS at the INR has provided us with a very demanding testbed.

The rest of this paper includes the following: in “The Babel Storage System in a Nutshell,” we introduce the main aspects of Babel itself. In “The Implementation Details of the Communication Interface,” we show the main software elements and components than resolve the communications between a DICOM storage server application entity and the Babel Storage System. Finally, in the“Discussion” and “Conclusion” sections, we analyze the results and the directions for further work.

The Babel Storage System in a Nutshell

Storage systems can be classified in three main categories: file oriented, block oriented, and object oriented.

Files and blocks are based on the concept of file system. Most of the users are familiar with them, such as FAT, NFS, CIFS, and EXT. They organize data into files and folders in a tree-like hierarchy and give a path to the file while also retaining a small amount of metadata about this central entity. The key difference between file access (deployed on a NAS) and block access (deployed on a SAN) is that in the first case, the file system resides on the disk array. Meanwhile, in block access, the file system is external to the array and I/O calls are handled by the file system on the server, with only block-level information required to access data from the SAN. From this distinction can be explained the different scenarios where they are applied. NAS is best fitted to retention and access of entire files and has locking systems that prevent simultaneous changes and corruption to files. Meanwhile, SAN systems allow changes to blocks within entire files and so are extremely well suited to database and transactional processing.

In the other hand, as the name suggests, object-based storage allocates data in isolated containers known as objects. Each single object has a unique identifier and is stored in a flat memory model. This is important for two reasons: (i) an object can be rapidly retrieved by simply presenting its identifier (ID), thus making information much easier to find in a large pool of data. (ii) The data could be physically stored on a local server, or a remote server in the cloud. Object storage also lends much greater flexibility to metadata. Scalability is where object-based storage does its most impressive work. Scaling out an object architecture is as simple as adding additional nodes to the storage cluster.

As we have already pointed out, scalability has been a driving force in our design. Based on the fact that the DICOM standard enforces the compliance on the procedures, independently from technology, we consider that object storage (which is supported by Babel using REST-ful services) has been the right approach that will provide a long-lasting storage service.

We have built a distributed system that can be initially deployed on a small group of storage devices and offers a capacity of a few terabytes, but, if required, it may incorporate additional devices to achieve a massive scale, even of tens of petabytes (Fig. 1).

The key performance goals to consider in our construction are availability, fault-tolerance, and scalability. To meet these goals, we have considered two basic mechanisms: (a) two alternative information redundancy techniques and (b) a modular data placement algorithm. The former refers to the redundant information produced in order to attain availability and fault-tolerance. The latter refers to the (re)allocation of data within the available storage devices, in order to support changes and enforce load balance.

1. Redundant information can be obtained from simple replication that generates n copies of the original file. This is the principle followed by GFS [6] and HDFS [24]. Alternatively, redundancy can also be produced using error-correcting techniques such as the IDA [16, 25]. Traditional systems allocate several copies (replicas) of the object been uploaded, on the different devices that make up the cluster of disks. For instance, let us assume that a medical image of 1 GB is uploaded. Normally the system takes two additional copies, and finally, three instances are allocated on three different devices. The intention of data redundancy is to guarantee that even if various storage devices go out of service, with a good probability, there will be a device that remains available. Chances improve with a bigger number of copies of the same object. Nevertheless, the price to pay is the effective overall storage capacity. In our example, we say that each object occupies 300% of its original size, in storage capacity. In other words, it reduces the initial raw capacity to 1/3. In contrast, Babel uses a coding technique known as the IDA. Let O be an object to be stored, which occupies |O| bytes. Using IDA, O is transformed into n objects called dispersals, each of size |O|/m. O can be recovered provided that any m out of n dispersals remain available. In this case, the object occupies n/m × 100% of its original size and it is possible to tolerate the absence of any n-m storage devices. Compared with the initial example, we can transform the same object into n = 5 dispersals (for instance), each of size |O|/3; therefore, we produce an excess of information equal to (5/3) × 100% (vs 300% of the initial approach), and even if we lose any 2 (=5-3) of them (Fig. 2), it will be possible to recover the original object. Using IDA, the effective capacity is 3/5 of the raw capacity (vs 1/3 of the initial approach). Our current implementation of IDA achieves processing times that are similar to those observed in replica-based systems. Nevertheless, considering different scenarios, we decided to accommodate replication and IDA to offer alternatives to fit the requirements of different applications.

2. Data placement schemes (dps) must be designed considering the number of data units produced by the redundancy mechanism, herein referred to as “blocks.” Besides, any pair of blocks resulting from the same redundancy operation must be allocated on two different storage devices. Otherwise, the failure of this very device could compromise the availability of redundant information and, therefore, the recovery of the initial object.

   A data placement scheme is required to answer a very simple question: what is the storage device in charge of a given block? This question is issued at different moments during the block’s lifetime, (i) the first time it is uploaded, (ii) every time it is retrieved. Nevertheless, a distributed system is a dynamic entity whose components may change over time. Indeed, there is a third condition to invoke the dps, (iii) when a new storage device has been incorporated either to replace a faulty one, or to scale up the overall system capacity. On this last circumstance, the dps helps the system to recover its load balance. A dps builds a link between each block and the storage device where it is allocated. In our solution, this link is calculated each time that it is required, based on a fixed attribute of the block, that we call its signature (such as its name, or its date of creation). This attribute is introduced to a given hash function that returns a position on a virtual address space. In turn, the virtual space is partitioned into parcels, and each parcel is assigned to a different device.

One of the key design principles in object storage systems is the decoupling between data and metadata. The former are redundantly stored at the cluster of disks, while the latter correspond to a replicated database in charge of the system servers also known as proxies. This decision is the base on which scalability is achieved: data may be reallocated over time, but the information required to retrieve this data (i.e., the metadata) remains always at the same places. Notice that a redundant set of proxies prevents bottlenecks, but mainly introduce redundant functions that build fault-tolerance. Nevertheless, they also imply a potential inconsistency on the replicated information that each server has recorded. Also, as the load balance procedure is invoked, metadata may become out of date. The consistency of a replicated DB has been guaranteed by means of a middleware that enforces the updating of any DB replica, even if the corresponding server experiences temporal failures. This means that thanks to its redundant design, Babel is a dependable system that remains in full operation even when some of its components are degraded.

The Implementation Details of the Communication Interface

Discussion

Health providers increasingly rely on medical imaging instruments, such as ultrasound (US), magnetic resonance (MR), computed tomography (CT), positron emission tomography (PET), endoscopy (ES), and computed radiography (CR), to provide quality care to patients. Within this context, a PACS is normally deployed within the hospitals’ protected internal network. However, the lack of scalability and high-level disaster recovery provisions involve significant risks and costs for onsite systems. These systems usually generate a huge amount of data, which implies an increasing stress on the hospitals’ computing, storage, and network infrastructures. Although many concerns exist about security, privacy, and liability issues involving highly sensitive medical information, external storage may represent a promising approach that can be regarded as the first step towards a private cloud, where all stakeholders may profit from a shared infrastructure.

As online PACS storage server requires more storage space, efficient storage strategies are becoming a key component in healthcare informatics. However, these strategies can be costly to manage and complex to access. The PACSINR-Babel interface (Fig. 6) simplifies management procedures for dependable archiving (redundant, highly available, and secure) which, from the PACS server perspective, perceive Babel as a part of a local file system.

In turn, Babel is a large-scale, highly dependable, software-defined, storage system that can be considered as a LEGO-type family of solutions, this means that it is a hardware-agnostic system that allows IT managers the possibility to settle a trade-off between price and performance, depending on their particular priorities. It has been designed to address three basic non-functional requirements that provide the foundations of a long-lasting system: (1) reliability, (2) scalability, and (3) service times. In addition, it supports a standardized set of communications protocols which guarantees interoperability.

Within this context, this paper shows the coupling between the PACSINR (a system in production) and an instance of Babel, by means of a standard web service. This interface immediately extends the value of an onsite central storage to external locations. The interface caches the active datasets, which means that application entities have immediate access to this information. Additionally, management is based on a web centralized model, which ensures full data consistency and solves data integrity issues on site. In other words, the administrators have the ability to control all file access policies and hierarchies that allow the growth of a secure access model.

Conclusion

In this paper, we described the design of a communication interface between a PACS and a massive storage system. In short, the main contributions of this work are

• The representation of the proposal is done in the UML (Unified Modeling Language), which facilitates its implementation in any programming language.

• The structure of the model (Fig. 6), based on multi-tier layers, supports the development of independent entities each offering a well-defined service (JPAEntities, DAOInterfaces, DAOImplementation, CRUDRepository, RESTservices, BabelServices, etc.). This design principle, also known as “component loose coupling,” prevents the propagation of errors when layers are being updated.

• The system is extensible, offering an interface for building new services to implement new functions (model and views are decoupled).

• Our proposal, on one hand, tries to provide an alternative to PACS storage self-management by the definition of storage automated mechanisms between both systems and, on the other hand, to implement the solution which allows creating an auxiliary system that facilitates the migration processes (PACS updates or disaster recovery) of medical images, regardless of libraries or third-party systems.

From a broad perspective, we have developed a new model that can be employed to manage any type of cataloged collection. This design is mainly focused on high availability and scalability. We have considered that the amount of information to be preserved imposes a structure that divides the storage capabilities in two main categories: primary and secondary servers. We propose the usage of Babel as an alternative to secondary storage. In our model, each document that is received at the primary server is automatically backed up at Babel in a transparent way. Also, the catalog that describes the collection (logs and metadata from the PACS) is regularly backed up, enabling disaster recovery procedures. Should a document that has been eliminated from the primary server is required, it will be automatically recovered from Babel in a transparent way. We call this organization “the closed library model.” It is worth pointing that both storage subsystems may evolve independently from each other, without compromising availability, which offers IT managers a long-lasting profit from the involved infrastructure.

The future work is oriented along two main directions: (1) run stress tests to evaluate latency times to retrieve historical studies and compare them to a trading application such as EMC Centera. (2) Link of two health institutes, to share medical images based on Babel as a private storage cloud.