Cloud Storage Orchestration, the automation technology for agility and flexibility
- 작성자 Joo Jaemin
Automating technology for servers, networks, and storage is essential for creating a cloud system with agility and flexibility. Yet, storage automation is very challenging because each manufacturer controls it with different systems. For that reason, cloud service providers use storage from one manufacturer only for automation or develop their own storage automation technology. Let’s look at the Cloud Storage Orchestration technology developed by Samsung SDS for storage automation of the enterprise cloud service, Samsung Cloud Platform.
Why is cloud storage important?
Automating technology for servers, networks, and storage is essential for creating a cloud system with agility and flexibility.
In particular, storage is largely divided into block storage, file storage, and object storage according to data access methods. It is very challenging to automate storage because each vendor controls it with different systems. In general, cloud providers either designate and use one vendor storage exclusively or develop and use their own cloud storage based on their situations. Samsung SDS also developed its own technology called Storage Orchestration for storage automation of Samsung Cloud Platform.
What is storage orchestration?
Storage Orchestration is a technology that integrates, controls, and operates the storage of multiple vendors and device types, and it collects and analyzes storage data to provide the optimal storage volume to users.
In the past, we used to store data on individual servers or manage it using separate storage. Today, much more data can be stored and managed in the cloud. Therefore, the cloud system should be able to configure the system using multiple storage solutions rather than a single storage solution. Multiple storage products are provided by many vendors, such as NetApp, Dell EMC, and HPE, and there are various storage solutions available depending on the methods and specifications.
Limitations of using individual storage products
In each storage product, different storage technologies are embedded, and we can see how rapid technological changes replace old storage technologies and how new technologies appear. Accordingly, vendors are releasing various storage solutions using the latest technologies. Thus, cloud providers are committed to improving performance and reliability using the new technologies. However, it is required to understand the characteristics of the unique storage vendor system before applying it to the cloud no matter how advanced it is. Moreover, as engineers have to work on volume creation, modification, and deletion whenever it is needed, this may harm the agility of the cloud service. Now, let us suppose you created a cloud system with an automation solution provided by a vendor. In that case, you are likely to be locked into the vendor's products, and even if new storage solutions are provided at lower prices, you hardly give up the locked-in product, making it difficult to have competitive cloud system.
In the past, only a small amount of data could be stored on the server’s internal disk. But these days, large amounts of data can be stored in the server’s internal disk and many other storage platforms.
[Figure 1] Change of data storage method
The main characteristics of storage orchestration
Storage Orchestration is a solution that controls the storage of multiple vendors and device types upon a single request and enables storage control management automation by integrating with applications. In addition, it applies autonomous volume allocation technology to analyze storage data, select optimal storage, and create volumes to use multiple storage devices more efficiently, regardless of the storage vendors and device types. With the solution, you can also solve the aforementioned lock-in problems and create a more advanced cloud storage system. Moreover, storage orchestration allows not only the cloud, but any applications with the storage of many vendors and device types to configure a system, and a storage system can be easily configured.
The infographic shows the integration of applications (e.g., cloud system) and the storage orchestration system. Storage orchestration is a solution that controls a variety of storage solutions for many vendors with a single request.
[Figure 2] Storage Orchestration that connects applications and storage
[Wait!] What is autonomous volume allocation?
Autonomous volume allocation is a technology that allows a computer system to automatically recognize a storage device and allocate storage volume on demand.
This technology is generally used in a virtual environment and helps manage storage volume for virtual servers or virtual machines more efficiently.
For example, when a virtual machine is running, it can automatically detect additional storage devices and allocate them to the virtual machine. In addition, when the storage device lacks volume, the necessary volume is automatically allocated to the virtual machine for business continuity. Such technology automates storage management and minimizes manual intervention, which increases system stability and availability.
Storage orchestration container storage interface (CSI)
We developed Storage Orchestration CSI to make it easier to use storage in applications running in the Kubernetes environment. Kubernetes uses objects called persistent volume (PV) and storage class (SC) to create and manage storage volume. But you cannot create and change the volume in storage just by using PV and SC. Storage Orchestration CSI is a solution that enables integrated control over storage in the Kubernetes environment based on a standard library called container storage interface (CSI) provided by Kubernetes, which was made to address such issues. With Storage Orchestration CSI, you can create and change volumes in storage simply by creating PVs after SC configuration.
In addition, storage orchestration volume (SOV) was developed based on the Kubernetes custom resource definition (CRD), which makes storage volume management easier by registering and managing each volume's characteristics and unique information.
The infographic shows the structure of storage orchestration CSI to help you easily use the storage for applications running in the Kubernetes environment. Storage orchestration CSI is designed to enable integrated control over various storage options in the Kubernetes environment.
- Volume request for training data storage
High capacity SOV
- Volume request for general data sharing
- Volume request for checkpoint processing
Storage Orchestration CSI
- VOL: File Storage (NAS)
- VOL: Block Storage
- VOL: Object Storage
- Storage Orchestration
Autonomous volume allocation technology
Autonomous volume allocation uses a group called a storage service pool to classify storage by type (based on storage services such as performance, function, etc.). It analyzes metric data from each storage solution to determine which solution is the most affordable and underloaded to create volumes in the storage. This minimizes capacity and performance variations across storage, and allows the storage scaling by simply registering with a storage service pool. This function ultimately makes it easier to run multiple storage solutions more efficiently.
Technical elements of storage orchestration
The followings are the components of storage orchestration:
- · REST API: A system that provides integrated control of storage and storage services via the storage orchestration REST API
- · Storage orchestration CSI: A system that provides storage volumes to Pods through the storage orchestration volume (SOV) in a Kubernetes environment
- · Storage service management: A system that configures storage by type, such as gold, silver, or bronze, and provides storage volume according to the users’ needs
- · Autonomous volume allocation: A system that provides optimal storage volume to users based on the storage status analysis data using storage data
- · Storage data analytics: A system that collects data from each storage device and analyzes them to examine and monitor storage status and issues
- · Storage & volume management: A system for managing physical storage and its internal componen
- · Storage orchestration driver: A module that connects the physical storage of each vendor and storage orchestration
Systems using storage orchestration
Storage orchestration is a foundational technology for cloud services and applications that provides storage volume to many users. It is currently used for block storage and file storage services of Samsung Cloud Platform (SCP), which is the Samsung SDS cloud service for enterprise, storage part of Infra Manager, and the private/group volume service of x.Cloud(Kubernetes) in the R&D center. It can also be used in systems that use multiple storage devices or systems with storage vendor lock-in issues, such as HPCs that use a lot of petabyte-level storage solutions and cloud big data that requires massive data processing.
Cloud services, technologies that connect applications and storage
INTERNAL SERVICE ‐ SAMSUNG SDS
- INFRA MANAGER
- CLOUD VOLUME SERVICE ‐ R&D Cloud FOR AI Kubernetes
- REST API, STORAGE ORCHESTRATION CSI
- STORAGE SERVICE MANAGEMENT
- AUTONOMOUS VOLUME ALLOCATION
- STORAGE DATA ANALYTICS
- STORAGE & VOLUME MANAGEMENT
- STORAGE ORCHESTRATION DRIVER
- DELL EMC
[Figure 3] Storage Orchestration concept and technology elements
What differentiates Samsung SDS technology?
It is easy to install and use the storage orchestration technology developed by Samsung SDS. Since it was developed by Samsung SDS without relying on any other vendors, it works faster and offers various storage services by allocating the volume autonomously according to the storage data and optimizing backup.
Storage vendors previously tried to develop a solution for integrated storage control, but they failed to do so because they were only focused on the product of a leading vendor due to their interests in the process of designing and developing a solution. Some businesses have tried to develop similar solutions, but since their products were made based on storage vendor solutions, everything from installation to actual usage was too complicated. To resolve such problems, we came up with our own solution, storage orchestration. Using the solution, we were able to build a faster and more efficient cloud storage service, which has become the basis for providing seamless services to our clients.
Let's take a deeper look into the Samsung Cloud Platform where storage orchestration technology developed by our own technologies was applied for storage automation?
Go to Samsung Cloud Platform (SCP) >
- Professional, Joo Jaemin / Samsung SDS
- As a storage expert, Joo Jaemin has years of R&D experience and is currently focusing on cloud storage solution research, including Storage Orchestration, based on his extensive experience related to software-based storage.