Building UI for DevOps operations with Cluster.dev and Streamlit
In the modern tech landscape, the “Platform Engineering” pattern emerges as a beacon of efficiency. Its primary goal? To streamline infrastructure operations, offering a product-like experience for Developers, QA professionals, and other engineers. With this approach, tasks such as launching environments, deploying services, conducting tests, and more become seamless.
Central to this transformation is the use of a unified CI/CD system. Now, most interactions can be executed by simply committing to Git repositories and initiating jobs. However, the traditional experience of using Git and managing tasks via the command line can feel somewhat lacking for end-users.
Enter a more intuitive approach: Imagine defining variables through drop-down menus, adjusting settings with sliders, and selecting multiple options with a simple click of the mouse. This not only enhances user experience but also aligns with the intuitive nature of modern interfaces.
Example Task
Let’s imagine we need to provide some interface that allows the product team to bootstrap EKS Kubernetes cluster in AWS Cloud with a set of preconfigured addons, domains, network policies, logging and monitoring and other. We need to cover everything as a code and allow users to change only required settings, like cluster size, node scaling, domain, project name in UI. It should be bootstrapped just with one click!
Selecting the UI Framework — Streamlit
There are plenty of options on how you can build a UI with a framework like Backstage or commercial platform providers, but from my standpoint it looks overwhelming. I’d like to find a solution that could be easily scripted, tested and could be launched standalone. From all options I have selected Streamlit.
Streamlit is an open-source Python library that allows developers to create web applications for data science and machine learning with minimal effort. It’s designed to help Python developers and data scientists turn data scripts into shareable web applications without requiring any knowledge of web development.
Key Features:
Python-centric UI Development: Streamlit is built for Python developers. You can create a full-fledged web application using just Python, without the need for HTML, CSS, or JavaScript.
Quick Launch: With a simple command (streamlit run your_script.py), your Streamlit app is up and running in your browser.
Hot-reloading: Streamlit apps automatically update as you modify your script, making it easy to see changes in real-time without manually refreshing the browser.
Quick Prototyping: Streamlit’s intuitive API and interactive widgets allow for rapid prototyping and experimentation. You can quickly iterate on your app design and functionality.
So it looks like the best candidate for creating DevOps UI.
Selecting Infrastructure Framework — Cluster.dev
DevOps tasks for infrastructure development can be succinctly described as “technology integrations.” This means we have a plethora of modules and libraries, and we need to integrate them to deliver high-level abstractions to users who prefer not to delve into the underlying tech.
Given that everything should be codified, we typically use popular technologies like Terraform, Helm, Kubernetes, and perhaps the most versatile tool — shell scripting :) To maximize efficiency, we won’t craft infrastructure code from scratch. Instead, we’ll leverage public open-source patterns for infrastructure (such as Terraform modules) and for addons and software deployment (like Helm charts).
Cluster.dev is a framework (and I’m one of its developers) that can seamlessly combine various high-level open-source technologies, like Terraform Modules and Helm Charts, into the StackTemplate.
The StackTemplate is a static object that can be maintained in a repository with its distinct lifecycle. This allows for easy testing, upgrading of components, and setting default values. For reference, check out this example StackTemplate relevant to our task: StackTemplate Example.
All that’s left is to specify the variables necessary for this StackTemplate to initialize the infrastructure. This can be achieved with a straightforward YAML file (referred to as the Stack object in Cluster.dev). Given its simplicity, it’s an excellent foundation for building a user interface on top.
Another cool point is that ChatGPT could easily create UI components with Streamit providing Python code for generating components like switches, sliders, multi-selects, etc…
I just dropped an example YAML — and It provides me with ready to use UI code!
Selecting Execution Environment — GitHub Actions
Now we have chosen the UI and Infrastructure frameworks. But we need some storage for configurations and executors that could execute the resulting code. Following the best GitOps practices we would rely on the CI/CD system and in our case I have chosen a GitHub Actions, since it widely operated, Workflow syntax could be easily generated and distributed.
User Scenario for Cluster Configuration and Deployment
Before we starting our project its a good practice to create a walkthrough the process and indicate the steps needs to be completed. Here was my list:
UI Selection: Users have the flexibility to customize their cluster through the UI. They can select essential parameters such as node size, region, domain, addons, and other specific requirements.
Repository & Credentials: Users specify their repository and provide an access token for it. Additionally, they need to furnish credentials for their AWS account.
Secure AWS Credentials: The system ensures the security of user data by storing AWS credentials in GitHub’s secret storage.
Dynamic YAML Generation: Based on the user’s input, the system dynamically generates the necessary YAML files. These, along with GitHub workflow files, are then pushed to the specified repository.
Initiate Pull Request: A Pull Request (PR) is automatically created. Following this, GitHub initiates a workflow that triggers the Plan step. This step involves creating an S3 bucket for Terraform (TF) and cluster.dev states. It also tests and provides an output preview of the planned cluster and associated resources set to be created.
Review & Merge: Users can review the details in the PR. Once they’re satisfied and merge it, GitHub executes the cdev apply. This command activates Terraform, Helm, and other units, bootstrapping the cluster into action.
Summary
At this part one we covered the task and list of technologies and steps to implement required UI and functionality. In the next chapters we can deep dive into technical implementation.
In meanwhile you can check the ready to use EKS Cluster creation demo.
And if you interested to dig into UI code, check it in template repository.
Thanks!
Volodymyr Tsap,
CTO Cluster.dev
