DZone Spotlight

At 3 a.m., the office is filled only with the dim glow of the computer screens. Data engineer Xiao Ming is struggling with two "heavyweights" — Doris and Hive. "Export, clean, import..." He mechanically repeats these steps between different components, his eyes starting to see stars. This scene is all too common in data teams, making one wonder: Do we really have to manually shuffle data for the rest of our lives? Just then, Doris extended an "olive branch" to Hive — the Hive Catalog made its dazzling debut! It's like arranging a perfect marriage for this "data couple," allowing Doris to directly read and write Hive data, enabling the two systems to "fly together." Whether it's HDFS or object storage, simple queries or complex analyses, one Catalog can handle it all! This amazing feature caught Xiao Ming's attention, and he could finally say goodbye to those cumbersome data synchronization tasks. Let's uncover this "life-saving tool" for data engineers together! The Perfect Encounter of Doris and Hive Late at night, Xiao Ming was staring at the screen, worried. As a data engineer, he faced a tricky problem: The company's data was scattered between Doris and Hive systems, and every cross-system data analysis required manual export and import, which was cumbersome and inefficient. "If only Doris could directly read and write Hive data..." he muttered to himself. Xiao Ming is not the only one with this concern. With the explosive growth of data, enterprise data architectures have become increasingly complex, with data storage scattered across various systems. How to connect these data silos and achieve unified data access and analysis has become a common technical pain point. The good news is that Apache Doris has perfectly solved this problem through the Hive Catalog feature. It's like building a bridge between Doris and Hive, enabling seamless collaboration between the two systems. Starting from version 2.1.3, through the Hive Catalog, Doris can not only query and import data from Hive but also perform operations such as creating tables and writing data back, truly realizing the architecture design of a unified lakehouse. The core value of Hive Catalog lies in its provision of a unified data access layer. For data developers, there is no need to worry about where the data is specifically stored; all data operations can be completed through Doris. For example, you can directly create a Hive table in Doris: SQL CREATE CATALOG hive PROPERTIES ( 'type'='hms', 'hive.metastore.uris' = 'thrift://172.21.16.47:7004' ); Once set up, you can operate Hive tables just like regular Doris tables. Not only does it support queries, but it also allows write operations such as INSERT and CREATE TABLE AS SELECT. The system automatically handles complex details such as partition management and file format conversion. Even more excitingly, Doris provides a comprehensive security mechanism. By integrating Kerberos authentication and Ranger permission management, enterprises do not have to worry about data security issues. Fine-grained access control down to the column level can be achieved to ensure compliance with data access. Now, Xiao Ming finally smiled. With Hive Catalog, his daily work efficiency has improved significantly. Cross-system data analysis has become so simple, as smooth as operating within the same system. This is just the beginning. In the following sections, we will explore more powerful features of Hive Catalog. Let's take a look at the new chapter of Doris + Hive data lakehouse integration! Core Features of Doris-Hive Catalog Xiao Ming recently faced a new challenge. The company's data analysis scenarios are becoming increasingly complex, with both traditional HDFS storage and the introduction of object storage. How can Doris elegantly handle these different storage media? Let's delve into the powerful features of Doris Hive Catalog in a simple and understandable way. Diverse Storage Support Each storage system has its own strengths. HDFS + Hive is suitable for large-scale offline processing of historical full data, while object storage offers high scalability and low-cost advantages... But, Hive Catalog provides a unified access interface, shielding the differences of the underlying storage: SQL -- Connect to S3 CREATE CATALOG hive_s3 PROPERTIES ( "type"="hms", "hive.metastore.uris" = "thrift://172.0.0.1:9083", "s3.endpoint" = "s3.us-east-1.amazonaws.com", "s3.region" = "us-east-1", "s3.access_key" = "ak", "s3.secret_key" = "sk", "use_path_style" = "true" ); -- Optional properties: -- s3.connection.maximum: Maximum number of S3 connections, default 50 -- s3.connection.request.timeout: S3 request timeout, default 3000ms -- s3.connection.timeout: S3 connection timeout, default 1000ms -- Connect to OSS CREATE CATALOG hive_oss PROPERTIES ( "type"="hms", "hive.metastore.uris" = "thrift://172.0.0.1:9083", "oss.endpoint" = "oss.oss-cn-beijing.aliyuncs.com", "oss.access_key" = "ak", "oss.secret_key" = "sk" ); Intelligent Metadata Management Doris employs an intelligent metadata caching mechanism to provide high-performance queries while ensuring data consistency: Local Cache Policy Doris caches table metadata locally to reduce the frequency of access to HMS. When the cache exceeds the threshold, it uses the LRU (Least-Recent-Used) strategy to remove some caches. Smart Refresh [Notification Event Diagram] By subscribing to HMS's Notification Event, Doris can promptly detect metadata changes. For example, you can set the Catalog's timed refresh when creating the Catalog: SQL CREATE CATALOG hive PROPERTIES ( 'type'='hms', 'hive.metastore.uris' = 'thrift://172.0.0.1:9083', 'metadata_refresh_interval_sec' = '3600' ); You can also manually refresh as needed: SQL -- Refresh the specified Catalog. REFRESH CATALOG ctl1 PROPERTIES("invalid_cache" = "true"); -- Refresh the specified Database. REFRESH DATABASE [ctl.]db1 PROPERTIES("invalid_cache" = "true"); -- Refresh the specified Table. REFRESH TABLE [ctl.][db.]tbl1; Enterprise-Level Security Features Security is always a top priority in enterprise data management. Hive Catalog also provides a complete security solution: Ranger Permission Control Apache Ranger is a security framework for monitoring, enabling services, and comprehensive data access management on the Hadoop platform. Doris supports using Apache Ranger for authorization for a specified External Hive Catalog. Currently, it supports Ranger's authorization for databases, tables, and columns, but does not support encryption, row-level permissions, Data Mask, and other functions. You only need to configure the FE environment and add it when creating the Catalog: SQL -- access_controller.properties.ranger.service.name refers to the type of service -- For example, hive, hdfs, etc. It is not the value of ranger.plugin.hive.service.name in the configuration file. "access_controller.properties.ranger.service.name" = "hive", "access_controller.class" = "org.apache.doris.catalog.authorizer.ranger.hive.RangerHiveAccessControllerFactory", Kerberos Authentication In addition to integrating with Ranger, Doris Hive Catalog also supports seamless integration with the existing Kerberos authentication system in enterprises. For example: SQL CREATE CATALOG hive_krb PROPERTIES ( 'type'='hms', 'hive.metastore.uris' = 'thrift://172.0.0.1:9083', 'hive.metastore.sasl.enabled' = 'true', 'hive.metastore.kerberos.principal' = 'your-hms-principal', 'hadoop.security.authentication' = 'kerberos', 'hadoop.kerberos.keytab' = '/your-keytab-filepath/your.keytab', 'hadoop.kerberos.principal' = 'your-principal@YOUR.COM', 'yarn.resourcemanager.principal' = 'your-rm-principal' ); Xiao Ming can now flexibly choose storage methods and security modes according to different business needs, truly achieving unified management and efficient analysis of Doris + Hive data. The boundaries between data lakes and data warehouses are blurring, and Doris has built a bridge connecting the two worlds through Hive Catalog. With the continuous evolution of technology, we look forward to seeing more innovative application scenarios.

In one of my earlier posts, we discussed how to best find memory leaks and the reasons behind them. It's best to use a focused and modern tool like HeapHero to detect OutOfMemory errors and many other performance bottlenecks, as it can pinpoint the real culprits and suggest ways to optimize the usage of computing resources. Above, you can see that there are a few thousand objects of byte[], String, int[], etc. Let's discuss some ways of fixing OutOfMemoryErrors in Java. You can see which fixes are applicable in your scenario/code and apply them to save memory and run your programs better. Some of the ways discussed below may seem trivial to you, but remember that a few small corrections may add up to a big gain. 1. Use ByteBuffer from java.nio Instead of allocating large byte arrays that may be underutilized, it allows direct memory allocation (ByteBuffer.allocateDirect(size)) to reduce GC pressure and avoid unnecessary heap allocations. If you are dealing with dynamically growing byte arrays, avoid starting with an unnecessarily large array. Java ByteBuffer buffer = ByteBuffer.allocateDirect(1024); // Allocates 1KB in off-heap memory For example, instead of: Java ByteArrayOutputStream baos = new ByteArrayOutputStream(10000);// Too large Let Java handle the resizing when needed because JVMs and JREs are continuously improved to consume minimal resources and manage their resource cycle well. Java ByteArrayOutputStream baos = new ByteArrayOutputStream(); // Starts small and grows as needed Or provide a small initial size if you know the minimum required length beforehand. Java ByteArrayOutputStream baos = new ByteArrayOutputStream(2048); // Starts small and grows as needed 2. Use Streams to Process Data in Chunks Instead of reading an entire file into memory using a huge byte array, For example, don’t use: Java byte[] data = Files.readAllBytes(Path.of("myLargeFile.txt")); // Loads entire file into memory Instead, try this: Java try (BufferedInputStream bis = new BufferedInputStream(new FileInputStream("myLargeFile.txt")); ByteArrayOutputStream baos = new ByteArrayOutputStream()) { byte[] buffer = new byte[2048]; // Read in smaller chunks int bytesRead; while ((bytesRead = bis.read(buffer)) != -1) { baos.write(buffer, 0, bytesRead); } byte[] data = baos.toByteArray(); } 3. Using the New MemorySegment Interface in Java 21 You can access off-heap or on-heap memory with the Foreign Function and Memory (FFM) API efficiently. It introduces the concept of an Arena. You use an Arena to allocate a memory segment and control the lifecycle of native memory segments. SegmentAllocator from Project Panama (Java 21) allows better control over memory allocation. Instead of large heap-based arrays, allocate memory using MemorySegment, which reduces garbage collection overhead. When you use the try-with-resources, the Arena will be closed as soon as the try block ends, all memory segments associated with its scope are invalidated, and the memory regions backing them are deallocated. For example: Java import java.lang.foreign.*; String s = "My LARGE ......... LARGE string"; try (Arena arena = Arena.ofConfined()) { // Allocate off-heap memory MemorySegment nativeText = arena.allocateUtf8String(s); // Access off-heap memory for (int i = 0; i < s.length(); i++ ) { System.out.print((char)nativeText.get(ValueLayout.JAVA_BYTE, i)); } } // Off-heap memory is deallocated 4. Use Singleton Objects Wherever Possible Some utility classes need not be instantiated per request; there can just be a single static instance for the whole application/session. For example, Unmarshallers and Marshallers. Unmarshallers are a part of JAXB specification. They are used to convert XML data into Java objects to its XML representations. Similarly, Marshallers are used to convert Java objects into XML representations. These help in processing XML data in Java programs by mapping XML elements and attributes to Java fields and properties, using Java annotations. If you look closely into the JAXBContext class, you will see that it has static methodscreateUnmarshaller() / createMarshaller() methods, which is a clear indication that these could be better handled as a single static instance for the whole application/session. 5. Use Singleton Scope In Your Spring-Based Applications This way, the container creates a single instance of that bean for the whole application to share, wherever possible, keeping your business logic intact. If coding a web application, remember that the @application scope creates the bean instance for the lifecycle of a ServletContext, the @request scope creates a bean instance for a single HTTP request, while the session scope creates a bean instance for a particular HTTP session. Java @Bean @Scope("singleton") public SomeService someService() { return new SomeService(); } 6. Use Faster and Memory-Efficient Alternatives to Popular Collections Use Collections.singletonMap and Collections.singletonList (for Small Collections) For example, if you only need a single key-value pair or item, avoid using a full HashMap or ArrayList, which have overhead. Use ArrayDeque Instead of LinkedList LinkedList has high memory overhead due to storing node pointers (next/prev references). Instead, use ArrayDeque, which is faster and memory-efficient. Java import java.util.ArrayDeque; ArrayDeque<Integer> deque = new ArrayDeque<Integer>(); deque.add(22); deque.removeFirst(); Use Map.of() and List.of() (Immutable Collections) If you don't need to modify a collection, use immutable collections, which are compact and optimized. Java Map<String, Integer> map = Map.of("A", 1, "B", 2); List<String> list = List.of("X", "Y", "Z"); Use WeakHashMap for Caching If you store temporary data in a HashMap, it may never get garbage collected, so use WeakHashMap, which automatically removes entries when keys are no longer referenced. 7. Close Objects as Soon as Their Utility Finishes Unclosed network sockets, I/O streams, database connections, and database/network objects keep using memory and CPU resources, adding to the running cost of the application. We have discussed some time-tested ways of dealing with OutOfMemory errors in Java. Your comments are welcome.

Forms are some of the easiest things to build in React, thanks to the useForm hook. For simple forms such as login, contact us, and newsletter signup forms, hard coding works just fine. But, when you have apps that require frequent updates to their forms, for example, surveys or product configuration tools, hard coding becomes cumbersome. The same goes for forms that require consistent validation or forms in apps that use micro frontends. For these types of forms, you need to build them dynamically. Fortunately, JSON and APIs provide a straightforward way to define and render these types of forms dynamically. In this guide, we’ll go over how you can use JSON and APIs (REST endpoints) to do this and how to set up a UI form as a service. Let’s start with creating dynamic forms based on JSON. Dynamic Forms in React Based on JSON What are Dynamic Forms in React? In React, dynamic forms based on JSON are forms where the structure (fields, labels, validation rules, etc.) is generated at runtime based on a JSON configuration. This means you don’t hard-code the form fields, labels, etc. Instead, you define all of this information in a JSON file and render your form based on the JSON file’s content. Here’s how this works: You start by defining your JSON schema. This will be your form’s blueprint. In this schema, you define the input field types (text, email, checkboxes, etc.), field labels and placeholders, whether the fields are required, and so on, like below: JSON { "title": "User Registration", "fields": [ { "name": "fullName", "label": "Full Name", "type": "text", "placeholder": "Enter your full name", "required": true }, { "name": "email", "label": "Email Address", "type": "email", "placeholder": "Enter your email", "required": true }, { "name": "gender", "label": "Gender", "type": "select", "options": ["Male", "Female", "Other"], "required": true }, { "name": "subscribe", "label": "Subscribe to Newsletter", "type": "checkbox", "required": false } ] } Create a form component (preferably in Typescript).Import your JSON schema into your component and map over it to create and render the form dynamically. Note: When looking into dynamic forms in React, you will likely come across them as forms where users can add or remove fields based on their needs. For example, if you’re collecting user phone numbers, they can choose to add alternative phone numbers or remove these fields entirely. This is a feature you can hard-code into your forms using the useFieldArray hook inside react-hook-form. But in our case, we refer to the dynamic forms whose renders are dictated by the data passed from JSON schema to the component. Why Do We Need Dynamic Forms? The need for dynamic forms stems from the shortcomings of static forms. These are the ones you hard-code, and if you need to change anything in the forms, you have to change the code. But dynamic forms are the exact opposite. Unlike static forms, dynamic forms are flexible, reusable, and easier to maintain. Let’s break these qualities down: Flexibility. Dynamic forms are easier to modify. Adding or removing fields is as easy as updating the JSON scheme. You don’t have to change the code responsible for your components.One form, many uses. One of React’s key benefits is how its components are reusable. With dynamic forms, you can take this further and have your forms reusable in the same way. You have one form component and reuse it for different use cases. For example, create one form but with a different schema for admins, employees, and customers on an e-commerce site. Custom, consistent validation. You also define the required fields, regex patterns (for example, if you want to validate email address formats), and so on in JSON. This ensures that all forms follow the same validation logic. These features make dynamic forms ideal for enterprise platforms where forms are complex and need constant updates. Why JSON for Dynamic Forms? JSON (short for Javascript Object Notation) is ideal for defining dynamic forms. Its readability, compatibility, and simplicity make it the best option to easily manipulate, store, and transmit dynamic forms in React. You can achieve seamless integration with APIs and various systems by representing form structures as JSON. With that in mind, we can now go over how to build dynamic forms in React with JSON. Building Dynamic Forms in React With JSON JSON Structure for Dynamic Forms The well-structured JSON schema is the key to a highly useful dynamic form. A typical JSON structure looks as follows: JSON { "title": "Registration", "fields": [ { "fieldType": "text", "label": "First Name", "name": "First_Name", "placeholder": "Enter your first name", "validationRules": { "required": true, "minLength": 3 } }, { "fieldType": "text", "label": "Last Name", "name": "Last_Name", "placeholder": "Enter your Last Name", "validationRules": { "required": true, "minLength": 3 } }, { "fieldType": "email", "label": "Email", "name": "email", "placeholder": "Enter your email", "validationRules": { "required": true, "pattern": "^[a-zA-Z0-9+_.-]+@[a-zA-Z0-9.-]+$" } }, { "fieldType": "text", "label": "Username", "name": "username", "placeholder": "Enter your username", "validationRules": { "required": true, "minLength": 3 } }, { "fieldType": "select", "label": "User Role", "name": "role", "options": ["User", "Admin"], "validationRules": { "required": true } } ], "_comment": "Add more fields here." } Save the above code as formSchema.JSON. Now that we have the JSON schema, it's time to implement and integrate it into the React form. Implementing JSON Schema in React Dynamic Forms Here is a comprehensive guide for implementing dynamic forms in React. Step 1: Create React Project Run the following script to create a React project: Plain Text npx create-react-app dynamic-form-app cd dynamic-form-app After creating your React app, start by installing the React Hook Form this way: Plain Text npm install react-hook-form Then, destructure the useForm custom hook from it at the top. This will help you to manage the form’s state. Step 2: Render the Form Dynamically Create a React Dynamic Forms component and map it through the JSON schema by importing it. JavaScript import React from 'react'; import { useForm } from 'react-hook-form'; import formSchema from './formSchema.json'; const DynamicForm = () => { const { register, handleSubmit, formState: { errors }, } = useForm(); const onSubmit = (data) => { console.log('Form Data:', data); }; const renderField = (field) => { const { fieldType, label, name, placeholder, options, validationRules } = field; switch (fieldType) { case 'text': case 'email': return ( <div key={name} className="form-group"> <label>{label}</label> <input type={fieldType} name={name} placeholder={placeholder} {...register(name, validationRules)} className="form-control" /> {errors[name] && ( <p className="error">{errors[name].message}</p> )} </div> ); case 'select': return ( <div key={name} className="form-group"> <label>{label}</label> <select name={name} {...register(name, validationRules)} className="form-control" > <option value="">Select...</option> {options.map((option) => ( <option key={option} value={option}> {option} </option> ))} </select> {errors[name] && ( <p className="error">{errors[name].message}</p> )} </div> ); default: return null; } }; return ( <form onSubmit={handleSubmit(onSubmit)} className="dynamic-form"> <h2>{formSchema.title}</h2> {formSchema.fields.map((field) => renderField(field))} <button type="submit" className="btn btn-primary"> Submit </button> </form> ); }; export default DynamicForm; Please note that you must handle different input types in dynamic forms with individual cases. Each case handles a different data type: JavaScript const renderField = (field) => { switch (field.type) { case 'text': case 'email': case 'password': // ... other cases ... break; default: return <div>Unsupported field type</div>; } }; Step 3: Submit the Form When the form is submitted, the handleSubmit function processes the data and sends it to the API and the state management system. JavaScript const onSubmit = (data) => { // Process form data console.log('Form Data:', data); // Example: Send to API // axios.post('/api/register', data) // .then(response => { // // Handle success // }) // .catch(error => { // // Handle error // }); }; So that’s how you can create dynamic forms using JSON to use in your React app. Remember that you can integrate this form component in different pages or different sections of a page in your app. But, what if you wanted to take this further? By this, we mean having a dynamic form that you can reuse across different React apps. For this, you’ll need to set up a UI form as a service. Setting Up Your Dynamic Form as a UI Form as a Service First things first, what is a UI form as a service? This is a solution that allows you to render dynamic forms by fetching the form definition from a backend service. It is similar to what we’ve done previously. Only here, you don’t write the JSON schema yourself — this is provided by a backend service. This way, anytime you want to render a dynamic form, you just call a REST endpoint, which returns the UI form component ready to render. How This Works If you want to fetch a REST API and dynamically render a form, here’s how you can structure your project: Set up a backend service that provides the JSON schema.The frontend fetches the JSON schema by calling the API.Your component creates a micro frontend to render the dynamic form. It maps over the schema to create the form fields.React hook form handles state and validation. Step 1: Set Up a Back-End Service That Provides JSON Schema There are two ways to do this, depending on how much control you want: You can build your own API using Node.j, Django, or Laravel. Here’s an example of what this might look like with Node.js and Express backend. JavaScript const express = require("express"); const cors = require("cors"); const app = express(); app.use(cors()); // Enable CORS for frontend requests // API endpoint that serves a form schema app.get("/api/form", (req, res) => { res.json({ title: "User Registration", fields: [ { name: "username", label: "Username", type: "text", required: true }, { name: "email", label: "Email", type: "email", required: true }, { name: "password", label: "Password", type: "password", required: true, minLength: 8 }, { name: "age", label: "Age", type: "number", required: false }, { name: "gender", label: "Gender", type: "select", options: ["Male", "Female", "Other"], required: true } ] }); }); app.listen(5000, () => console.log("Server running on port 5000")); To run this, you’ll save it as sever.js, install dependencies (express CORS), and finally run node server.js. Now, your react frontend can call http://localhost:5000/api/form to get the form schema. If you don’t want to build your own backend, you can use a database service, such as Firebase Firestore, that provides APIs for structured JSON responses. If you just want to test this process you can use mock APIs from JSON Placeholder. This is a great example of an API you can use: https://jsonplaceholder.typicode.com/users. Step 2: Create Your Dynamic Form Component You’ll create a typical React component in your project. Ensure to destructure the useEffect and useForm hooks to help in handling side effects and the form’s state, respectively. JavaScript import React, { useState, useEffect } from "react"; import { useForm } from "react-hook-form"; const DynamicForm = ({ apiUrl }) => { const [formSchema, setFormSchema] = useState(null); const { register, handleSubmit, formState: { errors } } = useForm(); // Fetch form schema from API useEffect(() => { fetch(apiUrl) .then((response) => response.json()) .then((data) => setFormSchema(data)) .catch((error) => console.error("Error fetching form schema:", error)); }, [apiUrl]); const onSubmit = (data) => { console.log("Submitted Data:", data); }; if (!formSchema) return <p>Loading form...</p>; return ( <form onSubmit={handleSubmit(onSubmit)}> <h2>{formSchema.title}</h2> {formSchema.fields.map((field) => ( <div key={field.name}> <label>{field.label}:</label> {field.type === "select" ? ( <select {...register(field.name, { required: field.required })} > <option value="">Select</option> {field.options.map((option) => ( <option key={option} value={option}> {option} </option> ))} </select> ) : ( <input type={field.type} {...register(field.name, { required: field.required, minLength: field.minLength })} /> )} {errors[field.name] && <p>{field.label} is required</p>} </div> ))} <button type="submit">Submit</button> </form> ); }; export default DynamicForm; This form will fetch the schema from the API and generate fields dynamically based on it. React hook form will handle state management and validation. Step 3: Use the Form Component in Your App This step is quite easy. All you have to do is pass the API endpoint URL as a prop to the dynamic form component. JavaScript import React from "react"; import DynamicForm from "./DynamicForm"; const App = () => { return ( <div> <h1>Form as a Service</h1> <DynamicForm apiUrl="https://example.com/api/form" /> </div> ); }; export default App; React will create a micro-frontend and render the form on the frontend. Why Would You Want to Use This? As mentioned earlier, a UI form as a service is reusable, not only across different pages/page sections of your app, but also across different apps. You can pass the REST endpoint URL as a prop in a component of another app. What’s more, it keeps your application lean. You manage your forms centrally, away from your main application. This can have some significant performance advantages. Advantages and Limitations of Dynamic Forms Advantages Reduced redundant code enables developers to manage and handle complex forms conveniently.Dynamic forms are easier to update, as changing the JSON schema automatically updates the form.JSON schemas can be reused across different parts of the application. You can take this further with a UI form as a service that is reusable across different applications.Dynamic forms can handle the increased complexity as the application scales. Limitations Writing validation rules for multiple fields and external data can be cumbersome. Also, if you want more control with a UI form as a service, you’ll need to set up a custom backend, which in itself is quite complex.Large or highly dynamic forms affect the performance of the application. With the first method where you’re creating your own JSON file, you still have to write a lot of code for each form field.Finding and resolving bugs and errors in dynamically generated forms can be challenging. Bonus: Best Practices for Dynamic Forms in React On their own, dynamic forms offer many advantages. But to get the best out of them, you’ll need to implement the following best practices. Modular Programming Divide the rendering logic into modules for better navigation and enhanced reusability. This also helps reduce the code complexity. This is something you easily achieve with a UI form as a service. It decouples the form’s logic from your application logic. In the event that one of the two breaks down, the other won’t be affected. Use the Validation Library It is best to use a validation library to streamline the process for complex validation rules. This will abstract you from writing validation rules for every possible scenario you can think of. Extensive Testing Test your dynamic forms extensively to cover all possible user inputs and scenarios. Include various field types, validation rules, and submission behaviors to avoid unexpected issues. Performance Optimization As mentioned earlier, the increased dynamicity affects the application's performance. Therefore, it is crucial that you optimize the performance by implementing components like memoization, lazy loading, and minimizing the re-renders. Define Clear and Consistent JSON Schemas Stick to a standard structure for defining all the JSON schemas to ensure consistency and enhance maintainability. Moreover, clear documentation and schema validation can also help prevent unexpected errors and faults. Furthermore, it aids team collaboration. With these best practices, you can achieve highly robust, efficient, and maintainable dynamic forms in React with JSON. Conclusion Dynamic forms in React based on JSON serve as a powerful tool for designing flexible user interfaces. By defining the form structure in JSON schemas, you can streamline form creation and submission dynamically. Moreover, this helps enhance the maintainability and adaptability of the application. Although this process has a few limitations, the benefits heavily outweigh them. In addition, you can work around some of the limitations by using the UI form as a service. This solution allows you to manage your dynamic forms independently of your application. Because of this, you can reuse these forms across multiple apps. With JSON-based dynamic forms, you can achieve seamless integration with APIs and ensure consistency throughout the project.

Why Would One Even Construct an AI Meme Generator? Memes are literally the one thing on the internet that anyone can understand. Whether you want to take a jab at your friend or want to show how coding sometimes gives you brain freezes, memes will always come to your rescue. The issue? Manually doing everything takes ages. You need to source the right picture, come up with snarky lines, and then figure out how to stick everything together without making it look like something a 5-year-old put together. But now, there are tools such as OpenAI and DeepSeek. With these, you don’t just automate comedy; you also automate currently trending formats and allow users to create memes in a matter of seconds. Here is how we approached our tasks: To generate engaging captions from memes, we created a context-specific approach.We built a super simple and straightforward drag-and-drop design interface. Finding new ways to economize API expenses allowed us to stay within the budget.Allowing users to store their most liked memes and adding a text-to-image meme feature made it possible. Looking Back at My Favorite Tools Before diving into the nitty-gritty details of the code, let’s discuss the tech stack a bit. And, just as a side note, assuming that you’ll be constructing a house without knowing what tools you’ll use is impractical. React + TypeScript. For the smooth, responsive UI React had brought into the world, TypeScript enabled the Team to catch many bugs that would have previously occurred.OpenAI/DeepSeek APIs. As long as there was a budget, the rest was history, with Division-04 being capable of delivering incisive, funny captions at will using GPT-4 Turbo. When they were limited, DeepSeek saved the day.Fabric.js. This library helps to have images with text dragged around easily, rather than feeling like one is trying to wrestle with a piglet drenched in oil.Vercel. Deployment utopia. It was also great during peak times because edge caching softened the blow.Redis. Low barrier to entry for protecting against rate limit and API abuse enforcement. Step 1: Set Up Your Own AI Brain What is clear is that an AI copying phrases from the internet will not work for memes like an AI telling you the response is “that’s hilarious” can. Memes require an amalgam of attitude, phrasing, and some level of restraint. This brings us to the more fundamental problem of how you tell an AI to make jokes. Tweaking the prompts of the AI itself, of course. Here’s a snip of the code used to create the captions: JavaScript // src/services/aiService.ts type MemePrompt = { template: string; // e.g., "Distracted Soul" context: string; // e.g., "When your code works on the first try" }; const generateMemeCaption = async ({ template, context }: MemePrompt) => { const prompt = ` Generate a sarcastic meme caption for the "${template}" template about "${context}". Rules: - Use Gen-Z slang (e.g., "rizz", "sigma") - Max 12 words - Add emojis related to the context `; const response = await openai.chat.completions.create({ model: "gpt-4-turbo", messages: [{ role: "user", content: prompt }], temperature: 0.9, // Higher = riskier jokes max_tokens: 50, }); return stripEmojis(response.choices[0].message.content); // No NSFW stuff allowed }; Pro tip: For humor, keep it around 0.7 to 0.9, but make sure to always moderate the response through OpenAI’s moderation endpoint for safety reasons. Step 2: Constructing the Meme Canvas If you attempted to deal with the HTML5 Canvas APIs, you understand how not straightforward they are to deal with. Luckily, Fabric.js came to the savior. It gave us Photoshop-like controls directly inside React with the added bonus of drag-and-drop. Take a look at this simplified version of our canvas component: TypeScript // src/components/MemeCanvas.tsx import { FabricJSCanvas, useFabricJSEditor } from "fabricjs-react"; export default function MemeCanvas() { const { editor, onReady } = useFabricJSEditor(); const [textColor, setTextColor] = useState("#FFFFFF"); const addTextLayer = (text: string) => { editor?.addText(text, { fill: textColor, fontFamily: "Impact", fontSize: 40, stroke: "#000000", strokeWidth: 2, shadow: "rgba(0,0,0,0.5) 2px 2px 2px", }); }; return ( <> <button onClick={() => addTextLayer("Why React, why?!")}>Add Default Text</button> <input type="color" onChange={(e) => setTextColor(e.target.value)} /> <FabricJSCanvas className="canvas" onReady={onReady} /> </> ); } Perks of this: Frees the text layers to be dragged anywhere on the document.Add color with stroke and shadow using the advanced color picker.Double-click to edit text to streamline the editing process. Step 3: Rate Limitations Imagine this for a moment: You release your app, and all of a sudden, everybody wants to make memes. Sounds fun, right? Until the OpenAI bill shoots up more than the price of Bitcoin. To address this, we put in place sliding window rate limiting with Redis. This is how we did it on Vercel Edge Functions: JavaScript // src/app/api/generate-caption/route.ts import { Ratelimit } from "@upstash/ratelimit"; import { Redis } from "@upstash/redis"; const ratelimit = new Ratelimit({ redis: Redis.fromEnv(), limiter: Ratelimit.slidingWindow(15, "86400s"), // 15 requests/day per IP }); export async function POST(request: Request) { const ip = request.headers.get("x-forwarded-for") ?? "127.0.0.1"; const { success } = await ratelimit.limit(ip); if (!success) { return new Response("Slow down, meme lord! Daily limit reached.", { status: 429, }); } // Proceed with OpenAI call } Hack to Save Costs Cache popular prompts such as "Hotline Bling" and "When Pull Request gets approved."Use CloudFlare to cache generated images. AI-Generated Meme Images From DALL-E 3 Sometimes, we learn the hard way that selecting the perfect meme template is an impossible task. JavaScript // src/services/aiService.ts const generateCustomMemeImage = async (prompt: string) => { const response = await openai.images.generate({ model: "dall-e-3", prompt: ` A meme template about "${prompt}". Style: Flat vector, bold outlines, no text. Background: Solid pastel color. `, size: "1024x1024", quality: "hd", }); return response.data[0].url; }; Changing the Output Prompt: "Two developers in a dispute over Redux and Zustand frameworks."Final product: An argument between Redux and Zustand cartoon characters is illustrated by two metabolizing icons fighting against a purple background. Meme History Feature (Zustad + LocalStorage) In order to enable users to keep collages, we added a meme history feature with the help of Zustand. JavaScript // src/stores/memeHistory.ts import { create } from "zustand"; import { persist } from "zustand/middleware"; type Meme = { id: string; imageUrl: string; caption: string; timestamp: number; }; interface MemeHistoryState { memes: Meme[]; saveMeme: (meme: Omit<Meme, "id" | "timestamp">) => void; } export const useMemeHistory = create<MemeHistoryState>()( persist( (set, get) => ({ memes: [], saveMeme: (meme) => { const newMeme = { ...meme, id: crypto.randomUUID(), timestamp: Date.now(), }; set({ memes: [newMeme, ...get().memes].slice(0, 100) }); }, }), { name: "meme-history" } ) ); User Occupational Stream Let’s create a meme, and then we can click on save.The meme will be kept locally and will be presented in a grid format.The meme that has been saved can be reloaded in the editor by clicking on it. Closing Thoughts Building an AI meme generator helped deepen my understanding, not just of coding, but of how to handle certain unexpected scenarios. I learned the hard way the importance of preparation from implementing harsh limit rates to enduring Reddit traffic surges. So, give it a try, work from the bottom up while making changes based on the feedback received, and enjoy yourself in the process. Perhaps your app might become popular, making you the next meme millionaire.

Database consistency is a fundamental property that ensures data remains accurate, valid, and reliable across transactions. In traditional databases, consistency is often associated with the ACID (atomicity, consistency, isolation, durability) properties, which guarantee that transactions transition the database from one valid state to another. However, in distributed databases, consistency takes on a broader meaning, balancing trade-offs with availability and partition tolerance, as described in the CAP theorem. With the rise of cloud computing, global-scale applications, and distributed architectures, database consistency models have become critical for ensuring seamless and reliable data operations. This article explores different types of database consistency models, their trade-offs, and their relevance in modern distributed systems. Quick Recap of CAP Theorem The CAP theorem states that in a distributed system, it is impossible to achieve all three properties simultaneously: Consistency (C). Every read receives the latest write or an error. This means that all nodes in the system see the same data at the same time.Availability (A). Every request receives a response, even if some nodes are down. The system remains operational.Partition tolerance (P). The system continues to function despite network partitions (i.e., communication failures between nodes). In practice: CP systems (consistency + partition tolerance). Prioritize consistency over availability. During a network partition, some requests may be blocked to ensure all nodes have up-to-date data. For example, Google Spanner, Zookeeper, and RDBMS-based systems.AP systems (availability + partition tolerance). Prioritize availability over consistency. The system responds to requests even if some nodes return outdated data. For example, DynamoDB, Cassandra, S3, CouchDB.CA systems (consistency + availability). CA systems are not possible in distributed systems because network failures will eventually occur, requiring partition tolerance. It's only possible in non-distributed, single-node systems. Database Consistency Different distributed databases achieve consistency through either CP or AP systems, commonly referred to as strong consistency and eventual consistency, respectively. Several consistency models fall within these categories, each with different guarantees and trade-offs. 1. Strong Consistency Strong consistency ensures that all replicas of the database reflect the latest updates immediately after a transaction is committed. This guarantees that every read operation retrieves the most recent write, providing a linear and predictable experience for users. Usage These systems are used in scenarios where maintaining a single, agreed-upon state across distributed nodes is critical. Leader election. Ensures a single active leader in distributed systems (e.g., Kafka, ZooKeeper).Configuration management. Synchronizes configs across nodes (e.g., ZooKeeper, etcd). Distributed locks. Prevents race conditions, ensuring exclusive access (e.g., ZooKeeper, Chubby). Metadata management. Maintains consistent file system metadata (e.g., HDFS NameNode, Chubby). Service discovery. Tracks live services and their locations (e.g., Consul, etcd). Transaction coordination. Ensures ACID transactions across distributed nodes (e.g., Spanner, CockroachDB). Trade-Offs Ensures correctness but increases latency and reduces availability during network failures.Difficult to scale in highly distributed environments.Can require complex distributed consensus protocols like Paxos or Raft, which can slow down system performance. 2. Eventual Consistency Eventual consistency allows data to be temporarily inconsistent across different replicas but guarantees that all replicas will converge to the same state over time, given that no new updates occur. This model prioritizes availability and partition tolerance over immediate consistency. Usage Eventual consistency databases (AP systems in CAP theorem) are used where availability is prioritized over strict consistency. These databases allow temporary inconsistencies but ensure data eventually synchronizes across nodes. Global-scale applications. Replicated across multiple regions for low-latency access (e.g., DynamoDB, Cosmos DB). Social media feeds. Updates can be slightly delayed but must remain highly available (e.g., Cassandra, Riak). E-commerce shopping carts. Allow users to add items even if some nodes are temporarily inconsistent (e.g., DynamoDB, CouchDB). Content delivery networks (CDNs). Serve cached content quickly, even if the latest version isn’t immediately available (e.g., Akamai, Cloudflare). Messaging and notification systems. Ensure messages are eventually delivered without blocking (e.g., RabbitMQ, Kafka). Distributed caches. Store frequently accessed data with eventual sync (e.g., Redis in AP mode, Memcached). IoT and sensor networks. Handle high write throughput and sync data over time (e.g., Apache Cassandra, InfluxDB). Trade-Offs Provides low latency and high availability but may serve stale data.Requires conflict resolution mechanisms to handle inconsistencies.Some systems implement tunable consistency, allowing applications to choose between strong and eventual consistency dynamically. 3. Causal Consistency Causal consistency ensures that operations that have a cause-and-effect relationship appear in the same order for all clients. However, independent operations may be seen in different orders. Usage If Alice posts a comment on Bob’s post, all users should see Bob’s post before Alice’s comment.Facebook’s TAO (graph database) maintains causal consistency for social interactions.Collaborative editing platforms like Google Docs may rely on causal consistency to ensure edits appear in the correct order.Cassandra (with lightweight transactions - LWTs) uses causal consistency with timestamps in some configurations to ensure operations dependent on each other are ordered correctly.Riak (with causal contexts) uses vector clocks to track causal dependencies and resolve conflicts. Trade-Offs Weaker than strong consistency but avoids anomalies in causally related events.Can be challenging to implement in systems with high user concurrency. 4. Monotonic Consistency Monotonic reads. Ensures that if a process reads a value of a data item, it will never see an older value in future reads.Monotonic writes. Ensures that writes are applied in the order issued by a single process. This model is useful for applications requiring ordered updates, such as Google Drive synchronization or distributed caching systems. Usage User sessions. Ensures users always see the latest updates across servers (Google Spanner, DynamoDB, Cosmos DB).Social media feeds. Prevents older posts from reappearing after seeing a newer version (Cassandra, Riak, DynamoDB).E-commerce transactions. Ensures order statuses don’t revert (e.g., "Shipped" never goes back to "Processing") (Google Spanner, Cosmos DB).Distributed caching. Avoids serving stale cache entries once a newer version is seen (Redis, DynamoDB). Trade-Offs Prevents inconsistency issues but does not enforce strict global ordering.Can introduce delays in synchronizing replicas across different regions. 5. Read-Your-Writes Consistency Read-Your-Writes consistency ensures that once a user writes (updates) data, any subsequent read by the same user will always reflect that update. This prevents users from seeing stale data after their own modifications. Usage: User profile updates. Ensures a user sees their latest profile changes immediately (Google Spanner, DynamoDB (session consistency), Cosmos DB).Social media posts. Guarantees users always see their latest posts or comments after submitting them (Cassandra, DynamoDB, Riak).Document editing applications. Guarantees users see the latest version of their document after saving (Google Drive (Spanner-based), Dropbox). Trade-Offs Can result in different consistency guarantees for different users.Works well in session-based consistency models but may not always ensure global consistency. Choosing the Right Consistency Model The choice of consistency model depends on the application’s requirements: Financial transactions, banking, and inventory systems require strong consistency to prevent anomalies.Social media feeds, recommendation engines, and caching layers benefit from eventual consistency to optimize scalability.Messaging systems and collaborative applications often require causal consistency to maintain the proper ordering of dependent events.E-commerce platforms might prefer read-your-writes consistency to ensure users see their most recent purchases.Distributed file systems and version control may rely on monotonic consistency to prevent rollback issues. Conclusion Database consistency is a critical aspect of data management in both traditional and distributed systems. While strong consistency ensures correctness, it comes at the cost of performance and availability. Eventual consistency prioritizes scalability and fault tolerance but may introduce temporary inconsistencies. Different models, such as causal, monotonic, and read-your-writes consistency, offer intermediate solutions tailored to specific use cases. Understanding the trade-offs of each model is essential for designing robust and efficient data architectures in modern applications. With the increasing complexity of distributed systems, the choice of the right consistency model is more critical than ever.

Hey, folks. I’m an AI geek who’s spent years wrestling with large language models (LLMs) like GPT-4. They’re incredible — chatting, coding, reasoning like champs — but they’ve got a flaw: they’re trained on the wild web, soaking up biases like gender stereotypes or racial skews. Picture an LLM skipping a top-notch female data scientist because it’s hung up on “tech = male.” That’s a real danger in hiring or healthcare apps, and it’s why I’ve poured my energy into Knowledge Graph-Augmented Training (KGAT). In this tutorial, I’ll share my approach. Straight from my work, like Detecting and Mitigating Bias in LLMs through Knowledge Graph-Augmented Training (Zenodo) with code and steps to try it yourself! The Bias Mess: Why I Dug In LLMs feast on internet chaos — tweets, blogs, the works — and inherit our messy biases. Feed one resumes, and it might favor “Mike” over “Maya” for a coding gig, echoing old patterns. My experiments with Bias in Bios showed this isn’t just talk — gender and racial skews pop up fast. Old fixes like data tweaks or fairness rules? They’re quick patches that don’t tackle the root or keep the model’s spark alive. That’s why I turned to knowledge graphs (KGs) — my game-changer. KGAT: My Fix for Better AI Imagine a knowledge graph as a fact-web — nodes like “engineer” or “woman” linked by edges like “works as.” My KGAT method, detailed in my enterprise intelligence paper, pairs this structured map with LLMs to cut bias and boost smarts. Here’s my playbook: Pick an LLM: I start with a beast like GPT-4.Add a KG: I hook it to a factual graph (Wikidata or custom) full of real connections.Train smart: Fine-tune it to cross-check text guesses with KG facts. This isn’t just about ethics — my enterprise pilots hit a 20% productivity spike! It’s in my Detecting and Mitigating Bias in LLMs talk at AIII 2025 (schedule). KGAT’s a business turbocharger, too. Hands-On: Build It With Me Let’s code up my KGAT pipeline. Here’s how I roll: 1. Prep the Data I use datasets like these to test bias and brains: Bias in Bios: Resumes with job/gender tags (source).FairFace: Faces with race/gender labels (source).COMPAS: Recidivism data for fairness (source). Clean lowercase text, ditch noise, and link entities (e.g., “data scientist”) to Wikidata. I keep it basic with simple entity matching for starters. 2. Wire Up the KG I lean on graph neural networks (GNNs) to turn KGs into vectors that LLMs can digest. My setup: Python import torch from torch_geometric.nn import GCNConv from transformers import GPT2Tokenizer, GPT2Model # Load LLM (GPT-2 for this demo) tokenizer = GPT2Tokenizer.from_pretrained('gpt2') model = GPT2Model.from_pretrained('gpt2') # My GNN layer (KG—swap in yours) gcn = GCNConv(in_channels=128, out_channels=768) # Match LLM dims kg_nodes = torch.rand(10, 128) # 10 nodes, 128-dim features kg_edges = torch.tensor([[0, 1], [1, 2], [2, 0]]) # Simple edges kg_emb = gcn(kg_nodes, kg_edges) # KG vectors ready 3. Blend and Train I merge LLM and KG embeddings with my formula: E_integrated = E_LLM ⊕ E_KG (just glue ‘em together). Training kickoff: Python # text embeddings (use your tokenized data) text_emb = torch.rand(32, 768) # Batch of 32, 768-dim integrated_emb = torch.cat([text_emb, kg_emb[:32]], dim=1) # Match sizes # Fine-tune (super simplified) outputs = model(inputs_embeds=integrated_emb) loss = outputs.loss # Add a real loss later loss.backward() # Optimize with Adam soon print("KGAT’s rolling!") For real runs, I use Adam (learning rate 3e-5, batch size 32, 10 epochs) — my go-to from the bias work. 4. Hunt Down Bias I track bias with metrics I swear by: Demographic parity: Equal positives across groups.Equal opportunity: Fair true-positive rates. Quick test: Python from sklearn.metrics import confusion_matrix # Dummy preds vs. truth y_true = [0, 1, 0, 1] y_pred = [0, 1, 1, 0] tn, fp, fn, tp = confusion_matrix(y_true, y_pred).ravel() equal_opp = tp / (tp + fn) print(f"Equal Opportunity: {equal_opp:.2f}") My results? Bias in Bios parity up 15%, COMPAS fairness up 10% — huge for trust in real apps. Why This Fires Me Up (and Should You) KGAT’s my passion because: Fairness counts: Biased AI can tank your app or harm users — I’m here to stop that.Scales big: My framework flexes with Wikidata or your own KG — enterprise-ready.Smarter AI: That 20% productivity lift? It’s KGs making LLMs brilliant, not just nice. Picture a hiring bot without KGAT; it skips “Priya” for “Pete.” With my method, it sees “data scientist” isn’t gendered and picks the best. Watch Out: My Hard-Earned Tips KGAT’s not perfect — I’ve hit snags: KG quality: A weak graph (e.g., outdated roles) can flop. I vet mine hard.Compute load: GNNs and LLMs need power — I lean on GPUs or the cloud.Big data: Millions of records? I chunk it or go parallel. Try It Out: My Challenge to You Start small with my approach: Grab Bias in Bios and a Wikidata slice.Use torch-geometric for GNNs and transformers for GPT-2 (or GPT-4 if you can).Tweak my code. Add real embeddings and a loss like cross-entropy. My pilots and bias talks show these scales — your next project could rock with it. My Take: Let’s Build Better AI KGAT’s my ticket to LLMs that don’t just dazzle but deliver — fair, smart, and ready to roll. It’s not just research; it’s hands-on and proven in my work. Fire up that code, test a dataset, and share your wins below. I’m stoked to see what you do with it! Dig deeper? Check my presentation on Zenodo or join me at DZone!

DZone events bring together industry leaders, innovators, and peers to explore the latest trends, share insights, and tackle industry challenges. From Virtual Roundtables to Fireside Chats, our events cover a wide range of topics, each tailored to provide you, our DZone audience, with practical knowledge, meaningful discussions, and support for your professional growth. DZone Events Happening Soon Below, you’ll find upcoming events that you won't want to miss. Best Practices for Building Secure Data Pipelines with Apache Airflow® Date: April 15, 2025Time: 1:00 PM ET Register for Free! Security is a critical but often overlooked aspect of data pipelines. Effective security controls help teams protect sensitive data, meet compliance requirements with confidence, and ensure smooth, secure operations. Managing credentials, enforcing access controls, and ensuring data integrity across systems can become overwhelming—especially while trying to keep Airflow environments up–to-date and operations running smoothly. Whether you're working to improve access management, protect sensitive data, or build more resilient pipelines, this webinar will provide the knowledge and best practices to enhance security in Apache Airflow. Generative AI: The Democratization of Intelligent Systemsive Date: April 16, 2025Time: 1:00 PM ET Register for Free! Join DZone, alongside industry experts from Cisco and Vertesia, for an exclusive virtual roundtable exploring the latest trends in GenAI. This discussion will dive into key insights from DZone's 2025 Generative AI Trend Report, focusing on advancements in GenAI models and algorithms, their impact on code generation, and the evolving role of AI in software development. We’ll examine AI adoption maturity, intelligent search capabilities, and how organizations can optimize their AI strategies for 2025 and beyond. Measuring CI/CD Transformations with Engineering Intelligence Date: April 23, 2025Time: 1:00 PM ET Register for Free! Ready to Measure the Real Impact of Your CI/CD Pipeline? CI/CD pipelines are essential, but how do you know they’re delivering the results your team needs? Join our upcoming webinar: Measuring CICD Transformations with Engineering Intelligence. We’ll be breaking down key metrics for speed, stability, and efficiency—and showing you how to take raw CI/CD data and turn it into real insights that power better decisions. What's Next? DZone has more in store! Stay tuned for announcements about upcoming Webinars, Virtual Roundtables, Fireside Chats, and other developer-focused events. Whether you’re looking to sharpen your skills, explore new tools, or connect with industry leaders, there’s always something exciting on the horizon. Don’t miss out — save this article and check back often for updates!

Data migration is like moving house — every data engineer has faced this headache: a pile of SQL statements that need rewriting, as if you have to disassemble and reassemble all the furniture. Different systems' SQL syntax is like different dialects. Although they all speak the SQL language, each has its own "accent" and habits. "If only there were a 'translator'!" This is probably the wish of every engineer who has experienced system migration. Today, I want to introduce a magical "translator" — Apache Doris's SQL dialect conversion feature. It can understand more than ten SQL dialects, including Presto, Trino, Hive, ClickHouse, and Oracle, and can automatically complete the conversion for you! Doris SQL Dialect Compatibility: Smooth Data Migration Like Silk "Facing system migration, SQL rewriting is like playing Tetris — one wrong move and you're in trouble." This sentence voices the sentiment of many data engineers. As data scales grow and businesses evolve, companies often need to migrate data from one system to another. The most painful part of this process is undoubtedly the compatibility of SQL syntax. Each data system has its unique SQL dialect, just like each place has its own dialect. Although they all speak SQL, each has its own "accent." When you need to migrate data from Presto/Trino, ClickHouse, or Hive to Doris, hundreds or even thousands of SQL statements need to be rewritten, which is undoubtedly a huge project. Apache Doris understands this pain. In version 2.1, Doris introduced the SQL dialect compatibility feature, supporting more than ten mainstream SQL dialects, including Presto, Trino, Hive, ClickHouse, and Oracle. Users only need to set a simple session variable to let Doris directly understand and execute the SQL syntax of other systems. Compatibility tests show that in some users' actual business scenarios, Doris' compatibility with Presto SQL reaches as high as 99.6%, and with the ClickHouse dialect, it reaches 98%. This means that the vast majority of SQL statements can run directly in Doris without modification. For data engineers, it is like holding a universal translator. No matter which SQL "dialect" it is, it can be automatically converted into a language that Doris can understand. System migration no longer requires manually rewriting a large number of SQL statements, greatly reducing the cost and risk of migration. From "Dialect Dilemma" to "Language Master" Zhang Gong is an experienced data engineer who recently received a challenging task — to migrate the company's data analysis platform from ClickHouse to Apache Doris. Faced with hundreds of SQL statements, he couldn't help but rub his temples. "If only there were a tool to directly convert ClickHouse SQL to Doris," Zhang Gong muttered to himself. It was then that he discovered Doris' SQL dialect compatibility feature. Let's follow Zhang Gong's steps to see how he solved this problem: First, download the latest version of the SQL dialect conversion tool. On any FE node, start the service with the following commands: Shell # config port vim apiserver/conf/config.conf # start SQL Converter for Apache Doris sh apiserver/bin/start.sh # webserver vim webserver/conf/config.conf # webserver start sh webserver/bin/start.sh Start the Doris cluster (version 2.1 or higher), and after the service is started, set the SQL conversion service address in Doris: SQL set global sql_converter_service_url = "http://127.0.0.1:5001/api/v1/convert" Then, switch the SQL dialect with just one command: SQL set sql_dialect=clickhouse; That's it! Zhang Gong found that SQL statements that originally needed to be manually rewritten could now be executed directly in Doris: SQL mysql> select toString(start_time) as col1, arrayCompact(arr_int) as col2, arrayFilter(x -> x like '%World%',arr_str)as col3, toDate(value) as col4, toYear(start_time)as col5, addMonths(start_time, 1)as col6, extractAll(value, '-.')as col7, JSONExtractString('{"id": "33"}' , 'id')as col8, arrayElement(arr_int, 1) as col9, date_trunc('day',start_time) as col10 FROM test_sqlconvert where date_trunc('day',start_time)= '2024-05-20 00:00:00' order by id; +---------------------+-----------+-----------+------------+------+---------------------+-------------+------+------+---------------------+ | col1 | col2 | col3 | col4 | col5 | col6 | col7 | col8 | col9 | col10 | +---------------------+-----------+-----------+------------+------+---------------------+-------------+------+------+---------------------+ | 2024-05-20 13:14:52 | [1, 2, 3] | ["World"] | 2024-01-14 | 2024 | 2024-06-20 13:14:52 | ['-0','-1'] | "33" | 1 | 2024-05-20 00:00:00 | +---------------------+-----------+-----------+------------+------+---------------------+-------------+------+------+---------------------+ 1 row in set (0.02 sec) "This is simply amazing!" Zhang Gong was pleasantly surprised to find that this seemingly complex ClickHouse SQL statement was perfectly executed. Not only that, but he also discovered that Doris provides a visual interface that supports both text input and file upload modes. For a single SQL statement, users can directly input text in the web interface. If there are a large number of existing SQL statements, you can upload files for one-click batch conversion of multiple SQL statements: Through the visual interface, Zhang Gong can upload SQL files in batches and complete the conversion with one click. "This is like having a universal translator that can seamlessly switch between ClickHouse and other SQL dialects," Zhang Gong exclaimed. What's more, he was delighted to find that the accuracy of this "translator" is quite high. In actual testing, the compatibility with Presto SQL reaches 99.6%, and with ClickHouse, it reaches 98%. This means that the vast majority of SQL statements can be used directly, greatly improving migration efficiency. The pressure of the data migration project was greatly reduced, and Zhang Gong could finally get a good night's sleep. However, he still had a small concern: "What if there are unsupported syntaxes?" At this point, he found that Doris' development team values user feedback highly. Through communities, Ask forums, GitHub Issues, or mailing lists, users can provide feedback anytime to promote the continuous optimization and improvement of the SQL dialect conversion feature. This open and user feedback-oriented attitude gives Zhang Gong great confidence for the future. "Next time I encounter a data migration project, I know which 'magic tool' to use!" Stay tuned for more interesting, useful, and valuable content in the next issue!

In 2023, a generative AI-powered chatbot for a financial firm mistakenly gave investment advice that violated compliance regulations, triggering regulatory scrutiny. Around the same time, an AI-powered medical summary tool misrepresented patient conditions, raising serious ethical concerns. As businesses rapidly adopt generative AI (GenAI), these incidents highlight a critical question: Can AI-generated content be trusted without human oversight? Generative AI is reshaping industries like retail, healthcare, and finance, with 65% of organizations already using it in at least one critical function, according to a 2024 McKinsey report (McKinsey, 2024). The speed and scale of AI-driven content generation are unprecedented, but with this power comes risk. AI-generated content can be misleading, biased, or factually incorrect, leading to reputational, legal, and ethical consequences if left unchecked. While it might be tempting to let large language models (LLMs) like GPT-4 operate autonomously, research highlights significant performance variability. A study testing GPT-4 across 27 real-world annotation tasks found that while the model performed well in structured settings, achieving precision and recall rates above 0.7, its performance dropped significantly in complex, context-dependent scenarios, sometimes falling below 0.5 (Pangakis & Wolken, 2024). In one-third of the tasks, GPT-4’s errors were substantial enough to introduce biases and inaccuracies, an unacceptable risk in high-stakes domains like healthcare, finance, and regulatory compliance. Key results from automated annotation performance using GPT-4 across 27 tasks (Pangakis & Wolken, 2024) Think of GPT-4 as an incredibly efficient research assistant, it rapidly gathers information (high recall) but lacks the precision or contextual awareness to ensure its outputs always meet the required standard. For instance, an AI writing tool for a skincare brand might generate an enticing but misleading product description: "Erases wrinkles in just 24 hours!". Such overpromising can violate advertising laws, mislead consumers, and damage brand credibility. Why Human Oversight Matters AI-generated content is reshaping how businesses communicate, advertise, and engage with customers, offering unparalleled efficiency at scale. However, without human oversight, AI-driven mistakes can lead to serious consequences, eroding trust, damaging reputations, or even triggering legal issues. According to Accenture’s Life Trends 2025 report, 59.9% of consumers now doubt the authenticity of online content due to the rapid influx of AI-generated material (Accenture, 2024). This growing skepticism raises a critical question: How can businesses ensure that AI-generated content remains credible and trustworthy? Meta has introduced AI-generated content labels across Facebook, Instagram, and Threads to help users distinguish AI-created images, signaling a growing recognition of the need for transparency in AI-generated content. But transparency alone isn’t enough — companies must go beyond AI disclaimers and actively build safeguards that ensure AI-generated content meets quality, ethical, and legal standards. Human oversight plays a defining role in mitigating these risks. AI may generate content at scale, but it lacks real-world context, ethical reasoning, and the ability to understand regulatory nuances. Without human review, AI-generated errors can mislead customers, compromise accuracy in high-stakes areas, and introduce ethical concerns, such as AI-generated medical content suggesting treatments without considering patient history. These risks aren’t theoretical; businesses across industries are already grappling with the challenge of balancing AI efficiency with trust. This is where Trust Calibration comes in, a structured approach to ensuring AI-generated content is reliable while maintaining the speed and scale that businesses need. Trust Calibration: When to Trust AI and When to Step In AI oversight shouldn’t slow down innovation; it should enable responsible progress. The key is determining when and how much human intervention is needed, based on the risk level, audience impact, and reliability of the AI model. Organizations can implement Trust Calibration by categorizing AI-generated content based on its risk profile and defining oversight strategies accordingly: High-risk content (medical guidance, financial projections, legal analysis) requires detailed human review before publication.Moderate-risk content (marketing campaigns, AI-driven recommendations) benefits from automated checks with human validation for anomalies.Low-risk content (social media captions, images, alt text) can largely run on AI with periodic human audits. Fine-tuning AI parameters, such as prompt engineering or temperature adjustments, modifying how deterministic or creative the AI's responses are by adjusting the probability distribution of generated words, can refine outputs, but research confirms these tweaks alone can’t eliminate fundamental AI limitations. AI models, especially those handling critical decision-making, must always have human oversight mechanisms in place. However, knowing that oversight is needed isn’t enough, organizations must ensure practical implementation to prevent getting stuck in analysis paralysis, where excessive review slows down decision-making. Many organizations are therefore adopting AI monitoring dashboards to track precision, recall, and confidence scores in production, helping ensure AI reliability over time. Use Cases: Areas Where AI Needs a Second Opinion Understanding when and how to apply oversight is just as important as recognizing why it’s needed. The right approach depends on the specific AI application and its risk level. Here are four major areas where AI oversight is essential, along with strategies for effective implementation. 1. Content Moderation and Compliance AI is widely used to filter inappropriate content on digital platforms, from social media to customer reviews. However, AI often misinterprets context, flagging harmless content as harmful or failing to catch actual violations. How to build oversight: Use confidence scoring to classify content as low, medium, or high risk, escalating borderline cases to human moderators.Implement reinforcement learning feedback loops, allowing human corrections to continuously improve AI accuracy. 2. AI-Generated Product and Marketing Content AI-powered tools generate product descriptions, ad copy, and branding materials, but they can overpromise or misrepresent features, leading to consumer trust issues and regulatory risks. How to build oversight: Use fact-checking automation to flag exaggerated claims that don’t align with verified product specifications.Set confidence thresholds, requiring human review for AI-generated content making strong performance claims.Implement "guardrails" in the prompt design or model training to prevent unverifiable claims like "instant results," "guaranteed cure," or "proven to double sales." 3. AI-Powered Customer Support and Sentiment Analysis Chatbots and sentiment analysis tools enhance customer interactions, but they can misinterpret tone, intent, or urgency, leading to poor user experiences. How to build oversight: Implement escalation workflows, where the AI hands off low-confidence responses to human agents.Train AI models on annotated customer interactions, ensuring they learn from flagged conversations to improve future accuracy. 4. AI in Regulated Industries (Healthcare, Finance, Legal) AI is increasingly used in medical diagnostics, financial risk assessments, and legal research, but errors in these domains can have serious real-world consequences. How to build oversight: Require explainability tools so human reviewers can trace AI decision-making before acting on it.Maintain audit logs to track AI recommendations and human interventions.Set strict human-in-the-loop policies, ensuring AI assists but does not finalize high-risk decisions. Before You Deploy AI, Check These Six Things While Trust Calibration determines the level of oversight, organizations still need a structured AI evaluation process to ensure reliability before deployment. #Stepkey actionimplementation strategy 1 Define the objective and risks Identify AI’s purpose and impact What is the task? What happens if AI gets it wrong? 2 Select the right model Match AI capabilities to the task Generative models for broad tasks, fine-tuned models for factual accuracy. 3 Establish a human validation set Create a strong benchmark Use expert-labeled data to measure AI performance. 4 Test performance Evaluate AI with real-world data Check precision, recall, and F1 score across varied scenarios. 5 Implement oversight mechanisms Ensure reliability & transparency Use confidence scoring, explainability tools, and escalation workflows. 6 Set deployment criteria Define go-live thresholds Establish minimum accuracy benchmarks and human oversight triggers. By embedding structured evaluation and oversight into AI deployment, organizations move beyond trial and error, ensuring AI is both efficient and trustworthy. Final Thoughts The question isn’t just “Can we trust AI?” It’s “How can we build AI that deserves our trust?” AI should be a partner in decision-making, not an unchecked authority. Organizations that design AI oversight frameworks today will lead the industry in responsible AI adoption, ensuring innovation doesn’t come at the cost of accuracy, ethics, or consumer trust. In the race toward AI-driven transformation, success won’t come from how fast we deploy AI; it will come from how responsibly we do it.

In my last post, I wrote about how quick and easy it is to turn an idea into reality. I built a Spring Boot API service using Gradle as my build management tool and then deployed it to Heroku. But what about my readers who have Maven in their toolchain? In this post, I’ll walk through the same project, but we'll look at how to accomplish the same result with Maven. And we'll see how Heroku makes deploying your Java apps and services seamless, regardless of the build tool you use. The Motivational Quotes API In my prior article, I sent the following request to ChatGPT: With some minor tweaks, I settled on the following OpenAPI specification in YAML format (saved as openapi.yaml): YAML openapi: 3.0.0 info: title: Motivational Quotes API description: An API that provides motivational quotes. version: 1.0.0 servers: - url: https://api.example.com description: Production server paths: /quotes: get: summary: Get all motivational quotes operationId: getAllQuotes responses: '200': description: A list of motivational quotes content: application/json: schema: type: array items: $ref: '#/components/schemas/Quote' /quotes/random: get: summary: Get a random motivational quote operationId: getRandomQuote responses: '200': description: A random motivational quote content: application/json: schema: $ref: '#/components/schemas/Quote' /quotes/{id}: get: summary: Get a motivational quote by ID operationId: getQuoteById parameters: - name: id in: path required: true schema: type: integer responses: '200': description: A motivational quote content: application/json: schema: $ref: '#/components/schemas/Quote' '404': description: Quote not found components: schemas: Quote: type: object required: - id - quote properties: id: type: integer quote: type: string Assumptions Like last time, we’re going to keep things simple. We’ll use Java 17 and Spring Boot 3 to create a RESTful API. This time, we’ll use Maven for our build automation. Like before, we won’t worry about adding a persistence layer, and we’ll continue to allow anonymous access to the API. Building the Spring Boot Service Using API-First Again, I’ll use the Spring Boot CLI to create a new project. Here’s how you can install the CLI using Homebrew: Shell $ brew tap spring-io/tap $ brew install spring-boot Create a new Spring Boot Service Using Maven We’ll call our new project quotes-maven and create it with the following command: Shell $ spring init --build=maven \ --package-name=com.example.quotes \ --dependencies=web,validation quotes-maven Notice how we specify the use of Maven for the build system instead of the default, Gradle. I also specify the com.example.quotes package name so that I can simply copy and paste the business code from the Gradle-based service to this service. Here are the contents of the quotes-maven folder: Shell $ cd quotes-maven && ls -la total 72 drwxr-xr-x 10 johnvester 320 Mar 15 10:49 . drwxrwxrwx 89 root 2848 Mar 15 10:49 .. -rw-r--r-- 1 johnvester 38 Mar 15 10:49 .gitattributes -rw-r--r-- 1 johnvester 395 Mar 15 10:49 .gitignore drwxr-xr-x 3 johnvester 96 Mar 15 10:49 .mvn -rw-r--r-- 1 johnvester 1601 Mar 15 10:49 HELP.md -rwxr-xr-x 1 johnvester 10665 Mar 15 10:49 mvnw -rw-r--r-- 1 johnvester 6912 Mar 15 10:49 mvnw.cmd -rw-r--r-- 1 johnvester 1535 Mar 15 10:49 pom.xml drwxr-xr-x 4 johnvester 128 Mar 15 10:49 src Next, we edit the pom.xml file to adopt the API-First approach. The resulting file looks like this: XML <?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <parent> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-parent</artifactId> <version>3.4.3</version> <relativePath/> <!-- lookup parent from repository --> </parent> <groupId>com.example</groupId> <artifactId>quotes-maven</artifactId> <version>0.0.1-SNAPSHOT</version> <name>demo</name> <description>Demo project for Spring Boot</description> <url/> <licenses> <license/> </licenses> <developers> <developer/> </developers> <scm> <connection/> <developerConnection/> <tag/> <url/> </scm> <properties> <java.version>17</java.version> </properties> <dependencies> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-web</artifactId> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-validation</artifactId> </dependency> <dependency> <groupId>org.springdoc</groupId> <artifactId>springdoc-openapi-starter-webmvc-ui</artifactId> <version>2.8.5</version> </dependency> <dependency> <groupId>org.openapitools</groupId> <artifactId>jackson-databind-nullable</artifactId> <version>0.2.6</version> </dependency> <dependency> <groupId>org.projectlombok</groupId> <artifactId>lombok</artifactId> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-test</artifactId> <scope>test</scope> </dependency> </dependencies> <build> <plugins> <plugin> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-maven-plugin</artifactId> </plugin> <plugin> <groupId>org.openapitools</groupId> <artifactId>openapi-generator-maven-plugin</artifactId> <version>7.12.0</version> <!-- Use the latest version --> <executions> <execution> <goals> <goal>generate</goal> </goals> </execution> </executions> <configuration> <inputSpec>${project.basedir}/src/main/resources/static/openapi.yaml</inputSpec> <output>${project.build.directory}/generated-sources/openapi</output> <generatorName>spring</generatorName> <apiPackage>com.example.api</apiPackage> <modelPackage>com.example.model</modelPackage> <invokerPackage>com.example.invoker</invokerPackage> <configOptions> <dateLibrary>java8</dateLibrary> <interfaceOnly>true</interfaceOnly> <useSpringBoot3>true</useSpringBoot3> <useBeanValidation>true</useBeanValidation> <skipDefaultInterface>true</skipDefaultInterface> </configOptions> </configuration> </plugin> </plugins> </build> </project> Then, we place openapi.yaml into the resources/static folder and create a file called application.yaml, placing it in the resources folder: YAML server: port: ${PORT:8080} spring: application: name: demo springdoc: swagger-ui: path: /swagger-docs url: openapi.yaml Finally, we create the following banner.txt file and place it into the resources folder: Shell ${AnsiColor.BLUE} _ __ _ _ _ ___ | |_ ___ ___ / _` | | | |/ _ \| __/ _ \/ __| | (_| | |_| | (_) | || __/\__ \ \__, |\__,_|\___/ \__\___||___/ |_| ${AnsiColor.DEFAULT} :: Running Spring Boot ${AnsiColor.BLUE}${spring-boot.version}${AnsiColor.DEFAULT} :: Port #${AnsiColor.BLUE}${server.port}${AnsiColor.DEFAULT} :: We can start the Spring Boot service to ensure everything works as expected. Looks good! Add the Business Logic With the base service ready and already adhering to our OpenAPI contract, we add the business logic to the service. To avoid repeating myself, you can refer to my last article for implementation details. Clone the quotes repository, then copy and paste the controllers, repositories, and services packages into this project. Since we matched the package name from the original project, there should not be any updates required. We have a fully functional Motivational Quotes API with a small collection of responses. Now, let’s see how quickly we can deploy our service. Using Heroku to Finish the Journey Since Heroku is a great fit for deploying Spring Boot services, I wanted to demonstrate how using the Maven build system is just as easy as using Gradle. Going with Heroku allows me to deploy my services quickly without losing time dealing with infrastructure concerns. To match the Java version we’re using, we create a system.properties file in the root folder of the project. The file has one line: Properties files java.runtime.version = 17 Then, I create a Procfile in the same location for customizing the deployment behavior. This file also has one line: Shell web: java -jar target/quotes-maven-0.0.1-SNAPSHOT.jar It’s time to deploy. With the Heroku CLI, I can deploy the service using a few simple commands. First, I authenticate the CLI and then create a new Heroku app. Shell $ heroku login $ heroku create Creating app... done, polar-caverns-69037 https://polar-caverns-69037-f51c2cc7ef79.herokuapp.com/ | https://git.heroku.com/polar-caverns-69037.git My Heroku app instance is named polar-caverns-69037, so my service will run at https://polar-caverns-69037-f51c2cc7ef79.herokuapp.com/. One last thing to do … push the code to Heroku, which deploys the service: Shell $ git push heroku master Once this command is complete, we can validate a successful deployment via the Heroku dashboard: We’re up and running. It’s time to test. Motivational Quotes in Action With our service running on Heroku, we can send some curl requests to make sure everything works as expected. First, we retrieve the list of quotes: Shell $ curl \ --location \ 'https://polar-caverns-69037-f51c2cc7ef79.herokuapp.com/quotes' JSON [ { "id":1, "quote":"The greatest glory in living lies not in never falling, but in rising every time we fall." }, { "id":2, "quote":"The way to get started is to quit talking and begin doing." }, { "id":3, "quote":"Your time is limited, so don't waste it living someone else's life." }, { "id":4, "quote":"If life were predictable it would cease to be life, and be without flavor." }, { "id":5, "quote":"If you set your goals ridiculously high and it's a failure, you will fail above everyone else's success." } ] We can retrieve a single quote by its ID: Shell $ curl \ --location \ 'https://polar-caverns-69037-f51c2cc7ef79.herokuapp.com/quotes/3' JSON { "id":3, "quote":"Your time is limited, so don't waste it living someone else's life." } We can retrieve a random motivational quote: Shell $ curl --location \ 'https://polar-caverns-69037-f51c2cc7ef79.herokuapp.com/quotes/random' JSON { "id":4, "quote":"If life were predictable it would cease to be life, and be without flavor." } We can even browse the Swagger Docs too: Returning to the Heroku dashboard, we see some activity on our new service: Gradle Versus Maven Using either Gradle or Maven, we quickly established a brand new RESTful API and deployed it to Heroku. But which one should you use? Which is a better fit for your project? To answer this question, I asked ChatGPT again. Just like when I asked for an OpenAPI specification, I received a pretty impressive summary: Gradle is great for fast builds, flexibility, and managing multi-projects or polyglot environments. It's ideal for modern workflows and when you need high customization.Maven is better for standardized builds, simplicity, and when you need stable, long-term support with strong dependency management. I found this article from Better Projects Faster, which was published in early 2024 and focused on Java build tools with respect to job descriptions, Google searches, and Stack Overflow postings. While this information is a bit dated, it shows users continue to prefer (worldwide) Maven over Gradle: Over my career, I’ve been fortunate to use both build management tools, and this has helped minimize the learning curve associated with a new project. Even now, I find my team at Marqeta using both Gradle and Maven (nearly a 50/50 split) in our GitHub organization. Conclusion My readers may recall my personal mission statement, which I feel can apply to any IT professional: “Focus your time on delivering features/functionality that extends the value of your intellectual property. Leverage frameworks, products, and services for everything else.” — J. Vester In this article, we saw how Spring Boot handled everything required to implement a RESTful API using the Maven build management tool. Once our code was ready, we realized our idea quickly by deploying to Heroku with just a few CLI commands. Spring Boot, Maven, and Heroku provided the frameworks and services so that I could remain focused on realizing my idea, not distracted by infrastructure and setup. Having chosen the right tools, I could deliver my idea quickly. If you’re interested, the source code for this article can be found on GitLab. Have a really great day!

By John Vester CORE