ballerina/etl Ballerina library

Categorizes a dataset based on a numeric field and specified ranges.

Order[] dataset = [
    { orderId: 1, customerName: "Alice", totalAmount: 5.3 },
    { orderId: 2, customerName: "Bob", totalAmount: 10.5 },
    { orderId: 3, customerName: "John", totalAmount: 15.0 },
    { orderId: 4, customerName: "Charlie", totalAmount: 25.0 },
    { orderId: 5, customerName: "David", totalAmount: 29.2 }
];
CategoryRanges categoryRanges = [0, [10,20], 30];
Order[][] categorized = check etl:categorizeNumeric(dataset, "totalAmount", categoryRanges);
=>[[{ orderId: 1, customerName: "Alice", totalAmount: 5.3 }],
   [{ orderId: 2, customerName: "Bob", totalAmount: 10.5 }, { orderId: 3, customerName: "John", totalAmount: 15.0 }],
   [{ orderId: 4, customerName: "Charlie", totalAmount: 25.0 }, { orderId: 5, customerName: "David", totalAmount: 29.2 }]]

Categorizes a dataset based on a string field using a set of regular expressions.

Customer[] dataset = [
    { name: "Alice", city: "New York" },
    { name: "Bob", city: "Colombo" },
    { name: "Charlie", city: "Los Angeles" },
    { name: "John", city: "Boston" }
];
regexp:RegExp[] regexArray = [re `A.*$`, re `^B.*$`, re `^C.*$`];
Customer[][] categorized = check etl:categorizeRegex(dataset, "city", regexArray);
=>[[{ name: "Alice", city: "New York" }],
   [{ name: "Bob", city: "Colombo" }],
   [{ name: "Charlie", city: "Los Angeles" }]]

Categorizes a dataset based on a string field using semantic classification.

Review[] dataset = [
    { id: 1, comment: "Great service!" },
    { id: 2, comment: "Good service!" },
    { id: 3, comment: "blh blh blh" },
    { id: 4, comment: "Terrible experience" },
];
Review[][] categorized = check etl:categorizeSemantic(dataset, "comment", ["Positive", "Negative"]);
=>[[{ id: 1, comment: "Great service!" }, { id: 2, comment: "Good service!" }],
   [{ id: 4, comment: "Terrible experience" }]]

Returns a new dataset with specified fields decrypted using AES-ECB decryption with a given symmetric key.

Customer[] encryptedDataset = [
    { name: "kHKa63v98rbDm+FB2DJ3ig==", age: 23 },
    { name: "S0x+hpmvSOIT7UE8hOGZkA==", age: 35 }
];
byte[16] key = [78, 45, 73, 76, 56, 73, 116, 116, 72, 70, 105, 108, 97, 110, 65, 100];
DecryptedCustomer[] decryptedData = check etl:decryptData(encryptedDataset, ["name"], key);
=> [{ name: "Alice", age: 23 },
    { name: "Bob", age: 35 }]

Returns a new dataset with specified fields encrypted using AES-ECB encryption with a given symmetric key.

Customer[] dataset = [
    { id: 1, name: "Alice", age: 25 },
    { id: 2, name: "Bob", age: 30 }
];
byte[16] key = [78, 45, 73, 76, 56, 73, 116, 116, 72, 70, 105, 108, 97, 110, 65, 100];
EncryptedCustomer[] encryptedData = check etl:encryptData(dataset, ["name"], key);
=>[{ id: 1, name: "kHKa63v98rbDm+FB2DJ3ig==", age: 25 },
   { id: 2, name: "S0x+hpmvSOIT7UE8hOGZkA==", age: 30 }]

Extracts structured data from a raw text input and maps it to a Ballerina record.

type Review record{|
    string goodPoints;
    string badPoints;
    string improvements;
|};
string reviews = "The smartphone has an impressive camera and smooth performance, making it great for photography and gaming. However, the battery drains quickly, and the charging speed could be improved. The UI is intuitive, but some features feel outdated and need a refresh.";
Review extractedDetails = check etl:extractFromText(reviews);
=> { goodPoints: "The smartphone has an impressive camera and smooth performance, making it great for photography and gaming.",
     badPoints: "However, the battery drains quickly, and the charging speed could be improved.",
     improvements: "The UI is intuitive, but some features feel outdated and need a refresh." }

Filters a random set of records from a dataset based on a specified ratio.

Customer[] dataset = [
    { id: 1, name: "Alice" },
    { id: 2, name: "Bob" },
    { id: 3, name: "Charlie" },
    { id: 4, name: "David" }
];
Customer[] filteredDataset = check etl:filterDataByRatio(dataset, 0.75);
=> [{ id: 4, name: "David" }, { id: 2, name: "Bob" }, { id: 3, name: "Charlie" }]

Filters a dataset based on a regex pattern match.

Customer[] dataset = [
    { id: 1, city: "New York" },
    { id: 2, city: "Los Angeles" },
    { id: 3, city: "San Francisco" }
];
string fieldName = "city";
regexp:RegExp regexPattern = re `^New.*$`;
Customer[] filteredDataset = check etl:filterDataByRegex(dataset, "city", regexPattern);
=> [{ id: 1, city: "New York"}]

Filters a dataset based on a relative numeric comparison expression.

Customer[] dataset = [
    { id: 1, name: "Alice", age: 25 },
    { id: 2, name: "Bob", age: 30 },
    { id: 3, name: "Charlie", age: 22 },
    { id: 4, name: "David", age: 28 }
];
Customer[] filteredDataset = check etl:filterDataByRelativeExp(dataset, "age", etl:GREATER_THAN, 25);
=> [{ id: 2, name: "Bob", age: 30}, {id: 4, name: "David", age: 28}]

Identifies and groups approximate duplicates in a dataset, returning a nested array with unique records first, followed by groups of similar records.

Customer[] dataset = [
    { name: "Alice", city: "New York" },
    { name: "Alice", city: "new york" },
    { name: "Bob", city: "Boston" },
    { name: "Charlie", city: "Los Angeles" },
    { name: "Charlie", city: "los angeles - usa" },
    { name: "John", city: "Chicago" }
];
Customer[][] result = check etl:groupApproximateDuplicates(dataset);
=> [[{ name: "Bob", city: "Boston" },{ name: "John", city: "Chicago" }],
    [{ name: "Alice", city: "New York" },{ name: "Alice", city: "new york" }],
    [{ name: "Charlie", city: "Los Angeles" },{ name: "Charlie", city: "los angeles - usa" }]]

Returns a new dataset with all extra whitespace removed from string fields.

Customer[] dataset = [
    { name: "  Alice  ", city: "  New   York  " },
    { name: "  Bob  ", city: "  Los Angeles  " },
    { name: "  Charlie  ", city: "  Chicago  " }
];
Customer[] cleanedData = check etl:handleWhiteSpaces(dataset);
=> [{ name: "Alice", city: "New York" },
    { name: "Bob", city: "Los Angeles" },
    { name: "Charlie", city: "Chicago" }]

Merges two datasets based on a common specified field and returns a new dataset with the merged records.

CustomerPersonalDetails[] dataset1 = [{ id: 1, name: "Alice" }, { id: 2, name: "Bob" }];
CustomerContactDetails[] dataset2 = [{ id: 1, phone: 0123456789 }, { id: 2, phone: 0987654321 }];
Customer[] mergedData = check etl:joinData(dataset1, dataset2, "id");
=> [{ id: 1, name: "Alice", phone: 0123456789 },
    { id: 2, name: "Bob", phone: 0987654321 }]

Returns a new dataset with PII (Personally Identifiable Information) fields masked using a specified character.

Customer[] dataset = [
    { id: 1, name: "John Doe", email: "john@example.com" },
    { id: 2, name: "Jane Smith", email: "jane@example.com" }
];
MaskedCustomer[] maskedData = check etl:maskSensitiveData(dataset);
=> [{ id: 1, name: "XXX XXX", email: "XXXXXXXXXXXXXXX" },
    { id: 2, name: "XXXX XXXX", email: "XXXXXXXXXXXXXXX" }]

Merges multiple datasets into a single dataset by flattening a nested array of records.

Customer[][] dataSets = [
    [{ id: 1, name: "Alice" }, { id: 2, name: "Bob" }],
    [{ id: 3, name: "Charlie" }, { id: 4, name: "David" }]
];
Customer[] mergedData = check etl:mergeData(dataSets);
=> [{ id: 1, name: "Alice" },
    { id: 2, name: "Bob" },
    { id: 3, name: "Charlie" },
    { id: 4, name: "David" }]

Returns a new dataset with all duplicate records removed.

Customer[] dataset = [
    { name: "Alice", city: "New York" },
    { name: "Alice", city: "New York" },
    { name: "Bob", city: "Los Angeles" },
    { name: "Charlie", city: "Chicago" },
    { name: "Charlie", city: "Chicago" }
];
Customer[] uniqueData = check etl:removeDuplicates(dataset);
=> [{ name: "Alice", city: "New York" },
    { name: "Bob", city: "Los Angeles" },
    { name: "Charlie", city: "Chicago" }]

Returns a new dataset with all records containing nil or empty string values removed.

Customer[] dataset = [
    { name: "Alice", city: "New York" },
    { name: "Bob", city: null },
    { name: "", city: "Los Angeles" },
    { name: "Charlie", city: "Boston" },
    { name: "David", city: () }
];
NewCustomer[] filteredData = check etl:removeNull(dataset);
=> [{ name: "Alice", city: "New York" },
    { name: "Charlie", city: "Boston" }]

Returns a new dataset with a specified field removed from each record.

Customer[] dataset = [
    { name: "Alice", city: "New York", age: 25 },
    { name: "Bob", city: "Los Angeles", age: 30 },
    { name: "Charlie", city: "Chicago", age: 35 }
];
NewCustomer[] updatedData = check etl:removeField(dataset, "age");
=> [{ name: "Alice", city: "New York" },
    { name: "Bob", city: "Los Angeles" },
    { name: "Charlie", city: "Chicago" }]

Returns a new dataset where matches of the given regex pattern in a specified string field are replaced with a new value.

Customer[] dataset = [
    { name: "Alice", city: "New York" },
    { name: "Bob", city: "Los Angeles" },
    { name: "Charlie", city: "Chicago" }
];
Customer[] updatedData = check etl:replaceText(dataset, "city", re `New York`, "San Francisco");
=> [{ name: "Alice", city: "San Francisco" },
    { name: "Bob", city: "Los Angeles" },
    { name: "Charlie", city: "Chicago" }]

Returns a new dataset sorted by a specified field in ascending or descending order.

Customer[] dataset = [
    { name: "Alice", age: 25 },
    { name: "Bob", age: 30 },
    { name: "Charlie", age: 22 }
];
Customer[] sortedData = check etl:sort(dataset, "age");
=> [{ name: "Charlie", age: 22 },
    { name: "Alice", age: 25 },
    { name: "Bob", age: 30 }]

Returns a new dataset with all string values in a specified field standardized to a set of standard values.

Customer[] dataset = [
    { name: "Alice", city: "New York" },
    { name: "Bob", city: "new york" },
    { name: "Charlie", city: "los-angeles" },
    { name: "John", city: "newyork -usa" }    
];
Customer[] standardizedData = check etl:standardizeData(dataset, "city", ["New York", "Los Angeles"]);
=> [{ name: "Alice", city: "New York" },
    { name: "Bob", city: "New York" },
    { name: "Charlie", city: "Los Angeles" },
    { name: "John", city: "New York" }]