Supported Functions¶
Out of the box, the engine packs a large collection of functions the user can utilize. The engine has been written to be extended. A user can provide his own function(s). Please follow the User Defined Functions chapter for understanding how to enrich the existing functions set.
| Name | Description | Usage |
|---|---|---|
| ANONYMIZE | Obfuscates the data, in order to protect it. If the anonymized
value is not provided it will use
* for String and 0 for Numbers |
anonymize(expression [,Anonymized value]) |
| CAST | Enables conversion of values from one data type to another | cast(dt as int) |
| COALESCE | Returns the first non-null expression in the expression list.
You must specify two expressions
|
coalesce(value, prevValue) |
| CONCAT | Returns the string representation of concatenating each
expressionin the list. Null fields are left out
|
concatenate(weather.main, 'fixed', dt,temp.eve) |
| CONCATENATE | An alias for concat.
|
concatenate(weather.main, 'fixed', dt,temp.eve) |
| LEN | Returns the length of a string. LEN calculates length
using characters as defined by UTF-16
|
len(name) |
| POSITION | Return the position of the first occurrence of substring | position('math' in fieldA) |
| REPLACE | Returns string with every occurrence of
search_stringreplaced with
replacement_string |
replace(fieldA, 'math', 'mathematics') |
| SUBSTRING | Return a portion of the string, beginning at the given position | substring(field, 2) |
| CONTAINS | Return true if an expression contains the given substring | contains(field1, field2) |
| STARTSWITH | Return true if an expression starts with the given substring | startswith(field1, field2) |
| ENDSWITH | Return true if an expression ends with the given substring | endswith(field1, field2) |
| TRIM | Removes leading and trailing spaces from the input expression | trim(expression) |
| CHOP | Returns the last character from an expression of type string | chop(expression) |
| UNIX_TIMESTAMP | Converts the given date into the timestamp as a long | unix_timestamp(field) |
| DELETEWHITESPACE | Removes all whitespace from an expression of type string | deletewhitespace(expression) |
| LOWER | Returns the expression in lowercase | lower(expression) |
| UPPER | Returns the expression in uppercase | upper(expression) |
| LOWERCASE | An alias for lower | lowercase(expression) |
| UPPERCASE | An alias for upper | uppercase(expression) |
| TAKELEFT | Returns the left most ‘length’ characters from a string expression | takeleft(expression, length) |
| TAKERIGHT | Returns the right most ‘length’ characters from a string expression | takeright(expression, length) |
| DROPLEFT | Removes the left most ‘length’ characters from a string expression | dropleft(expression, length) |
| DROPRIGHT | Removes the left most ‘length’ characters from a string expression | dropright(expression, length) |
| DUMP | Show the internal representation of a value | dump(expression) |
| SIZEOF | Returns the number of elements in an array of object | sizeof(expression) |
| DIGITS | Retains only the digits from a string expression | digits(expression) |
| TYPEOF | Returns the object type of a complex expression. This can only be used when
the format on the wire includes the details of the
objects full name.
|
typeof(expression) |
| CAPITALIZE | Capitalizes the first letter of the expression | capitalize(expression) |
| CAMELCASE | Uncapitalizes the first letter of the expression | camelcase(expression) |
| TRUNCATE | Truncates the expression by removing whitespace from the beginning and end | truncate(expression) |
| ABBREVIATE | Abbreviates the expression to the given length and appends ellipses | abbreviate(expression, length) |
| DISTANCE | Calculate the distance between two points using the haversine method | distance(x1, y1, x2, y2) |
| INDEXOF | Returns the index of a substring in an expression | abbreviate(expression, substring) |
| RADIANS | Converts the input expression to radians | radians(expression) |
| DEGREES | Converts the input expression to degrees | degrees(expression) |
| SIN | Returns the trigonometric sine of an expression | sin(expression) |
| COS | Returns the trigonometric cosine of an expression | cos(expression) |
| TAN | Returns the trigonometric tangent of an expression | tan(expression) |
| ASIN | Returns the trigonometric arc sine of an expression | asin(expression) |
| ACOS | Returns the trigonometric arc cosine of an expression | acos(expression) |
| ATAN | Returns the trigonometric arc tangent of an expression | atan(expression) |
| SINH | Returns the hyperbolic sine of an expression | sinh(expression) |
| COSH | Returns the hyperbolic cosine of an expression | cosh(expression) |
| TANH | Returns the hyperbolic tangent of an expression | tanh(expression) |
| STRIPACCENTS | Removes diacritics (approximately the same as accents)
from an expression. The case will not be altered.
|
stripaccents(field1) |
| UUID | Returns an universally unique identifier | uuid() |
| RANDINT | Returns a random integer | randint() |
| SWAPCASE | Swaps the case of a string expression | swapcase() |
| ABS | Returns the absolute value of an
expressionit has to evaluate to a number type
|
abs(field1 + field2) |
| NEG | Returns the negative value of an
expressionit has to evaluate to a number type
|
neg(field1) |
| SIGN | Returns +1 if a value is positive or -1 if a value is negative
it has to evaluate to a number type
|
sign(field1) |
| ROUND | Returns the closest integer of an
expression , with tiesrounding towards positive infinity. it has to evaluate
to a number type
|
round(field1) |
| CEIL | Returns the absolute value of an
expressionit has to evaluate to a number type
|
ceil(field1) |
| FLOOR | Return the largest value not greater than the argument | floor(field1) |
| ‘%’ | The remainder operator (%) computes the remainder after
dividing its first operand by its second. Each expression has
to evaluate to a number
|
a % b |
| ‘/’ | Divides one number by another (an arithmetic operator)
Each expression has to evaluate to a number
|
a / b |
| ‘-‘ | Subtracts one number from another (an arithmetic operator)
Each expression has to evaluate to a number
|
a - b |
| ‘*’ | Multiplies one number from another (an arithmetic operator)
Each expression has to evaluate to a number
|
a * b |
| ‘+’ | Adds one number to another (an arithmetic operator)
Each expression has to evaluate to a number
|
a + b |
| ‘-‘ (negative) | Returns the negative of the value of a numeric expression
(a unary operator). The expression has to evaluate to a number
|
a * b |
| POW | Returns expression1 raised to the expression2 power.
Each expression has to evaluate to a number
|
pow(a, b) |
| SQRT | Returns the square root of expression.
The expression has to evaluate to a number
|
sqrt(a) |
| CBRT | Returns the cube root of expression.
The expression has to evaluate to a number
|
cbrt(a) |
| COUNT | Returns the number of records read
ONLY FOR STREAMING when grouping is involved |
SELECT count(*) |
| MAX | Returns maximum value of expression.
ONLY FOR STREAMING when grouping is involved |
max(field1) |
| MIN | Returns minimum value of expression.
ONLY FOR STREAMING when grouping is involved |
min(field1) |
| SUM | Returns the sum of expression for each record read.
ONLY FOR STREAMING when grouping is involved |
sum(field1) |
| IF | Evaluates the boolean result for condition. If true it will
return the value of expr1; otherwise, it evaluates and
returns the value of expr2
|
if(condition, expr1,expr2) |
| LPAD | Prepends the value of padExpr to the value of strExpr
until the total length is lengthExpr
|
LPAD(strExpr, lengthExpr, padExpr) |
| RPAD | Appends the value of padExpr to the value of strExpr
until the total length is lengthExpr
|
LPAD(strExpr, lengthExpr, padExpr) |
| EPOCH_TO_DATETIME | Converts an epoch into a datetime
yyyy-MM-dd'T'HH:mm:ss.SSSZstrExpr needs to be a LONG value containing the
milliseconds since 1 January 1970 00:00:00)
|
EPOCH_TO_DATETIME(strExpr) |
| EPOCH_TO_DATE | Converts an epoch into a date
yyyy-MM-ddThe strExpr needs to resolve to INT/LONG value. If the
value is an INT it is expected to be the day represented
as days since epoch
|
EPOCH_TO_DATE(strExpr) |
DATETIME_TO_EPOCH
FROM_DATETIME
FROMDATETIME
|
Converts a string representation of a datetime into epoch value.
The first parameter strExpr needs to be a STRING value.
The second parameter pattern must be a DateTime Format
|
FROM_DATETIME(strExpr, pattern) |
| DATE_TO_EPOCH | Converts a string representation of a date into epoch value.
For AVRO records, this will emit a field with a logical type of
Date. This means the output value is an INT and it represents
the days since Epoch. For JSON records the resulted value is the
epoch expressed as LONG. The first parameter strExpr needs to
resolve to a STRING value. The second parameter pattern,
must be a Date Format
|
DATE_TO_EPOCH(strExpr, pattern) |
| CONVERT_TIME | Converts the string format of a date [and time]. The first
parameter strExpr needs be a STRING value. The second
parameter fromPattern is the incoming date[and time] format.
The last parameter represents the target date[and time] format.
|
CONVERT_TIME(strExpr, fromPattern, toPattern) |
CURDATE
CURRENT_DATE
|
Provides the current ISO date value
|
CURDATE()/CURRENT_DATE() |
CURRENT_DATETIME
CURRENT_DATETIME
CURTIME
|
Provides the current ISO date and time.
|
CURTIME()/CURRENT_DATETIME() |
| YESTERDAY | Returns the current date time minus 1 day | yesterday() |
| TOMORROW | Returns the current date time plus 1 day | tomorrow() |
| NEXTWEEK | Return the current date time plus 1 week | nextweek() |
| LASTWEEK | Return the current date time minus 1 week | lastweek() |
| NEXTMONTH | Return the current date time plus 1 month | nextmonth() |
| LASTMONTH | Return the current date time minus 1 month | lastmonth() |
| DAY | Extracts the day component of an expression that is of type timestamp | day(expression) |
| HOUR | Extracts the hour component of an expression that is of type timestamp | hour(expression) |
| MINUTE | Extracts the minute component of an expression that is of type timestamp | minute(expression) |
| SECOND | Extracts the second component of an expression that is of type timestamp | second(expression) |
| MONTH | Extracts the month component of an expression that is of type timestamp | month(expression) |
| YEAR | Extracts the year component of an expression that is of type timestamp | year(expression) |
| ARRAY_SIZE | Returns the count of elements contained by the parameter.
It is expected the parameter, evaluates to
an array structure.
|
ARRAY_SIZE(EXPR) |
| REGEXP | Returns the matched groups otherwise null.
Consider this regular expression
(Mr|Mrs|Ms) ([A-Z][a-z]+) ([A-Z][a-z]+)and this input
Ms Jane Smith; the return values are:["Ms Jane Smith", "Ms", "Jane", "Smith"]. |
REGEXP(SQL_EXPR, REGEX_EXPR) |
| REGEXP_LIKE | Returns true if the input value matches the expression
otherwise false.
|
REGEXP_LIKE(SQL_EXPR, REGEX_EXPR) |
| REGEXP_REPLACE | Replace the matched text with the provided value.
If there’s no match the input will
remain the same.
|
REGEXP_REPLACE(SQL_EXPR, SQL_EXPR, REGEX_EXPR) |
| EXISTS | Returns true if the given field is present;
false otherwise.
|
EXISTS(FIELD_EXPR) |
| HEADERASSTRING | Returns the value of the record header key as
a STRING value.
|
HEADERASSTRING('HEADER_KEY') |
| HEADERASINT | Returns the value of the record header key as
an INT value.
|
HEADERASINT('HEADER_KEY') |
| HEADERASLONG | Returns the value of the record header key as
a LONG value.
|
HEADERASLONG('HEADER_KEY') |
| HEADERASDOUBLE | Returns the value of the record header key as
a DOUBLE value.
|
HEADERASDOUBLE('HEADER_KEY') |
| HEADERASFLOAT | Returns the value of the record header key as a FLOAT value.
|
HEADERASFLOAT('HEADER_KEY') |
| HEADERKEYS | Returns all the header keys for the current record.
|
HEADERKEYS() |