Skip to content

Commit

Permalink
[SPARK-50692] Add RPAD pushdown support
Browse files Browse the repository at this point in the history
### What changes were proposed in this pull request?
This PR makes RPAD and LPAD functions translatable across connectors. Individual connectors can now support them via their dialects.

### Why are the changes needed?
When trying to compare CHAR(...) and a literal, Spark injects a RPAD function to make the CHAR parameter the same length. This behaviour is per ANSI standard.

The problem is that RPAD is not only an unsupported, but an untranslatable function. This prevents CHAR + literal comparison pushdown.

Solution is to support RPAD pushdowns and implement it when needed.

### Does this PR introduce _any_ user-facing change?
No.

### How was this patch tested?
This PR only introduces the ability to support the RPAD pushdown. Individual connectors will have their own tests.

### Was this patch authored or co-authored using generative AI tooling?
No

Closes #49325 from andrej-db/SPARK-50692-RPAD.

Authored-by: andrej-gobeljic_data <andrej.gobeljic@databricks.com>
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
  • Loading branch information
andrej-db authored and cloud-fan committed Dec 31, 2024
1 parent 8a09817 commit e1fb18d
Show file tree
Hide file tree
Showing 2 changed files with 3 additions and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -118,7 +118,7 @@ yield visitBinaryArithmetic(
"COT", "ASIN", "ASINH", "ACOS", "ACOSH", "ATAN", "ATANH", "ATAN2", "CBRT", "DEGREES",
"RADIANS", "SIGN", "WIDTH_BUCKET", "SUBSTRING", "UPPER", "LOWER", "TRANSLATE",
"DATE_ADD", "DATE_DIFF", "TRUNC", "AES_ENCRYPT", "AES_DECRYPT", "SHA1", "SHA2", "MD5",
"CRC32", "BIT_LENGTH", "CHAR_LENGTH", "CONCAT" ->
"CRC32", "BIT_LENGTH", "CHAR_LENGTH", "CONCAT", "RPAD", "LPAD" ->
visitSQLFunction(name, expressionsToStringArray(e.children()));
case "CASE_WHEN" -> visitCaseWhen(expressionsToStringArray(e.children()));
case "TRIM" -> visitTrim("BOTH", expressionsToStringArray(e.children()));
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -305,6 +305,8 @@ class V2ExpressionBuilder(e: Expression, isPredicate: Boolean = false) extends L
case _: Md5 => generateExpressionWithName("MD5", expr, isPredicate)
case _: Sha1 => generateExpressionWithName("SHA1", expr, isPredicate)
case _: Sha2 => generateExpressionWithName("SHA2", expr, isPredicate)
case _: StringLPad => generateExpressionWithName("LPAD", expr, isPredicate)
case _: StringRPad => generateExpressionWithName("RPAD", expr, isPredicate)
// TODO supports other expressions
case ApplyFunctionExpression(function, children) =>
val childrenExpressions = children.flatMap(generateExpression(_))
Expand Down

0 comments on commit e1fb18d

Please sign in to comment.