-
Notifications
You must be signed in to change notification settings - Fork 5.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add SPI for delegating row expression optimizer #24144
base: master
Are you sure you want to change the base?
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @tdcmeehan.
Just some initial comments
presto-main/src/main/java/com/facebook/presto/server/testing/TestingPrestoServer.java
Outdated
Show resolved
Hide resolved
presto-main/src/main/java/com/facebook/presto/sql/expressions/ExpressionOptimizerManager.java
Show resolved
Hide resolved
17fd8e1
to
563611e
Compare
Codenotify: Notifying subscribers in CODENOTIFY files for diff 0720f17...8c8d1f4. No notifications. |
563611e
to
948cf13
Compare
948cf13
to
d25b1b6
Compare
d25b1b6
to
6235f50
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
At a high-level, the changes make sense. I had a few questions about the configuration and how we might be able to simplify it to be more straightforward for users
presto-tests/src/test/java/com/facebook/presto/memory/TestMemoryManager.java
Outdated
Show resolved
Hide resolved
presto-tests/src/test/java/com/facebook/presto/memory/TestMemoryManager.java
Outdated
Show resolved
Hide resolved
@@ -385,6 +388,7 @@ public TestingPrestoServer( | |||
eventListenerManager = ((TestingEventListenerManager) injector.getInstance(EventListenerManager.class)); | |||
clusterStateProvider = null; | |||
planCheckerProviderManager = injector.getInstance(PlanCheckerProviderManager.class); | |||
expressionManager.loadExpressionOptimizerFactory(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I feel like having an additional method to configure/load the expression optimizer is a bit of an anti-pattern. It creates the potential for bugs if someone is trying to properly create the expression manager by requiring this additional method to be called. I haven't dug much deeper, but is there a reason we can't perform the loading of the factory in the constructor of the expressionManager
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I believe the reason for this convention is these loading methods might entail something expensive, or that could fail, and it's helpful to do such things outside of Guice's bootstrapping process. Such failures are often very verbose because Guice returns the error in the context of where it failed during the bootstrapping process, and I think it's cleaner to do it outside of the Guice dependency graph so that the failure is clear and isolated.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Would it be possible to load it in a more lazy manner? Such as using guava's Suppliers.memoize
instead?
Otherwise, if we don't want to defer loading configuration like that, then I think we should do some runtime assertion when ExpressionOptimizerManager.getExpressionOptimizer
is called to verify that loadExpressionOptimizerFactory
method has been called first.
presto-main/src/main/java/com/facebook/presto/sql/expressions/ExpressionOptimizerManager.java
Outdated
Show resolved
Hide resolved
} | ||
|
||
@Override | ||
public Object optimize(RowExpression rowExpression, Level level, ConnectorSession session, Function<VariableReferenceExpression, Object> variableResolver) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
not a big fan of having the duplicated code in these methods. Consider refactoring to a private method?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's difficult as it stands because these two methods return different types. I raised #24287 which would make it easier to share code between the two methods.
presto-main/src/main/java/com/facebook/presto/sql/expressions/ExpressionOptimizerManager.java
Show resolved
Hide resolved
...ain/src/main/java/com/facebook/presto/sql/planner/iterative/rule/SimplifyRowExpressions.java
Outdated
Show resolved
Hide resolved
...-main/src/main/java/com/facebook/presto/sql/relational/DelegatingRowExpressionOptimizer.java
Outdated
Show resolved
Hide resolved
a7b6402
to
2d609d0
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Some comments,
...main/src/test/java/com/facebook/presto/sql/planner/iterative/rule/TestRemoveMapCastRule.java
Show resolved
Hide resolved
...s/src/test/java/com/facebook/presto/tests/expressions/TestDelegatingExpressionOptimizer.java
Outdated
Show resolved
Hide resolved
presto-tests/src/test/java/com/facebook/presto/tests/expressions/TestExpressionOptimizers.java
Outdated
Show resolved
Hide resolved
presto-tests/src/test/java/com/facebook/presto/tests/expressions/TestExpressionOptimizers.java
Show resolved
Hide resolved
presto-main/src/main/java/com/facebook/presto/sql/relational/ConnectorRowExpressionService.java
Show resolved
Hide resolved
97d0c3b
to
8c8d1f4
Compare
The runtime should consolidate to the `ExpressionOptimizerProvider` factory so that it can be customized without significant refactoring.
Description
As part of RFC-0006, we need to support out of process expression evaluation. Add support for pluggable expression optimization and planner support to utilize the new SPI.
Motivation and Context
RFC-0006. See #24126 for larger changes that include the Presto sidecar as described in the RFC.
Impact
No impact by default as the old in-memory evaluation is the default.
Test Plan
Tests have been added.
Contributor checklist
Release Notes