Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[SPARK-50671][BUILD] Error compiling spark-protobuf module using user…
…-defined protoc ### What changes were proposed in this pull request? People may use user-defined protoc(as below) for various reasons, for example, if they're on macOS 11, they have to use it since the default version of protoc and protoc-gen-grpc-java plugin do not work on macOS 11. ``` $ export SPARK_PROTOC_EXEC_PATH=/Users/foobar/dev/protoc-4.29.1/protoc $ export CONNECT_PLUGIN_EXEC_PATH=/Users/foobar/dev/protoc-gen-grpc-java-1.59.1/protoc-gen-grpc-java $ ./build/mvn -DskipTests clean package -Puser-defined-protoc -DskipDefaultProtoc ``` But this results in compilation failure: ``` [ERROR] Failed to execute goal com.github.os72:protoc-jar-maven-plugin:3.11.4:run (default) on project spark-protobuf_2.13: Execution default of goal com.github.os72:protoc-jar-maven-plugin:3.11.4:run failed: Cannot read the array length because "<local6>" is null ``` This PR aims to fix this compilation failure, the error is caused by `protocol-jar-maven-plugin` bug: - os72/protoc-jar-maven-plugin#104 ### Why are the changes needed? Before: ``` $ export SPARK_PROTOC_EXEC_PATH=/Users/foobar/dev/protoc-4.29.1/protoc $ export CONNECT_PLUGIN_EXEC_PATH=/Users/foobar/dev/protoc-gen-grpc-java-1.59.1/protoc-gen-grpc-java $ ./build/mvn -DskipTests clean package -Puser-defined-protoc -DskipDefaultProtoc ... [ERROR] Failed to execute goal com.github.os72:protoc-jar-maven-plugin:3.11.4:run (default) on project spark-protobuf_2.13: Execution default of goal com.github.os72:protoc-jar-maven-plugin:3.11.4:run failed: Cannot read the array length because "<local6>" is null ``` After: ``` $ export SPARK_PROTOC_EXEC_PATH=/Users/foobar/dev/protoc-4.29.1/protoc $ export CONNECT_PLUGIN_EXEC_PATH=/Users/foobar/dev/protoc-gen-grpc-java-1.59.1/protoc-gen-grpc-java $ ./build/mvn -DskipTests clean package -Puser-defined-protoc -DskipDefaultProtoc ... [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary for Spark Project Parent POM 4.0.0-SNAPSHOT: [INFO] [INFO] Spark Project Parent POM ........................... SUCCESS [ 7.539 s] [INFO] Spark Project Tags ................................. SUCCESS [ 7.527 s] [INFO] Spark Project Sketch ............................... SUCCESS [ 8.413 s] [INFO] Spark Project Common Utils ......................... SUCCESS [ 29.923 s] [INFO] Spark Project Local DB ............................. SUCCESS [ 10.732 s] [INFO] Spark Project Networking ........................... SUCCESS [ 16.691 s] [INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 12.653 s] [INFO] Spark Project Variant .............................. SUCCESS [ 5.262 s] [INFO] Spark Project Unsafe ............................... SUCCESS [ 16.246 s] [INFO] Spark Project Connect Shims ........................ SUCCESS [ 4.356 s] [INFO] Spark Project Launcher ............................. SUCCESS [ 8.666 s] [INFO] Spark Project Core ................................. SUCCESS [02:56 min] [INFO] Spark Project ML Local Library ..................... SUCCESS [ 30.063 s] [INFO] Spark Project GraphX ............................... SUCCESS [ 33.989 s] [INFO] Spark Project Streaming ............................ SUCCESS [ 51.897 s] [INFO] Spark Project SQL API .............................. SUCCESS [ 39.508 s] [INFO] Spark Project Catalyst ............................. SUCCESS [03:15 min] [INFO] Spark Project SQL .................................. SUCCESS [04:56 min] [INFO] Spark Project ML Library ........................... SUCCESS [02:30 min] [INFO] Spark Project Tools ................................ SUCCESS [ 4.463 s] [INFO] Spark Project Hive ................................. SUCCESS [01:33 min] [INFO] Spark Project Connect Common ....................... SUCCESS [ 55.308 s] [INFO] Spark Avro ......................................... SUCCESS [ 33.350 s] [INFO] Spark Protobuf ..................................... SUCCESS [ 38.981 s] [INFO] Spark Project REPL ................................. SUCCESS [ 21.165 s] [INFO] Spark Project Connect Server ....................... SUCCESS [ 58.033 s] [INFO] Spark Project Connect Client ....................... SUCCESS [ 59.078 s] [INFO] Spark Project Assembly ............................. SUCCESS [ 6.703 s] [INFO] Kafka 0.10+ Token Provider for Streaming ........... SUCCESS [ 20.047 s] [INFO] Spark Integration for Kafka 0.10 ................... SUCCESS [ 26.882 s] [INFO] Kafka 0.10+ Source for Structured Streaming ........ SUCCESS [ 41.397 s] [INFO] Spark Project Examples ............................. SUCCESS [ 41.308 s] [INFO] Spark Integration for Kafka 0.10 Assembly .......... SUCCESS [ 11.300 s] [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS ``` ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Run command: ``` $ export SPARK_PROTOC_EXEC_PATH=/Users/foobar/dev/protoc-4.29.1/protoc $ export CONNECT_PLUGIN_EXEC_PATH=/Users/foobar/dev/protoc-gen-grpc-java-1.59.1/protoc-gen-grpc-java $ ./build/mvn -DskipTests clean package -Puser-defined-protoc -DskipDefaultProtoc ``` ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#49296 from morvenhuang/SPARK-50671. Authored-by: morvenhuang <morven.huang@gmail.com> Signed-off-by: yangjie01 <yangjie01@baidu.com>
- Loading branch information