Skip to content

Commit

Permalink
[SPARK-50671][BUILD] Error compiling spark-protobuf module using user…
Browse files Browse the repository at this point in the history
…-defined protoc

### What changes were proposed in this pull request?
People may use user-defined protoc(as below) for various reasons, for example, if they're on macOS 11, they have to use it since the default version of protoc and protoc-gen-grpc-java plugin do not work on macOS 11.

```
$ export SPARK_PROTOC_EXEC_PATH=/Users/foobar/dev/protoc-4.29.1/protoc
$ export CONNECT_PLUGIN_EXEC_PATH=/Users/foobar/dev/protoc-gen-grpc-java-1.59.1/protoc-gen-grpc-java
$ ./build/mvn -DskipTests clean package -Puser-defined-protoc -DskipDefaultProtoc
```

But this results in compilation failure:

```
[ERROR] Failed to execute goal com.github.os72:protoc-jar-maven-plugin:3.11.4:run (default) on project spark-protobuf_2.13: Execution default of goal com.github.os72:protoc-jar-maven-plugin:3.11.4:run failed: Cannot read the array length because "<local6>" is null
```

This PR aims to fix this compilation failure, the error is caused by `protocol-jar-maven-plugin` bug:

- os72/protoc-jar-maven-plugin#104

### Why are the changes needed?
Before:
```
$ export SPARK_PROTOC_EXEC_PATH=/Users/foobar/dev/protoc-4.29.1/protoc
$ export CONNECT_PLUGIN_EXEC_PATH=/Users/foobar/dev/protoc-gen-grpc-java-1.59.1/protoc-gen-grpc-java
$ ./build/mvn -DskipTests clean package -Puser-defined-protoc -DskipDefaultProtoc
...
[ERROR] Failed to execute goal com.github.os72:protoc-jar-maven-plugin:3.11.4:run (default) on project spark-protobuf_2.13: Execution default of goal com.github.os72:protoc-jar-maven-plugin:3.11.4:run failed: Cannot read the array length because "<local6>" is null
```

After:
```
$ export SPARK_PROTOC_EXEC_PATH=/Users/foobar/dev/protoc-4.29.1/protoc
$ export CONNECT_PLUGIN_EXEC_PATH=/Users/foobar/dev/protoc-gen-grpc-java-1.59.1/protoc-gen-grpc-java
$ ./build/mvn -DskipTests clean package -Puser-defined-protoc -DskipDefaultProtoc
...
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Spark Project Parent POM 4.0.0-SNAPSHOT:
[INFO]
[INFO] Spark Project Parent POM ........................... SUCCESS [  7.539 s]
[INFO] Spark Project Tags ................................. SUCCESS [  7.527 s]
[INFO] Spark Project Sketch ............................... SUCCESS [  8.413 s]
[INFO] Spark Project Common Utils ......................... SUCCESS [ 29.923 s]
[INFO] Spark Project Local DB ............................. SUCCESS [ 10.732 s]
[INFO] Spark Project Networking ........................... SUCCESS [ 16.691 s]
[INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 12.653 s]
[INFO] Spark Project Variant .............................. SUCCESS [  5.262 s]
[INFO] Spark Project Unsafe ............................... SUCCESS [ 16.246 s]
[INFO] Spark Project Connect Shims ........................ SUCCESS [  4.356 s]
[INFO] Spark Project Launcher ............................. SUCCESS [  8.666 s]
[INFO] Spark Project Core ................................. SUCCESS [02:56 min]
[INFO] Spark Project ML Local Library ..................... SUCCESS [ 30.063 s]
[INFO] Spark Project GraphX ............................... SUCCESS [ 33.989 s]
[INFO] Spark Project Streaming ............................ SUCCESS [ 51.897 s]
[INFO] Spark Project SQL API .............................. SUCCESS [ 39.508 s]
[INFO] Spark Project Catalyst ............................. SUCCESS [03:15 min]
[INFO] Spark Project SQL .................................. SUCCESS [04:56 min]
[INFO] Spark Project ML Library ........................... SUCCESS [02:30 min]
[INFO] Spark Project Tools ................................ SUCCESS [  4.463 s]
[INFO] Spark Project Hive ................................. SUCCESS [01:33 min]
[INFO] Spark Project Connect Common ....................... SUCCESS [ 55.308 s]
[INFO] Spark Avro ......................................... SUCCESS [ 33.350 s]
[INFO] Spark Protobuf ..................................... SUCCESS [ 38.981 s]
[INFO] Spark Project REPL ................................. SUCCESS [ 21.165 s]
[INFO] Spark Project Connect Server ....................... SUCCESS [ 58.033 s]
[INFO] Spark Project Connect Client ....................... SUCCESS [ 59.078 s]
[INFO] Spark Project Assembly ............................. SUCCESS [  6.703 s]
[INFO] Kafka 0.10+ Token Provider for Streaming ........... SUCCESS [ 20.047 s]
[INFO] Spark Integration for Kafka 0.10 ................... SUCCESS [ 26.882 s]
[INFO] Kafka 0.10+ Source for Structured Streaming ........ SUCCESS [ 41.397 s]
[INFO] Spark Project Examples ............................. SUCCESS [ 41.308 s]
[INFO] Spark Integration for Kafka 0.10 Assembly .......... SUCCESS [ 11.300 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
```

### Does this PR introduce _any_ user-facing change?
No.

### How was this patch tested?
Run command:
```
$ export SPARK_PROTOC_EXEC_PATH=/Users/foobar/dev/protoc-4.29.1/protoc
$ export CONNECT_PLUGIN_EXEC_PATH=/Users/foobar/dev/protoc-gen-grpc-java-1.59.1/protoc-gen-grpc-java
$ ./build/mvn -DskipTests clean package -Puser-defined-protoc -DskipDefaultProtoc
```

### Was this patch authored or co-authored using generative AI tooling?
No.

Closes apache#49296 from morvenhuang/SPARK-50671.

Authored-by: morvenhuang <morven.huang@gmail.com>
Signed-off-by: yangjie01 <yangjie01@baidu.com>
  • Loading branch information
morvenhuang authored and LuciferYang committed Jan 6, 2025
1 parent 459483a commit b3d2f28
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion connector/protobuf/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -189,7 +189,7 @@
<inputDirectories>
<include>src/test/resources/protobuf</include>
</inputDirectories>
<includeStdTypes>true</includeStdTypes>
<includeMavenTypes>direct</includeMavenTypes>
<outputTargets>
<outputTarget>
<type>java</type>
Expand Down

0 comments on commit b3d2f28

Please sign in to comment.