[SPARK-48773][FOLLOW-UP] spark.conf.set should not fail when setting …

…`spark.default.parallelism`

### What changes were proposed in this pull request?

spark.session.set should not fail when setting `spark.default.parallelism`.

### Why are the changes needed?

This is to fix a behavior change where before `SPARK-48773`, set `spark.default.parallelism` through spark session does not fail and is a no op.

### Does this PR introduce _any_ user-facing change?

Yes.

before `SPARK-48773`, spark.conf.set("spark.default.parallelism") does not fail and is a no-op.
after ``SPARK-48773`, spark.conf.set("spark.default.parallelism") will fail with a `CANNOT_MODIFY_CONFIG` exception.
With this followup, we restore the behavior to spark.conf.set("spark.default.parallelism") does not fail and is a no-op.

### How was this patch tested?

manually testing.

### Was this patch authored or co-authored using generative AI tooling?

No

Closes #48526 from amaliujia/SPARK-48773.

Authored-by: Rui Wang <rui.wang@databricks.com>
Signed-off-by: Wenchen Fan <wenchen@databricks.com>

sql/core/src/main/scala/org/apache/spark/sql/internal/RuntimeConfigImpl.scala

@@ -21,7 +21,7 @@ import scala.jdk.CollectionConverters._
 
 import org.apache.spark.SPARK_DOC_ROOT
 import org.apache.spark.annotation.Stable
-import org.apache.spark.internal.config.ConfigEntry
+import org.apache.spark.internal.config.{ConfigEntry, DEFAULT_PARALLELISM}
 import org.apache.spark.sql.RuntimeConfig
 import org.apache.spark.sql.errors.QueryCompilationErrors
 
@@ -85,6 +85,15 @@ class RuntimeConfigImpl private[sql](val sqlConf: SQLConf = new SQLConf) extends
   }
 
   private[sql] def requireNonStaticConf(key: String): Unit = {
+    // We documented `spark.default.parallelism` by SPARK-48773, however this config
+    // is actually a static config so now a spark.conf.set("spark.default.parallelism")
+    // will fail. Before SPARK-48773 it does not, then this becomes a behavior change.
+    // Technically the current behavior is correct, however it still forms a behavior change.
+    // To address the change, we need a check here and do not fail on default parallelism
+    // setting through spark session conf to maintain the same behavior.
+    if (key == DEFAULT_PARALLELISM.key) {
+      return
+    }
     if (SQLConf.isStaticConfigKey(key)) {
       throw QueryCompilationErrors.cannotModifyValueOfStaticConfigError(key)
     }

sql/core/src/test/scala/org/apache/spark/sql/RuntimeConfigSuite.scala

@@ -19,6 +19,7 @@ package org.apache.spark.sql
 
 import org.apache.spark.SparkFunSuite
 import org.apache.spark.internal.config
+import org.apache.spark.internal.config.DEFAULT_PARALLELISM
 import org.apache.spark.sql.internal.{RuntimeConfigImpl, SQLConf}
 import org.apache.spark.sql.internal.SQLConf.CHECKPOINT_LOCATION
 import org.apache.spark.sql.internal.StaticSQLConf.GLOBAL_TEMP_DATABASE
@@ -101,4 +102,10 @@ class RuntimeConfigSuite extends SparkFunSuite {
     // Get the unset config entry, which should return its defaultValue again.
     assert(conf.get(key) == SQLConf.SESSION_LOCAL_TIMEZONE.defaultValue.get)
   }
+
+  test("SPARK-48773: set spark.default.parallelism does not fail") {
+    val conf = newConf()
+    // this set should not fail
+    conf.set(DEFAULT_PARALLELISM.key, "1")
+  }
 }