Changes

Summary

  1. [SPARK-33294][SQL] Add query resolved check before analyze InsertIntoDir (commit: d59f6a7) (details)
  2. [SPARK-33297][BUILD] Switch to use flat class loader strategy in SBT (commit: 3af1651) (details)
  3. [MINOR][CORE][DOCS] Fix typo in (commit: 7c897c1) (details)
  4. [SPARK-33262][K8S][FOLLOWUP] Verify pod allocation does not stall (commit: 491a0fb) (details)
  5. [SPARK-32037][CORE] Rename blacklisting feature (commit: 72ad9dc) (details)
Commit d59f6a709586ff0d1bfbfda50c4e4cf17d5a50ff by wenchen
[SPARK-33294][SQL] Add query resolved check before analyze InsertIntoDir
### What changes were proposed in this pull request?
Add `query.resolved` before analyze `InsertIntoDir`.
### Why are the changes needed?
For better error msg.
``` INSERT OVERWRITE DIRECTORY '/tmp/file' USING PARQUET SELECT * FROM (
SELECT c3 FROM (
SELECT c1, c2 from values(1,2) t(c1, c2)
)
)
```
Before this PR, we get such error msg
``` org.apache.spark.sql.catalyst.analysis.UnresolvedException: Invalid
call to toAttribute on unresolved object, tree: *
at
org.apache.spark.sql.catalyst.analysis.Star.toAttribute(unresolved.scala:244)
at
org.apache.spark.sql.catalyst.plans.logical.Project$$anonfun$output$1.apply(basicLogicalOperators.scala:52)
at
org.apache.spark.sql.catalyst.plans.logical.Project$$anonfun$output$1.apply(basicLogicalOperators.scala:52)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.immutable.List.foreach(List.scala:392)
```
### Does this PR introduce _any_ user-facing change?
Yes, error msg changed.
### How was this patch tested?
New test.
Closes #30197 from ulysses-you/SPARK-33294.
Authored-by: ulysses <youxiduo@weidian.com> Signed-off-by: Wenchen Fan
<wenchen@databricks.com>
(commit: d59f6a7)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/sources/InsertSuite.scala (diff)
Commit 3af1651e50be3bc2e441be8827441f87d34d99cc by gurwls223
[SPARK-33297][BUILD] Switch to use flat class loader strategy in SBT
### What changes were proposed in this pull request?
This PR proposes to switch the class loader strategy from `ScalaLibrary`
to `Flat` (see
https://www.scala-sbt.org/1.x/docs/In-Process-Classloaders.html):
https://github.com/apache/spark/runs/1314691686
``` Error:  java.util.MissingResourceException: Can't find bundle for
base name org.scalactic.ScalacticBundle, locale en Error:  at
java.util.ResourceBundle.throwMissingResourceException(ResourceBundle.java:1581)
Error:  at
java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1396) Error:
at java.util.ResourceBundle.getBundle(ResourceBundle.java:782)
Error:  at
org.scalactic.Resources$.resourceBundle$lzycompute(Resources.scala:8)
Error:  at
org.scalactic.Resources$.resourceBundle(Resources.scala:8) Error:  at
org.scalactic.Resources$.pleaseDefineScalacticFillFilePathnameEnvVar(Resources.scala:256)
Error:  at
org.scalactic.source.PositionMacro$PositionMacroImpl.apply(PositionMacro.scala:65)
Error:  at
org.scalactic.source.PositionMacro$.genPosition(PositionMacro.scala:85)
Error:  at sun.reflect.GeneratedMethodAccessor34.invoke(Unknown
Source) Error:  at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Error:  at java.lang.reflect.Method.invoke(Method.java:498)
```
See also https://github.com/sbt/sbt/issues/5736
### Why are the changes needed?
To make the build unflaky.
### Does this PR introduce _any_ user-facing change?
No, dev-only.
### How was this patch tested?
GitHub Actions build in this test.
Closes #30198 from HyukjinKwon/SPARK-33297.
Authored-by: HyukjinKwon <gurwls223@apache.org> Signed-off-by:
HyukjinKwon <gurwls223@apache.org>
(commit: 3af1651)
The file was modifiedproject/SparkBuild.scala (diff)
Commit 7c897c1216dd23e4a973bd82063a88ea9a6f7ca5 by dhyun
[MINOR][CORE][DOCS] Fix typo in
"spark.storage.decommission.shuffleBlocks.enabled" description
### What changes were proposed in this pull request? Small typo fix in
the description of `spark.storage.decommission.shuffleBlocks.enabled`
property.
Closes #30208 from dsabanin/patch-1.
Authored-by: Dmitry Sabanin <sdmitry@gmail.com> Signed-off-by: Dongjoon
Hyun <dhyun@apple.com>
(commit: 7c897c1)
The file was modifiedcore/src/main/scala/org/apache/spark/internal/config/package.scala (diff)
Commit 491a0fb08b0c57a99894a0b33c5814854db8de3d by dhyun
[SPARK-33262][K8S][FOLLOWUP] Verify pod allocation does not stall
### What changes were proposed in this pull request?
Add a test that pending executor does not stall pod allocation.
### Why are the changes needed?
Better test coverage
### Does this PR introduce _any_ user-facing change?
Test only change.
### How was this patch tested?
New test passes.
Closes #30205 from holdenk/verify-pod-allocation-does-not-stall.
Authored-by: Holden Karau <hkarau@apple.com> Signed-off-by: Dongjoon
Hyun <dhyun@apple.com>
(commit: 491a0fb)
The file was modifiedresource-managers/kubernetes/core/src/test/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsAllocatorSuite.scala (diff)
Commit 72ad9dcd5d484a8dd64c08889de85ef9de2a6077 by tgraves
[SPARK-32037][CORE] Rename blacklisting feature
### What changes were proposed in this pull request?
this PR renames the blacklisting feature. I ended up using
"excludeOnFailure" or "excluded" in most cases but there is a mix. I
renamed the BlacklistTracker to HealthTracker, but for the
TaskSetBlacklist HealthTracker didn't make sense to me since its not the
health of the taskset itself but rather tracking the things its excluded
on so I renamed it to be TaskSetExcludeList.  Everything else I tried to
use the context and in most cases excluded made sense. It made more
sense to me then blocked since you are basically excluding those
executors and nodes from scheduling tasks on them. Then can be
unexcluded later after timeouts and such. The configs I changed the name
to use excludeOnFailure which I thought explained it.
I unfortunately couldn't get rid of some of them because its part of the
event listener and history files.  To keep backwards compatibility I
kept the events and some of the parsing so that the history server would
still properly read older history files.  It is not forward compatible
though - meaning a new application write the "Excluded" events so the
older history server won't properly read display them as being
blacklisted.
A few of the files below are showing up as deleted and recreated even
though I did a git mv on them. I'm not sure why.
### Why are the changes needed?
get rid of problematic language
### Does this PR introduce _any_ user-facing change?
Config name changes but the old configs still work but are deprecated.
### How was this patch tested?
updated tests and also manually tested the UI changes and manually
tested the history server reading older versions of history files and
vice versa.
Closes #29906 from tgravescs/SPARK-32037.
Lead-authored-by: Thomas Graves <tgraves@nvidia.com> Co-authored-by:
Thomas Graves <tgraves@apache.org> Signed-off-by: Thomas Graves
<tgraves@apache.org>
(commit: 72ad9dc)
The file was modifiedresource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/SparkConf.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/deploy/history/HistoryAppStatusStore.scala (diff)
The file was addedcore/src/test/resources/HistoryServerExpectations/executor_node_excludeOnFailure_expectation.json
The file was modifiedcore/src/main/scala/org/apache/spark/status/LiveEntity.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/status/AppStatusListener.scala (diff)
The file was addedcore/src/main/scala/org/apache/spark/scheduler/TaskSetExcludeList.scala
The file was removedcore/src/test/scala/org/apache/spark/scheduler/TaskSetBlacklistSuite.scala
The file was modifiedcore/src/main/scala/org/apache/spark/status/AppStatusSource.scala (diff)
The file was modifiedresource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala (diff)
The file was removedcore/src/test/resources/HistoryServerExpectations/blacklisting_for_stage_expectation.json
The file was modifiedcore/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala (diff)
The file was addedcore/src/test/resources/HistoryServerExpectations/excludeOnFailure_node_for_stage_expectation.json
The file was modifiedresource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackend.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/scheduler/SparkListenerBus.scala (diff)
The file was modifiedcore/src/test/resources/HistoryServerExpectations/stage_with_accumulable_json_expectation.json (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/ui/ToolTips.scala (diff)
The file was addedcore/src/test/scala/org/apache/spark/scheduler/TaskSetExcludelistSuite.scala
The file was addedcore/src/test/scala/org/apache/spark/scheduler/HealthTrackerIntegrationSuite.scala
The file was modifiedcore/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala (diff)
The file was modifiedcore/src/test/scala/org/apache/spark/deploy/StandaloneDynamicAllocationSuite.scala (diff)
The file was modifiedcore/src/test/scala/org/apache/spark/scheduler/CoarseGrainedSchedulerBackendSuite.scala (diff)
The file was modifiedresource-managers/yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnSchedulerBackend.scala (diff)
The file was modifiedresource-managers/yarn/src/test/scala/org/apache/spark/scheduler/cluster/YarnSchedulerBackendSuite.scala (diff)
The file was modifiedcore/src/main/resources/org/apache/spark/ui/static/executorspage.js (diff)
The file was modifiedcore/src/test/resources/HistoryServerExpectations/executor_resource_information_expectation.json (diff)
The file was modifiedcore/src/test/scala/org/apache/spark/ExecutorAllocationManagerSuite.scala (diff)
The file was removedcore/src/test/resources/HistoryServerExpectations/executor_node_blacklisting_unblacklisting_expectation.json
The file was modifiedcore/src/main/scala/org/apache/spark/TaskEndReason.scala (diff)
The file was modifiedcore/src/test/resources/HistoryServerExpectations/executor_list_json_expectation.json (diff)
The file was modifiedcore/src/test/scala/org/apache/spark/status/AppStatusListenerSuite.scala (diff)
The file was modifiedcore/src/test/scala/org/apache/spark/deploy/history/BasicEventFilterSuite.scala (diff)
The file was removedresource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocatorBlacklistTracker.scala
The file was removedresource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/YarnAllocatorBlacklistTrackerSuite.scala
The file was modifiedcore/src/main/scala/org/apache/spark/status/api/v1/api.scala (diff)
The file was addedcore/src/test/resources/HistoryServerExpectations/executor_node_excludeOnFailure_unexcluding_expectation.json
The file was modifiedcore/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedClusterMessage.scala (diff)
The file was modifiedcore/src/test/scala/org/apache/spark/deploy/history/HistoryServerSuite.scala (diff)
The file was removedcore/src/main/scala/org/apache/spark/scheduler/TaskSetBlacklist.scala
The file was modifiedcore/src/test/scala/org/apache/spark/util/JsonProtocolSuite.scala (diff)
The file was modifiedcore/src/main/resources/org/apache/spark/ui/static/stagepage.js (diff)
The file was addedresource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocatorNodeHealthTracker.scala
The file was modifiedcore/src/main/scala/org/apache/spark/scheduler/SparkListener.scala (diff)
The file was modifiedcore/src/main/java/org/apache/spark/SparkFirehoseListener.java (diff)
The file was addedcore/src/test/resources/HistoryServerExpectations/excludeOnFailure_for_stage_expectation.json
The file was modifiedcore/src/test/scala/org/apache/spark/serializer/KryoSerializerDistributedSuite.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/internal/config/package.scala (diff)
The file was addedcore/src/test/scala/org/apache/spark/scheduler/HealthTrackerSuite.scala
The file was modifieddocs/running-on-yarn.md (diff)
The file was modifiedcore/src/main/resources/org/apache/spark/ui/static/executorspage-template.html (diff)
The file was modifiedcore/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala (diff)
The file was modifiedcore/src/test/scala/org/apache/spark/deploy/history/EventLogFileCompactorSuite.scala (diff)
The file was modifieddocs/monitoring.md (diff)
The file was removedcore/src/test/scala/org/apache/spark/scheduler/BlacklistIntegrationSuite.scala
The file was modifiedcore/src/test/resources/HistoryServerExpectations/one_stage_attempt_json_expectation.json (diff)
The file was removedcore/src/main/scala/org/apache/spark/scheduler/BlacklistTracker.scala
The file was modifiedresource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/config.scala (diff)
The file was addedcore/src/main/scala/org/apache/spark/scheduler/HealthTracker.scala
The file was removedcore/src/test/resources/HistoryServerExpectations/executor_node_blacklisting_expectation.json
The file was modifiedcore/src/main/scala/org/apache/spark/scheduler/ExecutorFailuresInTaskSet.scala (diff)
The file was modifiedcore/src/test/scala/org/apache/spark/scheduler/TaskSetManagerSuite.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/deploy/history/BasicEventFilterBuilder.scala (diff)
The file was modifiedcore/src/test/scala/org/apache/spark/HeartbeatReceiverSuite.scala (diff)
The file was modifiedcore/src/test/scala/org/apache/spark/status/api/v1/ExecutorSummarySuite.scala (diff)
The file was modifiedcore/src/main/resources/org/apache/spark/ui/static/stagespage-template.html (diff)
The file was modifiedcore/src/test/resources/HistoryServerExpectations/executor_memory_usage_expectation.json (diff)
The file was removedcore/src/test/scala/org/apache/spark/scheduler/BlacklistTrackerSuite.scala
The file was modifiedresource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/YarnAllocatorSuite.scala (diff)
The file was modifiedcore/src/test/resources/HistoryServerExpectations/one_stage_json_expectation.json (diff)
The file was modifiedresource-managers/mesos/src/test/scala/org/apache/spark/scheduler/cluster/mesos/MesosCoarseGrainedSchedulerBackendSuite.scala (diff)
The file was removedcore/src/test/resources/HistoryServerExpectations/blacklisting_node_for_stage_expectation.json
The file was modifiedcore/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala (diff)
The file was modifiedcore/src/test/resources/HistoryServerExpectations/executor_list_with_executor_metrics_json_expectation.json (diff)
The file was addedresource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/YarnAllocatorHealthTrackerSuite.scala
The file was modifiedcore/src/test/resources/HistoryServerExpectations/stage_with_peak_metrics_expectation.json (diff)
The file was modifieddocs/configuration.md (diff)
The file was modifiedresource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosCoarseGrainedSchedulerBackend.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala (diff)