Changes

Summary

  1. [SPARK-33358][SQL] Return code when command process failed (commit: 1ae6d64) (details)
  2. [SPARK-33254][PYTHON][DOCS] Migration to NumPy documentation style in (commit: 52073ef) (details)
  3. [SPARK-33451][DOCS] Change to (commit: 0933f1c) (details)
  4. [SPARK-33253][PYTHON][DOCS] Migration to NumPy documentation style in (commit: 236c6c9) (details)
  5. [SPARK-33288][YARN][FOLLOW-UP][TEST-HADOOP2.7] Fix type mismatch error (commit: f660946) (details)
  6. [SPARK-33454][INFRA] Add GitHub Action job for Hadoop 2 (commit: 10105b5) (details)
  7. [SPARK-33456][SQL][TEST] Add end-to-end test for subexpression (commit: d4cf148) (details)
  8. [SPARK-33394][SQL][TESTS] Throw `NoSuchNamespaceException` for not (commit: 4e5d2e0) (details)
  9. [SPARK-33458][SQL] Hive partition pruning support Contains, StartsWith (commit: cdcbdae) (details)
  10. [SPARK-30985][K8S] Support propagating SPARK_CONF_DIR files to driver (commit: 8615f35) (details)
  11. [SPARK-33456][SQL][TEST][FOLLOWUP] Fix SUBEXPRESSION_ELIMINATION_ENABLED (commit: 10b011f) (details)
  12. [MINOR][SQL][DOCS] Fix a reference to (commit: 71a29b2) (details)
  13. [SPARK-33140][SQL][FOLLOW-UP] Revert code that not use passed-in (commit: aa508fc) (details)
  14. [SPARK-33389][SQL] Make internal classes of SparkSession always using (commit: dfa6fb4) (details)
  15. [SPARK-33453][SQL][TESTS] Unify v1 and v2 SHOW PARTITIONS tests (commit: 6883f29) (details)
  16. [SPARK-33460][SQL] Accessing map values should fail if key is not found (commit: b5eca18) (details)
Commit 1ae6d64b5fb9c20e41deffaf670a42b0b545477f by dongjoon
[SPARK-33358][SQL] Return code when command process failed
Exit Spark SQL CLI processing loop if one of the commands (sub sql
statement) process failed
This is a regression at Apache Spark 3.0.0.
```
$ cat 1.sql select * from nonexistent_table; select 2;
```
**Apache Spark 2.4.7**
``` spark-2.4.7-bin-hadoop2.7:$ bin/spark-sql -f 1.sql 20/11/15 16:14:38
WARN NativeCodeLoader: Unable to load native-hadoop library for your
platform... using builtin-java classes where applicable Error in query:
Table or view not found: nonexistent_table; line 1 pos 14
```
**Apache Spark 3.0.1**
```
$ bin/spark-sql -f 1.sql Error in query: Table or view not found:
nonexistent_table; line 1 pos 14;
'Project [*]
+- 'UnresolvedRelation [nonexistent_table]
2 Time taken: 2.786 seconds, Fetched 1 row(s)
```
**Apache Hive 1.2.2**
``` apache-hive-1.2.2-bin:$ bin/hive -f 1.sql
Logging initialized using configuration in
jar:file:/Users/dongjoon/APACHE/hive-release/apache-hive-1.2.2-bin/lib/hive-common-1.2.2.jar!/hive-log4j.properties
FAILED: SemanticException [Error 10001]: Line 1:14 Table not found
'nonexistent_table'
```
Yes. This is a fix of regression.
Pass the UT.
Closes #30263 from artiship/SPARK-33358.
Authored-by: artiship <meilziner@gmail.com> Signed-off-by: Dongjoon Hyun
<dongjoon@apache.org>
(commit: 1ae6d64)
The file was modifiedsql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkSQLCLIDriver.scala (diff)
Commit 52073ef8acae84ff94680af71bb3172237d4783b by gurwls223
[SPARK-33254][PYTHON][DOCS] Migration to NumPy documentation style in
Core (pyspark.*, pyspark.resource.*, etc.)
### What changes were proposed in this pull request?
This PR proposes migration of Core to NumPy documentation style.
### Why are the changes needed?
To improve documentation style.
### Does this PR introduce _any_ user-facing change?
Yes, this changes both rendered HTML docs and console representation
(SPARK-33243).
### How was this patch tested?
dev/lint-python and manual inspection.
Closes #30320 from zero323/SPARK-33254.
Authored-by: zero323 <mszymkiewicz@gmail.com> Signed-off-by: HyukjinKwon
<gurwls223@apache.org>
(commit: 52073ef)
The file was modifiedpython/pyspark/rdd.py (diff)
The file was modifiedpython/pyspark/resource/profile.py (diff)
The file was modifiedpython/pyspark/taskcontext.py (diff)
The file was modifiedpython/pyspark/statcounter.py (diff)
The file was modifiedpython/pyspark/testing/streamingutils.py (diff)
The file was modifiedpython/pyspark/accumulators.py (diff)
The file was modifiedpython/docs/source/reference/pyspark.rst (diff)
The file was modifiedpython/pyspark/util.py (diff)
The file was modifiedpython/pyspark/broadcast.py (diff)
The file was modifiedpython/pyspark/profiler.py (diff)
The file was modifiedpython/pyspark/worker.py (diff)
The file was modifiedpython/pyspark/__init__.py (diff)
The file was modifiedpython/pyspark/shuffle.py (diff)
The file was modifiedpython/pyspark/testing/mlutils.py (diff)
The file was modifiedpython/pyspark/install.py (diff)
The file was modifiedpython/pyspark/serializers.py (diff)
The file was modifiedpython/pyspark/conf.py (diff)
The file was modifiedpython/pyspark/context.py (diff)
The file was modifiedpython/pyspark/resource/requests.py (diff)
The file was modifiedpython/pyspark/resource/information.py (diff)
The file was modifiedpython/pyspark/testing/utils.py (diff)
The file was modifiedpython/pyspark/java_gateway.py (diff)
Commit 0933f1c6c27c7a087c8dbe485bd08371ce5f5695 by gurwls223
[SPARK-33451][DOCS] Change to
'spark.sql.adaptive.skewJoin.skewedPartitionThresholdInBytes' in
documentation
### What changes were proposed in this pull request?
In the 'Optimizing Skew Join' section of the following two pages: 1.
[https://spark.apache.org/docs/3.0.0/sql-performance-tuning.html](https://spark.apache.org/docs/3.0.0/sql-performance-tuning.html)
2.
[https://spark.apache.org/docs/3.0.1/sql-performance-tuning.html](https://spark.apache.org/docs/3.0.1/sql-performance-tuning.html)
The configuration 'spark.sql.adaptive.skewedPartitionThresholdInBytes'
should be changed to
'spark.sql.adaptive.skewJoin.skewedPartitionThresholdInBytes', The
former is missing the 'skewJoin'.
### Why are the changes needed?
To document the correct name of configuration
### Does this PR introduce _any_ user-facing change?
Yes, this is a user-facing doc change.
### How was this patch tested?
Jenkins / CI builds in this PR.
Closes #30376 from aof00/doc_change.
Authored-by: aof00 <x14562573449@gmail.com> Signed-off-by: HyukjinKwon
<gurwls223@apache.org>
(commit: 0933f1c)
The file was modifieddocs/sql-performance-tuning.md (diff)
Commit 236c6c9f7c932fdcd285988e2dff1185187f6617 by gurwls223
[SPARK-33253][PYTHON][DOCS] Migration to NumPy documentation style in
Streaming (pyspark.streaming.*)
### What changes were proposed in this pull request?
This PR proposes to migrate to [NumPy documentation
style](https://numpydoc.readthedocs.io/en/latest/format.html), see also
[SPARK-33243](https://issues.apache.org/jira/browse/SPARK-33243).
### Why are the changes needed?
For better documentation as text itself, and generated HTMLs
### Does this PR introduce _any_ user-facing change?
Yes, they will see a better format of HTMLs, and better text format. See
[SPARK-33243](https://issues.apache.org/jira/browse/SPARK-33243).
### How was this patch tested?
Manually tested via running ./dev/lint-python.
Closes #30346 from itholic/SPARK-32085.
Lead-authored-by: itholic <haejoon309@naver.com> Co-authored-by: Hyukjin
Kwon <gurwls223@gmail.com> Signed-off-by: HyukjinKwon
<gurwls223@apache.org>
(commit: 236c6c9)
The file was modifiedpython/pyspark/streaming/dstream.pyi (diff)
The file was modifiedpython/pyspark/streaming/kinesis.py (diff)
The file was modifiedpython/pyspark/streaming/util.py (diff)
The file was modifiedpython/pyspark/streaming/kinesis.pyi (diff)
The file was modifiedpython/pyspark/streaming/context.pyi (diff)
The file was modifiedpython/pyspark/streaming/dstream.py (diff)
The file was modifiedpython/pyspark/streaming/context.py (diff)
Commit f660946ef29a985b53ceff948001a9db64279d67 by yumwang
[SPARK-33288][YARN][FOLLOW-UP][TEST-HADOOP2.7] Fix type mismatch error
### What changes were proposed in this pull request?
This pr fix type mismatch error:
```
[error]
/home/jenkins/workspace/spark-master-test-sbt-hadoop-2.7-hive-2.3/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala:320:52:
type mismatch;
[error]  found   : Long
[error]  required: Int
[error]         Resource.newInstance(resourcesWithDefaults.totalMemMiB,
resourcesWithDefaults.cores)
[error]                                                    ^
[error] one error found
```
### Why are the changes needed?
Fix compile issue.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Existing test.
Closes #30375 from wangyum/SPARK-33288.
Authored-by: Yuming Wang <yumwang@ebay.com> Signed-off-by: Yuming Wang
<yumwang@ebay.com>
(commit: f660946)
The file was modifiedresource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala (diff)
Commit 10105b555d0f51a680302b438d0fb06b463bd530 by gurwls223
[SPARK-33454][INFRA] Add GitHub Action job for Hadoop 2
### What changes were proposed in this pull request?
This PR aims to protect `Hadoop 2.x` profile compilation in Apache Spark
3.1+.
### Why are the changes needed?
Since Apache Spark 3.1+ switch our default profile to Hadoop 3, we had
better prevent at least compilation error with `Hadoop 2.x` profile at
the PR review phase. Although this is an additional workload, it will
finish quickly because it's compilation only.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Pass the GitHub Action.
- This should be merged after https://github.com/apache/spark/pull/30375
.
Closes #30378 from dongjoon-hyun/SPARK-33454.
Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by:
HyukjinKwon <gurwls223@apache.org>
(commit: 10105b5)
The file was modified.github/workflows/build_and_test.yml (diff)
Commit d4cf1483fd940fad1ffcc8910fb10f6b894d2f68 by gurwls223
[SPARK-33456][SQL][TEST] Add end-to-end test for subexpression
elimination
### What changes were proposed in this pull request?
This patch proposes to add end-to-end test for subexpression
elimination.
### Why are the changes needed?
We have subexpression elimination feature for expression evaluation but
we don't have end-to-end tests for the feature. We should have one to
make sure we don't break it.
### Does this PR introduce _any_ user-facing change?
No, dev only.
### How was this patch tested?
Unit tests.
Closes #30381 from viirya/SPARK-33456.
Authored-by: Liang-Chi Hsieh <viirya@gmail.com> Signed-off-by:
HyukjinKwon <gurwls223@apache.org>
(commit: d4cf148)
The file was addedsql/core/src/test/resources/sql-tests/results/subexp-elimination.sql.out
The file was addedsql/core/src/test/resources/sql-tests/inputs/subexp-elimination.sql
Commit 4e5d2e06950ab60d2442a43e0656d47aca4a87f7 by wenchen
[SPARK-33394][SQL][TESTS] Throw `NoSuchNamespaceException` for not
existing namespace in `InMemoryTableCatalog.listTables()`
### What changes were proposed in this pull request? Throw
`NoSuchNamespaceException` in `listTables()` of the custom test catalog
`InMemoryTableCatalog` if the passed namespace doesn't exist.
### Why are the changes needed? 1. To align behavior of V2
`InMemoryTableCatalog` to V1 session catalog. 2. To distinguish two
situations:
   1. A namespace **does exist** but does not contain any tables. In
that case, `listTables()` returns empty result.
   2. A namespace **does not exist**. `listTables()` throws
`NoSuchNamespaceException` in this case.
### Does this PR introduce _any_ user-facing change? Yes. For example,
`SHOW TABLES` returns empty result before the changes.
### How was this patch tested? By running V1/V2 ShowTablesSuites.
Closes #30358 from MaxGekk/show-tables-in-not-existing-namespace.
Authored-by: Max Gekk <max.gekk@gmail.com> Signed-off-by: Wenchen Fan
<wenchen@databricks.com>
(commit: 4e5d2e0)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/TableCatalogSuite.scala (diff)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/connector/InMemoryTableCatalog.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/command/ShowTablesSuiteBase.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/command/v1/ShowTablesSuite.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/command/v2/ShowTablesSuite.scala (diff)
Commit cdcbdaeb0d78e083d12e7c33bb4e5d0746635bcc by wenchen
[SPARK-33458][SQL] Hive partition pruning support Contains, StartsWith
and EndsWith predicate
### What changes were proposed in this pull request?
This pr add support Hive partition pruning on `Contains`, `StartsWith`
and `EndsWith` predicate.
### Why are the changes needed?
Improve query performance.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Unit test.
Closes #30383 from wangyum/SPARK-33458.
Authored-by: Yuming Wang <yumwang@ebay.com> Signed-off-by: Wenchen Fan
<wenchen@databricks.com>
(commit: cdcbdae)
The file was modifiedsql/hive/src/test/scala/org/apache/spark/sql/hive/client/HivePartitionFilteringSuite.scala (diff)
The file was modifiedsql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala (diff)
Commit 8615f354a41d0b5ea8e24670b332c8023f9a326b by dongjoon
[SPARK-30985][K8S] Support propagating SPARK_CONF_DIR files to driver
and executor pods
### What changes were proposed in this pull request? This is an
improvement, we mount all the user specific configuration files(except
the templates and spark properties files) from `SPARK_CONF_DIR` at the
point of spark-submit, to both executor and driver pods. Currently, only
`spark.properties` is mounted, only on driver.
### Why are the changes needed?
`SPARK_CONF_DIR` hosts several configuration files, for example, 1)
`spark-defaults.conf` - containing all the spark properties. 2)
`log4j.properties` - Logger configuration. 3) `core-site.xml` - Hadoop
related configuration. 4) `fairscheduler.xml` - Spark's fair scheduling
policy at the job level. 5) `metrics.properties` - Spark metrics. 6) Any
user specific - library or framework specific configuration file.
At the moment, we can cannot propagate these files to the driver and
executor configuration directory.
There is a design doc, with more details, and this patch is currently
providing a reference implementation. Please take a look at the doc and
comment, how we can improve. [google docs link to the
doc](https://bit.ly/spark-30985)
### Further scope Support user defined configMaps.
### Does this PR introduce any user-facing change? Yes, previously the
user configuration files(e.g. hdfs-site.xml, log4j.properties etc...)
were not propagated by default, now after this patch it is propagated to
driver and executor pods' `SPARK_CONF_DIR`.
### How was this patch tested? Added tests.
Also manually tested, by deploying it to a minikube cluster and
observing the additional configuration files were present, and taking
effect. For example, changes to log4j.properties was properly applied to
executors.
Closes #27735 from ScrapCodes/SPARK-30985/spark-conf-k8s-propagate.
Authored-by: Prashant Sharma <prashsh1@in.ibm.com> Signed-off-by:
Dongjoon Hyun <dongjoon@apache.org>
(commit: 8615f35)
The file was addedresource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/KubernetesClientUtils.scala
The file was modifiedresource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackend.scala (diff)
The file was modifiedresource-managers/kubernetes/core/src/test/scala/org/apache/spark/deploy/k8s/features/BasicExecutorFeatureStepSuite.scala (diff)
The file was modifiedresource-managers/kubernetes/core/src/test/scala/org/apache/spark/deploy/k8s/submit/ClientSuite.scala (diff)
The file was modifiedresource-managers/kubernetes/docker/src/main/dockerfiles/spark/entrypoint.sh (diff)
The file was modifiedresource-managers/kubernetes/core/src/test/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackendSuite.scala (diff)
The file was modifiedresource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Constants.scala (diff)
The file was modifiedresource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicExecutorFeatureStep.scala (diff)
The file was modifiedresource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/KubernetesClientApplication.scala (diff)
The file was modifiedresource-managers/kubernetes/core/src/test/scala/org/apache/spark/deploy/k8s/Fabric8Aliases.scala (diff)
Commit 10b011f837756cb3cfdb17b7bb593f1636880d10 by gurwls223
[SPARK-33456][SQL][TEST][FOLLOWUP] Fix SUBEXPRESSION_ELIMINATION_ENABLED
config name
### What changes were proposed in this pull request?
To fix wrong config name in `subexp-elimination.sql`.
### Why are the changes needed?
`CONFIG_DIM` should use config name's key.
### Does this PR introduce _any_ user-facing change?
No, dev only.
### How was this patch tested?
Unit test.
Closes #30384 from viirya/SPARK-33456-followup.
Authored-by: Liang-Chi Hsieh <viirya@gmail.com> Signed-off-by:
HyukjinKwon <gurwls223@apache.org>
(commit: 10b011f)
The file was modifiedsql/core/src/test/resources/sql-tests/inputs/subexp-elimination.sql (diff)
Commit 71a29b2eca0b5f3c7bd51259796ca007774d8fd6 by gurwls223
[MINOR][SQL][DOCS] Fix a reference to
`spark.sql.sources.useV1SourceList`
### What changes were proposed in this pull request? Replace
`spark.sql.sources.write.useV1SourceList` by
`spark.sql.sources.useV1SourceList` in the comment for
`CatalogManager.v2SessionCatalog()`.
### Why are the changes needed? To have correct comments.
### Does this PR introduce _any_ user-facing change? No
### How was this patch tested? By running `./dev/scalastyle`.
Closes #30385 from MaxGekk/fix-comment-useV1SourceList.
Authored-by: Max Gekk <max.gekk@gmail.com> Signed-off-by: HyukjinKwon
<gurwls223@apache.org>
(commit: 71a29b2)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogManager.scala (diff)
Commit aa508fcc0328366c578f65a4d4b8a7c7358140ea by wenchen
[SPARK-33140][SQL][FOLLOW-UP] Revert code that not use passed-in
SparkSession to get SQLConf
### What changes were proposed in this pull request?
Revert code that does not use passed-in SparkSession to get SQLConf in
[SPARK-33140]. The change scope of [SPARK-33140] change passed-in
SQLConf instance and place using SparkSession to get SQLConf to be
unified to use SQLConf.get. And the code reverted in the patch, the
passed-in SparkSession was not about to get SQLConf, but using its
catalog, it's better to be consistent.
### Why are the changes needed?
Potential regression bug.
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
Existing UT.
Closes #30364 from leanken/leanken-SPARK-33140.
Authored-by: xuewei.linxuewei <xuewei.linxuewei@alibaba-inc.com>
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
(commit: aa508fc)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala (diff)
The file was modifiedsql/hive/src/main/scala/org/apache/spark/sql/hive/HiveSessionStateBuilder.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/internal/BaseSessionStateBuilder.scala (diff)
The file was modifiedsql/hive/src/main/scala/org/apache/spark/sql/hive/HiveStrategies.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/CoalesceShufflePartitions.scala (diff)
The file was modifiedsql/hive/src/main/scala/org/apache/spark/sql/hive/execution/PruneHiveTablePartitions.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala (diff)
The file was modifiedsql/hive/src/test/scala/org/apache/spark/sql/hive/execution/PruneHiveTablePartitionsSuite.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/QueryExecution.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/connector/V2CommandsCaseSensitivitySuite.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FallBackFileSourceV2.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/subquery.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/dynamicpruning/PlanDynamicPruningFilters.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/datasources/rules.scala (diff)
Commit dfa6fb46f4238792bff6a0201da201be1b42620e by wenchen
[SPARK-33389][SQL] Make internal classes of SparkSession always using
active SQLConf
### What changes were proposed in this pull request?
This PR makes internal classes of SparkSession always using active
SQLConf. We should remove all `conf: SQLConf`s from ctor-parameters of
this classes (`Analyzer`, `SparkPlanner`, `SessionCatalog`,
`CatalogManager` `SparkSqlParser` and etc.) and use  `SQLConf.get`
instead.
### Why are the changes needed?
Code refine.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Existing test
Closes #30299 from luluorta/SPARK-33389.
Authored-by: luluorta <luluorta@gmail.com> Signed-off-by: Wenchen Fan
<wenchen@databricks.com>
(commit: dfa6fb4)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/StarSchemaDetection.scala (diff)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/AnalysisSuite.scala (diff)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/plans/PlanTest.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/QueryPlan.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/V2SessionCatalog.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/V2SessionCatalogSuite.scala (diff)
The file was modifiedsql/hive/src/main/scala/org/apache/spark/sql/hive/HiveSessionCatalog.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/rules/Rule.scala (diff)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/DataSourceV2AnalysisSuite.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/PropagateEmptyRelation.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/ParseDriver.scala (diff)
The file was modifiedsql/hive/src/main/scala/org/apache/spark/sql/hive/HiveSessionStateBuilder.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/internal/BaseSessionStateBuilder.scala (diff)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/BooleanSimplificationSuite.scala (diff)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/DecimalPrecisionSuite.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogManager.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/streaming/IncrementalExecution.scala (diff)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/CatalogManagerSuite.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/internal/VariableSubstitution.scala (diff)
The file was addedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/SQLConfHelper.scala
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/view.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala (diff)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/OptimizerStructuralIntegrityCheckerSuite.scala (diff)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/LookupFunctionsSuite.scala (diff)
The file was modifiedsql/hive/src/main/scala/org/apache/spark/sql/hive/TableReader.scala (diff)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalogSuite.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/command/CommandCheck.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/SparkPlanner.scala (diff)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/AnalysisExternalCatalogSuite.scala (diff)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/EliminateSortsBeforeRepartitionSuite.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/timeZoneAnalysis.scala (diff)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/AnalysisTest.scala (diff)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/RewriteDistinctAggregatesSuite.scala (diff)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/TableLookupCacheSuite.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/command/PlanResolutionSuite.scala (diff)
Commit 6883f29465f67b5c052a3e6d137bec250d5cc8ef by wenchen
[SPARK-33453][SQL][TESTS] Unify v1 and v2 SHOW PARTITIONS tests
### What changes were proposed in this pull request? 1. Move `SHOW
PARTITIONS` parsing tests to `ShowPartitionsParserSuite` 2. Place Hive
tests for `SHOW PARTITIONS` from `HiveCommandSuite` to the base test
suite `v1.ShowPartitionsSuiteBase`. This will allow to run the tests w/
and w/o Hive.
The changes follow the approach of
https://github.com/apache/spark/pull/30287.
### Why are the changes needed?
- The unification will allow to run common `SHOW PARTITIONS` tests for
both DSv1 and Hive DSv1, DSv2
- We can detect missing features and differences between DSv1 and DSv2
implementations.
### Does this PR introduce _any_ user-facing change? No
### How was this patch tested? By running:
- new test suites `build/sbt -Phive-2.3 -Phive-thriftserver
"test:testOnly *ShowPartitionsSuite"`
- and old one `build/sbt -Phive-2.3 -Phive-thriftserver "test:testOnly
org.apache.spark.sql.hive.execution.HiveCommandSuite"`
Closes #30377 from MaxGekk/unify-dsv1_v2-show-partitions-tests.
Authored-by: Max Gekk <max.gekk@gmail.com> Signed-off-by: Wenchen Fan
<wenchen@databricks.com>
(commit: 6883f29)
The file was modifiedsql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveCommandSuite.scala (diff)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/DDLParserSuite.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2SQLSuite.scala (diff)
The file was addedsql/core/src/test/scala/org/apache/spark/sql/execution/command/ShowPartitionsParserSuite.scala
The file was addedsql/core/src/test/scala/org/apache/spark/sql/execution/command/ShowPartitionsSuiteBase.scala
The file was addedsql/core/src/test/scala/org/apache/spark/sql/execution/command/v1/ShowPartitionsSuite.scala
The file was addedsql/core/src/test/scala/org/apache/spark/sql/execution/command/v2/ShowPartitionsSuite.scala
The file was addedsql/hive/src/test/scala/org/apache/spark/sql/hive/execution/command/ShowPartitionsSuite.scala
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLParserSuite.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/SQLViewSuite.scala (diff)
Commit b5eca18af050718ca23c0bb9c171f2352c171790 by wenchen
[SPARK-33460][SQL] Accessing map values should fail if key is not found
### What changes were proposed in this pull request?
Instead of returning NULL, throws runtime NoSuchElementException towards
invalid key accessing in map-like functions, such as element_at,
GetMapValue, when ANSI mode is on.
### Why are the changes needed?
For ANSI mode.
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
Added UT and Existing UT.
Closes #30386 from leanken/leanken-SPARK-33460.
Authored-by: xuewei.linxuewei <xuewei.linxuewei@alibaba-inc.com>
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
(commit: b5eca18)
The file was addedsql/core/src/test/resources/sql-tests/inputs/ansi/map.sql
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ProjectionOverSchema.scala (diff)
The file was modifieddocs/sql-ref-ansi-compliance.md (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeExtractors.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/SelectedField.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/ComplexTypes.scala (diff)
The file was addedsql/core/src/test/resources/sql-tests/results/ansi/map.sql.out
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CollectionExpressionsSuite.scala (diff)
The file was addedsql/core/src/test/resources/sql-tests/results/map.sql.out
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ComplexTypeSuite.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala (diff)
The file was addedsql/core/src/test/resources/sql-tests/inputs/map.sql