Changes

Summary

  1. [SPARK-32786][SQL][TEST] Improve performance for some slow DPP tests (commit: e693df2) (details)
  2. [SPARK-32772][SQL][FOLLOWUP] Remove legacy silent support mode for (commit: f1f7ae4) (details)
  3. [SPARK-32762][SQL][TEST] Enhance the verification of (commit: 1de272f) (details)
  4. [SPARK-32783][DOCS][PYTHON] Development - Testing PySpark (commit: 32d87c2) (details)
Commit e693df2a07da16208178df944c127160f018fd43 by yamamuro
[SPARK-32786][SQL][TEST] Improve performance for some slow DPP tests
### What changes were proposed in this pull request?
The whole `DynamicPartitionPruningSuite` takes about 2 min on my laptop
(either AE on or off). The slowest tests are `test("simple inner join
triggers DPP with mock-up tables")` and `test("cleanup any DPP filter
that isn't pushed down due to expression id clashes")`, which totally
take about 1 min.
We can reuse existing test tables or use smaller tables to reduce the
cost. After that, the two tests takes only about 1 sec in total, leading
to 2x speedup for the suite.
### Why are the changes needed?
To speedup DPP test suites.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Modified two existing tests.
Closes #29636 from wzhfy/improve_dpp_test.
Authored-by: Zhenhua Wang <wzh_zju@163.com> Signed-off-by: Takeshi
Yamamuro <yamamuro@apache.org>
(commit: e693df2)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/DynamicPartitionPruningSuite.scala (diff)
Commit f1f7ae420ee5cfc5755141d5ff25604bb78be465 by yumwang
[SPARK-32772][SQL][FOLLOWUP] Remove legacy silent support mode for
spark-sql CLI
### What changes were proposed in this pull request?
Remove legacy silent support mode for spark-sql CLI.
### Why are the changes needed?
https://github.com/apache/spark/pull/29619 add new silent mode. We can
remove legacy silent support mode.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Manual test:
``` spark-sql> LM-SHC-16508156:spark yumwang$ bin/spark-sql -S NOTE:
SPARK_PREPEND_CLASSES is set, placing locally compiled Spark classes
ahead of assembly. 20/09/03 09:06:12 WARN NativeCodeLoader: Unable to
load native-hadoop library for your platform... using builtin-java
classes where applicable Using Spark's default log4j profile:
org/apache/spark/log4j-defaults.properties Setting default log level to
"WARN". To adjust logging level use sc.setLogLevel(newLevel). For
SparkR, use setLogLevel(newLevel). 20/09/03 09:06:16 WARN HiveConf:
HiveConf of name hive.stats.jdbc.timeout does not exist 20/09/03
09:06:16 WARN HiveConf: HiveConf of name hive.stats.retries.wait does
not exist 20/09/03 09:06:19 WARN ObjectStore: Version information not
found in metastore. hive.metastore.schema.verification is not enabled so
recording the schema version 2.3.0 20/09/03 09:06:19 WARN ObjectStore:
setMetaStoreSchemaVersion called but recording version is disabled:
version = 2.3.0, comment = Set by MetaStore yumwang10.226.196.190
spark-sql> select * from test1; 1 spark-sql> select * from test1; 1
```
Closes #29631 from wangyum/SPARK-32772.
Authored-by: Yuming Wang <yumwang@ebay.com> Signed-off-by: Yuming Wang
<yumwang@ebay.com>
(commit: f1f7ae4)
The file was modifiedsql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkSQLCLIDriver.scala (diff)
Commit 1de272f98d0ff22d0dd151797f22b8faf310963a by yamamuro
[SPARK-32762][SQL][TEST] Enhance the verification of
ExpressionsSchemaSuite to sql-expression-schema.md
### What changes were proposed in this pull request?
`sql-expression-schema.md` automatically generated by
`ExpressionsSchemaSuite`, but only expressions entries are checked
in `ExpressionsSchemaSuite`. So if we manually modify the contents of
the file,  `ExpressionsSchemaSuite` does not necessarily guarantee the
correctness of the it some times. For example,
[Spark-24884](https://github.com/apache/spark/pull/27507) added
`regexp_extract_all`  expression support, and manually modify the
`sql-expression-schema.md` but not change the content of `Number of
queries` cause file content inconsistency.
Some additional checks have been added to `ExpressionsSchemaSuite` to
improve the correctness guarantee of `sql-expression-schema.md` as
follow:
- `Number of queries` should equals size of `expressions entries` in
`sql-expression-schema.md`
- `Number of expressions that missing example` should equals size of
`Expressions missing examples` in `sql-expression-schema.md`
- `MissExamples` from case should same as  `expectedMissingExamples`
from `sql-expression-schema.md`
### Why are the changes needed? Ensure the correctness of
`sql-expression-schema.md` content.
### Does this PR introduce _any_ user-facing change? No
### How was this patch tested? Enhanced ExpressionsSchemaSuite
Closes #29608 from LuciferYang/sql-expression-schema.
Authored-by: yangjie <yangjie@MacintoshdeMacBook-Pro.local>
Signed-off-by: Takeshi Yamamuro <yamamuro@apache.org>
(commit: 1de272f)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/ExpressionsSchemaSuite.scala (diff)
The file was modifiedsql/core/src/test/resources/sql-functions/sql-expression-schema.md (diff)
Commit 32d87c2b595b4aac2d9274424a43697299638f61 by gurwls223
[SPARK-32783][DOCS][PYTHON] Development - Testing PySpark
### What changes were proposed in this pull request?
This PR proposes to add a page to describe how to test PySpark. Note
that it avoids duplication of
https://spark.apache.org/developer-tools.html and it more aims to add
put the relevant links together.
I made a demo site to review more effectively:
https://hyukjin-spark.readthedocs.io/en/stable/development/testing.html
### Why are the changes needed?
To guide PySpark developers easily test.
### Does this PR introduce _any_ user-facing change?
Yes, it will adds a new documentation page.
### How was this patch tested?
Manually tested.
Closes #29634 from HyukjinKwon/SPARK-32783.
Authored-by: HyukjinKwon <gurwls223@apache.org> Signed-off-by:
HyukjinKwon <gurwls223@apache.org>
(commit: 32d87c2)
The file was addedpython/docs/source/development/testing.rst
The file was modifiedpython/docs/source/development/index.rst (diff)