1. [SPARK-33281][SQL] Return SQL schema instead of Catalog string from the (commit: b409025) (details)
  2. [SPARK-33263][SS] Configurable StateStore compression codec (commit: 056b622) (details)
Commit b409025641133fe7f352de4beaa2c0b037be3f56 by gurwls223
[SPARK-33281][SQL] Return SQL schema instead of Catalog string from the
`SchemaOfCsv` expression
### What changes were proposed in this pull request? Return schema in
SQL format instead of Catalog string from the SchemaOfCsv expression.
### Why are the changes needed? To unify output of the
`schema_of_json()` and `schema_of_csv()`.
### Does this PR introduce _any_ user-facing change? Yes, they can but
`schema_of_csv()` is usually used in combination with `from_csv()`, so,
the format of schema shouldn't be much matter.
> SELECT schema_of_csv('1,abc');
> SELECT schema_of_csv('1,abc');
STRUCT<`_c0`: INT, `_c1`: STRING>
### How was this patch tested? By existing test suites
`CsvFunctionsSuite` and `CsvExpressionsSuite`.
Closes #30180 from MaxGekk/schema_of_csv-sql-schema.
Authored-by: Max Gekk <> Signed-off-by: HyukjinKwon
(commit: b409025)
The file was modifiedR/pkg/tests/fulltests/test_sparkSQL.R (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/CsvFunctionsSuite.scala (diff)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CsvExpressionsSuite.scala (diff)
The file was modifieddocs/ (diff)
The file was modifiedsql/core/src/test/resources/sql-tests/results/csv-functions.sql.out (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/csvExpressions.scala (diff)
The file was modifiedpython/pyspark/sql/ (diff)
Commit 056b62264b024c83840f2bf23f4bb9cabd13e136 by dhyun
[SPARK-33263][SS] Configurable StateStore compression codec
### What changes were proposed in this pull request?
This patch proposes to make StateStore compression codec configurable.
### Why are the changes needed?
Currently the compression codec of StateStore is not configurable and
hard-coded to be lz4. It is better if we can follow Spark other modules
to configure the compression codec of StateStore. For example, we can
choose zstd codec and zstd is configurable with different compression
### Does this PR introduce _any_ user-facing change?
Yes, after this change users can config different codec for StateStore.
### How was this patch tested?
Unit test.
Closes #30162 from viirya/SPARK-33263.
Authored-by: Liang-Chi Hsieh <> Signed-off-by: Dongjoon
Hyun <>
(commit: 056b622)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStoreConf.scala (diff)
The file was addedsql/core/src/test/resources/structured-streaming/checkpoint-version-3.0.0-streaming-statestore-codec/metadata
The file was addedsql/core/src/test/resources/structured-streaming/checkpoint-version-3.0.0-streaming-statestore-codec/state/0/0/
The file was addedsql/core/src/test/resources/structured-streaming/checkpoint-version-3.0.0-streaming-statestore-codec/commits/0
The file was addedsql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/StateStoreCompatibilitySuite.scala
The file was addedsql/core/src/test/resources/structured-streaming/checkpoint-version-3.0.0-streaming-statestore-codec/offsets/0
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/streaming/OffsetSeq.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/StateStoreSuite.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreProvider.scala (diff)