Changes

Summary

  1. [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode (commit: 9a4c790) (details)
  2. [SPARK-33304][R][SQL] Add from_avro and to_avro functions to SparkR (commit: 56a8510) (details)
  3. [SPARK-33479][DOC] Make the API Key of DocSearch configurable (commit: 4267ca9) (details)
  4. [SPARK-33427][SQL][FOLLOWUP] Prevent test flakyness in (commit: 97d2cee) (details)
  5. [SPARK-33473][SQL] Extend interpreted subexpression elimination to other (commit: e518008) (details)
  6. [SPARK-31255][SQL][FOLLOWUP] Add missing license headers (commit: 66a7637) (details)
  7. [SPARK-33441][BUILD] Add unused-imports compilation check and remove all (commit: e3058ba) (details)
  8. [SPARK-27421][SQL] Fix filter for int column and value class (commit: 014e1fb) (details)
  9. [SPARK-33400][SQL] Normalize sameOrderExpressions in SortOrder to avoid (commit: 0b0fb70) (details)
  10. [SPARK-33483][INFRA][TESTS] Fix rat exclusion patterns and add a LICENSE (commit: d5e7bd0) (details)
  11. [SPARK-33183][SQL][FOLLOW-UP] Update rule RemoveRedundantSorts config (commit: ef2638c) (details)
  12. [SPARK-33472][SQL] Adjust RemoveRedundantSorts rule order (commit: a03c540) (details)
  13. [SPARK-33442][SQL] Change Combine Limit to Eliminate limit using max row (commit: 21b1350) (details)
  14. [SPARK-33045][SQL] Support build-in function like_all and fix (commit: 3695e99) (details)
Commit 9a4c79073bbd741f14e2babcbe140a7b4e56d558 by yamamuro
[SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
### What changes were proposed in this pull request?
In section 6.13 of the ANSI SQL standard, there are syntax rules for
valid combinations of the source and target data types.
![image](https://user-images.githubusercontent.com/1097932/98212874-17356f80-1ef9-11eb-8f2b-385f32db404a.png)
Comparing the ANSI CAST syntax rules with the current default behavior
of Spark:
![image](https://user-images.githubusercontent.com/1097932/98789831-b7870a80-23b7-11eb-9b5f-469a42e0ee4a.png)
To make Spark's ANSI mode more ANSI SQL Compatible,I propose to
disallow the following casting in ANSI mode:
``` TimeStamp <=> Boolean Date <=> Boolean Numeric <=> Timestamp Numeric
<=> Date Numeric <=> Binary String <=> Array String <=> Map String <=>
Struct
``` The following castings are considered invalid in ANSI SQL standard,
but they are quite straight forward. Let's Allow them for now
``` Numeric <=> Boolean String <=> Binary
```
### Why are the changes needed?
Better ANSI SQL compliance
### Does this PR introduce _any_ user-facing change?
Yes, the following castings will not be allowed in ANSI mode:
``` TimeStamp <=> Boolean Date <=> Boolean Numeric <=> Timestamp Numeric
<=> Date Numeric <=> Binary String <=> Array String <=> Map String <=>
Struct
```
### How was this patch tested?
Unit test
The ANSI Compliance doc preview:
![image](https://user-images.githubusercontent.com/1097932/98946017-2cd20880-24a8-11eb-8161-65749bfdd03a.png)
Closes #30260 from gengliangwang/ansiCanCast.
Authored-by: Gengliang Wang <gengliang.wang@databricks.com>
Signed-off-by: Takeshi Yamamuro <yamamuro@apache.org>
(commit: 9a4c790)
The file was modifieddocs/sql-ref-ansi-compliance.md (diff)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuite.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/sources/InsertSuite.scala (diff)
Commit 56a8510e19b3e0349e41d2a8903f4bf05ca00a28 by gurwls223
[SPARK-33304][R][SQL] Add from_avro and to_avro functions to SparkR
### What changes were proposed in this pull request?
Adds `from_avro` and `to_avro` functions to SparkR.
### Why are the changes needed?
Feature parity.
### Does this PR introduce _any_ user-facing change?
New functions exposed in SparkR API.
### How was this patch tested?
New unit tests.
Closes #30216 from zero323/SPARK-33304.
Authored-by: zero323 <mszymkiewicz@gmail.com> Signed-off-by: HyukjinKwon
<gurwls223@apache.org>
(commit: 56a8510)
The file was modifiedR/pkg/R/generics.R (diff)
The file was modifiedR/pkg/tests/fulltests/test_sparkSQL.R (diff)
The file was modifiedR/pkg/R/functions.R (diff)
The file was modifieddocs/sql-data-sources-avro.md (diff)
The file was modifiedR/pkg/NAMESPACE (diff)
The file was modifiedR/run-tests.sh (diff)
Commit 4267ca98faf377715f2eb19216165b0046a338e0 by yamamuro
[SPARK-33479][DOC] Make the API Key of DocSearch configurable
### What changes were proposed in this pull request?
Make the API key of DocSearch configurable and avoid hardcoding in the
HTML template
### Why are the changes needed?
After https://github.com/apache/spark/pull/30292, our Spark
documentation site supports searching. However, the default API key
always points to the latest release doc. We have to set different API
keys for different releases. Otherwise, the search results are always
based on the latest documentation(https://spark.apache.org/docs/latest/)
even when visiting the documentation of previous releases.
As per discussion in
https://github.com/apache/spark/pull/30292#issuecomment-725613417, we
should make the API key configurable and set different values for
different releases.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Manual test
Closes #30409 from gengliangwang/apiKey.
Authored-by: Gengliang Wang <gengliang.wang@databricks.com>
Signed-off-by: Takeshi Yamamuro <yamamuro@apache.org>
(commit: 4267ca9)
The file was modifieddocs/_layouts/global.html (diff)
The file was modifieddocs/_config.yml (diff)
Commit 97d2cee4af4ad8882334e2b680ab75dc73e29336 by viirya
[SPARK-33427][SQL][FOLLOWUP] Prevent test flakyness in
SubExprEvaluationRuntimeSuite
### What changes were proposed in this pull request?
This followup is to prevent possible test flakyness of
`SubExprEvaluationRuntimeSuite`.
### Why are the changes needed?
Because HashMap doesn't guarantee the order, in `proxyExpressions` the
proxy expression id is not deterministic. So in
`SubExprEvaluationRuntimeSuite` we should not test against it.
### Does this PR introduce _any_ user-facing change?
No, dev only.
### How was this patch tested?
Unit test.
Closes #30414 from viirya/SPARK-33427-followup.
Authored-by: Liang-Chi Hsieh <viirya@gmail.com> Signed-off-by: Liang-Chi
Hsieh <viirya@gmail.com>
(commit: 97d2cee)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/SubExprEvaluationRuntimeSuite.scala (diff)
Commit e518008ca9dc8a4950e2655ed9b35ce95ffe5acb by dongjoon
[SPARK-33473][SQL] Extend interpreted subexpression elimination to other
interpreted projections
### What changes were proposed in this pull request?
Similar to `InterpretedUnsafeProjection`, this patch proposes to extend
interpreted subexpression elimination to `InterpretedMutableProjection`
and `InterpretedSafeProjection`.
### Why are the changes needed?
Enabling subexpression elimination can improve the performance of
interpreted projections, as shown in `InterpretedUnsafeProjection`.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Unit test.
Closes #30406 from viirya/SPARK-33473.
Authored-by: Liang-Chi Hsieh <viirya@gmail.com> Signed-off-by: Dongjoon
Hyun <dongjoon@apache.org>
(commit: e518008)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/MutableProjectionSuite.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/InterpretedSafeProjection.scala (diff)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/codegen/GeneratedProjectionSuite.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/InterpretedMutableProjection.scala (diff)
Commit 66a76378cf9aa049c9281fc099721904942fa5ee by dongjoon
[SPARK-31255][SQL][FOLLOWUP] Add missing license headers
### What changes were proposed in this pull request?
Add missing license headers for new files added in #28027.
### Why are the changes needed?
To fix licenses.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
This is a purely non-functional change.
Closes #30415 from rdblue/license-headers.
Authored-by: Ryan Blue <blue@apache.org> Signed-off-by: Dongjoon Hyun
<dongjoon@apache.org>
(commit: 66a7637)
The file was modifiedsql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/SupportsMetadataColumns.java (diff)
The file was modifiedsql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/MetadataColumn.java (diff)
Commit e3058ba17cb4512537953eb4ded884e24ee93ba2 by gurwls223
[SPARK-33441][BUILD] Add unused-imports compilation check and remove all
unused-imports
### What changes were proposed in this pull request? This pr add a new
Scala compile arg to `pom.xml` to defense against new unused imports:
- `-Ywarn-unused-import` for Scala 2.12
- `-Wconf:cat=unused-imports:e` for Scala 2.13
The other fIles change are remove all unused imports in Spark code
### Why are the changes needed? Cleanup code and add guarantee to
defense against new unused imports
### Does this PR introduce _any_ user-facing change? No
### How was this patch tested? Pass the Jenkins or GitHub Action
Closes #30351 from LuciferYang/remove-imports-core-module.
Authored-by: yangjie01 <yangjie01@baidu.com> Signed-off-by: HyukjinKwon
<gurwls223@apache.org>
(commit: e3058ba)
The file was modifiedsql/hive/src/test/scala/org/apache/spark/sql/hive/HiveShowCreateTableSuite.scala (diff)
The file was modifiedmllib/src/test/scala/org/apache/spark/ml/tree/impl/RandomForestSuite.scala (diff)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/UnsupportedOperationsSuite.scala (diff)
The file was modifiedexternal/kafka-0-10-token-provider/src/main/scala/org/apache/spark/kafka010/KafkaTokenUtil.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/RuntimeConfig.scala (diff)
The file was modifiedexternal/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisInputDStream.scala (diff)
The file was modifiedresource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerUtils.scala (diff)
The file was modifiedresource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnSparkHadoopUtil.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingAggregationSuite.scala (diff)
The file was modifiedresource-managers/kubernetes/core/src/test/scala/org/apache/spark/deploy/k8s/features/KerberosConfDriverFeatureStepSuite.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/command/DataWritingCommand.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/status/AppStatusStore.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/DemoteBroadcastHashJoin.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/ObjectAggregationMap.scala (diff)
The file was modifiedsql/hive/src/test/scala/org/apache/spark/sql/hive/HiveSparkSubmitSuite.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/expressions/UserDefinedFunction.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/sources/BucketedReadSuite.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/basicPhysicalOperators.scala (diff)
The file was modifiedstreaming/src/main/scala/org/apache/spark/status/api/v1/streaming/ApiStreamingRootResource.scala (diff)
The file was modifiedresource-managers/kubernetes/core/src/test/scala/org/apache/spark/deploy/k8s/features/DriverServiceFeatureStepSuite.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/SortBasedAggregationIterator.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/PlannerSuite.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/datasources/ReadSchemaTest.scala (diff)
The file was modifiedresource-managers/kubernetes/core/src/test/scala/org/apache/spark/deploy/k8s/PodBuilderSuite.scala (diff)
The file was modifiedresource-managers/kubernetes/core/src/test/scala/org/apache/spark/deploy/k8s/features/HadoopConfDriverFeatureStepSuite.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/ArrayData.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/NormalizeFloatingNumbers.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveTestUtils.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/connector/V1ReadFallbackSuite.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/ComplexTypes.scala (diff)
The file was modifiedexternal/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaOffsetReader.scala (diff)
The file was modifiedexternal/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSourceProvider.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/Statistics.scala (diff)
The file was modifiedsql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/ui/ThriftServerPage.scala (diff)
The file was modifiedresource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/MountSecretsFeatureStep.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/streaming/StreamTest.scala (diff)
The file was modifiedcore/src/test/scala/org/apache/spark/scheduler/BarrierTaskContextSuite.scala (diff)
The file was modifiedmllib/src/main/scala/org/apache/spark/ml/feature/package.scala (diff)
The file was modifiedexternal/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchStream.scala (diff)
The file was modifiedstreaming/src/test/scala/org/apache/spark/streaming/ReceiverInputDStreamSuite.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/streaming/sources/memory.scala (diff)
The file was modifiedsql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveTableScanSuite.scala (diff)
The file was modifiedsql/hive/src/test/scala/org/apache/spark/sql/hive/HiveShimSuite.scala (diff)
The file was modifiedsql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/LogicalQueryStage.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/shuffle/sort/SortShuffleManager.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamExecution.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/bucketing/DisableUnnecessaryBucketedScan.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/higherOrderFunctions.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/CollectMetricsExec.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/columnar/ColumnStats.scala (diff)
The file was modifiedrepl/src/test/scala/org/apache/spark/repl/ReplSuite.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/command/cache.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingJoinSuite.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/storage/BlockManager.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/r/ArrowRRunner.scala (diff)
The file was modifiedstreaming/src/test/scala/org/apache/spark/streaming/TestSuiteBase.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/exchange/Exchange.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/DataFrameTimeWindowingSuite.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/Dataset.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/trees/TreeNode.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/encoders/ExpressionEncoder.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/streaming/HDFSMetadataLog.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/AnalysisHelper.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetInteroperabilitySuite.scala (diff)
The file was modifiedsql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveScriptTransformationSuite.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetPartitionDiscoverySuite.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVDataSource.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamingSymmetricHashJoinHelper.scala (diff)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/ScalaReflectionSuite.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/streaming/StreamingQueryManager.scala (diff)
The file was modifiedcore/src/test/scala/org/apache/spark/scheduler/TaskSetManagerSuite.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/streaming/FileStreamSinkLogSuite.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/status/KVUtils.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/ReplaceNullWithFalseInPredicate.scala (diff)
The file was modifiedexternal/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaBatch.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/TPCDSQueryBenchmark.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningAwareFileIndex.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcV2SchemaPruningSuite.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/TableCapabilityCheck.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/resource/ResourceAllocator.scala (diff)
The file was modifiedstreaming/src/main/scala/org/apache/spark/streaming/scheduler/ReceivedBlockTracker.scala (diff)
The file was modifiedmllib/src/test/scala/org/apache/spark/mllib/clustering/LDASuite.scala (diff)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeBlockSuite.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetOutputWriter.scala (diff)
The file was modifiedexternal/avro/src/main/scala/org/apache/spark/sql/avro/SchemaConverters.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala (diff)
The file was modifiedresource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsAllocator.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/bucketing/CoalesceBucketsInJoin.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/columnar/ColumnStatsSuite.scala (diff)
The file was modifiedcore/src/test/scala/org/apache/spark/StatusTrackerSuite.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/DataFrameWindowFramesSuite.scala (diff)
The file was modifiedcore/src/test/scala/org/apache/spark/deploy/StandaloneDynamicAllocationSuite.scala (diff)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/SimplifyCastsSuite.scala (diff)
The file was modifiedmllib/src/main/scala/org/apache/spark/ml/recommendation/ALS.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/FilterPushdownBenchmark.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStore.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaSuite.scala (diff)
The file was modifiedresource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/LocalityPreferredContainerPlacementStrategy.scala (diff)
The file was modifiedsql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkGetSchemasOperation.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/command/functions.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/rdd/ParallelCollectionRDD.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/hints.scala (diff)
The file was modifiedsql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/SymmetricHashJoinStateManager.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/IntegratedUDFTestUtils.scala (diff)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/DDLParserSuite.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2SQLSuite.scala (diff)
The file was modifiedstreaming/src/main/scala/org/apache/spark/streaming/State.scala (diff)
The file was modifiedcore/src/test/scala/org/apache/spark/util/UtilsSuite.scala (diff)
The file was modifiedcore/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/datasources/RowDataSourceStrategySuite.scala (diff)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/SubExprEvaluationRuntimeSuite.scala (diff)
The file was modifiedcore/src/test/scala/org/apache/spark/network/netty/NettyBlockTransferServiceSuite.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/WholeStageCodegenExec.scala (diff)
The file was modifiedsql/hive/src/test/scala/org/apache/spark/sql/hive/execution/PrunePartitionSuiteBase.scala (diff)
The file was modifiedmllib/src/test/scala/org/apache/spark/ml/util/PMMLReadWriteTest.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ShowTablePropertiesExec.scala (diff)
The file was modifiedresource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/backend/cloud/KubeConfigBackend.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/SparkPlanTest.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/internal/CatalogSuite.scala (diff)
The file was modifiedsql/hive/src/test/scala/org/apache/spark/sql/hive/HiveExternalCatalogSuite.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala (diff)
The file was modifiedsql/hive/src/test/scala/org/apache/spark/sql/hive/HiveUserDefinedTypeSuite.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/ParquetNestedPredicatePushDownBenchmark.scala (diff)
The file was modifiedcore/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/DatasetPrimitiveSuite.scala (diff)
The file was modifiedmllib/src/main/scala/org/apache/spark/mllib/classification/LogisticRegression.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/Column.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/RebaseDateTime.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/datasources/RecordReaderIterator.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/deploy/master/Master.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/network/BlockTransferService.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FallBackFileSourceV2.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/streaming/continuous/ContinuousSuite.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/streaming/FlatMapGroupsWithStateSuite.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/command/SetCommand.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/BaseScriptTransformationSuite.scala (diff)
The file was modifiedmllib/src/main/scala/org/apache/spark/ml/recommendation/TopByKeyAggregator.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/internal/SharedState.scala (diff)
The file was modifiedexternal/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisCheckpointer.scala (diff)
The file was modifiedresource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala (diff)
The file was modifiedexternal/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchSourceSuite.scala (diff)
The file was modifiedmllib/src/test/scala/org/apache/spark/ml/stat/SummarizerSuite.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/network/BlockDataManager.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/streaming/test/DataStreamReaderWriterSuite.scala (diff)
The file was modifiedexamples/src/main/scala/org/apache/spark/examples/mllib/RankingMetricsExample.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/higherOrderFunctions.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/streaming/MemorySinkSuite.scala (diff)
The file was modifiedcore/src/test/scala/org/apache/spark/storage/BlockInfoManagerSuite.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala (diff)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/complexTypesSuite.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DropNamespaceExec.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/streaming/statefulOperators.scala (diff)
The file was modifiedpom.xml (diff)
The file was modifiedresource-managers/kubernetes/core/src/test/scala/org/apache/spark/deploy/k8s/KubernetesTestConf.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/resource/ResourceUtils.scala (diff)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/aggregate/ApproximatePercentileSuite.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/orc/OrcScanBuilder.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/FlatMapGroupsWithStateExecHelperSuite.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/exchange/EnsureRequirements.scala (diff)
The file was modifiedsql/hive/src/test/scala/org/apache/spark/sql/hive/execution/UDAQuerySuite.scala (diff)
The file was modifiedcore/src/test/scala/org/apache/spark/scheduler/WorkerDecommissionSuite.scala (diff)
The file was modifiedcore/src/test/scala/org/apache/spark/executor/ExecutorSuite.scala (diff)
The file was modifiedmllib/src/test/scala/org/apache/spark/ml/fpm/PrefixSpanSuite.scala (diff)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/StreamingJoinHelperSuite.scala (diff)
The file was modifiedresource-managers/kubernetes/core/src/test/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsAllocatorSuite.scala (diff)
The file was modifiedexamples/src/main/scala/org/apache/spark/examples/ml/DeveloperApiExample.scala (diff)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/FilterPushdownSuite.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStoreRDD.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/udaf.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetCommitterSuite.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveCatalogs.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/command/createDataSourceTables.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/internal/SQLConfSuite.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/deploy/JsonProtocol.scala (diff)
The file was modifiedexamples/src/main/scala/org/apache/spark/examples/sql/SparkSQLExample.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/ExternalCatalog.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/catalog/Catalog.scala (diff)
The file was modifiedsql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveUDFSuite.scala (diff)
The file was modifiedcore/src/test/scala/org/apache/spark/input/WholeTextFileRecordReaderSuite.scala (diff)
The file was modifiedsql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkSQLCLIService.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/simpleCosting.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JsonFilters.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/datasources/SchemaPruning.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/TextBasedFileScan.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonFileFormat.scala (diff)
The file was modifiedcore/src/test/scala/org/apache/spark/deploy/master/ui/MasterWebUISuite.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonParsingOptionsSuite.scala (diff)
The file was modifiedsql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveSQLViewSuite.scala (diff)
The file was modifiedsql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/datasources/binaryfile/BinaryFileFormatSuite.scala (diff)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/RowJsonSuite.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/predicates.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/datasources/binaryfile/BinaryFileFormat.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala (diff)
The file was modifiedresource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosCoarseGrainedSchedulerBackend.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/connector/SupportsCatalogOptionsSuite.scala (diff)
The file was modifiedsql/hive/src/test/scala/org/apache/spark/sql/hive/client/HiveClientUserNameSuite.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDF.scala (diff)
The file was modifiedsql/hive/src/test/scala/org/apache/spark/sql/execution/benchmark/InsertIntoHiveTableBenchmark.scala (diff)
The file was modifiedresource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/Utils.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/connector/TableCapabilityCheckSuite.scala (diff)
The file was modifiedmllib/src/main/scala/org/apache/spark/ml/attribute/package.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/ObjectAggregationIterator.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/PlanStabilitySuite.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/streaming/streamingLimits.scala (diff)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/CreateTablePartitioningValidationSuite.scala (diff)
The file was modifiedresource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/YarnSparkHadoopUtilSuite.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statsEstimation/ProjectEstimation.scala (diff)
The file was modifiedstreaming/src/test/scala/org/apache/spark/streaming/receiver/BlockGeneratorSuite.scala (diff)
The file was modifiedresource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterManager.scala (diff)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/PullupCorrelatedPredicatesSuite.scala (diff)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/statsEstimation/FilterEstimationSuite.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcDeserializer.scala (diff)
The file was modifiedresource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerBackendUtil.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/types/DataType.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/internal/SessionState.scala (diff)
The file was modifiedsql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/ThriftServerQueryTestSuite.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/datasources/rules.scala (diff)
The file was modifiedstreaming/src/test/scala/org/apache/spark/streaming/scheduler/ExecutorAllocationManagerSuite.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/DataFrameWriterV2Suite.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/GenerateUnsafeRowJoiner.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCRDD.scala (diff)
The file was modifiedresource-managers/mesos/src/main/scala/org/apache/spark/deploy/mesos/ui/MesosClusterUI.scala (diff)
The file was modifiedsql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkGetTablesOperation.scala (diff)
The file was modifiedresource-managers/yarn/src/main/scala/org/apache/spark/launcher/YarnCommandBuilderUtils.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/BarrierTaskContext.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/types/Decimal.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/python/EvalPythonExec.scala (diff)
The file was modifiedresource-managers/kubernetes/core/src/test/scala/org/apache/spark/deploy/k8s/KubernetesUtilsSuite.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/GeneratePredicate.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/storage/BlockManagerDecommissioner.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/RelationalGroupedDataset.scala (diff)
The file was modifiedsql/hive/src/test/scala/org/apache/spark/sql/hive/HiveParquetSourceSuite.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/sources/PathOptionSuite.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedClusterMessage.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/rdd/ReliableCheckpointRDD.scala (diff)
The file was modifiedcore/src/test/scala/org/apache/spark/ui/StagePageSuite.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/streaming/ui/UIUtils.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/DecimalPrecision.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/FileDataSourceV2.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/window/WindowExec.scala (diff)
The file was modifiedexternal/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaTestUtils.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingDeduplicationSuite.scala (diff)
The file was modifiedsql/hive/src/test/scala/org/apache/spark/sql/hive/StatisticsSuite.scala (diff)
The file was modifiedmllib/src/main/scala/org/apache/spark/ml/feature/VarianceThresholdSelector.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/v2ResolutionPlans.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/util/Utils.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/datasources/HadoopFsRelation.scala (diff)
The file was modifiedsql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkSQLSessionManager.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/UDFRegistration.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveSessionCatalog.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/ReuseAdaptiveSubquery.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/MapOutputTracker.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/HiveResult.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamMetadata.scala (diff)
The file was modifiedmllib/src/test/scala/org/apache/spark/ml/regression/GeneralizedLinearRegressionSuite.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/command/PlanResolutionSuite.scala (diff)
The file was modifiedsql/hive/src/test/scala/org/apache/spark/sql/hive/HiveUtilsSuite.scala (diff)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ObjectExpressionsSuite.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/expressions/scalalang/typed.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/util/ThreadUtils.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/finishAnalysis.scala (diff)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/ResolveNaturalJoinSuite.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/deploy/master/ui/MasterWebUI.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/StatisticsCollectionSuite.scala (diff)
The file was modifiedresource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/KubernetesTestComponents.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FlatMapGroupsWithStateExec.scala (diff)
The file was modifiedsql/hive/src/main/scala/org/apache/spark/sql/hive/execution/SaveAsHiveFile.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/api/python/SerDeUtil.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/dynamicpruning/PlanDynamicPruningFilters.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2DataFrameSessionCatalogSuite.scala (diff)
The file was modifiedcore/src/test/scala/org/apache/spark/deploy/history/EventLogFileCompactorSuite.scala (diff)
The file was modifiedrepl/src/test/scala/org/apache/spark/repl/ExecutorClassLoaderSuite.scala (diff)
The file was modifiedexternal/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaRelation.scala (diff)
The file was modifiedmllib/src/test/scala/org/apache/spark/mllib/linalg/distributed/BlockMatrixSuite.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/deploy/history/HybridStore.scala (diff)
The file was modifiedsql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/CliSuite.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/datasources/SaveIntoDataSourceCommandSuite.scala (diff)
The file was modifiedsql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveQuerySuite.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/streaming/sources/PackedRowWriterFactory.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/window/WindowExecBase.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/functions.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/deploy/history/BasicEventFilterBuilder.scala (diff)
The file was modifiedrepl/src/test/scala-2.12/org/apache/spark/repl/Repl2Suite.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/RemoveRedundantProjects.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningUtils.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/SameResultSuite.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/test/GenericFunSpecSuite.scala (diff)
The file was modifiedexternal/kafka-0-10/src/test/scala/org/apache/spark/streaming/kafka010/mocks/MockScheduler.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/sources/DisableUnnecessaryBucketedScanSuite.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/datasources/OutputWriter.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetRowConverter.scala (diff)
The file was modifiedcore/src/test/scala/org/apache/spark/scheduler/CoarseGrainedSchedulerBackendSuite.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DescribeTableExec.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala (diff)
The file was modifiedexternal/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/streaming/sources/ForeachBatchSink.scala (diff)
The file was modifiedsql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/network/netty/NettyBlockTransferService.scala (diff)
The file was modifiedsql/hive/src/test/scala/org/apache/spark/sql/hive/test/TestHive.scala (diff)
The file was modifiedsql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala (diff)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/EliminateDistinctSuite.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/api/python/PythonRunner.scala (diff)
The file was modifiedresource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/EnvSecretsFeatureStep.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/CacheManager.scala (diff)
The file was modifiedmllib/src/test/scala/org/apache/spark/ml/util/DefaultReadWriteTest.scala (diff)
The file was modifiedsql/hive/compatibility/src/test/scala/org/apache/spark/sql/hive/execution/HiveCompatibilitySuite.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/api/r/RRunner.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala (diff)
The file was modifiedresource-managers/kubernetes/core/src/test/scala/org/apache/spark/deploy/k8s/features/MountVolumesFeatureStepSuite.scala (diff)
The file was modifiedcore/src/test/scala/org/apache/spark/resource/ResourceUtilsSuite.scala (diff)
The file was modifiedsql/hive/src/test/scala/org/apache/spark/sql/hive/QueryPartitionSuite.scala (diff)
The file was modifiedexternal/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSource.scala (diff)
The file was modifiedresource-managers/kubernetes/core/src/test/scala/org/apache/spark/deploy/k8s/features/DriverCommandFeatureStepSuite.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetIOSuite.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/command/CommandUtils.scala (diff)
Commit 014e1fbb3aba81a803c963fc0b7f4a8d1d70e253 by yumwang
[SPARK-27421][SQL] Fix filter for int column and value class
java.lang.String when pruning partition column
### What changes were proposed in this pull request?
This pr fix filter for int column and value class java.lang.String when
pruning partition column.
How to reproduce this issue:
```scala spark.sql("CREATE table test (name STRING) partitioned by (id
int) STORED AS PARQUET") spark.sql("CREATE VIEW test_view as select
cast(id as string) as id, name from test") spark.sql("SELECT * FROM
test_view WHERE id = '0'").explain
```
``` 20/11/15 06:19:01 INFO audit: ugi=root ip=unknown-ip-addr
cmd=get_partitions_by_filter : db=default tbl=test 20/11/15 06:19:01
INFO MetaStoreDirectSql: Unable to push down SQL filter: Cannot push
down filter for int column and value class java.lang.String 20/11/15
06:19:01 ERROR SparkSQLDriver: Failed in [SELECT * FROM test_view WHERE
id = '0'] java.lang.RuntimeException: Caught Hive MetaException
attempting to get partition metadata by filter from Hive. You can set
the Spark configuration setting
spark.sql.hive.manageFilesourcePartitions to false to work around this
problem, however this will result in degraded performance. Please report
a bug: https://issues.apache.org/jira/browse/SPARK
at
org.apache.spark.sql.hive.client.Shim_v0_13.getPartitionsByFilter(HiveShim.scala:828)
at
org.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$getPartitionsByFilter$1(HiveClientImpl.scala:745)
at
org.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$withHiveState$1(HiveClientImpl.scala:294)
at
org.apache.spark.sql.hive.client.HiveClientImpl.liftedTree1$1(HiveClientImpl.scala:227)
at
org.apache.spark.sql.hive.client.HiveClientImpl.retryLocked(HiveClientImpl.scala:226)
at
org.apache.spark.sql.hive.client.HiveClientImpl.withHiveState(HiveClientImpl.scala:276)
at
org.apache.spark.sql.hive.client.HiveClientImpl.getPartitionsByFilter(HiveClientImpl.scala:743)
```
### Why are the changes needed?
Fix bug.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Unit test.
Closes #30380 from wangyum/SPARK-27421.
Authored-by: Yuming Wang <yumwang@ebay.com> Signed-off-by: Yuming Wang
<yumwang@ebay.com>
(commit: 014e1fb)
The file was modifiedsql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala (diff)
The file was modifiedsql/hive/src/test/scala/org/apache/spark/sql/hive/client/HivePartitionFilteringSuite.scala (diff)
Commit 0b0fb70b09c7424805478a261e264d9df044fb96 by wenchen
[SPARK-33400][SQL] Normalize sameOrderExpressions in SortOrder to avoid
unnecessary sort operations
### What changes were proposed in this pull request? This pull request
tries to normalize the SortOrder properly to prevent unnecessary sort
operators. Currently the sameOrderExpressions are not normalized as part
of AliasAwareOutputOrdering.
Example: consider this join of three tables:
      """
       |SELECT t2id, t3.id as t3id
       |FROM (
       |    SELECT t1.id as t1id, t2.id as t2id
       |    FROM t1, t2
       |    WHERE t1.id = t2.id
       |) t12, t3
       |WHERE t1id = t3.id
     """.
The plan for this looks like:
      *(8) Project [t2id#1059L, id#1004L AS t3id#1060L]
     +- *(8) SortMergeJoin [t2id#1059L], [id#1004L], Inner
        :- *(5) Sort [t2id#1059L ASC NULLS FIRST ], false, 0       
<-----------------------------
        :  +- *(5) Project [id#1000L AS t2id#1059L]
        :     +- *(5) SortMergeJoin [id#996L], [id#1000L], Inner
        :        :- *(2) Sort [id#996L ASC NULLS FIRST ], false, 0
        :        :  +- Exchange hashpartitioning(id#996L, 5), true,
[id=#1426]
        :        :     +- *(1) Range (0, 10, step=1, splits=2)
        :        +- *(4) Sort [id#1000L ASC NULLS FIRST ], false, 0
        :           +- Exchange hashpartitioning(id#1000L, 5), true,
[id=#1432]
        :              +- *(3) Range (0, 20, step=1, splits=2)
        +- *(7) Sort [id#1004L ASC NULLS FIRST ], false, 0
           +- Exchange hashpartitioning(id#1004L, 5), true, [id=#1443]
              +- *(6) Range (0, 30, step=1, splits=2)
In this plan, the marked sort node could have been avoided as the data
is already sorted on "t2.id" by the lower SortMergeJoin.
### Why are the changes needed? To remove unneeded Sort operators.
### Does this PR introduce any user-facing change? No
### How was this patch tested? New UT added.
Closes #30302 from prakharjain09/SPARK-33400-sortorder.
Authored-by: Prakhar Jain <prakharjain09@gmail.com> Signed-off-by:
Wenchen Fan <wenchen@databricks.com>
(commit: 0b0fb70)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/PlannerSuite.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/AliasAwareOutputExpression.scala (diff)
Commit d5e7bd0cc497a5ea2baab0046501a804e7d42aa4 by dongjoon
[SPARK-33483][INFRA][TESTS] Fix rat exclusion patterns and add a LICENSE
### What changes were proposed in this pull request?
This PR fixes the RAT exclusion rule which was originated from
SPARK-1144 (Apache Spark 1.0)
### Why are the changes needed?
This prevents the situation like
https://github.com/apache/spark/pull/30415.
Currently, it missed `catalog` directory due to `.log` rule.
```
$ dev/check-license Could not find Apache license headers in the
following files:
!?????
/Users/dongjoon/APACHE/spark-merge/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/MetadataColumn.java
!?????
/Users/dongjoon/APACHE/spark-merge/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/SupportsMetadataColumns.java
```
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Pass the CI with the new rule.
Closes #30418 from dongjoon-hyun/SPARK-RAT.
Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon
Hyun <dongjoon@apache.org>
(commit: d5e7bd0)
The file was modifieddev/.rat-excludes (diff)
The file was modifiedsql/hive/src/test/resources/data/scripts/test_transform.py (diff)
Commit ef2638c3e3aa1d2ce137f1c50c9697a7877d1719 by dongjoon
[SPARK-33183][SQL][FOLLOW-UP] Update rule RemoveRedundantSorts config
version
### What changes were proposed in this pull request? This PR is a follow
up for #30093 to updates the config
`spark.sql.execution.removeRedundantSorts` version to 2.4.8.
### Why are the changes needed? To update the rule version it has been
backported to 2.4. #30194
### Does this PR introduce _any_ user-facing change? No
### How was this patch tested? N/A
Closes #30420 from allisonwang-db/spark-33183-follow-up.
Authored-by: allisonwang-db
<66282705+allisonwang-db@users.noreply.github.com> Signed-off-by:
Dongjoon Hyun <dongjoon@apache.org>
(commit: ef2638c)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala (diff)
Commit a03c540cf7fe92160caf41ef6d2e2993f667dc59 by wenchen
[SPARK-33472][SQL] Adjust RemoveRedundantSorts rule order
### What changes were proposed in this pull request?
This PR switched the order for the rule `RemoveRedundantSorts` and
`EnsureRequirements` so that `EnsureRequirements` will be invoked before
`RemoveRedundantSorts` to avoid IllegalArgumentException when
instantiating PartitioningCollection.
### Why are the changes needed?
`RemoveRedundantSorts` rule uses SparkPlan's `outputPartitioning` to
check whether a sort node is redundant. Currently, it is added before
`EnsureRequirements`. Since `PartitioningCollection` requires left and
right partitioning to have the same number of partitions, which is not
necessarily true before applying `EnsureRequirements`, the rule can fail
with the following exception:
``` IllegalArgumentException: requirement failed: PartitioningCollection
requires all of its partitionings have the same numPartitions.
```
### Does this PR introduce _any_ user-facing change? No
### How was this patch tested? Unit test
Closes #30373 from allisonwang-db/sort-follow-up.
Authored-by: allisonwang-db
<66282705+allisonwang-db@users.noreply.github.com> Signed-off-by:
Wenchen Fan <wenchen@databricks.com>
(commit: a03c540)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/RemoveRedundantSortsSuite.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/QueryExecution.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/SparkPlan.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala (diff)
Commit 21b13506cd822ed7db343bff4ca25d9555178f10 by wenchen
[SPARK-33442][SQL] Change Combine Limit to Eliminate limit using max row
### What changes were proposed in this pull request?
Change `CombineLimits` name to `EliminateLimits` and add check if
`Limit` child max row <= limit.
### Why are the changes needed?
In Add-hoc scene, we always add limit for the query if user have no
special limit value, but not all limit is nesessary.
A general negative example is
``` select count(*) from t limit 100000;
```
It will be great if we can eliminate limit at Spark side.
Also, we make a benchmark for this case
``` runBenchmark("Sort and Limit") {
val N = 100000
val benchmark = new Benchmark("benchmark sort and limit", N)
  benchmark.addCase("TakeOrderedAndProject", 3) { _ =>
   spark.range(N).toDF("c").repartition(200).sort("c").take(200000)
}
  benchmark.addCase("Sort And Limit", 3) { _ =>
   withSQLConf("spark.sql.execution.topKSortFallbackThreshold" -> "-1")
{
     spark.range(N).toDF("c").repartition(200).sort("c").take(200000)
   }
}
  benchmark.addCase("Sort", 3) { _ =>
   spark.range(N).toDF("c").repartition(200).sort("c").collect()
}
benchmark.run()
}
```
and the result is
``` Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.15.6
Intel(R) Core(TM) i5-5257U CPU  2.70GHz benchmark sort and limit:      
         Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per
Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
TakeOrderedAndProject                              1833           2259 
      382          0.1       18327.1       1.0X Sort And Limit         
                          1417           1658         285          0.1 
    14167.5       1.3X Sort                                            
1324           1484         225          0.1       13238.3       1.4X
```
It shows that it makes sense to replace `TakeOrderedAndProjectExec` with
`Sort + Project`.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Add test.
Closes #30368 from ulysses-you/SPARK-33442.
Authored-by: ulysses <youxiduo@weidian.com> Signed-off-by: Wenchen Fan
<wenchen@databricks.com>
(commit: 21b1350)
The file was modifiedsql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q95.sf100/simplified.txt (diff)
The file was modifiedsql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q96/explain.txt (diff)
The file was modifiedsql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q95/explain.txt (diff)
The file was modifiedsql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q97.sf100/simplified.txt (diff)
The file was modifiedsql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q23a.sf100/explain.txt (diff)
The file was modifiedsql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q94/explain.txt (diff)
The file was modifiedsql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q94.sf100/simplified.txt (diff)
The file was modifiedsql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q94/simplified.txt (diff)
The file was modifiedsql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q38/explain.txt (diff)
The file was modifiedsql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q96/simplified.txt (diff)
The file was modifiedsql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q97.sf100/explain.txt (diff)
The file was modifiedsql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q16.sf100/simplified.txt (diff)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/LimitPushdownSuite.scala (diff)
The file was modifiedsql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q92.sf100/simplified.txt (diff)
The file was modifiedsql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q96.sf100/explain.txt (diff)
The file was modifiedsql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q94.sf100/explain.txt (diff)
The file was modifiedsql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q23a/explain.txt (diff)
The file was modifiedsql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q95/simplified.txt (diff)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/CombiningLimitsSuite.scala (diff)
The file was modifiedsql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q38.sf100/explain.txt (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala (diff)
The file was modifiedsql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q95.sf100/explain.txt (diff)
The file was modifiedsql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q38/simplified.txt (diff)
The file was modifiedsql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q96.sf100/simplified.txt (diff)
The file was modifiedsql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q23a.sf100/simplified.txt (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/streaming/StreamSuite.scala (diff)
The file was modifiedsql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q38.sf100/simplified.txt (diff)
The file was modifiedsql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q16/explain.txt (diff)
The file was modifiedsql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q16.sf100/explain.txt (diff)
The file was modifiedsql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q92/simplified.txt (diff)
The file was modifiedsql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q97/explain.txt (diff)
The file was modifiedsql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q23a/simplified.txt (diff)
The file was modifiedsql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q92.sf100/explain.txt (diff)
The file was modifiedsql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q16/simplified.txt (diff)
The file was modifiedsql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q92/explain.txt (diff)
The file was modifiedsql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q97/simplified.txt (diff)
Commit 3695e997d5d436be086235505bbb030c87ae8eef by wenchen
[SPARK-33045][SQL] Support build-in function like_all and fix
StackOverflowError issue
### What changes were proposed in this pull request? Spark already
support `LIKE ALL` syntax, but it will throw `StackOverflowError` if
there are many elements(more than 14378 elements). We should implement
built-in function for LIKE ALL to fix this issue.
Why the stack overflow can happen in the current approach ? The current
approach uses reduceLeft to connect each `Like(e, p)`, this will lead
the the call depth of the thread is too large, causing
`StackOverflowError` problems.
Why the fix in this PR can avoid the error? This PR support built-in
function for `LIKE ALL` and avoid this issue.
### Why are the changes needed? 1.Fix the `StackOverflowError` issue.
2.Support built-in function `like_all`.
### Does this PR introduce _any_ user-facing change?
'No'.
### How was this patch tested? Jenkins test.
Closes #29999 from beliefer/SPARK-33045-like_all.
Lead-authored-by: gengjiaan <gengjiaan@360.cn> Co-authored-by: beliefer
<beliefer@163.com> Co-authored-by: Jiaan Geng <beliefer@163.com>
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
(commit: 3695e99)
The file was modifiedsql/core/src/test/resources/sql-tests/inputs/like-all.sql (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/regexpExpressions.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala (diff)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/RegexpExpressionsSuite.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/dsl/package.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala (diff)