Changes

Summary

  1. [SPARK-33105][INFRA] Change default R arch from i386 to x64 and (commit: 018811f) (details)
  2. [SPARK-33102][SQL] Use stringToSeq on SQL list typed parameters (commit: 1e63dcc) (details)
  3. [SPARK-33108][BUILD] Remove sbt-dependency-graph SBT plugin (commit: dfb7790) (details)
  4. [SPARK-32881][CORE] Catch some race condition errors and log them more (commit: 7696ca5) (details)
  5. [SPARK-33107][SQL] Remove hive-2.3 workaround code (commit: 5e17014) (details)
Commit 018811f9747d063a44543ceb265351377f0bc917 by gurwls223
[SPARK-33105][INFRA] Change default R arch from i386 to x64 and
parametrize BINPREF
### What changes were proposed in this pull request?
- Change default R `arch` from `i386` to `x64`, to match Rtools
version.
- Parameterize `BINPREF` with `WIN`
(https://stackoverflow.com/a/44035904)
Reported on dev:
http://apache-spark-developers-list.1001551.n3.nabble.com/Broken-rlang-installation-on-AppVeyor-td30294.html
### Why are the changes needed?
It seems like update from rlang 0.4.7 to 0.4.8 exposed an issue, where
build fails because of incompatible ddl
```
c:/Rtools40/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/8.3.0/../../../../x86_64-w64-mingw32/bin/ld.exe:
skipping incompatible C:/R/bin/i386/R.dll when searching for -lR
[00:01:52]
c:/Rtools40/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/8.3.0/../../../../x86_64-w64-mingw32/bin/ld.exe:
skipping incompatible C:/R/bin/i386/R.dll when searching for -lR
[00:01:52]
c:/Rtools40/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/8.3.0/../../../../x86_64-w64-mingw32/bin/ld.exe:
cannot find -lR
[00:01:52] collect2.exe: error: ld returned 1 exit status
```
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Existing tests.
Closes #29991 from zero323/APPVEYOR-DEAFAULT-ARCH.
Authored-by: zero323 <mszymkiewicz@gmail.com> Signed-off-by: HyukjinKwon
<gurwls223@apache.org>
(commit: 018811f)
The file was modifieddev/appveyor-install-dependencies.ps1 (diff)
Commit 1e63dcc8f0b267f3d835db5b59c60360c04d9c0a by gurwls223
[SPARK-33102][SQL] Use stringToSeq on SQL list typed parameters
### What changes were proposed in this pull request? While I've
implemented JDBC provider disable functionality it has been popped up
[here](https://github.com/apache/spark/pull/29964#discussion_r501786746)
that `Utils.stringToSeq` must be used when String list type SQL
parameter handled. In this PR I've fixed the problematic parameters.
### Why are the changes needed?
`Utils.stringToSeq` must be used when String list type SQL parameter
handled.
### Does this PR introduce _any_ user-facing change? No.
### How was this patch tested? Existing unit tests.
Closes #29989 from gaborgsomogyi/SPARK-33102.
Authored-by: Gabor Somogyi <gabor.g.somogyi@gmail.com> Signed-off-by:
HyukjinKwon <gurwls223@apache.org>
(commit: 1e63dcc)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/streaming/MicroBatchExecution.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/streaming/DataStreamWriter.scala (diff)
Commit dfb7790a9dad8e98bd27001a613b4e13a5eb9d51 by dhyun
[SPARK-33108][BUILD] Remove sbt-dependency-graph SBT plugin
### What changes were proposed in this pull request?
This PR aims to remove `sbt-dependency-graph` SBT plugin.
### Why are the changes needed?
`sbt-dependency-graph` officially doesn't support SBT 1.3.x and it's
broken due to `NoSuchMethodError`. This cannot be fixed in
`sbt-dependency-graph` side at SBT 1.3.x
- https://github.com/sbt/sbt-dependency-graph
   > Note: Under sbt >= 1.3.x some features might currently not work as
expected or not at all (like dependencyLicenses).
```
$ build/sbt dependencyTree Launching sbt from
build/sbt-launch-1.3.13.jar
[info] welcome to sbt 1.3.13 (AdoptOpenJDK Java 1.8.0_252)
...
[error] java.lang.NoSuchMethodError:
sbt.internal.LibraryManagement$.cachedUpdate(Lsbt/librarymanagement/DependencyResolution;Lsbt/librarymanagement/ModuleDescriptor;Lsbt/util/CacheStoreFactory;Ljava/lang/String;Lsbt/librarymanagement/UpdateConfiguration;Lscala/Function1;ZZZLsbt/librarymanagement/UnresolvedWarningConfiguration;Lsbt/librarymanagement/EvictionWarningOptions;ZLsbt/internal/librarymanagement/CompatibilityWarningOptions;Lsbt/util/Logger;)Lsbt/librarymanagement/UpdateReport;
```
**ALTERNATIVES**
- One alternative is `coursier`, but it requires `coursier-based sbt
launcher` which is more intrusive.
- https://get-coursier.io/docs/sbt-coursier.html#sbt-13x
   > you'll have to use the coursier-based sbt launcher, via its custom
sbt-extras launcher for example.
- Another alternative is moving to `SBT 1.4.0` which uses
`sbt-dependency-graph` as a built-in, but it's still new and will
requires many change.
So, this PR aims to remove the broken plugin simply.
### Does this PR introduce _any_ user-facing change?
No. This is a dev-only change.
### How was this patch tested?
Manual.
```
$ build/sbt dependencyTree
...
[error] Not a valid command: dependencyTree
[error] Not a valid project ID: dependencyTree
[error] Not a valid key: dependencyTree (similar: dependencyOverrides,
sbtDependency, dependencyResolution)
[error] dependencyTree
[error]               ^
```
Closes #29997 from dongjoon-hyun/remove_depedencyTree.
Lead-authored-by: Dongjoon Hyun <dongjoon@apache.org> Co-authored-by:
Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon Hyun
<dhyun@apple.com>
(commit: dfb7790)
The file was modifiedproject/plugins.sbt (diff)
Commit 7696ca56732166977642a777f1d94cfba67b4151 by dhyun
[SPARK-32881][CORE] Catch some race condition errors and log them more
clearly
### What changes were proposed in this pull request?
Decommissioning can run out of time resulting in some race condition,
these race conditions result in confusing error messages but not
negative impact.
### Why are the changes needed?
The NPE & element missing errors in the log can create a
missunderstanding.
### Does this PR introduce _any_ user-facing change? Logs change.
### How was this patch tested? Existing tests pass.
Closes #29992 from
holdenk/SPARK-32881-error-messaging-on-decom-race-messages.
Authored-by: Holden Karau <hkarau@apple.com> Signed-off-by: Dongjoon
Hyun <dhyun@apple.com>
(commit: 7696ca5)
The file was modifiedcore/src/main/scala/org/apache/spark/MapOutputTracker.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala (diff)
Commit 5e170140b0374762087b204008da141febaacee3 by dhyun
[SPARK-33107][SQL] Remove hive-2.3 workaround code
### What changes were proposed in this pull request?
This pr remove `hive-2.3` workaround code.
### Why are the changes needed?
Make code more clear and readable.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Existing unit tests.
Closes #29996 from wangyum/SPARK-33107.
Authored-by: Yuming Wang <yumwang@ebay.com> Signed-off-by: Dongjoon Hyun
<dhyun@apple.com>
(commit: 5e17014)
The file was modifiedsql/hive/src/main/scala/org/apache/spark/sql/hive/HiveShim.scala (diff)
The file was modifiedsql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkExecuteStatementOperation.scala (diff)
The file was modifiedsql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/SparkMetadataOperationSuite.scala (diff)
The file was modifiedsql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/GetCatalogsOperationMock.scala (diff)
The file was modifiedsql/hive/src/main/scala/org/apache/spark/sql/hive/hiveUDFs.scala (diff)
The file was modifiedsql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkSQLSessionManager.scala (diff)
The file was modifiedsql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkSQLCLIDriver.scala (diff)
The file was modifiedsql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2Suites.scala (diff)
The file was removedsql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/ThriftserverShimUtils.scala
The file was modifiedsql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/SparkExecuteStatementOperationSuite.scala (diff)
The file was modifiedsql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkGetTypeInfoOperation.scala (diff)
The file was modifiedsql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/HiveSessionImplSuite.scala (diff)
The file was modifiedsql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/SharedThriftServer.scala (diff)
The file was modifiedsql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala (diff)
The file was modifiedsql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/SparkThriftServerProtocolVersionsSuite.scala (diff)