Changes

Summary

  1. [SPARK-36269][SQL] Fix only set data columns to Hive column names config (details)
Commit 6740d07f89201f6fbc322a2d08be1ee0ad8bcd92 by wenchen
[SPARK-36269][SQL] Fix only set data columns to Hive column names config

### What changes were proposed in this pull request?

When reading Hive table, we set the Hive column id and column name configs (`hive.io.file.readcolumn.ids` and `hive.io.file.readcolumn.names`). We should set non-partition columns (data columns) for both configs, as Spark always [appends partition columns in its own Hive reader](https://github.com/apache/spark/blob/master/sql/hive/src/main/scala/org/apache/spark/sql/hive/TableReader.scala#L240). The column id config has only non-partition columns, but column name config has both partition and non-partition columns. We should keep them to be consistent with only non-partition columns. This does not cause issue for public OSS Hive file format for now. But for customized internal Hive file format, it causes the issue as we are expecting these two configs to be same.

### Why are the changes needed?

Fix the code logic to be more consistent.

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Existing Hive tests.

Closes #33489 from c21/hive-col.

Authored-by: Cheng Su <chengsu@fb.com>
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
(cherry picked from commit e5616e32eecb516a6b46ae9bc5c2c850c18210a2)
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
The file was modifiedsql/hive/src/main/scala/org/apache/spark/sql/hive/execution/HiveTableScanExec.scala (diff)