Progress:
Changes

Summary

  1. [SPARK-37529][K8S][TESTS] Support K8s integration tests for Java 17 (details)
Commit 32922eeb3250408523991f21c943de02cd1ebea3 by dongjoon
[SPARK-37529][K8S][TESTS] Support K8s integration tests for Java 17

### What changes were proposed in this pull request?

This PR aims to support K8s integration tests for Java 17 using Maven and SBT.
The new system property `spark.kubernetes.test.dockerFile` is introduced to specify a Dockerfile.
By setting `Dockerfile.java17` to the property, the integration tests run with Java 17.

This PR also revised the change brought by SPARK-37354 (#34628) by changing `SparkBuild.scala` so that it can recognize the system property `spark.kubernetes.test.javaImageTag`, like the integration tests with Maven do.

If both `spark.kubernetes.test.dockerFile` and `spark.kubernetes.test.javaImageTag` are set, `spark.kubernetes.test.dockerFile` is preferred.

### Why are the changes needed?

To ensure Spark works on K8s with Java 17.

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Confirmed that the intended version of Java is used for each pattern.
* Neither `spark.kubernetes.test.javaImageTag` nor `spark.kubernetes.test.dockerFile` is set (SBT)
```
# Run the integration tests to create the container image
$ build/sbt -Pkubernetes  -Pkubernetes-integration-tests "kubernetes-integration-tests/test"

# Create and login the container.
docker run -it <hash> /bin/bash

185b40ef8aaa56c:/opt/spark/work-dir$ java -version
openjdk version "1.8.0_312"
OpenJDK Runtime Environment (build 1.8.0_312-b07)
OpenJDK 64-Bit Server VM (build 25.312-b07, mixed mode)
```

* Neither `spark.kubernetes.test.javaImageTag` nor `spark.kubernetes.test.dockerFile` is set (Maven)
```
# Run the integration tests to create the container image
$ build/mvn -Pkubernetes -Pkubernetes-integration-tests -pl resource-managers/kubernetes/integration-tests integration-test

# Create and login the container.
docker run -it <hash> /bin/bash

18536ae5c5c21f4:/opt/spark/work-dir$ java -version
openjdk version "1.8.0_312"
OpenJDK Runtime Environment (build 1.8.0_312-b07)
OpenJDK 64-Bit Server VM (build 25.312-b07, mixed mode)
```

* `spark.kubernetes.test.javaImageTag` is set (SBT):
```
# Run the integration tests to create the container image
$ build/sbt -Dspark.kubernetes.test.javaImageTag=11-jre-slim -Pkubernetes  -Pkubernetes-integration-tests "kubernetes-integration-tests/test"

# Create and login the container.
docker run -it <hash> /bin/bash

1855b896d0e5dd8:/opt/spark/work-dir$ java -version
openjdk version "11.0.13" 2021-10-19
OpenJDK Runtime Environment 18.9 (build 11.0.13+8)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.13+8, mixed mode, sharing)
```

* `spark.kubernetes.test.javaImageTag` is set (Maven):
```
# Run the integration tests to create the container image
$ build/mvn -Dspark.kubernetes.test.javaImageTag=11-jre-slim -Pkubernetes -Pkubernetes-integration-tests -pl resource-managers/kubernetes/integration-tests integration-test

# Create and login the container.
docker run -it <hash> /bin/bash

1855d3c228e7521:/opt/spark/work-dir$ java -version
openjdk version "11.0.13" 2021-10-19
OpenJDK Runtime Environment 18.9 (build 11.0.13+8)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.13+8, mixed mode, sharing)
```

* `spark.kubernetes.test.dockerFile` is set (SBT)
```
$ build/sbt -Dspark.kubernetes.test.dockerFile=resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile.java17 -Pkubernetes  -Pkubernetes-integration-tests package "kubernetes-integration-tests/test"

# Create and login the container.
docker run -it <hash> /bin/bash

185550515c76582:/opt/spark/work-dir$ java -version
openjdk version "17.0.1" 2021-10-19
OpenJDK Runtime Environment (build 17.0.1+12-Debian-1deb11u2)
OpenJDK 64-Bit Server VM (build 17.0.1+12-Debian-1deb11u2, mixed mode, sharing)
```

* `spark.kubernetes.test.dockerFile` is set (Maven)
```
build/mvn -Dspark.kubernetes.test.dockerFile=resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile.java17 -Pkubernetes -Pkubernetes-integration-tests -pl resource-managers/kubernetes/integration-tests integration-test

# Create and login the container.
docker run -it <hash> /bin/bash

185be35bde0ebfa:/opt/spark/work-dir$ java -version
openjdk version "17.0.1" 2021-10-19
OpenJDK Runtime Environment (build 17.0.1+12-Debian-1deb11u2)
OpenJDK 64-Bit Server VM (build 17.0.1+12-Debian-1deb11u2, mixed mode, sharing)
```

* Both `spark.kubernetes.test.javaImageTag` and `spark.kubernetes.test.dockerFile` are set (SBT)
```
$ build/sbt -Dspark.kubernetes.test.dockerFile=resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile.java17 -Dspark.kubernetes.test.javaImageTag=11-jre-slim -Pkubernetes  -Pkubernetes-integration-tests package "kubernetes-integration-tests/test"

# Create and login the container.
docker run -it <hash> /bin/bash

1857ed1891b2921:/opt/spark/work-dir$ java -version
openjdk version "17.0.1" 2021-10-19
OpenJDK Runtime Environment (build 17.0.1+12-Debian-1deb11u2)
OpenJDK 64-Bit Server VM (build 17.0.1+12-Debian-1deb11u2, mixed mode, sharing)
```

* Both `spark.kubernetes.test.javaImageTag` and `spark.kubernetes.test.dockerFile` are set (Maven)
```
$ build/mvn -Dspark.kubernetes.test.dockerFile=resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile.java17 -Dspark.kubernetes.test.javaImageTag=11-jre-slim -Pkubernetes -Pkubernetes-integration-tests -pl resource-managers/kubernetes/integration-tests integration-test

# Create and login the container.
docker run -it <hash> /bin/bash

18584f3a8b7b4a6:/opt/spark/work-dir$ java -version
openjdk version "17.0.1" 2021-10-19
OpenJDK Runtime Environment (build 17.0.1+12-Debian-1deb11u2)
OpenJDK 64-Bit Server VM (build 17.0.1+12-Debian-1deb11u2, mixed mode, sharing)
```

Closes #34790 from sarutak/kube-integration-test-java17.

Authored-by: Kousuke Saruta <sarutak@oss.nttdata.com>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
The file was modifiedproject/SparkBuild.scala (diff)
The file was modifiedresource-managers/kubernetes/integration-tests/pom.xml (diff)
The file was modifiedresource-managers/kubernetes/integration-tests/scripts/setup-integration-test-env.sh (diff)