Skip to content

Commit 9b4a811

Browse files
xiangfu0xiangfu0
committed
Run shared integration profiles in CI
1 parent 9c1425a commit 9b4a811

3 files changed

Lines changed: 94 additions & 11 deletions

File tree

‎.github/workflows/scripts/pr-tests/.pinot_tests_integration.sh‎

Lines changed: 31 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -50,23 +50,43 @@ print_surefire_dumps() {
5050
done <<< "${dump_files}"
5151
}
5252

53-
# Integration Tests
54-
cd pinot-integration-tests || exit 1
55-
if [ "$RUN_TEST_SET" == "1" ]; then
56-
mvn test \
57-
-P github-actions,codecoverage,integration-tests-set-1 || {
53+
run_integration_profile() {
54+
local profile="$1"
55+
shift
56+
57+
mvn test "$@" \
58+
-P "github-actions,codecoverage,${profile}" || {
5859
print_surefire_dumps
5960
exit 1
6061
}
62+
}
63+
64+
# Integration Tests
65+
cd pinot-integration-tests || exit 1
66+
if [ "$RUN_TEST_SET" == "1" ]; then
67+
run_integration_profile integration-tests-set-1
68+
shared_profiles=(
69+
shared-hybrid-cluster-integration-test-suite
70+
shared-llc-realtime-cluster-integration-test-suite
71+
)
72+
for shared_profile in "${shared_profiles[@]}"; do
73+
run_integration_profile "${shared_profile}"
74+
done
6175
exit 0
6276
fi
6377
if [ "$RUN_TEST_SET" == "2" ]; then
64-
mvn test \
65-
-DargLine="-Xms1g -Xmx2g -Dlog4j2.configurationFile=log4j2.xml" \
66-
-P github-actions,codecoverage,integration-tests-set-2 || {
67-
print_surefire_dumps
68-
exit 1
69-
}
78+
run_integration_profile integration-tests-set-2 \
79+
-DargLine="-Xms1g -Xmx2g -Dlog4j2.configurationFile=log4j2.xml"
80+
# Keep this lane to the shared profiles that have shown a local wall-clock win.
81+
shared_profiles=(
82+
shared-no-override-offline-cluster-integration-test-suite
83+
shared-realtime-manager-cluster-integration-test-suite
84+
shared-controller-only-cluster-integration-test-suite
85+
shared-offline-cluster-integration-test-suite
86+
)
87+
for shared_profile in "${shared_profiles[@]}"; do
88+
run_integration_profile "${shared_profile}"
89+
done
7090
exit 0
7191
fi
7292

‎pinot-integration-tests/INTEGRATION_TEST_SETUP_GROUPS.md‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -296,6 +296,29 @@ instead of the main no-override suite:
296296
| Shared upsert preload suite | `./mvnw -pl pinot-integration-tests -Pshared-upsert-preload-cluster-integration-test-suite test` | 1 | 27.68s |
297297
| Disabled manual suite | `./mvnw -pl pinot-integration-tests -Pdisabled-manual-cluster-integration-test-suite test` | 0 | 12.14s |
298298

299+
### GitHub Actions Wiring
300+
301+
The first CI wiring pass keeps the regular alphabetical integration sets, but
302+
excludes the most profitable migrated classes from those sets and runs them in
303+
the existing GitHub Actions integration lanes.
304+
305+
Set 1 runs these shared profiles after its reduced alphabetical pass:
306+
307+
- `shared-hybrid-cluster-integration-test-suite`
308+
- `shared-llc-realtime-cluster-integration-test-suite`
309+
310+
Set 2 runs these shared profiles after its reduced alphabetical pass:
311+
312+
- `shared-no-override-offline-cluster-integration-test-suite`
313+
- `shared-realtime-manager-cluster-integration-test-suite`
314+
- `shared-controller-only-cluster-integration-test-suite`
315+
- `shared-offline-cluster-integration-test-suite`
316+
317+
The six selected shared profiles total 17.45 minutes locally versus 20.39 minutes
318+
for the same per-class lifecycles. This is intentionally the profitable subset
319+
rather than every shared profile; several exact-config profiles are
320+
setup-correctness buckets and would increase CI wall time if wired immediately.
321+
299322
The four cursor/empty-response broker-config suites are exact-config buckets, so
300323
they are not yet a wall-clock improvement when run as four separate profiles.
301324
The same 41 tests passed in a single per-class lifecycle command in 112.06s,

‎pinot-integration-tests/pom.xml‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -103,6 +103,26 @@
103103
<include>**/org/apache/pinot/integration/tests/M*Test.java</include>
104104
<include>**/org/apache/pinot/integration/tests/N*Test.java</include>
105105
</includes>
106+
<excludes>
107+
<exclude>**/org/apache/pinot/integration/tests/DateTimeFieldSpecHybridClusterIntegrationTest.java</exclude>
108+
<exclude>**/org/apache/pinot/integration/tests/DimensionTableIntegrationTest.java</exclude>
109+
<exclude>**/org/apache/pinot/integration/tests/HelixZNodeSizeLimitTest.java</exclude>
110+
<exclude>**/org/apache/pinot/integration/tests/HybridClusterIntegrationTest.java</exclude>
111+
<exclude>**/org/apache/pinot/integration/tests/LLCRealtimeClusterIntegrationTest.java</exclude>
112+
<exclude>**/org/apache/pinot/integration/tests/LLCRealtimeKafka3ClusterIntegrationTest.java</exclude>
113+
<exclude>**/org/apache/pinot/integration/tests/LLCRealtimeKafka4ClusterIntegrationTest.java</exclude>
114+
<exclude>**/org/apache/pinot/integration/tests/OfflineClusterIntegrationTest.java</exclude>
115+
<exclude>**/org/apache/pinot/integration/tests/PinotLLCRealtimeSegmentManagerIntegrationTest.java</exclude>
116+
<exclude>**/org/apache/pinot/integration/tests/QueryQuotaClusterIntegrationTest.java</exclude>
117+
<exclude>**/org/apache/pinot/integration/tests/RetentionManagerIntegrationTest.java</exclude>
118+
<exclude>**/org/apache/pinot/integration/tests/SegmentUploadIntegrationTest.java</exclude>
119+
<exclude>**/org/apache/pinot/integration/tests/SegmentWriterUploaderIntegrationTest.java</exclude>
120+
<exclude>**/org/apache/pinot/integration/tests/ServerStarterIntegrationTest.java</exclude>
121+
<exclude>**/org/apache/pinot/integration/tests/SparkSegmentMetadataPushIntegrationTest.java</exclude>
122+
<exclude>**/org/apache/pinot/integration/tests/StarTreeFunctionParametersIntegrationTest.java</exclude>
123+
<exclude>**/org/apache/pinot/integration/tests/tpch/TPCHQueryIntegrationTest.java</exclude>
124+
<exclude>**/org/apache/pinot/server/realtime/ControllerLeaderLocatorIntegrationTest.java</exclude>
125+
</excludes>
106126
</configuration>
107127
</plugin>
108128
</plugins>
@@ -136,6 +156,26 @@
136156
<include>**/org/apache/pinot/integration/tests/Z*Test.java</include>
137157
<include>**/org/apache/pinot/server/realtime/**</include>
138158
</includes>
159+
<excludes>
160+
<exclude>**/org/apache/pinot/integration/tests/DateTimeFieldSpecHybridClusterIntegrationTest.java</exclude>
161+
<exclude>**/org/apache/pinot/integration/tests/DimensionTableIntegrationTest.java</exclude>
162+
<exclude>**/org/apache/pinot/integration/tests/HelixZNodeSizeLimitTest.java</exclude>
163+
<exclude>**/org/apache/pinot/integration/tests/HybridClusterIntegrationTest.java</exclude>
164+
<exclude>**/org/apache/pinot/integration/tests/LLCRealtimeClusterIntegrationTest.java</exclude>
165+
<exclude>**/org/apache/pinot/integration/tests/LLCRealtimeKafka3ClusterIntegrationTest.java</exclude>
166+
<exclude>**/org/apache/pinot/integration/tests/LLCRealtimeKafka4ClusterIntegrationTest.java</exclude>
167+
<exclude>**/org/apache/pinot/integration/tests/OfflineClusterIntegrationTest.java</exclude>
168+
<exclude>**/org/apache/pinot/integration/tests/PinotLLCRealtimeSegmentManagerIntegrationTest.java</exclude>
169+
<exclude>**/org/apache/pinot/integration/tests/QueryQuotaClusterIntegrationTest.java</exclude>
170+
<exclude>**/org/apache/pinot/integration/tests/RetentionManagerIntegrationTest.java</exclude>
171+
<exclude>**/org/apache/pinot/integration/tests/SegmentUploadIntegrationTest.java</exclude>
172+
<exclude>**/org/apache/pinot/integration/tests/SegmentWriterUploaderIntegrationTest.java</exclude>
173+
<exclude>**/org/apache/pinot/integration/tests/ServerStarterIntegrationTest.java</exclude>
174+
<exclude>**/org/apache/pinot/integration/tests/SparkSegmentMetadataPushIntegrationTest.java</exclude>
175+
<exclude>**/org/apache/pinot/integration/tests/StarTreeFunctionParametersIntegrationTest.java</exclude>
176+
<exclude>**/org/apache/pinot/integration/tests/tpch/TPCHQueryIntegrationTest.java</exclude>
177+
<exclude>**/org/apache/pinot/server/realtime/ControllerLeaderLocatorIntegrationTest.java</exclude>
178+
</excludes>
139179
</configuration>
140180
</plugin>
141181
</plugins>

0 commit comments

Comments
 (0)