Repository navigation
Run RNTester's iOS unit tests in the test_ios_rntester CI job - #59666
Open
shreyanshj10 wants to merge 2 commits into
Open
shreyanshj10 wants to merge 2 commits into
shreyanshj10 wants to merge 2 commits into
Conversation
Contributor
|
I don't think that these tests are actually passing, and that's why we disabled them in the first place. I allowed a run in CI. Let's see what CI returns. |
Run Ruby Tests is the only caller of run_ruby_tests.sh, and it has never executed because nothing set run-unit-tests: true. It cannot pass as written either: the script loads all ten *-test.rb into one process, where PodMock.rb's Pod::UI class collides with the real cocoapods Pod::UI module that spm-test.rb requires. Also host run-unit-tests on a dynamic-frameworks cell, so a failing suite cannot skip Upload RNTester App and starve test_e2e_ios_rntester.
shreyanshj10
force-pushed
the
fix/issue-59596-ios-unit-tests
branch
from
November 3, 2026 04:33
9eab6b8 to
301394b
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary:
.github/actions/test-ios-rntesterrunsyarn test-iosonly when itsrun-unit-testsinput is'true', and the input defaults tofalse. Three jobs use the action and none of them turns it on:test_ios_rntester_ruby_3_2_0passesruby-versionandflavor,test_ios_rntesterpassesuse-frameworks,use-prebuildsandflavor, andtest_ios_rntester_dynamic_frameworkspassesrun-unit-tests: falsebeside a comment saying the tests "are already run in the test_ios_rntester job". They aren't, so RNTester's iOS unit tests run in no job at all and all five of the action's gated steps are skipped in every run. A change that breaks an iOS unit test goes green.This wasn't an oversight, and that's the main thing I'd like you to weigh. They were switched off deliberately in #45133, "Disable unit tests for flakyness in CircleCI": "we migrated these jobs to M1 machines, as circleCI is deprecating intel ones, and on these machines the simulators tend to freeze... We are going to move to GHA, so that should not be a big issue." The move to GHA has since happened, and
test_ios_rntesteriscontinue-on-error: truewithfail-fast: false, so re-enabling cannot make flakiness block anyone. Worth noting against that rationale: the test plan this enables already sets"testRepetitionMode": "retryOnFailure", so per-test retry is in place.Three changes.
1. Wire the input up on
test_ios_rntester, the job the existing comment points at, in exactly one of its four matrix cells:One cell, because
yarn test-iosignores the cell:scripts/objc-test.shdoes its ownbundle exec pod installandxcodebuild build-for-testingwith no-configurationflag and noUSE_FRAMEWORKSexport,runTestspasses no-configurationeither, andRNTester.xcschemepinsTestAction buildConfiguration = "Debug". So it builds and tests Debug against static source pods whichever cell invokes it, and the other three would be identical reruns on amacos-15-largerunner.A
frameworks: truecell specifically, becauseUpload RNTester Appis gated onuse-frameworks == 'false', sits after the test steps and carries noalways(). On aframeworks: falsecell, any failure in the suite skips that upload and starvestest_e2e_ios_rntesterof its app — which is what happened on the run you allowed, below. A dynamic-frameworks cell publishes no app artifact at all, so it cannot perturb the e2e jobs however the suite fares.That cell does get materially slower, since
objc-test.sh's ownpod installbuilds from source rather than reusing the prebuilds the job downloaded. That is the cost of running the suite at all, and it is one cell rather than four. I wrote&& matrix.frameworksrather than&& matrix.frameworks == truedeliberately:frameworks: [false, true]are unquoted YAML booleans today, so both spellings work, but the comparison form would silently become permanently false if those values were ever quoted — reintroducing exactly the defect this issue reports.2. Remove the
Run Ruby Testsstep from.github/actions/prepare-ios-tests. This is the one change here that deletes a test invocation, so in full:That step runs
sh run_ruby_tests.sh, and it is what failed on the CI run you allowed — notyarn test-ios.test_ios_rntester (Release, false)died inPrepare IOS Testswith:run_ruby_tests.shhas exactly one call site in the repository,prepare-ios-tests, andprepare-ios-testshas exactly one consumer,test-ios-rntester, gated oninputs.run-unit-tests == 'true'. Nothing in the tree sets that input to true, so the step has never executed in CI, and it has drifted in three stacked ways:bundle installagainst the rootGemfile, the one that declarescocoapods. The job's onlybundle installcomes later, inInstall CocoaPods dependencies, and runs frompackages/rn-testeragainst that directory's own Gemfile. Hence theLoadError.bundle exec, and it moves tocannot load such file -- test/unitfromall_tests.rb:1:test-unitis a bundled gem, sobundler/setupputs it out of reach unless the root Gemfile asks for it.gem 'test-unit'and it reacheswrong argument type Class (expected Module) (TypeError)fromcocoapods/sources_manager.rb:163.run_ruby_tests.shconcatenates all ten*-test.rbinto oneall_tests.rband runs them in a single process. Five of them loadcocoapods/__tests__/test_utils/PodMock.rb, which definesPod::UIas a class;spm-test.rbalone loads the real gem, which doesextend Pod::UIand needs a module. The mock and the real CocoaPods cannot coexist in one interpreter, so the split has to happen in the runner, not in a Gemfile.Excluding
spm-test.rb, the remainder runs but is not green onmaineither: 118 tests, 6 failures and 1 error, the error beingtest_setupFabric_installsPods,undefined method 'rncore_pod'atcocoapods/fabric.rb:14.So reviving that suite is separate work with a real design question in it, and I did not want it to decide whether the Objective-C tests get to run.
run_ruby_tests.shis untouched and still in the tree, nothing that runs in CI today loses coverage, and I left the reasoning in a comment at the removal site. Glad to open the revival as its own PR if you want it — splitting the mocked tests fromspm-test.rbinto two runner invocations looks like the smallest fix, but that call is yours.3. Point
scripts/.tests.envat a simulator runtime that exists. It pinnedIOS_TARGET_OS="18.3.1", whichobjc-test.shpasses straight into-destination "platform=iOS Simulator,name=$IOS_DEVICE,OS=$IOS_TARGET_OS". The action takessetup-xcode's default of Xcode 16.4.0 onmacos-15, and that image carries iOS simulator runtimes 18.5, 18.6 and 26.0–26.2 — no 18.3.1. So the step I'm enabling would have failed to resolve a destination on every run, andcontinue-on-errorwould have hidden it. The pin had no live consumer in CI to keep it honest: the only other reader isrunXcodeTests.sh, which takes an override argument and is invoked by no workflow.OS=latestwould float instead, if you prefer that.What the enabled cell actually runs, and where I'd expect it to go red
Worth stating before you spend a run on it.
RNTester.xcscheme's test action points atpackages/rn-tester/RNTester/RNTester.xctestplan, whosetestTargetsisRNTesterUnitTestsonly — no integration target, despite the step being named "iOS Unit and Integration Tests" — and whose single configuration is named "Address Sanitizer", withaddressSanitizer.enabled: true,detectStackUseAfterReturn: trueandundefinedBehaviorSanitizerEnabled: true. ASan and UBSan over a suite dormant since #45133 is the likeliest source of a red result.The other candidate is derived data.
buildForTestingre-runspod installwith neitherUSE_FRAMEWORKSnorRCT_USE_LOCAL_RN_DEP/RCT_TESTONLY_RNCORE_TARBALL_PATH— all of whichInstall CocoaPods dependenciesexports only into its own shell — and then builds into the same/tmp/RNTesterBuildthe job has already filled. If that reuse across a dynamic-to-static switch is the failure, the fix is a separate-derivedDataPathinobjc-test.shrather than a different cell.Two things I left alone on purpose. The report's other half —
React-RCTFabric.podspec'sTeststest spec being built by no job, because no Podfile asks for it with:testspecs— needs apackages/rn-tester/Podfilechange and is a bigger job than this, so I haven't writtenFixes, to avoid closing the issue on half of it. AndStore test resultsuploads./reports/junitwhileobjc-test.sh'sxcbeautifyFormatwrites to$HOME/react-native/reports/junitunder CI;if-no-files-founddefaults towarn, so it can't fail the job, but expect that artifact to come up empty until someone reconciles the two paths.Changelog:
[INTERNAL] [FIXED] - Run RNTester's iOS unit tests in CI
Test Plan:
This is CI wiring, so the check that matters is this PR's own
test_ios_rntester (Debug, true)job:Prepare IOS TestsandRun Tests: iOS Unit and Integration Testsshould execute instead of reportingoutcome=skipped, and the run should publishxcresults-Debug-dynamic-frameworks-Ruby2.6.10, which no run produces today. I have not runyarn test-iosmyself, and I'm making no claim that the suite passes — that is the thing the run would establish.What I did check:
git grep -n "run-unit-tests" -- .githubfinds the action's input with itsfalsedefault, its five gated steps, and the callers, none of which passedtrue.git grep -nE "objc-test|xcodebuild test|RNTesterUnitTests|only-testing" -- .githubfinds nothing, so no other workflow runs the suite either.test-ios-rntesterpins, against the rootGemfileandpackages/react-native/scriptsfrommain: the samecannot load such file -- cocoapodsatspm-test.rb:8, then the two further failures described above, then thePod::UIclass-versus-moduleTypeError, and 6 failures plus 1 error across the remaining 118 tests.run_ruby_tests.shhas one call site and that nothing setsrun-unit-tests: true, by grep over the whole tree.actions/runner-imagesmanifest formacos-15rather than by inference: its "Installed Simulators" table lists 18.5, 18.6, 26.0, 26.1 and 26.2, andiPhone 16under each.frameworks: [false, true]are booleans in the matrix context and the interpolated input stringifies to"true"/"false"— the behaviouruse-frameworks: ${{ matrix.frameworks }}already relies on for the action'sinputs.use-frameworks == 'false'checks. Only(Debug, true)resolves totrue.CI_USE_PACKAGER=1, andobjc-test.sh mainstarts the WebSocket server andyarn start, waiting on both, beforebuildForTesting.scripts/.tests.envstill parses as shell (bash -n) and exportsIOS_TARGET_OS=18.5,IOS_DEVICE=iPhone 16.prettier --checkat the pinned 3.9.4 thatyarn format-check-javascriptruns over*.ymlis clean.Addresses #59596 — the CI wiring half. The
React-RCTFabrictest-spec half is still open.