Skip to content

Run RNTester's iOS unit tests in the test_ios_rntester CI job - #59666

Open
shreyanshj10 wants to merge 2 commits into
react:mainfrom
shreyanshj10:fix/issue-59596-ios-unit-tests
Open

shreyanshj10 wants to merge 2 commits into
react:mainfrom
shreyanshj10:fix/issue-59596-ios-unit-tests

Conversation

@shreyanshj10

@shreyanshj10 shreyanshj10 commented Oct 30, 2026 •

Copy link
Copy Markdown

Summary:

.github/actions/test-ios-rntester runs yarn test-ios only when its run-unit-tests input is 'true', and the input defaults to false. Three jobs use the action and none of them turns it on: test_ios_rntester_ruby_3_2_0 passes ruby-version and flavor, test_ios_rntester passes use-frameworks, use-prebuilds and flavor, and test_ios_rntester_dynamic_frameworks passes run-unit-tests: false beside a comment saying the tests "are already run in the test_ios_rntester job". They aren't, so RNTester's iOS unit tests run in no job at all and all five of the action's gated steps are skipped in every run. A change that breaks an iOS unit test goes green.

This wasn't an oversight, and that's the main thing I'd like you to weigh. They were switched off deliberately in #45133, "Disable unit tests for flakyness in CircleCI": "we migrated these jobs to M1 machines, as circleCI is deprecating intel ones, and on these machines the simulators tend to freeze... We are going to move to GHA, so that should not be a big issue." The move to GHA has since happened, and test_ios_rntester is continue-on-error: true with fail-fast: false, so re-enabling cannot make flakiness block anyone. Worth noting against that rationale: the test plan this enables already sets "testRepetitionMode": "retryOnFailure", so per-test retry is in place.

Three changes.

1. Wire the input up on test_ios_rntester, the job the existing comment points at, in exactly one of its four matrix cells:

run-unit-tests: ${{ matrix.flavor == 'Debug' && matrix.frameworks }}

One cell, because yarn test-ios ignores the cell: scripts/objc-test.sh does its own bundle exec pod install and xcodebuild build-for-testing with no -configuration flag and no USE_FRAMEWORKS export, runTests passes no -configuration either, and RNTester.xcscheme pins TestAction buildConfiguration = "Debug". So it builds and tests Debug against static source pods whichever cell invokes it, and the other three would be identical reruns on a macos-15-large runner.

A frameworks: true cell specifically, because Upload RNTester App is gated on use-frameworks == 'false', sits after the test steps and carries no always(). On a frameworks: false cell, any failure in the suite skips that upload and starves test_e2e_ios_rntester of its app — which is what happened on the run you allowed, below. A dynamic-frameworks cell publishes no app artifact at all, so it cannot perturb the e2e jobs however the suite fares.

That cell does get materially slower, since objc-test.sh's own pod install builds from source rather than reusing the prebuilds the job downloaded. That is the cost of running the suite at all, and it is one cell rather than four. I wrote && matrix.frameworks rather than && matrix.frameworks == true deliberately: frameworks: [false, true] are unquoted YAML booleans today, so both spellings work, but the comparison form would silently become permanently false if those values were ever quoted — reintroducing exactly the defect this issue reports.

2. Remove the Run Ruby Tests step from .github/actions/prepare-ios-tests. This is the one change here that deletes a test invocation, so in full:

That step runs sh run_ruby_tests.sh, and it is what failed on the CI run you allowed — not yarn test-ios. test_ios_rntester (Release, false) died in Prepare IOS Tests with:

.../rubygems/core_ext/kernel_require.rb:54:in `require': cannot load such file -- cocoapods (LoadError)
	from .../rubygems/core_ext/kernel_require.rb:54:in `require'
	from .../packages/react-native/scripts/cocoapods/__tests__/spm-test.rb:8:in `<top (required)>'
	from ./all_tests.rb:11:in `require_relative'
	from ./all_tests.rb:11:in `<main>'

run_ruby_tests.sh has exactly one call site in the repository, prepare-ios-tests, and prepare-ios-tests has exactly one consumer, test-ios-rntester, gated on inputs.run-unit-tests == 'true'. Nothing in the tree sets that input to true, so the step has never executed in CI, and it has drifted in three stacked ways:

  1. Nothing runs bundle install against the root Gemfile, the one that declares cocoapods. The job's only bundle install comes later, in Install CocoaPods dependencies, and runs from packages/rn-tester against that directory's own Gemfile. Hence the LoadError.
  2. Install the root bundle and run under bundle exec, and it moves to cannot load such file -- test/unit from all_tests.rb:1: test-unit is a bundled gem, so bundler/setup puts it out of reach unless the root Gemfile asks for it.
  3. Add gem 'test-unit' and it reaches wrong argument type Class (expected Module) (TypeError) from cocoapods/sources_manager.rb:163. run_ruby_tests.sh concatenates all ten *-test.rb into one all_tests.rb and runs them in a single process. Five of them load cocoapods/__tests__/test_utils/PodMock.rb, which defines Pod::UI as a class; spm-test.rb alone loads the real gem, which does extend Pod::UI and needs a module. The mock and the real CocoaPods cannot coexist in one interpreter, so the split has to happen in the runner, not in a Gemfile.

Excluding spm-test.rb, the remainder runs but is not green on main either: 118 tests, 6 failures and 1 error, the error being test_setupFabric_installsPods, undefined method 'rncore_pod' at cocoapods/fabric.rb:14.

So reviving that suite is separate work with a real design question in it, and I did not want it to decide whether the Objective-C tests get to run. run_ruby_tests.sh is untouched and still in the tree, nothing that runs in CI today loses coverage, and I left the reasoning in a comment at the removal site. Glad to open the revival as its own PR if you want it — splitting the mocked tests from spm-test.rb into two runner invocations looks like the smallest fix, but that call is yours.

3. Point scripts/.tests.env at a simulator runtime that exists. It pinned IOS_TARGET_OS="18.3.1", which objc-test.sh passes straight into -destination "platform=iOS Simulator,name=$IOS_DEVICE,OS=$IOS_TARGET_OS". The action takes setup-xcode's default of Xcode 16.4.0 on macos-15, and that image carries iOS simulator runtimes 18.5, 18.6 and 26.0–26.2 — no 18.3.1. So the step I'm enabling would have failed to resolve a destination on every run, and continue-on-error would have hidden it. The pin had no live consumer in CI to keep it honest: the only other reader is runXcodeTests.sh, which takes an override argument and is invoked by no workflow. OS=latest would float instead, if you prefer that.

What the enabled cell actually runs, and where I'd expect it to go red

Worth stating before you spend a run on it. RNTester.xcscheme's test action points at packages/rn-tester/RNTester/RNTester.xctestplan, whose testTargets is RNTesterUnitTests only — no integration target, despite the step being named "iOS Unit and Integration Tests" — and whose single configuration is named "Address Sanitizer", with addressSanitizer.enabled: true, detectStackUseAfterReturn: true and undefinedBehaviorSanitizerEnabled: true. ASan and UBSan over a suite dormant since #45133 is the likeliest source of a red result.

The other candidate is derived data. buildForTesting re-runs pod install with neither USE_FRAMEWORKS nor RCT_USE_LOCAL_RN_DEP/RCT_TESTONLY_RNCORE_TARBALL_PATH — all of which Install CocoaPods dependencies exports only into its own shell — and then builds into the same /tmp/RNTesterBuild the job has already filled. If that reuse across a dynamic-to-static switch is the failure, the fix is a separate -derivedDataPath in objc-test.sh rather than a different cell.

Two things I left alone on purpose. The report's other half — React-RCTFabric.podspec's Tests test spec being built by no job, because no Podfile asks for it with :testspecs — needs a packages/rn-tester/Podfile change and is a bigger job than this, so I haven't written Fixes, to avoid closing the issue on half of it. And Store test results uploads ./reports/junit while objc-test.sh's xcbeautifyFormat writes to $HOME/react-native/reports/junit under CI; if-no-files-found defaults to warn, so it can't fail the job, but expect that artifact to come up empty until someone reconciles the two paths.

Changelog:

[INTERNAL] [FIXED] - Run RNTester's iOS unit tests in CI

Test Plan:

This is CI wiring, so the check that matters is this PR's own test_ios_rntester (Debug, true) job: Prepare IOS Tests and Run Tests: iOS Unit and Integration Tests should execute instead of reporting outcome=skipped, and the run should publish xcresults-Debug-dynamic-frameworks-Ruby2.6.10, which no run produces today. I have not run yarn test-ios myself, and I'm making no claim that the suite passes — that is the thing the run would establish.

What I did check:

  • The defect, against the tree rather than the issue text. git grep -n "run-unit-tests" -- .github finds the action's input with its false default, its five gated steps, and the callers, none of which passed true. git grep -nE "objc-test|xcodebuild test|RNTesterUnitTests|only-testing" -- .github finds nothing, so no other workflow runs the suite either.
  • The Ruby failure, reproduced away from CI on the Ruby version test-ios-rntester pins, against the root Gemfile and packages/react-native/scripts from main: the same cannot load such file -- cocoapods at spm-test.rb:8, then the two further failures described above, then the Pod::UI class-versus-module TypeError, and 6 failures plus 1 error across the remaining 118 tests.
  • That run_ruby_tests.sh has one call site and that nothing sets run-unit-tests: true, by grep over the whole tree.
  • The destination claim, against the actions/runner-images manifest for macos-15 rather than by inference: its "Installed Simulators" table lists 18.5, 18.6, 26.0, 26.1 and 26.2, and iPhone 16 under each.
  • The expression types. frameworks: [false, true] are booleans in the matrix context and the interpolated input stringifies to "true"/"false" — the behaviour use-frameworks: ${{ matrix.frameworks }} already relies on for the action's inputs.use-frameworks == 'false' checks. Only (Debug, true) resolves to true.
  • That the packager requirement is met: the test plan sets CI_USE_PACKAGER=1, and objc-test.sh main starts the WebSocket server and yarn start, waiting on both, before buildForTesting.
  • scripts/.tests.env still parses as shell (bash -n) and exports IOS_TARGET_OS=18.5, IOS_DEVICE=iPhone 16.
  • Formatting. prettier --check at the pinned 3.9.4 that yarn format-check-javascript runs over *.yml is clean.

Addresses #59596 — the CI wiring half. The React-RCTFabric test-spec half is still open.

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Oct 30, 2026
@facebook-github-tools facebook-github-tools Bot added the Shared with Meta Applied via automation to indicate that an Issue or Pull Request has been shared with the team. label Oct 30, 2026
@cipolleschi

Copy link
Copy Markdown
Contributor

I don't think that these tests are actually passing, and that's why we disabled them in the first place. I allowed a run in CI. Let's see what CI returns.

Run Ruby Tests is the only caller of run_ruby_tests.sh, and it has never
executed because nothing set run-unit-tests: true. It cannot pass as
written either: the script loads all ten *-test.rb into one process,
where PodMock.rb's Pod::UI class collides with the real cocoapods
Pod::UI module that spm-test.rb requires.

Also host run-unit-tests on a dynamic-frameworks cell, so a failing
suite cannot skip Upload RNTester App and starve test_e2e_ios_rntester.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. Shared with Meta Applied via automation to indicate that an Issue or Pull Request has been shared with the team.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants