Build iOS Backup Boundary Regression Tests on a Cloud Mac

After an update, a news app began storing several hundred megabytes of redownloadable offline resources in Application Support. All functional tests passed and no user data was lost, but backup size increased sharply. The problem was not the download logic. The team had never turned the distinction between files that must be restored and files that can be rebuilt into executable rules. A cloud Mac is well suited to this type of regression testing: the environment remains consistent over time, Simulators can be reset, and container inventories can be retained for investigation when a test fails.

Define data recovery semantics first

Do not start by writing tests around directories. Classify data by purpose first. The same JSON file might contain user-created content or merely cache an API response, and those cases have completely different backup requirements.

Application Support is particularly easy to misuse. It is appropriate for data managed by the app over the long term, but that does not mean everything stored there belongs in a backup. If large models, map packages, or media proxy files can be obtained again, explicitly mark them as excluded or move them to Caches.

The deciding question is not “Is this file important?” but “After device restoration, must this file come back from the backup, and is there no reliable source from which it can be rebuilt?”

Document these rules in a repository-owned inventory. For example, record each item’s logical name, relative path, recovery requirement, and owner. Tests should enforce this agreement rather than treating incidental paths scattered throughout the codebase as the specification.

Enforce exclusion attributes with XCTest

Foundation provides a more stable interface than directly reading low-level extended attributes. The helper below sets isExcludedFromBackup after creating a file, then reads the resource value again to confirm that the setting was persisted.

import XCTest

final class BackupBoundaryTests: XCTestCase {
    private func markExcluded(_ url: URL) throws {
        var values = URLResourceValues()
        values.isExcludedFromBackup = true
        var mutableURL = url
        try mutableURL.setResourceValues(values)
    }

    func testDownloadPackageIsExcludedFromBackup() throws {
        let root = FileManager.default.urls(
            for: .applicationSupportDirectory,
            in: .userDomainMask
        )[0]
        let package = root.appendingPathComponent(
            "Downloads/catalog.bundle",
            isDirectory: false
        )

        try FileManager.default.createDirectory(
            at: package.deletingLastPathComponent(),
            withIntermediateDirectories: true
        )
        try Data("fixture".utf8).write(to: package)
        try markExcluded(package)

        let values = try package.resourceValues(
            forKeys: [.isExcludedFromBackupKey]
        )
        XCTAssertEqual(values.isExcludedFromBackup, true)
    }
}

The point of the test is not to prove that Foundation works. It is to ensure that production code follows the same attribute-setting path after creating a file. A more robust design encapsulates directory creation, atomic writes, and exclusion marking in a storage component, then calls that component directly from the test. Otherwise, the test may set the attribute successfully while the production downloader omits it, leaving the gate ineffective.

Check for misplaced files as well

Add inverse assertions too: user drafts must not be written to Caches, and temporary exports must not remain in Documents. Test fixtures should cover creation, upgrade migration, and failed retries because path drift often appears in migration branches rather than during first installation.

For directory-level exclusions, also spot-check newly created child files. Do not assume that the parent directory’s current attribute can permanently replace explicit write-time logic. Having the component actively verify the attribute whenever it creates a directory makes the system more resilient to legacy-data migration and directory recreation.

Preserve container evidence from the Simulator

Unit tests are well suited to verifying rules. Inspecting the Simulator container answers a different question: what was actually written when the test failed? Start by running tests against a fixed device with separate DerivedData so that concurrent jobs do not share state.

set -euo pipefail

DERIVED_DATA="$PWD/.ci/DerivedData-backup"
RESULT_BUNDLE="$PWD/.ci/BackupBoundary.xcresult"

rm -rf "$DERIVED_DATA" "$RESULT_BUNDLE"

xcodebuild test \
  -workspace Example.xcworkspace \
  -scheme Example \
  -destination 'platform=iOS Simulator,name=iPhone 16' \
  -derivedDataPath "$DERIVED_DATA" \
  -resultBundlePath "$RESULT_BUNDLE"

APP_DATA="$(
  xcrun simctl get_app_container booted com.example.app data
)"
find "$APP_DATA" -type f -print | LC_ALL=C sort \
  > "$PWD/.ci/app-container-files.txt"

get_app_container succeeds only when the target app is installed and the corresponding Simulator is booted. CI must not silently select any device that happens to be running. It should create or explicitly select a device at the start of the job, wait for it to finish booting, and shut it down afterward. When multiple test suites run in parallel on Runner, assign a separate Simulator device set to each job to prevent container state from crossing between jobs.

A file inventory containing only relative paths is safer and easier to compare. Do not include home directories, absolute workspace paths, or test credentials in artifacts. If extended attributes need inspection, attach xattr output as diagnostic evidence, but base the gate’s result on Foundation resource values and the product inventory. This avoids depending on undocumented low-level behavior.

Turn size anomalies into explainable failures

Checking Boolean attributes alone is not enough. A path refactor could place every cache file in Documents without causing any individual exclusion-attribute test to fail. You can define size budgets for the container’s top-level directories, but those budgets should identify structural anomalies rather than enforce an exact byte count.

For example, after the test fixture runs, require Documents to contain only approved samples and tmp to be empty when the task finishes. Allow Caches to grow, but prohibit file extensions associated with user drafts. Use an explicit allowlist for large test resources. Failure messages should include the relative path, file size, applicable rule, and creation stage.

Do not immediately delete the failure state

The first response to a failure should not be wiping the Simulator. Archive the following evidence first:

• the XCTest result bundle and the name of the failed test;
• an inventory of relative paths in the app container;
• aggregate sizes for each top-level directory;
• the tested commit, Scheme, and Simulator runtime;
• the test step that triggered file creation.

This evidence is enough to distinguish a missing attribute from a misplaced file or a cleanup step that never ran. Delete the device only after all attachments have been captured, so an intermittent failure does not leave behind nothing more than a single assertion message.

Design a stable CI gate

Backup boundary tests should run whenever the storage layer, downloader, database migration, or export flow changes. They can also be included in a lightweight pre-merge test suite. Do not tie them to end-to-end flows that require external services. Local fixtures and fixed-size data are necessary for reproducible results.

Consider dividing failures into three categories: misplaced user data that must block the build, missing exclusion attributes that must also block it, and storage-size trends that should only produce warnings. Size thresholds must be controlled by repository configuration, and any adjustment should be justified during review. A script must not automatically loosen a threshold after detecting that it has been exceeded.

Finally, retain an acceptance check on a physical device. The Simulator can reliably verify directories, attributes, and migration code, but it cannot cover every aspect of the real restoration path. Release checks should use a minimal dataset to confirm that non-reproducible data can be restored and that reproducible data is not treated as a prerequisite for recovery. In this model, the automated gate handles frequent regression testing while the physical-device workflow validates the final boundary. Neither replaces the other.

Frequently asked questions