Adapt Semantic Versioning

[rlindner]

Last week there's been a new version 1.10.0 released which included breaking changes. This violates semantic versioning, a standard you should stick to when releasing a package for SPM.

see https://docs.swift.org/package-manager/PackageDescription/PackageDescription.html:

The version rule requires Swift packages to conform to semantic versioning. To learn more about the semantic versioning standard, visit semver.org.

The problem which arises here is that this update broke every setup where this package was referenced with upToNextMajor which is unexpected when following semantic versioning.

I would therefore like to encourage you to consider this for future releases.

[stephencelis]

Hi @rlindner! We actually do our best to maintain backwards compatibility, regardless of SemVer. Our swift-composable-architecture library is pre-1.0 but has maintained source compatibility over all of its releases, with large windows for deprecations to be migrated.

SemVer guidelines are generally taken to mean changes to "source compatibility," of which this Snapshot Testing release has none: we don't believe there are any APIs that have changed that would lead to existing projects not building. We try to be very aware of non-source breaking changes, though, and try to communicate them appropriately. In the case of updates listed under "breaking changes," there are a few categories:

• Bug fixes that may change the output of a snapshot: bug fixes are generally "patch" concerns, but almost always come with behavioral changes. This is actually a big criticism of SemVer, since a patch update that fixes a bug can still break an app when it updates. Please note that snapshots already "break" frequently due to environmental changes: Apple regularly changes how UI renders over OS updates (minor and major), and different devices may output different results. In fact this release comes with a new API that will hopefully provide users with a more stable snapshot suite when run on both Intel and M1 devices.
• Deprecations to tooling we no longer support, i.e. CocoaPods and Carthage, or changes to tooling, i.e. SPM. These tools are ancillary to the library source.

Is there a particular breaking change that you are having issues with or consider source-incompatible?

[rlindner]

To be honest, the issue we've been facing is not one of the listed breaking changes. I've just seen that we were having issues with the latest release here and have then taken a look at the changelog and stumbled upon the "Breaking Changes" section. But I'm unsure if I would follow your argumentation that changes should only be considered "breaking", when they're source-incompatible, so that the project doesn't compile anymore.

But maybe just let me describe the actual issue we were having in some more detail:

With 1.9.0 we could use the assertSnapshot in a test helper function, like:

final class TestCase: XCTestCase {
    func assert(with color: UIColor) {
        let view = UIView(frame: .init(origin: .zero, size: .init(width: 50, height: 50)))
        view.backgroundColor = color

        assertSnapshot(matching: view, as: .image)
    }

    func testRed() throws {
        assert(with: .red)
    }

    func testGreen() throws {
        assert(with: .green)
    }

    func testBlue() throws {
        assert(with: .blue)
    }
}

This led to three snapshots:

After the update to 1.10.0 all those tests were failing because this only creates a single snapshot reference now:

So we had to update our test code accordingly. The code still compiled fine, yes, but the behavior changed. And this error just appeared on CI without any changes on our side, because the package was automatically updated "minor" from 1.9.0 to 1.10.0.

[siejkowski]

According to https://semver.org, you need to update major when the API changes in a breaking manner:

Given a version number MAJOR.MINOR.PATCH, increment the:

1. MAJOR version when you make incompatible API changes
2. MINOR version when you add functionality in a backwards compatible manner
3. PATCH version when you make backwards compatible bug fixes

So as long as API stays compatible, it's either a minor or a patch update.

[rlindner]

I would say you could interpret "API" in either way here. Your definition seems to only refer to the API-Calls (the code still compiles). But my expectation also includes changes to existing behavior. Otherwise you could also remove the whole implementation from a function and just keep a stub and it still compiles "non breaking".

I more like the definitions from here: https://learn.microsoft.com/en-us/linkedin/shared/breaking-change-policy

A breaking change is a change that may require you to make changes to your application in order to avoid disruption to your integration.

A non-breaking change is a change that you can adapt to at your own discretion and pace without disruption.

[stephencelis]

@rlindner I think what we're all seeing here is that SemVer doesn't define things clearly, and that there is room for a lot of interpretation, and a lot of nuance 😄

We are going with the interpretation that we believe is most common, and what most people expect a SemVer to focus on, which is the API interface. There are even static package managers out there that "enforce" SemVer using this definition (see Elm's package manager, which will not allow you to release a patch version with a new public API, nor will it allow you to release a minor version with an existing API that has changed or been removed). I don't think you can divide major/minor/patch along "behavior change" as reason for a major release, because most bug fixes are behavior changes, and so really all releases would have to be major. As I noted in another reply, the "breaking change" you experienced is due to a bug fix related to undefined/undocumented behavior of the library that your helper code was relying on, and in this case there is a fix you can apply to make your helper more reliable.

With all that said, we do understand that the temporary breakage to any integration is a bummer, but we will always do our best to guide you to a fix when these things occur, and try to minimize them in general! In this case I do think it was appropriate to merge the bug fix, and I also think that the breakage around undefined, undocumented behavior did not merit a major release, though I also understand that others may have made another decision.

[stephencelis]

the issue we've been facing is not one of the listed breaking changes.

The change you are seeing is a result of the following bug fix:

• Fixed: Support for repeatedly running tests (Add support for tests executed repeatedly #585).

Snapshot Testing manages some global state as a convenience for counting up over the course of a specific test. Relying on this count outside a particular test is not advertised or intentionally supported behavior, but supporting an Xcode feature definitely should be.

In your case, you can use the named parameter of assertSnapshot to give each artifact a unique name, or you can use the magic #function parameter in your helper and pass it along to get the library behavior of naming your snapshot based off the test it is invoked from. See how the library does that here:

https://github.com/pointfreeco/swift-snapshot-testing/blob/a02ef9ab8270fa2623af0bf89b26dcb29aaf3290/Sources/SnapshotTesting/AssertSnapshot.swift