Skip Contract Tests

[jpjpjp]

Maybe this is a silly question, but is it possible to skip the generation of contract tests?

With no configuration, Portman generates contract tests. I know that it's possible to configure your own contract tests, but is there a way to ask for NO contract tests to be generated?

Background and Use Case: When I'm working on a new endpoint, I often want to build a collection that only includes the integration tests for that endpoint during the iterative development phase. I can eliminate the variation and integration tests for other endpoints by commenting them out in my configuration like so:

tests:
  contractTests:
    $ref: ./specs/v2/tests/configs/contracts/generic-contract-tests.yaml
  integrationTests:
    # - name: /me tests
    #   $ref: ./specs/v2/tests/configs/variations/me-variations.yaml
    # - name: /categories tests
    #   $ref: ./specs/v2/tests/configs/variations/categories-variations.yaml
    # - name: /manual_accounts tests
    #   $ref: ./specs/v2/tests/configs/variations/manual_account-variations.yaml
    # - name: /tags tests
    #   $ref: ./specs/v2/tests/configs/variations/tags-variations.yaml
    # - name: /recurring tests
    #   $ref: ./specs/v2/tests/configs/variations/recurring-variations.yaml
    - name: /transactions tests
      $ref: ./specs/v2/tests/configs/variations/transactions-variations.yaml

But if I comment out the contractTests configuration the default contract tests will still be generated.

[thim81]

I've not validated this but I think you have 2 options:

--includeTests, -t Inject Portman test suite (default: true)

example
portman crm.yml --includeTests false

BUT i'm not sure if this will still generate the other tests.

or

disable the contract tests.

tests:
  contractTests:
  - openApiOperation: "*::/*"
    statusSuccess:
      enabled: false
  - openApiOperation: "*::/*"
    responseTime:
      enabled: false
      maxMs: 300
  - openApiOperation: "*::/*"
    contentType:
      enabled: false
  - openApiOperation: "*::/*"
    jsonBody:
      enabled: false
  - openApiOperation: "*::/*"
    schemaValidation:
      enabled: false
  - openApiOperation: "*::/*"
    headersPresent:
      enabled: false

[jpjpjp]

Hi @thim81 , thanks for the quick answer, sorry it took me a while to try these options out.

What I found with the includeTests false parameter is that it completely eliminates the generation of the Integration Tests folder (meaning that both the requests and the tests are not generated). This makes sense, but it doesn't help me with the goal that I was shooting for, which was to ONLY generate the Integration Tests for the particular endpoint I'm currently working on.

Both the includeTests false options and the manual disabling of the contractTests DO eliminate the post-request test scripts for the Contract Tests, but the per-endpoint folders, with the requests, ARE still generated.

What I was hoping for (and I may be a unique user, so don't consider this a high-priority feature request) is to eliminate the generation of the contract test folders completely (both the requests and the tests), so that the resulting collection would only include tests that are explicitly specified in the config. One could imagine a skipNonConfiguredRequests parameter which is false by default. If set to true and if there are no other tests specified in the config, Portman would generate an empty collection!

Having said that, the manual elimination of the tests (if not the requests) is still helpful! Thank you!

[thim81]

if you want to skip certain request from being converted, you can use the filter option.
See the https://github.com/apideck-libraries/portman/tree/main/examples/cli-filtering

Which is powered by https://github.com/thim81/openapi-format#openapi-filter-options

[jpjpjp]

Thanks @thim81 just saw this today. This seems really handy, especially in cases where you are developing a server against a full spec but don't have a complete implementation yet. I'll definitely use this!

Unfortunately, it doesn't solve the problem I'm trying to address, which is that I want to run variation tests for certain endpoints but not contract tests (which Portman generates by default). I ended up with a configuration that manually allows or disallows contract tests and variation tests on a per-endpoint basis. It's verbose, but it works!

#  Enable contract tests for all endpoints except transactions
tests:
  contractTests:
    - openApiOperation: "*::/me"
      $ref: ./specs/v2/tests/configs/contracts/generic-contract-tests.yaml
    - openApiOperation: "*::/categories"
      $ref: ./specs/v2/tests/configs/contracts/generic-contract-tests.yaml
    - openApiOperation: "*::/manual_accounts"
      $ref: ./specs/v2/tests/configs/contracts/generic-contract-tests.yaml
    - openApiOperation: "*::/plaid_accounts"
      $ref: ./specs/v2/tests/configs/contracts/generic-contract-tests.yaml
    - openApiOperation: "*::/recurring"
      $ref: ./specs/v2/tests/configs/contracts/generic-contract-tests.yaml
    - openApiOperation: "*::/tags"
      $ref: ./specs/v2/tests/configs/contracts/generic-contract-tests.yaml
    - openApiOperation: "*::/transactions"
      $ref: ./specs/v2/tests/configs/contracts/disable-contract-tests.yaml

  integrationTests:
    # These are the detailed positive and negative tests for each endpoint
    # Before checking in any changes they should all be passing but during 
    # iterative development it can be helpful to comment out all the endpoints
    # that you aren't actively working on to speed up the test runs
    - name: /me tests
      $ref: ./specs/v2/tests/configs/variations/me-variations.yaml
    - name: /categories tests
      $ref: ./specs/v2/tests/configs/variations/categories-variations.yaml
    - name: /manual_accounts tests
      $ref: ./specs/v2/tests/configs/variations/manual_account-variations.yaml
    - name: /tags tests
      $ref: ./specs/v2/tests/configs/variations/tags-variations.yaml
    - name: /recurring tests
      $ref: ./specs/v2/tests/configs/variations/recurring-variations.yaml
    - name: /transactions tests
      $ref: ./specs/v2/tests/configs/variations/transactions-variations.yaml

[thim81]

hi @jpjpjp

I think I understand, what you want to achieve.
How could it look like as Portman Config?
Do you have a suggestion?

[jpjpjp]

I'm not sure this is the best design, but perhaps a skipDefaultGeneratedRequests directive? My thought is that any operation listed under this direction would have NO requests generated unless they are specified in one of the subsequent test directives (ie: integrationTests).

If that existed, the configuration above could be written like this:

tests:
  # Generate contract requests for all endpoints except transactions
  skipDefaultGeneratedRequests:
    - openApiOperation: : "*::/transactions"

  integrationTests:
    # For simplicity I omit the integration tests for all the non skipped
    # endpoints in this example config 
    - name: /transactions tests
      $ref: ./specs/v2/tests/configs/variations/transactions-variations.yaml

The resulting generated collection would contain a folder for each of the endpoints defined in the spec, which includes requests to all defined paths/methods, each with post-request scripts for the default contract tests. The exception would be for the ones that match "*::/transactions". For these endpoints there would be NO folder and NO requests generated for these endpoints. (Today there is no way to do this, only to disable the tests as in the configuration above, if you want to also include integration tests for that path).

Then, because an integration test was requested under the "Integration Tests" folder, I would expect to find a folder called "/transactions tests".

I hope that makes sense.

And thanks for considering this! I may be unique in needing this, so please know that I DO have my use case addressed with the configuration in the above post. The only awkward thing I had to do was specify examples for the POST /transactions* endpoints that I knew would fail so that the generated requests would not create data that could not be cleaned up by subsequent requests..