Newman stop after first test assertion fail

[MarcoFierimonte]

Hi all, thanks for this tool it is really useful !

I have a problem with the newman run, basically I run newman with portman and after the first assertion fail the execution is stopped.
I remember that newman continue to execute all the api also if there are fails.

This is the command I run:
npx @apideck/portman --cliOptionsFile ./portman-cli-options.json --runNewman --newmanOptionsFile ./newman-options.json

This is the output of the command:

---

1. AssertionError [GET]::/v1/scenario/projects/:projectCode/scenario/:scenarioCode/revisions/:revisionCode/file - Status code is 2xx
   expected response code to be 2XX but found 500
   at assertion:0 in test-script
   inside "/v1/scenario / Download windcoe file"

2. AssertionFailure [GET]::/v1/scenario/projects/:projectCode/scenario/:scenarioCode/revisions/:revisionCode/file - Status code is 2xx
   at test-script:38:17
   inside "/v1/scenario / Download windcoe file"
   Newman run failed with:
   [GET]::/v1/scenario/projects/:projectCode/scenario/:scenarioCode/revisions/:revisionCode/file - Status code is 2xx

---

So it stops after the first fails also if there are many other run to execute. Any idea about how to solve it?

Thanks.
Marco

[MarcoFierimonte]

Hi @thim81

thanks for your reply.
I am using these newman options (I tried also with defaults values):

{
  "iteration-count": 1,
  "suppress-exit-code" : true,
  "ignore-redirects": true,
  "insecure": true
}

and the newman embedded in portman.
I just tried also to run generated collection directly from newman and it worked correctly.
Let me know if something else is needed.

Thanks.
Marco F.

[nicklloyd]

Hi @MarcoFierimonte - by default, Portman passes abortOnFailure:true to Newman. You should be able to overwrite this to see all fails in a run.

[chyde80]

I am running in a Jenkins pipeline and am seeing "collection run completed" when the run actually has failed tests. Why is the exit code not coming back correctly w/ an error? I have "abortOnFailure" set to false so that I can run the entire collection w/ tests before failing the pipeline stage. I can see in the console that newman/portman is reporting "Collection run completed" instead of what I would expect from Newman's docs which should throw "collection run encountered an error."

[thim81]

Which version of Newman are you using? So we can try to reproduce the issue.
We wrap the Newman CLI in Portman, so perhaps the variable has changed in a newer version of Newman.

[chyde80]

looks like the one packaged with portman in node-modules that it pulled in is newman 5.3.2

[chyde80]

I opened this issue as well #504

[chyde80]

I made a comment on #504 to explain how I got around this, but would be good if it could be fixed.

[thim81]

hi @MarcoFierimonte,

For your information we improve Portman towards the processing of Newman options in Portman 1.18.1
See the changelog