introduce configure abortSuiteOnFailure so that test suite can stop on first failure

[ptrthomas]

has been requested a few times in the last 5 years, and makes sense to me. for example this stack overflow q: https://stackoverflow.com/q/73385367/143475

so the idea is to set a suite-level flag and check its state before commencing any scenario, that should do it

[ptrthomas]

thinking aloud there should be logic and a setting to stop the entire suite on the first failure

[ptrthomas]

• thinking more, there is no need for karate.abortSuite()
• we will introduce a new setting abortSuiteOnFailure - this should take care of most folks
• and we already have a karate.fail() to do this programmatically if needed

[joelpramos]

Could be interesting if there’s a way to force a certain feature to be the first one to run and then conditionally fail the entire suite?

I’m thinking a pre condition for the rest of the tests, almost a classic smoke test to ensure your environment is good to go.

I’m sure there are ways to achieve this programmatically but don’t think there’s a pure karate way

[ptrthomas]

@joelpramos good point. I think the karate-way to do that would be a karate.callSingle() that can call karate.abortSuite() (maybe we need it after all)

[joelpramos]

Could we lean in more into tags and having a @beforeSuite or something for the feature file?

or, assuming you meant a callSingle from karate-config, another JS option introduced at the end of that chain for this behaviour

the concept does sound useful and something I’ve seen done in shell scripts or in bit of hacky way

[ptrthomas]

@joelpramos yes I meant calling karate.abortSuite() in karate-config.js

but I see where you are coming from. it can be argued that karate.callSingle() is confusing and a more "feature compatible" way of running something at the start of a suite is missing today

what I really am not looking forward to do is

• figuring out how this extra stuff gets into the report
• do we allow this routine to update global variables (I guess we already do that)

for now I propose we keep this simple and consider adding a beforeSuite('path/to/some.feature') method to the Runner.Builder API

[joelpramos]

selfishly not my preference but that is the quickest easiest path which will work for most people

For your questions, my thoughts for future reference:

• I'd let it sit in the report as just another Feature/Scenario being executed with no special treatment
• Preference would be not to update global variables (to discourage using this as a workaround when someone wants to add stuff "before all other tests") but would just take the quickest path to get it implemented and if that's allowing global variable manipulation, why not. A note could be added with direction in the docs and if folks choose to disregard it's either their problem (unless to many questions and issues start being raised).

My first thoughts would literally be chaining this in the run() method of the Suite before executing the main set of Features

[joelpramos]

Was looking into some sections today and the Runner.parallel() encapsulates the Suite instance.

When refactoring, do you mind making that accessible from the caller of Runner? The use case I was looking into was pretty much this thread but something more hacky (a quick and dirty) but struggled without the Suite access.

[ptrthomas]

@joelpramos the suite is designed to be immutable, if you want the suite you can always do Results.getSuite() - but if you mean you want to set something, before the tests run - it has to be via some setting on the Runner.Builder

[joelpramos]

I was trying to do the goal of this issue but in a hacky way but unfortunately I had no way to access the futures. Maybe the Runner should provide access to runtime elements within the suite (at a first glance the futures, JobManager, the features - to obtain what the suite actually calculates - drivers, etc.).

[ptrthomas]

@joelpramos yeah, maybe the Runner should return a Future. we don't yet do a build() method like everybody else, maybe that is the way to go

[ptrthomas]

1.3.0 released