Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replace allure-xunit's custom attributes with runner reporter (fixes #344) #366

Merged
merged 14 commits into from
Jul 6, 2023

Conversation

delatrie
Copy link
Contributor

@delatrie delatrie commented Jun 19, 2023

This is the first piece of work to improve allure-xunit out-of-box experience. The main goal here is to drop the [AllureXunit] and [AllureXunitTheory] attributes in favor of the built-in ones: [Fact] and [Theory].

Context

Basically, to generate a report we need to listen for xunit's execution events and extract data provided by xunit in event messages. There are a couple of ways to achieve this.

Previous implementation

Custom attributes ([AllureXunit] and [AllureXunitTheory]) were used to discover tests via the AllureXunitDiscover creating an instance of AllureXunitTestCase for each and injecting the event listener (namely, AllureMessageBus) to handle the events.

The disadvantage is that in order to use allure-xunit one has to modify their test code first replacing built-in xunit attributes with custom ones.

New implementation

We drop the currently implemented extension mechanism in favor of a custom runner reporter (AllureRunnerReporter) that may be used by xunit to directly create an event handler (in our case, an instance of AllureMessageSink) allowing us to bypass the need of custom attributes.

The [AllureXunit] and [AllureXunitTheory] attributes are marked as obsoleted.

Usage

Just install the package and run the tests as usual.

In some cases, allure reporter may not run automatically. In that case, enable the reporter explicitly:

  • For VS runner (in Visual Studio, or via the dotnet CLI) use RunConfiguration.ReporterSwitch=allure runsettings option.
  • For console runner use -noautoreporters -allure CLI arguments (make sure Allure.XUnit.dll and xunit.console.exe are in the same directory).
  • For MSBuild runner speficy the NoAutoReporters="true" Reporter="allure" properties in your xunit task definition (make sure Allure.XUnit.dll and xunit.runner.msbuild.net452.dll are in the same directory).

Additional changes

The PR also adds support for static test methods:

using Xunit;

public class MyTestClass
{
    [Fact]
    public static void StaticMethod()
    {
        Assert.Equal(3, 1 + 2);
    }
}

Previously, such a method caused a test run to be aborted.

Known issues

One reporter per test run

Xunit allows only one active runner reporter. Currently it's impossible to produce allure output files AND, say, send the output to TeamCity at the same test run.

Runners are quite composable though. It should not be that hard to wrap 3rd party's event listener in our own. Support for that will be added in near future.

Arguments of some theories might potentially be unreported

Xunit doesn't provide test arguments to event listeners if one the following conditions is true:

  1. A user explicitly disabled theory data pre-enumeration during discovery phase (either on per-test basis or globally).
  2. Xunit can't serialize one of a test's argument.

In that case xunit creates one ITestCase instance with null TestMethodArguments property, and multiple ITest instances, one per each argument set. There is no public API allowing to get these arguments.

The PR workarounds this problem by attaching a hook to the XunitTestRunner's constructor at runtime using Harmony. Harmony has its own limitations, the most concerning of which is the inability of Harmony to patch an inlined method. If the constructor of XunitTestRunner is inlined, the patch doesn't work, and arguments of a test that falls into one of the categories above are not reported by allure-xunit. The warning is issued in such a case advising to switch to debug mode.

Checklist

@delatrie delatrie marked this pull request as ready for review June 19, 2023 18:38
@delatrie delatrie changed the title Replace allure-xunit's custom attributes with runner reporter Replace allure-xunit's custom attributes with runner reporter (fixes #344) Jun 20, 2023
@neparij
Copy link
Contributor

neparij commented Jun 30, 2023

Add README notes.
Create issues from Known issues section

Allure.XUnit/Allure.XUnit.csproj Show resolved Hide resolved
@@ -31,142 +31,17 @@ static AllureXunitHelper()
Environment.SetEnvironmentVariable(allureConfigEnvVariable, allureConfigPath);
}

public static void StartTestContainer(ITestCaseStarting testCaseStarting)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should be either [Obsolete] with for-removal either we should have migration notice

Copy link
Contributor Author

@delatrie delatrie Jul 3, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've moved the code related to allure model manipulation back to AllureXunitHelper to simplify AllureMessageSink a little. Public methods are refactored and marked as [Obsolete].

@delatrie delatrie self-assigned this Jun 30, 2023
  - Allure-xunit's code that creates, updates, starts, stops and writes
    allure objects is factored out from AllureMessageSink to
    AllureXunitHelper .
  - Removal of AllureXunitHelper's public methods was reverted. The
    methods are marked as obsolete.
  - Add missing await to an allure-xunit example to suppress the
    compilation warning.
  - Mark obsoleted allure-xunit attributes as non-browsable to prevent
    them from appearing in IDE's suggestions.
@delatrie
Copy link
Contributor Author

delatrie commented Jul 3, 2023

Add README notes.
Create issues from Known issues section

Done!
See #368 and #369 and the updated readme.

@delatrie delatrie requested a review from neparij July 4, 2023 08:39
"This method wasn't supposed to be in the public API. It's not " +
"relevant anymore and will be removed in a future release. You " +
"should create model classes by yourself and use " +
"AllureLyfecycle.Instance methods instead"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AllureLifecycle is a part of Allure.Net.Commons.

@neparij neparij merged commit 4a55fce into preview Jul 6, 2023
@delatrie delatrie deleted the xunit-native-attributes4 branch July 6, 2023 09:15
neparij added a commit that referenced this pull request Oct 16, 2023
* Adds AllureBefore and AllureAfter to Allure.NUnit package

* NUnit: Ability to change testresult on fixture executions

* Preserve testResult.start time if it exists in update container

* SpecFlowNunit: add custom labels

* Fix missing output attachments in NUnit adapter

* Added step logging for NUnit (closes #312)

* Fixed NullPointerException if NUnit StepLogger was null (#315)

* embed untracked sources

* build process update

* NUnit Fix async behaviour in [AllureStep] aspect + Allow '{obj.prop}' placeholders in [AllureStep] step name (#329)

Co-authored-by: Кабанов Константин Юрьевич <[email protected]>

* Package assets fix

* NUnit - Add an ability to save OneTimeSetUp step results (#333)

* Handle case when underlying step returns Task<T> masked as non-generic Task (#343)

* Unify xunit and nunit step acpects

* Replace allure-xunit's custom attributes with runner reporter (fixes #344) (#366)

* Add support for native xunit attributes

* Fix AllureXunitTheoryAttribute base class

* Fix missing theories

* Fix missing parameters and fixture duplication for some tests

* Add support for static test functions

* Update xunit examples. Obsolete allure-xunit attributes

* Update message sink to match new steps implementation

* Remove irrelevant log method from runner

* Bump harmony to 2.3-prerelease.2. Warn if no args reported

Change Lib.Harmony dependency to 2.3-prerelease.2 for .net 7 support.
Add warning if the patch via harmony didn't work for theory args and
args aren't reported on the testcase level.

* Fix reference to lib.harmony to be listed by dotnet

* Factor out allure-related code. Tidying up obsolescence

  - Allure-xunit's code that creates, updates, starts, stops and writes
    allure objects is factored out from AllureMessageSink to
    AllureXunitHelper .
  - Removal of AllureXunitHelper's public methods was reverted. The
    methods are marked as obsolete.
  - Add missing await to an allure-xunit example to suppress the
    compilation warning.
  - Mark obsoleted allure-xunit attributes as non-browsable to prevent
    them from appearing in IDE's suggestions.

* Add deprecation info and known issues to allure-xunit README

* Fix issue links in allure-xunit readme

* Fix obsolescence message for AllureXunitHelper's public methods

* New context-based state management model for allure-csharp (#371)

* Add AllureContext lass to isolate allure state from threads and async tasks

* Add concurrency tests. Reformulate context in terms of active/inactive

* Add tests on the context capturing by child threads/tasks

* Modify lifecycle/storage to use new context model. Add context get/set API

* Make CoreStepsHelper use AllureLifecycle's context

* Make lifecycle thread safe. Fix multithreading issues in lifecycle tests

* Several improvements on context. Allure-xunit migration

- Add bool props to test context to context's public API
- Fix context string conversion
- Migrate allure-xunit's code to the new context implementation

* Bump C# language version to 11 for all projects

* Enable null checks in some classes of allure-xunit

* Make lifecycle methods with explicit uuids obsoleted

* Minor changes in context and lifecycle

  - Improve error messages and doc comments
  - Add DebuggerDisplay for context
  - Uuid in string representation of context if names are empty
  - Old storage uuid-based methods obsoleted
  - Make RunInContext return modified context
  - Replace unsupported container nesting with test nested
    into each container in chain
  - Container and test starting now add objects into storage by
    uuids (transition period)

* Rewrite Allure.NUnit to support the new context scheme

* Fix Allure.XUnit usage of RunInContext

* Rewrite Allure.SpecFlowPlugin to support new context scheme

Additionally:
  - Change broken status to failed for some scenario and step conditions
  - Add extra test case instead of changing state of existing one in case
    after feature failed
  - Enable nullable check project-wide

* Separate context from storage

  - AllureContext moved to Allure.Net.Commons namespace
  - Context and storage are separated; Storage will be removed later
  - File scope namespaces for lifecycle and context

* Fix obsolete messages. Add file scoped namespaces

* Revert obsoleted StopFixtureSuppressTestCase to be intact

* Add new info to commons README

* Fix invalid test start. Remove rudimentary test start check

* Remove dotnet SDK 3.1 installation from build pipeline

* Change .NET SDK version to 7.0 in build pipeline

* Github pipelines fix

  - bump actions/checkout to 3.5.3
  - bump actions/setup-dotnet to 3.2.0
  - add dotnet 3.1 and 6.0 SDKs back to build and test
  - Change --no-restore to --no-build in dotnet test step

* Rename a placeholder scenario

* Implement requested changes for PR #371

  - Add allure directory clean before all tests in Allure.NUnit.Examples
  - NUnit.Allure.Core.AllureExtensions.AddScreenDiff is back
    and marked as obsolete (it just directly calls
    AllureLifecycle.AddScreenDiff now).
  - Intendation in Allure.SpecFlowPlugin.csproj fixed.

* NUnit: Fix inconsistent one-time fixtures behavior (fixes #286, #374 and #375) (#380)

* Fix inconsistent one-time fixtures behavior (fixes #374)

* Fix crash when running test from empty namespace (fixes #375)

* Fix indentation for AllureAsyncOneTimeSetUoTests.cs

* Allure-xunit: support for the second reporter and xunit 2.5.0. Updated packaging and AspectInjector (fixes 368) (#382)

* xunit: change sink base, remove redundant err logging

* Add ability to compose xunit runner reporters (implements #368)

* Extract allure-xunit runner reporter into separate project

* Factor out common props. Bump and tune injector

* Reflect new project structure in solution

* Enable fully deterministic build in CI

* Add startup message to Allure.XUnit

* Fix second reporter property name in config for Allure.XUnit's example

* Rewrite how to run section and fix allure-xunit's README

* Fix comment in Directory.Build.props

* Revert AspectInjector build optimization

* Add link to allureConfig.json issue

* Fix release workflow to apply version via .props

* Change net.commons description

It's quite different from allure-java-commons now.

* Improve readability of ResolveExplicitReporter

* Fix typo in allure-xunit README

* release 2.10.0-preview.1

* set next development version 2.11

* Selective run support: commons, nunit, xunit (#390)

* Implement selective run commons

* Change GetAllureId to take an enumeration

* Implement selective run for allure-nunit

* Add a shorthand for testplan's IsMatch

* Implement selective run for allure-xunit

* Move TestPlan property to commons. Rename selection check method

* Allure-specflow: selective run support (#392)

* Fix regression: non-working lambda-steps and lambda-fixtures API (#393)

* allure-xunit: provide default config when allureConfig.json is missing

Fixes #381

* Fix async step/fixture wrapper functions

  - make async Step, Before and After obey .NET's ExecutionContext capturing rules
  - refactor CoreStepsHelper to be more testable
  - write tests for CoreStepsHelper's Step, Before and After

More tests are needed to fully cover CoreStepsHelper.

Fixes #383
Fixed #388

* Add async suffix to private step methods

This helps to make distinction more clear to a reader.

* Fix identification properties and parameter formatting (#395)

* Bump NUnit for test and example projects

* Add json schemas for allureConfig and testplan

* Fix schema for allure-xunit and allure-specflow config

* Add legacy naming switch to config

* Update schema links to current branch

* Add schema links update task to release workflow

* Implement UUID and method-based fullName algorithms

* Common fns to calculate testCaseId, historyId and parameter value

* Add docstrings to new id-related functions

* Add function to create fullName from class

* Fix allureConfig item type for allure-nunit examples

* Add missing setter for UseLegacyIds config property

* Add framework label factory

* Fix uuid, fullName, historyId, testCaseId for allure-nunit

UseLegacyIds option is honored.

Additionally:
  - fix uuid for containers
  - fix missing historyId for ignored tests. Same id rules apply as for non-ignored tests (fixes #345)
  - add NUnit 3 framework label to test results
  - remove suites from AllureDisplayIgnored (now suites can be applied directly)
  - fix parameter names for parametrized test methods

* Apply custom formatters to step title and parameters

Now step title interpolation and step parameter value conversion uses the same
algorithm as test parameter conversions.

Fixes #377

* Add language label to commons and NUnit

* Fix uuid, fullName, historyId, testCaseId for allure-xunit

Legacy identifiers switch is honored.

Additionally:
  - Add framework and language labels to allure-xunit test results
  - Fix test parameters formatting. Custom type formatters are also used now

* Add no-formatters step helper overloads back to public API

* Revert autoedit this removal from StepFunctionTests.cs

* Fix uuid, fullName, historyId, testCaseId for allure-specflow

Legacy ids switch is honored.

Implements #387

* Implement requested changes

* Lower the required AspectInjector version for users back to 2.8.1 (workaround for #391) (#396)

* Add workaround for #391

* Comment AspectInjector workaround step in publish.yml

* Add Rosetta warning in package READMEs. Fix allure-nunit readme

* Update allure-xunit and allure-specflow README (#397)

---------

Co-authored-by: qameta-ci <[email protected]>
Co-authored-by: Andrey Gurenkov <[email protected]>
Co-authored-by: Constantine <[email protected]>
Co-authored-by: Кабанов Константин Юрьевич <[email protected]>
Co-authored-by: Alexander Kulakov <[email protected]>
Co-authored-by: Dmitry Pavlushin <[email protected]>
Co-authored-by: Maxim <[email protected]>
@NikitaEgorov
Copy link

NikitaEgorov commented Nov 20, 2023

This PR has broken Rider support completely

https://youtrack.jetbrains.com/issue/RSRP-488382/Load-custom-reporter-dll

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Get rid of allure-xunit's [AllureXunit] and [AllureXunitTheory] attributes
3 participants