Prevent unrelated error from being displayed if a scenario step has failed by craig-mcmahon · Pull Request #6743 · Codeception/Codeception

craig-mcmahon · 2024-02-29T13:50:56Z

This fixes #6716 which results in incorrect test failures being reported where a scenario has a failed assertion that has been caught within the test itself or via expectException

…ailed

craig-mcmahon · 2024-02-29T14:22:37Z

Unsure how to add a test for this as it only occurs on the failure of a test.

I used the following test

<?php

namespace Tests\Unit;

use Codeception\Test\Unit;
use PHPUnit\Framework\AssertionFailedError;
use Tests\Support\UnitTester;

class TestTest extends Unit {

	protected UnitTester $tester;

	public function test1(){
		static::fail(':-(');
	}

	public function test2(){
		$this->expectException(AssertionFailedError::class);
		$this->tester->assertFalse(true);
	}

}

Output before fix

Codeception PHP Testing Framework v5.1.0 https://stand-with-ukraine.pp.ua

Tests.Unit Tests (2) -------------------------------------------------------------------------------------------------------------
✖ TestTest: Test1(0.01s)
✖ TestTest: Test2(0.00s)
----------------------------------------------------------------------------------------------------------------------------------
Time: 00:00.102, Memory: 16.87 MB

There was 1 failure:
1) TestTest: Test2
 Test  tests/Unit/TestTest.php:test2
:-(
#1  /usr/src/app/tests/Unit/TestTest.php:14
#2  /usr/src/app/vendor/bin/codecept:119

FAILURES!
Tests: 2, Assertions: 2, Failures: 1.

Output after fix

Codeception PHP Testing Framework v5.1.0 https://stand-with-ukraine.pp.ua

Tests.Unit Tests (2) -------------------------------------------------------------------------------------------------------------
✖ TestTest: Test1(0.01s)
✔ TestTest: Test2(0.00s)
----------------------------------------------------------------------------------------------------------------------------------
Time: 00:00.087, Memory: 16.87 MB

There was 1 failure:
1) TestTest: Test1
 Test  tests/Unit/TestTest.php:test1
:-(
#1  /usr/src/app/tests/Unit/TestTest.php:14
#2  /usr/src/app/vendor/bin/codecept:119

FAILURES!
Tests: 2, Assertions: 2, Failures: 1.

Naktibalda · 2024-02-29T20:23:26Z

Your change broke conditional assertions
Probably because condition assertion implementation passes a clone of test to FailEvent:

Codeception/src/Codeception/Scenario.php

Line 79 in fcfa1d0

$failEvent = new FailEvent(clone($this->test), $f, 0);

Naktibalda · 2024-02-29T20:42:37Z

I think that the best way to compare if 2 variables contain the same test is to compare results of \Codeception\Test\Descriptor::getTestSignatureUnique().

if ($lastFailure && Descriptor::getTestSignatureUnique($lastFailure->getTest()) === Descriptor::getTestSignatureUnique($this)

For ideas how to implement test for your change see

Codeception/tests/cli/RunCest.php

Lines 606 to 669 in fcfa1d0

    
           public function testsWithConditionalFails(CliGuy $I) 
        
           { 
        
               $I->executeCommand('run scenario ConditionalCept --no-exit'); 
        
               $I->seeInShellOutput('There were 3 failures'); 
        
               $I->seeInShellOutput('Fail  File "not-a-file" not found'); 
        
               $I->seeInShellOutput('Fail  File "not-a-dir" not found'); 
        
               $I->seeInShellOutput('Fail  File "nothing" not found'); 
        
           } 
        
           public function runTestWithAnnotationDataprovider(CliGuy $I) 
        
           { 
        
               $I->executeCommand('run scenario -g dataprovider --steps'); 
        
               $I->seeInShellOutput('OK (18 tests'); 
        
           } 
        
           public function runFailedTestAndCheckOutput(CliGuy $I) 
        
           { 
        
               $I->executeCommand('run scenario FailedCept', false); 
        
               $testPath = implode(DIRECTORY_SEPARATOR, ['tests', 'scenario', 'FailedCept.php']); 
        
               $I->seeInShellOutput('1) FailedCept: Fail when file is not found'); 
        
               $I->seeInShellOutput('Test  ' . $testPath); 
        
               $I->seeInShellOutput('Step  See file found "games.zip"'); 
        
               $I->seeInShellOutput('Fail  File "games.zip" not found at ""'); 
        
           } 
        
           public function runTestWithCustomSetupMethod(CliGuy $I) 
        
           { 
        
               $I->executeCommand('run powers PowerUpCest'); 
        
               $I->dontSeeInShellOutput('FAILURES'); 
        
           } 
        
           public function runCestWithTwoFailedTest(CliGuy $I) 
        
           { 
        
               $I->executeCommand('run scenario PartialFailedCest', false); 
        
               $I->seeInShellOutput('See file found "testcasetwo.txt"'); 
        
               $I->seeInShellOutput('See file found "testcasethree.txt"'); 
        
               $I->seeInShellOutput('Tests: 3,'); 
        
               $I->seeInShellOutput('Failures: 2.'); 
        
           } 
        
           public function failedTestFollowedByExceptionReportsCorrectStep(CliGuy $I) 
        
           { 
        
               $I->executeCommand('run scenario FailureAndExceptionCest', false); 
        
               $I->seeInShellOutput('1. $I->throwException("test exception")'); 
        
               $I->seeInShellOutput('Step  Assert same 1,2'); 
        
               $I->seeInShellOutput('Failed asserting that 2 is identical to 1.'); 
        
           } 
        
           public function displaysAllFailedConditionalStepsInOneCest(CliGuy $I) 
        
           { 
        
               $I->executeCommand('run scenario MultipleConditionalFailsCest --no-ansi', false); 
        
               $I->seeInShellOutput('x MultipleConditionalFailsCest: Multiple fails 3x[F]'); 
        
               $I->dontSeeInShellOutput('MultipleConditionalFailsCest: Multiple fails 2x[F]'); 
        
               $I->dontSeeInShellOutput('MultipleConditionalFailsCest: Multiple fails [F]'); 
        
               $I->dontSeeInShellOutput('MultipleConditionalFailsCest: Multiple fails[F]'); 
        
               $I->seeInShellOutput('There were 3 failures:'); 
        
               $I->seeInShellOutput('Step  Can see file found "not-a-file"'); 
        
               $I->seeInShellOutput('Step  Can see file found "not-a-dir"'); 
        
               $I->seeInShellOutput('Step  Can see file found "nothing"'); 
        
               $filename = implode(DIRECTORY_SEPARATOR, ['tests', 'scenario','MultipleConditionalFailsCest.php']); 
        
               $I->seeInShellOutput(' 1. $I->canSeeFileFound("not-a-file") at ' . $filename . ':7'); 
        
               $I->seeInShellOutput(' 7. $I->canSeeFileFound("not-a-dir") at ' . $filename . ':13'); 
        
               $I->seeInShellOutput(' 13. $I->canSeeFileFound("nothing") at ' . $filename . ':19'); 
        
           }

Codeception/tests/cli/OrderCest.php

Lines 16 to 75 in fcfa1d0

    
           public function checkForFails(CliGuy $I) 
        
           { 
        
               $I->amInPath('tests/data/sandbox'); 
        
               $I->executeCommand('run order FailedCept.php --no-exit'); 
        
               $I->seeFileFound('order.txt', 'tests/_output'); 
        
               $I->expect('global bootstrap, initialization, beforeSuite, before, bootstrap, test, fail, after, afterSuite'); 
        
               $I->seeFileContentsEqual("BIB([STF])"); 
        
           } 
        
           public function checkForCanCantFails(CliGuy $I) 
        
           { 
        
               $I->amInPath('tests/data/sandbox'); 
        
               $I->executeCommand('run order CanCantFailCept.php --no-exit'); 
        
               $I->seeFileFound('order.txt', 'tests/_output'); 
        
               $I->expect( 
        
                   'global bootstrap, initialization, beforeSuite, before, bootstrap, test,' 
        
                   . ' test, fail, after, afterSuite' 
        
               ); 
        
               $I->seeFileContentsEqual("BIB([STTF])"); 
        
           } 
        
           public function checkForCanCantFailsInCest(CliGuy $I) 
        
           { 
        
               $I->amInPath('tests/data/sandbox'); 
        
               $I->executeCommand('run order CanCantFailCest.php --no-ansi --no-exit'); 
        
               $I->seeInShellOutput('x CanCantFailCest: Test one [F]'); 
        
               $I->seeInShellOutput('x CanCantFailCest: Test two [F]'); 
        
               $I->dontSeeInShellOutput('+ CanCantFailCest: One'); 
        
               $I->dontSeeInShellOutput('+ CanCantFailCest: Two'); 
        
               $I->seeFileFound('order.txt', 'tests/_output'); 
        
               $I->expect( 
        
                   'global bootstrap, initialization, beforeSuite, before, bootstrap, test,' 
        
                   . ' test, fail, test, test, fail, after, afterSuite' 
        
               ); 
        
               $I->seeFileContentsEqual("BIB([TTF][TTF])"); 
        
           } 
        
           public function checkForCanCantFailsInTest(CliGuy $I) 
        
           { 
        
               $I->amInPath('tests/data/sandbox'); 
        
               $I->executeCommand('run order CanCantFailTest.php --no-ansi --no-exit'); 
        
               $I->seeInShellOutput('x CanCantFailTest: One'); 
        
               $I->seeInShellOutput('x CanCantFailTest: Two'); 
        
               $I->dontSeeInShellOutput('+ CanCantFailTest: One'); 
        
               $I->dontSeeInShellOutput('+ CanCantFailTest: Two'); 
        
               $I->seeFileFound('order.txt', 'tests/_output'); 
        
               $I->expect( 
        
                   'global bootstrap, initialization, beforeSuite, before, bootstrap, test,' 
        
                   . ' test, fail, test, test, fail, after, afterSuite' 
        
               ); 
        
               $I->seeFileContentsEqual("BIB([TTF][TTF])"); 
        
           } 
        
           public function checkSimpleFiles(CliGuy $I) 
        
           { 
        
               $I->amInPath('tests/data/sandbox'); 
        
               $I->executeCommand('run order --no-exit --group simple'); 
        
               $I->seeFileFound('order.txt', 'tests/_output'); 
        
               $I->seeFileContentsEqual("BIBP([ST][STTF][STF][ST])"); 
        
           }

contains some similar tests too, but they are more difficult to understand.

…comparison due to cloning

craig-mcmahon · 2024-03-01T10:39:06Z

Thanks for the feedback, your suggestion fixed the broken test. I have also refactored to avoid the loop when not needed.

I will look at adding a new test in the next few days

craig-mcmahon · 2024-03-04T13:46:46Z

I have added a test to cover the new change, and fixed the code style check that failed in the last run

craig-mcmahon · 2024-03-05T17:52:41Z

And fix pushed for the code style in the tests

Naktibalda · 2024-03-07T07:28:59Z

Released as 5.1.2

Prevent unrelated error from being displayed if a scenario step has f…

b3292eb

…ailed

Fix conditional asserts by checking test signature instead of object …

62c9f16

…comparison due to cloning

craig-mcmahon added 2 commits March 4, 2024 13:17

Code style

041a653

Add test

81ad458

Fix code style for test

d1622ff

Naktibalda merged commit 066deb0 into Codeception:5.1 Mar 5, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Prevent unrelated error from being displayed if a scenario step has failed#6743

Prevent unrelated error from being displayed if a scenario step has failed#6743
Naktibalda merged 5 commits intoCodeception:5.1from
craig-mcmahon:prevent-incorrect-failure-being-logged

craig-mcmahon commented Feb 29, 2024

Uh oh!

craig-mcmahon commented Feb 29, 2024

Uh oh!

Naktibalda commented Feb 29, 2024

Uh oh!

Naktibalda commented Feb 29, 2024

Uh oh!

craig-mcmahon commented Mar 1, 2024

Uh oh!

craig-mcmahon commented Mar 4, 2024

Uh oh!

craig-mcmahon commented Mar 5, 2024

Uh oh!

Naktibalda commented Mar 7, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

craig-mcmahon commented Feb 29, 2024

Uh oh!

craig-mcmahon commented Feb 29, 2024

Uh oh!

Naktibalda commented Feb 29, 2024

Uh oh!

Naktibalda commented Feb 29, 2024

Uh oh!

craig-mcmahon commented Mar 1, 2024

Uh oh!

craig-mcmahon commented Mar 4, 2024

Uh oh!

craig-mcmahon commented Mar 5, 2024

Uh oh!

Naktibalda commented Mar 7, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants