MegaBites-AI/Windows-powershell
0308
1# Testing Guidelines2 3Testing is a critical and required part of the PowerShell project.4 5The Microsoft PowerShell team created nearly 100,000 tests over the last 12 years which we run as part of the release process for Windows PowerShell.6Having all of those tests available for the initial release of PowerShell was not feasible,7and we have targeted those tests which we believe will provide us the ability to catch regressions in the areas which have had the largest changes for PowerShell.8It is our intent to continue to release more and more of our tests until we have the coverage we need.9 10For creating new tests, please review the [documents](https://github.com/PowerShell/PowerShell/tree/master/docs/testing-guidelines) on how to create tests for PowerShell.11There is a best practices document for [writing Pester tests](https://github.com/PowerShell/PowerShell/tree/master/docs/testing-guidelines/WritingPesterTests.md).12When adding new tests, place them in the directories as [outlined below](#test-layout).13 14## CI System15 16We use [Azure DevOps](https://azure.microsoft.com/en-us/solutions/devops) as a continuous integration (CI) system for Windows17and non-Windows platforms.18 19In the `README.md` at the top of the repository, you can see Azure CI badge.20It indicates the last build status of `master` branch.21Hopefully, it's green:22 2324 25This badge is **clickable**; you can open corresponding build page with logs, artifacts, and tests results.26From there you can easily navigate to the build history.27 28### Getting CI Results29 30CI System builds and runs tests on every pull request and provides quick feedback about it.31 3233 34These green check boxes and red crosses are **clickable** as well.35They will bring you to the corresponding page with details.36 37## Test Frameworks38 39### Pester40 41Our script-based test framework is [Pester](https://github.com/Pester/Pester).42This is the framework which we are using internally at Microsoft for new script-based tests,43and a large number of the tests which are part of the PowerShell project have been migrated from that test base.44Pester tests can be used to test most of PowerShell behavior (even some API operations can easily be tested in Pester).45 46Substantial changes were required, to get Pester executing on non-Windows systems.47These changes are not yet in the official Pester code base.48Some features of Pester may not be available or may have incorrect behavior.49Please make sure to create issues in [PowerShell/PowerShell](https://github.com/PowerShell/PowerShell/issues) (not Pester) for anything that you find.50 51#### Test Tags52 53The Pester framework allows `Describe` blocks to be tagged, and our CI system relies on those tags to invoke our tests.54 55One of the following tags must be used:56 57* `CI` - this tag indicates that the tests in the `Describe` block will be executed as part of the CI/PR process58* `Scenario` - this tag indicates a larger scale test interacting with multiple areas of functionality and/or remote resources, these tests are also run daily.59* `Feature` - tests with this tag will not be executed as part of the CI/PR process,60 but they will be executed on a daily basis as part of a `cron` driven build.61 They indicate that the test will be validating more behavior,62 or will be using remote network resources (ex: package management tests)63 64Additionally, the tag:65 66* `SLOW` indicates that the test takes a somewhat longer time to execute (97% of our `CI` tests take 100ms or less), a test which takes longer than 1 second should be considered as a candidate for being tagged `Slow`67 68#### Requesting additional tests for a PR69 70In our CI systems, we normally run only run tests tagged with `CI`.71If in the first line of the last (most recent) commit description you add `[Feature]`,72we will ensure that we will also run the tests tagged with `Feature`.73When you would want to do this:74 75- You have added or changed a `Feature` test.76- A maintainer asks you to run the `Feature` tests.77- Based on experience, you are confident that a maintainer will ask you to run the `Feature` tests.78 79#### Validating packaging changes for a PR80 81By default, our CI system does a build and run tests for a PR and does not exercise code to create a package.82If your PR includes changes to packaging, you can have the CI system exercise the packaging code by83using `[Package]` as the first line in the commit message.84When you would want to do this:85 86- You made change to PowerShell Core packaging87- A maintainer asks you to run as `[Package]`88 89### xUnit90 91For those tests which are not easily run via Pester, we have decided to use [xUnit](https://xunit.net/) as the test framework.92Currently, we have a minuscule number of tests which are run by using xUnit.93 94## Running tests outside of CI95 96When working on new features or fixes, it is natural to want to run those tests locally before making a PR.97These helper functions are part of the build.psm1 module to help with that:98 99* `Start-PSPester` will execute all Pester tests which are run by the CI system100* `Start-PSxUnit` will execute the available xUnit tests run by the CI system101 102Our CI system runs these as well; there should be no difference between running these on your dev system, versus in CI.103 104When running tests in this way, be sure that you have started PowerShell with `-noprofile` as some tests will fail if the105environment is not the default or has any customization.106 107For example, to run all the Pester tests for CI (assuming you are at the root of the PowerShell repo):108 109```PowerShell110Import-Module ./build.psm1111Start-PSPester112```113 114If you wish to run specific tests, that is possible as well:115 116```PowerShell117Start-PSPester -Path test/powershell/engine/Api118```119 120Or a specific Pester test file:121 122```PowerShell123Start-PSPester -Path test/powershell/engine/Api/XmlAdapter.Tests.ps1124```125 126### What happens after your PR?127 128When your PR has successfully passed the CI test gates, your changes will be used to create PowerShell binaries which can be run129in Microsoft's internal test frameworks.130The tests that you created for your change and the library of historical tests will be run to determine if any regressions are present.131If these tests find regressions, you'll be notified that your PR is not ready, and provided with enough information to investigate why the failure happened.132 133## Test Layout134 135We have taken a functional approach to the layout of our Pester tests and you should place new tests in their appropriate location.136If you are making a fix to a cmdlet in a module, the test belongs in the module directory.137If you are unsure, you can make it part of your PR, or create an issue.138 139The current layout of tests is:140 141* test/powershell/engine142* test/powershell/engine/Api143* test/powershell/engine/Basic144* test/powershell/engine/ETS145* test/powershell/engine/Help146* test/powershell/engine/Logging147* test/powershell/engine/Module148* test/powershell/engine/ParameterBinding149* test/powershell/engine/Remoting150* test/powershell/engine/Runspace151* test/powershell/engine/Logging/MessageAnalyzer152* test/powershell/Host153* test/powershell/Host/ConsoleHost154* test/powershell/Host/TabCompletion155* test/powershell/Language156* test/powershell/Modules157* test/powershell/Provider158* test/powershell/SDK159* test/powershell/Security160* test/powershell/Language/Classes161* test/powershell/Language/Interop162* test/powershell/Language/Operators163* test/powershell/Language/Parser164* test/powershell/Language/Interop/DotNet165* test/powershell/Language/Scripting166* test/powershell/Language/Scripting/Debugging167* test/powershell/Language/Scripting/NativeExecution168* test/powershell/Modules/Microsoft.PowerShell.Archive169* test/powershell/Modules/Microsoft.PowerShell.Core170* test/powershell/Modules/Microsoft.PowerShell.Diagnostics171* test/powershell/Modules/Microsoft.PowerShell.Management172* test/powershell/Modules/Microsoft.PowerShell.Security173* test/powershell/Modules/Microsoft.PowerShell.Utility174* test/powershell/Modules/PSReadLine175