MegaBites-AI/Windows-powershell

1# Overview2While we have done a fairly good job providing adequate coverage for the highest priority areas of PowerShell Core, we still have lots of gaps when we compare what is available in the OSS project vs what we have available as part of the Windows release test infrastructure.3This is to be expected as that infrastructure was built up over more than a decade.4It is not clear, however, how many of the current tests actually provide value, and how much duplication we have.5In any event, there is still a large difference in what we have in PowerShell Core and our older tests.6 7# Data Visibility8In both PowerShell Core or our proprietary tests we can't actually determine how and what cmdlets and engine functions (language/scripting/debugging/remoting) are being tested because we do not gather any usage telemetry on what our tests are doing during test execution.9We have no way via automation to determine how thoroughly we are testing our code except by inspecting the actual tests or looking at code coverage data (which isn't gathered often and only available on full PowerShell).10Further, this doesn't provide us data with regard to the test quality, only that we've covered a specific block of code.11The areas of greatest risk are those where we have no data at all, by collecting coverage data we can illuminate those code paths that are not being used and fill those gaps with new tests.12 13## Telemetry and Logging14While we have some telemetry feeds on Windows, we have _no_ telemetry capabilities on non-Windows platforms.15We should enable telemetry on PowerShell Core on all platforms and review the usage data as soon as possible.16This will provide us much needed visibility in how PowerShell Core is being used and can help us identify areas we should track more closely.17We already have infrastructure in place to allow us see how PowerShell Core is being used, by collecting telemetry from PowerShell Core, we can improve our confidence as we drive to production quality.18 19### Logging20The code which on Windows create ETW logging has been completely stubbed out on Linux/macOS.21We should take advantage of the native logging mechanisms on Linux/macOS and implement a logger similar to the ETW logger on Windows using Syslog (or equivalent).22We could use this data during test runs to identify test gaps.23Simply by capturing the cmdlets and their parameters which are invoked during test would illuminate the gaps we have in our current tests, and allow us to easily fill them.24It is not sufficient to support only one platform because we have many tests which determine at runtime whether or not it should run based on OS, so data from Windows will not be the same as that from Linux or MacOS.25We can also determine engine coverage gaps by measuring test of operators and other language elements.26Without that data, we are simply shooting in the dark.27 28## Code coverage29Even in our lab (STEX) environment we run tests to get code coverage data irregularly, and  PowerShell Core has _no_ tools to gather code coverage data.30We need to investigate possible solutions for code coverage on PowerShell Core.31There are a small number of solutions available:32* [OpenCover](https://github.com/OpenCover/opencover)33    * OpenCover is currently used by corefx to produce code coverage data, also visualization is available via [CoverAlls](https://coveralls.io/github/OpenCover/opencover).34    We should investigate `OpenCover` and determine it's feasibility for us.35    Unfortunately I haven't been able to find a solution for Linux36* DotCover37    * I have contacted `JetBrains` to see if they have any solutions which may be used with .NET Core.38 39If we can get code coverage on PowerShell Core on Windows, we would at least be able to have _some_ data to illuminate our test gaps.40 41Running code coverage more often on full PowerShell is something that we should consider, if it will help us close existing test coverage gaps, but the issue is _not_ test coverage for full PowerShell, but rather PowerShell Core where we have only a small percentage of tests in comparison.42 43## Daily Test Runs44We currently run only those tests which are tagged `CI` excluding the tag `SLOW` as part of our continuous integration systems.45This means roughly 1/3rd of our github tests are not being run on any regular schedule.46In order to provide us with higher confidence in our code, we should be running *ALL* of our tests on a regular basis.47However, running the tests is only the first step, we need an easy way to be notified of test failures, and to track progress of those runs over time.48Tracking this over time affords us the ability to see how our test count increases, implying an improvement in coverage.49It also provides us mechanism whereby we can see trends in instability.50 51### Pending and Skipped Tests52We currently have approximately 300 tests which are marked either as `skipped` or `pending`.53`Pending` tests represent those tests which should be run but are not currently being executed, usually because the underlying functionality is not present.54Over time, the number of `Pending` tests should drive to zero, and we should be tracking the list of tests which are marked `pending` and track those to be sure that they do not spend too long in this state.55`Skipped` tests due to platform applicability are certainly valid, and it is important that we track how many tests are skipped and for what reason.56A test which is skipped for all platforms should be considered carefully as to whether it should exist at all.57In either case, `pending` or `skipped` tests should be tracked over time so we can determine whether we are improving our ability to measure quality.58 59In the best case, the _total_ number of tests is the same across all platforms.60The count of Skipped/Pending tests would naturally be different as not all tests will be applicable to every platform, but if we can have consistency of test count across platforms, it will be easier to compare test results on one platform vs another.61 62## Remoting Considerations63Given that one of the targeted scenarios is to manage heterogeneous environments remotely, we need a test environment which encompasses the available platforms and protocols.64Our current test infrastructure does not have comprehensive support for remote testing except for loopback operations.65We need a matrix of protocols and platforms to ensure that we have adequate coverage to ensure quality.66 67In addition to loopback tests using both WSMan and SSH protocols, we should have the following _minimum_ matrix of client/server connections for both WSMan and SSH (where available) protocols.68* Windows Client->Nano Server69* Windows Client->Linux Server70* Linux Client -> Windows Server71* macOS Client -> Nano Client72* PowerShell Core Client -> Full PowerShell Server73* Full PowerShell Client -> PowerShell Core Server74* Downlevel Full PowerShell Client -> PowerShell Core Server75 76### Challenges77* On Windows, we have cmdlets to enable and disable remote access, those cmdlets do not exist on non-Windows platforms (they rely on configuration stored in the registry), nor do we support configuration for the SSH protocol.78We need to be sure that we can easily enable remoting for the non-Windows platforms and support both WSMan _and_ SSH configurations.79* Our current multi-machine tests do not test the connection code, they simply execute test code remotely and retrieve results and assume a good connection.80The infrastructure used for these tests is STEX which is not an open environment.81We will need to create automation to create and configure the test systems in the test matrix and then invoke tests on them.82It is not clear that our current CI systems can accommodate our needs here as Azure DevOps can supply us with all of the OS images needed.83We may need to create our own heterogeneous environment in Azure, or look to other teams (MS Build Lab/Jenkins) for assistance.84 85We need to investigate whether there are solutions available, and if not, design/implement an environment to meet our needs.86 87# Reporting88Currently, we report against the simplest of KPI:89* is the CI build error free (which is part of the PR/Merge process - and reported on our landing page)90 91There are a number of KPIs which we could report on:92* Code KPIs93    * What is the coverage (% blocks covered) of our `CI` tests94    * What is the coverage of all tests95    * How long is the CI system taking96        * build time97        * test time98    * What is the percentage of pull requests pass `CI` tests99    * What is the trend for `pending` tests100* Process KPIs101    * What is the time delta between a posted PR (with no test errors) and merge102    * How many revisions are needed before a PR is accepted103    * How many PRs are rejected104* Product KPIs105    * What is the usage of PowerShell Core on the various platforms106    * What are the common operations performed by PowerShell Core (cmdlet usage)107    * How many remote connections are created in a session108 109## Reporting Priorities1101. As a baseline, we should report code coverage on current Full PowerShell from the latest Windows release1112. We should report on code coverage as soon as possible.112This is the tool that will enable us to at least determine where we have _no_ data and illuminate test gaps.1133. We should track our CI execution time, which will allow us to keep an eye on our code submission workflow and how much our developers are waiting for builds114 115### Public Dashboard116Because we collect the test results as a build artifact, it is possible to collect and collate this data and provide a number of dash-board reports.117Since we are an OSS project, it makes sense that our reports should be also public.118A public dashboard provides evidence that we are not collecting PII, increasing trust in the project and shows that we are using data to drive product decisions.119We could easily create a web presence in Azure which would enable us to provide reporting on the current and historical state of our tests.120Now that we are running all of our tests on a nightly basis, we should be communicating the results of those test runs.121PowerBI could be used to create the visualizations, which would reduce the time and effort.122 123In order to achieve this we need to:124* Designate an Azure instance to run services and populate Azure tables:125    * Create tools to retrieve and transform git data126    * Create tools to retrieve and transform the test data127* Create PowerBI queries to visualize our KPIs128 129# Release Criteria130We must start defining the release criteria for a production ready release of PowerShell Core, as an initial proposal:131* No open issues for the release132* 80% code coverage of high use cmdlets (cmdlets used by 70% of users, as captured via telemetry)133* 90% code coverage of language elements (coverage error code paths may not be 100%)134* 60% code Coverage on Windows via Github tests135* 100% of our minimum remoting matrix tested136* Acceptance by 50% PowerShell MVPs (via Survey)137* Acceptance by Partners (via Survey)138 139A couple of assumptions have been made for this list:1401) we actually know the high use cmdlets - this implies we have telemetry to determine our cmdlet use1412) we have designated our Partners142 143# Microsoft Release Process144If PowerShell Core is targeted to be released as part of Windows Server or Nano, we will have additional requirements which must be met with regard to acceptance by the Windows org of the PowerShell Core package.145This represents additional work which needs planning and allocated resources.146 147## Replace STEX tests with PowerShell Core tests148Currently we have two distinct set of test artifacts, those used in PowerShell Core and our traditional lab (STEX) based tests.149We are creating new tests for PowerShell Core using the existing tests as guidance, but this is creating duplication between the two sets so we need to find ways to invoke our current PowerShell Core tests as part of our STEX based runs.150When we do this, we can start to delete our existing tests and reduce the maintenance burden due to duplicated tests.151 152**Proposal**153 154* Create a way to invoke our github based tests in lab runs155 156## Historical Tests157The PowerShell team created roughly 90,000 tests since the beginning of the project starting in 2003.158Initially tests were created in a C# framework which was used internally at Microsoft.159After that, a number of script based frameworks were created to run script based tests.160We have decided not to release those for a number of reasons:161* All the frameworks have a core assumption of running in the Microsoft Lab environment and rely on internal Microsoft tools162* All the frameworks rely on specific logging file formats used by internal Microsoft reporting tools163* Some of the frameworks have limitations which we can avoid by recasting/recreating them as `Pester` or `xUnit` tests164 165While, these reasons are not insurmountable, releasing these tests with their custom frameworks does not help us with our desire to promulgate open solutions.166Further, by creating/releasing `xUnit` and `Pester` tests we can take the opportunity to review existing tests and improve them, or eliminate them if they are poor, or duplicate tests.167It is obvious that the review of more than 90,000 tests is an enormous undertaking, and will take some time, and we are committed to creating a set of tests which can provide confidence that PowerShell Core has the same quality as earlier releases of PowerShell.168 169## Challenges ##170While there is a test gap between what is available on Github vs what is available via our internal proprietary tests, it may be possible to use our existing tests (and frameworks) against PowerShell Core.171This is an attractive potential because we have very high confidence of our current tests.172Moreover, we could also collect code coverage data during a test pass.173It would be expected to have a number of test failures because different functionality between PowerShell Core and Full PowerShell but collecting this data would enable us to enumerate those difference.174Unfortunately, the current logging mechanism used in our in-lab tests are not available in PowerShell Core, and a new logging mechanism would need to be created175 176**Proposal**177 178* Investigate the effort to run our historical tests in a PowerShell Core environment179 180# Action Plan181This document represents a number of initiatives, all of which will take resources.182Below is my suggestion for prioritization to reduce risk and improve confidence in our overall quality:183 1841. Implement telemetry feeds for PowerShell Core1852. Implement data collection feeds from our CI systems1863. Create a public dashboard to visualize our KPIs187    * including data from 1, 2 above1884. Implement Code Coverage on PowerShell Core1895. Design/Implement remoting test infrastructure1906. Replace in-lab tests with PowerShell Core tests1917. Investigate feasibility of running current in-lab tests on PowerShell Core192 193These are [tracked](https://github.com/PowerShell/PowerShell/issues?utf8=%E2%9C%93&q=is%3Aissue%20%23testability%20) as issues194