MegaBites-AI/Windows-powershell

1# Internals of build process2 3The purpose of this document is to explain build process **internals** with subtle nuances.4This document is not by any means complete.5The ultimate source of truth is the code in `.\build.psm1` that's getting executed on the corresponding CI system.6 7This document assumes that you can successfully build PowerShell from sources for your platform.8 9## Top directory10 11We are calling `dotnet` tool build for `$Top` directory12 13- `src\powershell-win-core` for CoreCLR on Windows.14- `src\powershell-unix` for CoreCLR on Linux and macOS.15 16### Dummy dependencies17 18We use dummy dependencies between projects to leverage `dotnet` build functionality.19For example, `src\powershell-win-core\powershell-win-core.csproj` has dependency on `Microsoft.PowerShell.Commands.Diagnostics.csproj`,20but in reality, there is no build dependency.21 22Dummy dependencies allow us to build just `$Top` folder, instead of building several folders.23 24### Dummy dependencies rules25 26If assembly is part of CoreCLR build,27it should be listed as a dependency for `$Top` folder (`src\powershell-unix` or `src\powershell-win-core`)28 29## Preliminary steps30 31### ResGen32 33Until the .NET CLI `dotnet-resgen` tool supports the generation of strongly typed resource access classes34(tracked by [Microsoft/msbuild #2272](https://github.com/Microsoft/msbuild/issues/2272)),35we run our own C# [ResGen tool](../../src/ResGen).36While the `Start-PSBuild` command runs this automatically via the `Start-ResGen` function,37it does *not* require PowerShell.38The same command can be run manually:39 40```sh41cd src/ResGen42dotnet restore43dotnet run44```45 46Running the program does the following work:47 48- For each project, given a `resources` folder, create a `gen` folder.49- For each `*.resx` file from the `resources` folder,50  create a strongly typed C# resource access class,51  and write it to the corresponding `*.cs` file in the `gen` folder.52 53These files are *not* automatically updated on each build,54as the project lacks the ability to detect changes.55Thus, running it for every build would break incremental recompilation.56 57If you pull new commits and get an error about missing strings,58you likely need to delete the `gen` folders and re-run the tool.59 60### Type Catalog61 62A pre-generated catalog of C# types is used in PowerShell to help type resolution.63Generating this catalog is a pre-build step that is run via `Start-TypeGen`,64which `Start-PSBuild` calls.65Again, however, PowerShell is not required.66The necessary steps can be run manually:67 68```sh69cd ../TypeCatalogGen70dotnet restore71dotnet run ../System.Management.Automation/CoreCLR/CorePsTypeCatalog.cs powershell.inc72```73 74The file `powershell.inc` is generated by running a custom MSBuild target.75`Start-TypeGen` handles generating this file,76but you can also do it manually by navigating to the `src` directory and running the following commands:77 78```sh79targetFile="Microsoft.PowerShell.SDK/obj/Microsoft.PowerShell.SDK.csproj.TypeCatalog.targets"80cat > $targetFile <<-"EOF"81<Project>82    <Target Name="_GetDependencies"83            DependsOnTargets="ResolveAssemblyReferencesDesignTime">84        <ItemGroup>85            <_RefAssemblyPath Include="%(_ReferencesFromRAR.HintPath)%3B"  Condition=" '%(_ReferencesFromRAR.NuGetPackageId)' != 'Microsoft.Management.Infrastructure' "/>86        </ItemGroup>87        <WriteLinesToFile File="$(_DependencyFile)" Lines="@(_RefAssemblyPath)" Overwrite="true" />88    </Target>89</Project>90EOF91dotnet msbuild Microsoft.PowerShell.SDK/Microsoft.PowerShell.SDK.csproj /t:_GetDependencies "/property:DesignTimeBuild=true;_DependencyFile=$(pwd)/TypeCatalogGen/powershell.inc" /nologo92```93 94`powershell.inc` contains the resolved paths to the DLLs of each dependency of PowerShell,95and is taken as input to the [`TypeCatalogGen`](../../src/TypeCatalogGen) tool,96which generates the source file `CorePsTypeCatalog.cs` for the `System.Management.Automation` project.97 98The error `The name 'InitializeTypeCatalog' does not exist in the current context`99indicates that the `CorePsTypeCatalog.cs` source file does not exist,100so follow the steps to generate it.101 102## Native Components103 104On Windows, PowerShell Core depends on the WinRM plugin `pwrshplugin.dll` to enable remoting over WinRM.105On Linux/macOS, PowerShell Core depends on the binary `libpsl-native.so/libpsl-native.dylib` to provide some necessary supports.106 107Building those native components requires setting up additional dependencies,108which could be a burden to those who don't seek to make changes to the native components.109In the meantime, the native component code seldom changes,110so it doesn't make sense to always build them with `Start-PSBuild`.111Therefore, we decided to wrap the native components into NuGet packages,112so that we only need to build them once when changes are made,113and then reuse the produced binaries for many builds subsequently.114 115The NuGet package for `pwrshplugin.dll` is `psrp.windows`,116and the NuGet package for `libpsl-native` is `libpsl`.117 118### Windows packages: PSRP.Windows and PowerShell.Core.Instrumentation119 120To build `pwrshplugin.dll` and `PowerShell.Core.Instrumentation.dll`, you need to install Visual Studio 2017 and run `Start-PSBootstrap -BuildWindowsNative` to install the prerequisites.121 122Ensure the following individual components are selected:123 124- [ ] VC++ 2017 v141 toolset (x86, x64)125- [ ] Visual C++ compilers and libraries for ARM126- [ ] Visual C++ compilers and libraries for ARM64127- [ ] Visual C++ tools for CMake128- [ ] Visual C++ ATL Support129- [ ] Windows 10 SDK (10.0.16299.0) for Desktop C++ (ARM and ARM64)130- [ ] Windows 10 SDK (10.0.16299.0) for Desktop C++ (x86 and x64)131 132Ensure [CMake](https://cmake.org/download/) 3.10.0 or newer is installed which supports VS2017 and ARM64 generator.133 134Then run `Start-BuildNativeWindowsBinaries` to build the binary.135For example, the following builds the release flavor of the binary targeting arm64 architecture.136 137```powershell138Start-BuildNativeWindowsBinaries -Configuration Release -Arch x64_arm64139```140 141Be sure to build and test for all supported architectures: `x86`, `x64`, `x64_arm`, and `x64_arm64`.142 143The `x64_arm` and `x64_arm64` architectures mean that the host system needs to be x64 to cross-compile to ARM.144When building for multiple architectures, be sure to use the `-clean` switch as cmake will cache the previous run and the wrong compiler will be used to generate the subsequent architectures.145 146After that, the binary `pwrshplugin.dll`, its PDB file, and `powershell.core.instrumentation.dll` will be placed under `src\powershell-win-core`.147 148To create a new NuGet package for `pwrshplugin.dll`, first you need to get the `psrp.windows.nuspec` from an existing `psrp.windows` package.149You can find it at `~/.nuget/packages/psrp.windows` on your windows machine if you have recently built PowerShell on it.150Or you can download the existing package from [powershell-core feed](https://powershell.myget.org/feed/powershell-core/package/nuget/psrp.windows).151Once you get `psrp.windows.nuspec`, copy it to an empty folder and update the `<version>` element.152 153After building successfully, copy the produced files to the same folder,154and create the same layout of files as in the existing package.155The layout of files should look like this:156 157```none158\---runtimes159    +---win-x64160    |   \---native161    |           pwrshplugin.dll162    |           pwrshplugin.pdb163    |164    +---win-x86165    |   \---native166    |           pwrshplugin.dll167    |           pwrshplugin.pdb168    +---win-arm169    |   \---native170    |           pwrshplugin.dll171    |           pwrshplugin.pdb172    \---win-arm64173        \---native174                pwrshplugin.dll175                pwrshplugin.pdb176```177 178Have the DLLs signed with `authenticode dual` certificate and run `nuget pack` from the parent of the `runtimes` folder where `psrp.windows.nuspec` resides.179Be sure to use the latest recommended version of [nuget.exe](https://www.nuget.org/downloads).180 181Publish the latest nupkg to https://powershell.myget.org/feed/powershell-core/package/nuget/psrp.windows.182 183`PowerShell.Core.Instrumentation.dll` NuGet package is created the same way, but in a separate directory following the same layout above.184To create a new NuGet package for `PowerShell.Core.Instrumentation.dll`, you will need the `PowerShell.Core.Instrumentation.nuspec` found in the repo under `src\PowerShell.Core.Instrumentation`.185 186Publish the latest nupkg to https://powershell.myget.org/feed/powershell-core/package/nuget/PowerShell.Core.Instrumentation.187 188### libpsl189 190For `linux-arm`, you need to run `Start-PSBootstrap -BuildLinuxArm` to install additional prerequisites to build `libpsl-native`.191Note that currently you can build `linux-arm` only on a Ubuntu machine.192 193For `linux-x64` and macOS, the initial run of `Start-PSBootstrap` would be enough -- no additional prerequisite required.194 195After making sure the prerequisites are met, run `Start-BuildNativeUnixBinaries` to build the binary:196 197```powershell198## Build targeting linux-x64 or macOS199Start-BuildNativeUnixBinaries200 201## Build targeting linux-arm202Start-BuildNativeUnixBinaries -BuildLinuxArm203```204 205After the build succeeds, the binary `libpsl-native.so` (`libpsl-native.dylib` on macOS) will be placed under `src/powershell-unix`.206 207To create a new NuGet package for `libpsl-native`, first you need to get the `libpsl.nuspec` from an existing `libpsl` package.208You can find it at `~/.nuget/packages/libpsl` on your Linux or macOS machine if you have recently built PowerShell on it.209Or you can download the existing package from [powershell-core feed](https://powershell.myget.org/feed/powershell-core/package/nuget/libpsl).210Once you get `psrp.windows.nuspec`, copy it to an empty folder on your Windows machine.211 212Then you need to build three binaries of `libpsl-native` targeting `linux-x64`, `linux-arm` and `osx` respectively.213**Please note that, in order for the `linux-x64` binary `libpsl-native.so` to be portable to all other Linux distributions,214the `linux-x64` binary needs to be built on CentOS 7**215(.NET Core Linux native binaries are also built on CentOS 7  to ensure that they don't depend on newer `glibc`).216 217After building successfully, copy those three binaries to the same folder,218and create the same layout of files as in the existing package.219The layout of files should look like this:220 221```none222└── runtimes223    ├── linux-arm224    │   └── native225    │       └── libpsl-native.so226    ├── linux-x64227    │   └── native228    │       └── libpsl-native.so229    └── osx230        └── native231            └── libpsl-native.dylib232```233 234Lastly, run `nuget pack .` from within the folder. Note that you may need the latest `nuget.exe`.235