Sunday, April 15, 2018

.NET runtime information in different systems

.NET Core is great multi-platform framework. In my previous job, most of the team members were working on Windows, I worked on a MacBook and we deployed stuff to CentOS and Windows Servers. Build servers had a mix of Windows Server 2012 R2, Windows Server 2016 and a lonely CentOS and it all worked just fine.

The truth is that there are lots of business applications which are built with .NET Framework in a variety of versions with active development. At my new job (it's been 2 weeks), there's a focus in integrating all kinds of technologies and frameworks/versions to make developer's life's easier. If for example you maintain a Windows Form application built 6 years ago, targeting .NET Framework 4.5 which Microsoft has dropped support to 2 years ago, it's possible that a tool like Sentry is even more vital to you. I'd still recommend you upgrade to 4.5.2 though, at least, if possible.

Because of that, it's one of Sentry's .NET SDKs goals to target as many framework versions as possible. It currently even supports .NET 3.5 which was released 10 years ago and cannot be built with the new msbuild SDK.

While working on adding a feature to Sentry's SDK which augments the events sent to Sentry with data about the system where the app is running, I was presented with a different set of APIs available to different versions of the framework. For example, when targeting netstandard2.0, it's possible to retrieve the framework information with:

System.Runtime.InteropServices.RuntimeInformation.FrameworkDescription

This API does not exist in versions prior to 4.5. My first idea was to copy the CoreFX implementation but once I saw the #ifdef's I realized it's not possible as I'm not generating assemblies just to have .NET Native or .NET Framework etc. This was just one example, there are many more APIs I'd like to use that will differ depending on the target framework.

CoreThrower


To help me test different scenarios, I wrote a script (batch and bash versions) to build and run a simple app that throws an error with the Sentry .NET SDK restored from some local path. It has the following targets:

  1. netcoreapp2.0
  2. net471
  3. net46
  4. net452
  5. net45
  6. net40
Considering that all .NET 4.x are in-place installations, all I'm doing here is making sure I can build the app and load the NuGet package against those versions. Compiling uses reference assemblies to ensure compatibility. In runtime, it'll use the v4.0 folder with the CLR 4.0.
The .NET Core 2 version runs with CoreCLR on all platforms. The full framework versions will run with Mono on macOS and Linux and on Windows, the desktop (full) .NET Framework .

This tool is useful to me to get a locally built Sentry SDK and run in different environments.
I ran it on:

  1. macOS 10.13.3 High Sierra
  2. Windows 10 Pro Fall Creator
  3. Ubuntu 16.04.3 LTS (Windows Subsystem for Linux)
  4. Windows 7 32bit
  5. CentOS 7
All of them have .NET Core SDK 2.1.104.

While dealing with the full framework builds and runs I had a few surprises though:

Windows 7 x86


I verified (through the Windows registry) that the Windows 7 machine has version .NET Release number 378389. That means the plain .NET Framework 4.5. Although 4.7.1 is compatible with Windows 7 32 bits, I decided not to update it to verify which versions of the program and the SDK would work properly. I copied the reference assemblies to allow me to build all versions:

Windows 10 (64 bits): C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework
To the Win7 (32 bits): C:\Program Files\Reference Assemblies\Microsoft\Framework

At this point running run.cmd successfully compiled all versions and ran all versions! That was to my surprise since the versions ahead of 4.5 could very well fail. It does make sense though since the application itself is not using any .NET 4.6+ API and Sentry's SDK is targeting net45 as the latest besides netstandard2.0. 

That also means that the netstandard2.0 version was not restored to any of the net4x versions. Since Considering I'm using the .NET Core 2.0 SDK I would expect the targets net46 and net471 to restore the netstandard2.0 version of the Sentry SDK. Another possibility is that I have no idea why it worked ;)

At sentry.io I see:

That surely needs some work: The events sent by the .NET Core version of the app shows the CoreCLR version (4.6.26212.01) followed by the .NET Framework 4.6+ version (4.0.30319.42000) which is not even installed on the machine. The middle entry shows 4.0.30319.18063. At least the value looks correct. It represents .NET 4.5 on Windows 7 SP1.

The values presented have two parts where the first is runtime name and the second is the version which was coming from the API:

System.Environment.Version

Microsoft does have a disclaimer on this API:

For the .NET Framework 4.5 and later, we do not recommend using the Version property to detect the version of the runtime; instead, you can determine the version of the common language runtime by querying the registry. For more information, see How to: Determine Which .NET Framework Versions Are Installed.
From MSDN

The version name and number get combined and depending on the version of the framework the code compiled against it shows the latest .NET Framework installed or simply the CLR version.

Needs work but for now lets see what shows up on the other systems:

Ubuntu 16.04 (WSL)


The WSL installation of Ubuntu I have, included OOTB Mono version 4.6.2:


$ mono --version

Mono JIT compiler version 4.6.2 (Debian 4.6.2.7+dfsg-1)

Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com

       TLS:       __thread

       SIGSEGV:       altstack
       Notifications: epoll
       Architecture:  amd64
       Disabled:      none
       Misc:       softdebug
       LLVM:       supported, not enabled.
       GC:       sgen

This version does not implement netstandard2.0. That was only introduced in version 5.4 back in November 2017. I took the same approach I had with the Win7 box: Try to build and run with an older version.

... /usr/share/dotnet/sdk/2.1.101/Microsoft.Common.CurrentVersion.targets(2052,5): warning MSB3245: Could not resolve this reference. Could not locate the assembly "System.Net.Http". Check to make sure the assembly exists on disk. If this reference is required by your code, you may get compilation errors. [/mnt/c/Users/bruno/git/CoreThrower/CoreThrower.csproj] CSC : error CS0006: Metadata file '/mscorlib.dll' could not be found [/mnt/c/Users/bruno/git/CoreThrower/CoreThrower.csproj] CSC : error CS0006: Metadata file '/mscorlib.dll' could not be found [/mnt/c/Users/bruno/git/CoreThrower/CoreThrower.csproj] CSC : error CS0006: Metadata file '/mscorlib.dll' could not be found [/mnt/c/Users/bruno/git/CoreThrower/CoreThrower.csproj] CSC : error CS0006: Metadata file '/mscorlib.dll' could not be found [/mnt/c/Users/bruno/git/CoreThrower/CoreThrower.csproj] CSC : error CS0006: Metadata file '/mscorlib.dll' could not be found [/mnt/c/Users/bruno/git/CoreThrower/CoreThrower.csproj] 63 Warning(s) 5 Error(s)

With the exception of the netcoreapp2.0 target, nothing would build. Using FrameworkPathOverride I configured the projects' csproj to look into mono's path for the reference assemblies but realized that there was actually nothing there. Considering Mono dropped support to pre-net45 reference assemblies on Mono 4 I had to install anyway the package for net40:

apt-get install mono-reference-assemblies-4.0

After that, I was still missing the newer packages. Makes sense since I was on an older version of Mono. This can be solved by taking them directly from GitHub:


cd /usr/lib/mono


sudo cp -r reference-assemblies/v* .

sudo rm -rf reference-assemblies/
The directories still needs renaming, to the expected format: remove v prefix and add -api suffix:

sudo sh -c 'for dir in v*; do mv "$dir" "${dir#v}-api"; done'

Build successful! That doesn't mean it will run and indeed it didn't.
First error:

Running: bin/Release/net46/CoreThrower.exe

Unhandled Exception:
System.IO.FileNotFoundException: Could not load file or assembly 'System.Core, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089' or one of its dependencies.
File name: 'System.Core, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089'
  at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine] (System.Runtime.CompilerServices.TStateMachine& stateMachine) <0x7ffc1650 + 0x0010f> in <filename unknown>:0
  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[TResult].Start[TStateMachine] (System.Runtime.CompilerServices.TStateMachine& stateMachine) <0x7ffc1610 + 0x00019> in <filename unknown>:0
  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[TStateMachine] (System.Runtime.CompilerServices.TStateMachine& stateMachine) <0x7ffc15e0 + 0x00019> in <filename unknown>:0
  at CoreThrower.Program.Main (System.String[] args) <0x7ffc1080 + 0x001db> in <filename unknown>:0
  at CoreThrower.Program.<Main> (System.String[] args) <0x7ffc0f10 + 0x0001b> in <filename unknown>:0
[ERROR] FATAL UNHANDLED EXCEPTION: System.IO.FileNotFoundException: Could not load file or assembly 'System.Core, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089' or one of its dependencies.
File name: 'System.Core, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089'
  at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine] (System.Runtime.CompilerServices.TStateMachine& stateMachine) <0x7ffc1650 + 0x0010f> in <filename unknown>:0
  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[TResult].Start[TStateMachine] (System.Runtime.CompilerServices.TStateMachine& stateMachine) <0x7ffc1610 + 0x00019> in <filename unknown>:0
  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[TStateMachine] (System.Runtime.CompilerServices.TStateMachine& stateMachine) <0x7ffc15e0 + 0x00019> in <filename unknown>:0
  at CoreThrower.Program.Main (System.String[] args) <0x7ffc1080 + 0x001db> in <filename unknown>:0
  at CoreThrower.Program.<Main> (System.String[] args) <0x7ffc0f10 + 0x0001b> in <filename unknown>:0

The first thread I found on the topic pointed out that I need to have libmoono-system-core4.0-cil installed. 30 seconds later I was running it again:


Running: bin/Release/net46/CoreTh

rower.exe


Unhandled Exception:

System.IO.FileNotFoundException: Could not load file or assembly 'System.Core, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089' or one of its dependencies.

File name: 'System.Core, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089'
 at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine] (System.Runtime.CompilerServices.TStateMachine& stateMachine) <0x7ffc1650 + 0x0010f> in <filename unknown>:0
 at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[TResult].Start[TStateMachine] (System.Runtime.CompilerServices.TStateMachine& stateMachine) <0x7ffc1610 + 0x00019> in <filename unknown>:0
 at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[TStateMachine] (System.Runtime.CompilerServices.TStateMachine& stateMachine) <0x7ffc15e0 + 0x00019> in <filename unknown>:0
 at CoreThrower.Program.Main (System.String[] args) <0x7ffc1080 + 0x001db> in <filename unknown>:0
 at CoreThrower.Program.<Main> (System.String[] args) <0x7ffc0f10 + 0x0001b> in <filename unknown>:0
[ERROR] FATAL UNHANDLED EXCEPTION: System.IO.FileNotFoundException: Could not load file or assembly 'System.Core, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089' or one of its dependencies.
File name: 'System.Core, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089'
 at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine] (System.Runtime.CompilerServices.TStateMachine& stateMachine) <0x7ffc1650 + 0x0010f> in <filename unknown>:0
 at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[TResult].Start[TStateMachine] (System.Runtime.CompilerServices.TStateMachine& stateMachine) <0x7ffc1610 + 0x00019> in <filename unknown>:0
 at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[TStateMachine] (System.Runtime.CompilerServices.TStateMachine& stateMachine) <0x7ffc15e0 + 0x00019> in <filename unknown>:0
 at CoreThrower.Program.Main (System.String[] args) <0x7ffc1080 + 0x001db> in <filename unknown>:0
 at CoreThrower.Program.<Main> (System.String[] args) <0x7ffc0f10 + 0x0001b> in <filename unknown>:0

As pointed out further down in the same thread I linked above, mono-complete does the job:

sudo apt-get install mono-complete
Setting up mono-complete (4.2.1.102+dfsg2-7ubuntu4) ...

Wait, what? Mono 4.2? I didn't change any of Ubuntu's package feed I don't think, did it just downgrade from 4.6.2 to 4.2.1? 

Mono JIT compiler version 4.2.1 (Debian 4.2.1.102+dfsg2-7ubuntu4)
Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
        TLS:           __thread
        SIGSEGV:       altstack
        Notifications: epoll
        Architecture:  amd64
        Disabled:      none
        Misc:          softdebug
        LLVM:          supported, not enabled.
        GC:            sgen

It looks like it did and if I now: ./run.sh
Finally, it works! 

At sentry.io I see:


Although I've installed the same SDK version on both Win7 and Ubuntu, the latter is returning only 4.6.0.0 compared to 4.6.26212.01 returned on Windows.

Windows 10 Pro



Straight forward. I have .NET Framework 4.7.1 and Visual Studio on this box so it built and ran without issues.


The first item is created by the net40 version of the app. Since it doesn't have the same APIs as net45+. It's using Environment.Version mentioned above which is far from ideal.
The rest of the results are OK with the exception of the "or higher" which came as a result of the version being higher than 461308. I'll need to rework this too.

CentOS 7


Using the official .NET Core and Mono install links I got their SDKs installed really quickly.
Latest Mono version: 5.10.1.20 and .NET Core SDK 2.1.101

Pull the code, pushed the nuget package to the checkout directory and ran ./run.sh
Works fine and I get some events at sentry.io.


Like the Ubuntu version, .NET Core only include the major and minor version numbers of the CoreCLR.

macOS


The macOS device is my work computer so I might have installed reference assemblies before, I don't remember. Things ran just fine and the Mono version there is 5.8:

It seems that only on Windows the API RuntimeInformation.FrameworkDescription returns the whole version number including build and revision.

Other system information


Above I've discussed only the 'Runtime' context information that Sentry handles. There's much more data that Sentry's SDK will extract and send with events. Lots of them are challenging when multi-targeting from netstandard2.0 all the way down to net35 and running on Windows, macOS, Linux, Tyzen, iOS, WebAssembly (blazor?) and whatever else we can use a .NET NuGet package in now and in the future.

Conclusion


I still have a lot of work ahead to get this context data sent to sentry in a unambiguous way. This was just the first try though.

Sunday, March 4, 2018

Getting started with SonarQube and .NET Core

SonarQube 


I decided to check SonarQube now that it finally supports .NET Core and the new MSBuild SDK format.

SonarQube is an open source product from SonarSource. They have a hosted instance called SonarCloud which is free for open source projects! Besides SonarQube, there's also an IDE plugin called SonarLint.

SonarLint

I started testing SonarLint out to see if it was even worth setting up SonarCloud. As a test project I picked a tiny OSS library I wrote a few months ago for ASP.NET Core Basic Authentication

SonarLint will run the analysis while using the IDE's Error/Warning panels to notify you when it finds something. Although they support IntelliJ IDEA, the support doesn't include Rider. The support to VSCode doesn't include C# either. Bummer. But honestly there's still no code coverage for CoreCLR yet outside of Windows either.

I tested the Visual Studio plugin and ironically, I was happy to see a warning:


Indeed. I should make that class static.

I happen to have ReSharper which is a great (paid) tool by JetBrains. I ran its Inspection/Code issues in Solution and it didn't pick this one up. JetBrains Rider which I use on my work computer (macOS) has the same inspection feature. They do give awesome feedback even though there's no warning for this specific issue.
Please note that both are amazing tools. In any case, it's hard to expect that all contributors of an OSS project will have licenses. JetBrains provide licenses to open source project maintainers but not for such small projects like the ones I have.

Running the analysis


Now that I had a project with a warning, I could test running the analysis scanner on my machine and see what the results look like on the hosted instance SonarCloud. The analysis scanner (although requires Mono, .NET Core and Javacan be executed on macOS and Linux but as today I'm on a Windows 10 box, I followed this guide. The only thing I did differently was that I didn't invoke MSBuild directly, I used the CLI: dotnet build instead. It's really just 3 commands including your build command.

I was sad to discover that it requires Java in order to run the MSBuild scanner. I've got a clean Windows 10 install a few months ago and I was very happy not to have Java installed. SonarQube is nice enough to convince me to install it though. Can't wait to see those Java update notifications every week!

The analysis took quite a while to run (over 5 minutes). Perhaps because it was the first run, but the report was sent!

The link to the results is part of the output:


Now I can browse the results at: https://sonarcloud.io/dashboard?id=Bazinga.AspNetCore.Authentication.Basic



It was a good quick getting started for me. I'll consider it now for bigger projects too and explore the many features it has.

Monday, February 12, 2018

CentOS build agent and .NET Core SDK versions

Our build server running on CentOS ends up building different repos, which use different versions of the .NET Core SDK.

The SDK version is defined via global.json and instructs the CLI which version to load in that context.

Surely the CLI is smart enough to roll forward. That means in case you target version 2.0.2 and the machine only has 2.0.3, it will (should) work just fine with that version.

I've tested setting my global.json to 2.0.2 where only SDK version 2.0.0 was available:
$ dotnet --version
2.0.0

Surprising to me it built and tests passed without warnings. So it also rolls backwards somehow.

When the exact SDK version defined via global.json is available, that's the one used by the CLI.
I believe that's a safer approach to have on a build server. Which means having all SDK versions in all agents.

That is, regardless which agent the build runs, it'll always use the same SDK version. Otherwise potentially one agent, created at a later time, could lack the exact SDK version used via global.json.

Installing all SDK versions


Using yum's wildcard, it's easy enough to install every single .NET Core SDK available on the official Microsoft feed, simply by:

$ sudo yum install dotnet-sdk-*

As an example, this agent had only the SDK version 2.1.4 (the latest one).

By running yum install dotnet-sdk-* on this machine:


Easy enough.

All SDK versions made available by Microsoft are now installed.



Sunday, February 4, 2018

Hosting a static site over HTTPS for free

I own the domain brunogarcia.com for over 10 years now. The use I have of it is basically an e-mail address and this blog which for years I didn't post new content to.

Yesterday I decided it was time to add a landing page to it.

With the amazing artistic skills I possess, I sat in front of the PC and spent more time than I'm willing to admit thinking of what to do and messing around with HTML and CSS. After a lot of back and forth I deleted any framework from it and the result was 1 image, a favicon and 5 links (sigh).

I have no intention of making profit from my art so I've published my favicon via Creative Commons.
Feel free to use, redistribute and make lots of money from it.

At that point I had a couple files to publish.
As a .NET guy, I started off considering Microsoft's Azure. I've hosted stuff there for free before. That was when I used it for the first time back in 2011 or 2012 when they had a Silverlight portal. There were not many features (compared to now) but it was straight forward to host a .NET app.
Azure grew to an amazing cloud service with so many options and amazing (and slow) portal to configure everything.

This time though, all I had was just a handful static files. I didn't need an app server!

I couldn't find a way to host at either Azure nor Google Cloud, for free, a static site under my own domain so I could use Let's Encrypt and have a free certificate. There were options only using CNAME but that was not an option since the TLS certificate would invalidate.

After a bit of a research I discovered Firebase. Super simple setup. The portal just walks you through. Free hosting of static files and a nice nodejs deployment tool that you can pull off of npm. It allows you add your own domain with TLS. Easy redirect from HTTP to HTTPS. It just took 5 minutes!

After that I decided to check out Cloudflare. Again super easy to setup. Fast and free and again with a valid certificate and all traffic through HTTPS. With a single rule I was able to enable caching of the HTML file too. To hit the back end directly, bypassing Cloudflare, there's a single button: Enable development mode. That changes the DNS records from Cloudflare servers directly to Firebase. It's virtually instant too.

Another cool thing about Cloudflare is the Overview page. I don't have 24 hours of data yet so part of the analytics is empty but I have some DNS statistics. Including Top Traffic Origins:




The only problem I had was that by default I had an endless redirect! It was quick to find a solution though which was setting via the Crypto menu the Full mode instead of the default Flexible. That means that Cloudflare will contact the backend (Firebase) via HTTPS only instead of hitting the plain HTTP, which returns a redirect to HTTPS and hence the endless redirect.

Finally, I've pushed it to GitHub, which serves as my personal backup.

It's worth noting that at this point another option was to use GitHub pages instead of Cloudflare. Especially since I'm pushing the files there anyway. One limitation GitHub pages have is that you can't use your custom domain with HTTPS. But that's done by Cloudflare unless I enable development mode.

It's nice to have https://brunogarcia.com returning a couple of tiny static files without having to pay for it.

Big Thanks to Firebase, Cloudflare and also Optimizilla which I used to compress the image, Fontastic for creating icons bundle and favicon.cc to allow me showing off my artistic gift.

Looking forward to dive into the paid features they offer with a real application!

Monday, December 4, 2017

ISerializer with Span<T>

While trying out the new Span<T> type from System.Memory package I encountered an error:

   typeof(System.InvalidProgramException): Common Language Runtime detected an invalid program.

At first I thought that must be a bug since System.Memory package is still a preview release but it turns out that it's by design.

I was experimenting with using ReadOnlySpan<T> instead of byte[] on Greentube.Serialization.

The contract became:

Once I fixed the compiler related problems, I ran all tests but 6 of them failed:



After learning a bit more about Span<T>, mostly thankfully to great documentation from Adam Sitnik I was able to understand what was going on.

Firstly the Assert.Throws<T> case:

The problem here is that the compiler will resolve () => sut.Serialize(object)) as Func<ReadOnlySpan<byte>>. That's unfortunately illegal since Span<T> (and ReadOnlySpan<T>) are stack-only types. It can't end up on the heap so it's not allowed to be a generic type.
I'm not going to try to explain better than Adam Sitnik so please refer to his doc for more.

The fix:


The work around here is simple enough. I've just added: { }

What the compiler resolves it as now is a simple Action since I'm ignoring the return of the Serialize method which is of type ReadOnlySpan<T>. Now the test passes again.

I've also changed the implementation from a null check to a default value check since Span is a value type. Now the exception thrown is ArgumentException instead.

The other problem was related to NSubstitute. Having a subtitute of ISerializer is a problem since internally NSubstitute will keep a reference to it which would require boxing the argument of type ReadOnlySpan<T> and that results on the same 'Common Language Runtime detected an invalid program.' error.

To solve that, I used a Stub instead of an NSubstitute mock.

Saturday, December 2, 2017

CI/CD for .NET with GitHub/Travis-CI/AppVeyor/Codecov/NuGet

I recently setup continuous integration for a few open source projects. I used GitHub, AppVeyor, Travis-CI and Codecov. The resulting packages are deployed to NuGet.org. 

Here I aim to describe the process in an attempt to document it both for myself and hopefully help others in getting started with it.


Scenario

I have some .NET Core projects on GitHub that are actually just .NET Standard libraries.

Ultimately, once they are built and tested on Windows, MacOS and Linux, I want them published to NuGet.org.
I also want to use GitHub's Release feature to document what I am pushing to NuGet.

Some of these projects actually generate more than one NuGet package. That's the case for Greentube.MessagingGreentube.Serialization and Greentube.Monitoring. Depending on the change I make, I might be looking at publishing any number of those packages. I don't want to publish all packages created while building the repository every time.

TL;DR 

Travis-CI is used only to build and run tests on MacOS and Linux.
AppVeyor, which runs on Windows, builds, run tests while tracking code coverage and sends the results to codecov.io. The result of AppVeyor's build are artifacts, .nupkg files (aka: NuGet packages) ready to be sent to Nuget.org.

If AppVeyor is building a tag, it changes the build version to be the value of the tag. That sets the actual version of the NuGet packages created. Also, AppVeyor then creates a GitHub release in draft mode.

At that point from AppVeyor I can publish individual packages to NuGet with a button click.
The release notes can be added to the GitHub Release and then it can be published.

How?

Nearly all settings to AppVeyor, Travis-CI and Codecov are defined on their respective configuration files. Although it's possible to configure all of those via their Web UI, I personally rather use the text configuration file which is version controlled.

Travis-CI

Previously I mentioned there's no code coverage coming from the Travis-CI build. That’s because as of today there's no way to do it. OpenCover and vstest don't support Linux or MacOS because there's still no released profiling API for those platforms. That seems to be going to change soon though.
Nonetheless, Travis-CI is a very nice, free for open source, CI system. Changes pushed to GitHub automatically trigger a build in both Linux and MacOS:



dotnet core build on Linux and MacOS

Using the Greentube.Messaging Travis-CI configuration file as an example: .travis.yml



Firstly note that mono is set to none since I'm building with dotnet here. Since tests run CoreCLR, only .NET Core SDK is required. I'm bulding the specific projects although from the SDK version 2.0.0 forward or so it's possible to build a solution file. That is because all I want here is to run my tests. Running dotnet test will build the dependent projects which in turn will run a dotnet restore if required. Again, this was not possible with .NET Core SDK 1.x.

The ulimit -n 512 is there to solve a problem when restoring NuGet packages which open too many connections. Also while trying to solve a problem, I've added osx_image: xcode8.1 to be able to build with .NET Core SDK 2.0.

There're tons of other examples on how to get Travis-CI to build a .NET Core project.

AppVeyor

Here there's more going on. Besides building all commits, including pull requests, AppVeyor will take code coverage and send it to codecov, create GitHub Releases and publish packages to NuGet.

I'm going to start by describing things before spitting the configuration file in front of you:

Code coverage

I've tried to summarize the code coverage setup on StackOverflow before but it was never marked as an answer so here goes another try. :)

Configuration

All project files have an extra configuration called Coverage with: DebugType=full

The reason for the full PDB configuration is that by default it compiles with portable pdbs and those are still not supported.

Run

Code coverage is tracked with OpenCover. I wrote a simple powershell script that will download the necessary tools and, using the .NET Core CLI, it runs all tests while tracking code coverage.

It has two optional arguments:

  1. generateReport
    • Generates an HTML report useful for testing things before sending to Codecov
  2. uploadCodecov

Previously I mentioned that nearly all settings are source controlled. That is not true to the API KEY used to send code coverage results to Codecov. That value is defined as an environment variable (named CODECOV_TOKEN) on AppVeyor. 

Packaging

The projects being built don't have their version on csproj defined before committing to GitHub.
They are always set to 0.0.0 and that gets replaced (aka patched) by AppVeyor before building and packaging it.

As mentioned before, the result of AppVeyor builds, as other CI systems, are called artifacts. What is included in the artifacts list is specified via the configuration key with that same name. A glob pattern to include app nupkg is enough then.: '**\*.nupkg'

Packages are created with the dotnet pack command. That is true to all but 1 package: metapackage

Metapackage

One thing I've left out of the summary is the creation of a metapackage. That's an attempt to ease into the API adoption by providing the simplest way to get started.

Note that obviously the packages referenced by this metapackage are not included in the runtime package store. That means that using this will in fact include the dependent packages in your publish output.

I'm publishing two metapackages, each from their own repository:
Someone wants to try the API?  Run a single dotnet add package command and all of the building blocks are available to them. 

I don't expect it to be used in production though. I advise the fine grained approach instead (pay-for-play).

Create

Runing dotnet pack on a project that has no code yields no package! 
The CLI will not build up an empty package. To create a metapackage you need to use a nuspec file.

To pack this one up, there's a line in the configuration invoking nuget pack instead.

Creating a Release

I've added to .appveyor.yml (see below) a piece of powershell which resets the version of the build to the tag if that's what's being built.

Pushing a tag is the method to express the intent of a creating release.

I'm using semver and NuGet already takes packages versioned N.N.N-something as a pre-release package.

That means a tag 1.0.0-beta will publish a pre-release package. On the other hand, if the tag is simply 1.0.0, that'd be a plain release.

GitHub on the other hand doesn't handle that automatically. AppVeyor creates the draft release on GitHub but at the bottom of the page before publishing, I still need to select that option:


AppVeyor will push every artifact of that build to GitHub's drafted release. Considering that I do not always publish all packages to NuGet, I remove whatever package I didn't publish to NuGet before finalizing the GitHub release.

In other words:
  1. Push a tag to GitHub, 
  2. Publish desired packages to NuGet through AppVeyor UI (more below).
  3. Remove from the GitHub release the packages not deployed on the next step
  4. Publish release on GitHub (green button above)

NuGet

Publishing to NuGet is not done automatically as a result of the release creation mentioned above. The reason is that each build results in multiple packages as artifacts and publishing is a all or nothing approach.


I'm looking for publishing packages in a selective way. AppVeyor has environments which can be configured to deploy specific artifacts from specific projects. Those can also be defined via the configuration file but you'd still need to define which 'environment' (aka: package) you want to deploy anyway. At that point I've chosen to use AppVeyor's UI to set-up these environments.

The way I've done it is: one AppVeyor's environment for each package:


To get something deployed I can click a single button which pushes that single package to NuGet.org.

Differentiating each of those is done via regular expressions. For example the RegEx for the first item on that image is:

/Greentube\.Messaging\.\d.*\.nupkg/


I'm still trying to push the markdown docs automatically together to NuGet. I've asked about it on StackOverflow a few days ago but still not answer.

One downside of this approach is that the dependencies between the packages have to be tracked by yourself.

For example:


If I want to publish Greentube.Messaging.DependencyInjection.Redis, I need to take into consideration that it depends on Greentube.Messaging.DependencyInjection which in turn depends on Greentube.Messaging of the same version.

For that reason, the 3 packages have to deployed otherwise restoring the first one will fail.


Configuration

Again using the Greentube.Messaging as an example, here's the .appveyor.yml


Note about the pace of changes

Everything about .NET Core changes rapidly. The tooling around it specially. A lot of configuration I've come up with was added as a reaction to some issue I encountered. I'm sure many of these issues will be solved soon so configuration could be simplified. An example is the full pdbs for coverage and the profiling API on MacOS. Take that into account when getting your own setup ready. 

Sunday, April 21, 2013

Wake on Lan in C# and Windows 8

About 8 years ago I was writing scripts to run on a network with over 130.000 computers (of which 5000 I administered).
The scripts ran 24/7, parsing computer's inventory log files, which they sent to a central server. It was  possible to detect and fix a whole bunch of issues, most of the time even before a user would notice something was wrong.

Note that most of those computers were running Windows NT 4, including the domain controllers. The task to install application in all those computers and keep their anti-virus signature up-to-date was not as trivial as it is today. There were times we needed to perform tasks on computers that weren't even switched on. And I must admit, back then I was quite proud of the solution I came up with for this particular case. Although it's not my goal to go into details on how I managed to get any of those 5000 computers, spread in 130 different offices, powered-on at any time; I want to write a little about the core of the solution: Wake on Lan

Whether to be able to power on your computer at home when you are away, or to manage a corporate network, the ability to switch a computer on out of sending a magic packet is at least quite interesting. It does require basic knowledge of computer networks and hardware to understand it, but to make it work, all you need is to know how to find your computer's MAC address. That's all you need to use WoL on a local network. However if you are willing to use it over the Internet, you'll need the IP address of the gateway of which the target computer belongs to, and have that gateway configured to forward the magic packet to the private network. The Port number is used in this case (if you use NAT, you'll need it), so that you can forward incoming UDP datagram on that predefined port of your router to an internal network IP address, or even the broadcast address if you'd like.

If the computer is switched off, it's likely the network switch's CAM table won't have an entry to your target computer's MAC address, and once your router forwards the datagram received from the external interface to your internal network, to the specific IP address you defined on the forwarding rule, the network switch will broadcast that to all ports.

You'll find many over architectured implementations of Wake on Lan out there, even in C#. But the fact is that it's really simple thing.

Let's see some C# code:


using System;
using System.Net;
using System.Net.NetworkInformation;
using System.Net.Sockets;
using System.Text.RegularExpressions;

namespace BrunoGarcia.Net
{
    public sealed class WolManager
    {
        const int _payloadSize = 102;

        public static void SendMagicPacket(string mac, IPEndPoint ipEndPoint)
        {
            var macBytes = PhysicalAddress.Parse(mac).GetAddressBytes();

            var payload = new byte[_payloadSize];
            Buffer.BlockCopy(new byte[] { 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF }, 0, payload, 0, 6);
            
            for (int i = 1; i < 17; i++)
                Buffer.BlockCopy(macBytes, 0, payload, 6 * i, 6);

            using (var udp = new UdpClient())
                udp.Send(payload, _payloadSize, ipEndPoint);
        }
    }
}

As you can see, it's a fire-and-forget operation, a single UDP datagram. The MAC address of the target machine goes in the payload. As I mentioned before, the port is not really important if you are sending the packet on the same subnet of your target computer (without the need of routing).  Port on wikipedia you find to be 7 or 9. However 0 can be used:


WolManager.SendMagicPacket("B8-AC-6F-59-56-55", 
    new IPEndPoint(IPAddress.Parse("255.255.255.255"), 0));

This is fun, and I've used it to power on my computer at home from the Internet a few times. But now, when I got Windows 8, the fun wasn't quite working.

This KB from Microsoft describes the change:

Windows 7:  In Windows 7, the default shutdown operation puts the system into classic shutdown (S5) and all devices are put into the lowest power state D3. Wake-On-LAN is not officially supported from S5 in Windows 7. However, some network adapters can be left armed for wake if enough residual power is available. As a result, wake from the S5 state is possible on some systems where enough residual power was supplied to the network adapter even though the system is in S5 and devices are in D3. 

Windows 8:  In Windows 8, the default shutdown behavior puts the system into hybrid shutdown (S4) and all devices are put into D3. Remote Wake-On-LAN from hybrid shutdown (S4) or classic shutdown (S5) is unsupported. Network adapters are explicitly not armed for Wake-On-LAN in both the classic shutdown (S5) and hybrid shutdown (S4) cases because users expect zero power consumption and battery drain in the shutdown state. This behavior removes the possibility of spurious wakes when explicit shutdown was requested. As a result, Wake-On-LAN is only supported from sleep (S3) or hibernate (S4) in Windows 8.

What a bummer. Anyway, not time to give up, right? Digging a bit deeper one understands better the changes, and enabling wake on lan on windows 8 is possible again:

Check out this post from Phil Pendlebury which talks about it.