Monday, December 4, 2017

ISerializer with Span<T>

While trying out the new Span<T> type from System.Memory package I encountered an error:

   typeof(System.InvalidProgramException): Common Language Runtime detected an invalid program.

At first I thought that must be a bug since System.Memory package is still a preview release but it turns out that it's by design.

I was experimenting with using ReadOnlySpan<T> instead of byte[] on Greentube.Serialization.

The contract became:

Once I fixed the compiler related problems, I ran all tests but 6 of them failed:



After learning a bit more about Span<T>, mostly thankfully to great documentation from Adam Sitnik I was able to understand what was going on.

Firstly the Assert.Throws<T> case:

The problem here is that the compiler will resolve () => sut.Serialize(object)) as Func<ReadOnlySpan<byte>>. That's unfortunately illegal since Span<T> (and ReadOnlySpan<T>) are stack-only types. It can't end up on the heap so it's not allowed to be a generic type.
I'm not going to try to explain better than Adam Sitnik so please refer to his doc for more.

The fix:


The work around here is simple enough. I've just added: { }

What the compiler resolves it as now is a simple Action since I'm ignoring the return of the Serialize method which is of type ReadOnlySpan<T>. Now the test passes again.

I've also changed the implementation from a null check to a default value check since Span is a value type. Now the exception thrown is ArgumentException instead.

The other problem was related to NSubstitute. Having a subtitute of ISerializer is a problem since internally NSubstitute will keep a reference to it which would require boxing the argument of type ReadOnlySpan<T> and that results on the same 'Common Language Runtime detected an invalid program.' error.

To solve that, I used a Stub instead of an NSubstitute mock.

Saturday, December 2, 2017

CI/CD for .NET with GitHub/Travis-CI/AppVeyor/Codecov/NuGet

I recently setup continuous integration for a few open source projects. I used GitHub, AppVeyor, Travis-CI and Codecov. The resulting packages are deployed to NuGet.org. 

Here I aim to describe the process in an attempt to document it both for myself and hopefully help others in getting started with it.

Scenario

I have some .NET Core projects on GitHub that are actually just .NET Standard libraries.

Ultimately, once they are built and tested on Windows, MacOS and Linux, I want them published to NuGet.org.
I also want to use GitHub's Release feature to document what I am pushing to NuGet.

Some of these projects actually generate more than one NuGet package. That's the case for Greentube.MessagingGreentube.Serialization and Greentube.Monitoring. Depending on the change I make, I might be looking at publishing any number of those packages. I don't want to publish all packages created while building the repository every time.

TL;DR 

Travis-CI is used only to build and run tests on MacOS and Linux.
AppVeyor, which runs on Windows, builds, run tests while tracking code coverage and sends the results to codecov.io. The result of AppVeyor's build are artifacts, .nupkg files (aka: NuGet packages) ready to be sent to Nuget.org.

If AppVeyor is building a tag, it changes the build version to be the value of the tag. That sets the actual version of the NuGet packages created. Also, AppVeyor then creates a GitHub release in draft mode.

At that point from AppVeyor I can publish individual packages to NuGet with a button click.
The release notes can be added to the GitHub Release and then it can be published.

How?

Nearly all settings to AppVeyor, Travis-CI and Codecov are defined on their respective configuration files. Although it's possible to configure all of those via their Web UI, I personally rather use the text configuration file which is version controlled.

Travis-CI

Previously I mentioned there's no code coverage coming from the Travis-CI build. That’s because as of today there's no way to do it. OpenCover and vstest don't support Linux or MacOS because there's still no released profiling API for those platforms. That seems to be going to change soon though.
Nonetheless, Travis-CI is a very nice, free for open source, CI system. Changes pushed to GitHub automatically trigger a build in both Linux and MacOS:



dotnet core build on Linux and MacOS

Using the Greentube.Messaging Travis-CI configuration file as an example: .travis.yml



Firstly note that mono is set to none since I'm building with dotnet here. Since tests run CoreCLR, only .NET Core SDK is required. I'm bulding the specific projects although from the SDK version 2.0.0 forward or so it's possible to build a solution file. That is because all I want here is to run my tests. Running dotnet test will build the dependent projects which in turn will run a dotnet restore if required. Again, this was not possible with .NET Core SDK 1.x.

The ulimit -n 512 is there to solve a problem when restoring NuGet packages which open too many connections. Also while trying to solve a problem, I've added osx_image: xcode8.1 to be able to build with .NET Core SDK 2.0.

There're tons of other examples on how to get Travis-CI to build a .NET Core project.

AppVeyor

Here there's more going on. Besides building all commits, including pull requests, AppVeyor will take code coverage and send it to codecov, create GitHub Releases and publish packages to NuGet.

I'm going to start by describing things before spitting the configuration file in front of you:

Code coverage

I've tried to summarize the code coverage setup on StackOverflow before but it was never marked as an answer so here goes another try. :)

Configuration

All project files have an extra configuration called Coverage with: DebugType=full

The reason for the full PDB configuration is that by default it compiles with portable pdbs and those are still not supported.

Run

Code coverage is tracked with OpenCover. I wrote a simple powershell script that will download the necessary tools and, using the .NET Core CLI, it runs all tests while tracking code coverage.

It has two optional arguments:

  1. generateReport
    • Generates an HTML report useful for testing things before sending to Codecov
  2. uploadCodecov

Previously I mentioned that nearly all settings are source controlled. That is not true to the API KEY used to send code coverage results to Codecov. That value is defined as an environment variable (named CODECOV_TOKEN) on AppVeyor. 

Packaging

The projects being built don't have their version on csproj defined before committing to GitHub.
They are always set to 0.0.0 and that gets replaced (aka patched) by AppVeyor before building and packaging it.

As mentioned before, the result of AppVeyor builds, as other CI systems, are called artifacts. What is included in the artifacts list is specified via the configuration key with that same name. A glob pattern to include app nupkg is enough then.: '**\*.nupkg'

Packages are created with the dotnet pack command. That is true to all but 1 package: metapackage

Metapackage

One thing I've left out of the summary is the creation of a metapackage. That's an attempt to ease into the API adoption by providing the simplest way to get started.

I'm publishing two metapackages, each from their own repository:
Someone wants to try the API?  Run a single dotnet add package command and all of the building blocks are available to them. 

I don't expect it to be used in production though. I advise the fine grained approach instead (pay-for-play).

Create

Runing dotnet pack on a project that has no code yields no package! 
The CLI will not build up an empty package. To create a metapackage you need to use a nuspec file.

To pack this one up, there's a line in the configuration invoking nuget pack instead.

Creating a Release

I've added to .appveyor.yml (see below) a piece of powershell which resets the version of the build to the tag if that's what's being built.

Pushing a tag is the method to express the intent of a creating release.

I'm using semver and NuGet already takes packages versioned N.N.N-something as a pre-release package.

That means a tag 1.0.0-beta will publish a pre-release package. On the other hand, if the tag is simply 1.0.0, that'd be a plain release.

GitHub on the other hand doesn't handle that automatically. AppVeyor creates the draft release on GitHub but at the bottom of the page before publishing, I still need to select that option:


AppVeyor will push every artifact of that build to GitHub's drafted release. Considering that I do not always publish all packages to NuGet, I remove whatever package I didn't publish to NuGet before finalizing the GitHub release.

In other words:
  1. Push a tag to GitHub, 
  2. Publish desired packages to NuGet through AppVeyor UI (more below).
  3. Remove from the GitHub release the packages not deployed on the next step
  4. Publish release on GitHub (green button above)

NuGet

Publishing to NuGet is not done automatically as a result of the release creation mentioned above. The reason is that each build results in multiple packages as artifacts and publishing is a all or nothing approach.


I'm looking for publishing packages in a selective way. AppVeyor has environments which can be configured to deploy specific artifacts from specific projects. Those can also be defined via the configuration file but you'd still need to define which 'environment' (aka: package) you want to deploy anyway. At that point I've chosen to use AppVeyor's UI to set-up these environments.

The way I've done it is: one AppVeyor's environment for each package:


To get something deployed I can click a single button which pushes that single package to NuGet.org.

Differentiating each of those is done via regular expressions. For example the RegEx for the first item on that image is:

/Greentube\.Messaging\.\d.*\.nupkg/


I'm still trying to push the markdown docs automatically together to NuGet. I've asked about it on StackOverflow a few days ago but still not answer.

One downside of this approach is that the dependencies between the packages have to be tracked by yourself.

For example:


If I want to publish Greentube.Messaging.DependencyInjection.Redis, I need to take into consideration that it depends on Greentube.Messaging.DependencyInjection which in turn depends on Greentube.Messaging of the same version.

For that reason, the 3 packages have to deployed otherwise restoring the first one will fail.


Configuration

Again using the Greentube.Messaging as an example, here's the .appveyor.yml


Note about the pace of changes

Everything about .NET Core changes rapidly. The tooling around it specially. A lot of configuration I've come up with was added as a reaction to some issue I encountered. I'm sure many of these issues will be solved soon so configuration could be simplified. An example is the full pdbs for coverage and the profiling API on MacOS. Take that into account when getting your own setup ready. 

Sunday, April 21, 2013

Wake on Lan in C# and Windows 8

About 8 years ago I was writing scripts to run on a network with over 130.000 computers (of which 5000 I administered).
The scripts ran 24/7, parsing computer's inventory log files, which they sent to a central server. It was  possible to detect and fix a whole bunch of issues, most of the time even before a user would notice something was wrong.

Note that most of those computers were running Windows NT 4, including the domain controllers. The task to install application in all those computers and keep their anti-virus signature up-to-date was not as trivial as it is today. There were times we needed to perform tasks on computers that weren't even switched on. And I must admit, back then I was quite proud of the solution I came up with for this particular case. Although it's not my goal to go into details on how I managed to get any of those 5000 computers, spread in 130 different offices, powered-on at any time; I want to write a little about the core of the solution: Wake on Lan

Whether to be able to power on your computer at home when you are away, or to manage a corporate network, the ability to switch a computer on out of sending a magic packet is at least quite interesting. It does require basic knowledge of computer networks and hardware to understand it, but to make it work, all you need is to know how to find your computer's MAC address. That's all you need to use WoL on a local network. However if you are willing to use it over the Internet, you'll need the IP address of the gateway of which the target computer belongs to, and have that gateway configured to forward the magic packet to the private network. The Port number is used in this case (if you use NAT, you'll need it), so that you can forward incoming UDP datagram on that predefined port of your router to an internal network IP address, or even the broadcast address if you'd like.

If the computer is switched off, it's likely the network switch's CAM table won't have an entry to your target computer's MAC address, and once your router forwards the datagram received from the external interface to your internal network, to the specific IP address you defined on the forwarding rule, the network switch will broadcast that to all ports.

You'll find many over architectured implementations of Wake on Lan out there, even in C#. But the fact is that it's really simple thing.

Let's see some C# code:


using System;
using System.Net;
using System.Net.NetworkInformation;
using System.Net.Sockets;
using System.Text.RegularExpressions;

namespace BrunoGarcia.Net
{
    public sealed class WolManager
    {
        const int _payloadSize = 102;

        public static void SendMagicPacket(string mac, IPEndPoint ipEndPoint)
        {
            var macBytes = PhysicalAddress.Parse(mac).GetAddressBytes();

            var payload = new byte[_payloadSize];
            Buffer.BlockCopy(new byte[] { 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF }, 0, payload, 0, 6);
            
            for (int i = 1; i < 17; i++)
                Buffer.BlockCopy(macBytes, 0, payload, 6 * i, 6);

            using (var udp = new UdpClient())
                udp.Send(payload, _payloadSize, ipEndPoint);
        }
    }
}

As you can see, it's a fire-and-forget operation, a single UDP datagram. The MAC address of the target machine goes in the payload. As I mentioned before, the port is not really important if you are sending the packet on the same subnet of your target computer (without the need of routing).  Port on wikipedia you find to be 7 or 9. However 0 can be used:


WolManager.SendMagicPacket("B8-AC-6F-59-56-55", 
    new IPEndPoint(IPAddress.Parse("255.255.255.255"), 0));

This is fun, and I've used it to power on my computer at home from the Internet a few times. But now, when I got Windows 8, the fun wasn't quite working.

This KB from Microsoft describes the change:

Windows 7:  In Windows 7, the default shutdown operation puts the system into classic shutdown (S5) and all devices are put into the lowest power state D3. Wake-On-LAN is not officially supported from S5 in Windows 7. However, some network adapters can be left armed for wake if enough residual power is available. As a result, wake from the S5 state is possible on some systems where enough residual power was supplied to the network adapter even though the system is in S5 and devices are in D3. 

Windows 8:  In Windows 8, the default shutdown behavior puts the system into hybrid shutdown (S4) and all devices are put into D3. Remote Wake-On-LAN from hybrid shutdown (S4) or classic shutdown (S5) is unsupported. Network adapters are explicitly not armed for Wake-On-LAN in both the classic shutdown (S5) and hybrid shutdown (S4) cases because users expect zero power consumption and battery drain in the shutdown state. This behavior removes the possibility of spurious wakes when explicit shutdown was requested. As a result, Wake-On-LAN is only supported from sleep (S3) or hibernate (S4) in Windows 8.

What a bummer. Anyway, not time to give up, right? Digging a bit deeper one understands better the changes, and enabling wake on lan on windows 8 is possible again:

Check out this post from Phil Pendlebury which talks about it.

Monday, March 4, 2013

Moq Return method not available after Setup

If you are familiar with the mocking framework Moq, you're used to call Setup with the overload taking a Func<T, TResult> and expect after that the Return<TResult> method to be available. And it's normally there.

However, I just ran into an interesting scenario, where calling the correct overload did not make available the Return<TResult> method.

In my case the code being mocked is a dependency that makes a request on a webservice.
A mockup of the wrapper class is created, and the Load method, which returns an XDocument, is setup.

But Return wasn't available:



Interesting that the overload resolution resolved my Func<T, TResult> to Action<T>, therefore ISetup<T> was being returned instead of ISetup<T, TResult> even though the method I setup had return type!

I inspect the method I setup to double check the return type defined:


Yes, it's missing a reference to System.Xml.Linq.
The Unit Test Project template doesn't include a reference to this assembly (and that makes sense).
Well, I added it, it works now.


Saturday, February 23, 2013

SignalR: Using a Hub instance not created by the HubPipeline is unsupported

When you need to push data to a SignalR hub from outside the hub (from a Controller for example), don't try to create a new instance of the Hub, like I did. Otherwise you'll see this nice exception:

"Using a Hub instance not created by the HubPipeline is unsupported"

From Microsoft.AspNet.SignalR.Core

Instead, the hub context must be retrieved:

var context = GlobalHost.ConnectionManager.GetHubContext<HubType>();
context.Clients.All.Whatever();

Saturday, February 16, 2013

Regex to match executing a stored procedure within a stored procedure

Unless you're working with a custom string format, which requires you writing a regular expression, I suppose (suggest?) you'll look it up online, right?

If you need a Regex to match an IP, MAC or an E-mail, would you spend time writing it? Chances are that you might leave room for false positives and/or false negatives, unless you really test it. That's why it's common to look it up online.

I needed to take a list of proc names and parse thousands of create procedure scripts, looking up if anything from the input list was used (executed) from those procs. A procedure executing another procedure.

This kind of call can be made in several different ways. It might have or not the database name (cross db call), server name (linked server), it might have or not the EXEC/EXECUTE keyword. It might have or not brackets or set the result to a variable.

Since I looked it up and couldn't find it, I wrote it.
In case you need it, here you go:

^\s*((exec(ute)?)\s+)?(@\w+\s+=\s+)?((\[?\w+\]?\.{1,2}){1,3})?\[?p_storedProcedure\]?\s.*$

Note that if you use SQL Server with default collation, proc names are case insensitive. So make sure you let your Regex engine know it should ignore cases! Otherwise you'll have to mind the case of the EXEC/EXECUTE keywords anyway.

Also note that cross database and linked server calls will also match. Some examples of valid proc calls (that will match against the Regex) are:

p_storedProcedure -- comments
p_storedProcedure @id, @anotherParam
[p_storedProcedure] @id, @anotherParam
EXEC p_storedProcedure @id, @anotherParam
EXECUTE p_storedProcedure @id, @anotherParam
EXEC p_storedProcedure
EXEC p_storedProcedure -- comments
EXEC [p_storedProcedure] @id, @anotherParam
EXEC dbo.p_storedProcedure @id, @anotherParam
EXEC dbo.[p_storedProcedure] @id, @anotherParam
EXEC anySchema.[p_storedProcedure] @id, @anotherParam
EXECUTE dbo.[p_storedProcedure] @id, @anotherParam
EXEC [dbo].[p_storedProcedure] @id, @anotherParam
EXEC [dbo].p_storedProcedure @id, @anotherParam
EXEC DBTEST..p_storedProcedure @id, @anotherParam
EXEC DBTEST..[p_storedProcedure]
EXEC DBTEST.dbo.p_storedProcedure @id, @anotherParam
EXEC [DBTEST]..[p_storedProcedure] @id, @anotherParam
EXEC [DBTEST]..p_storedProcedure @id, @anotherParam
EXEC [DBTEST].[dbo].[p_storedProcedure] @id, @anotherParam
EXEC [LINKEDDATABASE].[DBTEST].[dbo].[p_storedProcedure] @id, @anotherParam
EXEC LINKEDDATABASE.DBTEST.dbo.p_storedProcedure @id, @anotherParam
EXEC LINKEDDATABASE.DBTEST..p_storedProcedure @id, @anotherParam
EXEC @paramName = [dbo].[p_storedProcedure] @id, @anotherParam
EXEC @paramName = dbo.p_storedProcedure
EXEC @paramName = p_storedProcedure @id, @anotherParam
EXECUTE @paramName = p_storedProcedure @id, @anotherParam
EXECUTE    @paramName     =     p_storedProcedure     @id, @anotherParam

Wednesday, December 12, 2012

Pitfalls on WIF+SAML2 and Selenium

WIF and SAML 2.0


First some background: There is a known issue on WIF (Windows Identity Foundation) for SAML 2.0 that generates cookies with a name being a GUID and the value, base64 encoded data that grows every SAMLRequest the module handles. The decoded value looks like: 0;1;2;3;4;5;6;7;8;9;10;11;12;13;14;15
It starts with small ones but get really, really large.



Every client gets one of these cookies and each time they are bigger, to the point that when they are sent back to the server, an HTTP error is thrown: HTTP 400 - Bad Request (Request Header too long)

This msdn link has a comment with the first steps to take in case you end up with this problem. They are very straight forward and we did them even before ending up on that msdn page. Regarding their forth step (final "fix"), in our case, it was decided a different solution.

The solution here was to remove the cookies before they would be sent out to the user in the first place. This way, even though for some really short time the cookie existed in memory at the server, the client never got to know of its existence. Do achieve that, login and logout gotta be changed. That is: SignIn and RedirectingToIdentityProvider events from the Saml2AuthenticationModule. At that point in the event pipeline, the underlying Microsoft WIF code had already added the cookies to the Response, which gives us opportunity to remove them before the headers are sent out to the client.

Which takes us to Selenium:


The final solution had to be tested before dropping new build to production. And to test it, we had to reproduce it. The issue was not known during Dev or QA phases/environments, it did not happen, so the first step was to be able to reproduce it on a controlled environment.

Basically the idea was to use Selenium to simulate few dozens of users logging in and off in parallel until a cookie matching a GUID (plus a number?) would be received by one of the clients. There was no need to let it grow to the point of having: HTTP 400 - Bad Request (Request Header too long)

For that, I wrote a small application to spawn a thread for each IWebDriver (threads from the pool were conflicting the Drivers), each logging in with a different user account, removing the cookies (so user would be challenged again) and starting over.

The code would detect the existence of the cookie and stop the test, but to make visible (the cookies in and out) we can load Selenium driver with Firebug enabled and the cookie tab enabled and visible as default.

That goes like:

const string firebug = @"firebug-1.10.6-fx.xpi";
IWebDriver driver;
if (includeFireBug && File.Exists(firebug))
{
 var profile = new FirefoxProfile();
 profile.AddExtension(firebug);
 // Set default Firebug preferences
 profile.SetPreference("extensions.firebug.currentVersion", "1.10.6");
 profile.SetPreference("extensions.firebug.allPagesActivation", "on");
 profile.SetPreference("extensions.firebug.defaultPanelName", "cookies");
 profile.SetPreference("extensions.firebug.net.enableSites", true);
 profile.SetPreference("extensions.firebug.cookies.enableSites", true);
 driver = new FirefoxDriver(profile);

I mentioned the code would check the cookies to look for the GUID one, and with selenium API, it's very simple to do so:

Guid test;
if (driver.Manage().Cookies.AllCookies.Any(p => p.Name.Length >= 37 
    && Guid.TryParse(p.Name.Substring(0, 36), out test)))
{
...

Just checked if it's big enough to be GUID, then tried to parse the GUID part of it (note it appends some number to sequentially divide them into 2k sized each).

Two domains involved in this test. The service provider, let's call: service.com
And the identity provider: idp.com

Initially I set the IWebDriver Url property to the service provider: service.com
Find the element for Login and fire a click. That would call the SAML module that would redirect the client to the identity provider: idp.com

The login and password input elements would be filled up and login button triggered in the IdP page.
At this point, the session cookie from the IdP was sent to the browser, under idp.com domain, and client redirected back to service.com. SAML flow finished and session cookies from service.com also sent to the client.

That's all we need to reproduce the issue. However, these steps had to be done over and over, several times until the issue would happen. Particularly in our case, Logout was not possible since the accounts used were test account and thus not validated, so simply deleting the cookies would enable us restart the flow (and save us some requests/time). But this means deleting cookies from both service.com and idp.com.

Using Selenium API, I wrote:

driver.Manage().Cookies.DeleteAllCookies();

Even though the method is called DeleteAllCookies, it deletes all cookies from the current domain on which the WebDriver is located. In this case, service.com, since user just landed after the SAML login.
Looping the Cookies collection from within the WebDriver obviously would return only the cookies from the current domain.

It was the time for a second maneuver:
Setting the Url property of the WebDriver to anywhere under the idp.com domain that wouldn't return with a redirection, and call again DeleteAllCookies. That simple.
I browsed the root of the domain, without any resource id, which returned 403.14 - Directory listing denied. That was enough to run a code like:

// right after login flow finished (landed on service.com, logged in)
driver.Manage().Cookies.DeleteAllCookies(); // deletes service.com cookies
driver.Url = "idp.com";
driver.Manage().Cookies.DeleteAllCookies(); // deletes idp.com cookies

After that the flow could be re-initiated. After few hundreds of times, we could reproduce the issue, add the fix, run the test again with thousands of logins, without any issues.