Building secure CI/CD pipeline with Powershell DSC. Part 2: CI for IaaC

In the previous post, I described how to build CI-as-a-code with DSC Pull server with a focus on the security by using partial configurations and updating the on-demand.

Here comes the next challenge. Now, we want the DSC States that define Infrastructure configuration to be easily modifiable and deployable. As if we wanted to patch against Wannacry simply by adding a corresponding Windows update patch to the DSC Security state. “git push” – and in a short while all nodes in the CI pipeline would be secured. Below, I show how it can be done with script examples that can help you kick-start your CI for IaaC.

Before we start…

First, I’ll remind you some terminology.

DSC Configuration (State) is a Powershell-compiled file that needs to be placed on the Pull server. It defines the system configuration and is expected to get updates frequently.

Local Configuration Manager (LCM) is a Powershell-compiled file that has to be deployed to a target node. This file tells the node about the Pull server and which Configurations to get from it. This file is used once to register the node. Updates to LCM are rare and happen only in the case when the Pull server needs to be changed or new states added to the node configuration. However, we previously split the states into three groups – CI configuration state, Security state, and General system configuration. Given this structure, you can update only states without creating new types of them. Therefore, no LCM changes are required.

Also, keep in mind the fact that LCMs are totally different for targets with Powershell v4 (WMF4) and v5. And you need different machines to build them.

Looking from the delivery perspective, the difference between LCM and States is that LCMs need to be applied with administrative permissions and require running some node-side Powershell. In one of my previous posts, you can find more info on the most useful cmdlets for LCM.

On the contrary, States are easy to update and get to work – you only need to compile and drop them to the Configurations folder of the Pull server. No heavy-weight operations required. So, we design our CI for IaaC for frequent deployment of States in mind.

Building CI for IaaC

For the CI the first thing is always the source control. One of my former colleagues loved to ask the question at interviews: “For which type of a software project would you use source control?” And one and the only correct answer was: “For absolutely any”.

So, we don’t want to screw up that interview, and also our CI pipeline, therefore, we got the DSC States and LCMs under source control. Next, we want States and LCMs to be built on the code check-in time. The states will be published to the Pull server immediately,  while LCMs can be stored on the secure file share without direct applying them to the node.

ci_for_dsc

Building the artifacts is not a big deal – I’ll leave the commands out of this blog post. But what is still missing is how the nodes get LCMs. My way of doing it is to have a script that iterates over nodes and applies corresponding LCMs from the file share to them. I call it “Enroll to DSC”. Which is pretty fair since it happens when we need either to enroll a new node to a server or get some new states into it.

Here is the example of such script that uses Windows remoting in place from my Github. You can find details in README.md

Summary

By creating CI for IaaC we bring the best of DevOps practices to the way we handle the build infrastructure. In fact, having an abstract CI in place already simplifies our job, and after we are done – the CI itself becomes more reliable and controllable structure. You can deliver any updates to CI with CI it within seconds – isn’t it what CI supposed to be for? Quite a nice recursion example, I think 🙂

Building secure CI/CD pipeline with Powershell DSC. Part 1: overview

From my experience, way too often CI/CD pipelines suffer from the lack of security and general configuration consistency. There still might be an IaaC solution in place but it usually focuses on delivering a minimal functionality that is required for building a product and/or recreating the infrastructure if needed as fast as possible. Only a few of CI pipelines were built with security in mind.

I liked Powershell DSC for being native to the Windows stack and intensively developing feature modules to avoid gloomy scripting and hacking into the system’s guts. This makes it a good choice for delivering IaaC with Windows-specific security in mind.

DSC crash course

First, a short introduction of Powershell DSC in the Pull mode. In this configuration, DSC States or Configurations are deployed to and taken into use by the Pull server. States define what our node system configuration needs to look like, which features to have, which users to be admins, which apps installed etc – pretty much anything.

Configuration FirewallConfig
{


Import-DscResource -ModuleName PSDesiredStateConfiguration -ModuleVersion 1.1
Import-DscResource -ModuleName xNetworking -ModuleVersion 3.1.0.0

 xFirewall TCPInbound
 {
     Action = 'Allow'
     Direction = 'Inbound'
     Enabled = $true
     Name = 'TCP Inbound'
     LocalPort = '443'
     Protocol = 'TCP' 
 }

}

Each State needs to be built into the configuration resource of specific “.mof” format. Then, the state.mof need to be placed into “Configurations” folder on the Pull server together with its checksum file. Once the files are there, they can be used by nodes.

The second piece of config is the Local Configuration Manager file. This is the basic configuration for a Node that instructs it where to find the Pull server, how to get authorized with it and which states to use. More information is available in the official documentation.

To start using the DSC, you need to:

  • setup a pull server (once)
  • build a state .mof and checksum file and place it on the pull server (many times)
  • build a LocalConfigurationManager .mof and place it on the node (once or more)
  • instruct node to use LCM file (once or more)

After this, a node contacts the Pull server and receives one or more configuration according to its LCM. Then, a node starts a consistency against the states and correcting any difference it finds.

Splitting states from the Security perspective

It is the states that are going to be changed once we want to modify the configuration of enrolled nodes. I think it is a good practice to split a node state into pieces – to better control security and system settings of machines in the CI/CD cluster. For example, we can have the same security set of rules and patches that we want to apply to all our build machines but keep their tools and environment configurations corresponding to their actual build roles.

Let’s say, we split Configuration of Build node 1 into the following pieces:

  • Build type 1 state – all tools and environment settings required for performing a build (unique per build role)
  • Security and patches – updates state, particular patches we want to apply, firewall settings (same for all machines in the CI/CD)
  • General setting – system setting that (same for all machines in the CI/CD)

dsc_states

In this case, we make sure that security is consistent across CI/CD cluster no matter what role a build machine has. We can easily add more patches or rules, rebuild a configuration mof file and place it on the Pull server. The next time the node checks for the state, it will fetch the latest configuration and perform the update.

In the next post, I will explain how to build a simple CI pipeline for the CI – or how to deliver LCMs to nodes and configuration mof’s to Pull server with using a centralized CI server.

 

Scripts to find WannaCry vulnerable VMs in VMWare vCenter

WannaCry ransomware hit the news by infecting high-profile targets via a security hole that existed in Windows prior to 13.03.2017 when it was patched.

I created a script that connects to the vCenter and checks if the latest hotfix in the system was installed before or after Microsoft released the patch. This doesn’t give 100% protection since some fixes might have been installed manually but the required one omitted.

However, in centralized IT environments that rely on turned on Windows Update service that applies all important updates, it might be a good way to check for the vulnerability.

Link to my repo:

https://github.com/doshyt/Wannacry-UpdatesScan

I work on bringing similar functionality to PS Remoting and also looking for ways to figure out if a new patch was installed but not the one fixing the problem.

UPDATE 19.05:

I added a useful script that performs a remote check for SMBv1 being enabled for Windows 8 / Server 2012 + machines. It can be run against a list of computer names / FQDNs.

https://github.com/doshyt/Wannacry-UpdatesScan/blob/master/checkSmbOn.ps1

Returns $true if SMBv1 is enabled on a system level.

To turn off SMBv1, execute the following command on a remote machine

Set-SmbServerConfiguration -EnableSMB1Protocol $false

 

Benefiting from TeamCity Reverse dependencies

Reverse Dependencies is the feature of TeamCity that allows building more complex build workflows with setting parameters from “parent” build down to their own snapshot dependencies. I’ve been looking for this functionality for a while and recently accidentally discovered how to make it working. Back then, I felt like I found a treasure 🙂

Example: one of the builds can be run with code analysis turned on – for using the results in the “parent” code analysis build (i.e. with SonarQube).  But you only need it when manually triggering a SonarQube build, in any other case (i.e. code checked-in to the repo) you don’t want TC to spend time on running Code analysis. The code analysis is turned on with a System property system.RunCodeAnalysis=TRUE in the “child” build.

And here is the trick  – the “parent” wants to set a property of the “child” build but can’t access it outside of the scope of own parameters. How would you do it in a common way? Maybe, create two different

How would you do it in a common way? Maybe, create two different builds – where one has the system.RunCodeAnalysis always TRUE and trigger it from the SonarQube build.

In this case, you end up having two almost duplicated builds that only exist because the “parent” can’t set properties of a “child”.

Reverse dependencies are here to help

With Reverse dependencies, it can!

This feature is not intuitively simple, and it doesn’t support TeamCity auto-substitution (as with using %%). So, you need to be careful with naming. This is how it works:

In the “parent” build (let’s call it SonarQube), you set to TRUE a parameter named

reverse.dep.PROJECT_NAME.PARAMETER_NAME

Here, PARAMTER_NAME is the exact name of a parameter that you want to rewrite in the “child” build. I.e. “system.RunCodeAnalysis”.

For this to work, you need to have a snapshot dependency to the “child” build enabled. In the “child” build you simply set system.RunCodeAnalysis=FALSE

When someone triggers the “parent” build. First, it rewrites the default (existing) value of the specified parameter (system.RunCodeAnalysis) in the “child” build with TRUE and starts this build.

In this case, you can use one build definition for sharing different tasks that couldn’t be done with one “core” build previously. A great use case is setting up a build with an automated TC trigger in which you can set parameters of a “child” build. Let’s say, you want to deploy Nightly to a specific environment using the same build definition that all developers use for building their projects.
To do so, you can set up a build with a trigger that builds a specific branch and also sets a target environment parameter through the Reverse dependency.

When using Reverse Dependencies, a handy way to check the actual values submitted to a build si the “Parameters” tab in the executed build – it shows which values were assigned to parameters, you can make sure that Reverse dependencies work as expected.

Caveats

  1. Reverse Dependencies are not substituted with actual parameter names from “child” builds – it is easy to make a mistake in the definition.
  2. When a project name is changed, you also need to change it manually in all Reverse dependencies.
  3. Try not to modify the build flow with Reverse Dependencies, touching only features that don’t affect build results in any way – otherwise you will get non-deterministic build configuration, in which the same build produces totally different artifacts. The best way to use it – is to specify some parameters which will be used by external parties, like setting einvrionemnts for Deployment or Publishing services, getting code analysis results etc.

Setting up LCM with DSC PullServer – cmdlets you need to know

Powershell DSC has a slightly steep learning curve – from the beginning, it is not that straight-forward to figure out how to read logs, how to trigger a consistency check or how to get updated configurations from the Pull server.

After building LCM.meta.mof file, you need to apply it to the machine – so that it enrolls itself with DSC PullServer and determines which configuration states to pull and apply. In fact, LCM is the heart of DSC on each node – and it requires some special treatment in order to deliver predictable results. So, to start with LCM, you need to point DSC engine to the folder where .meta.nof of LCM is stored and register it in the system. This works as follows:

Set-DscLocalConfigurationManager -Path PATH_TO_FOLDER

As far as I noticed, starting from Powershell 5.1 update, this command automatically pulls the resources from Pull server. Previously, it was necessary to trigger the resource update without waiting for the standard DSC consistency check interval – 15-30 minutes.

Update-DscConfiguration -Verbose -Wait

This command also comes handy when there was an update to resources in the server (a new configuration state was uploaded or a PS module version got updated).

When we want to start DSC consistency check (don’t mix it with Update-DscConfiguration, the latter only updates resources, it doesn’t run the consistency check) without waiting for the DSC scheduler to do it for us:

Start-DscConfiguration -UseExisting -Verbose -Wait

After updating resources, registering LCM and starting DSC check, we need to check the status. Here comes the trick – first we need to make sure that there is no consistency check in place – otherwise, we can’t get the status of LCM. So, we run:

Get-DscLocalConfigurationManager

This command returns a bunch of parameters – where we are mainly interested in LCMState.

Get-DscLocalConfigurationManager | Select-Object -ExpandProperty LCMState

It can be either “Idle”, “Busy” or report inconsistent configuration which leaves the LCM in a blocked state. When it is “Idle”, we are good to go – and check the actual result of applying a configuration State pulled from the Pull server.

Get-DscConfigurationStatus

The outputs are either “Failed” or “Success” – and this gives us an answer to the question whether the machine is in the desired state or something went terribly wrong.

And the last command – how to get rid of the DSC configuration:

Remove-DscConfigurationDocument -Stage Current,Previous,Pending

DSC stores two configurations for LCM – current (the last applied) and previous. When it ends up in the “pending” state, most likely, you have a problem with your LCM or State. After using this command for clean up, you may go and set updated LCM.

Making VMWare Integration Plugin 5.5 work in modern browser

When deploying a new OVF template to the vCenter 5.5 host from exported one, you may find yourself in trouble getting it done.

The desktop client may complain about invalid configuration for device 7 which is an extremely misleading message.

error7

According to support forums, to import OVF templates you must use a web interface. However, the web interface complains about the VMWare Integration plugin being not installed. It even offers you to download it.

But here is the trick – whenever you try to install its version 5.5 or 5.6 (since you need ESXi 5.5 compatibility), it simply never works with modern browsers – the plugin is not added to the browser extension list and not detected as installed. I assume this is caused by their enhanced security requirements to the installed extensions. Integration plugin from VMWare requires disk and system access and is silently blocked in newer browsers.

On the original requirements specification of the plugin, its stated its compatibility with IE 7 and 8.

The solution is simple – run your modern IE in the compatibility mode as IE 8! It works like a charm.

Nevertheless, I would recommend quitting the session after you finished using the plugin and come back to IE 11 to not expose your system to risk.

 

The Security Development Lifecycle book is available for downloading

Very recently, Microsoft has published online the foundation book describing SDL (Security Development Lifecycle).

security-development-lifecycle-no-cd

The principles behind the SDL were born as a response to the Windows Longhorn project reset in the early 2000s. Back then, the entire project was wiped out and started from scratch due to the presence of critical vulnerabilities in various components – according to MS insiders. At the time, Microsoft had a questionable reputation with regards to security of its products. Therefore, the company made a huge investment in security improvement. SDL was created as the common approach to developing products, starting from the very bottom to top – from design to release.

The book was published in good old 2006, which can be seen as the Stone age comparing to the threats and attack vectors present nowadays. Nevertheless, it still remains a valuable source of knowledge and actions for the teams and companies that struggle with improving a security of the products. In my opinion, it is impossible to deliver a secure solution without integrating SDL principles into every chunk of the development process.

The most recent overview of SDL can be found at the dedicated Microsoft page.

The best part of it is the set of tools and instruments designed and used by MS at each of the steps of SDL – with links for downloads. It can be seen as a great reference to the spectrum of problems that SDL solves – you don’t have to replicate it to your organization in the exact way it works at MS but at least it helps understand the challenges and possible solutions.