Building secure CI/CD pipeline with Powershell DSC. Part 2: CI for IaaC

In the previous post, I described how to build CI-as-a-code with DSC Pull server with a focus on the security by using partial configurations and updating the on-demand.

Here comes the next challenge. Now, we want the DSC States that define Infrastructure configuration to be easily modifiable and deployable. As if we wanted to patch against Wannacry simply by adding a corresponding Windows update patch to the DSC Security state. “git push” – and in a short while all nodes in the CI pipeline would be secured. Below, I show how it can be done with script examples that can help you kick-start your CI for IaaC.

Before we start…

First, I’ll remind you some terminology.

DSC Configuration (State) is a Powershell-compiled file that needs to be placed on the Pull server. It defines the system configuration and is expected to get updates frequently.

Local Configuration Manager (LCM) is a Powershell-compiled file that has to be deployed to a target node. This file tells the node about the Pull server and which Configurations to get from it. This file is used once to register the node. Updates to LCM are rare and happen only in the case when the Pull server needs to be changed or new states added to the node configuration. However, we previously split the states into three groups – CI configuration state, Security state, and General system configuration. Given this structure, you can update only states without creating new types of them. Therefore, no LCM changes are required.

Also, keep in mind the fact that LCMs are totally different for targets with Powershell v4 (WMF4) and v5. And you need different machines to build them.

Looking from the delivery perspective, the difference between LCM and States is that LCMs need to be applied with administrative permissions and require running some node-side Powershell. In one of my previous posts, you can find more info on the most useful cmdlets for LCM.

On the contrary, States are easy to update and get to work – you only need to compile and drop them to the Configurations folder of the Pull server. No heavy-weight operations required. So, we design our CI for IaaC for frequent deployment of States in mind.

Building CI for IaaC

For the CI the first thing is always the source control. One of my former colleagues loved to ask the question at interviews: “For which type of a software project would you use source control?” And one and the only correct answer was: “For absolutely any”.

So, we don’t want to screw up that interview, and also our CI pipeline, therefore, we got the DSC States and LCMs under source control. Next, we want States and LCMs to be built on the code check-in time. The states will be published to the Pull server immediately,  while LCMs can be stored on the secure file share without direct applying them to the node.

ci_for_dsc

Building the artifacts is not a big deal – I’ll leave the commands out of this blog post. But what is still missing is how the nodes get LCMs. My way of doing it is to have a script that iterates over nodes and applies corresponding LCMs from the file share to them. I call it “Enroll to DSC”. Which is pretty fair since it happens when we need either to enroll a new node to a server or get some new states into it.

Here is the example of such script that uses Windows remoting in place from my Github. You can find details in README.md

Summary

By creating CI for IaaC we bring the best of DevOps practices to the way we handle the build infrastructure. In fact, having an abstract CI in place already simplifies our job, and after we are done – the CI itself becomes more reliable and controllable structure. You can deliver any updates to CI with CI it within seconds – isn’t it what CI supposed to be for? Quite a nice recursion example, I think 🙂

Building secure CI/CD pipeline with Powershell DSC. Part 1: overview

From my experience, way too often CI/CD pipelines suffer from the lack of security and general configuration consistency. There still might be an IaaC solution in place but it usually focuses on delivering a minimal functionality that is required for building a product and/or recreating the infrastructure if needed as fast as possible. Only a few of CI pipelines were built with security in mind.

I liked Powershell DSC for being native to the Windows stack and intensively developing feature modules to avoid gloomy scripting and hacking into the system’s guts. This makes it a good choice for delivering IaaC with Windows-specific security in mind.

DSC crash course

First, a short introduction of Powershell DSC in the Pull mode. In this configuration, DSC States or Configurations are deployed to and taken into use by the Pull server. States define what our node system configuration needs to look like, which features to have, which users to be admins, which apps installed etc – pretty much anything.

Configuration FirewallConfig
{


Import-DscResource -ModuleName PSDesiredStateConfiguration -ModuleVersion 1.1
Import-DscResource -ModuleName xNetworking -ModuleVersion 3.1.0.0

 xFirewall TCPInbound
 {
     Action = 'Allow'
     Direction = 'Inbound'
     Enabled = $true
     Name = 'TCP Inbound'
     LocalPort = '443'
     Protocol = 'TCP' 
 }

}

Each State needs to be built into the configuration resource of specific “.mof” format. Then, the state.mof need to be placed into “Configurations” folder on the Pull server together with its checksum file. Once the files are there, they can be used by nodes.

The second piece of config is the Local Configuration Manager file. This is the basic configuration for a Node that instructs it where to find the Pull server, how to get authorized with it and which states to use. More information is available in the official documentation.

To start using the DSC, you need to:

  • setup a pull server (once)
  • build a state .mof and checksum file and place it on the pull server (many times)
  • build a LocalConfigurationManager .mof and place it on the node (once or more)
  • instruct node to use LCM file (once or more)

After this, a node contacts the Pull server and receives one or more configuration according to its LCM. Then, a node starts a consistency against the states and correcting any difference it finds.

Splitting states from the Security perspective

It is the states that are going to be changed once we want to modify the configuration of enrolled nodes. I think it is a good practice to split a node state into pieces – to better control security and system settings of machines in the CI/CD cluster. For example, we can have the same security set of rules and patches that we want to apply to all our build machines but keep their tools and environment configurations corresponding to their actual build roles.

Let’s say, we split Configuration of Build node 1 into the following pieces:

  • Build type 1 state – all tools and environment settings required for performing a build (unique per build role)
  • Security and patches – updates state, particular patches we want to apply, firewall settings (same for all machines in the CI/CD)
  • General setting – system setting that (same for all machines in the CI/CD)

dsc_states

In this case, we make sure that security is consistent across CI/CD cluster no matter what role a build machine has. We can easily add more patches or rules, rebuild a configuration mof file and place it on the Pull server. The next time the node checks for the state, it will fetch the latest configuration and perform the update.

In the next post, I will explain how to build a simple CI pipeline for the CI – or how to deliver LCMs to nodes and configuration mof’s to Pull server with using a centralized CI server.

 

Scripts to find WannaCry vulnerable VMs in VMWare vCenter

WannaCry ransomware hit the news by infecting high-profile targets via a security hole that existed in Windows prior to 13.03.2017 when it was patched.

I created a script that connects to the vCenter and checks if the latest hotfix in the system was installed before or after Microsoft released the patch. This doesn’t give 100% protection since some fixes might have been installed manually but the required one omitted.

However, in centralized IT environments that rely on turned on Windows Update service that applies all important updates, it might be a good way to check for the vulnerability.

Link to my repo:

https://github.com/doshyt/Wannacry-UpdatesScan

I work on bringing similar functionality to PS Remoting and also looking for ways to figure out if a new patch was installed but not the one fixing the problem.

UPDATE 19.05:

I added a useful script that performs a remote check for SMBv1 being enabled for Windows 8 / Server 2012 + machines. It can be run against a list of computer names / FQDNs.

https://github.com/doshyt/Wannacry-UpdatesScan/blob/master/checkSmbOn.ps1

Returns $true if SMBv1 is enabled on a system level.

To turn off SMBv1, execute the following command on a remote machine

Set-SmbServerConfiguration -EnableSMB1Protocol $false