Update management in Hybrid cloud with Azure Automation and Log Analytics

It’s no news that Azure has a neat OMS integration and can be used to monitor update status of enrolled machines. What strikes me the most is the simplicity it provides to patching in the hybrid cloud infrastructure and the ability to get it under control in the minimal time.

Azure Update Management is part of an Automation Account and is tied to subscription it is created in. It means, you will be able to directly add Azure VMs from the same subscription. For other VMs – either Azure in a different subscription, or on-prem servers, you need to install OMS Agent. You can get one from the Log Analytics account associated with your Automation Account. You can find the link under the point 1 of the Getting started screen:

oms1

Proceed and grab an installer of the agent. You can unpack the agent MSI to a folder with /c /t:<folder> flags and install unattended – by providing the installer the following LogAnalytics parameters – Workspace ID and Workspace Key (you can find them . To simplify the distribution, I zipped the folder with unwrapped MSI content and wrote a small script for unattended installation that can run in memory (except for extracting zip archive). It requires the mentioned workspace details and a link to the agent zip archive.

Once you install the agent, it starts reporting to LogAnalytics and provides visibility into the update status for each machine.

oms2

The information is available in both Update Management blade and LogAnalytics workspace. The latter provides better stats but the data is non-actionable, one would need to get back to Update Management and trigger update from there.

oms3

oms4

oms5

 

From the Update Management blade, one can:

  • schedule update deployment
  • include specific machines or groups
  • include or exclude particular patches by KB
  • select update categories to apply
  • schedule continuous update that will check and install required categories regularly

And the last detail – LogAnalytics can be switched to a free tier and will still be able to serve the needs of Update Management bringing costs of patch management on hundreds of servers close to zero.

Together with Azure DSC (configuration management) and Machine Inventory that I reviewed earlier, Azure Automation provides a wide range of tools to replace expensive and complex tools for managing hybrid infrastructure.

Automating alert response with Azure Security Center and Azure Logic Apps

Responding a security event is the core practice in the modern security frameworks. After a potential threat was detected, it is time to act. The shorter the response time is the less damage an attacker can deal to your cloud.

Detection in Azure

Azure Security Center in the Standard pricing tier ($15/VM node per month) comes with automated detection mechanisms. The core detection capability is built around real-time traffic and system logs parsing and applying machine learning algorithms to it:

security-center-detection-capabilities-fig1

A single dashboard can be found under Security Center -> Security Alerts blade and also on the main page of the Security Center:

alertsdetection

Alerts represent single or multiple security events of the same nature and time span. Incidents are created from multiple alerts which are classified as related to each other – for example, an attacker runs a malicious script, extracts local password hashes and cleans the event log. This sequence of action will generate one incident.

Incident forensics

Incidents can be investigated in a forensics tool Investigation Dashboard (in the preview, as of May 2018). This tool draws the relationships between alerts, events that caused the alert, affected resources, users. It also can help when reconstructing lateral movements of attackers within the network.

investigation.PNG

Automated response

Incident forensics represents a post-mortem investigation. An adversary event did happen, and the attackers have already done some damage to the enterprise. We don’t have to wait until malicious actors finish their job – we can start acting right after getting the first signals about the intrusion. Alerts are generated by Azure in real-time, and recently Security Center got a powerful integration with Azure Logic Apps.

Logic Apps in Azure represent workflows of actions with pre-built triggers, conditions, and actions which include a wide range of both native and 3-rd components. For example, your logic app can listen to RSS feed and automatically tweet once new pages are published to the feed. Or, run a custom powershell through Azure Automation.

One of the recent additions to Logic Apps – Security Center triggers. This feature turns Azure security alerts into the powerful tool for fighting attackers once they trip a wire.

You can Security-related Azure Logic Apps under Security Center -> Playbooks (Preview).

Building the logic

After adding a new playbook, a user gets presented with Loic App Designer. The trigger is pre-populated – When a response to Azure Security Center alert is triggered. Once we get an alert, the playbook is executed. Then, we add a condition – there are multiple parameters that the alert arrives with. Let’s take “Alert Severity” and set the condition to High:

trigger

Other alert parameters include Confidence Level, Alert Body, Name, Start or End Time and many more. The range is quite broad which makes it possible to generate very specific responses to almost any imaginable event.

Now, if the condition is TRUE – Alert Severity is High, we want to contain the threat. One of the ways to do so is to isolate a VM under attack. Let’s say, assign it to a different Network Security Group which has no connection to the internal company network or some of its segments. To do it, we would need to get a VM name from the alert and run some Azure Powershell performing the NSG re-assignment.

Creating the Automation Job

Now, we can go to Azure Automation and create an Automation Job for our needs. This can be done through the blades Automation Accounts -> Runbooks -> Add a runbook. As Runbook type, choose “Powershell”.

Then, we insert the following code:

Param(
[string]$VMName
)

$connectionName = "AzureRunAsConnection"

try
{
# Get the connection "AzureRunAsConnection "
  $servicePrincipalConnection=Get-AutomationConnection -Name $connectionName

  Add-AzureRmAccount `
    -ServicePrincipal `
    -TenantId $servicePrincipalConnection.TenantId `
    -ApplicationId $servicePrincipalConnection.ApplicationId `
    -CertificateThumbprint $servicePrincipalConnection.CertificateThumbprint
}

catch {
  if (!$servicePrincipalConnection)
  {
     $ErrorMessage = "Connection $connectionName not found."
     throw $ErrorMessage
} else {
  Write-Error -Message $_.Exception
  throw $_.Exception
  }
}

# Get VM object
$vm = Get-AzureRmVM -Name $VMName -ResourceGroupName AzureBootcamp
# Get NIC
$Nic = Get-AzureRmNetworkInterface -ResourceGroupName AzureBootcamp | Where-Object {$_.VirtualMachine.Id -eq $vm.Id}
# Change Network Security group to IsolatedNetworkNSG
$Nic.NetworkSecurityGroup = Get-AzureRmNetworkSecurityGroup -ResourceGroupName AzureBootcamp -Name "IsolatedNetwork-NSG"
# Apply changes
Set-AzureRmNetworkInterface -NetworkInterface $Nic

This code gets the VMName as a parameter, authenticates to your Azure account with Azure Run-As connection (requires preliminary configuration). Then, it get’s VM’s NIC and assigns it to the security group “IsolatedNetwork-NSG”. Save the automation runbook with name IsolateVM, for instance, and don’t forget to publish the changes after editing Powershell.

Putting it all together

The last step, adding the action to the Azure Logic App we-ve been building. Select “Azure Automation – Create job” and point it to the IsolateVM automation book.

logicapptrue.PNG

Here, we specified “Host Name” as Runbook parameter (notice, it automatically picked up parameter name VMName that we created int he runbook).

Save the logic – and this is it. Once an alert is generated a VM is expelled to the isolated security group with limited access.

Testing and tuning the playbook

To test this integration before an actual event happens, go to any of previous events in Security Center – Security Alerts (you can generate them, for example, by trying to downloadMimikatz from Github), click on the event, then click on “View playbooks” button. In the new window find your Logic app workflow and press “Run” under “Run playbook”:

runpalybook

This will send exactly same trigger as this alert would have done. From the playbook run window or Run history, you will be presented with a static view similar to Logic App Designer with the only difference that it contains the logic path that was taken in this run:

logicexecution

Actual inputs that were submitted with the trigger can be viewed by expanding “When a response to an Azure Security Center alert is triggered” section.

alertdescription

The Azure Security Center alerts integration with Logic Apps provides limitless capabilities not only for informing about detections (via email, Slack, Skype) but also for an automated response to potential attacks with auto-tuning cloud infrastructure and isolating the threat, a show in the example.

Have fun building your own playbooks and fighting the threats before they become incidents.

Stay secure!

Identifying threats: Software inventory of Azure VMs

Azure VMs recently got a bunch of new features – Inventory, Change tracking and Update management (became GA on 8th of March, 2018). These features fill the gap in identifying the software that is deployed to IaaS clouds – the information necessary for securing these resources. The features are based on the Azure Automation capabilities and require an Automation account to run the workloads.

Inventory feature provides visibility into software installed on a VM (can be accessed from the individual VM blade), services, Linux daemons and also the timeline of events as part of the change the tracking view.

in2

VM view:

Inv1

In this example, we can see Adobe Flash Player and Steam client installed on the machine – both are increasing the attack surface for this infrastructure.

If you want to get more detail overview – proceed to Log Search tab. For example, this query will yield all non-Microsoft applications inventoried in the past day:

ConfigurationData
| where ConfigDataType == “Software”
| where SoftwareType == “Application”
| where Publisher != “Microsoft Corporation”
| order by TimeGenerated desc

logsearch1

The Change tracking view provides the visibility into the software changes and allows to track them efficiently. Also, you can add particular files or registry entries to watch. Watching entire folders is not yet supported for Windows VMs.

ch1.GIF

To get an overview of multiple machines either:

  • Click “Manage multiple machines” on Inventory or Change tracking view blade
    ch2
    This view also can be used to add non-Azure VMs through the Hybrid worker functionality.
  • Or go to the automation account that was associated with Inventory and Change tracking when first enabled. It also provides a convenient view of associated machines.

Update management can find machines with missing OS updates and fix it by scheduling a deployment.

upd1.PNG

Update management is integrated into the Security Center so that critical updates are never left out for all machines in the cloud. But it is important to remember to turn on these features first.

At the moment, only Update management allows taking an action based on the gathered data. Inventory doesn’t have a built-in functionality to remove unwanted applications or perform an on-demand scan. Also, it is lacking reporting capabilities and global overviews of applications in the cloud. Change tracking data can be used only for setting up custom alerts (requires Log Analytics knowledge).

At the same time, based on the direct connection of these features to the Azure Automation, I expect there will be more functions added to make it possible to fix found issues and secure a resource based on the Inventory data. It is clear that Microsoft is taking an important action in securing IaaS – and providing the data is only the first step.

P.S. For more advanced Inventory and Software Asset Management solutions, one may look into 3-rd party providers such as Snow Software (where I work at the moment).

 

Using ELK stack for vulnerability management

Have you tried to keep all data of discovered vulnerabilities in the same place? When you have about 10 different tools, and also manual records in task tracker, it becomes a bit of a nightmare.

My main needs were:

  1. Have all types of data in the same place – vulnerable computers, pentest findings, automated scan discoveries, software composition analysis etc.
  2. Have visibility over time in each of categories for each product – I need to talk to people with graphs and good visualizations to make my points.
  3. Have a birds-eye view on the state of security in products – to be able to act on the trend, before it materializes into the flaw or breach.
  4. Easily add new metrics.
  5. The answer to these questions needs to be free or inexpensive.

After weeks of thinking and looking for a tool that will save me, I figured out that the solution has always been just next to me. I simply spin up an instance of ELK (ElasticSearch, Logstash, Kibana) and set up a few simple rules with Logstash filters. They work as data hovers – by simply sucking in everything that is sent in as json payloads. Then, transformed data ends up in Elastic, and Kibana visualizes the vulnerabilities into trends and shows them in real-time dashboards. Also, it has a powerful search engine where I can find specific data in a matter of seconds.

I wanted to create a configuration that can process a generic json payload that was assembled by a script after parsing any vulnerability detection tool report.

Here is the example of logstash pipeline configuration:

input {
http { }
}

filter{
json{
source => “message”
}
}

filter {
date{
match => [ “date”, ISO8601 ]
}
}

output { elasticsearch {
hosts => localhost
index => “vuln-%{+YYYY.MM.dd}”
}
}

Here, we ingest http inputs posted to the logstash endpoint as json, set the @timestamp to the “date” field from the payload (it has to be submitted in ISO8601 format), and send this data to the ElasticSearch instance, to the index with name “vuln-<today’s date>”.

Let’s say we have 3 branches that are submitting open-source component analysis results on every build. Here is how the resulting report may look like (Y-axis represents discovered vulnerable components, X-axis represents scan dates, split in 3 sub-plots):

v1

Here, each of 3 subplots represents a separate branch analysis dynamics.

Or, for separate branches:

v2

And finally, we can build some nice data tables:

v3

This example only includes one case – OSS vulnerability reports. But various vulnerability data can be submitted to ELK, and then aggregated into powerful views for efficient management.