Lars-Erik's blog

Analyzing W3C logs with Excel and PowerQuery

Wed, 06 Jan 2021 00:34:42 Z

15 Years ago I wrote a .net WinForms tool called "IIS Log Viewer" to parse and visualize W3C log files in a nice color coded grid. Of course it topples over as soon as you analyze a file of some respectable size, so I'm sorry to say I abandoned maintenance of the program. Only today I realize I should have called it "W3C Log Viewer", but here's to hindsight. 🤣🍻

Today I use a different approach with Excel and PowerQuery instead. In this article I'll show how you can anlyze a fair amount of traffic (at least a million requests) from several logfiles possibly spanning your entire retention period. I'm a Microsoft fanboy, so I'll show how to do it with Azure App Services logging to Azure Storage accounts, but you can point to filesystem folders as well.

Setting up W3C logging on an Azure App Service

The nicest way of HTTP logging I know of on Azure is by logging to storage. It'll persist across slots when you swap, and you won't have to download them to analyze them.

You'll find the settings you want under Monitoring\App Service Logs:

Now, you'll production slot will happily rotate n days of logs in your storage account. They'll be stored in a hierarchy by slot name (?), year, month, date and finally hour. Totally horrible to analyze without a tool, but we got just the one! Also too difficult to screenshot, so you'll have to imagine it. 😇

Getting data into Excel

I believe PowerQuery really made it's break into Excel in the 2016 version. We'd had some "Power" tools (pivot?) from 2013, but this version really hit the target! I can't get over the fact I waited until 2020 to get more acquainted with it.

In the old versions, we more or less only had the horrible text import / split column wizard. In newer versions of Excel we've got a super rich "Get & Transform Data" section in the Data ribbon, not to speak of (almost) the full power of PowerQuery behind the scenes.

The "new" Excel data ribbon

I particularly like the "From Web" one, since it'll handle most modern formats such as JSON and XML. (Even CSV)

However, for our purposes, we'll find our source of choice by drilling into the "Get Data" dropdown:

We go on to enter our storage account name:

Then you'll need to find your account key and enter it in order to authorize with Azure:

Mind the security concerns and the data firewall when authenticating sources and sharing connected Excel files.

The next "wizard" we're faced with is a blob storage browser. We'll select the "logs" container, but leave it at that. We want to "load and transform" all the blobs we see on the right hand side.

So the final step is to resist the urge to click "Load", but rather click "Transform Data". This will launch the most powerful data transformation tool I know to date, the aptly named "PowerQuery" GUI.

Getting the contents of all the log files

Excel will immediately pop a new (sadly modal) window with a new Query aptly called "logs". The smiley up there is for sending feedback, but I like to think PowerQuery is just a happy place alltogether.

As you can see there's a bunch of options up top, but we're actually after a slightly more inconspicuous one down in the "Content" column header called "Combine files":

This will pop another dialog asking us about encoding and whether we want to skip things that bugs out. Also we can choose which file to use as our sample for further transformation:

We'll go ahead and use the first one, since the sequence of those don't matter. The're all in the same format. What follows is this:

We're now faced with a fairly more complex set of queries and steps in our "logs" query. The inner workings of those are out of the scope of this post, but I promise there's magic to be had if you'll dive deeper. The .net peeps reading this will also appreciate the familiar form of static method calling in the "formula bar". PowerQuery is a delightful blend of familiar syntaxes. 😇

The more interesting thing however is that (very quickly) the blob file contents are shown in a new column called "Column1". This is our next target.

Transforming the data and preparing for analysis

Before we can make anything meaningful out of the actual log data we have to get rid of some junk, and figure out what fields we've logged. These are the first two lines of the logfile. Alas they've got a bit of junk attached. The first line is just garbage, so we'll employ our first secret weapon. Remove top rows: (it's on the Home ribbon)

You'll be prompted with a dialog asking how many rows to remove. We want to remove the leading hash lines except the "#Fields" one. In my case, this is just the "#Software" row, so I remove 1 row.

The next step is to remove the leading "#Fields: " text of the column header specification. We'll drag out our second weapon of choice, "Replace Values". For this step we need to select "Column1" and move over to the "Transform" ribbon. Here we'll find the "Replace Values" dropdown where we select the values option:

We're faced with (yet) another dialog in which we specify that we want to get rid of the "#Fields: " string: (make sure to include the space)

Splitting columns

We're here! The good old "split column" phase is up!

Back at the home ribbon, we find our old friend "Split Column". For W3C log files we're dealing with space delimited files, so we go with the "by delimiter" option:

Yet another dialog where we specify space as the delimiter:

Now we're definitely getting somewhere:

Notice how we didn't have to specify any other options than how to split the data? No data types, no headers, nothing.

We do need to get those headers as column names, though. We're already on the home ribbon and if you've been looking closely you've likely noticed both "Remove Columns" and "Use First Row as Headers". We'll remove the now useless "Source.Name" column, and then use the first row as our headers:

Then we hit the "Use First Row as Headers" button, et voilá:

The utter sweetness (if you look in the bottom right corner) is that PowerQuery is awesome at guessing data types. Granted you have to nudge it a bit every now and then, but the date, times and numeric data is already properly typed.

Which leads us to the last necessary step for this to work flawlessly. PowerQuery has only sampled 1000 rows from the first file. It's not aware that several more lines starting with a hash is awaiting further down the road. That's gonna mess up it's date conversion for the first column. Here's for our final trick, "Remove errors". It's an option under our old friend "Remove rows" on the home ribbon. Make sure to select the date column before you hit it:

And that should be it! We're ready to hit that captivating "Close & Load" button up in the left corner of the home bar. Go ahead! Hit it!

Now depending on whether you've got less than a mil. lines total in your log files this will succeed or fail. Despair not though, there are ways to get around it. I'll add some tips at the bottom.

If you're lucky enough to escape the Excel row limit trap (unlike some unfortunate brits recently), you'll be faced with a fancy green table after a while. The "Queries & Connections" sidebar will appear and show progress of how many rows that's been loaded. (Including errors if any)
For me it takes a few minutes to load half a mil rows. (Granted I'm on a 1GB internet connection yielding .5 GB to my box. It is a few hundred MB to download.)
Here's how it looks:

So we've got half a million log rows loaded to Excel. I can't be bothered to investigate the 680 errors, but if you click the link you'll get a special query showing those and their causes. (❤)

However this is fairly hard to get some value out of in its present state. We need to pivot. If you've got less than a million you can go to the "Insert" ribbon and click "Pivot table". Make sure you've selected a cell in the log table first. Just accept the default options and skip to the next section.

Pivoting more than a million rows

Incidentally pivoting is also the trick (untested) if you've got more than a million rows. You'll have to right click the "logs" query in the "Queries & Connections" toolbar on the right and select "Load to". You can then change to a "Pivot Table" instead of table and skip loading all the details. I believe there's also options to build cached data models, but that's outside of my skillset and the scope of this post as well. :)

Pivoting W3C logs

Which ever way you opt to pivot the log data you'll end up with a worksheet like this:

I've got two favorite setups for analyzing logs. First of all, I'd like to examine whether a spike is due to a new bot of some sorts. We'll just block it if it's something we're not interested in. It can save an amazing amount of processing you possibly weren't aware you spent.

Though we imported all the log files, for this you'd likely want to look at a specific day. Dragging "date" to "Filters" lets you select a date in B2. (Full screenshot further down)

Next we want to look at agents, so we drag "cs(User-Agent)" to "Rows". (They'll all appear instantly)

Finally we want to see how many hits we've got from each agent and then sort. We can actually drag any field, but let's go with "time" for counting. The final dragging step is to drag "time" into "Values".

We should be left with something like this:

As you can see we still have quite a few agents to try to rid ourselves of, but the ones we see here have too acceptable request rates to bother. The one's we'd be interested in would have thousands in the "Count of time" column. To find them easily we have to sort, and this is were Excel can be a bit quirky. In order to sort the pivot by "Count of time" we have to open the dropdown menu on "Row Labels" and select "More Sort Options...":

What follows is a rather quirky dialog, but we're after "Descending (Z to A) by: Count of time":

We're now left with a table showing that todays winner was (bloody) iPhones!

Discovering facts

If we'd had an excessive agent, likely with "bot" in it's name, the next step would be to add "c-ip" as a secondary row dimension under "cs(User-Agent)". We could then drill down into the excessive agent and likely discover IPs to block or even report.

Today however, this process lead me to replacing "cs(User-Agent)" as my rows field with "cs-uri-stem". Repeating the sorting trick showed me that the customer obviously had had a campaign of some sorts leading to the sudden spike at three o'clock. Luckily our newly upgraded S3 plan managed to take it only dropping to 2 secs response time across 29 Umbraco sites, but I'm about to have a serious talk with this particular customer - leading to a slightly more expensive hosting fee for an individual plan. 😇

Another favourite setup of mine shows which URIs (or other interesting dimensions) have errors, redirects or excessive "access denieds".

Rows
- cs-uri-stem
Columns
- sc-status
Values
- time

Conclusion

Using custom tools for file based logs seems to be bygones. PowerQuery can do most popular serialization formats, and within Excel limits it's awesome combined with pivot tables. If the performance isn't good enough, the same techniques can be used with SQL Server, Analysis Services and Power BI, but I guess W3C log format parsing is unnecessary in those kinds of setups.

I find myself using PowerQuery for more and more these days, so don't be surprised if another PowerQuery post sees the light of day this year. Hope you enjoyed following along and maybe even learned something. 🤓

ARMLinker 1.0.0 released

Wed, 22 Jan 2020 23:22:41 Z

ARM What?

I've been having fun with ARM Templates the last couple of months. It's a wonderful way to keep your Azure Resource definitions in source control. Not to mention being able to parameterize deployment to different environments, and not least keeping your secrets neatly tucked away in a vault.

However, compiling a set of resources from multiple files currently requires you to put your templates online. I want to keep most of our customer products' templates private, and to do that one have to jump through hoops to copy the files over to a storage account and link to the given URLs. It kind of defeats the whole purpose for me.

So I went and created a small tool to be able to link templates locally.

How to use it

There's an installable project type for Visual Studio called "Azure Resource Group". When you create one you get a few files:

Deploy-AzureResourceGroup.ps1
azuredeploy.json
azuredeploy.parameters.json

You can stuff all of the resources you require in the azuredeploy.json file, and finally deploy them using a wizard, or run the PowerShell script in a CD pipeline.

By installing ARMLinker you can start running the tool to link other JSON files into the main azuredeploy.json file.

install-module ARMLinker

Let's say we have a Logic App (what I've been doing).
To deploy it and its connections and other needed resources, we often want a bounch of secret keys for different APIs and such.

Here's a trimmed down sample of a Logic App that runs a SQL command:

{
    "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "Tags": {
            "type": "object",
            "defaultValue": {
                "Customer": "My customer",
                "Product": "Their Logic App",
                "Environment": "Beta"
            }
        },
        "SQL-Server": {
            "defaultValue": "some.database.windows.net",
            "type": "string"
        },
        "SQL-User": {
            "defaultValue": "appuser",
            "type": "string"
        },
        "SQL-Password": {
            "defaultValue": "",
            "type": "securestring"
        },
        "SQL-Database-Name": {
            "defaultValue": "beta-database",
            "type": "string"
        }
    },
    "variables": {
        "ConnectionName": "[replace(concat(parameters('Tags').Customer, '-', parameters('Tags').Product, '-SQLConnection-', parameters('Tags').Environment), ' ', '')]",
        "LogicAppName": "[replace(concat(parameters('Tags').Customer, '-', parameters('Tags').Product, '-', parameters('Tags').Environment), ' ', '')]"
    },
    "resources": [
        {
            "type": "Microsoft.Web/connections",
            "apiVersion": "2016-06-01",
            "location": "westeurope",
            "name": "[variables('ConnectionName')]",
            "properties": {
                "api": {
                    "id": "[concat(subscription().id,'/providers/Microsoft.Web/locations/westeurope/managedApis/sql')]"
                },
                "displayName": "sql_connection",
                "parameterValues": {
                    "server": "[parameters('SQL-Server')]",
                    "database": "[parameters('SQL-Database-Name')]",
                    "authType": "windows",
                    "username": "[parameters('SQL-User')]",
                    "password": "[parameters('SQL-Password')]"
                }
            }
        }, 
        {
            "type": "Microsoft.Logic/workflows",
            "apiVersion": "2017-07-01",
            "name": "[variables('LogicAppName')]",
            "dependsOn": [ "[resourceId('Microsoft.Web/connections', variables('ConnectionName'))]" ], 
            "location": "westeurope",
            "properties": {
                "state": "Enabled",
                "definition": {
                    "$schema": "https://schema.management.azure.com/providers/Microsoft.Logic/schemas/2016-06-01/workflowdefinition.json#",
                    "contentVersion": "1.0.0.0",
                    "parameters": {
                        "$connections": {
                            "defaultValue": {},
                            "type": "Object"
                        },
                        "SQL-Server": {
                            "defaultValue": "",
                            "type": "string"
                        },
                        "SQL-Database-Name": {
                            "defaultValue": "",
                            "type": "string" 
                        }
                    },
                    "triggers": {
                        "Recurrence": {
                            "recurrence": {
                                "frequency": "Day",
                                "interval": 1
                            },
                            "type": "Recurrence"
                        }
                    },
                    "actions": {
                        "Execute_a_SQL_query_(V2)": {
                            "runAfter": {},
                            "type": "ApiConnection",
                            "inputs": {
                                "body": {
                                    "query": "select 'do something really useful' as task"
                                },
                                "host": {
                                    "connection": {
                                        "name": "@parameters('$connections')['sql']['connectionId']"
                                    }
                                },
                                "method": "post",
                                "path": "/v2/datasets/@{encodeURIComponent(encodeURIComponent(parameters('SQL-Server')))},@{encodeURIComponent(encodeURIComponent(parameters('SQL-Database-Name')))}/query/sql"
                            }
                        }
                    },
                    "outputs": {}
                },
                "parameters": {
                    "$connections": {
                        "value": {
                            "sql": {
                                "connectionId": "[resourceId('Microsoft.Web/connections', variables('ConnectionName'))]",
                                "connectionName": "variables('ConnectionName')",
                                "id": "/subscriptions/00000000-0000-0000-0000-000000000000/providers/Microsoft.Web/locations/westeurope/managedApis/sql"
                            }
                        }
                    },
                    "SQL-Server": {
                        "value": "[parameters('SQL-Server')]"
                    },
                    "SQL-Database-Name": {
                        "value": "[parameters('SQL-Database-Name')]"
                    }
                }
            }
        }
    ]
}

The parameters here are ARM template parameters. The most interesting one is the secret password for the database server. It's secret, so it's not supposed to live in our parameter file or source control. We've also got the ID of the connection, which will be the real ID in the actual deployed Logic App.

There's a fancy way to go about keeping the password in a key vault on Azure, and the Visual Studio Wizard is really helpful with putting it into a vault.

When we're done and ready for production, a parameter file may look like this:

{
    "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "Tags": {
            "value": {
                "Customer": "My customer",
                "Product": "Their Logic App",
                "Environment": "Production"
            }
        },
        "SQL-Database-Name": {
            "value": "production-database"
        },
        "SQL-Password": {
            "reference": {
                "keyVault": {
                    "id": "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/Vault-Group/providers/Microsoft.KeyVault/vaults/OurKeyVault"
                },
                "secretName": "CustomerSQLPassword"
            }
        }
    }
}

One of the beauties of using Logic Apps is that it have this nice GUI to work with in the portal. There's also an extension for Visual Studio to be able to edit them within Visual Studio.

However, the definition will look like this when viewed with the code editor. (I removed the bulk of it, but notice the parameters)

{
    "definition": {
        "$schema": "https://schema.management.azure.com/providers/Microsoft.Logic/schemas/2016-06-01/workflowdefinition.json#",
        "actions": {
            "Execute_a_SQL_query_(V2)": {
                "inputs": {
                    "body": {
                        "query": "select 'do something really useful' as task"
                    },
                    "host": {
                        "..."
                    },
                    "..."
                },
                "runAfter": {},
                "type": "ApiConnection"
            }
        },
        "...",
        "parameters": {
            "$connections": {
                "defaultValue": {},
                "type": "Object"
            },
            "..."
        },
        "triggers": {
            "..."
        }
    },
    "parameters": {
        "$connections": {
            "value": {
                "sql": {
                    "connectionId": "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/CustomerResourceGroup/providers/Microsoft.Web/connections/MyCustomer-TheirProduct-SQLConnection-Prod",
                    "connectionName": "MyCustomer-TheirProduct-SQLConnection-Prod",
                    "id": "/subscriptions/00000000-0000-0000-0000-000000000000/providers/Microsoft.Web/locations/westeurope/managedApis/sql"
                }
            }
        },
        "SQL-Database-Name": {
            "value": "production-database"
        },
        "SQL-Server": {
            "value": "some.database.windows.net"
        }
    }
}

Notice that the parameters are all filled out. We can't copy this into our ARM template since it's all real Resource ID references.

There's another way to get only the definition. We can use the Az.LogicApp powershell module:

(get-azlogicapp -resourcegroupname CustomerResourceGroup -name mycustomer-theirproduct-prod).definition.ToString()

It will give us only the definition part of the template.

Both gives us a means to put only the definition of the logic app into a file in our local project.

Now we can go back to the ARM template and replace the definition with a simple link to the file. Say we Set-Content the result of the statement above into a file called "logicapp.json". We can modify the ARM template as such:

{
    "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "..."
    },
    "variables": {
        "..."
    },
    "resources": [
        {
            "type": "Microsoft.Web/connections",
            "..."
        }, 
        {
            "type": "Microsoft.Logic/workflows",
            "apiVersion": "2017-07-01",
            "name": "[variables('LogicAppName')]",
            "dependsOn": [ "[resourceId('Microsoft.Web/connections', variables('ConnectionName'))]" ], 
            "location": "westeurope",
            "properties": {
                "state": "Enabled",
                "definition": {
                    "templateLink" {
                        "uri": ".\logicapp.json"
                    }
                },
                "parameters": {
                    "$connections": {
                        "value": {
                            "sql": {
                                "connectionId": "[resourceId('Microsoft.Web/connections', variables('ConnectionName'))]",
                            "connectionName": "variables('ConnectionName')",
                                "id": "/subscriptions/00000000-0000-0000-0000-000000000000/providers/Microsoft.Web/locations/westeurope/managedApis/sql"
                            }
                        }
                    },
                    "..."
                }
            }
        }
    ]
}

By running ARMLinker we will have the same generated file as we started with, but we can use the GUI for the logic app and easily fetch the new JSON for it.

Convert-TemplateLinks azuredeploy.json azuredeploy.linked.json

For now, I've actually turned those around and put the "linked" template in a file called azuredeploy.linked.json in order to generate the "conventional" azuredeploy.json file.

More options

When using the "copy content from the editor" method mentioned above, we have to make sure to copy only the definition object. Otherwise we'll bring the concrete parameters into the local file.

Do not despair!

There's another option that doesn't match the official schema for "templateLink". By adding a property called "jsonPath" we can point to an object deeper in the linked file. Say we copy the content from the online editor.

We can modify the linked template as such:

"definition": {
    "templateLink" {
        "uri": ".\logicapp.json",
        "jsonPath": "definition"
    }
},

It will now only merge the definition part from the logicapp.json file.

I only implemented dot separated paths for now, so exotic paths to arrays or paths with special characters won't work.

IE. resources[0]['very fancy'].thing won't work, but things.with.dots will work.

Plans and dreams

This is pretty much only a workaround while waiting for Microsoft to realise this is totally useful and obvious.

I originally intended it to be a Custom Tool for Visual Studio, but I could not for the life of me figure out how to enable Custom Tools in projects not of the C# or Visual Basic archetypes.

If anyone picks up on it, I'll happily discuss new features and even happierly receive meaningful pull requests.

Other than that, I believe it does the job properly. It can be used in CD pipelines. It should even work for any JSON, not necessarily ARM templates.

I would really appreciate your feedback, and hope you like it!

Now go commit and deploy something automagically while fetching coffee! 🤘😁🦄

Code and gallery links

Github repository
PowerShell gallery