Lars-Erik's blog

ARMLinker 1.0.0 released

Wed, 22 Jan 2020 23:22:41 Z

ARM What?

I've been having fun with ARM Templates the last couple of months. It's a wonderful way to keep your Azure Resource definitions in source control. Not to mention being able to parameterize deployment to different environments, and not least keeping your secrets neatly tucked away in a vault.

However, compiling a set of resources from multiple files currently requires you to put your templates online. I want to keep most of our customer products' templates private, and to do that one have to jump through hoops to copy the files over to a storage account and link to the given URLs. It kind of defeats the whole purpose for me.

So I went and created a small tool to be able to link templates locally.

How to use it

There's an installable project type for Visual Studio called "Azure Resource Group". When you create one you get a few files:

Deploy-AzureResourceGroup.ps1
azuredeploy.json
azuredeploy.parameters.json

You can stuff all of the resources you require in the azuredeploy.json file, and finally deploy them using a wizard, or run the PowerShell script in a CD pipeline.

By installing ARMLinker you can start running the tool to link other JSON files into the main azuredeploy.json file.

install-module ARMLinker

Let's say we have a Logic App (what I've been doing).
To deploy it and its connections and other needed resources, we often want a bounch of secret keys for different APIs and such.

Here's a trimmed down sample of a Logic App that runs a SQL command:

{
    "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "Tags": {
            "type": "object",
            "defaultValue": {
                "Customer": "My customer",
                "Product": "Their Logic App",
                "Environment": "Beta"
            }
        },
        "SQL-Server": {
            "defaultValue": "some.database.windows.net",
            "type": "string"
        },
        "SQL-User": {
            "defaultValue": "appuser",
            "type": "string"
        },
        "SQL-Password": {
            "defaultValue": "",
            "type": "securestring"
        },
        "SQL-Database-Name": {
            "defaultValue": "beta-database",
            "type": "string"
        }
    },
    "variables": {
        "ConnectionName": "[replace(concat(parameters('Tags').Customer, '-', parameters('Tags').Product, '-SQLConnection-', parameters('Tags').Environment), ' ', '')]",
        "LogicAppName": "[replace(concat(parameters('Tags').Customer, '-', parameters('Tags').Product, '-', parameters('Tags').Environment), ' ', '')]"
    },
    "resources": [
        {
            "type": "Microsoft.Web/connections",
            "apiVersion": "2016-06-01",
            "location": "westeurope",
            "name": "[variables('ConnectionName')]",
            "properties": {
                "api": {
                    "id": "[concat(subscription().id,'/providers/Microsoft.Web/locations/westeurope/managedApis/sql')]"
                },
                "displayName": "sql_connection",
                "parameterValues": {
                    "server": "[parameters('SQL-Server')]",
                    "database": "[parameters('SQL-Database-Name')]",
                    "authType": "windows",
                    "username": "[parameters('SQL-User')]",
                    "password": "[parameters('SQL-Password')]"
                }
            }
        }, 
        {
            "type": "Microsoft.Logic/workflows",
            "apiVersion": "2017-07-01",
            "name": "[variables('LogicAppName')]",
            "dependsOn": [ "[resourceId('Microsoft.Web/connections', variables('ConnectionName'))]" ], 
            "location": "westeurope",
            "properties": {
                "state": "Enabled",
                "definition": {
                    "$schema": "https://schema.management.azure.com/providers/Microsoft.Logic/schemas/2016-06-01/workflowdefinition.json#",
                    "contentVersion": "1.0.0.0",
                    "parameters": {
                        "$connections": {
                            "defaultValue": {},
                            "type": "Object"
                        },
                        "SQL-Server": {
                            "defaultValue": "",
                            "type": "string"
                        },
                        "SQL-Database-Name": {
                            "defaultValue": "",
                            "type": "string" 
                        }
                    },
                    "triggers": {
                        "Recurrence": {
                            "recurrence": {
                                "frequency": "Day",
                                "interval": 1
                            },
                            "type": "Recurrence"
                        }
                    },
                    "actions": {
                        "Execute_a_SQL_query_(V2)": {
                            "runAfter": {},
                            "type": "ApiConnection",
                            "inputs": {
                                "body": {
                                    "query": "select 'do something really useful' as task"
                                },
                                "host": {
                                    "connection": {
                                        "name": "@parameters('$connections')['sql']['connectionId']"
                                    }
                                },
                                "method": "post",
                                "path": "/v2/datasets/@{encodeURIComponent(encodeURIComponent(parameters('SQL-Server')))},@{encodeURIComponent(encodeURIComponent(parameters('SQL-Database-Name')))}/query/sql"
                            }
                        }
                    },
                    "outputs": {}
                },
                "parameters": {
                    "$connections": {
                        "value": {
                            "sql": {
                                "connectionId": "[resourceId('Microsoft.Web/connections', variables('ConnectionName'))]",
                                "connectionName": "variables('ConnectionName')",
                                "id": "/subscriptions/00000000-0000-0000-0000-000000000000/providers/Microsoft.Web/locations/westeurope/managedApis/sql"
                            }
                        }
                    },
                    "SQL-Server": {
                        "value": "[parameters('SQL-Server')]"
                    },
                    "SQL-Database-Name": {
                        "value": "[parameters('SQL-Database-Name')]"
                    }
                }
            }
        }
    ]
}

The parameters here are ARM template parameters. The most interesting one is the secret password for the database server. It's secret, so it's not supposed to live in our parameter file or source control. We've also got the ID of the connection, which will be the real ID in the actual deployed Logic App.

There's a fancy way to go about keeping the password in a key vault on Azure, and the Visual Studio Wizard is really helpful with putting it into a vault.

When we're done and ready for production, a parameter file may look like this:

{
    "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "Tags": {
            "value": {
                "Customer": "My customer",
                "Product": "Their Logic App",
                "Environment": "Production"
            }
        },
        "SQL-Database-Name": {
            "value": "production-database"
        },
        "SQL-Password": {
            "reference": {
                "keyVault": {
                    "id": "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/Vault-Group/providers/Microsoft.KeyVault/vaults/OurKeyVault"
                },
                "secretName": "CustomerSQLPassword"
            }
        }
    }
}

One of the beauties of using Logic Apps is that it have this nice GUI to work with in the portal. There's also an extension for Visual Studio to be able to edit them within Visual Studio.

However, the definition will look like this when viewed with the code editor. (I removed the bulk of it, but notice the parameters)

{
    "definition": {
        "$schema": "https://schema.management.azure.com/providers/Microsoft.Logic/schemas/2016-06-01/workflowdefinition.json#",
        "actions": {
            "Execute_a_SQL_query_(V2)": {
                "inputs": {
                    "body": {
                        "query": "select 'do something really useful' as task"
                    },
                    "host": {
                        "..."
                    },
                    "..."
                },
                "runAfter": {},
                "type": "ApiConnection"
            }
        },
        "...",
        "parameters": {
            "$connections": {
                "defaultValue": {},
                "type": "Object"
            },
            "..."
        },
        "triggers": {
            "..."
        }
    },
    "parameters": {
        "$connections": {
            "value": {
                "sql": {
                    "connectionId": "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/CustomerResourceGroup/providers/Microsoft.Web/connections/MyCustomer-TheirProduct-SQLConnection-Prod",
                    "connectionName": "MyCustomer-TheirProduct-SQLConnection-Prod",
                    "id": "/subscriptions/00000000-0000-0000-0000-000000000000/providers/Microsoft.Web/locations/westeurope/managedApis/sql"
                }
            }
        },
        "SQL-Database-Name": {
            "value": "production-database"
        },
        "SQL-Server": {
            "value": "some.database.windows.net"
        }
    }
}

Notice that the parameters are all filled out. We can't copy this into our ARM template since it's all real Resource ID references.

There's another way to get only the definition. We can use the Az.LogicApp powershell module:

(get-azlogicapp -resourcegroupname CustomerResourceGroup -name mycustomer-theirproduct-prod).definition.ToString()

It will give us only the definition part of the template.

Both gives us a means to put only the definition of the logic app into a file in our local project.

Now we can go back to the ARM template and replace the definition with a simple link to the file. Say we Set-Content the result of the statement above into a file called "logicapp.json". We can modify the ARM template as such:

{
    "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "..."
    },
    "variables": {
        "..."
    },
    "resources": [
        {
            "type": "Microsoft.Web/connections",
            "..."
        }, 
        {
            "type": "Microsoft.Logic/workflows",
            "apiVersion": "2017-07-01",
            "name": "[variables('LogicAppName')]",
            "dependsOn": [ "[resourceId('Microsoft.Web/connections', variables('ConnectionName'))]" ], 
            "location": "westeurope",
            "properties": {
                "state": "Enabled",
                "definition": {
                    "templateLink" {
                        "uri": ".\logicapp.json"
                    }
                },
                "parameters": {
                    "$connections": {
                        "value": {
                            "sql": {
                                "connectionId": "[resourceId('Microsoft.Web/connections', variables('ConnectionName'))]",
                            "connectionName": "variables('ConnectionName')",
                                "id": "/subscriptions/00000000-0000-0000-0000-000000000000/providers/Microsoft.Web/locations/westeurope/managedApis/sql"
                            }
                        }
                    },
                    "..."
                }
            }
        }
    ]
}

By running ARMLinker we will have the same generated file as we started with, but we can use the GUI for the logic app and easily fetch the new JSON for it.

Convert-TemplateLinks azuredeploy.json azuredeploy.linked.json

For now, I've actually turned those around and put the "linked" template in a file called azuredeploy.linked.json in order to generate the "conventional" azuredeploy.json file.

More options

When using the "copy content from the editor" method mentioned above, we have to make sure to copy only the definition object. Otherwise we'll bring the concrete parameters into the local file.

Do not despair!

There's another option that doesn't match the official schema for "templateLink". By adding a property called "jsonPath" we can point to an object deeper in the linked file. Say we copy the content from the online editor.

We can modify the linked template as such:

"definition": {
    "templateLink" {
        "uri": ".\logicapp.json",
        "jsonPath": "definition"
    }
},

It will now only merge the definition part from the logicapp.json file.

I only implemented dot separated paths for now, so exotic paths to arrays or paths with special characters won't work.

IE. resources[0]['very fancy'].thing won't work, but things.with.dots will work.

Plans and dreams

This is pretty much only a workaround while waiting for Microsoft to realise this is totally useful and obvious.

I originally intended it to be a Custom Tool for Visual Studio, but I could not for the life of me figure out how to enable Custom Tools in projects not of the C# or Visual Basic archetypes.

If anyone picks up on it, I'll happily discuss new features and even happierly receive meaningful pull requests.

Other than that, I believe it does the job properly. It can be used in CD pipelines. It should even work for any JSON, not necessarily ARM templates.

I would really appreciate your feedback, and hope you like it!

Now go commit and deploy something automagically while fetching coffee! 🤘😁🦄

Code and gallery links

Github repository
PowerShell gallery

Automating Umbraco with PowerShell

Tue, 07 Nov 2017 19:46:14 Z

Background

This particular example is for warming up a UCommerce site in staging, but the technique can be used for anything you can do in the Umbraco backoffice. Especially with the new REST APIs coming out.

In this particular case, I've sinned and not created a good test environment for the last few integration bits of a project. It was hard to tune the production behavior of some code without actually running it in production. However, it's running Umbraco 7.5.13 and UCommerce 7.7. It's probably missing the other performance fix too, but the result is that it takes quite a while to warm up everything. So we warm it up in staging and then swap slots to get it fresh, awake and blazing fast into production.

Resolve

After having done this a few times, I figured I wanted something to do while waiting. What better activity than automating the whole routine so I could do something else instead? (Like automating automation...) Here's a powershell script I ended up with to warm up everything in the backoffice. I'll go through the pieces below.

Param(
    [string]$password
)

$ws = New-Object Microsoft.PowerShell.Commands.WebRequestSession

$body = @{
    "username"="admin@admin.com"
    "password"=$password
}

$json = $body | ConvertTo-Json

Invoke-RestMethod `
    -Method Post `
    -ContentType "application/json" `
    -Uri "https://customer-staging.azurewebsites.net/umbraco/backoffice/UmbracoApi/Authentication/PostLogin" `
    -WebSession $ws `
    -Body $json

Write-Host "20%"

Invoke-RestMethod -Method Get -Uri "https://customer-staging.azurewebsites.net/umbraco/ucommerce/catalog/editcategory.aspx?id=718" -WebSession $ws

Write-Host "40%"

Invoke-RestMethod -Method Get -Uri "https://customer-staging.azurewebsites.net/umbraco/ucommerce/catalog/editproduct.aspx?id=465&parentcategoryId=718" -WebSession $ws

Write-Host "60%"
Invoke-RestMethod -Method Get -Uri "https://customer-staging.azurewebsites.net/umbraco/ucommerce/settings/orders/editpaymentmethod.aspx?id=8" -WebSession $ws

Write-Host "80%"

Invoke-RestMethod -Method Get -Uri "https://customer-staging.azurewebsites.net/umbraco/ucommerce/settings/orders/editshippingmethod.aspx?id=10" -WebSession $ws
Write-Host "100%"

The effect is that we log into Umbraco using a provided password, and instead of navigating and clicking everything, we fire a request triggering all the caching and JIT compilation for us. Even though I'm using the Invoke-RestMethod cmdlet, I can do regular web calls. The cmdlet has a sibling called Invoke-WebRequest, but the rest version is better for posting commands. It's mostly a matter of mental context, but they have a few differences.

Log into Umbraco

To set up an authorized session with Umbraco, we can call the PostLogin action. It's the same endpoint that is used from the login screen. An authorized session means that we need to get a cookie and pass it with all our requests. In order for each Invoke-RestMethod to pass this cookie, we can create a WebRequestSession we pass to each call:

$ws = New-Object Microsoft.PowerShell.Commands.WebRequestSession

Invoke-RestMethod -WebSession $ws -Uri "..."

If the response brings a cookie, it's kept in the WebRequestSession, and subsequently passed back with each new request. Just like a browser.

Then we need some JSON to pass our username and password. You can declare dictionaries of sorts in PowerShell like so:

$body = @{
    "username"="admin@admin.com"
    "password"=$password
}

And then convert it to JSON by piping it to the ConvertTo-Json cmdlet:

$json = $body | ConvertTo-Json

Finally we're ready to fire the request off to Umbraco, adding config for HTTP method, ContentType etc.:

Invoke-RestMethod `
    -Method Post `
    -ContentType "application/json" `
    -Uri "https://customer-staging.azurewebsites.net/umbraco/backoffice/UmbracoApi/Authentication/PostLogin" `
    -WebSession $ws `
    -Body $json

Automate Umbraco

For this I just needed to kick off a request to some pages, but posting messages around like rebuilding a grumpy index, running an ad hoc task, even publishing should be just as simple.

It takes a while, so I added a little status message. I'm sure PowerShell wizards would pack this stuff into better reusable parts.

Write-Host "20%"

Now reuse that WebRequestSession object to fire off new authenticated requests:

Invoke-RestMethod -WebSession $ws -Method Get -Uri "https://customer-staging.azurewebsites.net/umbraco/ucommerce/catalog/editcategory.aspx?id=718"

Do it everywhere

With all the love I can give to UCommerce, I ended up naming the script kick-ucommerce.ps1. It's like kicking your old belowed car to get it started, after you've polished it. Really!
Adding my source folder to the path environment variable makes the script available from any shell. Even the Package Manager console in Visual Studio.

Make a note that the password is a parameter. You do change it more often than you'd like to update the code right? How 'bout automating the process? ;)

Smoke test and swap the Azure slot

I'll leave the swap-slot script I run after warming up the site here too. The cool thing about warming up with the script is that it'll fail almost immediately on the Umbraco login if anything isn't like it should. So it doubles as a smoke test.

When everything looks good, I can just go:

swap-slot -from staging -to production -site customer-x

And here's the "simpleness" of that one. There are fairly good docs on all the Azure cmdlets over on Microsoft's sites. (Ask Google. ;) )

Param(
    [string]$site,
    [string]$from,
    [string]$to
)

$subscriptionId = "333112F4-4483-449C-A2DA-727E8D2E428D"
$resourcegroupname = "Common-Group"     # Might need to be param

Login-AzureRmAccount -SubscriptionId $subscriptionId

Swap-AzureRmWebAppSlot `
    -SourceSlotName $from `
    -DestinationSlotName $to `
    -Name $site `
    -ResourceGroupName $resourcegroupname

That's it! Now go automate something so you get more time to do fun stuff! :)