Azure Tagging with Powershell Example

Azure uses Tagging for easier management. Tagging is done with Key value pairs. You can create the manually in the Azure Portal but you can also do it programatically via Powershell (For Example)

For this example Log onto the Azure Portal, Open Cloud Shell (This is the new Azure Portal, Not classic)

And set it to run with powershell

Just to get warmed up, lets list all our Resource groups

Get-AzResourceGroup

You can right click in powershell and paste the code above. then click Enter

This gives you a list of all your resource Groups


You even get a list of tags for each resource group with the above command

However, the resource groups that are available to me currently are the ones on my personal visual studio subscription. the Current Subscription

Lets see what subscriptions we have available

Get-AzureRMSubscription

You get the Subscription Name, ID and TenantID

Lets find out the current Subscription Context

Get-AzureRmContext

You can also use the following command to get subscription details

(Get-AzureRmContext).Subscription

Now we want to change the current Subscription context. this example is going to look at a resource group and a resource in our Proof Of Concept Subscription

Set-AzContext -SubscriptionId "xxxx-xxxx-xxxx-xxxx"

The ID can be generated by using Get-AzureRMSubscription

And running Get-AzureRmContext again, provides you with confirmation that you have changed subscriptions

Now lets have a look at all the resource groups within this subscription with Get-AzResourceGroup again


These are the two resource groups we are interested in. POC has no tags and training Resources have a couple of tags at resource group Level.

Get-AzureRmResource | ft

This scrips allows you to get a list of all the resources and Resource groups within the current subscription

What I want to do now is check all the tags on the resource Group

(Get-AzResource -ResourceGroupName POC_Resource_Group).Tags

This function allows us to see every tag at resource level against the Resource group. So this script doesnt quite give us what we want. We know that there are no tags at resource group level. How do we establish this with Powershell?

$resourceGroup = Get-AzResourceGroup -Name POC_Resource_Group  
Get-AzTag -ResourceId $resourceGroup.ResourceId

This script allows us so see the tags on the Resource Group.

It would now be useful to get a list of Resources in the resource Group

Get-AzureRmResource -ResourceGroupName POC_Resource_Group | ft

So now we can see every resource in the Resource group

Next step is to have a look at tags per resource

(Get-AzureRmResource -Name "AdventureWorksDW").Tags

We know what our Resources are called now (See Above) so we can get specific tags of a resource group

Scripts to Add tags to a Resource Group

It would be really good to have the following tag at resource group level

  • businessProcess, Proof of Concept
  • environmentType, Dev
  • client, na
  • billingCategory, Internal Analytics

Lets have a go with powershell

$tags = @{"businessProcesst"="Proof of Concept"; "environmentType"="Dev"; "client"="na"; "billingCategory"="Internal Analytics"}
$resourceGroup = Get-AzResourceGroup -Name POC_Resource_GroupNew-AzTag -ResourceId 
$resourceGroup.ResourceId -tag $tags

The resource group now has Tags

Scripts to add tags to a resource

Completely update the List of Tags

We also want to add a tag to a resource.

If you were to use the following script

$tags = @{"costCentre"="000000"} $resource = Get-AzResource -Name AdventureWorksDW -ResourceGroup POC_Resource_Group  
New-AzTag -ResourceId $resource.id -Tag $tags 

And then run (Get-AzureRmResource -Name “AdventureWorksDW”).Tags to look at the tags, this script deletes all the tags and inserts just the one, this isn’t the logic we want to use. We want to Add tags to a Resource that already has tags

Add Tags to a resource that already has tags

$resource = Get-AzResource -Name AdventureWorksDW -ResourceGroup POC_Resource_Group 
$tags = @{"costCentre"="000000"} 
Update-AzTag -ResourceId $resource.id -Tag $tags -Operation Merge

Instead of New, Update has been used. and we gave the script a resource name and a resource group

(Get-AzureRmResource -Name “AdventureWorksDW”).Tags shows that the new tag has been added to the previous tags

Amend a Tag

After running (Get-AzureRmResource -Name “AdventureWorksDW”).Tags, adventureworkks is a value that has been inputted incorrectly and needs amending.

$resource = Get-AzResource -Name AdventureWorksDW -ResourceGroup POC_Resource_Group 
$tags = @{"applicationName"="adventureworks"} 
Update-AzTag -ResourceId $resource.id -Tag $tags -Operation Merge

Great. The incorrect value has been updated

Get Tags

a couple more really useful Scripts in regards to tags

Get-AzureRmTag

This script allows you to see all your tags and how many times they are used within the subscription

From the above script we can see that businessProcesst is a tag that has been added which needs amending to businessProcess (Its been incorrectly typed in)

Get more information about the Tag (Tag Values and Count of Use)

Get-AzureRmTag -Name "environmentType"

This script is great for looking at the values in a specific tag. And in this case we want to know more about businessProcesst

Get-AzureRmTag -Name "businessProcesst"

And to really tie it down we want to know what resource or resource Group it belongs too?

You can run the following Script to get the Specific resource Group for the tag

(Get-AzResourcegroup -Tag @{ “businessProcesst"="Proof of Concept"}).ResourceGroupName

Its in the Proof of Concept Resource group so all that is needed is to delete and recreate

And this Script checks if its against any resources

(Get-AzResource -Tag @{ “businessProcesst"="Proof of Concept"}).Name

In this case No

Remove-AzureRmTag -Name "businessProcesst"

Although we have checked, the error message stills says that the Tag is being referenced by resources.

It may be easier to update the Tag in Azure rather than using Code. there doesnt appear to be a way of changing the tag name in poweshell (Or at least I haven’t found it yet)

You can go to tags. Click on the Tag. Then click on … at the end of the tag Row and choose Edit tags.

Then Save

This just gives a flavor of how to use powershell to manage tags. This information can be saved into Scripts so we can reproduce the creation of tags if required.

Tags applied to resource Groups don’t get added to the Resources below. Powershell gives you move control over making sure the Resources and resources Groups have the correct tags applied

Introduction to Azure Tagging

Tagging is a feature that has been introduced into the Azure Resource Manager model (ARM). They can be used to Logically group and track resources. The old (Classic) version of Azure was Azure Service Manager.

Azure tagging can be done using Azure Portal, Azure Powershell, CLI (Command Line User Interface) or ARM (Azure Resource Manager) JSON templates

Tags can then be used to select resources or Resource Groups and are useful when you need to organize resources for billing or management

You apply tags to your Azure resources giving metadata to logically organize them into a taxonomy.

Each tag consists of a name and a value pair. For example, you can apply the name “Environment” and the value “Production” to all the resources in production.

Key Points

  • You can only apply tags to resources that support Resource Manager operations
    • VMs, Virtual Networks and Storage created through the classic deployment model must be re-deployed through Resource Manager to support tagging
    • A good way around this is to tag the resource group they belong to instead.
  • All ARM resources support tagging
  • Each resource or resource group can have a maximum of 15 tags.
  • Tags are key/value pairs, name is limited to 512 characters, value is limited to 256 characters
  • Tags are free-form text so consistent correct spelling is very important
  • Tags defined on Resource Groups exist only on the group object and do not flow down to the resources under them
    • Through the relationship you can easily find resource by filtering by tagged resource group Its recommended keeping the tags to the resource group unless they are resource specific.
  • Each tag is automatically added to the subscription-wide taxonomy
    • Application or resource specific tags will “pollute” the tag list for the entire subscription.

Issues with Tags

Using the recommended Tag procedure of tagging at resource group level causes issues because the Tags dont get inherited at Resource level.

The hope was that any tags that you apply at one level of the hierarchy will be inherited by the lower levels within the hierarchy and this doesnt happen.

You need to be careful that your tags stay Logical and don’t differ from higher levels. It may well be preferable to do this via Powershell Scrips that manually to ensure correct logic is maintained between resources and Resource Groups.

Resource Groups

  • The underlying technology that powers resource groups is the Azure Resource Manager (ARM).
  • ARM was built by Microsoft in response to the shortcomings of the old Azure Service Manager (ASM)
  • ARM requires that resources be placed in resource groups, which allows for logical grouping of related resources.
  • Although creating a resource group requires specifying a region for it to be stored in, the resources in that resource group could span multiple regions.
  • In the ARM architecture, resource groups not only become units of deployment, but also units of management of related resources.
  • It allows role-based access control (RBAC) at the resource group level, making it much easier to manage user access to the resources in the group.
  • When users log into the Azure Portal, they will only see resource groups they have access to and not others within the subscription. u Administrators will still be able to assign access control for users to individual resources within the resource group based on their roles. This is great to see costs associated with Each Resource Group

Successful Azure Resource Groups

If an application requires different resources that need to be updated together, such as having a SQL database, a web app, a mobile app, etc. then it makes sense to group these resource in the same resource group.

Use different resource groups for dev/test, staging, or production, as the resources in these groups have different lifecycles.

All the resources in the group should share the same environment (Dev, Test etc) because you deploy, update and delete together

If you have for example the marketing analytics database in one Resource Group and a demo database in another resource group, Each resource group needs its own server

You cant Rename a resource Group

A good naming convention to use is rg-projectorapp-subprojectorapp-dev or projectorapp-subprojectorapp-dev-rg

Examples of Resource groups

Adding Tags to a VM resource within a Resource Group

Tagging Examples

Now you have logical resource Groups set up we can set up tags which are good for larger organisations. 

Business Tags:

  • Cost centre
  • Responsible Person or Party
  • Application Name
  • Environment
    • Development
    • Testing
    • Staging
    • Production

Security Tags:

  • Data Profile

Automation Tags

  • Power Off
  • Maintenance Window
KeyExampleCommentType
costCenter12345This is your internal billing codeBusiness
managedBydebbie@peak.co.ukName or email addressBusiness
applicationNamemyappname of the Project Business
environment<production, Staging, QA>Identifies the environmentBusiness
dataProfile<Public, Confidential, Restricted, Internal>Data Sensitivity. Public: This information is public information, and can be openly shared on your website
Internal: Internal information is company-wide and should be protected with limited controls.
Confidential: Confidential information is team-wide and its use should be contained within the business.
Restricted: Restricted information is highly sensitive and its use should be limited on a need-to-know basis.
Security
powerOffyes, noCan this resource be shut down at 7pmAutomation
Example of a Subscription with Tags on the Resource groups.

Looking at this example, the Tagging has been added to the resource Groups. However if you look at a resource, you wont see the tags.

We need to look at a way of ensuring that the Tags applied to the resource group are also applied for each resource.

Adding tags to a resource in Azure Portal

Policies for Tags

  • Azure Policy is a service in Azure that you use to create, assign and, manage policies.
  • These policies enforce different rules and effects over your resources, so those resources stay compliant with your corporate standards and service level agreements https://docs.microsoft.com/en-us/azure/governance/policy/samples/apply-tag-default-value
  • Microsoft provides sample JSON scripts to create the tagging policies
  • JSON scripts can be deployed in the Azure Portal, Azure Powershell, Azure CLI or REST API.
  • When you create resource groups and Resources you should always apply tags and the default value for that tag.
  • There are Billing tags policy initiatives you can apply
  • You can also enforce tags and value via Policies on resource groups or Resources

https://github.com/Azure/azure-policy/tree/master/samples/built-in-policy/apply-default-tag-value

https://github.com/Azure/azure-policy/tree/master/samples/built-in-policy/enforce-tag-value

Locks

As an administrator, you may need to lock a subscription, resource group, or resource to prevent other users in your organization from accidentally deleting or modifying critical resources.

You can set the lock level to CanNotDelete or ReadOnly. In the portal, the locks are called Delete and Read-only respectively. it may be useful to add a tag for this

LockLevel <CanNotDelete, ReadOnly, NA>

Conclusion

We will look at tagging in more detail in other posts. What Policies you can apply. tagging via Powershell, CLI and ARM JSON Templates. How to manage and enforce good Tagging Logic.

We will also look at ways to use your tags

Links

Power BI Deployment Approaches When Dealing With A Large Scale Organisation

When you start working with Power BI at a smaller scale it can be fairly simple.

  • You can find your data sources
  • Import them into Power BI.
  • Do some work on the data in Power Query Editor.
  • Create some DAX to create measures
  • Add visuals into your report.
  • Publish the Reports and add dashboards.
  • Then Share them to Other Power BI Pro Users (Or free licence Users if you have Premium but at a smaller scale its very doubtful you would have Power BI Premium due to the costs.

However when you are dealing with a larger organisation there are other consideration. the last thing you want is a sprawling confusion on Power BI Reports and Dashboards, shared to anyone within the organisation (And possibly externally if external users have been added as guests)

Some reports will be really useful, others wont be used at all. Workspaces are being created everywhere and no governance is being applied to the entire set up.

This is where we need to start applying some governance and control. The first thing to understand is which Power BI Set up are you using?

If you are still at user Licence level then everyone will be working on Power BI Pro licenses, whether Contributor or consumer.

If your organisation is large, it will probably be working with a Power BI Premium capacity licence. Therefore if Workspaces are published in Premium, anyone with a Free Power BI Licence in your company can access Power BI content as a Reader

Anyone Contributing as a Developer will still need a Power BI Pro License.

There are three main deployment approaches to be aware of in a large scale environment

Business Led Self Service

This is where uses within the business teams lead on their own Power BI content

users create their own Reports, Dashboards and Datasets and add users when they want to share their content.

This is normally done by sharing at the User level

This is a great way of finding out who your Power Users are. They are the ones creating content and asking questions. get these people involved at all levels and you are on your way to a great adoption roadmap.

IT Led Self Service

This is where the BI team (IT) Create the datasets (Including all the DAX) for the business users. These users can then create all the Visualisations and reports over the data set

We can also now separate the data transformations (Dataflows) from the DAX (DataSets) which gives further opportunity to share work.

  • There will only be one owner of the dataflow and data set. The person who created it
  • because this is IT Led the data sets can be promoted (Or certified which is the highest level of attainment)
  • Promotion and certification is great for data discovery.
  • Data sets can be used by users (With Access) and Published to other workspaces
  • Users can also create their own DAX in their own published datasets.
  • Potentially, if this happens, the DAX, when proved useful should be added by the BI team into the original dataset.

At this level, the assumption is that access to the App Workspace and App with the different Power BI roles is managed as Azure AD Group Level

This means that we can centrally keep control of who has access to the App Workspaces and what Power BI roles are assigned

Certified

Only a small group of users should have certification rights, within the Power BI team. certification is the one we should always be working towards and you should have documentation that establishes the route to certification.

This means that you have control and your Small team of Power BI reviewers can check everything over and then change from promoted to certified

Promoted

If you own the data set you can promote it, to establish wider use of the data set. This can be done by BI Team members if IT Led self Service and BI team and Business Users with Pro Development License

Corporate power BI

Everything is created by the BI team centrally and consumed by the business users.

This is a great approach when you have to prioritise work and need to move fast. Its great when your report consumers possibly don’t have the skill sets to set up reports and dashboards.

Its also the best approach to take content that has been created Business led or IT Led and move it centrally, with all the security standards and governance in place. Only using the Corporate approach tends to create bottlenecks for business teams because of the longer set up period. it also disenfranchises

Which Deployment Approach to go for

Its always good to start of with the Corporate deployment Approach for your centralised analytics. Then have proper controls and governance in place to enable IT Led and Business Led deployment approaches

We need to aim for the Blended Approach which is IT Managed Self Service BI

So Ideally, you are looking at blending all three approaches. So long as you understand which approach you have used.

You still want users to get involve and want to create Power BI reports. You also want them to engage enough to want them to become certified and moved into central control when the content has proved important and useable.

Moving into the IT Managed model does involve more work. Imagine there is a self service App Workspace that has proved incredibly popular. Time to congratulate Power Users who pulled everything together and make sure they understand that moving into into Managed BI doesn’t mean they cant continue to work on their ideas and new content. Moving the content involves:

  • Checking that the data sources are good enough to work with.
  • Possibly updating the underlying Data Warehouse or Data Mart.
  • If the data is in excel spreadsheets, or text files, making sure this is managed.
  • There may be a lot of work done within the dataflow on data transformations. Some of this could be moved to the data source, freeing up Power BI resources.
  • To get the data set certified you may need extra involvement from BI team members to review the data set and verify it as acceptable for certification.

An Example Approach in Power BI

We are in an organisation called Adventureworks.

Within this We have various App Workspaces. All so far are Self Service. There has been no control over the process so far and there are three people with power BI Pro licenses (Imagine if this involved more Pro users. Some creating Separate Workspaces. Some reusing Workspaces)

Within these app workspace, users have been creating content, Reports, Dashboards, Datasets and dataflows

  • All three Workspaces have used the same data source
  • Adventureworks contains the most up to date data set and also contains dataflows
  • AT and JL workspaces contain similar information so on the whole there are three fairly identical separate data sets
  • For every workspace, users have been added, rather than user groups
  • Basically, we now have practically the same data source three times in Power BI, when there should only be one.

Bronze Silver and Gold Report template

One approach is to use the Bronze Silver and Gold Approach either via a template or a badge visual you can add to your reports

Bronze if you are doing Business led Self Service

Silver if you are doing IT led Self Service and you are using a dataset to create your own reports

Gold if its been fully checked and managed by the Central BI team. The Data set can be certified to use for Silver projects and all the reports and dashboards have been tested and have been proved as useful to the company.

The Process to entangle what has gone before

There is lots of work to do on the example above.

First of all we need to identify what type of approach have been used. For Jess and Amy, we can say that the Business led Self Service approach has been used. Each report and dashboard can be branded with the Bronze badge for business led.

For the Adventureworks we can again say that the Business led Self Service approach has been used. Dataflows within the service have been created and the data set has been promoted by the person creating the reports and dashboards but it was all done within the business.

In all three cases the user has created all the items for their reports and dashboards. therefore all reports and dashboards can be branded with Bronze.

So far we cant say that we have moved to the IT Managed blended approach, and its clear that users have been added ad hoc without much planning

Step 1. Identify if there is anything in Amy and Jess’s dataflow within Power BI that needs adding to the dataflow within Adventureworks. Then amend check off as complete.

Step 2. Identify any missing DAX Measures or calculated columns. Any identical measures with different names. Any data items that have been renamed between data sets. Any calculated Columns that can be moved into the dataflow. Ensure everything is added to the main dataset in Adventureworks and ensure the users of the other workspaces know what is happening

Step 3. Identify the differences between the reports. Hopefully each person has been working on different areas, but it may be that the reports need bringing together. This will take time. A report should never answer the same question twice.

Step 4. Amend appropriately. At the very least, all three reports should be using the Power BI Dataset that has been promoted as a usable source

Step 5. All these steps have led to a more IT managed deployment so we can add a silver badge to these reports.

Step 6. An extra review to check the dataflow and data sets within the central BI team. Also ensure that we aren’t duplicating work within the reports and dashboards, and this becomes Corporate Power BI, managed centrally. Our power users still have control and the ability to make changes. However everything is now more managed. We can now add a Gold badge to these reports and Certify the data sets

Step 7. Identify the users and the Power BI role levels they should have.

Create Groups within Azure AD for each Power BI Role and assign the users to these groups. This will mean that we are handling the contributors and readers centrally within Azure

I got lots of great help trying to put together an idea in my head. https://community.powerbi.com/t5/Service/Logic-with-dealing-with-Self-Service-and-Corporate-power-BI/m-p/1030137#M93044

Thanks so much to nickyvv from the Power BI Forums and Line Krogh providing some great info on Bronze Silver and Gold templates.

Azure keeping your costs under control with Spending Limits and Budgets

We can now start adding Subscriptions for all our different Business domains and environments. Azure allows you to quickly get started adding Resource groups and Resources to your subscriptions. Using Management groups above subscriptions allows for even more control. You can manage access, policy, and compliance across multiple subscriptions.

This is all great but what about controlling costs? You will very likely have team members that cant wait to try resources to see what they can do but what happens when you came from after a weekend and you test Cosmos DB has racked up a huge amount of money?

We want to be able to control things more so lets look at some options.

Azure Spending Limit

The following Subscriptions have the ability to apply Azure Spending Limits

  • 0044P Free trial
  • 0059P Visual Studio Professional subscribers
  • 0060P Visual Studio Test Professional subscribers
  • 0062P MSDN Platforms subscribers
  • 0063P Visual Studio Enterprise subscribers
  • 0064P Visual Studio Enterprise (BizSpark) subscribers
  • 0029P Visual Studio Enterprise (MPN) subscribers
  • 0025P Action Pack
  • 0111p Azure in Open Licensing
  • 0170p Azure for Students
  • 0144P Microsoft Azure for Students Starter

When you get a free account, or are for example a Visual Studio Enterprise Subscriber you will get personal use credits.

for example. Going to a personal Subscription and looking at the overview

This Visual Studio Enterprise Subscription allows the user £115 Credits. You cant increase set credits.

If you reach your spending Limit, all services are disabled until the next billing period.

If you need to uncap you need to remove the spending limit by clicking on the above message.


Remove the Spending Limit for the current billing period will allow you to continue and the credits will then reset.

Remove the Spending Limit indefinitely will allow you to go over your Spending limit every month by adding your credit card details (Pay as you go)

However the recommendation is to only remove if required to keep more control

This is obviously a great choice for personal Azure Subscriptions, used for testing and trying things out.

Monitor Costs When using Azure Services

You cant cap the following subscriptions

  • 0136P Microsoft Azure EA Sponsorship
  • 0003P Pay-as-you-go
  • 0023P Pay-as-you-go Dev/Test
  • 0148P Enterprise Dev/Test
  • 0036P Microsoft Azure Sponsored Offer
  • 0243P Azure Pass
  • 0145P Azure in CSP

Track Costs with Budgets

Budgets allow you to set thresholds for your spending. You can then get alerts on these thresholds.

If for example, you have a budget for £500 and a threshold for £400 you could also start adding in automation to start shutting down resources, like VMs at the threshold.

Setting the Budget at the Subscription Level

Within the Subscription Overview

Click Next

Setting Alerts

Application Insights Smart Detection has been used as the action Group if the costs go up to 90% of the budget that has been set

‘Smart Detection automatically warns you of potential performance problems and failure anomalies in your web application’

Then Create

Targeting a Resource with a Budget

Again going to a Subscription and Clicking Cost Analysis

The resource that has a habit of becoming expensive is the database in this subscription

We want to make sure it doesn’t get out of control

Create a new budget for this subscription and then add a filter (the filter will only work once the graph is displayed

Resource Type is selected

then Microsoft SQL Servers. You could also put tags to good use in Budgets. You could for example check for tags of resources created by specific people.

You can now have Budgets that specifically check on certain services. Great if you have Services that have a tendency to create lots of costs.

So now we can rest a little easier knowing that Azure is going to let us know what is happening. It would be great if we could apply automation to shut something down if it was accumulating costs over the threshold. We can look at this next time