After 14 years using Microsoft On Premise BI Tools (SQL Server, Reporting Services, Integration Services and Analysis Services) Its time to embrace Business Intelligence in the cloud.
We have been working with Power BI Premium for a few weeks. Simply switching it on and seeing how it goes.
However, Its worth paying some attention to Autoscaling that you can get with Premium Gen 2 (Preview)
Auto Scaling
Previously our Power BI Premium capacity could struggle when there was high capacity. For example If we reached full capacity, someone’s automatic refreshes would fail. Or if too many things were happening on the server, report users would find that reports were taking longer to render.
There are lots of use cases where this could happen so Auto scaling is definitely something that can help with these issues.
You can now scale and autoscale using Azure Pay as you go, which is around £62 per vCore for a 24 hour period
We use the DTU Pricing structure in Azure for SQL DBs. What is the difference between the DTU and the vCore pricing structure?
Autoscaling is an opt in feature and can be charged to an Azure Subscription
Once the spike is over, scale down happens and you stop paying for the scale up.
Autoscale Notifications
Toasts pop up in Admin Portals Capacity Settings to let your admin know when autoscale is running.
It would be good to be able to tell everyone working with Power BI when this is happening simply for reference
Get Started with setting up Autoscaling in Azure
Go into Azure and Select Subscriptions. You need to have made the decision before hand on which subscription can be used for Autoscaling billing.
Next create an auto scaling resource group
Enable Autoscale in Power BI Admin Portal
To do this you need to be the Power BI Admin (Or Global Admin)
Another addition is that the person needs to be at least a contributor on the Azure Subscription to go through all the steps succesfully
Go to Capacity Settings
Make sure Premium Generation 2 is already enabled
Then Select Manage Auto Scale
Enable auto scale and then select your Azure Subscription
And then assign the number of vCores to the Autoscale
Here we have set the max of 2.
How many vCores does a Premium P1 capacity have?
8 virtual Cores
Once completed you are all set. There are some questions. Apart from the Toast pop ups are there other ways to monitor and log the usage of Autoscaling?
The big takeaway from all this is that we should never be in a situation where we are surprised that we have reached capacity. Or, if we do set up Auto Scaling it should not be used on a day to day basis.
More investigation is needed on how to set up proper monitoring so we have full knowledge about what is going on in Premium Capacity. And we must never forget that there will be Pro workspaces already set up and these shouldn’t go under the radar either.
We will have a look at these issues in future posts
We can now start adding Subscriptions for all our different Business domains and environments. Azure allows you to quickly get started adding Resource groups and Resources to your subscriptions. Using Management groups above subscriptions allows for even more control. You can manage access, policy, and compliance across multiple subscriptions.
This is all great but what about controlling costs? You will very likely have team members that cant wait to try resources to see what they can do but what happens when you came from after a weekend and you test Cosmos DB has racked up a huge amount of money?
We want to be able to control things more so lets look at some options.
Azure Spending Limit
The following Subscriptions have the ability to apply Azure Spending Limits
0044P Free trial
0059P Visual Studio Professional subscribers
0060P Visual Studio Test Professional subscribers
0062P MSDN Platforms subscribers
0063P Visual Studio Enterprise subscribers
0064P Visual Studio Enterprise (BizSpark) subscribers
0029P Visual Studio Enterprise (MPN) subscribers
0025P Action Pack
0111p Azure in Open Licensing
0170p Azure for Students
0144P Microsoft Azure for Students Starter
When you get a free account, or are for example a Visual Studio Enterprise Subscriber you will get personal use credits.
for example. Going to a personal Subscription and looking at the overview
This Visual Studio Enterprise Subscription allows the user £115 Credits. You cant increase set credits.
If you reach your spending Limit, all services are disabled until the next billing period.
If you need to uncap you need to remove the spending limit by clicking on the above message.
Remove the Spending Limit for the current billing period will allow you to continue and the credits will then reset.
Remove the Spending Limit indefinitely will allow you to go over your Spending limit every month by adding your credit card details (Pay as you go)
However the recommendation is to only remove if required to keep more control
This is obviously a great choice for personal Azure Subscriptions, used for testing and trying things out.
Monitor Costs When using Azure Services
You cant cap the following subscriptions
0136P Microsoft Azure EA Sponsorship
0003P Pay-as-you-go
0023P Pay-as-you-go Dev/Test
0148P Enterprise Dev/Test
0036P Microsoft Azure Sponsored Offer
0243P Azure Pass
0145P Azure in CSP
Track Costs with Budgets
Budgets allow you to set thresholds for your spending. You can then get alerts on these thresholds.
If for example, you have a budget for £500 and a threshold for £400 you could also start adding in automation to start shutting down resources, like VMs at the threshold.
Setting the Budget at the Subscription Level
Within the Subscription Overview
Click Next
Setting Alerts
Application Insights Smart Detection has been used as the action Group if the costs go up to 90% of the budget that has been set
‘Smart Detection automatically warns you of potential performance problems and failure anomalies in your web application’
Then Create
Targeting a Resource with a Budget
Again going to a Subscription and Clicking Cost Analysis
The resource that has a habit of becoming expensive is the database in this subscription
We want to make sure it doesn’t get out of control
Create a new budget for this subscription and then add a filter (the filter will only work once the graph is displayed
Resource Type is selected
then Microsoft SQL Servers. You could also put tags to good use in Budgets. You could for example check for tags of resources created by specific people.
You can now have Budgets that specifically check on certain services. Great if you have Services that have a tendency to create lots of costs.
So now we can rest a little easier knowing that Azure is going to let us know what is happening. It would be great if we could apply automation to shut something down if it was accumulating costs over the threshold. We can look at this next time
The World of Azure Subscriptions and setting up a good Azure architecture can seem quite daunting (because it is). Here is a guide to Azure Subscriptions and how they work.
Free Subscription
When you first go into Azure you can start for free.
This is probably the most used subscription starting point. There are services that you can use free for 12 months. For example:
5GB Blob Storage
5 GB File Storage
250 GB SQL Database
750 hours of Windows VMs
30,000 transactions of AI and machine learning
The list goes on.
You also get 150 credits to use on other services. And in many cases, some services are always free, like functions and App Services
Monthly Azure Credits for Visual Studio Subscribers
Note that the Azure service states “Your own personal sandbox for dev/test”
In regards to running Production environments, the guidance states
“Monthly Azure credits for Visual Studio subscribers are specifically for individual dev/test usage only. To run production workloads, you’ll need to upgrade to a paid Azure subscription, such as pay-as-you-go pricing.”
The question is, what does Microsoft consider a Production environment?
And if these are supposed to be for individual use, what should you do when you have multiple developers working on a development project?
Answers coming up later…….
Visual Studio Professional subscription gives you £40 a month credits. Professional costs around £920 for the first year and then yearly renewal is around £613
MSDN Platforms subscription gives you £75 a month credits
This is a new Subscription that provides IT Department with a cost effective way to license Microsoft Software for individuals but don’t require the full suite of Visual Studio Development Tools. Pricing is more complex https://visualstudio.microsoft.com/msdn-platforms/
Create Subscription in Azure
When you go into Azure and search for Subscriptions +Add Subscription you are taken to the following screen. (these are the offers we have available to us. Depending upon your own environment you may have more or less)
Pay as you go
As advertised, this is simply the pay as you go model that many people move to after their free subscription has ended
Pay as you go is billed at the standard Pay-As-You-Go rates and service level agreements (SLAs) are offered for each of the Azure platform services
Pay as you go Dev/Test
Meets the need of teams of Visual Studio subscribers and is specifically for development and testing only.
Enterprise Costs around £4,602 for the first year and then £1,970 for every subsequent year
Visual Studio Enterprise Solution – MPN
Again, this is specifically for Visual Studio Enterprise Subscribers in the Microsoft Partner Network. This relates to the Monthly Azure Credits for Visual Studio Subscribers section above
Within this service, you get low dev/test rates on VM’s (You pay for the linux rate) You also get low rates on HDInsight and Web Apps.
As mentioned, this is specifically for personal Dev/test Environments. Usage within the subscription does not carry a financially-backed SLA
Azure in Open
You can buy Azure through open licensing programmes from your Microsoft Reseller. Small and medium sized businesses purchase in this way. Simply contact your Microsoft Partner or find one to work with https://www.microsoft.com/en-gb/solution-providers/home
To go for this option, you purchase Azure Monetary Commitment credits from your reseller in the form of an Online Service Activation (OSA) Key. You can use these credits for 12 months following Activation
Other Subscription Offerings
Enterprise Dev-test
Similar to Pay as you go Dev Test, If you have a Microsoft Enterprise Agreement
EA/SA (Enterprise Agreement/ Software Assurance) is a licensing package, targeting large organisations with 500 or more computers. The EA is a three year contract which covers software licensing and updates. Customers can renew for 1 year or 3 years.
Software under EA includes Windows 10, Microsoft office, Windows Server, Exchange, Sharepoint, System Center, Client Access Licenses etc.
To set up a subscription under this offer, you will have an Enterprise Administrator. They can add Account owners to their agreement via Azure Enterprise portal. Then the Account owner can create Subscriptions via the Azure Account portal.
Once in place, active Visual Studio subscribers can be added as co-administrators to manage and use resources.
Visual Studio Enterprise (Bizspark)
In February 2018, BizSpark was replaced by the Microsoft for Startups program and has the same functionality as the Visual Studio Enterprise Subscription with £115 free credits
Azure Hybrid Benefit
Azure Hybrid Use Benefit (HUB) is a discount program for Azure users with Windows Server licences. It can save up to 40% of the normal costs of running Services.
Windows Server is used in business settings. Windows Server is a line of operating system that Microsoft specifically creates for use on a server. It includes Enterprise software because its intended for businesses.
Microsoft Windows Server 2019 pricing is complex so I wont go into detail here, but moving to HUB does provide savings for VMs and databases.
Difference between Dev/ Test and Production Environments
Dev/Test environments and Visual Studio credits are exclusively for developing and testing applications. These subscriptions don’t carry financially backed SLA’s
Any Subscription over a Visual Studio Subscription is for Dev/test and demonstrate only
It should also be noted that Training is NOT considered a development activity so training resources should be on the appropriate Subscription.
MSDN Licensing is now Visual Studio licensing but the following applies
“The single most important concept when discussing MSDN licensing is to understand the difference between production systems (those which are used to actually make money or carry out the business of an organisation) and development systems (those which actually create the programs and applications used in the production systems)”
IT Asset Management .net (2011)
And there is also the Subscriptions in Dev/ Test being for specific individual useage. For actual Project work where teams are involved, services are cheaper but you don’t get credits.
Is this policed by Microsoft? It is certainly a violation of the terms of use.
c. Suspension. We may suspend your use of the Online Services if: (1) it is reasonably needed to prevent unauthorized access to Customer Data; (2) you fail to respond to a claim of alleged infringement under Section 5 within a reasonable time; (3) you do not pay amounts due under this agreement; (4) you do not abide by the Acceptable Use Policy or you violate other terms of this agreement; or (5) for Limited Offerings, the Subscription becomes inactive from your failure to access the Online Services as described in the Offer Details. If one or more of these conditions occurs, then:
Production subscriptions can be purchased via several means
PAYGO is specified above
EA . Again this is specific to having a EA/SA (Enterprise Agreement/ Software Assurance). See above for more information
CSP (Microsoft Cloud Solution Providers) See Azure in Open for more information above.
Management Groups, Subscriptions and Resource Groups
As an example, imagine you have an Azure environment with just Visual Studio Enterprise Accounts complete with credits for each developer
So far, no thought has been put into creating Management groups
lets have a look at this so far. the decision has been made to split the resource groups up by environment. In this case Proof of concept. However, because we know that this subscription can only be for Dev/Test and POC maybe we need to think about about the hierarchy in a little more detail, bringing in the Management group
Management Group by Environment
Management groups are containers and they help to manage access, policy and compliance across multiple subscriptions. You can use them with Azure policy and Azure Role Based Access Control.
Here we have Management groups split by environment. So under Production we may have all the pay as you go Subscriptions complete with SLA backing. The Test/Dev subscriptions under the other Management Groups when required.
The Resource Groups here as set at Apps level.
You could add a Resource Group for all your SQL Servers and Databases for example. And another one for all your machine learning requirements.
Management Group by Business Area
This is a different way of looking at your hierarchy, by Business Area. Management groups can be up to 6 levels deep which allows you to set up what ever meets your needs. If you worked for other companies you could have a management group for each company for example.
You could then have management groups under this for environment. Next Subscriptions and finally the Resource groups, which can bring together resources that are related.
Don’t forget that Tagging is also an important part of the architecture process
Hopefully that gives a little more structure to what you need to to when thinking about Subscriptions, which then leads into your Management Groups, Resource groups and resources.
Right at the top of your Hierarchy is your Tenant which represents your Organisation. Your tenant is your dedicated Azure AD Service instance.
Understanding what licencing you have to work with, what hierarchies will be the most logical and who you work with within other tenants will really help you get to grips with Azure.
Debbies Microsoft Power BI, SQL and Azure Blog 