Azure Logic App – Copying a file from Sharepoint to a Data Lake

I have been asked to set up a Logic app in Azure (That is Power Automate for anyone outside Azure) to copy specific file(s) from a Sharepoint folder and add to an Azure Data Lake.

The first example file is around 16,00 rows and not likely to grow too significantly. This is the same with the other files.

There is a specific use case behind this First logic app:

  • The Data in the csv file(s) is updated every day so the file name remains the same
  • We need to copy the File and overwrite the file in the data lake every day after the task has been done to update the Sharepoint File (Around 5PM every day)
  • we want the Logic App to run via Data Factory
  • Once the logic app has run we want to trigger the pipeline to populate the SQL database from the file in the data lake.

Set up the Logic App

In azure go to Logic App and New

Log Analytics: to get richer debugging information about your logic apps during runtime

Consumption Plan: Easiest to get started and fully managed (Pay as you go model). Workflows increase slowly or are fairly static

Standard Plan: Newer than the consumption plan. Works on a single tenant. Works on a flat monthly fee which gives you potential cost savings.

Create the Logic App

Once you have added tags and created its time to create the logic App

Because we want to trigger in Azure Data Factory we want to go for When a HTTP request is triggered

The HTTP Post URL will be used in Data Factory to trigger the Logic App.

I have added a JSON Schema that supports some of the important information for this project. Like Container for the data lake, Folder , File name and isFolder (Which becomes more important a little later.

{   
     "properties": {       
        "Container": {            
           "type": "string"        
        },        
        "fileName": {            
           "type": "string"        
        },       
       "folder": {           
            "type": "string"        
       },        
       "isFolder": {            
           "type": "boolean"        
       }    
  },    
"type": "object"
}

List Folder

Now we want to List Sharepoint folder. So create a new step and search for List Folder

Returns files contained in a Sharepoint Folder.

Next you have to Sign into Sharepoint with a valid account that has access to the Sharepoint site.

Here is where we have a question. For this test, my own username and password has been used but obviously I change my password at certain points which means that this will need manually updating when this happens.

What we need is a way of logging into Sharepoint that isn’t user related and we can use within the logic app. This needs further thought and investigation.

When you log in you create a Sharepoint API connection in Azure Resource Group

To get the site address you can go into Sharepoint, Click on the … against the file and copy link.

The link needed amending slightly because it needs to be tenant.sharepoint.com/sites/ProjectMainArea

If you have access you should then be able to click the folder against File Identifier and select the correct area

For Each

Next Stop, For each ‘Body’ from the List Folder step. We get the File Content. Go to Next Step and choose the For Each Condition (Because there will be multiple files)

Get File Content

Now We want to Get File Content From Sharepoint

Gets File contents using the File Identifier. The contents can be copied somewhere else or used as an attachment

You need to access the same Sharepoint site address as before. Then click on File identifier and choose ID from the Sharepoint Dynamic Content pop up

so here we can see that from the list folder step we have lots of file metadata we can use like DisplayName. ID, LastModified etc.

We know we need ID for Get File Content

We are at a point where we can run this now as a test.

Note that so far we have this set up

but we hit specific issues

Status 404 File not found

cannot write more bytes to the buffer than the configured maximum buffer size of 10457600

So we have two issues to resolve and after a bit of help on the Q&A Forums we find out that:

List Files “Returns files contained in a Sharepoint Folder. ” Actually also returns folders which are erroring because they are not files

Logic Apps aren’t really set up for large files. There doesn’t appear any way we can get past the size issue. So we need to check our files and also think of ways to bring through smaller data sets if needs be.

Thankfully our files are way below the threshold and the business thinks that they won’t increase too much.

So here is where we can start applying criteria, which we want to do anyway because we only want certain files.

  1. If its a folder we don’t want to use it
  2. If its over 10457600 in size we don’t want to use it
  3. Only bring through files called…….

So we need to change our For Each

Within For each add a new step and search for Condition

And add your conditions (And Or)

Then you can move the Get File content into True

So If IsFolder is false and the size is less than 10457600 we can grab file A OR File B.

When you now test this Logic App Get File content should succeed with most not even hitting the criteria.

Create Blob

Finally within the True section we need to add the file to our Data Lake.

Search for Create Blob

Here you have to sign into your Blob Storage which again creates another API Connection in Azure

You have to supply the Storage account name and choose an authentication type. Access Key has been used, the details added here. Normally in data Factory the Access Key is obtained through a Key Vault so, more information is needed to come up with the most secure way of doing this. There are two other authentication types to choose from.

More investigation is needed into these other approaches.

Now we can do a full test of the Logic App

Testing the Logic App

When you trigger the logic app

The Body contains a long list of every object. Really handy to know what the details are inside this action.

To test this was copied into a word document.

Next comes the Get File Content

Now most of the files don’t satisfy the condition.

Next was clicked to get to a file in Get File Content (first one appeared as number 32)

And now we can see the Body of the Create Blob. (This happens for every file specified in the criteria

And if you use Microsoft Storage Explorer app you can check that they have indeed been updated (Either its a new file or it updates what is already there)

Data Factory

Now we have saved the Logic App we want to trigger it in Data Factory

Create a pipeline and choose a web activity

Copy the URL from the Logic App and paste here

For the Body I simply used the Simply JSON at the start of this article.

Now you can trigger this pipeline along with all your other pipelines to run the Data into your Data Lake and then into SQL to be used for Analytics.

https://www.mssqltips.com/sqlservertip/5893/transfer-files-from-sharepoint-to-blob-storage-with-azure-logic-apps/

Power BI February 2022 Updates Dynamic M Query Parameters

Now supports SQL Server and more data sources

But what are Dynamic M Query Parameters and what does this mean?

It feels like they have been upgraded to use with direct query data sources so you can restrict the amount of data being asked for at the lowest level.

Lets have a look at a simple example using Taxi data from a Microsoft learning path.

First of all you need to open Power BI – Options and Settings – Options

Its in Preview so make sure that is ticked before continuing

Get Data Azure SQL Database (The guidance mentions SQL Server but it seems that both can be used for this test)

Load

Then go to Transform data.

Right Click on trip fares to get to advanced editor

let    
Source = Sql.Database("dataengineering-sqlserv.database.windows.net", "taxi-data-db"),    dbo_TripFares = Source{[Schema="dbo",Item="TripFares"]}[Data]
in   
 dbo_TripFares

Its currently bound to a table but we need to bind it to a query for this process.

Click the cog against source.

Go into Advanced Options and add the SQL Statement

SELECT * FROM dbo.TripFares

And then go back and look at advanced editor

let   
Source = Sql.Database("dataengineering-sqlserv.database.windows.net", "taxi-data-db", [Query="SELECT * FROM dbo.TripFares"]),    
dbo_TripFares = Source{[Schema="dbo",Item="TripFares"]}[Data]
in    
dbo_TripFares

So now, its nearly bound to a query but you will note that it looks like the table is erroring.

You can go back to Advanced Editor and change to

let
Source = Sql.Database("dataengineering-sqlserv.database.windows.net", "taxi-data-db", 
[Query="SELECT * FROM dbo.TripFares"])in   
 Source

 We only need the query and not dbo_TripFares

Now we can add the  Dynamic M Query parameters. I will go for an easy one first as a demo.

And then I change the advanced code again

let 
Source = Sql.Database("dataengineering-sqlserv.database.windows.net", "taxi-data-db", [Query="SELECT * FROM dbo.TripFares Where payment_type = '" & paramPaymentType & "'"])
in    
#"Source"

Note the new WHERE Clause that concatenates the value in our parameter

It will read in SQL SELECT * FROM dbo.TripFares Where payment_type = ‘CRD’

When it runs the first time you are asked to approve and you can actually see the SQL its going to use which is good. (Note I had to change to CSH to get the message up but I am running with CRD)

When it comes through its restricting to the selected payment type

We are going to change the code again

let    
filterQueryPaymentType = "SELECT * FROM dbo.TripFares Where payment_type = '" & paramPaymentType & "'",    
Source = Sql.Database("dataengineering-sqlserv.database.windows.net", "taxi-data-db", 
[Query=filterQueryPaymentType])
in  
#"Source"

This sets the SQL command first and we pass the Filter query into the data source

Now we know that the query works. Lets use it in Power BI Reporting and Assign to table.

This will need a lookup table of all the payment types to work

I am going to simply create the reference table in M

let
Source = Sql.Database("dataengineering-sqlserv.database.windows.net", "taxi-data-db", 
[Query="SELECT DISTINCT payment_Type FROM dbo.TripFares"])
in    #"Source"

Close and Apply

Now bind the table to the parameter in Modelling tab

Click on Payment_Type column. Advanced. Bind to parameter

Click Continue

A multi select is not going to be used for this demo

I have added a quick table. the metrics have come through as strings and there will be lots of things you need to test in direct query mode but I will ignore for the time being.

I dragged in Payment type from the Payment Type lookup into a slicer.

Click the slicer and see your data change. every time you click the slicer a direct query will happen but only for the payment type selected, hopefully making things much quicker.

And there you go. You have set up a restricted direct query. This will help with any direct query reports you need to create based on real time data.

You are still hitting the SQL DB though a lot so this would need thinking out.

And remember, Direct query doesnt give you the full Power BI reporting suite so your reports may be more basic. And usually I like to work with Star schemas but here we have the extra complexity of lookup tables to work with the parameters.

I will be looking at a date time example soon hopefully. This is clearly an extremely important piece of the Direct query real time puzzle.

Power BI Admin APIs to return a list of email subscriptions

Get Dashboard Subscriptions brings back a list of everyone who has subscribed to a dashboard

What is a Power BI Subscription?

Subscriptions are a great way to assign yourself and other users to get emails regarding report content.

There are certain governance rules we follow.

  1. Report viewers views content via an app. We don’t want report viewers coming into the App workspace. we want them to see carefully selected and brought together content.
  2. If we use Subscriptions we want to push though a really nice screen shot of a report that changes and gets the users wanting to come and see more content within that app. therefore we always have a report or dashboard with visuals that don’t need scroll bars to engage the viewer so they want to see more.
  3. because of this, we want to be able to subscribe people to App content

Go to an App. Note you can add your Subscription here which is a link to the dashboard

for this example, the App dashboard is subscribed to

then we go to try it out from the Microsoft API Page

Dashboard Subscriptions

https://docs.microsoft.com/en-gb/rest/api/power-bi/admin/dashboards-get-dashboard-subscriptions-as-admin

and try the API

Add the Dashboard ID to parameters

But this is where logic is not quite working (the hope is that this will resolve fairly quickly). The above API doesn’t give you information back if you subscribe via the app. Only when you subscribe to the actual Dashboard in the workspace.

We want all our report viewers accessing the pre built app so this is where the information is most required.

When the user is added to a dashboard subscription in the workspace. The API is tested again.

What this can show us is anyone in the workspaces that has subscribed to the actual dashboard.  We want all viewers with App access.

Get report Subscriptions as Admin

https://docs.microsoft.com/en-gb/rest/api/power-bi/admin/reports-get-report-subscriptions-as-admin

This is the same as above but with reports

Get user Subscriptions as Admin

https://docs.microsoft.com/en-gb/rest/api/power-bi/admin/users-get-user-subscriptions-as-admin

I get my user ID from Azure Active Directory

And see what I’m subscribed too but again, only workspace content

Logically, I feel like our viewers should be subscribing through the apps

this is really good stuff but I feel like they need to resolve the issue with Apps. Apps are the go to areas for users to view content so this is where we want people to subscribe too.

If you look at the information coming back. Here we can see the artifact type is report but there is no where that mentions if the report is in an App or in the workspace and I feel like this is actually important information. I only know because I have tested against both the App and the workspace.

If this could be resolved these APIs would be really useful to help us understand the subscription uptake.

Power BI Sparklines

Sparklines are small charts you can add to a table or Matrix and are new for the start of 2022.

Above we have a Matrix showing Products by country and I would like to also see this measure on a timeline.

Select the metric Value : Add a Sparkline Button

Here we choose the measure Total Sales which has already been summarised. The Date will be used as the X Axis so we can see the trend across time.

Rename to Sparkline if you want (double click the value to highlight as editable) and now you can see the trends

And you can also format the spark line (Right at the bottom of formatting)

So Sparklines are great for adding you your tabular information, I think the line charts are pretty good.

Current restrictions

Only supports 5 per visual so if you have more categories that wont work.

Apparently Date hierarchies don’t work at the moment but they will in the future

It would be good to assign conditional formatting to the colours. So going up could be green. Staying around the same amber and going down red.

they really do bring tables to life though and are are really great addition to power BI

Investigating the Power BI Scanner APIs Part 1

Delegating Permissions

Part 1 will be an introduction to Scanner APIs and how you delegate permissions in order to now have to use the power BI Admin Account

Power BI Scanner APIs are fairly new and there have been a lot of updates happening in the September October 21 Power BI Updates.

These scanner APIS scan, catalog and report on all the metadata of your organisations Power BI artifacts.

Scanner APIs are the Admin REST APIs that can extract tenant level meta data that support the Power PI good Governance Pillar of Discoverability and Monitoring

So the scanner APIs are a set of Admin REST APIs.

What is an API?

API is the acronym for Application Programming Interface, which is a software intermediary that allows two applications to talk to each other”

What is a REST API?

“an application programming interface that conforms to the constraints of REST. REST = Representational state transfer

Scanner APIs

When you look at the list of Power BI APIs these are the ones specific to the Scanner API group

Your Power BI Admin needs to set this up. I have used the following information for this

And as usual I’m extremely thankful to everyone providing such fantastic information.

Service Principal support for read-only Admin APIs

Service Principal Support to the Scanner admin APIS became available September 21.

“Service Principal is an authentication method  that can be used to let an Azure AD applications access Power BI APIs. This removes the need to maintain a service account with an admin role. To allow your app to use the admin API’s, you need to provide your approval once as part of the tenant settings configuration.”

The Scanner API’s and Service Principal Support could well be a game changer for governance if everything is actually in place.

So we are able to delegate permissions to use the APIs

I followed the documentation https://powerbi.microsoft.com/en-us/blog/power-bi-september-2021-feature-summary/#post-17174-_Toc819…

Create an Azure AD app.

 https://docs.microsoft.com/en-gb/power-bi/admin/read-only-apis-service-principal-authentication

So as a prerequisite here is my list of everything that needs to be done based on the documentation read

Prerequisites

Create Azure AD App

  • Go to Azure AD
  • App registration – New Registration
  • Make sure its a web application

Assign Role

  • You will need to choose an Azure Area for this project (Subscription and resource group)
  • Go to Azure Level (In my case the level is at resource group not subscription)
  • go to IAM. Add Role to the app
  • I selected contributor

Get Tenant and AppID

  • Go back to App
  • Get tenant ID
  • Get Application (Client ID)
  • These will be store both in the Application Code at a later date so keep this in mind.

Create Application Secret

  • App Registration – Select the app again
  • Client Secret – New Secret
  • I have added the secret to key vault for use later (the Key Vault is in the same Resource group as selected above)

Configure Access Policies on Resources

  • I have a Key vault.
  • Added the Service principal of the app in Access Policies with Get and List on Secrets

Create Security Group in Azure AD 

  • Go to Azure AD Groups
  • New Security Group
  • Add the App as a member in Members

Enable Power BI service Admin Settings

  • Power BI Admin – tenant Settings (Must be Power BI admin)
  • Allow Service principal to use ReadOnly Power BI admin APIs
  • Add the Security Group created above which has the service principal as a member

Start using read only admin APIs?

The documentation finishes at this point so how do you use these APIS?

We will look at this in part 2 but at this point we want to be able to set up and use in a Data factory.

Linked Service

Note that here we use the AAD Service Principal

We added the App Secret into Key Vault which we used here

The Service Principal ID (Blanked out here) Is taken from the App. Azure Active Directory

App registrations

And after a test its successful.

We can go onto swap out the authentication type of web services later in the process. Like this one for getting Workspace Info (Scanner API 2)

Later we will look at how to set up the Scanner API in a data factory, But in the meantime, Here is a possible error you can get when attempting to work with the Service Principal

Operation on target Post WorkspaceInfo failed: GetSpnAuthenticationToken: Failed while processing request for access token with error: Failed to get access token by using service principal. Error: unauthorized_client, Error Message: AADSTS700016: Application with identifier 'IDENTIFIER DETAILS' was not found in the directory 'Peak Indicators'. This can happen if the application has not been installed by the administrator of the tenant or consented to by any user in the tenant. You may have sent your authentication request to the wrong tenant. 

After closer inspection, I had missed the last number from the Application /Client ID so this was a quick fix.

When you use the Scanner APIs in Data factory you use all 4 in sequence. Lots more to think about here. The data factory Set up. How you switch modified workspaces to only take updates after a certain time.

so lots more to come

Power BI DAX – Create a percentage across days of week (ALLEXCEPT. ALL. DIVIDE)

We just looked at the following request

We have sold specific quantities of products. Can we look at a week for a year in a slicer and see the percentage of items sold for each day. Which days do we sell the most products overall?

We have the following model (Just the parts that will be required for this exercise

And thanks to the Power BI Forums for giving me some help on figuring this out.

What we want to end up with is:

  • A year Slicer
  • Product category on rows of a matrix
  • The Day Name from Date on the Columns of the Matrix
  • And % Quantity for that day based on all the days of the week as the value

Power Query Editor

At first I believed that I needed to add some kind of Ranking order for the DAX to use and as this will be a column its better to do in Power Query Editor (Or in the Sort DB)

To speed things up I created a Year Week Column in SQL DB consisting of the following examples

  • 202101
  • 202102
  • 202135

So the weeks 1 to 9 was padded out with a zero. I then formatted this to a number field and called it Year Week. Every seven rows within that week in the date table will have the same Year Week Number Applied

Measure

I can now create the measure. Lets have a look at it in a bit more detail

Order Quantity by Week % = Var varOrderQtyWeek = CALCULATE(SUM(FactResellerSales[Order Quantity]), FILTER(ALL(DimDate), DimDate[Year Week]))
RETURN DIVIDE(SUM(FactResellerSales[Order Quantity]), varOrderQtyWeek)

And this measure is then set as a percentage.

First of all we create a variable. It gets the Sum of Order Quantity and it filters by using an ALL on the column Year Week we just created in the Date Table

“ALL Returns all the rows in a table, or all the values in a column, ignoring any filters that might have been applied (On Year Week).”

And we return the sum of order Quantity (Which in the context is just for, for example Friday and Accessories by the Sum of Order Quantity

This appears to work

We have Implicitly filtered by Product Category and Day Name of Week.

the question here is, did we even need to set ALL on Year Week column in date. Could we have just said use the entire date

Order Quantity by Week % = Var 
varOrderQtyWeek = CALCULATE(SUM(FactResellerSales[Order Quantity]), ALL(DimDate)) 

RETURN DIVIDE(SUM(FactResellerSales[Order Quantity]), varOrderQtyWeek)

This is working in the same way and makes sense to me. We are using all the dates in the date table. And removing the Filter will create a faster DAX query.

Its looking at the entire data set and we can see that everything totals to 100%. So for us, taking into account all years Thursdays look like a good day especially for accessories.

However we don’t want to analyse the full data set and when we add in a year slicer the logic fails

As you can see, the story is still true but its telling the wrong story. Now for Accessories we have 31% of sales happening in 2021, attributing to smaller percentages across the days.

So we want to change the DAX to accept the Year Slicer

Order Quantity by Week % = 

Var VarALLEXCEPT = CALCULATE(SUM(FactResellerSales[Order Quantity]), ALLEXCEPT(DimDate,DimDate[CalendarYear]))

RETURN 
DIVIDE(SUM(FactResellerSales[Order Quantity]), VarALLEXCEPT)

And this appears to now work just taking the year 2012 into consideration because we are using ALLEXCEPT

Thursday is definitely a good day.

“ELLEXCEPT Removes all context filters in the table except filters that have been applied to the specified columns.”

So we create a variable to get the Sum or Order Quantity with ALLEXCEPT our Date Slicer which in this case is the calendar year slicer

So we are using 2012 from the date Dimension in the slicer. And Bikes from Product and Thursday as the Week Name from date in the Matrix

We Divide the Sum of Quantity which has all the filters applied. Bikes. Thursday. 2021

By the Sum Of Quantity with all, Except we do apply the Calendar Year.

  • DIVIDE(50/100)
  • 50 is Bikes on a Thursday in 2012
  • 100 is Bikes in 2012

DAX is something I do struggle with a little. You think you understand it. Turn your head from the screen and its gone again.

But hopefully this goes a little way to understanding how this DAX in this specific context has been approached

Power BI AI Visual Smart Narratives

Power BI has given us the following AI visuals:

  • Key influencers
  • Q&A Visualisation
  • Decomposition Tree

And now we also have smart narratives to work with. Many companies want to create report that includes text in regards to the report they are looking at to help the user understand the key takeaways. This usually involves someone manually adding in text to the reports.

Not great for interactive reports that update on a regular basis. this is where smart narratives come in. Smart narratives create the text in an interactive way for you.

If its good, it should eliminate the need to add in information manually for your users, which is a huge win.

Lets have a look at it in action on the Adventureworks Report

At the most basic level you can simply drag smart narratives into your report

Smart narratives describes your data to you. This is all really good information. what happens when I select a data point on the report like select 2011 in Sales amount and order quantity

The narrative updates to tell you about just 2011 so our smart narrative is even better, It updates with every click the user makes.

For my Top 10 Products table, I want a smart narrative that just gives me information on this visual and not every visual on the page so I right click on the visual and choose Summarize

And again, we are getting good information for that specific visual. You need to think whether the visual can tell this story on its own or if narrative would help. I tend to see smart narratives as something that sits on a page to tell a story across all the visuals.

Changing the Summary

This is great but there will be times when you do want specific information in your smart narrative. I am going to remove the smart narrative for the visual, and replace with one for the entire products report page.

Click on the smart narrative and then +Value to add a value

I want it to tell me how many items have been sold of top value products. the red line tells me that it doesn’t know what sold means and we need to define this item .

Click on Sold and then Define Sold

Here I go to Field Synonyms in the Q&A Setup to define a new synonym for order quantity because order quantity can be expressed as quantity sold. I have added quantity sold as this is what has been searched for.

Back in Teach Q&A I ask the question again

And this time it understands what I mean. I submit this to train the Q&A. and this time we find this specific query because its in Q&A. I can now format the question to get an answer for the smart narrative

After saving, the number simply appears in the smart narrative. I deleted it and moved the cursor to the beginning on the smart narrative. then I wrote.

The quantity sold of top value products is:

And then clicked review, found the item and + to insert item to place it exactly where I want it

from this I can see that I can use the Q&A tool to format questions that I can then add into the smart narrative, adding synonyms where necessary to create even more easy to understand questions.

Smart narratives and Q&A work perfectly together.

Power BI October 21 Updates GetUserArtifactAccessAsAdmin API through to PowerShell

There have been some new API updates (September October Power BI 2021) and I thought I would see what they were all about and how you can use them starting with the API mentioned in the October updates.

I haven’t used these APIs before and I thought it would be interesting to see how the documentation holds up.

So the first start at https://powerbi.microsoft.com/en-us/blog/power-bi-october-2021-feature-summary/#post-17563-_Toc84518701

And this links you you to https://docs.microsoft.com/en-gb/rest/api/power-bi/admin/users-get-user-artifact-access-as-admin

So this API returns all the artifacts for a given user which I imagine would be really useful when someone leaves the company and you need to assess the objects that they may own.

To use the API you need to be Power BI admin or Global Admin. I have the Power BI admin so I am good to go.

Delegated permissions are supported. I imagine this needs a lot more investigation as to why and how you would set this up.

There are some limitations, For example you can only make 200 requests per hour. You can click on Try it to try out the API from this URL by logging into your account

But this API’s need you to get the userGraphId. my first question is, What is a userGraphID and how can I get it?

It is mentioned in the documentation. ‘The graph ID of user’

And to be honest that is unhelpful. I have no idea how to get this information to run the API. Its just assumed. I have also asked on the forums and no one else seems to know either.

So to just get an idea of how you would use these APIs, I’m working with one that doesn’t need the userGraphIDhttps://docs.microsoft.com/en-us/rest/api/power-bi/admin/groups-get-groups-as-admin#code-try-0

This will get me out all the objects in power BI and for the time being I just want the top 10.

I added the $expand name and value by creating the name and value and hitting the +

Then hit Run to make sure it works

So where do we use it? Open up Windows Powershell

We can run the commands via Windows Powershell. https://docs.microsoft.com/en-us/powershell/scripting/overview?view=powershell-7.1 ISE (Integration Scripting Environment)

powershell_ise.exe

And type in

 login-PowerBI

And now you can create your code

I simply added the URL from the Try it above (Without the Authorisation Line)

Value[0] Simply means give me the first element inside this command and run.

I can go even further an parameterise the URL by doing the following

Param(
[string]$url = 'https://api.powerbi.com/v1.0/myorg/admin/groups?%24top=10&%24expand=users%2Creports%2Cdashboards%2Cdatasets'
)
login-PowerBI
$result = Invoke-PowerBIRestMethod -Url $url -Method Get 
$workspaceContents = $result | convertFrom-Json
$firstworkspace = $workspacecontents.value[1]
$firstworkspace

This is great

  • But what if I want to automate this and, for example create a json file from it once a month?
  • And what if I don’t want to have to run it as the Power BI Admin?

Back to the original question regarding userGraphId

It seems that many other people have asked this question. I got a possible answer on the Power BI Issues forum saying that the userGraphID is the users Object ID that you can find in azure Active Directory – Users.

So I went into Azure – Azure Active Directory – Users and took my Object ID

I added the Object ID and got the results back that I was hoping for.

Great. This gives all the artifacts that I own. Can I now do this in Powershell like the previous API? Open Powershell again.

powershell_ise.exe

Can I reuse the Previous Script.

Open my previous Script. InvokePowerBIRESTMethos.ps1

 Param(
    #commented out the other API
    #[string]$url = 'https://api.powerbi.com/v1.0/myorg/admin/groups?           %24top=10&%24expand=users%2Creports%2Cdashboards%2Cdatasets' 

    [string]$url = 'https://api.powerbi.com/v1.0/myorg/admin/users/objectID/artifactAccess'

   )

login-PowerBI


$result = Invoke-PowerBIRestMethod -Url $url -Method Get 

$workspaceContents = $result | convertFrom-Json

$firstworkspace = $workspacecontents.value[1]
$firstworkspace 

And this works. I get back all my objects

So this is just the initial investigation. To do

  • How to Delegate Permissions for this API?
  • How to Automate the Powershell script and create a dataset to work with?

And even further research

  • can we automate checking between a list of users from Yesterday and today to see who has gone to automate the process even further?
  • Is it possible that Azure can handle use cases where we know the user is leaving and have assigned a leave date in Azure Active Directory?
  • Then Feed this loop into the API to generate this list so we know what reports need to have their ownership taken over
  • Automate the handling of take over accounts

Tabular Editor and the Best Practice Analyser for Power BI

Finally I get a chance to have a look at the Best Practice Analyse in Tabular Editor

https://powerbi.microsoft.com/en-ca/blog/best-practice-rules-to-improve-your-models-performance/

I am so excited about this because we are looking at ways to improve and speed up the reviewing process on all the datasets being created.

Currently our Power BI Power Users go in to review and take a lot of time checking out the models, processing time, DAX Measures, etc etc.

This is great but if there are ways to speed up that process so we can review even more models then I am in.

So lets get started.

DAX Studio

I always go to DAX Studio connected to my model to get a look at the model Summary before I do any updates. Advanced Tab – View Metrics

Tabular Editor

With a dataset open in Power BI Desktop I open up Tabular Editor from External Tools

And then go to Advanced Scripting

And now I am ready to load in the Best Practice Rules from GitHub.

https://github.com/microsoft/Analysis-Services/tree/master/BestPracticeRules

I go for the Automated Set up approach and run the following code

I believe that these rules get updated so when you run it again your rules will get updated.

Then after running the code I create a custom action over the code by clicking the green + Button

And you now have it in Samples

I can then Close and Open tabular Editor again. (I am doing it with a model already open but you can open it separately and connect to a model.

Then I go to Tools and Best Practice Analyser

And here is where it gets interesting. It brings up a list of Best practice rules that have been worked on by Microsoft experts for a long time. I really want to get a good look at where they are at at 15/10/2021

At this point its fair to say that this has been incredibly easy to do and very useful so far.

Of course the rules come up very specific to my model. 172 objects are in violation of 17 best practice rules. Lets have a look at some of them

[Performance] Do not use Floating Point data Type

My Latitude and Longitude are in Violation here so I should be going back to the model and checking that I cant change these items and not get issues

Some fixes can be taken from best practice analyser.

In this case I am selecting generate fix Script

And pasting to Advanced Scripting window to have a look at it

I can run this and it will immediately update my data model. However it should be said that Tabular Editor requires the XMLA endpoint to allow both read/write access. This setting is controlled by your capacity admin.

In the Power BI Admin Portal

If you don’t have this enabled you need to check with your Admin. At worst you can gather all these recommendations and implement them manually in Power BI.

If you can use XMLA endpoints you can also simply choose apply fix but I prefer to see the fix before implementing.

[Performance] Set isAvailableInMDX to false on None-Attribute columns (6 Objects)

I wasn’t aware of this and all my Keys within the dimensions are flagged as issues here. Lets have a look at that in more detail.

This is rather more specific to Analysis Services. You can stop attribute hierarchies from being build on columns that don’t need them.

Basically all attributes have this set to on. It allows a column on a table to be used on a row or column axis of a visual so it can for example filter a measure. the important thing to note here is that these are structures only used queries tabular models (Analysis Services) IN MDX. Power BI always runs DAX queries so its not really specific to out Power BI Model. We can ignore this rule

And hit Show ignored if you want them back

[DAX Expressions] No two measures should have the same definitions

I have two measures in my DAX. I can right click on each to go to object. and I can then look at the expression

CALCULATE(DISTINCTCOUNT(fact[AccountID]),
FILTER(Accounts, Accounts[Name]<>"NA"))
CALCULATE(DISTINCTCOUNT(fact[AccountID]),
FILTER(Accounts, Accounts[Name]<>"NA"))

It goes to show that these things do happen. You can’t apply a fix script here. Its a case of finding out what uses these two items and ensuring we only use one. So the Question is here, is there an easy way of answering this question?

[DAX Expressions] Use the DIVIDE Function for Division

5 of my objects aren’t using the DIVIDE Function created especially for this. I clearly need to remember that I have a habit of doing this.

There is no fix script for this but I can go to object and to the Expression Editor

I have recreated as a proper DAX Function and you can simply hit the tick to update the code in your model.

[DAX Expressions] Filter Column Values with the Correct Syntax

I am unsure what this means so go to Manage Best Practice Rules in the top corner of the screen

Hit Edit Rule

And you can get to a description of the rule

Instead of using this pattern FILTER(‘Table’,’Table'[Column]=”Value”) for the filter parameters of a CALCULATE or CALCULATETABLE function, use one of the options below. As far as whether to use the KEEPFILTERS function, see the second reference link below.

Option 1: KEEPFILTERS(‘Table'[Column]=”Value”)
Option 2: ‘Table'[Column]=”Value”

Reference: https://docs.microsoft.com/power-bi/guidance/dax-avoid-avoid-filter-as-filter-argument
Reference: https://www.sqlbi.com/articles/using-keepfilters-in-dax/

This is fantastic. I can even go to the document reference to get more information. When I go to the object I can see that I have done this

CALCULATE([Number of Records],FILTER(Records, RecordType[Type]="Test"))

And immediately its clearly to me. CALCULATE is supposed to be fast for one implicit Filter. you use FILTER when its a little more complex and is slower. Although this works I have absolutely no need to use the filter.

CALCULATE([Number of Records],Records[Type]="Test")

And there you go. A faster bit of DAX. Current opinion of this tool. I am really quite taken with this. Its a bit of a gamechanger.

[Maintenance] Visible objects with no description

I have lots of these and absolutely. Each one should have a good description against it. I hadn’t even thought of that.

There are quite a few more rules that I have violated. I wont go through them all here but I am really excited to see how I can do so much more to streamline my model. this is absolutely fantastic.

Additional Script Rules

Note that there are several rules which require running an additional script. And these steps are specific to the Vertipaq Analyser which was added to DAX Studio. We can use the Vertipaq analyser combined with the best practice analyser

https://www.elegantbi.com/post/vertipaqintabulareditor

After reading the documentation above I took the script and ran it in Advanced Editor. I also saved it as a sample just in case.

This should create Annotations that you can see in Tabular Editor when you look at the objects but I cant see anything in my model. this was because you need to go to File – Preferences and Allow Unsupported Power BI features.

Now I can see Annotations

We are basically saying here that in Power BI desktop you can’t create Annotations. This can only be done within the external tool so there is a possibility that this could cause issues. From the documentation it does seem that this is fine to do, but you could now do other things that could cause problems in future. One to think about.

So, We have run report that basically creates the same stats as the Vertipaq Analyser in DAX Studio. How do we use it with Best Practice Analyser?

Avoid bi-directional relationships against high-cardinality columns

We need to create the rule over the script

Add a New rule. I’m simply Following the documentation at this point so lets get to the end of the rule creation

Its important to set the Applies to

If you don’t specify the Applies to you will get an error message (See image below)

I save the new Rule into the Rules for the Local user. But a question here is that, If I rerun the Advanced Scripting Code Sample again to get any new changes, will this wipe of the item created?

To test I go back to Power BI and Change a join to bi directional

And then go back to Tabular Editor and Update the Change Rerun the Best Practice Analyser and I am shown my one bi directional join as a best practice violation. this allows the user to say if its valid

Or if I should do something about it. I will do this in Power BI and rerun

Large tables should be partitioned

When you run the script for Avoid Bi Directional Relationships you also Create the Vertipaq rule for this best practice.

Frustratingly the documentation gets a little confusing at this point. the only rule that has a screen shot to show how to create it is Bi Directional functionality.

I’m making the assumption then that you don’t need to set up the rule and It should automatically appear. However my model isn’t big enough for this to be violated.

Is it a rule then? Got to ToolsManage PBA Rules .

Yes, there it is, I’m just not violating it. I was asked to look at this documentation for more information at this point but I am finding it a little confusing.

https://github.com/microsoft/Analysis-Services/tree/master/BestPracticeRules

Reduce usage of long-length columns with high cardinality

Again for this one, When you run the script for Avoid Bi Directional Relationships you also Create the Vertipaq rule for this best practice.

But you also have to run another script (again I saved as a sample)

Is this a rule then? Got to ToolsManage PBA Rules

This is good, the rule is there but I am not causing any violations to happen

Split date and time

In the above gitHub information you are told to run another script for this which I saved as a sample.

Best Practice Analyser was run again and it appears as a violation. However it is erroring

I know Split date and time is incredibly important to keep the models as speedy as possible so I would like to resolve this problem.

Rerunning the Scripts

Its been some time and we want to review another model. We also think it might be a good idea to rerun the original script in case the rules have changed.

Open another model in Desktop. then open Tabular Editor. Because I have enabled experimental Power BI features I get this warning.

Because we have already run the code we can simply open up best practice analyser to get the results. But what if we want to update for possible changes?

It would actually be good to have a way to know if there have been changes made to the rules?

Rerun the script. close and reopen like last time

Tools – Manage BPA Rules – Rules for the Local User

The additional rules are still there which is great. Lets run the analyser. 367 objects are in violation of 22 best practice rules.

Lots more to do then and all to make the model better and more efficient.

Conclusion

The First section of this just works and I’m incredibly pleased with it and I think it will really help.

The Vertipaq Analyser against Best Practice Analyser references are more difficult to follow but it seems to be that there are three extra scripts to run and one of the rules actually needs to be set up. the other rules are automatically created.

Obviously you have to allow for XMLA Endpoints in order to update your model using Tabular Editor so there are some things to think about here.

Having to set Allow Unsupported Power BI features is a worry for the Vertipac Analyser rules. Having gone through the process to try and figure out how it would sit in our review process I think you could do the following

Power BI – reporting on Items NOT in a filter

We are working on a report to look at Power BI activities and objects at the moment.

Lets have a look at the model

This is a test model with a date dimension (When the Activity occured)

  • A Report dimension (Details of the report used in the activity)
  • A Dataset dimension (Details about the dataset used in the activity)
  • And a Fact table of Activity metrics

We have a star schema with single direction joins to the fact. Power BI as we know loves a star.

Lets have a look at my sample data

Date

Just a sample of the date table

DataSet

Report

Fact

DataSet 3 and report 4 have never been part of an activity (Just for this exercise 01/01/2000 is our fake date because there is no activity)

The above logic is easy to do (Which items have never been used) but what happens when you want to look at the business question

Give me all the reports NOT used in September

As we can see from the metrics table report 2 and 4 were not used in September

So how do we create DAX that will allow us to look at these kind of Business Questions?

Base DAX

We want some base DAX measures that we can then build on top of

# reports = DISTINCTCOUNT(ActivityMetrics[ReportKey])
# Datasets = DISTINCTCOUNT(ActivityMetrics[DatasetKey])

And for the reports we never use we can either use the measure above with a filter

Or explicitly add a filter into a new measure

# Reports Never Used = CALCULATE([# reports],DimReport[neverusedFlag]=true)
# Datasets Never Used = CALCULATE([# Datasets],DimDataset[neverusedFlag]=true) 

Notice that I set Edit Interactions to off between the date slicer and the two cards because the cards are simply telling me how many reports and dashboards have never been used

So this part of the process is all fine.

Our Next Business Question is

Show me the reports and datasets used in September by date

This is another straight forward request

And both interact with the Date Filter.

And now to the question

Show me the reports and datasets NOT used in September

So we use our date slicer and we want to see items outside of that date slicer. We need to create a more complex measure to do what we need

Is Not in the Date Filter Flag = 
// assumes a slicer on DimDate[ActivityDateKey] and there is an implicit filter on the visual 
VAR VActivityDates = SELECTCOLUMNS(DimDate,"Dates", DimDate[DateKey]) 
VAR VDates = CALENDAR(MINX(DimDate,DimDate[DateKey]),MAXX(DimDate,DimDate[DateKey]))
RETURN IF(COUNTROWS(INTERSECT(VActivityDates,VDates)),0,1) 
// if no matching rows, return 1

Lets break the above down a little to understand what is happening

SELECTCOLUMNS

VAR VActivityDates = SELECTCOLUMNS(DimDate,”Dates”, DimDate[DateKey])

First of all create a variable. SELECTCOLUMNS

“Adds calculated columns to the given table or table expression.”

We start with the table they come from. then we give the name to the column in SELECTCOLUMNS “ColumnName”, Next comes the expression which in this case is the column reference.

CALENDAR

VAR VDates = CALENDAR(MINX(DimDate,DimDate[DateKey]),MAXX(DimDate,DimDate[DateKey]))

Next comes the VDates Variable. Calendar Returns a table with a single column named “Date” and it contains a contiguous set of dates. We need to provide the range of dates. In this case MINX finds the earliest date and MAXX finds the Maximum date in our date dimension

Now this should work with our slicer. In this case Min 01/09/2021 Max 30/09/2021

INTERSECT

INTERSECT(VActivityDates,VDates)

And now we get to the measure that will be RETURNED

“INTERSECT A table that contains all the rows in table_expression1 that are also in table_expression2”

COUNTROWS

COUNTROWS simply counts the number of rows within a table or an expression

IF

RETURN IF(COUNTROWS(INTERSECT(VActivityDates,VDates)),0,1)

So here we can see the intersect and if working together. If there are rows counted (True – There are rows then 0 else 1.

Still Struggling to understand? Me too. Lets draw it

Here is where I am a little confused. I’m not sure which part of the DAX works with the slicer. SELECTCOLUMNS or CALENDAR MINX MAXX? I’m making the assumption that CALENDAR takes the min and max of the slicer but its incredibly difficult to test so if anyone has any more insight on this that would be fantastic.

And we can now look at everything together in a report

We have used Is Not =1 in the ‘Is not in the Date Filter’ as out value and its always ever 0 or 1

But now we are asked another business question

How MANY reports did we not view in the current date slicer?

with the measure as it stands. It needs the Report Name or the Data set name to provide details

We can’t use what we have in a card because we cant add the implicit filter (Report Name etc).

So How can we do this one?

We can solve it using variables

Total Reports not in date Filter = 
//Clear all filters which are applied to the specified table.ALLCROSSFILTERED
var VAllReports =CALCULATE([# reports],ALLCROSSFILTERED(DimDate))
//Total within the date filtervar 
VInFilter = [# reports]
//All reports - Reports inside the slicer
Return VAllReports-VInFilter

Using ALLCROSSFILTERED brings back the total reports 4 as it removes all the filters

then we get back the total of reports in the date slicer and finally take in filter away from all the reports.

Out # reports is a measure made from distinct report IDs already. We now have everything we need to create these reports thanks to DAX.

Create your website with WordPress.com
Get started