Advent of 2023, Day 21 – Monitoring workspaces with Fabric

In this Microsoft Fabric series:

1. Dec 01: What is Microsoft Fabric?
2. Dec 02: Getting started with Microsoft Fabric
3. Dec 03: What is lakehouse in Fabric?
4. Dec 04: Delta lake and delta tables in Microsoft Fabric
5. Dec 05: Getting data into lakehouse
6. Dec 06: SQL Analytics endpoint
7. Dec 07: SQL commands in SQL Analytics endpoint
8. Dec 08: Using Lakehouse REST API
9. Dec 09: Building custom environments
10. Dec 10: Creating Job Spark definition
11. Dec 11: Starting data science with Microsoft Fabric
12. Dec 12: Creating data science experiments with Microsoft Fabric
13. Dec 13: Creating ML Model with Microsoft Fabric
14. Dec 14: Data warehouse with Microsoft Fabric
15. Dec 15: Building warehouse with Microsoft Fabric
16. Dec 16: Creating data pipelines for Fabric data warehouse
17. Dec 17: Exploring Power BI in Microsoft Fabric
18. Dec 18: Exploring Power BI in Microsoft Fabric
19. Dec 19: Event streaming with Fabric
20. Dec 20: Working with notebooks in Fabric

Monitoring workspaces, executions and checking logs is so quintessential, that one should get familiarized with this in the first place.

Monitoring Hub

The easy way to check, view and track your activities and execution and runs of notebooks, data pipelines, data factory executions, datasets refresh, and many others.

With filters and using search, you can find the desired logs and check the the details.

For example, check the notebook executions(which are run on Spark architecture) and double-click on the ellipses, you will get detailed information on the Spark engine, job runs, resources used, data touched (used, created,…) and snapshot items.

Should you want to examine each job run, you can double-click it and the Spark UI will open. You will be redirected to a standard URL: https://sparkui.fabric.microsoft.com/sparkui/ where further data will be displayed. Fabric Spark UI will behave as your normal Spark UI and will get the exact same experience as with any other Spark installation. And I am personally happy that Fabric gives you all these capabilities to dig deeper into Spark. And for a particular run you can go deep into the details as Spark has to offer. For example, previewing the Spark Applications and Job Graph.

Several of these diagnostics are also available within the notebook. After each execution of the cell (in the notebook), there is Log, Diagnostics and Job Runs information available and the same information can be shown as these in Spark UI.

Admin monitoring workspace

Admin workspace is available when you are granted the privileges.

Within this workspace, you will have the feature Usage and adoption, as well as the Purview Hub (Yaay! 🙂 )

You will have an overview of feature usage within your tenant. Besides activities, you can also check the analyse the usage activity logs in details.

Dataset and the report can always be edited, upgraded or tailored to your need.

The Purview hub report gives you the ability to check:

• Overview report: Overview of distribution and use of endorsement and sensitivity labeling.
• Endorsement report: Drill down and analyze distribution and use of endorsement.
• Sensitivity report: Drill down and analyze distribution and use of sensitivity labeling.
• Inventory report: Get details about labeled and endorsed items. Can apply date ranges, filter by workspace, item type, etc.
• Items page: Insights about the distribution of items throughout your organization, and endorsement coverage.
• Sensitivity page: Insights about sensitivity labeling throughout your entire organization.

Accessing purview and getting hands-on Purview is worth couple of blogposts and covers variety of super important topics, that should be relevant for every enterprise!

User Activity

Tracking user activity is not that straightforward in Fabric. One can use the Power BI Powershell modules to track the activities in workspaces, users, and all Power BI Items. This can be done using cdmlet:

Get-PowerBIActivityEvent

or Audit log in the Admin portal. I have written extensively using Powershell for Power BI (Blogposts; Github Repository). But this can also be used against all other Fabric items but only for CRUD operations.

And if you want to check all the user sign-in activities, Entra ID (or formally known as Azure AD) is also a good starting point.

Tomorrow we will look into Fabric Apps.

Complete set of code, documents, notebooks, and all of the materials will be available at the Github repository: https://github.com/tomaztk/Microsoft-Fabric

Happy Advent of 2023! 🙂