{"id":380773,"date":"2023-12-11T15:14:29","date_gmt":"2023-12-11T21:14:29","guid":{"rendered":"http:\/\/tomaztsql.wordpress.com\/?p=9622"},"modified":"2023-12-11T15:14:29","modified_gmt":"2023-12-11T21:14:29","slug":"advent-of-2023-day-11-starting-data-science-with-microsoft-fabric","status":"publish","type":"post","link":"https:\/\/www.r-bloggers.com\/2023\/12\/advent-of-2023-day-11-starting-data-science-with-microsoft-fabric\/","title":{"rendered":"Advent of 2023, Day 11 \u2013 Starting data science with Microsoft Fabric"},"content":{"rendered":"\r\n\r\n
\r\n[This article was first published on R \u2013 TomazTsql<\/a><\/strong>, and kindly contributed to R-bloggers<\/a>]. (You can report issue about the content on this page here<\/a>)\r\n
Want to share your content on R-bloggers? click here<\/a> if you have a blog, or here<\/a> if you don't.\r\n<\/div>\n\n

In this Microsoft Fabric series:<\/p>\n\n\n\n

    \n
  1. Dec 01:\u00a0What is Microsoft Fabric?<\/a><\/li>\n\n\n\n
  2. Dec 02:\u00a0Getting started with Microsoft Fabric<\/a><\/li>\n\n\n\n
  3. Dec 03:\u00a0What is lakehouse in\u00a0Fabric?<\/a><\/li>\n\n\n\n
  4. Dec 04:\u00a0Delta lake and delta tables in Microsoft\u00a0Fabric<\/a><\/li>\n\n\n\n
  5. Dec 05:\u00a0Getting data into lakehouse<\/a><\/li>\n\n\n\n
  6. Dec 06:\u00a0SQL Analytics\u00a0endpoint<\/a><\/li>\n\n\n\n
  7. Dec 07:\u00a0SQL commands in SQL Analytics\u00a0endpoint<\/a><\/li>\n\n\n\n
  8. Dec 08: Using Lakehouse REST\u00a0API<\/a><\/li>\n\n\n\n
  9. Dec 09: Building custom environments<\/a><\/li>\n\n\n\n
  10. Dec 10: Creating Job Spark definition<\/a><\/li>\n<\/ol>\n\n\n\n

    We have looked into creating the lakehouse, checked the delta lake and delta tables, got some data into the lakehouse, and created a custom environment and Spark job definition. And now we need to see, how to start working with the data.<\/p>\n\n\n\n

    There are four capabilities that one can explore, and since we have covered the Environment, let\u2019s go with the notebook capability.<\/p>\n\n\n\n

    \"\"<\/a><\/figure>\n\n\n\n

    Once you create a new (or open) notebook, add the lakehouse. I will add the existing one, we created on Day 3 \u2013 called Advent2023. You can also choose the preferred language of the notebook \u2013 PySpark (Python), Spark (Scala), Spark SQL and SparkR (R). On top of the language, you can also choose the environment; the default workspace or any custom-prepared environment (with custom libraries and compute).<\/p>\n\n\n\n

    Get the data into notebook, by reading the delta table.<\/p>\n\n\n

    \n# PySpark - show data\ndf = spark.read.load('Tables\/iris_data',\n    format='delta',\n    header=True\n)\ndisplay(df.limit(10))\n<\/pre>\n\n\n
    You can also get the data using SQL that is wrapped in PySpark command.<\/p>\n\n\n
    \ndf = spark.sql(\n    """\n    SELECT \n         CAST(`Sepal.Length` AS DECIMAL(5,1)) AS SepalLength\n        ,CAST(`Sepal.Width` AS  DECIMAL(5,1))  AS SepalWidth\n        ,CAST(`Petal.Length` AS  DECIMAL(5,1)) AS PetalLength\n        ,CAST(`Petal.Width` AS  DECIMAL(5,1)) AS PetalWidth\n       ,Species\n    FROM\n    Advent2023.iris_data LIMIT 3000\n    """\n    )\n\ndf.show()\n<\/pre>\n\n\n
    You can always visualize the data:<\/p>\n\n\n
    \nimport pandas as pd\nimport warnings\nwarnings.filterwarnings("ignore")\nimport seaborn as sns\nimport matplotlib.pyplot as plt\n\ndf2 = df.toPandas()\n\ndf2.plot(kind="scatter", x="SepalLength", y="SepalWidth")\nplt.show()\n\nsns.jointplot(x="SepalLength", y="SepalWidth", data=df2, size=5)\nplt.show()\n<\/pre>\n\n\n
    And the the visual perspective with great Seaborn joinplot:<\/p>\n\n\n\n
    \"\"<\/a><\/figure>\n\n\n\n
    And now do some features engineering. Create a Vector Assembler:<\/p>\n\n\n
    \nvectorAssembler = VectorAssembler(inputCols = ['SepalLength','SepalWidth','PetalLength','PetalWidth'], outputCol = 'features')\nv_iris_df = vectorAssembler.transform(df)\nv_iris_df.show(5)\n<\/pre>\n\n\n
    \"\"<\/a><\/figure>\n\n\n\n
    And convert the new feature to string:<\/p>\n\n\n
    \nindexer = StringIndexer(inputCol = 'Species', outputCol = 'label')\ni_v_iris_df = indexer.fit(v_iris_df).transform(v_iris_df)\ni_v_iris_df.show(5)\n<\/pre>\n\n\n
    Now, do the split:<\/p>\n\n\n
    \nsplits = i_v_iris_df.randomSplit([0.6,0.4],1)\ntrain_df = splits[0]\ntest_df = splits[1]\ntrain_df.count(), test_df.count(), i_v_iris_df.count()\n<\/pre>\n\n\n
    Load the Perceptron classifier and the evaluator function for evaluating the model.<\/p>\n\n\n
    \nfrom pyspark.ml.classification import MultilayerPerceptronClassifier\nfrom pyspark.ml.evaluation import MulticlassClassificationEvaluator\n<\/pre>\n\n\n
    Now, let\u2019s play with the neural network and create the layer definition. We are using two hidden layers of 5 nodes each and hence our layers array is [4,5,5,3] (input-4, 2 x hidden-5, output nodes-3). And do the fitting-<\/p>\n\n\n
    \nlayers = [4,5,5,3]\nmlp = MultilayerPerceptronClassifier(layers = layers, seed = 1)\n#fit\nmlp_model = mlp.fit(train_df)\n<\/pre>\n\n\n
    Once the training is completed, use the transform method on the test data frame using the model object from the previous step. We store the results in a data frame called pred_df<\/em> review some of the columns and check the probabilities<\/p>\n\n\n
    \npred_df = mlp_model.transform(test_df)\npred_df.select('features','label','rawPrediction','probability','prediction').show(10)\n\n<\/pre>\n\n\n
    And finally, evaluate the model:<\/p>\n\n\n
    \nevaluator = MulticlassClassificationEvaluator(labelCol = 'label', predictionCol = 'prediction', metricName = 'accuracy')\nmlpacc = evaluator.evaluate(pred_df)\nmlpacc\n<\/pre>\n\n\n
    And the results are pretty good:<\/p>\n\n\n\n
    \"\"<\/a><\/figure>\n\n\n\n
    Tomorrow we will look the continue with data science!<\/p>\n\n\n\n
    Complete set of code, documents, notebooks, and all of the materials will be available at the Github repository:\u00a0https:\/\/github.com\/tomaztk\/Microsoft-Fabric<\/a><\/p>\n\n\n\n
    Happy Advent of 2023! \"\ud83d\ude42\"<\/p>\n\n
    \r\n
    To leave a comment<\/strong> for the author, please follow the link and comment on their blog:  R \u2013 TomazTsql<\/a><\/strong>.<\/div>\r\n
    \r\n    R-bloggers.com<\/a> offers daily e-mail updates<\/a><\/strong> about R<\/a> news and tutorials about learning R<\/a> and many other topics. Click here if you're looking to post or find an R\/data-science job<\/a>.\r\n\r\n
    Want to share your content on R-bloggers?     click here<\/a> if you have a blog, or  here<\/a> if you don't.\r\n<\/div>","protected":false},"excerpt":{"rendered":"
     In this Microsoft Fabric series: We have looked into creating the lakehouse, checked the delta lake and delta tables, got some data into the lakehouse, and created a custom environment and Spark job definition. And now we need to see,\u2026Read more \u203a<\/div>\n
    <\/div>\n
    <\/div>\n","protected":false},"author":1281,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":[],"categories":[4],"tags":[],"aioseo_notices":[],"jetpack-related-posts":[],"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/www.r-bloggers.com\/wp-json\/wp\/v2\/posts\/380773"}],"collection":[{"href":"https:\/\/www.r-bloggers.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.r-bloggers.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.r-bloggers.com\/wp-json\/wp\/v2\/users\/1281"}],"replies":[{"embeddable":true,"href":"https:\/\/www.r-bloggers.com\/wp-json\/wp\/v2\/comments?post=380773"}],"version-history":[{"count":7,"href":"https:\/\/www.r-bloggers.com\/wp-json\/wp\/v2\/posts\/380773\/revisions"}],"predecessor-version":[{"id":385281,"href":"https:\/\/www.r-bloggers.com\/wp-json\/wp\/v2\/posts\/380773\/revisions\/385281"}],"wp:attachment":[{"href":"https:\/\/www.r-bloggers.com\/wp-json\/wp\/v2\/media?parent=380773"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.r-bloggers.com\/wp-json\/wp\/v2\/categories?post=380773"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.r-bloggers.com\/wp-json\/wp\/v2\/tags?post=380773"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}