Include Unity Catalog into the E2E Databricks sample

[ydaponte]

As a Data Engineer I would like to use the Unity Catalog on the Parking solution sample so I understand how it works and I don't need to work with the legacy Hive Metastore

DoD

• Successfully deploy and configure a Unity Catalog into the E2E Databricks + ADF sample
• All the tables should be generated under a "sensordata" metastore
• Hive metastore should not be used anymore
• Alter documentation accordingly

Ensure you've read the Contributing Guide (Microsoft Internal only.)

Top priority

• Modify ReadMe to reflect the pre-requisite of having a Regional Metastore - @thesqlpro [Unity Catalog] Modify ReadMe to reflect the pre-requisite of having a Regional Metastore #978
• Spike on impact of dbfs path change - @thesqlpro + hemaverma (might be oof) [Unity Catalog] Spike on impact of dbfs path change #979
• Change the import library process - ydaponte [Unity Catalog] - Change the import library process #980
• Upgrade Spark version to 15.4 LTS: @thesqlpro Upgrade Spark version to 15.4 LTS #896
• Research update creation of folder paths in ADLS to support Unity Catalog (POTENTIAL - might not be needed) @thesqlpro [Unity Catalog] - Research update creation of folder paths in ADLS to support Unity Catalog #981

Ready to be picked up when team is available (most probably Sprint 5)

• Change Setup.py notebook - @thesqlpro [Unity Catalog] - Change Setup.py notebook #982
• Change/test Standardize and Transform notebooks @thesqlpro [Unity Catalog] - Change/test Explore, Standardize and Transform notebooks #983
• Upgrade task of uploading Date/Time Dims to a notebook that creates them (deploy bash script) - @thesqlpro [Unity Catalog] - Upgrade task of uploading Date/Time Dims to a notebook that creates them (deploy bash script) #984

Nice to Have

• Document potential automation pieces (for future) for Unity Catalog [Unity Catalog] - Document potential automation pieces (for future) for Unity Catalog #985
• Create the Date/Time Dimensions programmatically in the SQL DB to remove reliance on Data file uploads (see [Unity Catalog] - Research update creation of folder paths in ADLS to support Unity Catalog #981 )

[ydaponte]

@thesqlpro - @hemaverma told me she is OOF the entire week. Can you please let me know if you will be able to work/finish this during this sprint or if it will spill to the next sprint? Thanks so much!

[thesqlpro]

Some notes from my research thus far:
Since Nov 2023 Unity Catalog is enabled by default (can check using this command in SQL editor SELECT CURRENT_METASTORE();)
Default name for the catalog is the name of the workspace (inherited - although it looks like hyphens are converted to underscores) - this will need to be addressed
Default is in the region the workspace was created (Ex. WS created in EastUS2 then Metastore is in EastUS2)
By default a legacy Hive Metastore exists - will research if this can be removed or not, so far looks like it is included for backward compatibility

[thesqlpro]

The Catalog will be named sensordata to be more generic than parkingsensor.