Data Discovery: Identify the SAP tables and fields used for existing Data Quality(DQ) rules & reports and locate the corresponding data in Databricks
Data Validation: Validate data across all the layers(bronze, silver,etc) in Databricks for completeness, structure and availability of required fields
DQ Rule Migration & BUILD: Select and build a set of existing DQ rules in Databricks as part of the PoC
PoC Testing & Reconciliation: Execute the rules in Databricks and compare results with the existing DQ results, investigating and resolving differences
End-to-End Rule Migration: Following the PoC, migrate / BUILD the agreed As-Is DQ rules to data bricks, including SQL logic, data mappings, rule outputs and required transformations
Data & Rule Validation: Perform end-to-end validation of source data, DQ rules, results and defects in Databricks to ensure the migrated process produces expected outcomes
DQ Reporting: Support the development of DQ outputs, KPIs and dashboard reporting from Databricks, including validation of report results
Documentation & BAU Handover: Document data mappings, DQ rules, validation results and migration processes
SQL & Pyspark/Python knowledge
Experience in Data Quality / Data Validation.
Experience working with SAP data, preferably master data
Good understanding of SAP tables
Experience with Databricks platform
Experience in data reconciliation and comparison between different data sources.
Ability to translate DQ requirements/rules into technical SQL or Python logic
Experience in DQ rule testing, defect analysis and troubleshooting
