Kate Williams (San Francisco, London) Email & Phone Number

Reading, Gb

London, England, United Kingdom

- Interact with business to capture business requirements, understand pain areas and challenges- Interact with IT stakeholders to understand system landscape, data sources & data journey- Identify use cases based on priority, and identify source systems and tables.- Design datalake Architecture, dataflow, data warehouse, identify tools & eco-systems, cloud hardware specification and Configuration.- Design data flow across Cloud, RDBMS, internal & external applications- configured Azure data lake (Gen2), Data factory, data bricks, delta files, and cosmosDB.- Configure Data Bricks on Cloud environment- Mount Data Bricks to Gen2 storage- Configure Spark cluster, versions, Nodes, Auto-Scaling etc.,- Data Ingestion, transformation and loading to Data Warehousing- Working with Notebooks, libraries- Extensive usage of PySpark Data frames and pandas for transformation.- Migrate & ingest data from disparate systems to cloud datalake and perform ETL- Align with business, data analysts, data scientists and Visualization teams.- Analyze data formats from various data sources, develop transformation logic that suites models from raw sets and store final output in COSMOS DB.- PySpark expert skills for ingestion, transformation & loading to transform massive data sets.- Working with DataBricks, Azure cloud storage and CI & CD.- Working with Azure DevOps, repository (git), power BI and QlikView.- Advisory and recommendations from strategy, solutions and technology to business.

- Design, develop Hadoop/Big Data infrastructure & landscape such as Servers, cloud Configuration etc.,- Design data flow across Cloud, RDBMS, internal & external applications- Design & implement Data lake, data flow - batch & streaming, security, Data warehousing.- Implement Hadoop eco-systems, Hive, Impala, oozie, spark, kafka- Develop Architecture, adopt development standards, governance process and access control.- Migrate from traditional analytics into Hadoop big data & cloud infrastructure.- Working with virtual servers, desktops etc.,- Configure Data Bricks on Cloud environment- Mount Data Bricks to Gen2 storage- Configure Spark cluster, versions, Nodes, Auto-Scaling etc.,- Data Ingestion, transformation and loading to Data Warehousing- Working with Notebooks, libraries- Align with business, data analysts, data scientists and Visualization teams.- Design data lake architecture and coordinate with segment architect for approval.- Leading the team of 6 data engineers and coordinate with Data science teams.- Analyze data formats from various data sets and understand requirements for models.- develop transformation logic that suites models from raw sets and store in COSMOS DB.- PySpark expert skills for ingestion, transformation & loading to transform massive data sets.- Working with DataBricks, Azure cloud storage and CI & CD.- Ingest Streaming & Batch data into azure datalake and store transformed data in DELTA file format, similar to parquet.- Extensive usage of PySpark Dataframes and pandas for transformation.- Working with Azure DevOps, repository (git) and Tablue for ETL.- Advisory and recommendations from strategy, solutions and technology to business.