Derek K Email & Phone Number

Phoenix, Arizona, United States

Exposure to Full Lifecycle (SDLC) of Data Warehouse projects including Dimensional Data Modeling. Worked with building data warehouse structures, and creating facts, dimensions, aggregate tables, by dimensional modeling, Star and Snowflake schemas. Extract Transform and Load data from Sources Systems to Azure Data Storage services using a combination of Azure Data Factory, Spark SQL, and U-SQL Azure Data Lake Analytics. Data Ingestion to one or more Azure Services - (Azure Data Lake… Show more Exposure to Full Lifecycle (SDLC) of Data Warehouse projects including Dimensional Data Modeling. Worked with building data warehouse structures, and creating facts, dimensions, aggregate tables, by dimensional modeling, Star and Snowflake schemas. Extract Transform and Load data from Sources Systems to Azure Data Storage services using a combination of Azure Data Factory, Spark SQL, and U-SQL Azure Data Lake Analytics. Data Ingestion to one or more Azure Services - (Azure Data Lake, Azure Storage, Azure SQL, Azure DW) and processing the data in In Azure Databricks. Developed JSON Scripts for deploying the Pipeline in Azure Data Factory (ADF) that process the data using the Sql Activity. As a Data Engineer/Data Modeler designed and deployed scalable, highly available, and fault tolerant systems on Azure. Experienced in writing and deploying UNIX Korn Shell Scripts as part of the standard ETL processes and for job automation purposes. Implement modules into NodeJs to integrate with designs and requirements. Show less

Dallas, Texas, United States

Responsible for the design, implementation, and architecture of very large-scale data intelligence solutions around big data platforms. Analyzed large and critical datasets using HDFS, HBase, Hive, HQL, Pig, Sqoop and Zookeeper. Developed multiple POC’s using Spark, Scala and deployed on the Yarn Cluster, compared the performance of Spark, with Hive and SQL. Use Amazon Elastic Cloud Compute (EC2) infrastructure for computational tasks and Simple Storage Service (S3) as storage… Show more Responsible for the design, implementation, and architecture of very large-scale data intelligence solutions around big data platforms. Analyzed large and critical datasets using HDFS, HBase, Hive, HQL, Pig, Sqoop and Zookeeper. Developed multiple POC’s using Spark, Scala and deployed on the Yarn Cluster, compared the performance of Spark, with Hive and SQL. Use Amazon Elastic Cloud Compute (EC2) infrastructure for computational tasks and Simple Storage Service (S3) as storage mechanism. Developed Rest API using Flask Framework (python) for the front end (UI) to consume. Migrated previously written cron jobs to airflow/composer in GCP. Show less

Dallas, Texas, United States

Executed entire Data Life Cycle and actively involved in all the phases of project life cycle including data acquisition, data cleaning, data engineering, feature scaling, feature engineering and statistical modeling. Created multiple workbooks, dashboards, and charts using calculated fields, quick table calculations, Custom hierarchies, sets & parameters to meet business needs using Tableau. Create HIVE queries to join multiple tables of a source system and load them into Elastic… Show more Executed entire Data Life Cycle and actively involved in all the phases of project life cycle including data acquisition, data cleaning, data engineering, feature scaling, feature engineering and statistical modeling. Created multiple workbooks, dashboards, and charts using calculated fields, quick table calculations, Custom hierarchies, sets & parameters to meet business needs using Tableau. Create HIVE queries to join multiple tables of a source system and load them into Elastic Search Tables and used HIVE QL scripts to perform the incremental loads. Developed ETL jobs using Python and SQL for data extraction, transformation, and loading from various sources into AWS data lake and data warehouse, ensuring data quality and integrity. Performed Real time event processing of data from multiple servers in the organization using Apache Storm by integrating with Apache Kafka. Involved in development of full life cycle implementation of ETL using Informatica, Oracle and helped with designing the Date warehouse by defining Facts, Dimensions and relationships between them and applied the Corporate Standards in Naming Conventions. Show less