Kumar D Email & Phone Number

Pleasanton, California, US

Greenwood Village, CO, US

• Responsible for ingesting large volumes of user behavioural data and customer profile data to Analytics Data store.
• Developing ETL pipelines to move on-prem data (data sources that include Flat Files, Mainframe Files, and Databases) to AWS S3.
• Created and embedded python modules in the ETL pipeline to automatically migrate data from S3 to Redshift using AWS Glue.
• Developed PySpark and Scala based Spark applications for performing data cleaning, event enrichment, data aggregation, de-normalization and data preparation needed for machine learning and reporting teams to consume.
• Wrote Spark-Streaming applications to consume the data from Kafka topics and write the processed streams to HBase and MongoDB.
• Worked on fine-tuning spark applications to improve the overall processing time for the pipelines.

Rochester, Minnesota, US

• Designing and Developing Apache NiFi jobs to get the files from transaction systems into data lake raw zone.
• Experienced in Databricks platform where it follows best practices for securing network access to cloud applications.
• Analysed the SQL scripts and designed it by using PySpark SQL for faster performance.
• Worked on developing ETL processes (Data Stage Open Studio) to load data from multiple data sources to HDFS using FLUME and SQOOP, and performed structural modifications using Map Reduce HIVE.
• Implemented large Lambda architectures using Azure Data platform capabilities like Azure Data Lake, Azure Data Factory, Azure Data Catalog, HDInsight, Azure SQL Server, Azure ML, and Power BI.
• Used Azure Databricks for fast, easy, and collaborative spark-based platform on Azure.

Roseland, New Jersey, US

• Responsible for building an Enterprise Data Lake to bring ML ecosystem capabilities to production and make it readily consumable for data scientists and business users.
• Processing and transforming the data using AWS EMR to assist the Data Science team as per business requirements.
• Developing Spark applications for cleaning and validation of the ingested data into the AWS cloud.
• Designing and building multi-terabyte, full end-to-end Data Warehouse infrastructure from the ground up on Confidential Redshift.
• Design Develop and test ETL Processes in AWS Glue to migrate Campaign data from external sources like S3, ORC/Parquet/Text Files into AWS Redshift.
• Develop Terraform scripts to create the AWS resources such as EC2, Auto Scaling Groups, ELB, S3, SNS and Cloud Watch Alarms.

San Diego, CA, US

• Worked on development of data ingestion pipelines using ETL tool, Talend & bash scripting with big data technologies including but not limited to Hive, Impala, Spark, Kafka, and Talend.
• Importing data from MS SQL Server and Teradata into HDFS using Sqoop.
• Implemented data streaming capability using Kafka and Talend for multiple data sources.
• Working knowledge of cluster security components like Kerberos, Sentry, SSL/TLS, etc.
• Worked with multiple storage formats (Avro, Parquet) and databases (Hive, Impala, Kudu).
• Implemented UNIX scripts to define the use case workflow and to process the data files and automate the jobs.