Kumar D Email & Phone Number

Pleasanton, California, Us

Greenwood Village, Co, Us

• Responsible for ingesting large volumes of user behavioural data and customer profile data to Analytics Data store. • Developing ETL pipelines to move on-prem data (data sources that include Flat Files, Mainframe Files, and Databases) to AWS S3.• Created and embedded python modules in the ETL pipeline to automatically migrate data from S3 to Redshift using AWS Glue.• Developed PySpark and Scala based Spark applications for performing data cleaning, event enrichment, data aggregation, de-normalization and data preparation needed for machine learning and reporting teams to consume. • Wrote Spark-Streaming applications to consume the data from Kafka topics and write the processed streams to HBase and MongoDB.• Worked on fine-tuning spark applications to improve the overall processing time for the pipelines.• Wrote Kafka producers to stream the data from external rest APIs to Kafka topics.• Experienced in handling large datasets using Spark in memory capabilities, using broadcasts variables in Spark, effective & efficient Joins, transformations, and other capabilities.• Worked extensively with Sqoop for importing data from Oracle. • Created private cloud using Kubernetes that supports DEV, TEST, and PROD environments.• Written HBase bulk load jobs to load processed data to Hbase tables by converting to Hfiles.• Designing and customizing data models for Data warehouse supporting data from multiple sources on real time.• Experience working for EMR cluster in AWS cloud and working with S3, Redshift, Snowflake.• Involved in creating Hive tables, loading, and analysing data using hive scripts. • Implemented Partitioning, Dynamic Partitions, Buckets in Hive. • Designed, documented operational problems by following standards and procedures using JIRA.• Used Reporting tools like Tableau to connect with Impala for generating daily reports of data. .

Rochester, Minnesota, Us

•Designing and Developing Apache NiFi jobs to get the files from transaction systems into data lake raw zone. •Experienced in Databricks platform where it follows best practices for securing network access to cloud applications.•Analysed the SQL scripts and designed it by using PySpark SQL for faster performance.•Worked on developing ETL processes (Data Stage Open Studio) to load data from multiple data sources to HDFS using FLUME and SQOOP, and performed structural modifications using Map Reduce HIVE. •Implemented large Lambda architectures using Azure Data platform capabilities like Azure Data Lake, Azure Data Factory, Azure Data Catalog, HDInsight, Azure SQL Server, Azure ML, and Power BI.•Used Azure Databricks for fast, easy, and collaborative spark-based platform on Azure.

Roseland, New Jersey, Us

•Responsible for building an Enterprise Data Lake to bring ML ecosystem capabilities to production and make it readily consumable for data scientists and business users.•Processing and transforming the data using AWS EMR to assist the Data Science team as per business requirements.•Developing Spark applications for cleaning and validation of the ingested data into the AWS cloud.•Designing and building multi-terabyte, full end-to-end Data Warehouse infrastructure from the ground up on Confidential Redshift.•Design Develop and test ETL Processes in AWS Glue to migrate Campaign data from external sources like S3, ORC/Parquet/Text Files into AWS Redshift.•Develop Terraform scripts to create the AWS resources such as EC2, Auto Scaling Groups, ELB, S3, SNS and Cloud Watch Alarms.•Implement Serverless architecture using AWS Lambda with Amazon S3 and Amazon Dynamo DB.•Develop Spark programs with PySpark and applied principles of functional programming to process the complex unstructured and structured data sets. Processed the data with Spark from Hadoop Distributed File System (HDFS).

San Diego, Ca, Us

•Worked on development of data ingestion pipelines using ETL tool, Talend & bash scripting with big data technologies including but not limited to Hive, Impala, Spark, Kafka, and Talend.•Importing data from MS SQL Server and Teradata into HDFS using Sqoop.•Implemented data streaming capability using Kafka and Talend for multiple data sources.•Working knowledge of cluster security components like Kerberos, Sentry, SSL/TLS, etc.•Worked with multiple storage formats (Avro, Parquet) and databases (Hive, Impala, Kudu).•Implemented UNIX scripts to define the use case workflow and to process the data files and automate the jobs.•Worked with SCRUM team in delivering agreed user stories on time for every Sprint.•Involved in creating Hive external tables to perform ETL on data that is produced on daily basis.