Umesh K Email & Phone Number

United States

United States

Connecticut, United States

• Designed and implemented scalable data processing pipelines using Hadoop MapReduce, Hadoop Hive, and HDFS.• Designed and optimized SQL queries for complex data retrieval and analysis.• Designed and engineered data solutions, collaborating with onshore and offshore teams for seamless data migration.• Developed Snow Pipe and complex data transformations using Snow SQL, ensuring efficient data ingestion.• Optimized performance through Snowflake external tables, staging, and… Show more • Designed and implemented scalable data processing pipelines using Hadoop MapReduce, Hadoop Hive, and HDFS.• Designed and optimized SQL queries for complex data retrieval and analysis.• Designed and engineered data solutions, collaborating with onshore and offshore teams for seamless data migration.• Developed Snow Pipe and complex data transformations using Snow SQL, ensuring efficient data ingestion.• Optimized performance through Snowflake external tables, staging, and scheduler for enhanced data processing.• Leveraged Snowflake's Time Travel and zero-copy cloning features to ensure robust data recovery and duplication management.• Integrated Snowflake with AWS cloud services for scalable, secure, and high-performing data platforms.• Utilized AWS services such as S3 for Snowflake staging and data storage to enable real-time data processing.• Deployed and maintained Snowflake solutions within AWS infrastructure, optimizing performance and cost.• Implemented network policies, clustering, and tasks to secure and enhance Snowflake data environments.• Managed containerized environments using Docker and Kubernetes for scalable, microservice-based data solutions.• Collaborated with business analysts and technical stakeholders to align data solutions with business objectives.• Created and scheduled complex ETL processes using Airflow and Snowflake’s scheduler for streamlined data pipelines.• Ensured low-latency, streaming, and micro-batch data processing using AWS, Snowflake, and Snow Pipe.• Hands-on experience with AWS services such as Lambda, EC2, and RDS for deploying and managing data solutions.• Performed production support and maintenance for Snowflake solutions deployed within AWS environments.• Employed Airflow for orchestration and scheduling of ETL jobs in Snowflake and AWS environments. Show less

Bellevue, Wa

• Led a big data processing project using Apache Spark, Spark SQL, and Streaming for real-time data processing, ensuring efficient analysis.• Managed Hadoop system, oversaw data import from various sources, transformed it using Hive and MapReduce, and loaded it into HDFS.• Deployed StreamSets in enterprise environments to support mission-critical data integration and data management initiatives.• Designed and implemented scalable and resilient data pipelines using StreamSets Data… Show more • Led a big data processing project using Apache Spark, Spark SQL, and Streaming for real-time data processing, ensuring efficient analysis.• Managed Hadoop system, oversaw data import from various sources, transformed it using Hive and MapReduce, and loaded it into HDFS.• Deployed StreamSets in enterprise environments to support mission-critical data integration and data management initiatives.• Designed and implemented scalable and resilient data pipelines using StreamSets Data Collector, ensuring high availability and fault tolerance.• Built data ingestion Spark streaming framework, integrating data from sources like REST API and Kafka.• Developed robust data models and optimized ETL processes to enhance data quality and accessibility.• Wrote UDFs in Hadoop PySpark for streamlined transformations, ensuring efficient data processing.• Managed Snowflake warehouses, scheduled jobs using NiFi, and established CI/CD pipeline using Jenkins and Airflow.• Utilized Airflow for scheduling Hadoop jobs, optimizing resource utilization and improving workflow efficiency.• Played a key role in migrating objects from Oracle, SAP/HANA, MongoDB, and Teradata using a custom framework.• Collaborated with cross-functional teams to define data requirements and implement best practices in data engineering. Show less

Plano, Texas, United States

• Played a pivotal role in developing solutions to meet business requirements by creating data pipelines using Apache Spark, orchestrating the extraction, transformation, and loading of data from various sources like SQL databases and HDFS.• Implemented data integration solutions for on-premises and cloud-based data warehouses and data lakes using platforms such as Amazon Redshift, Google BigQuery, and Snowflake.• Orchestrated data replication and synchronization tasks between… Show more • Played a pivotal role in developing solutions to meet business requirements by creating data pipelines using Apache Spark, orchestrating the extraction, transformation, and loading of data from various sources like SQL databases and HDFS.• Implemented data integration solutions for on-premises and cloud-based data warehouses and data lakes using platforms such as Amazon Redshift, Google BigQuery, and Snowflake.• Orchestrated data replication and synchronization tasks between on-premises databases and big data platforms using StreamSets Data Collector and Control Hub.• Utilized various data sources and destinations to efficiently manage and orchestrate data workflows, ensuring seamless data movement and transformation.• Built data pipelines using PySpark to ingest data from CSV and JSON sources, ensuring data integrity and compliance with organizational standards.• Managed ETL data import from diverse sources and performed transformations using Hadoop Hive and MapReduce, facilitating the extraction of data from MySQL into HDFS and vice versa using Sqoop.• Implemented custom ETL processes to ensure efficient data processing, optimizing data movement and transformation workflows.• Developed notebooks using Apache Spark and PySpark to process and analyze large datasets, leveraging streaming capabilities to capture real-time data from various sources.• Designed and maintained data pipelines with various activities such as data validation, transformation, and loading, ensuring flexible and scalable data workflows. Show less

Minneapolis, Minnesota, United States

• Collaborated within Agile Scrum methodology for development, ensuring effective teamwork and project management, facilitating continuous delivery of high-quality data engineering solutions.• Utilized Sqoop, Kafka, and Hadoop File System APIs to implement robust data ingestion pipelines, ensuring reliable and efficient data processing and storage.• Handled Hadoop cluster installations across various environments (Unix, Linux), assisting in upgrading, configuring, and maintaining Hadoop… Show more • Collaborated within Agile Scrum methodology for development, ensuring effective teamwork and project management, facilitating continuous delivery of high-quality data engineering solutions.• Utilized Sqoop, Kafka, and Hadoop File System APIs to implement robust data ingestion pipelines, ensuring reliable and efficient data processing and storage.• Handled Hadoop cluster installations across various environments (Unix, Linux), assisting in upgrading, configuring, and maintaining Hadoop infrastructures such as Pig, Hive, and HBase.• Worked with Hadoop from Cloudera Data Platform, managing services through Cloudera Manager to ensure smooth operation and performance optimization of Hadoop clusters.• Created Hive tables, loaded data into them, and crafted efficient Hive queries to support data analysis and reporting initiatives.• Demonstrated hands-on experience in Hadoop administration and support activities, including configuring and managing Hadoop clusters to optimize performance.• Implemented partitions and utilized bucketing on Hive tables, optimizing performance parameters to enhance query execution and resource utilization.• Developed custom UDFs in Hive and Spark to enable tailored data processing and meet specific business requirements.• Developed Spark scripts using Python in the PySpark shell for efficient data processing and analysis across large datasets.• Imported millions of structured data records from relational databases using Sqoop, processed them using Spark, and ensured data integrity and consistency throughout the pipeline. Show less

Hyderabad, Telangana, India

* Implemented solutions for data ingestion from diverse sources and processing stored data using Big Data technologies like Hadoop, MapReduce frameworks, HBase, and Hive.* Efficiently utilized Sqoop to transfer data between databases and HDFS, ensuring seamless data integration and management.* Employed Flume to stream log data from servers, enabling real-time data ingestion and analysis for monitoring and troubleshooting.* Led efforts to load and transform substantial volumes of… Show more * Implemented solutions for data ingestion from diverse sources and processing stored data using Big Data technologies like Hadoop, MapReduce frameworks, HBase, and Hive.* Efficiently utilized Sqoop to transfer data between databases and HDFS, ensuring seamless data integration and management.* Employed Flume to stream log data from servers, enabling real-time data ingestion and analysis for monitoring and troubleshooting.* Led efforts to load and transform substantial volumes of structured, semi-structured, and unstructured data from relational databases into HDFS using Sqoop imports.* Implemented an enterprise-grade platform (MarkLogic) for Extract, Transform, Load (ETL) processes, facilitating data migration from the mainframe to NoSQL (Cassandra).* Oversaw the importation of log files from various sources into HDFS using Flume, conducting data analysis with HiveQL to generate payer reports for transmitting payment summaries to payers.* Spearheaded the design, architecture, development, and engineering of Big Data solutions, including installing and deploying Hadoop clusters and managing node operations.* Leveraged the Data Frame API in Python to manipulate distributed collections of data, enabling efficient data processing and analysis.* Implemented enhancements to traditional data warehouses based on STAR schema, updated data models, and performed data analytics and reporting using Tableau.* Contributed to the migration of data from existing RDBMS systems (Oracle and SQL Server) to Hadoop using Sqoop for data processing.* Developed Python scripts to automate and control the flow of Hive scripts, supporting various file formats such as CSV, JSON, Parquet, and HDFS.* Developed Hive SQL scripts for executing transformation logic and loading data from the staging zone to the landing zone and semantic zone, ensuring efficient data processing and management. Show less