Lava Kumar Email & Phone Number

Michigan, United States

● Led the architecture and implementation of Google Cloud Platform (GCP) solutions, focusing on GCP Big Query, Python, and SQL, aligning with the client data engineering requirements.● Developed robust ETL processes, leveraging Python scripting for data extraction, transformation, and loading into GCP Big Query, ensuring optimal performance and efficiency.● Utilized Agile methodologies throughout the Software Development Life Cycle (SDLC), emphasizing flexibility and responsiveness to… Show more ● Led the architecture and implementation of Google Cloud Platform (GCP) solutions, focusing on GCP Big Query, Python, and SQL, aligning with the client data engineering requirements.● Developed robust ETL processes, leveraging Python scripting for data extraction, transformation, and loading into GCP Big Query, ensuring optimal performance and efficiency.● Utilized Agile methodologies throughout the Software Development Life Cycle (SDLC), emphasizing flexibility and responsiveness to evolving project requirements.● Worked on development of data ingestion pipelines using ETL tool, Talend & amp; bash scripting with big data technologies including Hive, Spark, Kafka, and Talend.● Created Databricks notebooks using SQL, Python and automated notebooks using jobs.● Responsible for estimating the cluster size and troubleshooting of the Spark data bricks cluster. ● Designed and implemented the Enterprise Data Warehouse and Analytic project from scratch utilizing cloud- based infrastructure and the Data Vault 2.0 design pattern to ensure agility and reliability.● Designed, developed and Implemented ETL Process using IICS Data Integration.● Collaborated closely with cross-functional teams, including data scientists, business analysts, and client stakeholders, to understand requirements and deliver tailored data solutions.● Developed data pipeline using Flume, Sqoop to ingest customer behavioral data into HDFS for analysis.● Used Python and Django creating graphics, XML processing, data exchange and business logic implementation.● Used Informatica Power Center for extraction, transformation, and loading (ETL) of data in the data warehouse.● Conducted regular performance tuning of SQL queries, Spark jobs, and data pipelines to optimize resource usage and reduce processing time.● Utilized tools like Cloud Monitoring and Logging for proactive system monitoring and issue resolution. Show less

Texas, United States

● Worked as a Sr. Data Engineer with Big Data and Hadoop ecosystem components.● Involved in converting Hive/SQL queries into Spark transformations using Scala.● Created Spark data frames using Spark SQL and prepared data for data analytics by storing it in AWS S3.● Responsible for loading data from Kafka into HBase using REST API.● Developed batch scripts to fetch the data from AWS S3 storage and perform required transformations in Scala using Spark framework and AWS Glue… Show more ● Worked as a Sr. Data Engineer with Big Data and Hadoop ecosystem components.● Involved in converting Hive/SQL queries into Spark transformations using Scala.● Created Spark data frames using Spark SQL and prepared data for data analytics by storing it in AWS S3.● Responsible for loading data from Kafka into HBase using REST API.● Developed batch scripts to fetch the data from AWS S3 storage and perform required transformations in Scala using Spark framework and AWS Glue ETL.● Utilized AWS Kinesis services for analytics using Apache Flink and Firehose to load data into target destinations with high availability and durability.● Used Spark streaming APIs to perform transformations and actions on the fly for building a common learner data model which gets the data from Kafka in near real time and persists it to the HBase.● Created Sqoop scripts to import and export customer profile data from RDBMS to S3 buckets.● Developed various enrichment applications in Spark using Scala for cleansing and enrichment of clickstream data with customer profile lookups.● Troubleshooted Spark applications for improved error tolerance and reliability.● Used Spark Data frame and Spark API to implement batch processing of Jobs.● Automated creation and termination of AWS EMR clusters.● Worked on fine tuning and performance enhancements of various spark applications and hive scripts.● Used various concepts in spark like broadcast variables, caching, dynamic allocation to design more scalable spark applications.● Maintained Kubernetes patches and upgrades.● Managed multiple Kubernetes clusters in a production environment.● Identify source systems, their connectivity, related tables, and fields and ensure data suitability for mapping, preparing unit test cases, and provide support to the testing team to fix defects. Show less

Hyderabad, Telangana, India

● Involved in Analysis, Design, and Implementation/translation of Business User requirements.● Worked on collection of large sets of Structured and Unstructured data using Python Script.● Involved in designing and developing data ingestion, aggregation, and integration into Hadoop.● Developed Sqoop scripts to import export data from relational sources and handled incremental loadingon the customer, transaction data by date.● Identified inconsistencies in data collected from… Show more ● Involved in Analysis, Design, and Implementation/translation of Business User requirements.● Worked on collection of large sets of Structured and Unstructured data using Python Script.● Involved in designing and developing data ingestion, aggregation, and integration into Hadoop.● Developed Sqoop scripts to import export data from relational sources and handled incremental loadingon the customer, transaction data by date.● Identified inconsistencies in data collected from different sources.● Designed object model, data model, tables, constraints, necessary stored procedures, functions, triggers, and packages for Oracle Database.● Wrote Spark applications for Data validation, cleansing, transformations, and custom aggregations.● Worked on installing cluster, commissioning & decommissioning of Data node, Name node high availability, capacity planning, and slots configuration.● Developed Spark applications for the entire batch processing by using Scala.● Stored time-series transformed data from Spark engine built on top of a Hive platform to S3.● Visualized the results using Tableau dashboards and the Python Seaborn libraries in deployment.● Provided insights and recommendations and implemented changes for positive business growth. Show less

Hyderabad, Telangana, India

● Created web-based applications using Python on Django framework for data processing.● Implemented preprocessing procedures along with deployment by creating virtual machines using EC2.● Analyzed user behavior data and collaborated with program managers, business analysts, developers and other key stakeholders to develop effective product solutions.● Extracted data from S3 through SQL database, performed ETL using AWS Glue and utilized kinesis Apache Flink for data… Show more ● Created web-based applications using Python on Django framework for data processing.● Implemented preprocessing procedures along with deployment by creating virtual machines using EC2.● Analyzed user behavior data and collaborated with program managers, business analysts, developers and other key stakeholders to develop effective product solutions.● Extracted data from S3 through SQL database, performed ETL using AWS Glue and utilized kinesis Apache Flink for data analytics.● Applied SQL Queries, procedures for data manipulation, extraction, and analysis for product optimization.● Created interactive dashboards using Power BI for data visualization, by translating complex model outputs into engaging, visual reports and enhanced business decision-making, and actions.● Defined data needs, evaluated data quality, and extracted/transformed data for analytic projects and research.● Designed and maintained databases using Python and developed Python based API (RESTful Web Service) using Flask, andPostgreSQL.● Worked on server-side applications using Python programming.● Employed Visual Studio Code, and Jupyter Notebook to streamline code writing and debugging processes, reducing development time and improving overall code quality.● Conducted through software maintenance, testing, and troubleshooting, to ensure smooth operations.● Researched and identified industry trends, providing valuable production improvement recommendations.● Performed efficient delivery of code and continuous integration to keep in line with Agile principles.● Experienced in Agile framework, from sprint planning and meetings to retrospectives, product backlog management and writing user stories.● Researched and identified industry trends, providing valuable production improvement recommendations and maintained program libraries, user's manuals and technical documentation.● Contributed to database management, cleaning, and optimization, resulting in a positive customer experience. Show less