• 9+ years of experience as a software developer, with strong emphasis on data engineering, data analysis and application development using Java.• 6+ Years of experience in utilizing Big data tools for building and automating large scale data pipelines.• Strong programming experiencing in developing rest services, automation scripts and data engineering jobs using Python, Scala, and Java.• Good understanding on architecture of Distributed Systems and Parallel processing frameworks for scalable data storage and distributed data processing.• Utilized various Hadoop ecosystem tools like HDFS, MapReduce, Hive, Yarn, Kafka, Oozie, Sqoop and HBase.• Strong experience using Spark framework for performing various data cleansing, data enrichment, data aggregation activities.• Strong experience using Scala and Python for building end to end spark applications.• Strong experience working with AWS Cloud services for building data lakes and automating the data pipelines.• Responsible for developing data pipelines using Azure data factory involving investigating raw JSON files, transactional and user profile information from on prem data warehouses and processing them using PySpark and Scala.• Written Python, PySpark, Scala transformations using Jupyter and Databricks on data loaded in Azure Data Lake Storage• Utilized Spark RDD APIs, Spark Dataframes, Spark SQL and Spark Streaming APIs.• Experienced in writing custom Hive UDF’s to incorporate Business logic with Hive Queries.• Strong experience using No-Sql Databases like HBase and Dynamo DB.• Experience in JIRA and tracked the test results and interacted with the developers to resolve issue.• Strong Java developer and developed Rest applications using Spring Boot.• Detailed understanding of Software Development Life Cycle (SDLC) and sound knowledge of project implementation methodologies including Scrum, Waterfall and Agile.
