Designed and implemented ETL pipelines using Apache Airflow and astronomer.ioManaged Data lake using Google Cloud Platform and Microsoft Azure. Created complex SQL queries, API calls, and transformations using Object oriented programming in Python. Worked on a interdisciplinary scrum team with daily meetings and high levels of team collaboration.

As a Technical Product Lead I facilitate Lambda students foray into the professional world using their technical skills while developing intrapersonal skills. We work with non profit organizations to bring the change we desire to the world, while preparing students for their future endeavors.Involved in designing and implementing the web application using HTML5, CSS3, Bootstrap, React, Redux and AJAX with RESTful web service.• Involved in Big data requirement analysis, develop and design solutions for ETL and Business Intelligence platforms.• Load real time data from various data sources into PostgreSQL using AWS S3.• Worked on reading multiple data formats on HDFS using python.• Implemented Spark using Python (pySpark) and SparkSQL for faster testing and processing of data.• Load the data into Spark RDD and do in memory data Computation.• Involved in converting Hive/SQL queries into Spark transformations using API’s like Spark SQL, Data Frames and python.• Analyzed the SQL scripts and designed the solution to implement using python.• Exploring the Spark by improving the performance and optimization of the existing algorithms in Hadoop using Spark Context, SparkSQL, Data Frame, Pair RDD & Spark YARN.• Performed transformations, cleaning and filtering on imported data using Spark Data Frame API, Hive, MapReduce, and loaded final data into Hive .• Involved in converting Map Reduce programs into Spark transformations using Spark RDD on python.• Developed Spark scripts by using python Shell commands as per the requirement.• Worked with NoSQL databases like HBase in creating tables to load large sets of semi structured data coming from source systems.• Design and develop the HBase target schema.

• Involved in designing and implementing the web application using HTML5, CSS3, Bootstrap, React, Redux and AJAX with RESTful web service.• Responsible for a setup of 5 node development cluster for a Proof of Concept which was later implemented as a fulltime project by Fortune Brands.• Responsible for Installation and configuration of Hive, Sqoop, Zookeeper, Knox and Oozie on the Hortonworks Hadoop cluster using Ambari.• Involved in extracting large sets of structured, semi structured and unstructured data.• Developed Sqoop scripts to import data from Oracle database and handled incremental loading on the point of sale tables.• Created Hive external tables and views, on the data imported into the HDFS.• Developed and implemented Hive scripts for transformations such as evaluation, filtering and aggregation.• Worked on partitioning Hive tables and running the scripts in parallel to reduce run-time of the scripts.• Developed User Defined Functions (UDF) in Java if required for hive queries.• Worked with data in multiple file formats including Avro, Parquet, Sequence files, ORC and Text/ CSV.• Used Oozie Operational Services for batch processing and scheduling workflows dynamically.• Worked on creating End-End data pipeline orchestration using Oozie.• Developed bash scripts to automate the above process of Extraction, Transformation and Loading.• Implemented Authentication and Authorization using Kerberos, Knox and Apache Ranger.• Very good experience in managing the Hadoop cluster using Ambari.• Created roles and user groups in Ambari for permitted access to Ambari functions.• Working knowledge of Mapreduce and YARN architectures.• Working knowledge on Zoo Keeper.

Worked closely with the project superintendents, vendors, and crew to facilitate smooth operations through logistical methods and planning. Ensured the appropriation and delivery of equipment, materials, and personnel on strict timeframes and minimal budget.