• Worked with Spark and improved the performance and optimized the existing algorithms in Hadoop using Spark Context, Spark-SQL, Data Frame, RDD's, Spark YARN.• Design, Develop, Implement the ETL objects by extracting the data using Sqoop from source system to Hadoop Files system (HDFS). • Designed and developed ETL code using Informatica Mappings to load data from heterogeneous Source systems flat files, XML’s, MS Access files, Oracle to target system Oracle under Stage, then to the data warehouse and then to Data Mart tables for reporting.• Developed scripts in Hive to perform transformations on the data and loaded it into target systems for reports.• Collecting and aggregating large amounts of log data using Flume and tagging data in HDFS for further analysis. • Worked extensively with importing metadata into Hive using Python and migrated existing tables and applications to work on the AWS cloud (S3). • Involved in complete Big Data flow of the application starting from data ingestion from upstream to HDFS, processing the data in HDFS, and analyzing the data.• Automated the process for the extraction of data from warehouses and weblogs by developing workflows and coordinator jobs in Oozie. • Worked on performance tuning of Apache Kafka workflow to optimize the data ingestion speeds.• Involved in the development of full life cycle implementation of ETL using Informatica, Oracle and helped with designing the Date warehouse by defining Facts, Dimensions, and relationships between them and applied the Corporate Standards in Naming Conventions.• Implemented ETL pipelines on AWS EMR.• Worked on AWS Lambda to run the code in response to events, such as changes to data in an Amazon

• Developed Spark scripts, UDF's using both Spark DSL and Spark SQL query for data aggregation, querying, and writing data back into RDBMS through Sqoop. • Designed and developed a Data Lake using Hadoop for processing raw and processed claims via Hive and Informatica. • Used Polybase for ETL/ELT process with Azure Data Warehouse to keep data in Blob Storage with almost no limitation on data volume. • Import data from sources like HDFS/HBase into Spark RDD. • Built Azure Data Warehouse Table Data sets for Power BI Reports. • Worked on BI reporting with At Scale OLAP for Big Data. • Implemented Kafka for streaming data and filtered, processed the data. • Created ETL/Talend jobs both design and code to process data to target databases. • Worked with Hadoop infrastructure to storage data in HDFS storage and use Spark / HIVE SQL to migrate underlying SQL codebase in Azure. • Wrote Pig Scripts to generate Map Reduce jobs and performed ETL procedures on the data in HDFS. • Experienced in loading the real-time data to NoSQL database like Cassandra. • Generate metadata, create Talend jobs, mappings to load data warehouse, data lake• Used Talend for Big data Integration using Spark and Hadoop. • Worked in developing Pig Scripts for data capture change and delta record processing between newly arrived data and already existing data in HDFS.

• Implemented and supported strategic data sourcing to automate the population of annual, monthly, and quarterly CCAR reports in the Big Data platform.• Developed the statistical models designed to forecast market variables under stress scenarios within Financial Models using R.• Created queries using Hive, SAS (Proc SQL), and PL/SQL to load large amounts of data from MongoDB and SQL Server into HDFS to spot data trends.• Wrote Hive-QL to retrieve, query, and process raw data.• Utilized K-means clustering technique to classify unlabeled data.• Worked on data pattern recognition, data cleaning as well as data visualizations such as Scatter Plot, Box Plot, and Histogram Plot to explore the data using packages Matplotlib, Seaborn in Python, ggplot in R and SAS.• Used LDA, PCA, and Factor Analysis to perform the dimensional reduction.• Modified and applied Machine Learning algorithms such as Neural Networks, SVM, Bagging, Gradient Boosting, K-Means, PySpark, and MLlib to detect target customers.• Worked on customer segmentation based on the similarities of the customers using an unsupervised learning technique - cluster analysis.• Used Pandas, Numpy, Scipy, Scikit-learn, NLTK in Python for scientific computing and data analysis.• Applied cross-validation to evaluate and compare the performance among different models. Validated the machine learning classifiers using ROC Curves and Lift Charts.• Configured Spark Streaming with Kafka to clean and aggregate real-time data. • Involved in Text Analytics such as analyzing text, language syntax, structure, and semantics.• Generated weekly and monthly reports and maintained, manipulated data using SAS macro, Tableau, and D3.js.• Involved in using Sqoop to load historical data from SQL Server into HDFS.• Used Git for version control.

• Worked on analyzing the Hadoop cluster using different big data analytic tools including Pig, Hive, and MapReduce. • Involved in loading data from the LINUX file system to HDFS.• Designed ETL process using Talend Tool to load from Sources to Targets through data Transformations. • Translated business requirements into working logical and physical data models for OLTP &OLAP systems. • Reviewed Stored Procedures for reports and wrote test queries against the source system (SQL Server-SSRS) to match the results with the actual report against the Datamart (Oracle) • Owned and managed all changes to the data models. Created data models, solution designs, and data architecture documentation for complex information systems. • Developed Advanced PL/SQL packages, procedures, triggers, functions, Indexes, and Collections to implement business logic using SQL Navigator. • Working on the OLAP for data warehouse and data mart developments using OLTP models, both and interacting with all the involved stakeholders and SMEs to derive the solution. • Created the best fit Physical Data Model based on discussions with DBAs and ETL developers. • Identified required dimensions and Facts using Erwin tool for the Dimensional Model. • Implemented ETL techniques for Data Conversion, Data Extraction, and Data Mapping for different processes as well as applications. • Validated and updated the appropriate Models to process mappings, screen designs, use cases, business object models, and system object models as they evolved and changed. • Created Model reports including Data dictionaries, Business reports. • Generated SQL scripts and implemented the relevant databases with related properties from keys, constraints, indexes & sequences. • Created conceptual, logical, and physical data models, data dictionaries, DDL, and DML to deploy and load database table structures in support of system requirements.