 Developed Spark programs to parse the raw data, populate staging tables, and store the refineddata in partitioned tables in the Enterprise Data warehouse. Experience in building power BI reports on Azure Analysis services for better performance. Developed Streaming applications using PySpark to read from the Kafka and persist the dataNoSQL databases such as HBase and Cassandra. Implemented PySpark Scripts using Spark SQL to access hive tables into a spark for… Show more  Developed Spark programs to parse the raw data, populate staging tables, and store the refineddata in partitioned tables in the Enterprise Data warehouse. Experience in building power BI reports on Azure Analysis services for better performance. Developed Streaming applications using PySpark to read from the Kafka and persist the dataNoSQL databases such as HBase and Cassandra. Implemented PySpark Scripts using Spark SQL to access hive tables into a spark for fasterprocessing of data. Worked on Big Data Hadoop cluster implementation and data integration in developing large-scale system software. Migrating an entire oracle database to Big Query and using of power BI for reporting. Build data pipelines in airflow in GCP for ETl related jobs using different airflow operators. Developed streaming and batch processing applications using PySpark to ingest data from thevarious sources into HDFS Data Lake. Developed DDLs and DMLs scripts in SQL and HQL for analytics applications in RDBMS and Hive. Developed and implemented HQL scripts to create Partitioned and Bucketed tables in Hive foroptimized data access. Used cloud shell SDK in GCP to configure the services Data Proc, Storage, BigQuery. Written Hive UDFs to implement custom functions in the Hive for aggregations. Worked extensively with Sqoop for importing and exporting the data from HDFS to RelationalDatabase systems/mainframe and vice-versa loading data into HDFS. Monitoring YARN applications Troubleshoot and resolve cluster related system problems. Created shell scripts to parameterize the Hive actions in Oozie workflow and for scheduling thejobs. Populated HDFS and Cassandra with huge amounts of data using Apache Kafka. Worked as a key role in a team of developing an initial prototype of a NiFi big data pipeline. Thispipeline demonstrated an end-to-end scenario of data ingestion, processing. Show less

 Participated in weekly release meetings with Technology stakeholders to identify and mitigatepotential risks associated with the releases. Implemented Responsible AWS solutions using EC2, S3, RDS, EBS, Elastic Load Balancer, andAuto scaling groups, Optimized volumes, and EC2 instances. Wrote Terraform templates for AWS Infrastructure as a code to build staging, productionenvironments & set up build & automations for Jenkins. Configured Elastic Load Balancers… Show more  Participated in weekly release meetings with Technology stakeholders to identify and mitigatepotential risks associated with the releases. Implemented Responsible AWS solutions using EC2, S3, RDS, EBS, Elastic Load Balancer, andAuto scaling groups, Optimized volumes, and EC2 instances. Wrote Terraform templates for AWS Infrastructure as a code to build staging, productionenvironments & set up build & automations for Jenkins. Configured Elastic Load Balancers (ELB) with EC2 Auto scaling groups. Created Amazon VPC to create a public-facing subnet for web servers with Internet access, andbackend databases & application servers in a private-facing subnet with no Internet access. Created AWS Launch configurations based on customized AMI and used this launch configurationto configure auto-scaling groups. Utilized Puppet for configuration management of hosted Instances within AWS Configuring andNetworking of Virtual Private Cloud (VPC). Utilized S3 bucket and Glacier for storage and backup on AWS. Using the Amazon Identity Access Management (IAM) tool created groups & permissions for users towork collaboratively. Implemented /set continuous project build and deployment delivery process usingSubversion, Git, Jenkins, IIS, Tomcat. Connected continuous integration system with GIT version control repository and continuallybuilt as the check-ins 's come from the developer. Knowledge in build tools Ant and Maven and writing build.xml and pom.xml respectively. Knowledge in authoring pom.xml files, performing releases with the Maven release plug-in, andmanaging Maven repositories. Implemented Maven builds to automate JAR and WAR files. Designed and built deployment using ANT/ Shell scripting and automate the overall processusing Git and MAVEN. Show less

 I was responsible for overall design and implementing enterprise datamigration process from Legacy Oracle/Db2 Sources to RDS-Postgres and Amazon Redshift usingAWS Data Migration Service, Schema Conversion tools and migration agents. Designed and implemented highly scalable ETL Using Matillion tool. Developed numerousOrchestration and transformation jobs and nested as master jobs Matillion. Dockized ETL components and deployed to Data Specific ECS clusters using… Show more  I was responsible for overall design and implementing enterprise datamigration process from Legacy Oracle/Db2 Sources to RDS-Postgres and Amazon Redshift usingAWS Data Migration Service, Schema Conversion tools and migration agents. Designed and implemented highly scalable ETL Using Matillion tool. Developed numerousOrchestration and transformation jobs and nested as master jobs Matillion. Dockized ETL components and deployed to Data Specific ECS clusters using Jenkins/GIT.Configured ETL Services, RDS and Redshift logs to Splunk and Steel Central for enterprisemonotiling. Implemented FIPS 140-2 Compliant Encryption standards for Data-At-Rest and Data-In-Transit. Worked as POC for Optimizing and performance tuning issues and Maintenance for CloudDatabases. Defined and deployed monitoring, metrics, and logging systems on AWS, primary configuringCloudWatch metrics for RDS and Redshift. Implemented Workload Management (WML) in Redshift to prioritize basic dashboard queriesover more complex longer running ad-hoc queries. This allowed for a more reliable and fasterreporting interface, giving sub-second query response for basic queries. Worked on developing various spark jobs for processing parquet data files; Responsible forDesigning Logical and Physical data modelling for various data sources on Amazon Redshift. Implemented Data extracts process between CME BIC (Beneficiary cloud) and CMS RASSProducts. Optimizing /tuning and automating Redshift DW environment using AWS Utility. For implementation in RASS Project- extensively used QLIK Replicate (Formerly Attunity) andQLIK Compose for Datawarehouse to automation Data Ingestion and Data Curation Processes. Show less

Involved in the implementation of the project went through several phases namely: data setanalysis, preprocessing data set, user-generated data extraction, and modeling. Participated in Data Acquisition with the Data Engineer team to extract historical and real-timedata by using Sqoop, Pig, Flume, Hive, MapReduce, and HDFS. Wrote user-defined functions (UDFs) in Hive to manipulate strings, dates, and other data. Performed Data Cleaning, features scaling, and features… Show more Involved in the implementation of the project went through several phases namely: data setanalysis, preprocessing data set, user-generated data extraction, and modeling. Participated in Data Acquisition with the Data Engineer team to extract historical and real-timedata by using Sqoop, Pig, Flume, Hive, MapReduce, and HDFS. Wrote user-defined functions (UDFs) in Hive to manipulate strings, dates, and other data. Performed Data Cleaning, features scaling, and features engineering using pandas, and NumPypackages in Python. Process Improvement: Analyzed error data of recurrent programs using Python and devised anew process to reduce the turnaround time of the problem's solutions by 60% Worked on production data fixes by creating and testing SQL scripts. Deep dive into complex data sets to analyze trends using Linear Regression, Logistic Regression,Decision Trees Prepared reports using SQL and Excel to track the performance of websites and apps Visualized data using Tableau to highlight abstract information Applied clustering algorithms i.e. Hierarchical, K-means using Scikit, and Scipy. Performed Data Collection, Data Cleaning, Data Visualization, and Feature Engineering usingPython libraries such as Pandas, Numpy, matplotlib, and Seaborn. Optimized SQL queries for transforming raw data into MySQL with Informatica to preparestructured data for machine learning. Used Tableau for data visualization and interactive statistical analysis. Worked with Business Analysts to understand the user requirements, layout, and look of theinteractive dashboard. Used SSIS to create ETL packages to Validate, Extract, Transform, and Load data into DataWarehouse and Data Mart. The lifetime values were classified based on the RFM model by using an XGBoost classifier. Show less

Responsibilities: Primarily worked on a project to develop internal ETL product to handle complex and largevolume healthcare claims data. Designed ETL framework and developed number of packages toExtract, Transform and Load data using SQL Server Integration Services (SSIS) into local MS SQL2012 databases to facilitate reporting operations. Involved in various Transformation and data cleansing activities using various Control flow anddata flow tasks in SSIS packages… Show more Responsibilities: Primarily worked on a project to develop internal ETL product to handle complex and largevolume healthcare claims data. Designed ETL framework and developed number of packages toExtract, Transform and Load data using SQL Server Integration Services (SSIS) into local MS SQL2012 databases to facilitate reporting operations. Involved in various Transformation and data cleansing activities using various Control flow anddata flow tasks in SSIS packages during data migration Applied various data transformations like Lookup, Aggregate, Sort, Multicasting, ConditionalSplit, Derived column etc. Developed Mappings, Sessions, and Workflows to extract, validate, and transform data per thebusiness rules using Informatica. Supported Data migration projects, migrated data from SQL Server to Netezza using NZ Migrateutility. Designed target tables as per the requirement from the reporting team and designed Extraction,Transformation and Loading (ETL) using Talend.  Worked on Netezza SQL scripts to load the data between Netezza tables. Schedule Talend Jobs using Job Conductor (Scheduling Tool in Talend) - available in TAC.  Querying, creating stored procedures and writing complex queries and T-SQL join to addressvarious reporting operations and ad-hoc data requests. Performance monitoring and Optimizing Indexes tasks by using Performance Monitor, SQLProfiler, Database Tuning Advisor and Index tuning wizard. Acted as point of contact to resolve locking/blocking and performance issues. Wrote scripts and indexing strategy for a migration to Amazon Redshift from SQL Server andMySQL databases Worked on AWS Data Pipeline to configure data loads from S3 to into Redshift Show less