• Designing the business requirement collection approach based on the project scope and SDLC methodology.• Created and worked Sqoop jobs with incremental load to populate Hive External tables.• Developed Scala, UDF's using both Data frames/SQL and RDD/MapReduce in Spark for Data Aggregation, queries and writing data back into RDBMS through Sqoop.• Loading data from different sources to a data ware house to perform some data aggregations for business Intelligence using python.• Used SQL Server Management Tool to check the data in the database as compared to the requirement give• Installing, configuring and maintaining Data Pipelines• Took proof of concept projects ideas from business, lead, developed and created production pipelines that deliver business value using Azure Data Factory• Administered Tableau server including creating User Rights Matrix for permissions and roles, monitoring report usage and creating sites for various departments• Worked on a clustered Hadoop for Windows Azure using HDInsight and Hortonworks Data Platform for Windows.• Performance tuning, code promotion and testing of application changes

• Prepared Design Documents (Request-Response Mapping Documents, Hive Mapping Documents). • Scheduled Oozie workflow engine to run multiple HiveQL, Sqoop, and Pig jobs.• Implement Spark Kafka streaming to pick up the data from Kafka and send to Spark pipeline.• Built performant, scalable ETL processes to load, cleanse and validate data• Collected data using Spark Streaming from AWS S3 bucket in near-real-time and performs necessary Transformations and Aggregation on the fly to build the common learner data model and persists the data in HDFS• Performs data analysis and design, and creates and maintains large, complex logical and physical data models, and metadata repositories using ERWIN and MB MDR• I have written shell script to trigger data Stage jobs.• Worked on AWS Data pipeline to configure data loads from S3 to into Redshift.

• Experience in designing and developing applications in Spark using Scala to compare the performance of Spark with Hive and SQL/Oracle.• Implemented python scripts which perform transformations and actions on tables and send incremental data to the next zone by using spark submit.• Designed and implemented Incremental Imports into Hive tables.• Experience in manipulating/analyzing large datasets and finding patterns and insights within structured and unstructured data.• Experienced in managing and reviewing the Hadoop log files.• Start working with AWS for storage and halding for tera byte of data for customer BI Reporting tool• Developed and Configured Kafka brokers to pipeline server logs data into spark streaming.

• Developed Spark applications using Scala for easy Hadoop transitions.• Worked on AWS CLI Auto Scaling and Cloud Watch Monitoring creation and update.• Created customized BI tool for manager team that perform Query analytics using HiveQL.• Migrated high-volume OLTP transactions from Oracle to Cassandra • Created Data Pipeline of Map Reduce programs using Chained Mappers.• Loaded the aggregated data onto DB2 for reporting on the dashboard.• Used Pig as ETL tool to do transformations, event joins, filters and some pre-aggregations before storing the data onto HDFS.• Modelled Hive partitions extensively for data separation and faster data processing and followed Pig and Hive best practices for tuning.• Allotted permissions, policies and roles to users and groups using AWS Identity and Access Management (IAM).• Used Hive to analyze the partitioned and bucketed data and compute various metrics for reporting on the dashboard.• Importing and exporting data into HDFS from database and vice versa using Sqoop.• Implemented optimization and performance tuning in Hive and Pig.

• Created batch jobs and configuration files to create automated process using SSIS.• Worked on different dataflow and control flow task, for loop container, sequence container, script task, executes SQL task and Package configuration.• Automated the process of extracting the various files like flat/excel files from various sources like FTP and SFTP (Secure FTP).• Created SSIS packages to pull data from SQL Server and exported to Excel Spreadsheets and vice versa.• Built SSIS packages, to fetch file from remote location like FTP and SFTP, decrypt it, transform it, mart it to data warehouse and provide proper error handling and alerting• Data validation and cleansing of staged input records was performed before loading into Data Warehouse• Deploying and scheduling reports using SSRS to generate daily, weekly, monthly and quarterly reports.