•Worked on AWS glue ETL tool, in which we used AWS Glue Crawler, AWS Glue Data catalog andConnections.•Worked on AWS Athena query service that makes it easy to analyze the data in Amazon S3 using standard SQL.•Perform validation and verify software at all testing phases which include Functional Testing, SystemIntegration Testing, End to End Testing, Regression Testing, Sanity Testing, User Acceptance Testing,Smoke Testing, Disaster Recovery Testing, Production Acceptance Testing, and Pre-prod Testing phases.•Built CI/CD pipelines using Azure DevOps.•Worked with Snowflake utilities, SnowSQL, SnowPipe, and Big Data model with Python.•Created ETL pipelines in and out of the data warehouse using a combination of Python and Snowflakes•SnowSQL Writing SQL queries against Snowflake.

• Developed highly complex Python and Scala code, which is maintainable, easy to use, and satisfies application requirements, data processing and analytics using inbuilt libraries.• Involved in designing optimizing Spark SQL queries, Data frames, import data from Data sources, perform transformations; perform read/write operations, save the results to output directory into HDFS/AWS S3.• Developed Pig Latin scripts to extract the data from the web server output files to load into HDFS.• Built dashboards using Tableau to allow internal and external teams to visualize and extract insights from big data platforms.

• Involved in Requirements and Analysis: Understanding the requirements of the client and the flow of the application.• Developed Simple to complex MapReduce Jobs using Hive and Pig.• Developed PIG scripts to transform the raw data into intelligent data as specified by business users.• Utilized Spark, Scala, Python for querying, preparing from big data sources.• Wrote pre-processing queries in python for internal spark jobs• Prepared dashboards using Tableau for summarizing Configuration, Quotes, Orders e - commerce data.• Extract, transform, and load (ETL) data from multiple federated data sources (JSON, relational database, etc.) with Data Frames in Spark.

• Developed PIG UDFs to provide Pig capabilities for manipulating the data according to Business Requirements and worked on developing custom PIG Loaders and Implemented various requirements using Pig scripts.• Worked on developing ETL workflows on the data obtained using Python for processing it in HDFS and HBase using Flume.• Developed simple and complex MapReduce programs in Hive, Pig and Python for Data Analysis on different data formats.• Developed various Python scripts to find vulnerabilities with SQL Queries by doing SQL injection, permission checks and analysis.• Written Terraform scripts to automate AWS services which include ELB, CloudFront distribution, RDS, EC2, database security groups, Route 53, VPC, Subnets, Security Groups, and S3 Bucket and converted existing AWS infrastructure to AWS Lambda deployed via Terraform and AWS Cloud Formation.• Imported data from AWS S3 into Spark RDD, Performed transformations and actions on RDD's.• Created scripts to read CSV, JSON and parquet files from S3 buckets in Python and load into AWS S3, DynamoDB and Snowflake.