Dev J Email & Phone Number

Arizona, United States

 Ingesting data from various sources both OnPrem and External sources into Azure Data Lake and increasing the availability from AzureSQL Data Warehouse. Involved in building Data pipelines using Azure Data Factory by connecting to different database sources using JDBC connectors. Good experience in processing Ingested data using Azure Data bricks. Designed and implemented data pipelines to transform retail transaction data from various sources, ensuring data accuracy, consistency,and availability for downstream analytics and reporting. Involvement in working with Azure cloud stage (HDInsight, Databricks, Data Lake, Blob, Data Factory, Synapse, SQL DB and SQL DWH). Developed API proxies, products using APIGEE platform Designed and implemented by configuring Topics in new Kafka cluster in all environment. Implemented Kafka Security Features using SSL and without Kerberos. Sticking to ANSI SQL language specification wherever possible, and providing context about similar functionality in other industry-standard engines (e.g. referencing PostgreSQL function documentation) Design and developed data loader developed using APIGEE, Denodo, Informatica. Expert in developing Data bricks notebooks for extracting data from various source systems such as DB2, Teradata, and performing datacleansing, wrangling, ETL processing, and loading it into Azure SQL DB. Installed and configured OpenShift platform in managing Docker containers and Kubernetes Clusters. Experience in data migration from Microsoft SQL server to Azure SQL database. Created Automated Databricks workflow using Python to run multiple data loads or increasing parallel processing. Built ETL data pipelines to input data from Blob storage to Azure Data Lake Gen2 using Azure Data Factory (ADF).

Developed solutions in Databricks for Data Extraction, transformation and aggregation from multiple data sources Wrote Lambda functions in python for AWS Lambda and invoked python scripts for data transformations and analytics on large data setsin EMR clusters and AWS Kinesis data streams. Development of RESTful WS web services using Java and spring boot. Worked on ETL Migration services by developing and deploying AWS Lambda functions for generating a serverless data pipeline whichcan be written to Glue Catalog and can be queried from Athena. Analysis and development of components using java / spring, for the generation and storage of payment receipts and payrolls in pdfformat. Designed and built data processing applications using Spark on AWS EMR cluster which consumes data from AWS S3 buckets, applynecessary transformations and store the curated business datasets. Wrote and executed several complex SQL queries to join tables in AWS glue for ETL operations in Spark data frame using SparkSQL. Extensively worked on AWS S3 data transfer and AWS Redshift was used for cloud data storage. Migrated an Oracle SQL ETL to run on google cloud platform using cloud dataproc & bigquery, cloud pub/sub for triggering the airflowjobs Wrote Python scripts to migrate data into from old system to AWS Redshift. Extensively worked with Pyspark / Spark SQL for querying data frames and RDDs. Implemented and maintained the monitoring and alerting of production and corporate servers/storage using AWS Cloudwatch.  Experienced in working on different file formats like XML, JSON, AVRO and Parquet Involved in maintaining user accounts (IAM), SQS, SNS Services on AWS cloud. Used Bit Bucket to collaboratively interact with the other team members. Used cloud pub/sub and cloud functions for some specific use cases such as triggering workflows upon messag.

Data transformations were performed in Hive, and partitions and buckets were used to improve performance. Experienced in handling HDFS, Job Tracker, Task Tracker, Name Node, Data Node, YARN, Spark and Map Reduce programming.  Configured and monitored resource utilization across the cluster using Cloudera Manager, Search, and Navigator. Create external Hive tables for consumption and store data in HDFS using the ORC, Parquet, and Avro file formats. ETL Pipelines were created using the Apache PySpark - Spark SQL and Data Frame APIs. Responsible for coding Java Batch, Restful Service, MapReduce program, Hive query's, testing, debugging, Peer code review,troubleshooting and maintain status report. Analyzed Hadoop clusters and various Big Data analytic tools such as Pig, Hive, HBase, Spark, and Sqoop. Used Sqoop to load data into the cluster from dynamically generated files and relational database management systems. Developed service classes, domain/DAOs, and controllers using JAVA/J2EE technologies Partitioning, dynamic partitions, and buckets had been implemented in HIVE. Developed HQL queries, Mappings, tables, and external tables in Hive for analysis across multiple banners, as well as worked onpartitioning, optimization, compilation, and execution. Cloudera Manager is used to continuously monitor and manage the Hadoop cluster. Migrated data successfully from on prem to AWS EMR and S3 buckets by writing shell scripts. Invoked Python scripts for data transformations on large data sets in AWS Kinesis. Mappings done with reusable components such as worklets and mapplets, as well as other transformations. Automated data movement between different components by using Apache NiFi. Loading data from multiple data sources (SQL, DB2, and Oracle) into HDFS using Sqoop and storing it in Hive tables. Migrated data from Teradata into HDFS using Sqoop.

 Designed, developed, and implemented Business Intelligence reports using Tableau.  Involved in building Interactive Dashboards of monthly performance reports using PowerBI. Troubleshooting, resolving, and escalating data- related issues and validating data to improve data quality. Defined data requirements and elements used in XML transactions. Performed Unit testing of reports. Perform Visualizations using Data engine, extracts and connect them.  Extensively worked on creating views and tables with MSSQL. Used Database Engine Tuning Advisor and monitoring tools for database analysis. Experience using data mining techniques to extract information from data sets and identify correlations and patterns.

Hyderabad, Telangana, India

 Analyzed, implemented, and solved research problems using various numerical and stochastic methods in C++ and Python modules forcomputer experiment design and analysis. Design and develop a user interface using HTML, AJAX, CSS, Java Script. Designed and developed data management system using MySQL.  Created class diagrams/sequence diagrams using UML and Rational Rose.  Modify and execute Python / Django modules to change data format.  Hands on with accessing database objects of Django Database API's.  Written python scripts to load the data in database for parsed XML documents.  Handled all the client-side validation using JavaScript.  Expertise in writing Constraints, Indexes, Views, Stored Procedures, Cursors, Triggers, and User Defined functions.  Worked with SQL and stored procedures development on MySQL and SQLite. Participated in the creation of SOAP Web Services for transmitting and receiving data in XML format from an external interface. Frontend and backend modules are tested using Django Web Framework. NumPy was used for numerical analysis, and MatPlotLib libraries from the sci-py kit were used for data analysis and plotting. JIRA was used to track agile/scrum process and development status.