Tej D Email and Phone Number

• Developed ETL pipelines in and out of data warehouse using a combination of Python and Snowflakes Snow SQL Writing SQL queries against Snowflake. • Define virtual warehouse sizing for Snowflake for different type of workloads• Used various sources to pull data into Power BI such as SQL Server, oracle and also used Power BI desktop to connect to various sources, working with different visualizations. • Helped individual teams to set up their repositories in bit bucket and maintain their code and help them setting up jobs which can make use of CI/CD environment.• I have developed data quality tests to assess various aspects of data quality, including accuracy, completeness, consistency, and timeliness, using frameworks like Great Expectations and Apache Griffin.

• Developed business logic using Kafka & Spark Streaming and implemented business transformations.• Triggering, Monitoring Harmonization, and curation Jobs in Production Environment. Also Scheduled few Jobs by using DHF jobs and ESP Jobs. • Raised the Change Request and SNOW Request in ServiceNow to deploy or send changes into Production.• Also Guiding the development of a team working on PySpark (Python and Spark) jobs.• Developed Spark Applications for various business logics using Python.• Extracted, Transformed and Loaded (ETL) data from disparate sources to Azure Data Storage services using a combination of Azure Data Factory, T-SQL, Spark SQL and Azure Data Lake Analytics.• Worked on different files like CSV, JSON, Flat, Parquet to load the data from source to raw tables.

•Analyze, develop, and build modern data solutions with the Azure PaaS service to enable data visualization. Understand the application's current Production state and the impact of new installation on existing business processes.•Worked on migration of data from On-prem SQL server to Cloud databases (Azure Synapse Analytics (DW) & Azure SQL DB).•Extract Transform and Load data from Sources Systems to Azure Data Storage services using a combination of Azure Data Factory, T-SQL, Spark SQL and U-SQL Azure Data Lake Analytics. •Data Ingestion to one or more Azure Services - (Azure Data Lake, Azure Storage, Azure SQL, Azure DW) and processing the data in In Azure Databricks.•Pipelines were created in Azure Data Factory utilizing Linked Services/Datasets/Pipeline/ to extract, transform, and load data from many sources such as Azure SQL, Blob storage, Azure SQL Data warehouse, write-back tool, and backwards.•Used Azure ML to build, test and deploy predictive analytics solutions based on data. •Used Snowflake cloud data warehouse for integrating data from multiple source system which include nested JSON formatted data into Snowflake table.•Created Build and Release for multiple projects (modules) in production environment using Visual Studio Team Services (VSTS).

•Developed Apache presto and Apache drill setups in AWS EMR (Elastic Map Reduce) cluster, to combine multiple databases like MySQL and Hive. This enables to compare results like joins and inserts on various data sources controlling through single platform.•The AWS Lambda functions were written in Scala with cross-functional dependencies that generated custom libraries for delivering the Lambda function in the cloud.•Performed raw data ingestion into S3 from kinesis firehouse, which triggered a lambda function and put refined data into another S3 bucket and wrote to SQS queue as aurora topics.•Writing to the Glue metadata catalog allows us to query the improved data from Athena, resulting in a serverless querying environment.•Created Pyspark frame to bring data from DB2 to Amazon S3.•Worked on Kafka Backup Index, Log4j appender minimized logs and pointed ambari server logs to NAS Storage.•Created AWS RDS (Relational database services) to work as Hive metastore and could combine 20 EMR cluster’s meta data into a single RDS, which avoids the data loss even by terminating the EMR.

• Imported real time weblogs using Kafka as a messaging system and ingested the data to Spark Streaming.• Implemented data quality checks using Spark Streaming and arranged bad and passable flags on the data.• Supported Continuous storage in AWS using Elastic Block Storage, S3, Glacier. Created Volumes and configured Snapshots for EC2 instances.• Developed Spark code for using Scala and Spark-SQL for faster processing and testing.• Worked on loading CSV/TXT/AVRO/PARQUET files using Scala language in Spark Framework and processed the data by creating Spark Data frame and RDD and save the file in parquet format in HDFS to load into fact table using ORC Reader.• Involved in data loading using PL/SQL and SQL*Loader calling UNIX scripts to download and manipulate files.• Involved in creating data-models for customer data using Cassandra Query Language. Performed benchmarking of the No-SQL databases, Cassandra and HBase • Involved in converting Hive/SQL queries into Spark transformations using Spark RDDs and Scala.

• Address the performance tuning of Hadoop ETL processes against very large data set work directly with statistically on implementing solutions involving predictive analytics.• Performed Linux operations on the HDFS server for data lookups, job changes if any commits were disabled, and rescheduling data storage jobs.• Created data processing pipelines for data transformation and analysis by developing spark jobs in Scala.• Testing and validating database tables in relational databases with SQL queries, as well as performing Data Validation and Data Integration. Worked on visualizing the aggregated datasets in Tableau.• Migrating code to version controllers using Git Commands for future use and to ensure a smooth development workflow.