With over 10 years in IT and 5+ years specializing in Data Engineering, I've designed, developed, and implemented Big Data Applications on both Microsoft Azure Cloud and AWS.My expertise lies in leveraging technologies like Apache Hive, Apache Spark, PySpark, SparkSQL, and various cloud services to optimize data workflows, build end-to-end data pipelines, and ensure data accuracy and integrity. I have extensive experience with Azure services such as Azure Data Factory, Azure Databricks, and Azure Synapse Analytics, optimizing Spark applications, and handling real-time streaming pipelines using Kafka and Spark-Streaming. Proficient in both on-premises and cloud environments, I have expertise in ETL/ELT pipelines, CI/CD pipelines, and automating data workflows using various tools and frameworks.Collaboration with cross-functional teams has been crucial in delivering scalable and reliable data solutions.TECHNICAL SKILLS:Big Data Technologies:HDFS, Hive, Map Reduce, Pig, Hadoop distribution, and HBase, Spark, Spark Streaming, Kafka.Cloud Services:AWS (EC2, S3, EMR, RDS, Lambda, Cloud Watch, Auto scaling, Redshift, Cloud Formation, Glue), Azure (Databricks, Azure Data Lake, Azure HDInsight)Databases:Oracle, MySQL, SQL Server, Mongo DB, Dynamo DB, Cassandra, Snowflake.Programming Languages:Python, Pyspark, Shell script, Perl script, SQL, Java.Tools:PyCharm, Eclipse, Visual Studio, SQL*Plus, SQL Developer, SQL Navigator, SQL Server Management Studio, Eclipse, Postman.Cloud Tech: Azure and AWSVersion Control:SVN, Git, GitHub, Maven.Big Data Ecosystems:HDFS, Map-Reduce, Hive, Pig, Sqoop, Flume, Zookeeper, Spark, Kafka, Spark, Hbase.Deployment Tools:Git, Jenkins, Terra form and Cloud FormationCloud Technologies:Azure Analysis Services, Azure SQL Server, Dynamo DB, Step Functions, Glue, Athena, Cloud Watch, Azure Data Factory, Azure Data Lake, Functions, Azure SQL Data Warehouse, Databricks and HDInsightData Visualization:Power BI, Tableau, BO Reports, Dremio.I love to build my network, happy to connect!
Norton Healthcare
View- Website:
- nortonhealthcare.com
- Employees:
- 9977
-
Azure Data EngineerNorton HealthcareUnited States
-
Azure Data Engineer/ Data ScientistFm Global Jul 2021 - Present• Utilized Azure Data Factory (ADF) extensively for ingesting diverse data sources, including relational and unstructured data, to meet business requirements.• Designed and implemented end-to-end pipelines and Spark applications in Azure Synapse Analytics, ensuring fault tolerance and scalability.• Integrated CI/CD pipelines by combining Azure Data Factory, Azure Databricks, and other Azure services, facilitating seamless deployment using Azure DevOps.• Implemented Change Data Capture (CDC) logic for incremental data loads in Azure Synapse Pipelines, enhancing data processing efficiency.• Developed and optimized PySpark and SparkSQL scripts for data transformations in Azure Synapse notebooks, meeting business needs.• Created Fact and Dimension tables with SCD type 2 implementation in Azure Synapse Analytics, optimizing data loading from Azure Data Lake storage.• Built complex ETL/ELT pipelines using Azure Data Factory V2 and Azure Synapse Dedicated SQL Pools for efficient data processing.• Implemented performance tuning techniques to enhance Spark application performance by 5 times, including optimization of computing time for streaming data processing.• Automated job scheduling and execution using various triggers in Azure Data Factory, improving operational efficiency.• Designed and developed solutions for real-time data processing using Azure Stream Analytics, Azure Event Hub, and Service Bus Queue, ensuring timely insights.• Established linked services to connect external resources to Azure Data Factory, enabling seamless data integration and management. -
Azure Data Engineer/ Data ScientistMolina Healthcare Sep 2018 - Jun 2021 -
Data EngineerCardinal Health Nov 2016 - Aug 2018• Designed and deployed cloud infrastructure using AWS CloudFormation templates and automated AWS environment provisioning with Jenkins and Auto Scaling for EC2 instances.• Developed workflows in Oozie and Airflow to automate data loading (HDFS) and pre-processing with Pig and Hive. Utilized Airflow DAGs with Python to execute custom data transformation tasks for scalable pipelines within Hadoop clusters.• Created Hive tables with dynamic/static partitioning and buckets for efficient data storage. Developed Python scripts to automate EMR cluster launch and Hadoop application configuration. Extensive experience working with Avro and Parquet data formats using PySpark dataframes.• Analyzed system failures, identified root causes, and recommended solutions for Hadoop clusters. Configured and load-balanced Hadoop clusters across nodes using the Hortonworks platform.• Developed Python scripts for data ingestion (web server outputs) and data cleansing tasks. Utilized Python libraries (pandas, NumPy) for data manipulation and analysis within Hadoop environments. Integrated Python-based machine learning models with Hadoop clusters for predictive analytics. Implemented Python unit tests for validating data processing pipeline functionality and used Python for automating administrative tasks in Hadoop (configuration management, monitoring).• Experience with Spark on YARN/MRv2 for interactive and batch data analysis. Managed and monitored Hadoop clusters using Cloud Era Manager. Proficient in Python and Shell scripting for building data pipelines. Supported existing GCP Data Management implementations. Utilized AWS Athena for data ingestion and report generation. -
Software EngineerWipro Nov 2013 - Jul 2016
Frequently Asked Questions about Nithish Reddy
What company does Nithish Reddy work for?
Nithish Reddy works for Norton Healthcare
What is Nithish Reddy's role at the current company?
Nithish Reddy's current role is Azure Data Engineer.
Who are Nithish Reddy's colleagues?
Nithish Reddy's colleagues are Jessica Lott, Taylor Livojevich, Ross Stephanie, Kay Polk, Lashay Mitchell, Dana Hayse, Delisha Hunt-Colbert.
Not the Nithish Reddy you were looking for?
-
NITHISH R.
United States -
Nithish Reddy
Full Stack Developer | Expertise In Banking, Healthcare, And Logistics | Skilled In Java, Spring Boot, React, Aws, And Azure | Proven Track Record In Scalable And Secure Applications |Seeking Full-Time OpportunitiesLas Vegas, Nv -
Nithish Reddy
Sr.Lead Golang Developer || Microservices & Cloud Solutions Architect || Expert In Api Development, Distributed Systems, Docker, And Kubernetes. Actively Looking For New Opportunity .Ready To RelocateUnited States -
Nithish Reddy
Innovative Tech Professional| Expertise In Java, Spring, And Agile MethodologiesFranklin Park, Nj
Free Chrome Extension
Find emails, phones & company data instantly
Aero Online
Your AI prospecting assistant
Select data to include:
0 records × $0.02 per record
Download 750 million emails and 100 million phone numbers
Access emails and phone numbers of over 750 million business users. Instantly download verified profiles using 20+ filters, including location, job title, company, function, and industry.
Start your free trial