Venkata B. Email & Phone Number

United States

I have extensive experience in developing data pipelines using PySpark, Spark, and Scala for batch and real-time processing. These pipelines read external data, merge and enrich it, and load it into Azure SQL Datawarehouse. I've worked with databases like Oracle, SQL Server, and MySQL, using PowerCenter for transformations.In Azure Synapse Analytics, I configured polybasic techniques for fast data loading and analysis with serverless SQL. I built robust ETL pipelines from Azure Blob and Data Lake storage using Azure HDInsights. I've also migrated On-premises SQL Server Databases to Azure Data Lake, Azure SQL Database, Databricks, and Azure SQL Datawarehouse.I've utilized PowerCenter transformations like Aggregator, Joiner, Lookup, Router, and Expression for data transformations and business logic. Azure services like Azure Data Factory, Azure Data Lake, and Azure SQL Database played a crucial role in data extraction and migration preparation.Designed end-to-end data integration solutions, leveraging IICS connectors and transformations, and integrated IICS with cloud applications like Salesforce, ServiceNow, and Workday for data synchronization and automation.I've optimized Spark job performance with job monitoring and logging solutions. For data migration, I've used BigQuery, Informatica Intelligent Cloud Services (IICS), Informatica PowerCenter, and Google Cloud Platform (GCP) services. Data ingestion pipelines into BigQuery used Cloud Dataflow, Cloud Storage, and Cloud Pub/Sub.Within Azure, I've ingested data into Azure Data Lake, Azure Blob Storage, and Azure Data Warehouses with Azure Databricks. Data extraction, transformation, and loading (ETL) from various source systems into Azure Data Storage used Azure Data Factory, Spark SQL, and Azure Data Lake Analytics.Designed and maintained interactive Power BI reports, drawing insights from Azure Synapse/Azure Data Warehouse, Azure Data Lake, and Azure SQL, enhancing data visualization capabilities.

I have extensive experience in data engineering and analytics, with a focus on cloud-based solutions. I've developed data pipelines using technologies like PySpark, Spark, and Scala to process large volumes of data, both in batch and real-time. These pipelines involve reading data from external sources, merging it, enriching it, and loading it into Azure SQL Data Warehouse.My work has spanned various database platforms, including Oracle, SQL Server, and MySQL, where I've utilized PowerCenter's connectivity and transformation capabilities. I've also configured efficient data loading and exporting techniques in Azure Synapse Analytics and conducted data analysis using serverless SQL.I've been involved in complex data migration projects, migrating on-premises SQL Server databases to Azure Data Lake, Azure SQL Database, Databricks, and Azure SQL Data Warehouse. I've leveraged PowerCenter transformations like Aggregator, Joiner, Lookup, Router, and Expression for data transformations and business logic implementation.Furthermore, I have experience with cloud platforms such as AWS and GCP, working on services like AWS Lambda, DynamoDB, and BigQuery. I've also designed and maintained data integration solutions, integrated with cloud applications, and optimized Spark jobs for performance.Utilized Python for developing data processing applications on AWS infrastructure.Implemented Python-based data encryption and decryption mechanisms using AWS Key Management Service (KMS) for secure data transmission and storage.In the realm of data reporting and visualization, I've designed interactive reports using Power BI, drawing insights from various data sources, including Azure Synapse, Data Lake, and SQL. My diverse skill set spans the entire data lifecycle, from ingestion to transformation, analysis, and reporting.

I developed and implemented Spark applications in the Hortonworks HDP 2 Hadoop ecosystem for large-scale data processing and analysis. To ensure efficient and reliable data transfer, I designed and optimized data ingestion processes using Sqoop, importing data from diverse sources into the HDFS Data Lake.For coordinating Spark job execution and maintaining data consistency, I implemented ZooKeeper-based distributed locks and synchronization mechanisms. As part of performance tuning, I introduced Dynamic Partitioning and Bucketing in Hive and automated tasks using the Oozie framework.My work involved designing and developing Spark applications using a range of Spark tools, including RDD transformations, Spark Core, Spark Machine Learning Library, Spark Streaming, and Spark SQL. I also converted Hive/SQL queries into Spark transformations using RDDs and Scala.Leveraging the Hadoop Distributed File System (HDFS), I ensured data integrity and high availability for extensive datasets used in Spark applications. I employed the Spark API over Hortonworks Hadoop YARN for analytics on Hive data and optimized existing Hadoop algorithms using Spark components like Spark context, spark-SQL, DataFrames, and RDDs.Using Python, Implemented Spark programs that harnessed Spark SQL, Spark Streaming, DataFrames, Accumulators, and Broadcast variables to process and transform data stored in HDFS files. Furthermore, I converted existing PIG and MapReduce jobs into Spark programs, resulting in enhanced performance, scalability, and code maintainability.I conducted performance tuning for Pig, Hive, and Spark jobs by implementing caching, persistence, and partitioning strategies, leading to significant performance improvements and performed optimization of Python code using tools like cProfile and line_profiler.Configured Linux-based monitoring tools such as Nagios and Zabbix for proactive system monitoring and alerting, ensuring high availability of our data infrastructure.

In my role, I crafted Technical Design Documents (TDD) and Detailed Design Documents (DDD) for efficient data loading into DataMarts using MS SQL Server Management Studio and PL/SQL. To enhance data transformation and migration,Developed Python scripts for data preprocessing tasks, including data cleansing, normalization, and feature engineering, using libraries such as Pandas and NumPy.Implemented Git branching strategies such as GitFlow to manage feature development and release cycles effectively.Leveraged Docker Compose for local development environments, ensuring consistency across development, testing, and production environments.Configured Linux-based cron jobs for scheduling data extraction and transformation tasks, automating repetitive data processing workflows.Conducted performance tuning of SQL queries and database indexes using tools like EXPLAIN ANALYZE and pg_stat_statements, optimizing query execution times and resource utilization.Furthermore, I delved into optimizing database speed and backend operations by creating efficient PL/SQL stored procedures. In the realm of real-time issue resolution, I diligently debugged and rectified problems with production server SSIS packages, ensuring smooth data processing and upholding data integrity. I leveraged SQL Server Reporting Services (SSRS) for the development and dissemination of advanced reports, empowering insightful decision-making based on robust data analysis. Additionally, Tableau played a vital role in my toolkit, enabling the creation of prototype data visualizations with various interactive features, showcasing the advantages of analytics.The implementation of tools like Jenkins for continuous integration and deployment and GitHub for version control ensured effective project management and streamlined collaboration throughout the development processes.