Venkata Subbarayudu Email & Phone Number

Chicago, Illinois, United States

As an Azure Data Engineer, I focused on designing, constructing, and maintaining scalable data pipelines. My responsibilities included integrating various data sources and ensuring data quality and integrity. I created and managed Azure Data Factory pipelines, leveraging Linked Services, Datasets, and Pipelines to extract, transform, and load data from diverse sources. I utilized Spark DataFrame operations for crucial data validations and implemented Synapse integration with Azure Databricks… Show more As an Azure Data Engineer, I focused on designing, constructing, and maintaining scalable data pipelines. My responsibilities included integrating various data sources and ensuring data quality and integrity. I created and managed Azure Data Factory pipelines, leveraging Linked Services, Datasets, and Pipelines to extract, transform, and load data from diverse sources. I utilized Spark DataFrame operations for crucial data validations and implemented Synapse integration with Azure Databricks, significantly improving performance and reducing development time. Additionally, I managed and manipulated configurations using MongoDB and PyMongo, imported real-time weblogs using Kafka, and conducted data quality checks with Spark Streaming. I deployed models as Python packages and services within a microservices architecture using Kubernetes. My role also involved automating deployments with Jenkins and maintaining CI/CD pipelines. Through these efforts, I contributed to enhancing data processing efficiency and ensuring robust data quality and consistency across the organization. Show less

Kansas, United States

In my role as an AWS Data Engineer, I worked closely with Solution Designers and Business Analysts to perform data ingestion, enrichment, and egression. I supported the development of web portals and completed database modeling in PostgreSQL. I created Airflow DAGs to schedule ingestion and ETL jobs and built Docker images to run Airflow locally for testing. I utilized business intelligence and data visualization tools like Tableau to connect to various sources and build graphs. My role… Show more In my role as an AWS Data Engineer, I worked closely with Solution Designers and Business Analysts to perform data ingestion, enrichment, and egression. I supported the development of web portals and completed database modeling in PostgreSQL. I created Airflow DAGs to schedule ingestion and ETL jobs and built Docker images to run Airflow locally for testing. I utilized business intelligence and data visualization tools like Tableau to connect to various sources and build graphs. My role involved provisioning high availability of AWS EC2 instances, migrating legacy systems to AWS, and developing Terraform plugins, modules, and templates for automating AWS infrastructure. Additionally, I performed raw data ingestion with AWS Lambda functions and integrated Kubernetes with cloud-native services for scalability and managed services. I also wrote queries in MySQL and developed automated monitoring and alerting systems using Kubernetes and Docker. Show less

Bengaluru, Karnataka, India

As an Application Developer/Data Engineer, I implemented Azure Data Lake, Azure Data Factory, and Azure Databricks to move and conform data from on-premises to the cloud. I developed multiple notebooks using PySpark and Spark SQL in Databricks for data extraction, analysis, and transformation. I designed and implemented infrastructure as code using Terraform for automated provisioning and scaling of cloud resources on Azure. My role involved fetching Twitter feeds using the python-twitter… Show more As an Application Developer/Data Engineer, I implemented Azure Data Lake, Azure Data Factory, and Azure Databricks to move and conform data from on-premises to the cloud. I developed multiple notebooks using PySpark and Spark SQL in Databricks for data extraction, analysis, and transformation. I designed and implemented infrastructure as code using Terraform for automated provisioning and scaling of cloud resources on Azure. My role involved fetching Twitter feeds using the python-twitter library, developing front-end GUIs, and working with Spark Core, Spark ML, Spark Streaming, and Spark SQL. I created MapReduce programs to parse data for claim report generation and coordinated with the Java team in creating MapReduce programs. I also wrote and executed various MySQL database queries from Python and developed data pipelines and workflows using Azure Databricks. Show less

Bengaluru, Karnataka, India

In my role as a Data Analyst, I was involved in the entire lifecycle of projects, including design, development, deployment, testing, implementation, and support. I wrote scripts to import and export data to CSV and Excel formats using Python and created Celery actions using REST API calls. I developed remote integration with third-party platforms using RESTful web services and used the Python library Beautiful Soup for web scraping. I managed large datasets using Pandas data frames and SQL… Show more In my role as a Data Analyst, I was involved in the entire lifecycle of projects, including design, development, deployment, testing, implementation, and support. I wrote scripts to import and export data to CSV and Excel formats using Python and created Celery actions using REST API calls. I developed remote integration with third-party platforms using RESTful web services and used the Python library Beautiful Soup for web scraping. I managed large datasets using Pandas data frames and SQL, wrote and executed various MySQL database queries from Python, and built Jenkins jobs for CI/CD infrastructure. Additionally, I developed methods for Create, Read, Update, and Delete (CRUD) operations in Active Record and used Django evolution and manual SQL modifications to modify Django models while retaining data in production. I also developed applications for the Android operating system using Android Studio, Eclipse IDE, SQLite, Java, XML, Android SDK, and ADT plugin. Show less