Automated data ingestion using AWS Lambda by creating trigger lambda functions for real-time ingestion.Build robust ETL pipelines to support various data sources in storage solutions like Amazon Redshift.Integrated diverse data sources using Apache Kafka and AWS Glue to create a unified data platform.Data transformation by leveraging PySpark on Amazon EMR to perform complex data transformations at scale, including joins, aggregations, and filtering.Created lambda functions to connect data across various data systems on-premise and on the cloud.Develop infrastructure to support real-time streaming of Kafka events into DynamoDB.Optimized ETL jobs to achieve 98% uptime and ensure latency of less than 24 hours.Ensured the data pipeline supported ML model training, deployment, and retraining, and assisted in feature engineering and data selection for model inputs. Set up monitoring and logging systems using tools like Airflow to track pipeline performance.Document notebooks for pipeline architecture, ETL processes, and infrastructure configurations.

Designed and implemented data pipelines in Azure Data Factory (ADF) using linked services and datasets to extract, transform, and load data from various sources including Azure SQL, Blob Storage, and Azure SQL Data Warehouse.Developed real-time data extraction processes using Spark Streaming and Kafka, enhancing the timeliness and accuracy of patient analytics.Ingested data from several IoT devices and application logs using Azure Streaming Analytics for real-time ingestion and leveraged Azure Databricks for advanced data processing and transformation.Used SparkSQL to efficiently read and process Parquet data, and loaded tables into Hive using Scala. Enabled improved querying performance and data management.Worked with SnowSQL and Snowpipe for continuous data loading and cross-account data sharing, ensuring up-to-date and consistent data availability.Developed an integration to consume consumer subscription data from Azure Queue Storage and loaded it into Snowflake tables, streamlining data processing and analysis.Implemented proofs-of-concept for integrating GraphQL and REST APIs.

Integrated data from various sources, including IoT devices and historical maintenance logs, into a unified dataset using custom Python scripts.Implemented data cleaning and preprocessing routines using Skit-learn, pandas, and Numpy by applying techniques like KNN, random forest regressor, capping, and data augmentation.Normalized and transformed data by feature engineering to prepare it for model training.Develop a Time series ML model to minimize downtime and maintenance costs by enabling timely interventions based on predictive insights using TensorFlow and PyTorch.Deployed model via Web Interface using Django to monitor performance in real time.Integrated the model’s development workflow with CI/CD tools like Jenkins to automatically build, test, and validate the model and its Docker container on every commit.Design and implement RESTful APIs to connect front-end elements to interact with back-end model logic.

Developed Power automate flows for extracting data from a legacy system using Rest API and integrated it with Informatica.Developed ML models, including regression, classification, and time-series models for customer segmentation, sales forecasting, and optimizing inventory metrics.Create data visualizations using libraries like Matplotlib, or Seaborn to present complex data insights.Developed interactive Power BI dashboards and integrated them into a Flask based web interface to present actionable insights, enhancing CXO engagement.Provide analytical support through 50+ Ad-hoc reports, resulting in $100K+ annual sales growth and a 10% improvement in key performance indicators (KPIs).Continuously monitor system performance post-development, conduct unit testing, and implement regular updates to ensure optimal operation.