Avinash E Email & Phone Number

Ohio, United States

•Led the initiation of an Azure Data Governance project to address data security and compliance requirements.•Implemented Dimensional Data Modeling to deliver Multi-Dimensional STAR schemas and Developed Snowflake Schemas by normalizing the dimension tables as appropriate.•Designed & implemented the new Azure Subscriptions, data factories, Virtual Machines, SQL Azure Instances, SQL Azure DW instances, and HD Insight clusters, and installed DMGs on VMs to connect to on-premises servers.•Developed the Spark Data Frames from various Datasets and applied business transformations and data cleansing operations in Azure Data Bricks

Nebraska, United States

•Collaborating with managers leads to understanding project requirements and creating functional specifications for Azure-based data platforms.•Utilized FiveTran for data integration and automation of the ETL process.•Integrated data from various sources such as Salesforce, and HubSpot.•Work closely with the data team to ensure data integrity, quality, and consistency across all DBT models and downstream applications.

Bengaluru, Karnataka, India

•Worked on building a centralized Data Lake on AWS Cloud utilizing primary services like S3, EMR, Redshift, and Athena.•Lead the team in deploying methodologies, ensuring cohesiveness and adherence to best practices.•Lead data curation efforts to ensure high-quality datasets, including data validation, cleansing, and monitoring processes.•Migrating the data from the local Teradata data warehouse to AWS S3 data lakes.• Implemented CI/CD by automating the spark jobs build and deployment using Jenkins and Ansible.• Highly proficient in developing lambda functions to automate tasks on AWS using CloudWatch triggers, S3 events as well as kinesis streams.• Developed the Python scripts using Boto3 to supplement automation provided by Terraform for tasks such as encrypting EBS volumes backing AMIs and scheduling Lambda functions for routine AWS tasks.• Worked on ETL Migration services by developing and deploying AWS Lambda functions for generating a serverless data pipeline which can be loaded to Glue Catalog and can be queried from AWS Athena.• Developed scalable data integration pipelines to transfer data from AWS S3 bucket to AWS Redshift database using Python and AWS Glue.• Analyzing files from the S3 data lake using AWS Athena and AWS Glue without importing the data into a database. • Created ad-hoc tables to add schema and structure data into AWS S3 bucket using lambda function and performed data validation, filtering, sorting, and transformations for every data change in a Dynamo DB table and load the transformed data to Postgres database.• Implemented Hadoop file system (HDFS), AWS S3 storage, and Bigdata formats including Parquet and AVRO JSON for enterprise data lake. •Optimized Druid query performance by tuning memory settings, caching strategies, and segment size.•Designed and implemented a real-time analytics system using Druid, Apache Kafka, and Apache Flink to analyze user behavior on our e-commerce platform.

Hyderabad, Telangana, India

• Enhanced Python programming to align with project needs, employing Flask REST Plus and APIs for crafting RESTful web services.• Deployed SSIS packages and reports in Dev and QA environments and created deployment documentation.• Created SSRS reports and dashboards based on customer specifications, ensuring satisfaction with data visualization.• Engineered interactive web-based solutions by designing database models, APIs, and views with Python.• Built a web application for data handling using Python, incorporating MYSQL and Flask REST Plus for backend processes.• Utilized SQL Alchemy and MYSQL to architect and model intricate details of applications.• Integrated NEO4J Graph Database within Python projects to form and analyze node relationships.• Constructed web services in Python using REST principles for efficient data communication with external systems in JSON format.• Integrated Flask-Application into OKTA.• Exported data from DB2 to HDFS using Sqoop and Developed MapReduce jobs using Python API.• Developed data reconciliation SQL queries and reports to validate overall data migration and support all scheduled ETL jobs for batch processing.• Installed and configured Pig and wrote Pig Latin scripts.• Assisted in exporting analyzed data to relational databases using Sqoop.• Involved in Developing monitoring and performance metrics for Hadoop clusters.• Continuous monitoring and managing of the Hadoop cluster through Cloudera Manager.• Built a GCP cost optimization dashboard and automated GCP instance shutdown/startup based on Stack driver metrics.• Created dashboard using Python Flask RESTful API.Environment: Python, Flask, Google Cloud Platform, AWS, S3, Elastic Search, OKTA, HDFS, Hive, Flume, Sqoop, HBase, PIG, Eclipse, Spark, My SQL and Ubuntu, Zookeeper, Maven, Jenkins, Java (JDK 1.6), Oracle, SSIS, SSRS, MS SQL Server, Azure VM,SQL