Data Engineer
Solutionsoft Inc
New Jersey, United States
- Executed ETL operations, leveraging Amazon S3, and Amazon Managed Workflows for Apache Airflow(MWAA), resulting in a seamless data transfer.- Developed and executed SQL queries to transform and optimize data for a data warehouse design across different platforms(Amazon RDS for Oracle), resulting in a reduction in processing time.- Processed 100k+ rows of data from 20+ CSV files using Python scripts and efficiently loaded the data into the database.- Fast-tracked user productivity by eliminating a 400-data limit through a scalable AWS Lambda solution for large dataset uploads.- Automated CI/CD pipelines, reducing manual errors and streamlining software releases using Atlassian Bamboo, Azure DevOps, and Git.Utilized a tech stack including Sumo Logic, Python, Django, Postgres, Jira, BitBucket, Runscope, and Atlassian Bamboo for testing.