Senior Software Engineer
Current• Orchestrated backend Python development, crafting a scalable, object-oriented codebase with thousands of lines, alongside multiple bug fixes.• Designed a Python library streamlining data retrieval from Google Monitoring and BigQuery via API calls, culminating in an 80% reduction in frontend loading time.• Engineered advanced algorithms and modules leveraging MQL and SQL queries, offering cost-saving recommendations and extensive insights for optimizing GCP resources.• Established an end-to-end pipeline by orchestrating GitHub repositories and integrating Cloud Build, Cloud Function, Cloud Scheduler, and Monitoring, automating deployment and synchronization tasks effectively.Developed a Python dataset using the Faker library, meticulously tailored to the schema requirements, while setting up two essential S3 buckets - the 'landing zone' and 'staging zone'. Configured Spark with a dedicated EMR cluster, enabling a Spark job to handle data transformation from the landing zone to the staging zone based on paths defined in the app config file. Deployed an EC2 instance housing Airflow, creating a DAG responsible for scheduling Spark job submissions via Livy, all parsed from the config file. Additionally, established a Redshift serverless cluster, orchestrated a Lambda function to efficiently transfer data from the S3 Staging zone to Redshift, equipped with necessary Lambda Layer and IAM Role for seamless data integration and analysis.
