Python Developer
Current• Worked on building PySpark algorithms for different aggregations of data based on the specifications. • Involved with performance and process enhancement of the PySpark framework.• Developed integration checks around the PySpark framework for Processing of large datasets. • Worked on migration of PySpark framework into AWS Glue for enhanced processing. • Wrote various automation scripts for automation of data processing on AWS Glue.• Developed data warehouse model in snowflake for over 100 datasets using whereScape.• Worked on Scripting CloudFormation Templates to auto provisioning of resources. • Worked of various Quality control checks for data processing using Spark SQL. • Build a data processing algorithm for incremental processing of data that’s aggregated each week.• Wrote ETL jobs to read from web APIs using REST and HTTP calls and loaded into HDFS using java and Talend.• Worked on data transformation like cleaning, partitioning of data for enhanced processing of data. • Worked on setting up of SEE5 data mining server on AWS for pattern finding in the data• Implemented Change Data Capture technology in Talend in order to load deltas to a Data Warehouse.• Created and maintained reports and dashboards to provide fast access to key business metrics.• Enhanced apex class and Visualforce page to create a custom related list, showing activities for selected contacts or clients.• Wrote an apex trigger on contact for cross object field update for reporting purposes.• Involved with migration of AI based project from Google Cloud Platform into AWS.• Build Schedule based AWS Lambdas, SageMaker for MLOps for automatic model build and run of data processing pipelines • Worked on building a standalone UI tool using wxPython for creating custom requirement JSON templets to feed into AWS Glue for ETL processing of data. • Worked on setting of automated loading of data into SQL database using AWS Glue and Step Functions.