Senior Python Data Engineer
Current• Automated workflows with Python scripts and UNIX shell scripting.• Utilized Python unit and functional testing modules like unittest, unit test2, mock, and custom frameworks in line with Agile Software Development methodologies.• Developed Sqoop scripts for change data capture to process incremental records between new and existing data in RDBMS tables.• Installed and managed Hadoop, MapReduce, and HDFS, and developed MapReduce jobs in PIG and Hive for data cleaning and preprocessing.• Managed datasets using Pandas Data Frames and queried MySQL databases from Python using MySQL Connector and MySQL DB package.• Worked on web/application development using Python 3.5, HTML5, CSS3, AJAX, JSON, and jQuery.• Developed and tested features for dashboards using Python, Java, Bootstrap, CSS, JavaScript, and jQuery.• Generated Python Django forms to record user data and wrote test cases using PyTest.• Cleaned and processed data using Excel macros and Python libraries like NumPy, SQLAlchemy, and matplotlib.• Utilized Pandas to handle data in time series and tabular formats for manipulation and retrieval.• Assisted in migrating old server data to Jira using Python scripts to transfer and verify information.• Designed and managed API system deployment using Fast HTTP Server and Amazon AWS architecture.• Integrated with third-party platforms using RESTful web services, successfully implementing Apache Spark and Spark Streaming for large-scale data.• Collaborated via platforms like Slack and Microsoft Teams for team communication.• Developed web applications with Python/Django, incorporating client-specific customizations.• Designed application architecture and developed APIs for Google Analytics aggregation and reporting.• Worked with Python OpenStack APIs and integrated AWS S3 and RDS for static/media file hosting and database management.