Data Engineer
CurrentAs a Data Engineer / Software Engineer at PLC Group, I have played a pivotal role in designing, developing, and maintaining highly efficient ETL (Extract, Transform, Load) pipelines to facilitate data processing and analysis. My expertise primarily revolves around leveraging Python programming and PostgreSQL database, with additional experience in Spark, Apache Superset, and Apache Airflow.Key Responsibilities:ETL Pipeline Development: Designed and implemented end-to-end ETL pipelines to ingest, process, and transform large volumes of data from diverse sources into a format suitable for analysis and reporting.Data Modeling: Utilized my strong understanding of database systems to design data models that optimize query performance and ensure data accuracy.Python Programming: Leveraged Python to develop custom data processing scripts, automating routine tasks, and enhancing data quality.Database Management: Managed PostgreSQL databases, including schema design, performance tuning, and data migration tasks.Data Visualization: Utilized Apache Superset to create interactive dashboards and visualizations, enabling stakeholders to gain valuable insights from data.Performance Optimization: Continuously monitored and optimized ETL processes for improved performance, ensuring timely data availability for analytics teams.Collaboration: Worked closely with cross-functional teams, including data scientists, analysts, and business stakeholders, to understand data requirements and deliver data solutions that drive informed decision-making.Documentation: Maintained comprehensive documentation of ETL processes, data models, and workflows to facilitate knowledge sharing and future maintenance.