Senior Data Engineer
Current• Conducted data analysis using Matplotlib and seaborn libraries to visualize data effectively.• Implemented Spark with Scala and utilized Data frames and Spark SQL API for accelerated data processing.• Established data and reporting infrastructure using Power BI and SQL from scratch.• Developed large-scale data structures and pipelines using SQL and Python to standardize and organize data for insights and reporting.• Conducted feature engineering, model training, and comparison, and back-testing using Python (Pandas, Numpy, and Scipy).• Designed ETL processes for loading prospects into data marts and applied complex marketing segmentation for targeted mailing.• Created and maintained Logical Data Model documentation, including entities, attributes, relationships, keys, and business rules.• Developed interactive and informative Power BI reports using advanced DAX features.• Conducted Data visualization using Snowflake and PostgreSQL for insightful reporting.• Developed SQL scripts for data visualization and reporting.• Implemented data warehousing concepts and techniques using SQL.• Assessed existing data, identified gaps, and worked with Data Strategy and business partners to devise solutions.• Wrote ETL processes, designed database systems, and developed tools for real-time and offline analytics processing.• Designed and managed ETL processes using SQL.• Built data-cleaning ETL pipelines to process raw text and extract information for market analysis data (Python, PostgreSQL).