Data Engineer
CurrentIn my role as a Data Engineer,I played a key role in designing, developing, and implementing automated data pipelines for marketing data. I leveraged my expertise in Python, Spark, and CI/CD methodologies to streamline the extraction, transformation, and ingestion of data from various marketing platforms. This included utilizing Spark Structured Streaming to enable real-time data processing, which facilitated faster insights generation and data-driven decision making.Furthermore, I played a critical role in optimizing Comcast's data infrastructure and processes. I successfully migrated on-premise data solutions to the cloud, establishing a data lake in cloud storage and a data warehouse in BigQuery. This migration improved scalability, accessibility, and efficiency for data management. Additionally, I optimized data modeling processes within TopBraid, resulting in enhanced efficiency and performance for data analysis tasks.Beyond technical infrastructure, I also focused on developing solutions to improve data accessibility and usability for various stakeholders across the organization. I created User-Defined Functions (UDFs) to automate complex calculations and data transformations, streamlining data analysis workflows. I also developed Sqoop scripts to facilitate data ingestion from diverse sources, including HDFS, Teradata, SQL Server, and PostgreSQL. This expanded data accessibility empowered informed decision making across various departments.Finally, I played a key role in building efficient data pipelines in Airflow using various operators for ETL automation. This automation ensured timely data availability and consistency, while also reducing manual intervention and potential errors.