• Developing data processing jobs using Python frameworks such as PySpark and Great Expectation to generate business entities, normalize data, and create features and metrics.• Creation and maintenance of large-scale data processing pipelines from various data sources using Apache Spark and orchestrated by a self-managed Airflow system.• Working closely with the Data Science team, translating their POCs and models to production-ready scalable data processing jobs.• Cluster management and optimizations. Optimizing AWS EMR cluster configurations and Yarn resource allocation to increase hardware utilization and reduce processing runtime.

• Developing data-driven intelligence products based on tabular, graph, and image data.• Implementation of ETL workflow within data pipelines that creates ~3 billion edges on Cloudera clusters using Dagster, PySpark, HDFS.• Improved query performance for core product by X10.• Improve processes efficiency by implementing web scrapers, which saves 30% of man-hours.• Software project development from the design phase to user usability.

• IT support (Hardware and peripheral equipment).• Manage network equipment based on ISO standards.• Software deployment for end-user computers.