Data Engineer
Current1. Data Acquisition- Optimize and scale ETL pipelines for seamless data ingestion across multiple sources.- Design and implement robust, scalable data pipelines adhering to best practices.- Monitor data ingestion with Change Data Capture (CDC) for real-time updates.- Resolve failed batch jobs and implement automated recovery processes.- Maintain technical documentation and a centralized data dictionary with metadata and lineage.2. Data Extraction and Cleaning- Automate data extraction from diverse sources, integrating with platforms like Hadoop and Redshift.- Apply advanced data cleaning techniques, leveraging machine learning for anomaly detection, handling missing data, and ensuring consistency.- Conduct data quality checks using tools like Apache Nifi and Talend, ensuring accuracy, completeness, and reliability.3. Data Integration, Aggregation, and Representation- Develop and expose optimized data views/models using Hive, Impala, and Presto.- Provide AI/ML teams with cleaned datasets for model training.- Implement real-time data aggregation with Spark and Storm, ensuring low-latency analytics.- Create dashboards using Grafana, Prometheus, and Kibana for real-time monitoring.Skill Set:- Expertise in SQL for complex queries, window functions, and data partitioning.- Proficient in big data tools: Hive, HBase, Impala, Presto, and Apache Flink.- Strong experience with Apache Kafka for event-driven architectures and real-time streaming.- Hands-on with Hadoop ecosystem: HDFS, YARN, MapReduce, and Pentaho.- Advanced knowledge of NoSQL databases like Cassandra, HBase, and OpenTSDB.- Skilled in Java Spring Boot for microservices and data pipeline integration.- Experienced with monitoring tools: Grafana, Prometheus, Zabbix, and Elastic Stack.
