Faaiz Siddiqui Email & Phone Number

New York, New York, United States

- Developed and maintained scalable data pipelines using Apache Spark (Java) and AWS for real-time analytics, optimizing data flow from diverse sources.- Automated and scheduled data processing workflows using Apache Airflow, ensuring reliable and consistent data management.- Ensured data security by implementing AWS IAM roles and policies, maintaining data integrity and fine- grained access control.- Extracted and transformed data from Oracle databases using PySpark, storing it in AWS S3 for efficient downstream processing.- Optimized performance of Spark jobs by performing stress testing and tuning, minimizing latency in real- time data pipelines.

Whippany, New Jersey, United States

- Architected and implemented data lake solutions on AWS, utilizing services like Redshift and S3 for efficient data storage and retrieval.- Developed and orchestrated complex ETL pipelines using Apache Airflow, facilitating seamless data integration and transformation.- Led the development of real-time streaming applications using Apache Kafka, Apache Flink, and Apache Spark, enabling real-time data processing and analytics.- Managed the migration of on-prem Hadoop systems to AWS, significantly improving data accessibility and processing efficiency.- Collaborated with cross-functional teams to ensure the smooth integration of AWS-based data solutions into the overall architecture, optimizing performance and scalability.

Weehawken, New Jersey, United States

- Designed and maintained real-time data processing pipelines using Apache Spark Streaming and Kafka, enabling the ingestion, processing, and analysis of high-velocity financial transaction and market data.- Implemented complex event processing and windowing functions using Apache Flink to perform real- time anomaly detection and pattern recognition on streaming data.- Integrated Spark Streaming with Hadoop, Hive, and Elasticsearch to build a comprehensive data analytics platform for end-to-end workflows.- Conducted performance tuning and optimization of Spark Streaming applications and Kafka clusters, ensuring scalability, low latency, and efficient resource utilization.- Collaborated with cross-functional teams to integrate data processing solutions, ensuring seamless interaction between Spark Streaming, Kafka, and downstream consumers.

Seattle, Washington, United States

- Developed and optimized Hadoop-based data processing applications using MapReduce for large-scale data transformation and analysis, ensuring efficient data handling.- Utilized Apache Hive for SQL-like querying and analysis of data stored in HDFS, optimizing HiveQL queries for improved performance.- Created and managed real-time data access solutions using HBase, including schema design and performance tuning for low-latency data access.- Developed custom User-Defined Functions (UDFs) in Hive to extend functionality and perform complex data transformations.- Implemented data ingestion workflows using Flume and Sqoop, facilitating efficient data import/export between Hadoop and relational databases.

Hyderabad, Telangana, India

- Developed Python-based applications and automation scripts to streamline business processes, significantly reducing manual intervention and operational errors.- Designed and implemented SQL queries, stored procedures, and database schema designs for efficient data manipulation and extraction from relational databases.- Conducted comprehensive testing, including unit testing, integration testing, and regression testing, utilizing PyTest and version control through Git.- Collaborated with cross-functional teams, including business analysts and software developers, to gather requirements and implement effective software solutions.- Provided technical support and training to end users and stakeholders, ensuring successful adoption and smooth usage of the developed applications.