Manoj Kumar Email & Phone Number

United States

Experienced Data Engineer with a strong background in designing and implementing end-to-end data solutions on the Azure cloud platform. Skilled in data ingestion, transformation, and loading (ETL) from diverse sources into Snowflake data warehouse, I specialize in orchestrating robust data processing workflows using Azure Databricks and Apache Spark. With expertise across a wide range of Azure services, including Azure Data Factory, Azure Event Hubs, and Azure Functions, I have consistently delivered scalable and reliable data solutions to meet business needs. Additionally, I have a proven track record in implementing security and authentication measures within Azure environments, leveraging Azure Active Directory (Azure AD) to ensure data protection. Collaborative and results-driven, I excel in communicating with cross-functional teams to align data requirements and deliver impactful insights.

California, United States

In my extensive career as a data professional, I've spearheaded transformative initiatives, demonstrating proficiency in designing, implementing, and optimizing data workflows across various cloud platforms. With a focus on real-time analytics and efficient data processing, I've built Spark Streaming applications that provide actionable insights from streaming data. Leveraging Spark scripts in Scala and Spark SQL, I've ensured the effectiveness of data processing tasks and utilized Spark Streaming to divide streaming data into batches for further processing by the Spark engine.A key aspect of my contributions involves the implementation of Continuous Integration/Continuous Deployment (CI/CD) pipelines for seamless project building and deployment in the Hadoop environment. The integration of Spark with Python (PySpark) and Spark SQL has enabled faster testing and processing of data, demonstrating a commitment to efficiency in the development lifecycle.My expertise extends to various Azure services, including Azure Data Factory, Synapse Analytics, SQL Azure, Data Factory, Azure Analysis Services, Application Insights, Azure Monitoring, Key Vault, and Azure Data Lake. I've successfully created batch and streaming pipelines for ETL processes, and enhanced ETL processes by integrating Snowflake, ensuring the efficient loading of cleansed data into Snowflake data tables for improved storage efficiency.The incorporation of Snowflake into the existing data ecosystem has elevated data warehousing capabilities, enabling efficient analytics with advanced features. I've applied Directed Acyclic Graph (DAG) representation within Apache Flink for structured and efficient data management, seamlessly integrating with AWS DynamoDB and other cloud services. Data ingestion into Azure Data Lake Storage (ADLS) Gen2 and loading it into Delta tables after cleansing showcases my commitment to data integrity and seamless integration within the Azure environment.

Fountain Valley, California, United States

Worked on the development of data ingestion pipelines using ETL tool, Informatica, and bash scripting with big data technologies including but not limited to Hadoop, Hive, Spark, and Kafka.• Experience in developing, scalable and secure data pipelines for large datasets.• Gathered requirements for ingestion of new data sources including life cycle, data quality check, transformations, and metadata enrichment.• Developed data pipeline using Flume, Sqoop, Pig, and Java Map Reduce to ingest customer behavioral data into HDFS for analysis. • Supported data quality management by implementing proper data quality checks in data pipelines.• Enhancing Data Ingestion Framework by creating more robust and secure data pipelines.• Implemented data streaming capability using Kafka and Informatica for multiple data sources.• Involved in Sqoop implementation which helps in loading data from various RDBMS sources to Hadoop systems and vice versa.• Experience with Neo4j Enterprise edition, incorporating advanced features for scalability, security, and management.• Able to maximize the capabilities of the Enterprise edition in enterprise-level projects.• Implemented query caching mechanisms to store and reuse frequently executed queries, reducing the need for redundant processing and improving overall application speed.• Established proficiency in enhancing data processing and analytics performance by effectively utilizing Snowflake's seamless scalability, ensuring the efficient scaling of computing resources as needed.• Worked with multiple storage formats (Avro, Parquet) and databases (Hive) Azure SQL.• Optimizing query performance in Hive using bucketing and partitioning techniques.• Demonstrated proficiency in leveraging Snowflake's multi-cloud capabilities to strategically deploy instances across diverse cloud providers and regions, effectively enhancing redundancy and ensuring optimal data availability.

California, United States

• I Installed, configured, and maintained Apache Hadoop clusters for application development.• Managed Hadoop tools such as HDFS, Hive, Zookeeper, and Sqoop.• Engineered and optimized complex ETL workflows, resulting in a 25% reduction in data processing times.• Integrated Spark Streaming with Kafka for real-time data processing, enabling instant insights and analytics for critical business applications.• Implemented partitioning, dynamic partitions, and buckets in Hive for efficient data storage.• Installed and configured Sqoop to facilitate data import/export between relational databases and Hive.• Administered large Hadoop environments, overseeing cluster setup, and providing support.• Conducted performance tuning and monitoring in enterprise environments to optimize cluster performance.• Monitored and analyzed MapReduce job executions at the task level, optimizing Hadoop cluster components for enhanced performance.• Utilized Python and SAS for extracting, transforming, and loading source data from transaction systems.• Generated reports, insights, and key conclusions from processed data.• Designed and developed data mapping procedures for ETL processes, integrating data using Python programming.• Worked on the extraction, analysis, and loading of data within Hadoop and RDBMS environments.• Created and managed Hive tables for efficient data storage.• Collaborated with cross-disciplinary teams, fostering effective communication between data engineering and server administration teams for seamless project execution.• Participated in data loading processes and developed custom Hive UDFs.• Collaborated with Linux server admin teams to administer server hardware and operating systems.• Configured Spark Streaming to receive real-time data from Kafka and store the stream data in HDFS.

New York, United States

• Created, manipulated, and supported SQL Server databases.• Contributed to data modeling and physical and logical design of the database.• Facilitated integration of the front end with the SQL Server backend.• Developed Stored Procedures, Triggers, Indexes, User-defined Functions, and Constraints to achieve specific results.• Implemented data import and export between servers using tools like Data Transformation Services (DTS).• Wrote T-SQL statements for data retrieval and engaged in performance tuning of T-SQL queries.• Facilitated seamless data transfer between various sources such as MS Excel, MS Access, and Flat Files to SQL Server, utilizing SSIS/DTS for improved data integration.• Supported the team in resolving SQL Reporting Services and T-SQL-related issues.• Proficient in creating various types of reports including Crosstab, Conditional, Drill-down, Top N, Summary, Form, OLAP, and Sub reports.• Designed and developed ETL (Extract, Transform, Load) processes using Informatica PowerCenter to efficiently move and transform data from source systems to the data warehouse.• Developed and tested Windows command files and SQL Server queries for Production database monitoring in 24/7 support.• Developed, monitored, and deployed SSIS packages.• Introduced semantic data modeling techniques to enhance the overall understanding and interpretation of database structures, fostering improved communication between stakeholders and ensuring alignment with business requirements.• Contributed to advanced data modeling techniques, refining both the physical and logical design of databases for improved scalability and performance.• Generated multiple Enterprise reports (SSRS/Crystal/Impromptu) from SQL Server Database (OLTP) and SQL Server Analysis Services Database (OLAP).• Applied custom scripting techniques for data retrieval, offering efficient alternatives within T-SQL statements and ensuring optimized query performance.

Weehawken, New Jersey, United States

• Developed intricate stored procedures, efficient triggers, and necessary functions.• Created indexes and indexed views to enhance database performance.• Monitored SQL Server performance and conducted tuning activities.• Designed ETL data flows using SSIS for data extraction from SQL Server.• Created mappings and workflows for Data Migration and Transformation from Access/Excel Sheets using SQL Server SSIS.• Worked on Dimensional Data Modeling for Data Mart design.• Identified Facts and Dimensions and developed fact tables, and dimension tables, using Slowly Changing Dimensions (SCD).• Implemented Error and Event Handling strategies in SSIS.• Utilized precedent constraints, Break Points, Check Points, and Logging.• Built Cubes and Dimensions with different Architectures and Data Sources for Business Intelligence.• Wrote MDX Scripting for Cube development.• Developed SSAS Cubes, including Aggregation, KPIs, Measures, and Partitioning Cube.• Worked on Data Mining Models and deployed and processed SSAS objects.• Developed ad hoc reports with complex formulas.• Executed database queries for Business Intelligence purposes.• Developed Parameterized, Chart, Graph, Linked, Dashboard, Scorecards, and Drill-down reports using SSRS.• Created cascading reports for enhanced data analysis.• Utilized SQL Server for database development and optimization.• Applied SSIS for ETL processes and SSAS for Cubes development.• Developed various types of reports using SSRS.