Nani T Email & Phone Number

Boise, Idaho, United States

I led the development of a complex data pipeline project, orchestrating various data operations and transformations. This pipeline encompassed actions like file movement, Sqoop data extraction from Teradata and SQL sources, and data export into Hive staging tables. I collaborated on ETL tasks, ensuring data integrity and pipeline stability, with a focus on data retrieval from file systems using Spark commands.Python was central to our project, with code written for exploratory data analysis using machine learning libraries like Scikit-learn, NumPy, Pandas, and Matplotlib. We implemented Snowflake's COPY and INSERT statements, integrating Snowpipe for real-time data ingestion and transforming raw data into actionable insights.Our team also leveraged Cloud services for data ingestion and transformation and building a pipeline. We employed Python scripts for data cleansing, mapping, aggregation, and quality reporting, facilitating effective data management.Throughout the project, I utilized a diverse technology stack, including Python, PySpark, Docker, Kafka, and various data formats, to deliver a robust and efficient data pipeline solution.

Columbia, South Carolina, United States

Spearheaded a transformative project focused on optimizing data analysis and performance within a financial institution. Central to the initiative was the design and implementation of robust data processing pipelines leveraging Apache Spark for enhanced efficiency. Harnessing Spark's capabilities such as Spark Context, Spark Sessions, Data Frame, and Pair RDD's, the team developed scalable solutions for data transformation and analysis.In addition to Spark, advanced cloud technologies, particularly Cloud-based services, were utilized to bolster scalability and performance. Integrating Cloud HDInsight and other cutting-edge components ensured seamless adaptation to evolving data volumes and business requirements. This involved fine-tuning ETL workflows and optimizing resource allocation to maximize processing capabilities.Expertise extended to various data storage and processing technologies, including Hadoop, NoSQL databases such as MongoDB, HBase, and Cassandra, and real-time data ingestion frameworks like Kafka and Flume. Additionally, RESTful middleware was engineered to facilitate seamless communication between components, and distributed messaging systems were implemented for efficient data flow.

India

I spearheaded a multifaceted project, leveraging the Hadoop ecosystem through Cloudera Manager to unlock actionable insights from big data. Our focus on cost efficiency led us to optimize data engineering on AWS, utilizing tools like AWS Cost Explorer and Trusted Advisor. We adeptly collected and processed vast log data using Apache Flume, stored it in HDFS, and developed a versatile Python API for enhanced data processing.Within the Hadoop ecosystem, we employed a spectrum of technologies including MapReduce, Spark, Hive, Pig, Sqoop, HBase, Oozie, Impala, and Kafka. I played a pivotal role in transforming Hive/SQL queries into Spark transformations, using Spark RDDs, Python, and Scala. Data quality was our bedrock, with rigorous ETL validation processes ensuring accuracy and integrity.We delved into machine learning, experimenting with ensemble methods and integrating Cassandra with Spark and Scala. User-friendly web interfaces were created using Django, while automation streamlined RabbitMQ cluster installations and IoT data ingestion into Kafka. We also authored Kafka producers for seamless data streaming from external REST APIs.

India

Implemented R and Shiny applications for business forecasting, alongside predictive models in Python and R to analyze customer behavior and classify customers. Leveraged SQL Server Integration Services (SSIS) for seamless data extraction, transformation, and loading from multiple sources. Proficient in various R and Python packages like ggplot2, caret, dplyr, pandas, NumPy, and Seaborn for data manipulation and analysis. Conducted data cleansing using Excel functions and performed data retrieval operations from file systems to S3 using Spark commands. Utilized AWS services for data management and migration, including AWS Lambda and Amazon RDS, and developed healthcare dashboards in Tableau for real-time insights.Documented complete process flows and developed reports supporting consultants in process improvements, performing analytical analysis for insights and identifying future areas of improvement. Prepared reports using MS Excel functions like VLOOKUP, pivot tables, and Macros. Leveraged AWS Redshift, S3, Spectrum, and Athena services for efficient querying of large datasets stored on S3 to create a Virtual Data Lake without the need for extensive ETL processes.