Tejaswini S Email & Phone Number

Chicago, Il, Us

• Involved in requirements gathering interacting with product owners and business analysts to understand the requirements. • Developed Spark Scripts using Spark Session, Python, Spark SQL to access hive tables into spark for faster processing of data.• Worked on Big Data Integration and Analytics based on Hadoop, SOLR, Spark, Kafka, Storm and web Methods technologies.• Collected data using Spark Streaming from AWS S3 bucket in near-real-time and performs necessary Transformations and Aggregation on the fly to build the common learner data model and persists the data in HDFS.• Creating pipelines using python with spark framework to ingest data from different sources to the client’s destination.• Developed Spark applications to implement various aggregation and transformation functions of Spark RDD and Spark SQL.• Developed ETL for ingestion pipelines which run on AWS infrastructure setup of EMR, S3, RedShift, Aurora, Glue, SQS and Lambda.• Used Broadcast Join in SPARK for making smaller datasets to large datasets without shuffling of data across nodes. • Used Oozie Scheduler systems to automate the pipeline workflow and orchestrate the Spark jobs.• Developed data processing applications in Scala using SparkRDD as well as Dataframes using SparkSQL APIs.• Implemented Nifi flow topologies to perform cleansing operations before moving data into HDFS. • Used Apache NiFi to copy data from local file system to HDP.• Designed and implemented Sqoop for the incremental job to read data from DB2 and load to Hive tables and connected to Tableau for generating interactive reports using Hive server2.• Used Spark Streaming to receive real time data from the Kafka and store the stream data to HDFS using Python and NoSQL databases such as HBase and Cassandra• Implemented Akka model for event driven architecture.• Followed Agile/SCRUM methodology for Software Development Life Cycle

Hyderabad, Telangana, India

• Designed and documented the new architecture and development process to convert existing ETL pipeline into Hadoop-based systems.• Extensively worked on Spark Scala to prepare data for building models which will be consumed by Data Science team.• Involved in loading data from UNIX file system to HDFS.• Extracted the data from Teradata into HDFS using Sqoop.• Developed Streaming data pipelines using Spark Scala, HBase and Kafka.• Developed a generic framework using Spark for processing/Flatten JSON data that is re-used by various applications within the enterprise.• Created Linked service to land the data from different sources to Azure Data Factory.• Handled structured and unstructured data and applying ETL processes. • Performance tuning of Spark Applications from code, resource and data point views.• Developed Oozie Workflows for daily incremental loads, which gets data from Teradata and then imported into hive tables using Sqoop.• Designed and performance tuned hive tables and queries from storage, file formats and query levels• Used Hive to analyses the partitioned and bucketed data and compute various metrics for reporting on the dashboard. Involved in Requirements gathering, Requirement analysis, Design, Development, Integration and Deployment.• Wrote and executed various MYSQL database queries from python using Python-MySQL connector and MySQL dB package.• Worked on Cloud databases such as Azure SQL Database, SQL managed instance, SQL Elastic pool on Azure, and SQL server.• Performed Data Analysis, Data Migration, Data Cleansing, Transformation, Integration, Data Import, and Data Export through Python.