Senior Data Engineer
Chennai, Tamil Nadu, India
US hardware org for cloud migrations for their AI acquisitionsLarge Language Model Implementation:• Objective: Implemented LLM open source model to facilitate data querying via simple English language.• Roles and Responsibilities: Architected the solution and developed a functional apps.• Technologies Utilized: Dolly2.0, Databricks, Google Cloud Platform, Python, Code-bison, Text-bison, Langchain.• Achievements: Integrated Google Cloud-based text-bison as the LLM model in conjunction with Langchain to generate SQL queries based on user input. Designed an architecture to execute these queries within the Spark engine, fetching outcomes from Databricks' tables.• Collaboration: Worked with a team of two data engineers to transform the architecture into a working reality.• Learnings/Outcome: Initiated the project with the Dolly model but shifted to more efficient models available on the Google Cloud Platform to meet customer demands.Extraction of Oracle Fusion Data to Google Bucket:• Objective: Designed and implemented the complete data flow from Oracle Fusion Cloud to Google Cloud Platform without third-party applications.• Roles and Responsibilities: Conducted a proof of concept, architected the entire design, and translated it into a functional application using available technologies.• Technologies Utilized: SOAP, REST, Google Cloud Platform, PySpark, Python, Databricks, OCI, BICC, Oracle Bucket Storage, BigQuery.• Achievements: Successfully extracted data from Fusion application using BICC, transferred it to object storage, then loaded and transformed it into bronze and silver layers before reaching BigQuery as the gold layer. Managed the archiving of extracted data to clear the landing bucket space.• Collaboration: Worked individually but maintained regular communication with the client on project progress.• Learnings/Outcome: Gained in-depth knowledge of Oracle Cloud applications, their capabilities, and exposure to Oracle bucket storage APIs.
