Data Engineer
CurrentWorking on a cost-effective, end-to-end data pipeline for audio transcription to text: 1) The entire pipeline is set up using Localstack an AWS Emulator on local machine2) Services like S3 for storage , AWS Lambda for processing and AWS Kinesis/Firehose for data movement are implemented 3) AWS infrastructure set up using terraform4) Open AI whisper is used for speech to text conversion with rate limit and size limit handling5) Multi-threading implementation to transcribe audio chunks in parallel5) Transcribed text written to ElasticSearch (Document Based) , Neo4J (Graph) and Chromadb (Vector) for comprehensive text analysis. 6) Entire set up is done using docker avoiding hassles of any installation 7) Entirely local setup, with no running costs apart from the OpenAI API key (which can be replaces with transcription service from Localstack ), making it a robust, budget-friendly solution for processing audio data8) Seamlessly shift to AWS cloud for scalability