Senior Architect, Data Analytics
CurrentCustomer: Point72 Asset Management, USARole: Technical Architect● Implemented security strategy for the ELK Stack. With controlled access to Elasticsearch and Kibana, users are prevented with unauthorized access to data across the departments within the same organization.● FpML Trades data (XML format) is consumed from Kafka topics, parsed using XQuery and XPath within Apache NiFi and loaded into Apache Phoenix NoSql Database. Also, few of the trades were transposed using XSL… Show more Customer: Point72 Asset Management, USARole: Technical Architect● Implemented security strategy for the ELK Stack. With controlled access to Elasticsearch and Kibana, users are prevented with unauthorized access to data across the departments within the same organization.● FpML Trades data (XML format) is consumed from Kafka topics, parsed using XQuery and XPath within Apache NiFi and loaded into Apache Phoenix NoSql Database. Also, few of the trades were transposed using XSL Template.● Processed large and complex XML files using StAX (Streaming XML) in Pentaho Data Integration. This resolved the out of memory issue with Carte Server.● Implemented Rolling logs using Log4J Framework for Pentaho Application Logs and Carte Server Logs. Logs are zipped, archived and time-stamped with house-keeping.● AVRO messages that are consumed from Kafka Topics are deserialized in the Apache NiFi data pipeline with the Hortonworks Schema Registry ID and converted to JSON format.● Created Apache NiFi Custom Processors (Plugins):• (a) to read Kafka AVRO Key with Base64 encoding as Attribute• (b) to read Kafka Timestamp as Attribute• (c) to convert Attributes to Flowfile• (d) Evaluate Null Json Paths● Data Migration from Apache Phoenix NoSql Database to Microsoft SQL Server with MERGE in real-time using Apache NiFi.● Data Validation between Apache Phoenix NoSql Database and Microsoft SQL Server using Pentaho Transformation.● Parsing Confidential PDF files to CSV using Apache Tika library in Pentaho and Regex in User Defined Java Class.● Developed Pentaho Custom Plugin for Elasticsearch Bulk Load that works over Elasticsearch REST API with options to provide batch size and timeout. Indexed 6 million documents with improved performance of 20x.● Provided resilience capability to Pentaho REST Step Plugin with configurable Max Retries and Wait Time.● Privileged Security Access to MSSql with Kerberos and MongoDB with CyberArk. Show less