build data pipeline to ingest security data

Socialcredits is a startup company specialize in Chinese company data. We crawl data from news, job posting, registration, patent, legal announcement websites and integrate with other official APIs. With all these data we provide search, reporting, data analysis and risk monitoring services for our clientsData Pipeline• Developed distributed crawler framework, using python, nodejs(simulate browser) for html parsing, using config service to manage all the urls, scheduling, queue and storage configuration• Developed data pipeline from web crawlers to data cleaning/validation and ETL based on AWS cloud technologies such as S3, SQS, RDS, EC2, EMR• Using python pandas for smaller size data ETL and analysis, using EMR sparksql for large dataset ETL and analysis, Jupyter notebook to expose dataset as dataframe for data scientists and reportingApplication• built search API and other business logic API using SpringBoot, NodeJs, data storage using ElasticSearch and Postgres• built rule engine using embedded python script engine and let business user define data abstraction, build business model and define business rulesDevops• optimize AWS usage by using Reserved Instance, Spot Instance; utilizing boto3 SDK and terraform to build AWS infrastructure• built CICD pipeline using Jenkins, fabrics for different environments code deployment

- Develop cloud based payment REST services for managing credit card token, using spring framework (spring MVC, spring data)- Develop credential REST services for managing user login and client apikey and shared secret, using spring framework (spring MVC, spring data)- Develop single page application using backbone and handlebars- Develop content repository based on jackrabbit and serves content for rails application

X.commerce teamresponsible for building web consoles for developer and merchant to access x.commerce fabric and other services.

• Member View website: led development team on UI design, implementation, deployment and maintenance of member view website The site was written in Python using Django framework.• Oracle Advance Queuing and Message Gateway: • Rest web service: design and developed restful web services for other teams to access member insurance information(eligibility, demographic, claims)

Member Enrollment webapp: design, develop and deploy member enrollment webapp forbenefits administrators to do batch enrollment for insurance plan. The web app was writtenin JAVA using Struts2 framework. - Data access service: design, developed internal data access layer using hibernate, jpaon top of oracle database to expose member eligibility and claims information.- PL/SQL: design, developed pl/sql procedures to generate export to external partners andconvert RX transactions into claim transactions.- Edge technology: used Seebeyond as edge technologies to establish connections withpartners via MQ and used Pervasive to handle batch transform process.