I am a Data Engineer with 5 years of Experience in migrating on-premise databases to GCP. I am a Google Certified Associate Cloud Engineer.I have worked on the following migrations.1. Oracle to Bigquery:In this project, I have replicated the oracle on-premise database to Bigquery to get insights from the data. I have used GCP VM to extract the data from the oracle database as CSV files and load them into Cloud Storage which I used as the staging location. As I am extracting the data from the source, I extracted them in chunks of size, 1 Million. Each extract is given a unique job id which will be used in the next step. As the next step, I took the job id generated and used that job id to load the data that is stored in cloud storage to Bigquery using the Bigquery python API. All the actions that we have performed till now will be logged and also there are control tables that will store the last updated timestamp so that I can help us in loading incremental data. I have automated the whole batch and incremental load pipelines without requiring no manual intervention at all.Python libraries used: cx_oracle, pandas, google-cloud-storage, google-cloud-bigquery, datetime, os.GCP services used: Bigquery, Cloud Storage, Cloud Scheduler, Cloud Functionsother technologies used : shell scripting2. Postgres to CloudSQL:As discussed in the previous project, The first part is also the same here i.e., loading the data from the source to the GCS bucket. Now from the second part i.e., loading the data from the GCS bucket to CloudSQL, I have used cloud functions which will automatically trigger on ingestion of the CSV file into the specified GCS bucket. And the python script will automatically load the data into CloudSQL. I have kept control tables which will help in the incremental load. This pipeline has been automated completely and doesn't require any manual intervention.Python libraries used: psycopg2, gcsfs. google-cloud-storage,GCP services used : Cloud SQL, Cloud FunctionsAlso I have worked on the below migrations3. MySQL to CloudSQL4. SQLServer to Bigquery#etl #googlecloud #spark #migration #machinelearning #dataengineering
