Running the OAuth Token Migration Utility

To run the OAuth Token Migration utility:
  1. Go to the directory where the utility is deployed, for example: /opt/mashery/migration.
  2. Go to the application bin directory.
  3. Run the startup command.
For example: 
[root@88dfda2eecab builder]# cd /opt/mashery/migration/
[root@88dfda2eecab migration]# cd tml-token-migration-v1.0.0/bin
[root@88dfda2eecab bin]# ./tml-token-migration ../conf/config.properties
[INFO ] 2019-03-17 07:23:41.403 [main] c.m.tml.token.migration.Launcher - Migration program starting now with configuration file: ../conf/config.properties
[INFO ] 2019-03-17 07:23:41.413 [pool-2-thread-2] c.m.t.t.m.s.job.BatchReadJob - Starting read thread now
[INFO ] 2019-03-17 07:23:41.874 [pool-2-thread-2] c.m.t.t.m.s.job.BatchReadJob - Reading data between interval : Fri Feb 22 10:00:02 PST 2019 - Mon Feb 25 07:05:02 PST 2019
[INFO ] 2019-03-17 07:23:41.879 [pool-2-thread-2] c.m.t.t.m.s.FilePeristenceManager - Reading File data between interval : Fri Feb 22 10:00:02 PST 2019 - Mon Feb 25 07:05:02 PST 2019
[INFO ] 2019-03-17 07:23:41.897 [pool-2-thread-2] c.m.t.t.m.s.FilePeristenceManager - Read of File source tokens finished, total no of file tokens fetched 26550
...
..
If the path to config.properties is not provided, it will use a default path, for example, tml-token-migration-v1.0.0/conf/config.properties. 
[root@64bfbb9d9706 bin]# ./tml-token-migration
[INFO ] 2019-04-05 13:13:52.265 [main] c.m.t.t.m.MigrationConfiguration - Migration is configured for SOURCE = CASSANDRA
[INFO ] 2019-04-05 13:13:52.274 [main] c.m.t.t.m.MigrationConfiguration - Migration is configured for DESTINATION = FILE
[INFO ] 2019-04-05 13:13:52.347 [main] c.m.tml.token.migration.Launcher - Migration program starting now with configuration file: ../conf/config.properties
...

Running the Utility for the First Time

Upon running the utility for the first time, it will try to migrate all tokens (oldest updated token date to latest updated token date).

Running the Utility for the Second Time

Once all tokens are migrated from Source to Destination, just before switching over the load balancer to point to the Destination cluster, the migration utility needs to be run once again to migrate those tokens which were created/updated during the first migration timeframe window.

Running the Utility for the Third Time

Now after second time migration run, load balancers are switched over to the Destination cluster, then the migration utility needs to be run once again to migrate any leftover tokens which were created/updated during the second run migration timeframe window.

The three runs will ensure that all tokens are migrated from source to destination.

Failover Run in Case Process was Stopped during any Migration Run

If the process gets killed or stopped on its own in between the migration process, it will start from where it finished in the last run.

For example:
  • In the first run, the utility started migrating all the tokens between time interval, 22-01-2019 02:00:00 to 15-03-2019 04:45:10 (both inclusive).
  • After some time, the process got killed or stopped on its own due to some reason, at that time, it migrated tokens till updated time, 01-02-2019 03:40:00. This time entry is written in the Journal file (location at /opt/mashery/migration/tml-token-migration-v1.0.0/data/MigrationJournal.txt) 
[root@64bfbb9d9706 bin]# cat ../data/MigrationJournal.txt
1548182063000    Migrate-Token    600000    Fri Apr 05 15:08:28 UTC 2019    Fri Apr 05 16:28:39 UTC 2019
  • So in the subsequent run, it will try to pick tokens between 01-02-2019 03:40:00 and 15-03-2019 04:45:10 (both inclusive).

Token Migration Notes

  • Create separate data directory for each run if doing FILE based migration.
  • Change destination file directory (created during MySQL to FILE data export) to source file directory when running migration from FILE to CASSANDRA.
  • Clear journal entry of last run (FILE to CASSANDRA import) when executing next migration run from FILE to CASSANDRA.

Checkpoint File

After every run, the utility creates an entry in a checkpoint file named MigrationJournal.txt. It essentially records the updated timestamp of last token which was written to destination.

Format (entries separated by tab): 
Last_written_record_updated_timestamp      Id_of_last_token      Source          No_of_records_written       Batch_start_time               Batch_end_time
For example: 
1548182063000                           234212           Migrate-Token    600000                        Fri Apr 05 15:08:28 UTC 2019    Fri Apr 05 16:28:39 UTC 2019
This entry will be used in subsequent runs to load the records from the point it stopped writing.

Sample Run Logs

root@d2d3421x1332:bin$ ./tml-token-migration ../conf/config.properties
[INFO ] 2019-04-01 18:58:10.283 [main] c.m.t.t.m.MigrationConfiguration - Migration is configured for SOURCE = MYSQL
[INFO ] 2019-04-01 18:58:10.287 [main] c.m.t.t.m.MigrationConfiguration - Migration is configured for DESTINATION = CASSANDRA
[INFO ] 2019-04-01 18:58:10.329 [main] c.m.tml.token.migration.Launcher - Migration program starting now with configuration file: ../conf/config.properties
[INFO ] 2019-04-01 18:58:10.894 [pool-2-thread-2] c.m.t.t.m.s.job.BatchReadJob - Starting read thread now
[INFO ] 2019-04-01 18:58:11.397 [pool-2-thread-2] c.m.t.t.m.s.job.BatchReadJob - Reading data between interval : 2019-01-22 23:47:41.0 - 2019-01-23 03:56:07.0
[INFO ] 2019-04-01 18:58:11.397 [pool-2-thread-2] c.m.t.t.m.s.MySQLPersistenceManager - Reading MySQL data between interval : 2019-01-22 23:47:41.0 - 2019-01-23 03:56:07.0
[INFO ] 2019-04-01 18:58:13.802 [pool-2-thread-2] c.m.t.t.m.s.MySQLPersistenceManager - Read of MYSQL source tokens finished, total no of file tokens fetched 211991
[INFO ] 2019-04-01 18:58:13.803 [pool-2-thread-2] c.m.t.t.m.s.MySQLPersistenceManager - Read of MySQL source tokens completed...
[INFO ] 2019-04-01 18:58:13.803 [pool-2-thread-2] c.m.t.t.m.s.job.BatchReadJob - Fetched data from source, size of 211991 no of records
[INFO ] 2019-04-01 18:58:13.833 [pool-2-thread-1] c.m.t.t.m.s.job.MigrationJob - Submitted Batch read Job to executor
[INFO ] 2019-04-01 18:58:13.833 [pool-2-thread-1] c.m.t.t.m.s.job.MigrationJob - Submitted Batch write Job with 1 threads
[INFO ] 2019-04-01 18:58:14.823 [pool-2-thread-2] c.m.t.t.m.s.job.BatchWriteJob - Starting write thread now
[INFO ] 2019-04-01 18:58:14.974 [pool-2-thread-2] c.m.t.t.m.s.job.BatchWriteJob - Total 211991 found to be migrated, migrating now.
[INFO ] 2019-04-01 18:58:20.917 [pool-2-thread-2] c.m.t.t.m.s.CassandraPersistenceManager - Cassandra Write Batch commit started at: 2019-04-01 18:58:20.917 and finished at: 2019-04-01 18:58:20.917. Total writes so far 5000
[INFO ] 2019-04-01 18:58:24.972 [pool-2-thread-2] c.m.t.t.m.s.CassandraPersistenceManager - Cassandra Write Batch commit started at: 2019-04-01 18:58:24.971 and finished at: 2019-04-01 18:58:24.971. Total writes so far 10000
[INFO ] 2019-04-01 18:58:28.889 [pool-2-thread-2] c.m.t.t.m.s.CassandraPersistenceManager - Cassandra Write Batch commit started at: 2019-04-01 18:58:28.889 and finished at: 2019-04-01 18:58:28.889. Total writes so far 15000
[INFO ] 2019-04-01 18:58:32.999 [pool-2-thread-2] c.m.t.t.m.s.CassandraPersistenceManager - Cassandra Write Batch commit started at: 2019-04-01 18:58:32.999 and finished at: 2019-04-01 18:58:32.999. Total writes so far 20000
[INFO ] 2019-04-01 18:58:37.974 [pool-2-thread-2] c.m.t.t.m.s.CassandraPersistenceManager - Cassandra Write Batch commit started at: 2019-04-01 18:58:37.974 and finished at: 2019-04-01 18:58:37.974. Total writes so far 25000
[INFO ] 2019-04-01 18:58:42.119 [pool-2-thread-2] c.m.t.t.m.s.CassandraPersistenceManager - Cassandra Write Batch commit started at: 2019-04-01 18:58:42.119 and finished at: 2019-04-01 18:58:42.119. Total writes so far 30000
[INFO ] 2019-04-01 18:58:46.175 [pool-2-thread-2] c.m.t.t.m.s.CassandraPersistenceManager - Cassandra Write Batch commit started at: 2019-04-01 18:58:46.175 and finished at: 2019-04-01 18:58:46.175. Total writes so far 35000
[INFO ] 2019-04-01 18:58:50.516 [pool-2-thread-2] c.m.t.t.m.s.CassandraPersistenceManager - Cassandra Write Batch commit started at: 2019-04-01 18:58:50.516 and finished at: 2019-04-01 18:58:50.516. Total writes so far 40000
[INFO ] 2019-04-01 18:58:55.163 [pool-2-thread-2] c.m.t.t.m.s.CassandraPersistenceManager - Cassandra Write Batch commit started at: 2019-04-01 18:58:55.163 and finished at: 2019-04-01 18:58:55.163. Total writes so far 45000
[INFO ] 2019-04-01 18:58:59.345 [pool-2-thread-2] c.m.t.t.m.s.CassandraPersistenceManager - Cassandra Write Batch commit started at: 2019-04-01 18:58:59.345 and finished at: 2019-04-01 18:58:59.345. Total writes so far 50000
[INFO ] 2019-04-01 18:59:03.493 [pool-2-thread-2] c.m.t.t.m.s.CassandraPersistenceManager - Cassandra Write Batch commit started at: 2019-04-01 18:59:03.492 and finished at: 2019-04-01 18:59:03.492. Total writes so far 55000
[INFO ] 2019-04-01 18:59:07.581 [pool-2-thread-2] c.m.t.t.m.s.CassandraPersistenceManager - Cassandra Write Batch commit started at: 2019-04-01 18:59:07.581 and finished at: 2019-04-01 18:59:07.581. Total writes so far 60000
[INFO ] 2019-04-01 18:59:11.670 [pool-2-thread-2] c.m.t.t.m.s.CassandraPersistenceManager - Cassandra Write Batch commit started at: 2019-04-01 18:59:11.67 and finished at: 2019-04-01 18:59:11.67. Total writes so far 65000
[INFO ] 2019-04-01 18:59:15.822 [pool-2-thread-2] c.m.t.t.m.s.CassandraPersistenceManager - Cassandra Write Batch commit started at: 2019-04-01 18:59:15.822 and finished at: 2019-04-01 18:59:15.822. Total writes so far 70000
[INFO ] 2019-04-01 18:59:19.963 [pool-2-thread-2] c.m.t.t.m.s.CassandraPersistenceManager - Cassandra Write Batch commit started at: 2019-04-01 18:59:19.963 and finished at: 2019-04-01 18:59:19.963. Total writes so far 75000
[INFO ] 2019-04-01 18:59:24.156 [pool-2-thread-2] c.m.t.t.m.s.CassandraPersistenceManager - Cassandra Write Batch commit started at: 2019-04-01 18:59:24.156 and finished at: 2019-04-01 18:59:24.156. Total writes so far 80000
[INFO ] 2019-04-01 18:59:29.087 [pool-2-thread-2] c.m.t.t.m.s.CassandraPersistenceManager - Cassandra Write Batch commit started at: 2019-04-01 18:59:29.086 and finished at: 2019-04-01 18:59:29.086. Total writes so far 85000
[INFO ] 2019-04-01 18:59:35.411 [pool-2-thread-2] c.m.t.t.m.s.CassandraPersistenceManager - Cassandra Write Batch commit started at: 2019-04-01 18:59:35.411 and finished at: 2019-04-01 18:59:35.411. Total writes so far 90000
[INFO ] 2019-04-01 18:59:39.609 [pool-2-thread-2] c.m.t.t.m.s.CassandraPersistenceManager - Cassandra Write Batch commit started at: 2019-04-01 18:59:39.609 and finished at: 2019-04-01 18:59:39.609. Total writes so far 95000
[INFO ] 2019-04-01 18:59:43.791 [pool-2-thread-2] c.m.t.t.m.s.CassandraPersistenceManager - Cassandra Write Batch commit started at: 2019-04-01 18:59:43.79 and finished at: 2019-04-01 18:59:43.79. Total writes so far 100000
[INFO ] 2019-04-01 18:59:48.208 [pool-2-thread-2] c.m.t.t.m.s.CassandraPersistenceManager - Cassandra Write Batch commit started at: 2019-04-01 18:59:48.208 and finished at: 2019-04-01 18:59:48.208. Total writes so far 105000
[INFO ] 2019-04-01 18:59:52.519 [pool-2-thread-2] c.m.t.t.m.s.CassandraPersistenceManager - Cassandra Write Batch commit started at: 2019-04-01 18:59:52.519 and finished at: 2019-04-01 18:59:52.519. Total writes so far 110000
[INFO ] 2019-04-01 18:59:56.775 [pool-2-thread-2] c.m.t.t.m.s.CassandraPersistenceManager - Cassandra Write Batch commit started at: 2019-04-01 18:59:56.775 and finished at: 2019-04-01 18:59:56.775. Total writes so far 115000
[INFO ] 2019-04-01 19:00:01.060 [pool-2-thread-2] c.m.t.t.m.s.CassandraPersistenceManager - Cassandra Write Batch commit started at: 2019-04-01 19:00:01.06 and finished at: 2019-04-01 19:00:01.06. Total writes so far 120000
[INFO ] 2019-04-01 19:00:05.392 [pool-2-thread-2] c.m.t.t.m.s.CassandraPersistenceManager - Cassandra Write Batch commit started at: 2019-04-01 19:00:05.392 and finished at: 2019-04-01 19:00:05.392. Total writes so far 125000
[INFO ] 2019-04-01 19:00:10.743 [pool-2-thread-2] c.m.t.t.m.s.CassandraPersistenceManager - Cassandra Write Batch commit started at: 2019-04-01 19:00:10.743 and finished at: 2019-04-01 19:00:10.743. Total writes so far 130000
[INFO ] 2019-04-01 19:00:15.069 [pool-2-thread-2] c.m.t.t.m.s.CassandraPersistenceManager - Cassandra Write Batch commit started at: 2019-04-01 19:00:15.069 and finished at: 2019-04-01 19:00:15.069. Total writes so far 135000
[INFO ] 2019-04-01 19:00:21.381 [pool-2-thread-2] c.m.t.t.m.s.CassandraPersistenceManager - Cassandra Write Batch commit started at: 2019-04-01 19:00:21.381 and finished at: 2019-04-01 19:00:21.381. Total writes so far 140000
[INFO ] 2019-04-01 19:00:25.702 [pool-2-thread-2] c.m.t.t.m.s.CassandraPersistenceManager - Cassandra Write Batch commit started at: 2019-04-01 19:00:25.702 and finished at: 2019-04-01 19:00:25.702. Total writes so far 145000
[INFO ] 2019-04-01 19:00:30.099 [pool-2-thread-2] c.m.t.t.m.s.CassandraPersistenceManager - Cassandra Write Batch commit started at: 2019-04-01 19:00:30.098 and finished at: 2019-04-01 19:00:30.098. Total writes so far 150000
[INFO ] 2019-04-01 19:00:34.516 [pool-2-thread-2] c.m.t.t.m.s.CassandraPersistenceManager - Cassandra Write Batch commit started at: 2019-04-01 19:00:34.516 and finished at: 2019-04-01 19:00:34.516. Total writes so far 155000
[INFO ] 2019-04-01 19:00:39.065 [pool-2-thread-2] c.m.t.t.m.s.CassandraPersistenceManager - Cassandra Write Batch commit started at: 2019-04-01 19:00:39.065 and finished at: 2019-04-01 19:00:39.065. Total writes so far 160000
[INFO ] 2019-04-01 19:00:43.510 [pool-2-thread-2] c.m.t.t.m.s.CassandraPersistenceManager - Cassandra Write Batch commit started at: 2019-04-01 19:00:43.51 and finished at: 2019-04-01 19:00:43.51. Total writes so far 165000
[INFO ] 2019-04-01 19:00:48.162 [pool-2-thread-2] c.m.t.t.m.s.CassandraPersistenceManager - Cassandra Write Batch commit started at: 2019-04-01 19:00:48.162 and finished at: 2019-04-01 19:00:48.162. Total writes so far 170000
[INFO ] 2019-04-01 19:00:52.686 [pool-2-thread-2] c.m.t.t.m.s.CassandraPersistenceManager - Cassandra Write Batch commit started at: 2019-04-01 19:00:52.686 and finished at: 2019-04-01 19:00:52.686. Total writes so far 175000
[INFO ] 2019-04-01 19:00:59.547 [pool-2-thread-2] c.m.t.t.m.s.CassandraPersistenceManager - Cassandra Write Batch commit started at: 2019-04-01 19:00:59.546 and finished at: 2019-04-01 19:00:59.547. Total writes so far 180000
[INFO ] 2019-04-01 19:01:04.368 [pool-2-thread-2] c.m.t.t.m.s.CassandraPersistenceManager - Cassandra Write Batch commit started at: 2019-04-01 19:01:04.368 and finished at: 2019-04-01 19:01:04.368. Total writes so far 185000
[INFO ] 2019-04-01 19:01:09.093 [pool-2-thread-2] c.m.t.t.m.s.CassandraPersistenceManager - Cassandra Write Batch commit started at: 2019-04-01 19:01:09.093 and finished at: 2019-04-01 19:01:09.093. Total writes so far 190000
[INFO ] 2019-04-01 19:01:13.832 [pool-2-thread-2] c.m.t.t.m.s.CassandraPersistenceManager - Cassandra Write Batch commit started at: 2019-04-01 19:01:13.831 and finished at: 2019-04-01 19:01:13.831. Total writes so far 195000
[INFO ] 2019-04-01 19:01:22.035 [pool-2-thread-2] c.m.t.t.m.s.CassandraPersistenceManager - Cassandra Write Batch commit started at: 2019-04-01 19:01:22.035 and finished at: 2019-04-01 19:01:22.035. Total writes so far 200000
[INFO ] 2019-04-01 19:01:26.945 [pool-2-thread-2] c.m.t.t.m.s.CassandraPersistenceManager - Cassandra Write Batch commit started at: 2019-04-01 19:01:26.945 and finished at: 2019-04-01 19:01:26.945. Total writes so far 205000
[INFO ] 2019-04-01 19:01:32.132 [pool-2-thread-2] c.m.t.t.m.s.CassandraPersistenceManager - Cassandra Write Batch commit started at: 2019-04-01 19:01:32.132 and finished at: 2019-04-01 19:01:32.132. Total writes so far 210000
[INFO ] 2019-04-01 19:01:34.202 [pool-2-thread-2] c.m.t.t.m.s.CassandraPersistenceManager - Last written Cassandra token is w32wz5q5mygu44ya59w5rrfk : Wed Jan 23 03:56:07 IST 2019
[INFO ] 2019-04-01 19:01:34.202 [pool-2-thread-2] c.m.t.t.m.s.job.BatchWriteJob - Finished write thread now
[ERROR] 2019-04-01 19:02:34.205 [pool-2-thread-1] c.m.t.t.m.s.job.MigrationJob - Could not terminate running Migration job cleanly, shutting down forcefully
[INFO ] 2019-04-01 19:02:34.208 [pool-2-thread-1] c.m.t.t.m.s.job.MigrationJob - Migration job completed, shutting down now
[INFO ] 2019-04-01 19:02:34.208 [main] c.m.t.t.m.service.Migrator - Migration program exiting with status code 0
[INFO ] 2019-04-01 19:02:34.209 [Thread-2] c.m.t.t.m.s.FileJournalService - Journal service got last written record updated: Wed Jan 23 03:56:07 IST 2019
[INFO ] 2019-04-01 19:02:34.210 [Thread-2] c.m.t.t.m.s.FileJournalService - File journal path is /opt/mashery/migration/data/MigrationJournal.txt
[INFO ] 2019-04-01 19:02:34.210 [Thread-2] c.m.t.t.m.s.FileJournalService - File journal is not existing, creating it /opt/mashery/migration/data/MigrationJournal.txt
[INFO ] 2019-04-01 19:02:34.214 [Thread-2] c.m.t.t.m.s.FileJournalService - Writing file journal entry for Migrate-Token
root@d2d3421x1332:bin$