Smarter Data Sync #78

New issue

Closed

opened 2022-03-22 22:26:37 +00:00 by jamie · 3 comments

jamie commented

2022-03-22 22:26:37 +00:00

(Migrated from git.hazaar.io)

Had an idea to make data sync a bit smarter and more useful when the database is not quite up to spec. Before sync'ing a table we can do the following:

Make sure the table exists and if not, skip the current sync stage.
Get a list of columns for each table and only sync data for columns that exist.

These two updates should all but stop any sync errors when the data sync files are ahead, behind, or just out of sync with the actual schema that is loaded into the database. Any tables or columns that have data but don't exist, will simply be filtered out.

Also, not 100% sure how this will work, but we may be able to check if there are NOT NULL columns and if they don't have data to sync we could "make something up", like NA for VARCHAR/TEXT, 0 for INTEGER or FALSE for BOOLEAN types. Or at a minimum we could display a data sync error that is more clear.

Had an idea to make data sync a bit smarter and more useful when the database is not quite up to spec. Before sync'ing a table we can do the following: * Make sure the table exists and if not, skip the current sync stage. * Get a list of columns for each table and only sync data for columns that exist. These two updates should all but stop any sync errors when the data sync files are ahead, behind, or just out of sync with the actual schema that is loaded into the database. Any tables or columns that have data but don't exist, will simply be filtered out. Also, not 100% sure how this will work, but we may be able to check if there are NOT NULL columns and if they don't have data to sync we could "make something up", like _NA_ for `VARCHAR/TEXT`, _0_ for `INTEGER` or _FALSE_ for `BOOLEAN` types. Or at a minimum we could display a data sync error that is more clear.

jamie commented

2022-03-22 22:26:37 +00:00

(Migrated from git.hazaar.io)

assigned to @jamie