Smarter Data Sync
Had an idea to make data sync a bit smarter and more useful when the database is not quite up to spec. Before sync'ing a table we can do the following:
- Make sure the table exists and if not, skip the current sync stage.
- Get a list of columns for each table and only sync data for columns that exist.
These two updates should all but stop any sync errors when the data sync files are ahead, behind, or just out of sync with the actual schema that is loaded into the database. Any tables or columns that have data but don't exist, will simply be filtered out.
Also, not 100% sure how this will work, but we may be able to check if there are NOT NULL columns and if they don't have data to sync we could "make something up", like NA for VARCHAR/TEXT
, 0 for INTEGER
or FALSE for BOOLEAN
types. Or at a minimum we could display a data sync error that is more clear.