*** This version of Confluence is for testing only and contains a copy of content from June 29th 2026. No changes will be preserved. ***
-
Created by
Sharon Helen Markus, last updated on Feb 23, 2024
4 minute read
Basics
The Metadb Test instance is currently set up to synchronize data with the FOLIO Test instance. The goal is to optimize, document, and understand this process to prepare to synchronize data between Metadb Test and FOLIO Production.
Debezium Connector, Synch, and Resynch
Our hosting team is preparing for synchronizing our Metadb Test instance with our FOLIO Production instance. The Debezium connector reads the FOLIO database and populates Kafka with topic messages, which takes about 31 hours for Cornell's data, so that is okay. The part that is really slow is where Metadb is consuming the Kafka topics and populating the instance with either new data or changes to data. The most recent synchronization test took 8 days. We are trying to figure out why this is happening so we can get the sync down to a more reasonable timeframe, like 2 days. Our hosting team is running Metadb version 1.2 on an instance that meets the specs per the metadb documentation. Getting the synching and resynching down to shorter timeframes is our top priority right now.
John Malconian from Index Data indicates that given that it takes 31 hours for Debezium to complete the initial snapshot, 6-8 days to complete the Metadb portion of the snapshot is about normal based on his experience. This also depends on the data set. He says Nassib is better equipped to comment on Metadb performance specifically, but, generally speaking, it’s going to take longer for Metadb to consume the snapshot than Debezium. Metadb is doing data transformation and database operations that just take longer than reading a logical replication slot and pushing messages into Kafka. From an Ops perspective, ensure that the Metadb database is adequately sized to handle the processing (CPU) and that it is not hindered by I/O limitations. If using RDS for the Metadb database, provisioned IOPS or gp3 storage is best (John prefers gp3).
Carole Godfrey from the hosting team says they have monitored cpu usage (it did not go above ~26% during the initial sync process) and we are using gp3 storage. One thing that they are trying to test is the resync process. The MetaDB database has been populated from the initial sync and now they need to adjust and get it in sync with an updated RDS instance (in this case - FOLIO has been upgrade from Orchid --> Poppy ). Carole wonders if John thinks a resync should take the same or more/less time to complete as the initial sync. John says It’s going to take about the same time whether it’s an initial sync or re-sync. It’s essentially the same process either way.
Carole says they did get 1 resync running and trying to understand if it completed successfully -- but wasnt sure how to confirm its success. We had gotten messages when running an endsync
I continued on from this step - but wasnt sure on queries or ? to confirm all is in syncTable sizes did not match up (when compared to FOLIO) -- but filtering __current = true
SELECT count(*) FROM folio_inventory.instance__ where __current = true;
SELECT count(*) FROM folio_inventory.instance__t__ where __current = true;
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
34% of current records have not been confirmed by the new snapshot.
The unconfirmed records will be marked as deleted.
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Are there any checks that can be done in MetaDB database to confirm that a resync was successful?https://metadb-project.slack.com/archives/C03ETMG3TGX/p1708107064841009?thread_ts=1708100301.261019&cid=C03ETMG3TGXFrom previous thread - we had a warning during the endsync - is it ok to run endsync with this type of warning?
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! 34% of current records have not been confirmed by the new snapshot. The unconfirmed records will be marked as deleted. !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
(From Nassib) This warning is not always a problem. It means that a lot of the current data in Metadb have not (so far) been confirmed by the new snapshot.This could happen if many records were deleted in the source (FOLIO) during the time when the databases were not synchronized.But typically it indicates that a lot of snapshot data have not yet been read from Kafka. If this is the reason for the warning, then running endsync would remove those records in Metadb until they are eventually streamed. So the records would temporarily be unavailable in the current tables in Metadb.
Ok thanks @nassibIn the case of a resync -- should counts from these 2 queries alignIn MetaDB:SELECT count(*) FROM folio_inventory.instance__ where __current = true;In FOLIOSELECT count(*) FROM tenant_mod_inventory_storage.instance;Also with each resync - is it expected that the count of records in Metadb tables will increase by the number of records in the corresponding FOLIO table?
(1) Yes.
(2) The counts should be the same when the tables are synchronized.
Ok -- thanksAfter the initial sync process - observed these 2 queries aligned
(1)SELECT count(*) FROM folio_inventory.instance__
(2)SELECT count(*) FROM tenant_mod_inventory_storage.instanceAfter a resync - observed results from (1) was ~2X the size of (2) and I needed to filter by _current to get counts that more alignedSafe to assume the filtering by _current is needed?
Yes, non-current records are not retained by FOLIO.
OK, thanks -- understood
In other words, FOLIO discards the non-current records but Metadb retains them. So we can only compare counts of the current records.