*** This version of Confluence is for testing only and contains a copy of content from June 29th 2026. No changes will be preserved. ***
Basics
The Metadb Test instance is currently set up to synchronize data with the FOLIO Test instance. The goal is to optimize, document, and understand this process to prepare to synchronize data between Metadb Test and FOLIO Production.
Debezium Connector, Synch, and Resynch
Our
Sharon Markus (Beltaine) 10:47 AM
@John Malconian - Thank you so much for your help with Debezium. @Carole Godfrey from our hosting team is preparing for synchronizing our Metadb Test instance with our FOLIO Production instance. The Debezium connector reads the FOLIO database and populates Kafka with topic messages, which takes about 31 hours for Cornell's data, so that is okay. The part that is really slow is where Metadb is consuming the Kafka topics and populating the instance with either new data or changes to data. The most recent synchronization test took 8 days. We are trying to figure out why this is happening so we can get the sync down to a more reasonable timeframe, like 2 days. If you might be able to provide some insights and general guidance, that would be great. As I understand it, our Our hosting team is following the metadb documentation closely and running Metadb version 1.2 on an instance that meets the specs per the metadb documentation. Getting the synching and resynching down to shorter timeframes is our top priority right now. @Carole Godfrey and @nassib may have more comments/details here.
John Malconian 11:18 AMGiven from Index Data indicates that given that it takes 31 hours for Debezium to complete the initial snapshot, 6-8 days to complete the Metadb portion of the snapshot is about normal based on my experience as well depending on the data set. I think Nassib is better equipped to comment on Metadb performance specifically, but, generally speaking, it’s going to take longer for Metadb to consume the snapshot than Debezium. Metadb is doing data transformation and database operations that just take longer than reading a logical replication slot and pushing messages into Kafka. From an Ops perspective, ensure that the Metadb database is adequately sized to handle the processing (CPU) and that it is not hindered by I/O limitations. If using RDS for the Metadb database, provisioned IOPS or gp3 storage is best (I prefer gp3).Given that it takes 31 hours for Debezium to complete the initial snapshot, 6-8 days to complete the Metadb portion of the snapshot is about normal based on my experience as well depending on the data set. I think his experience. This also depends on the data set. He says Nassib is better equipped to comment on Metadb performance specifically, but, generally speaking, it’s going to take longer for Metadb to consume the snapshot than Debezium. Metadb is doing data transformation and database operations that just take longer than reading a logical replication slot and pushing messages into Kafka. From an Ops perspective, ensure that the Metadb database is adequately sized to handle the processing (CPU) and that it is not hindered by I/O limitations. If using RDS for the Metadb database, provisioned IOPS or gp3 storage is best (I prefer John prefers gp3).
1
3 replies
Carole Godfrey 4 days ago
Ok, thank you -- we Carole Godfrey from the hosting team says they have monitored cpu usage (it did not go above ~26% during the initial sync process) and we are using gp3 storage. One thing that we they are trying to test is the resync processMetaDB the resync process. The MetaDB database has been populated from the initial sync and now we they need to adjust and get it in sync with an updated RDS instance (in this case - FOLIO has been upgrade from Orchid --> Poppy )In your experience, should . Carole wonders if John thinks a resync should take the same or or more/less time to complete as the initial sync?Thanks!
John Malconian 4 days ago
It’s . John says It’s going to take about the same time whether it’s an initial sync or re-sync. It’s essentially the same process either way.
11
Carole Godfrey 4 days ago
Ok thank youWe Carole says they did get 1 resync running and trying to understand if it completed successfully -- but wasnt sure how I could to confirm its successWe success. We had gotten messages when running an endsync
I continued on from this step - but wasnt sure on queries or ? to confirm all is in syncTable sizes did not match up (when compared to FOLIO) -- but filtering __current = true
...
The unconfirmed records will be marked as deleted.
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Are there any checks that can be done in MetaDB database to confirm that a resync was successful?https://metadb-project.slack.com/archives/C03ETMG3TGX/p1708107064841009?thread_ts=1708100301.261019&cid=C03ETMG3TGXFrom previous thread - we had a warning during the endsync - is it ok to run endsync with this type of warning?
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! 34% of current records have not been confirmed by the new snapshot. The unconfirmed records will be marked as deleted. !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
(From Nassib) This warning is not always a problem. It means that a lot of the current data in Metadb have not (so far) been confirmed by the new snapshot.This could happen if many records were deleted in the source (FOLIO) during the time when the databases were not synchronized.But typically it indicates that a lot of snapshot data have not yet been read from Kafka. If this is the reason for the warning, then running endsync would remove those records in Metadb until they are eventually streamed. So the records would temporarily be unavailable in the current tables in Metadb.
Ok thanks @nassibIn the case of a resync -- should counts from these 2 queries alignIn MetaDB:SELECT count(*) FROM folio_inventory.instance__ where __current = true;In FOLIOSELECT count(*) FROM tenant_mod_inventory_storage.instance;Also with each resync - is it expected that the count of records in Metadb tables will increase by the number of records in the corresponding FOLIO table?
(1) Yes.
(2) The counts should be the same when the tables are synchronized.
Ok -- thanksAfter the initial sync process - observed these 2 queries aligned
(1)SELECT count(*) FROM folio_inventory.instance__
(2)SELECT count(*) FROM tenant_mod_inventory_storage.instanceAfter a resync - observed results from (1) was ~2X the size of (2) and I needed to filter by _current to get counts that more alignedSafe to assume the filtering by _current is needed?
Yes, non-current records are not retained by FOLIO.
OK, thanks -- understood
In other words, FOLIO discards the non-current records but Metadb retains them. So we can only compare counts of the current records.(edited)