*** This version of Confluence is for testing only and contains a copy of content from June 29th 2026. No changes will be preserved. ***
...
Metadb is a new streaming data integration platform that will eventually replace the Library Data Platform (LDP 1.x) software used by FOLIO project participants for reporting on FOLIO data. Metadb is still under development. Some institutions are running early versions of Metadb for testing purposes. The Cornell Library is currently running LDP 1.x on its production reporting database to provide users with access to FOLIO data for reporting.
Data updates
The LDP software provides reporting users with a copy of yesterday's data from the FOLIO application server. Metadb will provide continuously streaming data updates from the FOLIO application server. This will provide reporting users with access to FOLIO data in near real time instead of yesterday's data. While LDP 1.x's architecture is limited to running on a single node, Metadb's architecture is horizontal, allowing for linear scaling of data updates. This allows updates to run in parallel within a single Metadb cluster.
...
- historical data - history tables are created as data are updated so the LDP or Metadb database retains data that was updated in the past, which can be used for trend reporting
- transformed data - FOLIO stores data in 3 different formats (JSON, ERM, MARC JSON mapped), all of which get transformed into relational data tables as part of the LDP/Metadb ETL (extract, transform, load) process, making it possible make joins and queries across those heterogeneous native data sources
- derived tables - the FOLIO reporting community writes derived table queries that transform the denormalized FOLIO data into joined tables with data arrays broken into columns to both improve query performance and simplify query creation
- external data sets - institutions can import their own data into the LDP or Metadb database, which Cornell has done in bringing select data tables from its legacy Voyager ILS into its LDP and Metadb reporting databasedatabases
- user-defined data sets - data analysts can use LDP and Metadb to create their own data tables to be used in queries, which the Cornell Library reporting team has done for queries in areas with large data sets (e.g., annual statistics)
...
Building report queries using the LDP software requires extracting some of the data from data arrays. By design, Metadb tables have data arrays broken out into columns, making it easier for query developers to use.
Special Features
Both LDP 1.x and Metadb include special features of interest to library reporting users: ldpmarc marct and the LDP Reporting app.
- Having access to MARC data is essential for Cornell Library reporting users. The
...
- marct tool currently converts SRS/MARC records from JSON to a tabular format for LDP
...
- and
...
- Metadb.
- The
...
- Reporting app provides a FOLIO interface for interacting with an LDP or Metadb database. The Query builder in the
...
- Reporting app allows you to build a simple query without using SQL with the option to save it for later use
...
Borrow Direct move from Relais to Reshare
...
- .
(Image provided by Nassib Nassar, Index Data)
Resources
- (draft) Metadb documentation is available here: https://d1f3dtrg62pavmetadb.cloudfront.netdev/doc/
- Metadb-based query development for FOLIO: https://github.com/folio-org/folio-analytics/tree/main/sql_metadb
- https://github.com/metadb-project/metadb
- Wolfcon 2022 LDP Presentation - (skip ahead to the 4-minute mark in the recording)
- Metadb software development Roadmap and Issues: https://github.com/metadb-project/metadb/issues
- LDP to Metadb table equivalents - https://metadb.dev/doc/#_table_names