Page History

...

• Will there be a feature to turn off the historical row creation for Metadb (as you could for LDP)?
  
  It's doable, no feature request in github now. There might be an easy way to do it.

have a clear idea of what we need to do, then get in touch with Nassib

Anonymization

• Given increasing concerns about data privacy, is data anonymization on the metadb software development roadmap? If so, when do you expect it to be released?

It would 

• Is there a plan to meet GDPR requirements with Metadb for German institutions, if so, when?

• There is some loan anonymization functionality in FOLIO settings and there are plans to do the same for Requests
  • Is metadb anonymization being developed with that interdependence in mind, or is it going to be an independent feature?
• Will there be a feature to make patron identifiers (e.g., user_id, requester_id) NULL in historical rows?
• If we need to recover from our data anonymization processes, what will be required and what might we lose?

User Permissions

• Support for more granular permissions has been added in Metadb 1.4: see https://gist.github.com/nassibnassar/9e8e63383db1dac0634b0cf371254dec
• Are there ways that this new functionality can support our goal of removing access to patron identifiers in our Metadb reporting database? 
• Does this mean we can use postgres permissions, groups, etc.?
• Would it be possible to develop user permission functionality for Metadb that corresponds to user permissions in FOLIO?

Table Creation

• When we look at the sets of 3 tables in each application area (e.g., loan__ , loan__t , and loan__t__), we get different totals when we count the id rows. Why would this be? Is this happening because just first level of json fields are extracted into the __t__ table?
• Is there a relationship among tables during the ETL process? Specifically, does it work like this, or another way?
  • main table (loan__) the json source
  • current and historical table (loan__t__) is created from the main table  
  • current table (loan__t) is made from the current and historical table (loan__t__)?

Other

...

• No timeline, but developing in the near future

see data privacy documentation on LDP, you can see how anonymization works, it allows you to specify columns and tables that should be anonymized; data are never written to the database

Anonymization in Metadb will probably be similar, but not absolutely sure

1 - LDP + keeping an eye on new columns so specification as to what should be anonymized continues to be 

2 - You allow data to come in, but you remove a person's specific data when they request it

(3) (Tom)Following anonymization processes in FOLIO

Anonymization should be done as part of the stream processing, so it never gets to the Metadb database

Anonymization will probably be similar to LDP anonymization

If FOLIO anonymizes a field, it will be anonymized 

If we can make sure that FOLIO never writes the data to its database, it will not come over to Metadb

• There is some loan anonymization functionality in FOLIO settings and there are plans to do the same for Requests
  • Is metadb anonymization being developed with that interdependence in mind, or is it going to be an independent feature?

Not sure because it captures historical data; probably independent because metadb used beyond FOLIO

• Will there be a feature to make patron identifiers (e.g., user_id, requester_id) NULL in historical rows?
• If we need to recover from our data anonymization processes, what will be required and what might we lose?

User Permissions

• Support for more granular permissions has been added in Metadb 1.4: see https://gist.github.com/nassibnassar/9e8e63383db1dac0634b0cf371254dec
• Are there ways that this new functionality can support our goal of removing access to patron identifiers in our Metadb reporting database? 
• Does this mean we can use postgres permissions, groups, etc.?
• Would it be possible to develop user permission functionality for Metadb that corresponds to user permissions in FOLIO?

If there is a concern that just the data sitting there is a problem, support 

What do you need from us for a feature to turn off history

-we need to remove the data from the database, so historical rows keep us from keeping our promise with patrons

-is turning history off going to meet Cornell's needs?

-turning off history is not something that could be done right now (Nassib); he is not sure how involved it would be

Table Creation

• When we look at the sets of 3 tables in each application area (e.g., loan__ , loan__t , and loan__t__), we get different totals when we count the id rows. Why would this be? Is this happening because just first level of json fields are extracted into the __t__ table?
• Is there a relationship among tables during the ETL process? Specifically, does it work like this, or another way?
  • main table (loan__) the json source
  • current and historical table (loan__t__) is created from the main table  
  • current table (loan__t) is made from the current and historical table (loan__t__)?

Other

• What is in the zzz_ tables in Metadb?  e.g., zzz___loan__t___
• Should we anonymize the zzz_tables, and if so, how would we do that?
• Do you have any additional advice for us as we work on data anonymization?
• Are there any questions we missed that are important to ask?

Risks

1st level risk - you may render the historical data unusable for queries you want to do, make it difficult to join

2nd level risk - could interfere with Metadb's ability to update the data

can send nassib queries to review each time

stay away from

in the tables, __id is the primary key; underlying data from FOLIO has its own primary key

primary key is a constraint, key cannot be null, must be unique; unique values are keys; primary key is a choice of one of those keys

candidate key, primary key - primary key is concretely a constraint; never nullify the primary keys

candidate key is the folio key, ID, this column should not be changed because metadb is using what folio thinks is the primary key to version the records

the database will not let you nullify a primary key, but the candidate key can be nullified (and should not be)

candidate keys, id keys are used for versioning

any column that might be a key in FOLIO should not be nullified

Pitfalls with NULLS

-in db theory, NULL = NULL is false , so you have to be careful if you do an equijoin on 2 columns being equal, cartisian product filtered by a condition, can get unexpected results if we join on NULL

...

Metadb Links

Metadb Project: https://github.com/metadb-project/metadb

...