Extracting Fields from Metadb JSON arrays

The following jsonb extract expressions will be used to get values from the json data array of a table. 

• jsonb_extract_path   ---- this expression specifies the PATH necessary to get the last field entered in the path; the result is a json object and will display in double quotes
• jsonb_extract_path_text   ---- this expression does the same thing as jsonb_extract_path, but returns a TEXT object (displays without quotes)
• jsonb_array_elements   ---- this expression is used for extracting values within an array object
• the operator #>>. The format for using this operator is: table_name.jsonb #>> '{path_of_fields_in_the_hierarchy_separated_by_commas}' 
  • Example: audit_loan.jsonb #>> '{loan, metadata, updatedByUserId}'  -- this will get the value for "updatedByUserId" (the result is in Text format)

Extracting "First Level" Data Fields 

Example: In the data array of the folio_users.user table (screenshot below), the field 'active' (a boolean value showing patron status) is a "first level" hierarchical field. Other first-level fields include: id, barcode, metadata, personal, proxyFor, username, createdDate, departments, patronGroup, updatedDate and externalSystemId.

You will see that some of the first-level fields have subordinate data in second-levels fields ("metadata" and "personal" have subordinate fields). The following query will get the data out of first-level fields, when those fields don't have subordinate fields:

SELECT
     users.id AS user_id,
     jsonb_extract_path_text (users.jsonb, 'status') AS patron_status
FROM folio_users.users

The following expression will also get the value for "status":

SELECT
     users.id AS user_id,
     users.jsonb #>> '{status}" AS patron_status
FROM folio_users.users

Extracting "Second Level" Data Fields

In the example below, 'lastName' and 'firstName' are both "second-level" fields found under "personal." 

SELECT
     users.id AS user_id,
     jsonb_extract_path_text (users.jsonb, 'personal', 'lastName') AS user_last_name,
     jsonb_extract_path_text (users.jsonb, 'personal', 'firstName') AS user_first_name
FROM
     folio_users.users

The following expression will also get "lastName" and "firstName":

SELECT
     users.id AS user_id,
     users.jsonb #>> '{personal, lastName}' AS user_last_name
     users.jsonb #>> '{personal, firstName}' AS user_first_name
FROM
     folio_users.users

Extracting "Third level" Data Fields

Some tables, such as the folio_circulation.audit_loan table, have third-level data fields that are comprised of text objects (they can also have array objects, but that is covered in the next section – note: array objects can appear at any level within the json data). The json statement for extracting third-level text fields is exactly the same as for first- and second-level data fields, except there are three fields listed in the parentheses, instead of just one or two. 

In this example, "name" and "updatedByUserId" are third-level fields.

This is the code to extract them. Note that the fields listed within the parentheses must be in the same hierarchical order as they appear in the json data.

SELECT
     audit_loan.id AS audit_loan_id,
     jsonb_extract_path_text (audit_loan.jsonb, 'loan', 'status', 'name') AS loan_status_name,
     jsonb_extract_path_text (audit_loan.jsonb, 'loan', 'metadata', 'updatedByUserId') AS loan_updated_by_user_id
FROM
     folio_circulation.audit_loan

The following expression will also get the third-level values:

SELECT
     audit_loan.id AS audit_loan_id,
     audit_loan.jsonb #>> '{loan, status, name}' AS loan_status__name,
     audit_loan.jsonb #>> '{loan, metadata, updatedByUserId}' AS loan_updated_by_user_id
FROM
     folio_circulation.audit_loan

Extracting Arrays 

-- 1. Arrays embedded in text objects

In the previous examples, the elements being extracted were "text objects" within the table's JSON array, and were denoted by curly brackets { }. In this next example, we need to extract multiple values from an "array" object (denoted by square brackets [ ] ) that is embedded in a hierarchy of text elements. This calls for nested json extract statements, a structure very similar to what we used in Access with nested "GetField" and "GetXXXBLOB" functions.

When extracting arrays embedded in text objects, it's usually best to use a "cross join lateral" statement in the FROM stanza. This cross join creates a "json object" that can then be further extracted in the Select stanza. Warning: the records retrieved will only be those that have the array elements you're looking for; records without those elements will drop out.

It's also possible to NOT use a cross join at all, but to nest the json extract statements in one long expression in the Select stanza. However, you lose the ability to include the ordinality of the values extracted. See Summary at the end of this page ("Three ways to extract arrays").

• Nested extract statements are applied from the outside in. So the left-most json statement applies to the right-most element in the parentheses
• The entire nested extract statement (created by the cross join) is a json object
• That json object is then used in the select stanza to get the needed fields within it (by using another json extract statement on it)
  • Note that SQL performs the work in the FROM stanza before it performs the work in the SELECT stanza; that's how it knows about the json object you created through the cross join, and why you can use that object in the Select stanza

The example below uses the folio_users.custom_fields table, which has an embedded array called "values" containg three fields: "id","value" and "default".  The "Values"array is embedded within two preceding text objects:

• "selectField" (text object) – topmost level
• "options" (text object) – middle level
• "values" (array object) – bottom level. This array object contains the fields we need: "id" and "value"

Here is what the custom_fields jsonb data array looks like:

The code to do that is:

SELECT

   cft.id AS custom_fields_id,

   cft.ref_id,

   cft.name AS custom_fields_name,

   jsonb_extract_path_text (values.jsonb,'id') AS value_id,  ----- this extracts the "id" element from the values object that we created in the cross join below

   jsonb_extract_path_text (values.jsonb,'value') AS value_name  ----- this extracts the "value" element from the values object that we careated in the cross join below
          
           ------ NOTE: the two json extract statements above can be replaced by a shorthand formula, using the "#>>" operator: 
                      values.jsonb #>> '{id}'
                      values.jsonb #>> '{value}'

FROM folio_users.custom_fields AS cf

CROSS JOIN LATERAL jsonb_array_elements (jsonb_extract_path (cf.jsonb, 'selectField', 'options', 'values')) AS values (jsonb)
----- "jsonb_array_elements" works on the 'values' array object at the end of the parentheses; we use this extract expression because "values" is an array
----- "jsonb_extract_path" looks for the path of elements to follow in order to get to the values array – the elements listed must be in the same hierarchical order as they appear in the table
----- the entire cross join statement produces a jsonb object which I named "values" (note: what you name this object is totally arbitrary - you could call it "stuff" if you felt like it – "stuff (jsonb)")

LEFT JOIN folio_users.custom_fields__t AS cft  

ON jsonb_extract_path_text (cf.jsonb,'id')::UUID = cft.id

         ------ the custom_fields__t table has the custom fields id, the ref_id and the custom fields name (joining to this table is not required – I just wanted to include those fields in the results)

ORDER BY name, jsonb_extract_path_text (values.jsonb,'id'), jsonb_extract_path_text (values.jsonb,'value')

• • Note:
    • If you want to include "ordinality" as part of your results – this is the order in which the extracted values appear in the source record – you would insert "WITH ordinality" before "AS values (jsonb)":
             CROSS JOIN LATERAL jsonb_array_elements (jsonb_extract_path (cf.jsonb, 'selectField', 'options', 'values')) WITH ordinality AS values (jsonb)
    • Then you would add a line in the Select stanza to show the ordinality:  values.ordinality AS custom_fields_ordinality

RESULT:

Extracting Arrays 

-- 2. Array is at the top-most hierarchical level in a table

Sometimes you want to extract an array that is at the top-most hierarchical level of a table – that is, it is not embedded within any other object. An example of this is "departments" element in the folio_users.users table:

Because "departments" is an array, you still have to extract it with a cross join statement, as in the previous example. In the code below, I am using the shorthand method to extract the various elements:

SELECT

users.id AS user_id,

(users.jsonb #>> '{active}')::BOOLEAN AS patron_status,

CASE

WHEN (users.jsonb #>> '{active}')::BOOLEAN = true

THEN 'Active'

ELSE 'Expired'

END AS patron_status_name, -------- this Case When statement is just for displaying the boolean value as "Active" or "Expired"

users.jsonb #>>'{personal, lastName}' AS last_name, ------- this extrcts the last name from the "personal" object (second-level extraction)

users.jsonb #>>'{personal, firstName}' AS first_name, ------- this extracts the first name from the "personal' object (second-level extraction)

users.jsonb #>>'{username}' AS net_id, ------ this extracts "username" from the table (first-level extraction)

users.jsonb #>>'{barcode}' AS user_barcode, ------ this extracts "barcode" from the users table (first-level extraction)

depts.jsonb #>>'{}' AS dept_id, ----- this extracts the array values from the "depts" json object created through the cross join. Note that the curly brackets are empty; this is because there is no tag for the values, it's just the values themselves. If there were a tag for the values, the tag name would go in the curly brackets

ud.name AS department_name,

depts.ordinality AS dept_ordinality ------ this displays the ordinality of the departments (the sequence of occurrence in the record)

FROM folio_users.users

CROSS JOIN LATERAL jsonb_array_elements (jsonb_extract_path (users.jsonb,'departments'))

WITH ordinality AS depts (jsonb)

LEFT JOIN folio_users.departments__t AS ud -------- I am joining to the the departments__t table to get the department name

ON (depts.jsonb #>> '{}')::UUID = ud.id -------- note that I'm using the shortcut expression to join to the id in the departments__t table

RESULT:

3. Extracting Arrays - Summary: Three ways to extract arrays

This illustrates three methods for extracting an embedded array from the folio_users.custom_fields table

-- 1. No cross join -- just use triple-nested json extract statements in the Select clause

SELECT

cft.id AS custom_fields_id,

cft.ref_id,

cft.name AS custom_fields_name,

jsonb_extract_path_text (jsonb_array_elements (jsonb_extract_path (cf.jsonb, 'selectField', 'options', 'values')),'id') AS value_id,

jsonb_extract_path_text (jsonb_array_elements (jsonb_extract_path (cf.jsonb, 'selectField', 'options', 'values')),'value') AS value_name

FROM folio_users.custom_fields AS cf

LEFT JOIN folio_users.custom_fields__t AS cft

ON jsonb_extract_path_text (cf.jsonb,'id')::UUID= cft.id

ORDER BY name, value_id, value_name

;

-- 2. Use a cross join and put the result in a single-level json extract statement in the Select clause

SELECT

cft.id AS custom_fields_id,

cft.ref_id,

cft.name AS custom_fields_name,

jsonb_extract_path_text (values.jsonb,'id') AS value_id,

jsonb_extract_path_text (values.jsonb,'value') AS value_name

FROM folio_users.custom_fields AS cf

CROSS JOIN LATERAL

jsonb_array_elements (jsonb_extract_path (cf.jsonb, 'selectField', 'options', 'values'))

AS values (jsonb)

LEFT JOIN folio_users.custom_fields__t AS cft

ON jsonb_extract_path_text (cf.jsonb,'id') :: UUID= cft.id

ORDER BY name, jsonb_extract_path_text (values.jsonb,'id'), jsonb_extract_path_text (values.jsonb,'value')

;

-- 3. Use a cross join but use the shortcut method in the Select clause to get the values in the array

SELECT

cft.id AS custom_fields_id,

cft.ref_id,

cft.name AS custom_fields_name,

values.jsonb #>> '{id}' AS value_id,

values.jsonb #>> '{value}' AS value_name

FROM folio_users.custom_fields AS cf

CROSS JOIN LATERAL

jsonb_array_elements (jsonb_extract_path (cf.jsonb, 'selectField', 'options', 'values'))

AS values (jsonb)

LEFT JOIN folio_users.custom_fields__t AS cft

ON jsonb_extract_path_text (cf.jsonb,'id') :: UUID= cft.id

ORDER BY name, jsonb_extract_path_text (values.jsonb,'id'), jsonb_extract_path_text (values.jsonb,'value')

;