FHIR BULK DATA API

Extending FHIR to Population Level Datasets

Dan Gottlieb, Central Square Solutions, LLC (@GotDan)

Josh Mandel, Microsoft (@JoshCMandel)

Revised September 2021

 slides: https://bit.ly/fhir-bulk-api

Agenda
• Why a FHIR Bulk Data API?
• Technical Architecture & what’s new in 2.0
• Open Source Tools
• Adoption!
• Next steps and how to get involved

2
 FHIR APIs

FHIR REST API FHIR Bulk Data API

Patient Panel Population

3
Use Cases
• Internal clinical data warehouse for study cohort identification
• Claims in EHR to provide comprehensive view
• Machine learning startup obtaining training data from cloud EHR
• Integration population health system with EHR system
• Transferring records from one EHR to another
• Payer database to assess care quality
• Reportable disease submission or other registry

4
Let’s enhance to support population level data
access
• FHIR Resources as a standard data model to simplify data parsing and
mapping
• FHIR Operation API to initiate the data extracts
• SMART Backend Services Authorization as security model

5
Focused Scope + Complementary Technologies
• Legal framework for sharing data between partners needs to be set
up out-of-band (BAAs, SLAs, DUAs)
• Real-time data - data loaded through bulk APIs can be supplemented
with real time FHIR REST API calls or subscriptions
• Patient matching - it’s possible to include identifiers like subscriber
number in Bulk Export FHIR resources
• Data transformation - can serve as a foundation for data pipeline

6
 Technical Architecture
Bulk Data Access Implementation Guide

7
Bulk Data Access IG Versions

STU1 (v1) STU2 (v2)

Initial version Incorporates experience from early

implementations
Published August 2019 Published November 2022

8
Kick-off Request
Bulk Data Kick-off Request Bulk Data
Client Server
(destination) (source)

9
Kick-off Request
• Asynchronous requests with status polling (HTTP GET only in v1, GET or POST in v2) Updated
Prefer: respond-async in v2

• FHIR Operation for all data on all patients (all data in the patient “compartment”)
[FHIR Server Base]/Patient/$export

• FHIR Operation for all data on a group of patients (eg. research cohort, plan members)
[FHIR Server Base]/Group/[group id]/$export

• FHIR Operation for all data on the server

[FHIR Server Base]/$export

10
Kick-off Operation Parameters
_outputFormat The format for the generated bulk data files
Currently, only ndjson is supported

_since Filter results by FHIR resource modified date Updated

in v2
FHIR instant timestamp
(required for servers to support)
_type Filter results by comma delimited list of FHIR resource types
(optional for servers to support)
_typeFilter Filter using FHIR REST queries
(optional and experimental)

11
Kick-off Operation Parameters New in
v2

_elements FHIR resource elements to return

e.g., Patient.id, Patient.identifier
(optional and experimental)
Patient FHIR Patient References to limit data returned
(optional, not valid for GET requests or system level requests)

includeAssociatedData Metadata resources to include with response

e.g., LatestProvenanceResources or
RelevantProvenanceResources
(optional and experimental)

12
Kick-off Response
Bulk Data Kickoff Request Bulk Data
Client Server
(destination) (source)
Content Location

13
Kick-off Response Header
Status: 202 Accepted
Content-Location: [URL for status or deleting request]

14
Status Request #1
Bulk Data Kickoff Request Bulk Data
Client Server
(destination) (source)
Content Location

GET Content Location

15
Status Response #1
Bulk Data Kickoff Request Bulk Data
Client Server
(destination) (source)
Content Location

GET Content Location

File Generation Status

16
In-Progress Status Response Header
Status: 202 Accepted
X-Progress: “50% complete”
Retry-After: 120

17
Status Request #2
Bulk Data Kickoff Request Bulk Data
Client Server
(destination) (source)
Content Location

GET Content Location

File Generation Status

GET Content Location

18
Status Response #2
Bulk Data Kickoff Request Bulk Data
Client Server
(destination) (source)
Content Location

GET Content Location

File Generation Status

GET Content Location

JSON manifest

19
Status Complete Response Body
1 "transactionTime" : "2020-07-13T13:28:17.239Z",
2 "request" : "https://example.com/Patient/$export?_type=Patient,Observation",
3 "requiresAccessToken" : true,
4 "output" : [{
"type" : "Patient",
"url" : "https://example.com/files/patient_file_1.ndjson"
},{
"type" : "Patient",
"url" : "https://example.com/files/patient_file_2.ndjson"
},{
"type" : "Observation",
"url" : "https://example.com/filesw/observation_file_1.ndjson"
}],
5 "deleted" : [{
New in "type" : "Bundle",
v2
"url" : "https://example.com/output/del_file_1.ndjson"
}],
6
"error" : [{
Updated "type" : "OperationOutcome",
in v2
"url" : "https://example.com/files/error_file_1.ndjson"
}]
20
File Request
Bulk Data Kickoff Request Bulk Data
Client Server
(destination) (source)
Content Location

GET Content Location

File Generation Status

GET Content Location

JSON manifest

GET File (eg. 0001.Observation.ndjson )

21
File Response
Bulk Data Kickoff Request Bulk Data
Client Server
(destination) (source)
Content Location

GET Content Location

File Generation Status

GET Content Location

JSON manifest

GET File (eg. 0001.Observation.ndjson )

FHIR Resources File

22
FHIR Resources
Data models representing discrete clinical and administrative units (patient, practitioner,
allergy, medication order, etc.)

• Currently around 100 have been defined

• Can reference other resources by their URL
• Don’t include the kitchen sink, but support extensions
“We only include data elements if we are confident that most normal implementations
using that resource will make use of the element”
– Grahame Grieve (FHIR Product Director)
• MU3 Common Clinical Dataset (and soon USCDI) defines subset

23
NDJSON

,
,

24
SMART Backend Services Authorization
• Out-of-band app registration (can use Dynamic Client Registration or
portal)
• Apps can register public key (JWKS format) or URL for public key
• Token requests signed with private key
• System level scope (parallels SMART “user” and “patient” scopes)
system/[resourceType].read

• Short-lived access tokens

25
Registration Flow (once)

Backend Service Configure Public Key and other OAuth settings Bulk
Admin Data
Server
OAuth Client Id

26
Authorization Flow (min. once per request)

Backend Service Configure Public Key and other OAuth settings Bulk
Admin Data
Server
OAuth Client Id

Bulk Data Client

Signed Token Request

Short Lived Access Token

27
Tutorial (Python): EHR Export to SQL Exploration
https://colab.research.google.com/drive/1HhEEB3MJ8LbMP2ta946s8OARPc5RflHu

28
Enhancements
in v2

29
Historical Group Data

• Server side with revised guidance on the “_since” parameter

“In the case of a Group level export, servers MAY return additional resources modified Updated
prior to the supplied time if the resources belong to the patient compartment of a patient in v2
added to the Group after the supplied time (this behavior should be clearly documented
by the server).”

• Client side with the “patients” and “_elements” parameters

• Make a request to get just the ids of patients in the group with “_elements” New in
• Use the “patients” parameter to get data for patients not previously retrieved v2
• Use the “patients” paramter and the “_since” parameter to get new data for remaining
patients

30
Attachments New in
v2

Resource elements of type Attachment must contain:

data element with a Base64 encoded version of the file

OR

url element with an absolute URL for the file accessible using the same
authentication as the ndjson files

31
Open Source Tools

32
SMART Reference Server Implementation
https://bulk-data.smarthealthit.org

33
SMART Sample Client
https://github.com/smart-on-fhir/sample-apps-stu3/tree/master/fhir-downloader

34
ONC Inferno Testing Tool
https://inferno.healthit.gov/community

35
FHIR Data Census Tool
https://github.com/sync-for-science/data-census

36
Adoption

37
Growing number of implementations!
Open Source FHIR Servers EHRs
• Microsoft • Epic
• HAPI • Cerner (prototype)
• IBM • T-System (prototype)

Commercial FHIR Servers Payor Data Servers

• Azure API for FHIR • CMS ACO Beneficiary Claims Data (pilot)
• CareEvolution • CMS Data at the Point of Care (pilot)
• Firely Server (SQL endpoint) • CMS Claims to Part D Sponsors (pilot)
• Google Healthcare API (in preview)

38
US Regulatory Requirements

170.215 [EHR Certification] Application Programming Interface

Standards.
The Secretary adopts the following application programming interface
(API) standards and associated implementation specifications […] FHIR
Bulk Data Access (Flat FHIR) (v1.0.0: STU 1), including mandatory
support for the “group-export” “OperationDefinition”

Implementation Date: 12/31/2022

39
Next Steps

40
Get Involved!
• Use the APIs in real-world use cases and collect ideas for v3
• Open source modules (eg. de-identification, filtering, NLP)
• Define Bulk Import Operation (updates coming soon!)
Early draft proposals at https://github.com/smart-on-fhir/bulk-import/blob/master/import-pnp.md

• Standardize analytic approaches (identify cohorts, quality measures)

41
 slides: https://bit.ly/fhir-bulk-api

Resources
• Bulk Data Implementation Guide
STU2 (v2): https://hl7.org/fhir/uv/bulkdata/
STU1 (v1): http://hl7.org/fhir/uv/bulkdata/

• SMART Server Reference Implementation

https://bulk-data.smarthealthit.org

• Bulk Data Discussion Group (Bulk Data Stream on FHIR Zulip Chat)
https://chat.fhir.org/#narrow/stream/bulk.20data

42