isatools_edit_microservices.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ISA API example using tab/json conversion microservices"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook aims to show how to use the ISA API for loading, editing, and writing ISA tabs, using the cloud infrastructure set up using the PhenoMeNal [workflow-demo tutorial](https://github.com/phnmnl/workflow-demo). The aim of this is to: \n",
    "\n",
    "1. Convert an ISA tab into a ISA JSON document;\n",
    "2. Edit values in the JSON objects;\n",
    "3. Convert the edited ISA JSON document back into an ISA tab.\n",
    "\n",
    "This gives us a basic framework for editing ISA tab files programmatically."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The code snippets in this notebook use Python to import the isatab2json and the json2isatab converters. Each converter is used to load from tab into some JSON objects, and them dump back to tab the modified JSON objects of ISA content. This example makes use of running microservices calls for the conversions inside a MANTL orchestrated cloud cluster.\n",
    "\n",
    "The ISA API is documented in this [page](http://www.github.com/ISA-tools/isa-api/wiki)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prerequisites\n",
    "\n",
    "There are some prerequisites to fulfill in order to successfully run the examples in this page."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Please run the following snippet and insert your MANTL control node URL."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "control.djcomlab.phenomenal.cloud\n"
     ]
    }
   ],
   "source": [
    "control = input()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Please run the following code snippet and insert your MANTL admin password."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import getpass\n",
    "password = getpass.getpass()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the following code snippets to get some sample input data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "N.B. You can ignore any warning about unverified HTTP requests. Response code 204 means that the REST call succeed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import urllib.request\n",
    "urllib.request.urlretrieve(\n",
    "    \"https://raw.githubusercontent.com/ISA-tools/isa-api/master/tests/data/BII-S-3/i_gilbert.txt\", \n",
    "    \"i_investigation.txt\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "urllib.request.urlretrieve(\n",
    "    \"https://raw.githubusercontent.com/ISA-tools/isa-api/master/tests/data/BII-S-3/s_BII-S-3.txt\", \n",
    "    \"s_BII-S-3.txt\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "urllib.request.urlretrieve(\n",
    "    \"https://raw.githubusercontent.com/ISA-tools/isa-api/master/tests/data/BII-S-3/a_gilbert-assay-Gx.txt\", \n",
    "    \"a_gilbert-assay-Gx.txt\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "urllib.request.urlretrieve(\n",
    "    \"https://raw.githubusercontent.com/ISA-tools/isa-api/master/tests/data/BII-S-3/a_gilbert-assay-Tx.txt\", \n",
    "    \"a_gilbert-assay-Tx.txt\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 1: isatab2json - Converting ISA tab to ISA JSON\n",
    "\n",
    "ISA tab files are a collection of comma separated variable files, typically loaded and edited in ISAcreator. \n",
    "\n",
    "In this step we convert an existing ISA tab to ISA JSON. This will allow us to more easily treat ISA content as objects. Instead of using the ISA API locally, we utlise a MANTL cluster to run the conversion as a docker microservice. This is to emulate how we might deploy such microservices within a cloud infrastructure."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import requests\n",
    "from requests.packages.urllib3.exceptions import InsecureRequestWarning\n",
    "requests.packages.urllib3.disable_warnings(InsecureRequestWarning) # suppress warnings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import json\n",
    "import time\n",
    "url = \"https://admin:\"+password+\"@\"+control+\"/chronos/scheduler/iso8601\"\n",
    "job_json = \"\"\"\n",
    "{ \n",
    "    \"cpus\": \"0.25\",\n",
    "    \"mem\": \"128\",  \n",
    "    \"epsilon\" : \"PT10M\",  \n",
    "    \"name\" : \"isatab2json\",\n",
    "    \"container\": {\n",
    "        \"type\": \"DOCKER\",\n",
    "        \"image\": \"djcomlab/isatab2json\",\n",
    "        \"volumes\": [{\n",
    "            \"hostPath\": \"/mnt/container-volumes/jupyter\",\n",
    "            \"containerPath\": \"/data\",\n",
    "            \"mode\": \"RW\"\n",
    "         }]\n",
    "    },\n",
    "    \"command\" : \"python run_tab2json.py /data/ /data/\",\n",
    "    \"owner\" : \"david.johnson@oerc.ox.ac.uk\"\n",
    "}\n",
    "\"\"\"\n",
    "response = requests.post(url, headers={'content-type': 'application/json'}, data=job_json, verify=False)\n",
    "print(\"HTTP response code: \" + str(response.status_code))\n",
    "if response.status_code == 204:\n",
    "    print(\"Waiting for ISA tab to JSON conversion to complete...\")\n",
    "    for x in range(0, 30): \n",
    "        if os.path.exists('BII-S-3.json'): break\n",
    "        time.sleep(1)  # Wait until the JSON is written by the microservice job\n",
    "    print(\"Conversion finished.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2: Edit ISA JSON\n",
    "\n",
    "In this step we edit up the JSON. We can modify existing existing values. The example below adds a new publication to the investigation object, which corresponds to the investigation file in an ISA tab. We load the ISA JSON output from the microservice in the previous step and append some JSON to the list of publications."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "isa_json = json.load(open('BII-S-3.json'))\n",
    "isa_json['publications'].append(\n",
    "    {\n",
    "        \"authorList\": \"Gilbert JA, Field D, Huang Y, Edwards R, Li W, Gilna P, Joint I.\",\n",
    "        \"doi\": \"10.1371/journal.pone.0003042\",\n",
    "        \"pubMedID\": \"18725995\",\n",
    "        \"status\": {\n",
    "            \"annotationValue\": \"indexed in PubMed\",\n",
    "            \"termAccession\": \"\",\n",
    "            \"termSource\": \"\"\n",
    "        },\n",
    "        \"title\": \"Detection of large numbers of novel sequences in the metatranscriptomes of complex marine microbial communities.\"\n",
    "    }\n",
    ")\n",
    "json.dump(open('BII-S-3.json', 'w'))  # BUG HERE: Chronos seems to write back file as root, causes problems\n",
    "print(json.dumps(isa_json['publications']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3: json2isatab - Convert the edited JSON back to ISA tab\n",
    "\n",
    "Once finished with editing the ISA content by directly modifying the JSON we can now write back to ISA tab format, effectively having loaded, edited, and now saving ISA tab files. Again, we utilise our cloud infrastructure to run a microservice to convert back to ISA tab."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "url = \"https://admin:\"+password+\"@\"+control+\"/chronos/scheduler/iso8601\"\n",
    "if not os.path.exists('out/'):\n",
    "    os.mkdir('out/')\n",
    "json = \"\"\"\n",
    "{ \n",
    "    \"cpus\": \"0.25\",\n",
    "    \"mem\": \"128\",  \n",
    "    \"epsilon\" : \"PT10M\",  \n",
    "    \"name\" : \"json2isatab\",\n",
    "    \"container\": {\n",
    "        \"type\": \"DOCKER\",\n",
    "        \"image\": \"djcomlab/json2isatab\",\n",
    "        \"volumes\": [{\n",
    "            \"hostPath\": \"/mnt/container-volumes/jupyter\",\n",
    "            \"containerPath\": \"/data\",\n",
    "            \"mode\": \"RW\"\n",
    "         }]\n",
    "    },\n",
    "    \"command\" : \"python run_json2tab.py /data/BII-S-3.json /data/out/\",\n",
    "    \"owner\" : \"david.johnson@oerc.ox.ac.uk\"\n",
    "}\n",
    "\"\"\"\n",
    "response = requests.post(url, headers={'content-type': 'application/json'}, data=json, verify=False)\n",
    "print(\"HTTP response code: \" + str(response.status_code))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If all went as planned, you should have written out some ISA tab files that correspond to the edit you made to the ISA JSON in Step 2."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.4.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}