docs/graphs.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SB93Ge748VQs"
      },
      "source": [
        "##### Copyright 2019 The TensorFlow Authors."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "cellView": "form",
        "id": "0sK8X2O9bTlz"
      },
      "outputs": [],
      "source": [
        "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HEYuO5NFwDK9"
      },
      "source": [
        "# Examining the TensorFlow Graph\n",
        "\n",
        "<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://www.tensorflow.org/tensorboard/graphs\"><img src=\"https://www.tensorflow.org/images/tf_logo_32px.png\" />View on TensorFlow.org</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/tensorboard/blob/master/docs/graphs.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://github.com/tensorflow/tensorboard/blob/master/docs/graphs.ipynb\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" />View source on GitHub</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a href=\"https://storage.googleapis.com/tensorflow_docs/tensorboard/docs/graphs.ipynb\"><img src=\"https://www.tensorflow.org/images/download_logo_32px.png\" />Download notebook</a>\n",
        "  </td>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "56V5oun18ZdZ"
      },
      "source": [
        "## Overview\n",
        "\n",
        "TensorBoard’s **Graphs dashboard** is a powerful tool for examining your TensorFlow model. You can quickly view a conceptual graph of your model’s structure and ensure it matches your intended design. You can also view a op-level graph to understand how TensorFlow understands your program. Examining the op-level graph can give you insight as to how to change your model. For example, you can redesign your model if training is progressing slower than expected."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TOSJ-4nteBYG"
      },
      "source": [
        "This tutorial presents a quick overview of how to generate graph diagnostic data and visualize it in TensorBoard’s Graphs dashboard. You’ll define and train a simple Keras Sequential model for the Fashion-MNIST dataset and learn how to log and examine your model graphs. You will also use a tracing API to generate graph data for functions created using the new `tf.function` annotation."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zNI1-dflrAo0"
      },
      "source": [
        "## Setup"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "6B95Hb6YVgPZ"
      },
      "outputs": [],
      "source": [
        "# Load the TensorBoard notebook extension.\n",
        "%load_ext tensorboard"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "id": "_wqSAZExy6xV"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "TensorFlow version:  2.2.0\n"
          ]
        }
      ],
      "source": [
        "from datetime import datetime\n",
        "from packaging import version\n",
        "\n",
        "import tensorflow as tf\n",
        "from tensorflow import keras\n",
        "\n",
        "print(\"TensorFlow version: \", tf.__version__)\n",
        "assert version.parse(tf.__version__).release[0] >= 2, \\\n",
        "    \"This notebook requires TensorFlow 2.0 or above.\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "id": "qRZlYIEcJ56Z"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "'2.2.1'"
            ]
          },
          "execution_count": 4,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "import tensorboard\n",
        "tensorboard.__version__"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "id": "Ao7fJW1Pyiza"
      },
      "outputs": [],
      "source": [
        "# Clear any logs from previous runs\n",
        "!rm -rf ./logs/ "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e25E37vd1xEW"
      },
      "source": [
        "## Define a Keras model\n",
        "\n",
        "In this example, the classifier is a simple four-layer Sequential model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "id": "skqORzvE3Egy"
      },
      "outputs": [],
      "source": [
        "# Define the model.\n",
        "model = keras.models.Sequential([\n",
        "    keras.layers.Flatten(input_shape=(28, 28)),\n",
        "    keras.layers.Dense(32, activation='relu'),\n",
        "    keras.layers.Dropout(0.2),\n",
        "    keras.layers.Dense(10, activation='softmax')\n",
        "])\n",
        "\n",
        "model.compile(\n",
        "    optimizer='adam',\n",
        "    loss='sparse_categorical_crossentropy',\n",
        "    metrics=['accuracy'])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qbjuoz9E3VC_"
      },
      "source": [
        "Download and prepare the training data."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "id": "6TDmc41z3g38"
      },
      "outputs": [],
      "source": [
        "(train_images, train_labels), _ = keras.datasets.fashion_mnist.load_data()\n",
        "train_images = train_images / 255.0"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8DV0xibO3bRC"
      },
      "source": [
        "## Train the model and log data\n",
        "\n",
        "Before training, define the [Keras TensorBoard callback](https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/TensorBoard), specifying the log directory. By passing this callback to Model.fit(), you ensure that graph data is logged for visualization in TensorBoard."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "id": "TU_L_u9SqQdH"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Epoch 1/5\n",
            "938/938 [==============================] - 2s 2ms/step - loss: 0.6955 - accuracy: 0.7618\n",
            "Epoch 2/5\n",
            "938/938 [==============================] - 2s 2ms/step - loss: 0.4877 - accuracy: 0.8296\n",
            "Epoch 3/5\n",
            "938/938 [==============================] - 2s 2ms/step - loss: 0.4458 - accuracy: 0.8414\n",
            "Epoch 4/5\n",
            "938/938 [==============================] - 2s 2ms/step - loss: 0.4246 - accuracy: 0.8476\n",
            "Epoch 5/5\n",
            "938/938 [==============================] - 2s 2ms/step - loss: 0.4117 - accuracy: 0.8508\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "<tensorflow.python.keras.callbacks.History at 0x7f656ecc3fd0>"
            ]
          },
          "execution_count": 8,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# Define the Keras TensorBoard callback.\n",
        "logdir=\"logs/fit/\" + datetime.now().strftime(\"%Y%m%d-%H%M%S\")\n",
        "tensorboard_callback = keras.callbacks.TensorBoard(log_dir=logdir)\n",
        "\n",
        "# Train the model.\n",
        "model.fit(\n",
        "    train_images,\n",
        "    train_labels, \n",
        "    batch_size=64,\n",
        "    epochs=5, \n",
        "    callbacks=[tensorboard_callback])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IRX5OIsi4TTV"
      },
      "source": [
        "## Op-level graph\n",
        "\n",
        "Start TensorBoard and wait a few seconds for the UI to load. Select the Graphs dashboard by tapping “Graphs” at the top. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "PFgFjlPEqXb9"
      },
      "outputs": [],
      "source": [
        "%tensorboard --logdir logs"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EGlOqRp54ufD"
      },
      "source": [
        "By default, TensorBoard displays the **op-level graph**. (On the left, you can see the “Default” tag selected.)  Note that the graph is inverted; data flows from bottom to top, so it’s upside down compared to the code. However, you can see that the graph closely matches the Keras model definition, with extra edges to other computation nodes.\n",
        "\n",
        "Graphs are often very large, so you can manipulate the graph visualization:\n",
        "\n",
        "* Scroll to **zoom** in and out\n",
        "* Drag to **pan**\n",
        "* Double clicking toggles **node expansion** (a node can be a container for other nodes)\n",
        "\n",
        "You can also see metadata by clicking on a node. This allows you to see inputs, outputs, shapes and other details.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F-2yw5qd7OpK"
      },
      "source": [
        "<!-- <img class=\"tfo-display-only-on-site\" src=\"https://github.com/tensorflow/tensorboard/blob/master/docs/images/graphs_computation.png?raw=1\"/> -->"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jDRynpVw53SJ"
      },
      "source": [
        "<!-- <img class=\"tfo-display-only-on-site\" src=\"https://github.com/tensorflow/tensorboard/blob/master/docs/images/graphs_computation_detail.png?raw=1\"/> -->"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Oj9FSPdz6SO2"
      },
      "source": [
        "## Conceptual graph\n",
        "\n",
        "In addition to the execution graph, TensorBoard also displays a **conceptual graph**. This is a view of just the Keras model. This may be useful if you’re reusing a saved model and you want to examine or validate its structure.\n",
        "\n",
        "To see the conceptual graph, select the “keras” tag. For this example, you’ll see a collapsed **Sequential** node. Double-click the node to see the model’s structure:"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Qw9rbEcE6eZB"
      },
      "source": [
        "<!-- <img class=\"tfo-display-only-on-site\" src=\"https://github.com/tensorflow/tensorboard/blob/master/docs/images/graphs_tag_selection.png?raw=1\"/> --> <br/>\n",
        "<!-- <img class=\"tfo-display-only-on-site\" src=\"https://github.com/tensorflow/tensorboard/blob/master/docs/images/graphs_conceptual.png?raw=1\"/> -->"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zVuaKBifu-qF"
      },
      "source": [
        "## Graphs of tf.functions\n",
        "\n",
        "The examples so far have described graphs of Keras models, where the graphs have been created by defining Keras layers and calling Model.fit().\n",
        "\n",
        "You may encounter a situation where you need to use the `tf.function` annotation to [\"autograph\"](https://www.tensorflow.org/guide/function), i.e., transform, a Python computation function into a high-performance TensorFlow graph. For these situations, you use **TensorFlow Summary Trace API** to log autographed functions for visualization in TensorBoard."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JIuhJnQ8w-dT"
      },
      "source": [
        "To use the Summary Trace API:\n",
        "\n",
        "*   Define and annotate a function with `tf.function`\n",
        "*   Use `tf.summary.trace_on()` immediately before your function call site.\n",
        "*    Add profile information (memory, CPU time) to graph by passing `profiler=True`\n",
        "*   With a Summary file writer, call `tf.summary.trace_export()` to save the log data\n",
        "\n",
        "You can then use TensorBoard to see how your function behaves.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "id": "woI67Stgv_uY"
      },
      "outputs": [],
      "source": [
        "# The function to be traced.\n",
        "@tf.function\n",
        "def my_func(x, y):\n",
        "  # A simple hand-rolled layer.\n",
        "  return tf.nn.relu(tf.matmul(x, y))\n",
        "\n",
        "# Set up logging.\n",
        "stamp = datetime.now().strftime(\"%Y%m%d-%H%M%S\")\n",
        "logdir = 'logs/func/%s' % stamp\n",
        "writer = tf.summary.create_file_writer(logdir)\n",
        "\n",
        "# Sample data for your function.\n",
        "x = tf.random.uniform((3, 3))\n",
        "y = tf.random.uniform((3, 3))\n",
        "\n",
        "# Bracket the function call with\n",
        "# tf.summary.trace_on() and tf.summary.trace_export().\n",
        "tf.summary.trace_on(graph=True, profiler=True)\n",
        "# Call only one tf.function when tracing.\n",
        "z = my_func(x, y)\n",
        "with writer.as_default():\n",
        "  tf.summary.trace_export(\n",
        "      name=\"my_func_trace\",\n",
        "      step=0,\n",
        "      profiler_outdir=logdir)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "zCArnWzP0VuZ"
      },
      "outputs": [],
      "source": [
        "%tensorboard --logdir logs/func"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WDl1PBFQ64xi"
      },
      "source": [
        "<!-- <img class=\"tfo-display-only-on-site\" src=\"https://github.com/tensorflow/tensorboard/blob/master/docs/images/graphs_autograph.png?raw=1\"/> -->"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1pLRaf3q6Nku"
      },
      "source": [
        "You can now see the structure of your function as understood by TensorBoard. Click on the \"Profile\" radiobutton to see CPU and memory statistics."
      ]
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [
        "SB93Ge748VQs"
      ],
      "name": "graphs.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}