Simplified_CC12M_Local.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Simplified CC12M Local.ipynb",
      "provenance": [],
      "private_outputs": true,
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/MSFTserver/AI-Colab-Notebooks-Local/blob/main/Simplified_CC12M_Local.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#**The MOST Powerful Art Generator you need to learn RIGHT NOW!!!**\n",
        "\n",
        "**Edited by [HostsServer](https://github.com/MSFTserver) for local runtime use!**"
      ],
      "metadata": {
        "id": "sThFl4fJtEQJ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#**1)** Download everything required for Art Generator ❗❗❗"
      ],
      "metadata": {
        "id": "U2wrXkaJg-r1"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# @title Licensed under the MIT License\n",
        "\n",
        "# Copyright (c) 2022 Katherine Crowson\n",
        "\n",
        "# Permission is hereby granted, free of charge, to any person obtaining a copy\n",
        "# of this software and associated documentation files (the \"Software\"), to deal\n",
        "# in the Software without restriction, including without limitation the rights\n",
        "# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n",
        "# copies of the Software, and to permit persons to whom the Software is\n",
        "# furnished to do so, subject to the following conditions:\n",
        "\n",
        "# The above copyright notice and this permission notice shall be included in\n",
        "# all copies or substantial portions of the Software.\n",
        "\n",
        "# THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n",
        "# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n",
        "# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\n",
        "# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n",
        "# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\n",
        "# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN\n",
        "# THE SOFTWARE."
      ],
      "metadata": {
        "cellView": "form",
        "id": "eb9FKu1rtH6R"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "xa4Er6nMgI1b"
      },
      "outputs": [],
      "source": [
        "# Check the GPU\n",
        "\n",
        "!mkdir steps\n",
        "!nvidia-smi"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Install dependencies (no need to rerun this section if you restart the notebook runtime)"
      ],
      "metadata": {
        "id": "0XknUUiEsPCL"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Install dependencies\n",
        "\n",
        "!pip install ftfy regex requests tqdm IProgress\n",
        "!git clone --recursive https://github.com/crowsonkb/v-diffusion-pytorch"
      ],
      "metadata": {
        "id": "xQVAY5-4g8ta"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#@markdown # Download the diffusion model\n",
        "#@markdown download this model and put it inside the folder `v-diffusion-pytorch/checkpoints` wich is created after running this cell\n",
        "\n",
        "#@markdown https://v-diffusion.s3.us-west-2.amazonaws.com/cc12m_1_cfg.pth\n",
        "# SHA-256: 4fc95ee1b3205a3f7422a07746383776e1dbc367eaf06a5b658ad351e77b7bda\n",
        "import os\n",
        "!mkdir v-diffusion-pytorch\\checkpoints\n",
        "os.startfile(f\"{os.getcwd()}/v-diffusion-pytorch/checkpoints\")"
      ],
      "metadata": {
        "id": "9410kZ2aipVg"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Import modules and load models"
      ],
      "metadata": {
        "id": "tbjtOS68sazS"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Imports\n",
        "\n",
        "import gc\n",
        "import math\n",
        "import sys\n",
        "\n",
        "from IPython import display\n",
        "import torch\n",
        "from torchvision import utils as tv_utils\n",
        "from torchvision.transforms import functional as TF\n",
        "from tqdm.notebook import trange, tqdm\n",
        "\n",
        "sys.path.append('v-diffusion-pytorch')\n",
        "\n",
        "from CLIP import clip\n",
        "from diffusion import get_model, sampling, utils"
      ],
      "metadata": {
        "id": "f_6OhF85i5vp"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Load the models\n",
        "\n",
        "model = get_model('cc12m_1_cfg')()\n",
        "_, side_y, side_x = model.shape\n",
        "model.load_state_dict(torch.load('v-diffusion-pytorch/checkpoints/cc12m_1_cfg.pth', map_location='cpu'))\n",
        "model = model.half().cuda().eval().requires_grad_(False)\n",
        "clip_model = clip.load(model.clip_model, jit=False, device='cpu')[0]"
      ],
      "metadata": {
        "id": "v7Zl0mg1jrBP"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# side_y = 512\n",
        "# side_x = 512"
      ],
      "metadata": {
        "id": "k0fSKOWaZLkz"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# **2)** Parameters for The Art Generator 🤖\n"
      ],
      "metadata": {
        "id": "HH60ZBz4hAyq"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "***-Only these heights and widths work: 4,8,16,32,64,128,256,512***\n",
        "\n",
        "-Change the seed if you want different results!"
      ],
      "metadata": {
        "id": "7rv8Axl4A-Yz"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#@title Settings\n",
        "\n",
        "prompt = 'San Francisco city skyline with trippy fog'  #@param {type:\"string\"}\n",
        "\n",
        "height =  256#@param {type:\"integer\"}\n",
        "width =  256#@param {type:\"integer\"}\n",
        "side_x = width\n",
        "side_y = height\n",
        "#@markdown Specify the number of diffusion timesteps (default is 500, can lower for faster but lower quality sampling).\n",
        "steps =   1000#@param {type:\"integer\"}\n",
        "\n",
        "\n",
        "#@markdown Sample this many images.\n",
        "n_images =   4#@param {type:\"integer\"}\n",
        "\n",
        "weight = 5 \n",
        "\n",
        "#@markdown Set to 0 for deterministic (DDIM) sampling, 1 (the default) for stochastic (DDPM) sampling, and in between to interpolate between the two. 0 is preferred for low numbers of timesteps.\n",
        "eta =   0#@param {type:\"number\"}\n",
        "\n",
        "#@markdown The random seed. Change this to sample different images.\n",
        "seed =   -1#@param {type:\"integer\"}\n",
        "\n",
        "#@markdown Display progress every this many timesteps.\n",
        "display_every = 25  #@param {type:\"integer\"}\n",
        "\n",
        "save_progress_video = True #@param {type:\"boolean\"}"
      ],
      "metadata": {
        "id": "4PSRj7REh0ZS",
        "cellView": "form"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# **3)** Generate Art 😎\n"
      ],
      "metadata": {
        "id": "fR420Ql1hC93"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#@title Run the Generator\n",
        "from PIL import ImageFile, Image\n",
        "import numpy as np\n",
        "import os\n",
        "target_embed = clip_model.encode_text(clip.tokenize(prompt)).float().cuda()\n",
        "from IPython.display import clear_output\n",
        "\n",
        "def cfg_model_fn(x, t):\n",
        "    \"\"\"The CFG wrapper function.\"\"\"\n",
        "    n = x.shape[0]\n",
        "    x_in = x.repeat([2, 1, 1, 1])\n",
        "    t_in = t.repeat([2])\n",
        "    clip_embed_repeat = target_embed.repeat([n, 1])\n",
        "    clip_embed_in = torch.cat([torch.zeros_like(clip_embed_repeat), clip_embed_repeat])\n",
        "    v_uncond, v_cond = model(x_in, t_in, clip_embed_in).chunk(2, dim=0)\n",
        "    v = v_uncond + (v_cond - v_uncond) * weight\n",
        "    return v\n",
        "save_name = 0.00000000\n",
        "\n",
        "def display_callback(info):\n",
        "    global save_name\n",
        "    save_name += 0.00000001\n",
        "    nrow = math.ceil(info['pred'].shape[0]**0.5)\n",
        "    grid = tv_utils.make_grid(info['pred'], nrow, padding=0)\n",
        "    utils.to_pil_image(grid).save(f\"./steps/%.8f.png\" % save_name)\n",
        "    \n",
        "    if info['i'] % display_every == 0:\n",
        "        nrow = math.ceil(info['pred'].shape[0]**0.5)\n",
        "        grid = tv_utils.make_grid(info['pred'], nrow, padding=0)\n",
        "        tqdm.write(f'Step {info[\"i\"]} of {steps}:')\n",
        "        display.display(utils.to_pil_image(grid))\n",
        "        # utils.to_pil_image(grid).save(\"./steps/Test.png\")\n",
        "        \n",
        "        tqdm.write(f'')\n",
        "\n",
        "\n",
        "def run():\n",
        "    print(\"Prompt is: \" + prompt)\n",
        "    print(\"Seed is: \" + str(seed))\n",
        "    gc.collect()\n",
        "    torch.cuda.empty_cache()\n",
        "    torch.manual_seed(seed)\n",
        "    x = torch.randn([n_images, 3, side_y, side_x], device='cuda')\n",
        "    t = torch.linspace(1, 0, steps + 1, device='cuda')[:-1]\n",
        "    step_list = utils.get_spliced_ddpm_cosine_schedule(t)\n",
        "    outs = sampling.sample(cfg_model_fn, x, step_list, eta, {}, callback=display_callback)\n",
        "    tqdm.write('Done!')\n",
        "    for i, out in enumerate(outs):\n",
        "        filename = f'out_{i}.png'\n",
        "        utils.to_pil_image(out).save(filename)\n",
        "        display.display(display.Image(filename))\n",
        "\n",
        "\n",
        "run()\n",
        "\n",
        "if save_progress_video:\n",
        "\n",
        "    init_frame = 1 #Este es el frame donde el vídeo empezará\n",
        "    last_frame = steps #Puedes cambiar i a el número del último frame que quieres generar. It will raise an error if that number of frames does not exist.\n",
        "\n",
        "    min_fps = 10\n",
        "    max_fps = 30\n",
        "\n",
        "    total_frames = last_frame-init_frame\n",
        "\n",
        "    length = 15 #Tiempo deseado del vídeo en segundos\n",
        "\n",
        "    # import required module\n",
        "    import os\n",
        "    # assign directory\n",
        "    directory = 'steps'\n",
        "    \n",
        "    files = []\n",
        "\n",
        "    # iterate over files in\n",
        "    # that directory\n",
        "    for filename in os.listdir(directory):\n",
        "        f = os.path.join(directory, filename)\n",
        "        # checking if it is a file\n",
        "        files.append(f)\n",
        "\n",
        "\n",
        "    files.sort()\n",
        "    # print(files)\n",
        "\n",
        "\n",
        "    frames = []\n",
        "    save_name2 = 0.00000000\n",
        "    tqdm.write('Generating video...')\n",
        "    for i in range(init_frame,last_frame): #\n",
        "        # save_name2 += 0.00000001\n",
        "        # filename = f\"steps/{save_name2}.png\"\n",
        "        # print(files[i])\n",
        "        frames.append(Image.open(files[i]))\n",
        "\n",
        "    #fps = last_frame/10\n",
        "    fps = np.clip(total_frames/length,min_fps,max_fps)\n",
        "\n",
        "    from subprocess import Popen, PIPE\n",
        "    p = Popen(['ffmpeg', '-y', '-f', 'image2pipe', '-vcodec', 'png', '-r', str(fps), '-i', '-', '-vcodec', 'libx264', '-r', str(fps), '-pix_fmt', 'yuv420p', '-crf', '17', '-preset', 'veryslow', 'video.mp4'], stdin=PIPE)\n",
        "    for im in tqdm(frames):\n",
        "        im.save(p.stdin, 'PNG')\n",
        "    p.stdin.close()\n",
        "\n",
        "    p.wait()\n",
        "    print(\"Video finished Rendering\")"
      ],
      "metadata": {
        "cellView": "form",
        "id": "I8xFXMdah4V_",
        "collapsed": true
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#@markdown Show Generated Video in Colab\n",
        "from base64 import b64encode\n",
        "mp4 = open('video.mp4','rb').read()\n",
        "data_url = \"data:video/mp4;base64,\" + b64encode(mp4).decode()\n",
        "display.HTML(\"\"\"\n",
        "<video width=400 controls>\n",
        "      <source src=\"%s\" type=\"video/mp4\">\n",
        "</video>\n",
        "\"\"\" % data_url)"
      ],
      "metadata": {
        "id": "7VE7oiMvh7fi",
        "cellView": "form"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "#🌟🌟🌟 **Beautiful Prompts you should try!** 🌟🌟🌟"
      ],
      "metadata": {
        "id": "wBAFNR5mhLIv"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "More prompts added everyday at: https://www.reddit.com/r/TheGenerat0rs/"
      ],
      "metadata": {
        "id": "eumpa3UWfFVA"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "    The Tower Of Illusions in the clouds by Thomas Kinkade\n",
        "    The Tower of Illusions in the clouds by Thomas Kinkade by Lisa Frank\n",
        "    The Tower of Illusions in the clouds by Thomas Kinkade WPAP\n",
        "    Starry Night in the style of Lisa Frank\n",
        "    a beautiful epic fantasy of a 🤖\n",
        "    a futuristic boat\n",
        "    fantasy world in unreal engine\n",
        "    retrowave castle city by Tyler Edlin in the style of John Constable\n",
        "    a new algorithm\n",
        "    lighthouse on the edge of tomorrow, surreal, digital art\n",
        "    futuristic Eiffel Tower, dslr\n",
        "    concept lanscape environment fantasy\n",
        "    voxel minecraft house\n",
        "    Architectural Visualizations in unreal engine\n",
        "    Architectural Concepts spaceship artstation hd\n",
        "    ahri fan art artstation hd\n",
        "    bmw automotive digital 3D in unreal engine 4K \n",
        "    concept art fan art artstation hd\n",
        "    the infinity realm concept art digital 2d illustration\n",
        "    hyperrealistic psychedelic ophanim"
      ],
      "metadata": {
        "id": "P6G9SC6nhO-A"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "-\n",
        "#**Want to Super-Enhance the Generated Image with another advanced A.I.? Enhances a 500x500 image to 1760x1760 with no quality loss!!!**: \n",
        "\n",
        "-\n",
        "\n",
        "https://colab.research.google.com/drive/1O8kr-iyRll5eQsaOH-vQIc3ImEKWsvF-?usp=sharing"
      ],
      "metadata": {
        "id": "_-7S6YSphV7o"
      }
    }
  ]
}