feat: cleanup

dev/inference/samples.csv

@@ -1,102 +0,0 @@
-Caption,Theme
-a cat seats on top of an alligator,Animals
-a dog eating worthlessness,Animals
-a dog playing with a ball,Animals
-a rat holding a red lightsable in a white background,Animals
-A unicorn is passing by a rainbow in a field of flowers,Animals
-an elephant made of carrots,Animals
-an elephant on a unicycle during a circus,Animals
-photography of a penguin watching television,Animals
-rat wearing a crown,Animals
-"a background consisting of colors blue, green, and red.",Art
-a colorful stairway to heaven,Art
-a graphite sketch of a gothic cathedral,Art
-a portrait of a nightmare creature watching at you,Art
-a white room full of a black substance,Art
-epic sword fight,Art
-"happy, happiness",Art
-painting of an oniric forest glade surrounded by tall trees,Art
-real painting of an alien from Monet,Art
-robots taking control over humans,Art
-"sad, sadness",Art
-still life in the style of Kandinsky,Art
-still life in the style of Picasso,Art
-the representation of infinity,Art
-a cute avocado armchair singing karaoke on stage in front of a crowd of strawberry shaped lamps,Avocado
-an armchair in the shape of an avocado,Avocado
-an avocado armchair,Avocado
-an avocado armchair flying into space,Avocado
-an illustration of an avocado in a christmas sweater staring at its reflection in a mirror,Avocado
-illustration of an avocado armchair,Avocado
-illustration of an avocado armchair getting married to a pineapple,Avocado
-logo of an avocado armchair,Avocado
-watercolor of the Eiffel tower on the moon,Avocado
-a cute pikachu teapot,Culture
-a picture of a castle from minecraft,Culture
-an illustration of pikachu seating on a bench,Culture
-mario eating an avocado while walking his baby koala,Culture
-star wars concept art,Culture
-a cartoon of a superhero bear,Illustrations
-an illustration of a cute skeleton wearing a blue hoodie,Illustrations
-Cartoon of a carrot with big eyes,Illustrations
-illustration of a baby shark swimming around corals,Illustrations
-logo of a robot wearing glasses and reading a book,Illustrations
-a beautiful sunset at a beach with a shell on the shore,Landscape
-a farmhouse surrounded by beautiful flowers,Landscape
-a photo of a fantasy version of New York City,Landscape
-a picture of fantasy kingdoms,Landscape
-a volcano erupting in the middle of New York city,Landscape
-aerial view of the beach at night,Landscape
-aerial view of the beach during daytime,Landscape
-big wave destroying a city,Landscape
-"London in a far future, futuristic London",Landscape
-sunset over green mountains,Landscape
-the last sunrise on earth,Landscape
-underwater cathedral,Landscape
-white snow covered mountain under blue sky during daytime,Landscape
-a bottle of coca-cola on a table,Objects
-a cactus lifitng weights,Objects
-a living room with two white armchairs and a painting of the collosseum. The painting is mounted above a modern fireplace.,Objects
-a long line of alternating green and red blocks,Objects
-a long line of green blocks on a beach at subset,Objects
-a long line of peaches on a beach at sunset,Objects
-a peanut,Objects
-a photo of a camera from the future,Objects
-a restaurant menu,Objects
-a skeleton with the shape of a spider,Objects
-"looking into the sky, 10 airplanes are seen overhead",Objects
-sheves filled with books and archemy potion bottles,Objects
-the communist statue of liberty,Objects
-this is a detailed high-resolution scan of a human brain,Objects
-a collection of glasses is sitting on a table,OpenAI
-a cross-section view of a walnut,OpenAI
-a painting of a capybara sitting on a mountain during fall in surrealist style,OpenAI
-a pentagonal green clock,OpenAI
-a photo of san francisco golden gate bridge,OpenAI
-a pixel art illustration of an eagle sitting in a field in the afternoon,OpenAI
-a professional high-quality emoji of a lovestruck cup of boba,OpenAI
-a small red block sitting on a large green block,OpenAI
-a storefront that has the word 'openai' written on it,OpenAI
-a tatoo of a black broccoli,OpenAI
-a variety of clocks is sitting on a table,OpenAI
-"an emoji of a baby fox wearing a blue hat, blue gloves, red shirt, and red pants",OpenAI
-"an emoji of a baby penguin wearing a blue hat, blue gloves, red shirt, and green pants",OpenAI
-an extreme close-up view of a capybara sitting in a field,OpenAI
-an illustration of a baby cucumber with a mustache playing chess,OpenAI
-an illustration of a baby daikon radish in a tutu walking a dog,OpenAI
-an illustration of a baby hedgehog in a cape staring at its reflection in a mirror,OpenAI
-an illustration of a baby panda with headphones holding an umbrella in the rain,OpenAI
-an illustration of an avocado in a beanie riding a motorcycle,OpenAI
-urinals are lined up in a jungle,OpenAI
-a human face,People
-"a person is holding a phone and a waterbottle, running a marathon.",People
-a photograph of Ellen G. White,People
-Mohammed Ali and Mike Tyson in a hypothetical match,People
-Pele and Maradona in a hypothetical match,People
-Young woman riding her bike through the forest,People
-a clown wearing a spacesuit floating in space,Space
-a photo of the French flag on the planet Saturn,Space
-a picture of the eiffel tower on the moon,Space
-illustration of an astronaut in a space suit playing guitar,Space
-the moon is a skull,Space
-view of mars from space,Space

dev/inference/samples.txt

@@ -0,0 +1,101 @@
+white snow covered mountain under blue sky during daytime
+aerial view of the beach at night
+aerial view of the beach during daytime
+a beautiful sunset at a beach with a shell on the shore
+a farmhouse surrounded by beautiful flowers
+a photo of a fantasy version of New York City
+a picture of fantasy kingdoms
+a volcano erupting in the middle of San Francisco
+big wave destroying a city
+Paris in a far future, futuristic Paris
+sunset over green mountains
+the last sunrise on earth
+underwater cathedral
+painting of an oniric forest glade surrounded by tall trees
+real painting of an alien from Monet
+a graphite sketch of a gothic cathedral
+still life in the style of Kandinsky
+still life in the style of Picasso
+a colorful stairway to heaven
+a background consisting of colors blue, green, and red
+the communist statue of liberty
+robots taking control over humans
+epic sword fight
+an avocado armchair
+an armchair in the shape of an avocado
+logo of an avocado armchair
+an avocado armchair flying into space
+a cute avocado armchair singing karaoke on stage in front of a crowd of strawberry shaped lamps
+an illustration of an avocado in a christmas sweater staring at its reflection in a mirror
+illustration of an avocado armchair
+illustration of an avocado armchair getting married to a pineapple
+Mohammed Ali and Mike Tyson in a hypothetical match
+Pele and Maradona in a hypothetical match
+view of mars from space
+illustration of an astronaut in a space suit playing guitar
+a clown wearing a spacesuit floating in space
+a picture of the eiffel tower on the moon
+watercolor of the Eiffel tower on the moon
+a photo of the French flag on the planet Saturn
+the moon is a skull
+a dog playing with a ball
+a cat sits on top of an alligator
+a rat holding a red lightsaber in a white background
+A unicorn is passing by a rainbow in a field of flowers
+a dog eating worthlessness
+an elephant made of carrots
+an elephant on a unicycle during a circus
+photography of a penguin watching television
+rat wearing a crown
+a portrait of a nightmare creature watching at you
+a white room full of a black substance
+happy, happiness
+sad, sadness
+the representation of infinity
+a cute pikachu teapot
+a picture of a castle from minecraft
+an illustration of pikachu sitting on a bench
+mario eating an avocado while walking his baby koala
+star wars concept art
+a cartoon of a superhero bear
+an illustration of a cute skeleton wearing a blue hoodie
+illustration of a baby shark swimming around corals
+Cartoon of a carrot with big eyes
+logo of a robot wearing glasses and reading a book
+a bottle of coca-cola on a table
+a cactus lifting weights
+a living room with two white armchairs and a painting of the collosseum. The painting is mounted above a modern fireplace.
+a long line of alternating green and red blocks
+a long line of green blocks on a beach at subset
+a long line of peaches on a beach at sunset
+a peanut
+a photo of a camera from the future
+a restaurant menu
+a skeleton with the shape of a spider
+looking into the sky, 10 airplanes are seen overhead
+shelves filled with books and alchemy potion bottles
+this is a detailed high-resolution scan of a human brain
+a collection of glasses is sitting on a table
+a cross-section view of a walnut
+a painting of a capybara sitting on a mountain during fall in surrealist style
+a pentagonal green clock
+a photo of san francisco golden gate bridge
+a pixel art illustration of an eagle sitting in a field in the afternoon
+a professional high-quality emoji of a lovestruck cup of boba
+a small red block sitting on a large green block
+a storefront that has the word 'openai' written on it
+a tatoo of a black broccoli
+a variety of clocks is sitting on a table
+an emoji of a baby fox wearing a blue hat, blue gloves, red shirt, and red pants
+an emoji of a baby penguin wearing a blue hat, blue gloves, red shirt, and green pants
+an extreme close-up view of a capybara sitting in a field
+an illustration of a baby cucumber with a mustache playing chess
+an illustration of a baby daikon radish in a tutu walking a dog
+an illustration of a baby hedgehog in a cape staring at its reflection in a mirror
+an illustration of a baby panda with headphones holding an umbrella in the rain
+an illustration of an avocado in a beanie riding a motorcycle
+urinals are lined up in a jungle
+a human face
+a person is holding a phone and a waterbottle, running a marathon
+a photograph of Ellen G. White
+Young woman riding her bike through the forest

dev/inference/wandb-backend.ipynb

@@ -7,7 +7,6 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "import csv\n",
     "import tempfile\n",
     "from functools import partial\n",
     "import random\n",
@@ -36,7 +35,8 @@
     "ENTITY, PROJECT = 'dalle-mini', 'dalle-mini'  # used only for training run\n",
     "VQGAN_REPO, VQGAN_COMMIT_ID = 'dalle-mini/vqgan_imagenet_f16_16384', None\n",
     "normalize_text = True\n",
-    "latest_only = False   # log only latest or all versions"
+    "latest_only = False   # log only latest or all versions\n",
+    "suffix = '_1'           # mainly for duplicate inference runs with a deleted version"
    ]
   },
   {
@@ -46,11 +46,12 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "run_ids = ['4oh3u7ca']\n",
+    "run_ids = ['3kaut6e8']\n",
     "ENTITY, PROJECT = 'wandb', 'hf-flax-dalle-mini'\n",
     "VQGAN_REPO, VQGAN_COMMIT_ID = 'dalle-mini/vqgan_imagenet_f16_16384', None\n",
     "normalize_text = False\n",
-    "latest_only = True   # log only latest or all versions"
+    "latest_only = True   # log only latest or all versions\n",
+    "suffix = '_2'           # mainly for duplicate inference runs with a deleted version"
    ]
   },
   {
@@ -78,8 +79,8 @@
    "outputs": [],
    "source": [
     "vqgan = VQModel.from_pretrained(VQGAN_REPO, revision=VQGAN_COMMIT_ID)\n",
-    "clip = FlaxCLIPModel.from_pretrained(\"openai/clip-vit-base-patch32\")\n",
-    "processor = CLIPProcessor.from_pretrained(\"openai/clip-vit-base-patch32\")\n",
+    "clip = FlaxCLIPModel.from_pretrained(\"openai/clip-vit-base-patch16\")\n",
+    "processor = CLIPProcessor.from_pretrained(\"openai/clip-vit-base-patch16\")\n",
     "clip_params = replicate(clip.params)\n",
     "vqgan_params = replicate(vqgan.params)"
    ]
@@ -108,13 +109,10 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "with open('samples.csv', newline='', encoding='utf8') as f:\n",
-    "    reader = csv.DictReader(f)\n",
-    "    samples = []\n",
-    "    for row in reader:\n",
-    "        samples.append(row)\n",
+    "with open('samples.txt', encoding='utf8') as f:\n",
+    "    samples = [l.strip() for l in f.readlines()]\n",
     "    # make list multiple of batch_size by adding elements\n",
-    "    samples_to_add = [{'Caption':padding_item, 'Theme':padding_item}] * (-len(samples) % batch_size)\n",
+    "    samples_to_add = [padding_item] * (-len(samples) % batch_size)\n",
     "    samples.extend(samples_to_add)\n",
     "    # reshape\n",
     "    samples = [samples[i:i+batch_size] for i in range(0, len(samples), batch_size)]"
@@ -160,7 +158,7 @@
     "# retrieve inference run details\n",
     "def get_last_inference_version(run_id):\n",
     "    try:\n",
-    "        inference_run = api.run(f'dalle-mini/dalle-mini/inference-{run_id}')\n",
+    "        inference_run = api.run(f'dalle-mini/dalle-mini/inf-{run_id}{suffix}')\n",
     "        return inference_run.summary.get('version', None)\n",
     "    except:\n",
     "        return None"
@@ -205,37 +203,7 @@
    "execution_count": null,
    "id": "bba70f33-af8b-4eb3-9973-7be672301a0b",
    "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "Processing artifact: model-4oh3u7ca:v54\n"
-     ]
-    },
-    {
-     "name": "stderr",
-     "output_type": "stream",
-     "text": [
-      "\u001b[34m\u001b[1mwandb\u001b[0m: Currently logged in as: \u001b[33mborisd13\u001b[0m (use `wandb login --relogin` to force relogin)\n"
-     ]
-    },
-    {
-     "data": {
-      "text/html": [
-       "\n",
-       "                    Syncing run <strong><a href=\"https://wandb.ai/dalle-mini/dalle-mini/runs/inference-4oh3u7ca\" target=\"_blank\">inference-4oh3u7ca</a></strong> to <a href=\"https://wandb.ai/dalle-mini/dalle-mini\" target=\"_blank\">Weights & Biases</a> (<a href=\"https://docs.wandb.com/integrations/jupyter.html\" target=\"_blank\">docs</a>).<br/>\n",
-       "\n",
-       "                "
-      ],
-      "text/plain": [
-       "<IPython.core.display.HTML object>"
-      ]
-     },
-     "metadata": {},
-     "output_type": "display_data"
-    }
-   ],
+   "outputs": [],
    "source": [
     "artifact_versions = get_artifact_versions(run_id, latest_only)\n",
     "last_inference_version = get_last_inference_version(run_id)\n",
@@ -247,10 +215,11 @@
     "    print(f'Processing artifact: {artifact.name}')\n",
     "    version = int(artifact.version[1:])\n",
     "    results = []\n",
-    "    columns = ['Caption', 'Theme'] + [f'Image {i+1}' for i in range(top_k)] + [f'Score {i+1}' for i in range(top_k)]\n",
+    "    columns = ['Caption'] + [f'Image {i+1}' for i in range(top_k)] + [f'Score {i+1}' for i in range(top_k)]\n",
     "    \n",
     "    if latest_only:\n",
-    "        assert last_inference_version is None or version > last_inference_version\n",
+    "        pass\n",
+    "        #assert last_inference_version is None or version > last_inference_version\n",
     "    else:\n",
     "        if last_inference_version is None:\n",
     "            # we should start from v0\n",
@@ -263,7 +232,7 @@
     "\n",
     "    # start/resume corresponding run\n",
     "    if run is None:\n",
-    "        run = wandb.init(job_type='inference', entity='dalle-mini', project='dalle-mini', config=training_config, id=f'inference-{run_id}', resume='allow')\n",
+    "        run = wandb.init(job_type='inference', entity='dalle-mini', project='dalle-mini', config=training_config, id=f'inf-{run_id}{suffix}', resume='allow')\n",
     "\n",
     "    # work in temporary directory\n",
     "    with tempfile.TemporaryDirectory() as tmp:\n",
@@ -284,8 +253,7 @@
     "\n",
     "        # process one batch of captions\n",
     "        for batch in tqdm(samples):\n",
-    "            prompts = [x['Caption'] for x in batch]\n",
-    "            processed_prompts = [text_normalizer(x) for x in prompts] if normalize_text else prompts\n",
+    "            processed_prompts = [text_normalizer(x) for x in batch] if normalize_text else list(batch)\n",
     "\n",
     "            # repeat the prompts to distribute over each device and tokenize\n",
     "            processed_prompts = processed_prompts * jax.device_count()\n",
@@ -306,7 +274,7 @@
     "\n",
     "            # get clip scores\n",
     "            print('Calculating CLIP scores')\n",
-    "            clip_inputs = processor(text=prompts, images=images, return_tensors='np', padding='max_length', max_length=77, truncation=True).data\n",
+    "            clip_inputs = processor(text=batch, images=images, return_tensors='np', padding='max_length', max_length=77, truncation=True).data\n",
     "            # each shard will have one prompt, images need to be reorganized to be associated to the correct shard\n",
     "            images_per_prompt_indices = np.asarray(range(0, len(images), batch_size))\n",
     "            clip_inputs['pixel_values'] = jnp.concatenate(list(clip_inputs['pixel_values'][images_per_prompt_indices + i] for i in range(batch_size)))\n",
@@ -318,11 +286,11 @@
     "\n",
     "            # add to results table\n",
     "            for i, (idx, scores, sample) in enumerate(zip(top_scores, logits, batch)):\n",
-    "                if sample['Caption'] == padding_item: continue\n",
+    "                if sample == padding_item: continue\n",
     "                cur_images = [images[x] for x in images_per_prompt_indices + i]\n",
     "                top_images = [wandb.Image(cur_images[x]) for x in idx]\n",
     "                top_scores = [scores[x] for x in idx]\n",
-    "                results.append([sample['Caption'], sample['Theme']] + top_images + top_scores)\n",
+    "                results.append([sample] + top_images + top_scores)\n",
     "\n",
     "    # log results\n",
     "    table = wandb.Table(columns=columns, data=results)\n",