markdaoust:

I’m not sure I see the advantage if doing this inside tf.data. WHgy not just filter it first?


But it likely won’t scale when you have millions of URLs.
Let me provide a better context.
I am working with parquet files. A sample can downloaded from here:
!wget -q http://deploy.laion.ai/8f83b608504d46bb81708ec86e912220/dataset/part-00000-5b54c5d5-bbcf-484d-a2ce-0d6f73df1a36-c000.snappy.parquet

Then I am reading it with tensorflow_io:
column_spec = {
    "URL": tf.TensorSpec(tf.TensorShape([]), tf.string),
}

files = tf.io.gfile.glob("*.parquet")

dataset = tfio.IODataset.from_parquet(files[0], columns=column_spec)

Then my idea was to read the images on the fly:
img_sz = (260, 260)

def read_image(features):
    raw = tf.io.read_file(features["URL"])
    image = tf.image.decode_jpeg(raw, channels=3)
    image = tf.image.resize(image, img_sz, antialias=True)
    return image

But as you can imagine raw = tf.io.read_file(features["URL"]) would fail if the URL itself is 404. So, I thought maybe we can also add a check on the fly to filter the URL.