Question: I've been trying to get the extract a JSON requests API result into a csv but I'm struggeling,

so far the below code return the value in the terminal

```
import requests
import csv
import json
from pprint import pprint
import urllib
import pandas as pd

url = "https://esi.evetech.net/latest/markets/10000002/orders/?datasource=tranquility&order_type=all&page=1"

payload={}
headers = {}
r = {}

response = urllib.request.urlopen(url)

text = response.read()
json_data = json.loads(text)

pprint(json_data)
df = pd.read_json(r)
df.to_csv("output.csv")

```

however Pandas return me with an error in regards of the Dictionary class? ( apologize I'm not really familiar when it come to coding )

Last question would be, what would be the logic to continue the url request beyond page=1 in the url variable? (order\_type=all&page=**1** ) I don't know how many page the system have

thanks

Answer: 
> 
> I want to migrate data that is in a bucket. I want to ensure I migrate
> all items that are in the bucket when the code starts. If I migrate
> items that arrive after it starts that is ok but I can't miss anything
> that is there prior to it starting
> 
> 
> 

That being the case, you are good. This will work.
The exception wold be if new data overwrites data that you'd like to copy before you get to it, but if there is no overwriting take place, this is just fine.

> 
> But how does the iterator work such that we know it will be ok.
> Doesn't adding items to a list with an iterator change its behaviour?
> 
> 
> 

As seem in the documentation,`.list_blobs` returns a result that is partly "offline": it will fetch one page of results using the API and transparently fetch more pages as the itens in each request are iterated. That is: a new blocking HTTP request is made at each N items, N being the default page size - *but not* on each item consumed on the iterator. If you are not explicitly setting `max_results` it looks like it uses an API default. This means it will make a few requests throughout iteration and new files added between these requests will either show up at the end, or do not show up at all - it would be a very buggy behavior if these new files would spoil the result fetching.

Actually, if the the operator would work as a "live view" of the bucket, fetching an item each time it is iterated, it would have to be fairly complicated - the app using the API would have to work with some sort of parallelism be it multi-threading or async, in which the API code would be able to run and retrieve network data *while* your code is also running. That requires code an order of magnitude more complicated to work - and, would of course, only make sense if it would *benefit* the API user. If a "live content update" call was available, copying with changes in the source bucket would not only be its duty, but also, the *only* reason for such an implementation.

TL;DR: just go for it - when `.list_blobs()` return you have an object that can be iterated and contain a local (in opposed to "remote") static set of the objects present in the bucket at the moment the call was processed.

<https://googleapis.dev/python/storage/latest/client.html>
I think this can be simpler. `list_blobs` returns a [lazy iterator](https://googleapis.dev/python/storage/latest/client.html). Why not just materialize the list of blobs at the start and then iterate over them? That way you will not see any of the new blobs added to the bucket.

```py
from google.cloud import storage
client = storage.Client()
bucket = client.get_bucket("my_bucket")
blobs = list(bucket.list_blobs())
print(type(blobs))

>>> <class 'list'>

```