Search and massive download

In this example, we will search for items using pygeodes, filter these items using geopandas dataframes and use a download queue to downloads these items and monitor the progress of the download.

Imports

Let’s start by importing geodes

from pygeodes import Geodes, Config

Configuration

We configure using a config file located in our cwd

conf = Config.from_file("config.json")
geodes = Geodes(conf=conf)

Searching products

We search for products in the T31TCK tile whose acquisition date is after 2023-01-01

from pygeodes.utils.datetime_utils import complete_datetime_from_str

query = {
    "grid:code": {"eq": "T31TCK"},
    "end_datetime": {"gte": complete_datetime_from_str("2023-01-01")},
}
items, dataframe = geodes.search_items(query=query)

/work/scratch/data/fournih/test_env/lib/python3.11/site-packages/urllib3/connectionpool.py:1099: InsecureRequestWarning: Unverified HTTPS request is being made to host 'geodes-portal.cnes.fr'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings
  warnings.warn(

Found 539 items matching your query, returning 80 as get_all parameter is set to False
80 item(s) found for query : {'grid:code': {'eq': 'T31TCK'}, 'end_datetime': {'gte': '2023-01-01T00:00:00.000000Z'}}

Exploring results

We get a list ot items and a dataframe, we can work with the dataframe for instance :

dataframe

	grid:code	end_datetime	id	collection	item	geometry
0	T31TCK	2024-05-02T10:56:21.024Z	URN:FEATURE:DATA:gdh:91c4d81a-0e4c-3dec-a9c2-4...	PEPS_S2_L1C	Item (URN:FEATURE:DATA:gdh:91c4d81a-0e4c-3dec-...	POLYGON ((0.95006 45.13349, 0.45686 45.12551, ...
1	T31TCK	2023-07-22T10:56:29.024Z	URN:FEATURE:DATA:gdh:e4e9708a-6005-3d57-9b36-0...	PEPS_S2_L1C	Item (URN:FEATURE:DATA:gdh:e4e9708a-6005-3d57-...	POLYGON ((0.49965 44.138, 0.55032 44.1388, 0.5...
2	T31TCK	2024-07-08T10:50:31.024Z	URN:FEATURE:DATA:gdh:48aeb103-69de-31b7-a3f3-d...	PEPS_S2_L1C	Item (URN:FEATURE:DATA:gdh:48aeb103-69de-31b7-...	POLYGON ((0.45686 45.12551, 0.49965 44.138, 1....
3	T31TCK	2023-04-08T10:56:21.024Z	URN:FEATURE:DATA:gdh:a6d9bb08-22ff-3be2-a715-9...	PEPS_S2_L1C	Item (URN:FEATURE:DATA:gdh:a6d9bb08-22ff-3be2-...	POLYGON ((0.49965 44.138, 0.54163 44.13866, 0....
4	T31TCK	2024-01-20T10:53:41.024Z	URN:FEATURE:DATA:gdh:084acd27-8d81-3dd1-adac-6...	PEPS_S2_L1C	Item (URN:FEATURE:DATA:gdh:084acd27-8d81-3dd1-...	POLYGON ((0.45686 45.12551, 0.49965 44.138, 1....
...	...	...	...	...	...	...
75	T31TCK	2023-07-19T10:46:29.024Z	URN:FEATURE:DATA:gdh:d38e0f8a-fa7b-352b-9c3f-b...	PEPS_S2_L1C	Item (URN:FEATURE:DATA:gdh:d38e0f8a-fa7b-352b-...	POLYGON ((0.49965 44.138, 1.87192 44.1598, 1.8...
76	T31TCK	2023-11-04T11:02:21.024Z	URN:FEATURE:DATA:gdh:444c28ff-2097-3fdf-b98e-7...	PEPS_S2_L1C	Item (URN:FEATURE:DATA:gdh:444c28ff-2097-3fdf-...	POLYGON ((0.86769 44.92037, 0.90268 45.00649, ...
77	T31TCK	2024-03-25T10:46:39.024Z	URN:FEATURE:DATA:gdh:7ad8089d-ac8d-33d9-b8c0-e...	PEPS_S2_L1C	Item (URN:FEATURE:DATA:gdh:7ad8089d-ac8d-33d9-...	POLYGON ((0.45686 45.12551, 0.49965 44.138, 1....
78	T31TCK	2023-06-04T10:46:21.024Z	URN:FEATURE:DATA:gdh:6236fb49-7523-30f7-8900-f...	PEPS_S2_L1C	Item (URN:FEATURE:DATA:gdh:6236fb49-7523-30f7-...	POLYGON ((0.49965 44.138, 1.87192 44.1598, 1.8...
79	T31TCK	2023-01-10T10:53:29.024Z	URN:FEATURE:DATA:gdh:d776f06a-f60a-3507-bdfd-6...	PEPS_S2_L1C	Item (URN:FEATURE:DATA:gdh:d776f06a-f60a-3507-...	POLYGON ((0.49965 44.138, 1.87192 44.1598, 1.8...

80 rows × 6 columns

Adding columns

We want to filter on cloudcover, so we need to add the column to the dataframe.

items[0].list_available_keys()

{'area',
 'continent_code',
 'dataset',
 'datetime',
 'end_datetime',
 'endpoint_description',
 'endpoint_url',
 'eo:cloud_cover',
 'grid:code',
 'hydrology.rivers',
 'id',
 'identifier',
 'instrument',
 'keywords',
 'latest',
 'platform',
 'political.continents',
 'processing:level',
 'product:timeliness',
 'product:type',
 'proj:bbox',
 'references',
 's2:datatake_id',
 'sar:instrument_mode',
 'sat:absolute_orbit',
 'sat:orbit_state',
 'sat:relative_orbit',
 'sci:doi',
 'start_datetime',
 'version'}

We find we can use spaceborne:cloudCover, so we add it to the dataframe :

from pygeodes.utils.formatting import format_items

dataframe_new = format_items(dataframe, {"eo:cloud_cover"})

Filtering our results

Now that the cloud cover is in our dataframe, we can filter on it.

dataframe_filtered = dataframe_new[dataframe_new["eo:cloud_cover"] < 30]

dataframe_filtered

	grid:code	end_datetime	id	collection	item	geometry	eo:cloud_cover
1	T31TCK	2023-07-22T10:56:29.024Z	URN:FEATURE:DATA:gdh:e4e9708a-6005-3d57-9b36-0...	PEPS_S2_L1C	Item (URN:FEATURE:DATA:gdh:e4e9708a-6005-3d57-...	POLYGON ((0.49965 44.138, 0.55032 44.1388, 0.5...	0.000000
2	T31TCK	2024-07-08T10:50:31.024Z	URN:FEATURE:DATA:gdh:48aeb103-69de-31b7-a3f3-d...	PEPS_S2_L1C	Item (URN:FEATURE:DATA:gdh:48aeb103-69de-31b7-...	POLYGON ((0.45686 45.12551, 0.49965 44.138, 1....	0.000000
3	T31TCK	2023-04-08T10:56:21.024Z	URN:FEATURE:DATA:gdh:a6d9bb08-22ff-3be2-a715-9...	PEPS_S2_L1C	Item (URN:FEATURE:DATA:gdh:a6d9bb08-22ff-3be2-...	POLYGON ((0.49965 44.138, 0.54163 44.13866, 0....	6.303467
4	T31TCK	2024-01-20T10:53:41.024Z	URN:FEATURE:DATA:gdh:084acd27-8d81-3dd1-adac-6...	PEPS_S2_L1C	Item (URN:FEATURE:DATA:gdh:084acd27-8d81-3dd1-...	POLYGON ((0.45686 45.12551, 0.49965 44.138, 1....	0.553047
6	T31TCK	2024-09-19T11:08:39Z	URN:FEATURE:DATA:gdh:7c5a6f36-d16f-3640-b38a-8...	MUSCATE_SENTINEL2_SENTINEL2_L2A	Item (URN:FEATURE:DATA:gdh:7c5a6f36-d16f-3640-...	POLYGON ((0.45686 45.12551, 1.85259 45.14807, ...	0.000000
7	T31TCK	2024-07-26T11:08:41Z	URN:FEATURE:DATA:gdh:2dc5d06f-cea7-3031-af0f-0...	MUSCATE_SENTINEL2_SENTINEL2_L2A	Item (URN:FEATURE:DATA:gdh:2dc5d06f-cea7-3031-...	POLYGON ((0.45686 45.12551, 1.85259 45.14807, ...	8.000000
10	T31TCK	2024-04-02T11:08:37Z	URN:FEATURE:DATA:gdh:0c18cbce-4e42-3df1-a557-9...	MUSCATE_SENTINEL2_SENTINEL2_L2A	Item (URN:FEATURE:DATA:gdh:0c18cbce-4e42-3df1-...	POLYGON ((0.45686 45.12551, 1.85259 45.14807, ...	6.000000
11	T31TCK	2024-08-05T11:08:41Z	URN:FEATURE:DATA:gdh:51739c13-41d7-328b-b4e5-1...	MUSCATE_SENTINEL2_SENTINEL2_L2A	Item (URN:FEATURE:DATA:gdh:51739c13-41d7-328b-...	POLYGON ((0.45686 45.12551, 1.85259 45.14807, ...	0.000000
12	T31TCK	2023-11-26T10:58:40Z	URN:FEATURE:DATA:gdh:6e03e00e-c4f0-343e-9a3e-8...	MUSCATE_SENTINEL2_SENTINEL2_L2A	Item (URN:FEATURE:DATA:gdh:6e03e00e-c4f0-343e-...	POLYGON ((0.45686 45.12551, 1.85259 45.14807, ...	0.000000
13	T31TCK	2023-06-04T10:58:53Z	URN:FEATURE:DATA:gdh:6a8cc460-c7c2-3cdf-9145-6...	MUSCATE_SENTINEL2_SENTINEL2_L2A	Item (URN:FEATURE:DATA:gdh:6a8cc460-c7c2-3cdf-...	POLYGON ((0.45686 45.12551, 1.85259 45.14807, ...	28.000000
14	T31TCK	2023-07-27T11:08:44Z	URN:FEATURE:DATA:gdh:fdeb23c0-113d-3819-a314-f...	MUSCATE_SENTINEL2_SENTINEL2_L2A	Item (URN:FEATURE:DATA:gdh:fdeb23c0-113d-3819-...	POLYGON ((0.45686 45.12551, 1.85259 45.14807, ...	0.000000
15	T31TCK	2023-09-10T11:08:42Z	URN:FEATURE:DATA:gdh:6bac51b5-e23c-3965-831e-c...	MUSCATE_SENTINEL2_SENTINEL2_L2A	Item (URN:FEATURE:DATA:gdh:6bac51b5-e23c-3965-...	POLYGON ((0.45686 45.12551, 1.85259 45.14807, ...	2.000000
16	T31TCK	2023-09-05T11:08:44Z	URN:FEATURE:DATA:gdh:a77faa14-ff6d-3d9e-a4c8-1...	MUSCATE_SENTINEL2_SENTINEL2_L2A	Item (URN:FEATURE:DATA:gdh:a77faa14-ff6d-3d9e-...	POLYGON ((0.45686 45.12551, 1.85259 45.14807, ...	4.000000
17	T31TCK	2023-06-24T10:58:53Z	URN:FEATURE:DATA:gdh:3c8bb792-9501-3d42-b8f3-b...	MUSCATE_SENTINEL2_SENTINEL2_L2A	Item (URN:FEATURE:DATA:gdh:3c8bb792-9501-3d42-...	POLYGON ((0.45686 45.12551, 1.85259 45.14807, ...	0.000000
18	T31TCK	2023-10-07T10:58:44Z	URN:FEATURE:DATA:gdh:69fec691-f643-30c4-99e3-5...	MUSCATE_SENTINEL2_SENTINEL2_L2A	Item (URN:FEATURE:DATA:gdh:69fec691-f643-30c4-...	POLYGON ((0.45686 45.12551, 1.85259 45.14807, ...	0.000000
20	T31TCK	2023-02-02T11:08:35Z	URN:FEATURE:DATA:gdh:564d14d2-b27d-317c-892e-5...	MUSCATE_SENTINEL2_SENTINEL2_L2A	Item (URN:FEATURE:DATA:gdh:564d14d2-b27d-317c-...	POLYGON ((0.45686 45.12551, 1.85259 45.14807, ...	28.000000
21	T31TCK	2024-09-15T00:00:00Z	URN:FEATURE:DATA:gdh:9030e38c-de3f-3206-a713-9...	MUSCATE_SENTINEL2_SENTINEL2_L3A	Item (URN:FEATURE:DATA:gdh:9030e38c-de3f-3206-...	POLYGON ((0.45686 45.12551, 1.85259 45.14807, ...	0.000000
24	T31TCK	2023-10-15T00:00:00Z	URN:FEATURE:DATA:gdh:2dfae9c4-4697-3c4b-a8f0-3...	MUSCATE_SENTINEL2_SENTINEL2_L3A	Item (URN:FEATURE:DATA:gdh:2dfae9c4-4697-3c4b-...	POLYGON ((0.45686 45.12551, 1.85259 45.14807, ...	0.000000
25	T31TCK	2023-04-15T00:00:00Z	URN:FEATURE:DATA:gdh:426693b5-9a6f-38ea-a7d8-0...	MUSCATE_SENTINEL2_SENTINEL2_L3A	Item (URN:FEATURE:DATA:gdh:426693b5-9a6f-38ea-...	POLYGON ((0.45686 45.12551, 1.85259 45.14807, ...	1.000000
29	T31TCK	2024-09-19T10:57:31.024Z	URN:FEATURE:DATA:gdh:f182e9be-de0e-34fa-94e0-f...	PEPS_S2_L1C	Item (URN:FEATURE:DATA:gdh:f182e9be-de0e-34fa-...	POLYGON ((0.95622 45.13358, 0.45686 45.12551, ...	0.000000
40	T31TCK	2024-10-31T10:51:09.024Z	URN:FEATURE:DATA:gdh:9f2ab752-b63e-3f7e-bc5c-d...	PEPS_S2_L1C	Item (URN:FEATURE:DATA:gdh:9f2ab752-b63e-3f7e-...	POLYGON ((1.1419 45.13659, 0.45686 45.12551, 0...	0.000000
41	T31TCK	2025-02-16T11:01:51.025Z	URN:FEATURE:DATA:gdh:ad5d3566-8930-3baa-b355-9...	PEPS_S2_L1C	Item (URN:FEATURE:DATA:gdh:ad5d3566-8930-3baa-...	POLYGON ((0.95337 45.13354, 0.45686 45.12551, ...	0.000000
42	T31TCK	2023-04-20T10:46:19.024Z	URN:FEATURE:DATA:gdh:f3cbdaf7-34ac-3a94-b7f4-b...	PEPS_S2_L1C	Item (URN:FEATURE:DATA:gdh:f3cbdaf7-34ac-3a94-...	POLYGON ((0.49965 44.138, 1.87192 44.1598, 1.8...	10.917376
49	T31TCK	2023-04-15T10:46:21.024Z	URN:FEATURE:DATA:gdh:968eada1-5eba-3526-8e38-6...	PEPS_S2_L1C	Item (URN:FEATURE:DATA:gdh:968eada1-5eba-3526-...	POLYGON ((1.87175 44.16864, 1.85259 45.14807, ...	21.794515
53	T31TCK	2024-04-12T10:56:21.024Z	URN:FEATURE:DATA:gdh:c3aa6a8a-c476-38d8-98a6-1...	PEPS_S2_L1C	Item (URN:FEATURE:DATA:gdh:c3aa6a8a-c476-38d8-...	POLYGON ((0.95063 45.13349, 0.45686 45.12551, ...	9.792046
57	T31TCK	2023-12-14T11:04:41.024Z	URN:FEATURE:DATA:gdh:feabfb13-c300-3e2b-af2f-1...	PEPS_S2_L1C	Item (URN:FEATURE:DATA:gdh:feabfb13-c300-3e2b-...	POLYGON ((0.96381 45.13371, 0.45686 45.12551, ...	0.000000
62	T31TCK	2023-09-10T10:56:29.024Z	URN:FEATURE:DATA:gdh:67ff3214-78c4-304e-82aa-2...	PEPS_S2_L1C	Item (URN:FEATURE:DATA:gdh:67ff3214-78c4-304e-...	POLYGON ((0.49965 44.138, 0.54246 44.13868, 0....	0.367962
66	T31TCK	2023-08-23T10:46:31.024Z	URN:FEATURE:DATA:gdh:7f4e618b-1d33-3017-9e83-d...	PEPS_S2_L1C	Item (URN:FEATURE:DATA:gdh:7f4e618b-1d33-3017-...	POLYGON ((0.49965 44.138, 1.87192 44.1598, 1.8...	0.000000
67	T31TCK	2023-06-24T10:46:21.024Z	URN:FEATURE:DATA:gdh:68922eac-baa5-3f4d-8f12-e...	PEPS_S2_L1C	Item (URN:FEATURE:DATA:gdh:68922eac-baa5-3f4d-...	POLYGON ((0.49965 44.138, 1.87192 44.1598, 1.8...	0.000000
69	T31TCK	2024-02-17T11:00:19.024Z	URN:FEATURE:DATA:gdh:5c1379a6-702e-3f27-9cc0-2...	PEPS_S2_L1C	Item (URN:FEATURE:DATA:gdh:5c1379a6-702e-3f27-...	POLYGON ((0.94737 45.13344, 0.45686 45.12551, ...	3.435898
71	T31TCK	2024-07-23T10:46:19.024Z	URN:FEATURE:DATA:gdh:985a0188-0e22-333c-ab1e-7...	PEPS_S2_L1C	Item (URN:FEATURE:DATA:gdh:985a0188-0e22-333c-...	POLYGON ((0.45686 45.12551, 0.49965 44.138, 1....	0.000000
73	T31TCK	2024-05-09T10:50:31.024Z	URN:FEATURE:DATA:gdh:bee1e3b7-84b7-332d-a06f-5...	PEPS_S2_L1C	Item (URN:FEATURE:DATA:gdh:bee1e3b7-84b7-332d-...	POLYGON ((0.45686 45.12551, 0.49965 44.138, 1....	0.000000
74	T31TCK	2023-04-15T10:46:21.024Z	URN:FEATURE:DATA:gdh:f7b216ae-64aa-313f-88e2-4...	PEPS_S2_L1C	Item (URN:FEATURE:DATA:gdh:f7b216ae-64aa-313f-...	POLYGON ((0.49965 44.138, 1.87192 44.1598, 1.8...	14.800485
75	T31TCK	2023-07-19T10:46:29.024Z	URN:FEATURE:DATA:gdh:d38e0f8a-fa7b-352b-9c3f-b...	PEPS_S2_L1C	Item (URN:FEATURE:DATA:gdh:d38e0f8a-fa7b-352b-...	POLYGON ((0.49965 44.138, 1.87192 44.1598, 1.8...	4.313566
78	T31TCK	2023-06-04T10:46:21.024Z	URN:FEATURE:DATA:gdh:6236fb49-7523-30f7-8900-f...	PEPS_S2_L1C	Item (URN:FEATURE:DATA:gdh:6236fb49-7523-30f7-...	POLYGON ((0.49965 44.138, 1.87192 44.1598, 1.8...	13.018869

Plotting

We can plot our results on a map :

dataframe_filtered.explore()

Make this Notebook Trusted to load map: File -> Trust Notebook

Downloading our items

We can download our results using the Profile system

from pygeodes.utils.profile import DownloadQueue, Profile

We reset our Profile to be sure to track only the downloads from the queue

Profile.reset()
items = dataframe_filtered["item"].values
queue = DownloadQueue(items)

In a separate cell, we run our queue

queue.run()