The model owes a lot to the volume of open Copernicus data distributed on HuggingFace via ๐บ Major TOM org:
https://huggingface.co/Major-TOM
It's a fantastic resource for those looking to experiment with Earth observation applications! ๐
Mikolaj Czerkawski
mikonvergence
AI & ML interests
None yet
Recent Activity
replied to
their
post
2 days ago
๐ต ๐๐๐-๐๐๐-๐๐๐ญ๐: ๐๐ง๐ข๐๐ข๐๐ ๐๐๐ง๐๐ซ๐๐ญ๐ข๐ฏ๐ ๐๐จ๐๐๐ฅ๐ฅ๐ข๐ง๐ ๐จ๐ ๐๐๐๐๐ซ๐ง๐ข๐๐ฎ๐ฌ ๐๐ฆ๐๐ ๐๐ซ๐ฒ ๐๐ก๐ฎ๐ฆ๐๐ง๐๐ข๐ฅ๐ฌ
Today we release a prototype of COP-GEN - a universal generative model for Copernicus data. ๐๐๐-๐๐๐-๐๐๐ญ๐ is a model trained globally on the thumbnails of the Major TOM Core datasets, including Sentinel-2 L1C, Sentinel-2 L2A, Sentinel-1 RTC, and COP-DEM GLO-30.
โ๏ธ ๐๐จ๐๐๐ฅ https://huggingface.co/mespinosami/COP-GEN-Beta
๐ฑ ๐๐๐ฆ๐จ https://huggingface.co/spaces/mikonvergence/COP-GEN-Beta
How is it universal? COP-GEN learns a joint generative process of all modalities, which means that it can reconstruct data from any subset of present observations. ๐๐ข๐ญ๐ก๐จ๐ฎ๐ญ ๐ญ๐ซ๐๐ข๐ง๐ข๐ง๐ ๐ฌ๐ฉ๐๐๐ข๐๐ข๐๐๐ฅ๐ฅ๐ฒ to perform any of these tasks it can be used to approximate:
โ
Sentinel-1 to Sentinel-2 translation
โ
Elevation estimation from Sentinel-2 or Sentinel-1
โ
Atmospheric Correction (L1C to L2A pipeline)
โ
Atmospheric Generation (L2A to L1C)
โ
...and any other task involving translation between the supported modalities
On its own, the model can be used as a useful prior for estimating the data likelihood distribution for Copernicus data. COP-GEN-Beta learns joint, conditional, and marginal distributions within a single unified backbone, allowing to flexibly sample any modality given any condition.
Why is it Beta? Because thumbnails are a low-cost representation of the data that scales well and we managed to develop this prototype quite fast. We are currently developing the more costly COP-GEN model that supports the original data. For now, we wanted to showcase the prototype and make it available to the community for a test!
๐ ๐๐๐๐ฌ๐ข๐ญ๐ https://miquel-espinosa.github.io/cop-gen
๐ป ๐๐จ๐๐ https://github.com/miquel-espinosa/COP-GEN-Beta
๐ ๐๐๐ฉ๐๐ซ https://arxiv.org/pdf/2504.08548
posted
an
update
2 days ago
๐ต ๐๐๐-๐๐๐-๐๐๐ญ๐: ๐๐ง๐ข๐๐ข๐๐ ๐๐๐ง๐๐ซ๐๐ญ๐ข๐ฏ๐ ๐๐จ๐๐๐ฅ๐ฅ๐ข๐ง๐ ๐จ๐ ๐๐๐๐๐ซ๐ง๐ข๐๐ฎ๐ฌ ๐๐ฆ๐๐ ๐๐ซ๐ฒ ๐๐ก๐ฎ๐ฆ๐๐ง๐๐ข๐ฅ๐ฌ
Today we release a prototype of COP-GEN - a universal generative model for Copernicus data. ๐๐๐-๐๐๐-๐๐๐ญ๐ is a model trained globally on the thumbnails of the Major TOM Core datasets, including Sentinel-2 L1C, Sentinel-2 L2A, Sentinel-1 RTC, and COP-DEM GLO-30.
โ๏ธ ๐๐จ๐๐๐ฅ https://huggingface.co/mespinosami/COP-GEN-Beta
๐ฑ ๐๐๐ฆ๐จ https://huggingface.co/spaces/mikonvergence/COP-GEN-Beta
How is it universal? COP-GEN learns a joint generative process of all modalities, which means that it can reconstruct data from any subset of present observations. ๐๐ข๐ญ๐ก๐จ๐ฎ๐ญ ๐ญ๐ซ๐๐ข๐ง๐ข๐ง๐ ๐ฌ๐ฉ๐๐๐ข๐๐ข๐๐๐ฅ๐ฅ๐ฒ to perform any of these tasks it can be used to approximate:
โ
Sentinel-1 to Sentinel-2 translation
โ
Elevation estimation from Sentinel-2 or Sentinel-1
โ
Atmospheric Correction (L1C to L2A pipeline)
โ
Atmospheric Generation (L2A to L1C)
โ
...and any other task involving translation between the supported modalities
On its own, the model can be used as a useful prior for estimating the data likelihood distribution for Copernicus data. COP-GEN-Beta learns joint, conditional, and marginal distributions within a single unified backbone, allowing to flexibly sample any modality given any condition.
Why is it Beta? Because thumbnails are a low-cost representation of the data that scales well and we managed to develop this prototype quite fast. We are currently developing the more costly COP-GEN model that supports the original data. For now, we wanted to showcase the prototype and make it available to the community for a test!
๐ ๐๐๐๐ฌ๐ข๐ญ๐ https://miquel-espinosa.github.io/cop-gen
๐ป ๐๐จ๐๐ https://github.com/miquel-espinosa/COP-GEN-Beta
๐ ๐๐๐ฉ๐๐ซ https://arxiv.org/pdf/2504.08548
liked
a model
3 days ago
mespinosami/COP-GEN-Beta
Organizations
mikonvergence's activity
replied to
their
post
2 days ago
posted
an
update
2 days ago
Post
639
๐ต ๐๐๐-๐๐๐-๐๐๐ญ๐: ๐๐ง๐ข๐๐ข๐๐ ๐๐๐ง๐๐ซ๐๐ญ๐ข๐ฏ๐ ๐๐จ๐๐๐ฅ๐ฅ๐ข๐ง๐ ๐จ๐ ๐๐๐๐๐ซ๐ง๐ข๐๐ฎ๐ฌ ๐๐ฆ๐๐ ๐๐ซ๐ฒ ๐๐ก๐ฎ๐ฆ๐๐ง๐๐ข๐ฅ๐ฌ
Today we release a prototype of COP-GEN - a universal generative model for Copernicus data. ๐๐๐-๐๐๐-๐๐๐ญ๐ is a model trained globally on the thumbnails of the Major TOM Core datasets, including Sentinel-2 L1C, Sentinel-2 L2A, Sentinel-1 RTC, and COP-DEM GLO-30.
โ๏ธ ๐๐จ๐๐๐ฅ mespinosami/COP-GEN-Beta
๐ฑ ๐๐๐ฆ๐จ mikonvergence/COP-GEN-Beta
How is it universal? COP-GEN learns a joint generative process of all modalities, which means that it can reconstruct data from any subset of present observations. ๐๐ข๐ญ๐ก๐จ๐ฎ๐ญ ๐ญ๐ซ๐๐ข๐ง๐ข๐ง๐ ๐ฌ๐ฉ๐๐๐ข๐๐ข๐๐๐ฅ๐ฅ๐ฒ to perform any of these tasks it can be used to approximate:
โ Sentinel-1 to Sentinel-2 translation
โ Elevation estimation from Sentinel-2 or Sentinel-1
โ Atmospheric Correction (L1C to L2A pipeline)
โ Atmospheric Generation (L2A to L1C)
โ ...and any other task involving translation between the supported modalities
On its own, the model can be used as a useful prior for estimating the data likelihood distribution for Copernicus data. COP-GEN-Beta learns joint, conditional, and marginal distributions within a single unified backbone, allowing to flexibly sample any modality given any condition.
Why is it Beta? Because thumbnails are a low-cost representation of the data that scales well and we managed to develop this prototype quite fast. We are currently developing the more costly COP-GEN model that supports the original data. For now, we wanted to showcase the prototype and make it available to the community for a test!
๐ ๐๐๐๐ฌ๐ข๐ญ๐ https://miquel-espinosa.github.io/cop-gen
๐ป ๐๐จ๐๐ https://github.com/miquel-espinosa/COP-GEN-Beta
๐ ๐๐๐ฉ๐๐ซ https://arxiv.org/pdf/2504.08548
Today we release a prototype of COP-GEN - a universal generative model for Copernicus data. ๐๐๐-๐๐๐-๐๐๐ญ๐ is a model trained globally on the thumbnails of the Major TOM Core datasets, including Sentinel-2 L1C, Sentinel-2 L2A, Sentinel-1 RTC, and COP-DEM GLO-30.
โ๏ธ ๐๐จ๐๐๐ฅ mespinosami/COP-GEN-Beta
๐ฑ ๐๐๐ฆ๐จ mikonvergence/COP-GEN-Beta
How is it universal? COP-GEN learns a joint generative process of all modalities, which means that it can reconstruct data from any subset of present observations. ๐๐ข๐ญ๐ก๐จ๐ฎ๐ญ ๐ญ๐ซ๐๐ข๐ง๐ข๐ง๐ ๐ฌ๐ฉ๐๐๐ข๐๐ข๐๐๐ฅ๐ฅ๐ฒ to perform any of these tasks it can be used to approximate:
โ Sentinel-1 to Sentinel-2 translation
โ Elevation estimation from Sentinel-2 or Sentinel-1
โ Atmospheric Correction (L1C to L2A pipeline)
โ Atmospheric Generation (L2A to L1C)
โ ...and any other task involving translation between the supported modalities
On its own, the model can be used as a useful prior for estimating the data likelihood distribution for Copernicus data. COP-GEN-Beta learns joint, conditional, and marginal distributions within a single unified backbone, allowing to flexibly sample any modality given any condition.
Why is it Beta? Because thumbnails are a low-cost representation of the data that scales well and we managed to develop this prototype quite fast. We are currently developing the more costly COP-GEN model that supports the original data. For now, we wanted to showcase the prototype and make it available to the community for a test!
๐ ๐๐๐๐ฌ๐ข๐ญ๐ https://miquel-espinosa.github.io/cop-gen
๐ป ๐๐จ๐๐ https://github.com/miquel-espinosa/COP-GEN-Beta
๐ ๐๐๐ฉ๐๐ซ https://arxiv.org/pdf/2504.08548
-
- 1 reply
posted
an
update
7 days ago
Post
1510
๐๐๐๐ ๐๏ธ ๐๐๐ฑ๐ญ-๐๐๐ฌ๐๐ ๐ญ๐๐ซ๐ซ๐๐ข๐ง ๐ ๐๐ง๐๐ซ๐๐ญ๐ข๐จ๐ง ๐ฆ๐จ๐๐๐ฅ
MESA is a novel generative model based on latent denoising diffusion capable of generating 2.5D representations (co-registered colour and depth maps) of terrains based on text prompt conditioning.
Work developed by Paul BorneโPons ( @NewtNewt ) during his joint internship at
Adobe & ESA, and in collaboration with asterisk labs.
๐๏ธ ๐๐ซ๐จ๐ฃ๐๐๐ญ ๐๐๐ ๐ : https://paulbornep.github.io/mesa-terrain/
๐ ๐๐ซ๐๐ฉ๐ซ๐ข๐ง๐ญ : https://arxiv.org/abs/2504.07210
๐ค ๐๐จ๐๐๐ฅ ๐๐๐ข๐ ๐ก๐ญ๐ฌ : NewtNewt/MESA
๐พ ๐๐๐ญ๐๐ฌ๐๐ญ : Major-TOM/Core-DEM
๐ง๐ปโ๐ปโ๐๐จ๐๐ : https://github.com/PaulBorneP/MESA
๐๐ ๐๐ฉ๐๐๐: mikonvergence/MESA
MESA is a novel generative model based on latent denoising diffusion capable of generating 2.5D representations (co-registered colour and depth maps) of terrains based on text prompt conditioning.
Work developed by Paul BorneโPons ( @NewtNewt ) during his joint internship at
Adobe & ESA, and in collaboration with asterisk labs.
๐๏ธ ๐๐ซ๐จ๐ฃ๐๐๐ญ ๐๐๐ ๐ : https://paulbornep.github.io/mesa-terrain/
๐ ๐๐ซ๐๐ฉ๐ซ๐ข๐ง๐ญ : https://arxiv.org/abs/2504.07210
๐ค ๐๐จ๐๐๐ฅ ๐๐๐ข๐ ๐ก๐ญ๐ฌ : NewtNewt/MESA
๐พ ๐๐๐ญ๐๐ฌ๐๐ญ : Major-TOM/Core-DEM
๐ง๐ปโ๐ปโ๐๐จ๐๐ : https://github.com/PaulBorneP/MESA
๐๐ ๐๐ฉ๐๐๐: mikonvergence/MESA
- 2 replies
reacted to
mkluczek's
post with ๐๐ฅ
4 months ago
Post
1753
First Global and Dense Open Embedding Dataset of Earth! ๐ ๐ค
Introducing the Major TOM embeddings dataset, created in collaboration with CloudFerro S.A. ๐ถ and ฮฆ-lab at the European Space Agency (ESA) ๐ฐ๏ธ. Together with @mikonvergence and Jฤdrzej S. Bojanowski, we present the first open-access dataset of Copernicus embeddings, offering dense, global coverage across the full acquisition areas of Sentinel-1 and Sentinel-2 sensors.
๐ก Highlights:
๐ Data: Over 8 million Sentinel-1 & Sentinel-2 images processed, distilling insights from 9.368 trillion pixels of raw data.
๐ง Models: Foundation models include SigLIP, DINOv2, and SSL4EO.
๐ฆ Scale: 62 TB of raw satellite data processed into 170M+ embeddings.
This project delivers open and free vectorized expansions of Major-TOM/README datasets, setting a new standard for embedding releases and enabling lightweight, scalable ingestion of Earth Observation (EO) data for countless applications.
๐ค Explore the datasets:
Major-TOM/Core-S2L1C-SSL4EO
Major-TOM/Core-S1RTC-SSL4EO
Major-TOM/Core-S2RGB-DINOv2
Major-TOM/Core-S2RGB-SigLIP
๐ Check paper: Global and Dense Embeddings of Earth: Major TOM Floating in the Latent Space (2412.05600)
๐ป Code notebook: https://github.com/ESA-PhiLab/Major-TOM/blob/main/05-Generate-Major-TOM-Embeddings.ipynb
Introducing the Major TOM embeddings dataset, created in collaboration with CloudFerro S.A. ๐ถ and ฮฆ-lab at the European Space Agency (ESA) ๐ฐ๏ธ. Together with @mikonvergence and Jฤdrzej S. Bojanowski, we present the first open-access dataset of Copernicus embeddings, offering dense, global coverage across the full acquisition areas of Sentinel-1 and Sentinel-2 sensors.
๐ก Highlights:
๐ Data: Over 8 million Sentinel-1 & Sentinel-2 images processed, distilling insights from 9.368 trillion pixels of raw data.
๐ง Models: Foundation models include SigLIP, DINOv2, and SSL4EO.
๐ฆ Scale: 62 TB of raw satellite data processed into 170M+ embeddings.
This project delivers open and free vectorized expansions of Major-TOM/README datasets, setting a new standard for embedding releases and enabling lightweight, scalable ingestion of Earth Observation (EO) data for countless applications.
๐ค Explore the datasets:
Major-TOM/Core-S2L1C-SSL4EO
Major-TOM/Core-S1RTC-SSL4EO
Major-TOM/Core-S2RGB-DINOv2
Major-TOM/Core-S2RGB-SigLIP
๐ Check paper: Global and Dense Embeddings of Earth: Major TOM Floating in the Latent Space (2412.05600)
๐ป Code notebook: https://github.com/ESA-PhiLab/Major-TOM/blob/main/05-Generate-Major-TOM-Embeddings.ipynb
- 1 reply
posted
an
update
8 months ago
Post
2339
๐๐๐ฐ ๐๐๐ฅ๐๐๐ฌ๐: ๐๐๐ฃ๐จ๐ซ ๐๐๐ ๐๐ข๐ ๐ข๐ญ๐๐ฅ ๐๐ฅ๐๐ฏ๐๐ญ๐ข๐จ๐ง ๐๐จ๐๐๐ฅ ๐๐ฑ๐ฉ๐๐ง๐ฌ๐ข๐จ๐ง ๐บ๏ธ
Dataset: Major-TOM/Core-DEM
Today with European Space Agency - ESA and Adobe Research, we release a global expansion to Major TOM with GLO-30 DEM data.
You can now instantly access nearly 2M of Major TOM samples with elevation data to build your next AI model for EO. ๐
๐ Browse the data in our usual viewer app: Major-TOM/MajorTOM-Core-Viewer
Fantastic work championed by Paul Borne--Pons @NewtNewt ๐
Dataset: Major-TOM/Core-DEM
Today with European Space Agency - ESA and Adobe Research, we release a global expansion to Major TOM with GLO-30 DEM data.
You can now instantly access nearly 2M of Major TOM samples with elevation data to build your next AI model for EO. ๐
๐ Browse the data in our usual viewer app: Major-TOM/MajorTOM-Core-Viewer
Fantastic work championed by Paul Borne--Pons @NewtNewt ๐
posted
an
update
about 1 year ago
Post
1564
๐ ๐ฎ๐ท๐ผ๐ฟ ๐ง๐ข๐ : ๐ฃ๐น๐ฎ๐ป๐ฒ๐ ๐๐ฎ๐ฟ๐๐ต ๐ถ๐ ๐ฏฬถ๐นฬถ๐ฬถ๐ฒฬถ ๐ฑ.๐ฐ๐ฌ๐ฑ ๐๐๐
๐จ EXPANSION RELEASE: ๐ฆ๐ฒ๐ป๐๐ถ๐ป๐ฒ๐น-๐ญ ๐ถ๐ ๐ป๐ผ๐ ๐ฎ๐๐ฎ๐ถ๐น๐ฎ๐ฏ๐น๐ฒ in the MajorTOM-Core!
Major-TOM/Core-S1RTC
๐ Together with @aliFrancis we've been racing to release the first official expansion to the Major TOM project.
MajorTOM-Core-S1RTC contains 1,469,955 of SAR images paired to Sentinel-2 images from Core-S2.
๐We cover more than 65% of the optical coverage with an average time shift of 7 days.
16 TB of radiometrically calibrated SAR imagery, available in the exact same format as the existing Major-TOM data.
๐บ๏ธ You can explore instantly in our viewing app:
Major-TOM/MajorTOM-Core-Viewer
So, what now?
๐งฑ ๐๐จ๐ฆ๐ฆ๐ฎ๐ง๐ข๐ญ๐ฒ ๐๐ซ๐จ๐ฐ๐ญ๐ก: our community continues to grow! To coordinate the upcoming expansions as well as use cases of the open data, we will organise a meet up on 23 April, you can ๐ซ๐๐ ๐ข๐ฌ๐ญ๐๐ซ ๐ฒ๐จ๐ฎ๐ซ ๐ข๐ง๐ญ๐๐ซ๐๐ฌ๐ญ here: https://forms.gle/eBj8JvibJx9b6PLf9
๐ ๐๐ฉ๐๐ง ๐๐๐ญ๐ ๐๐จ๐ซ ๐๐ฉ๐๐ง ๐๐จ๐๐๐ฅ๐ฌ: Major-TOM Core dataset is currently supporting several strands of ongoing research within and outwith our lab and we are looking forward to the time when we can release models that take advantage of that data!
Major-TOM
๐ ๐๐จ๐ฌ๐ญ๐๐ซ ๐๐ญ ๐๐๐๐๐๐: We will present Major TOM project as a poster at IGARSS in Athens (July) - come talk to us if you're there! You can access the paper here: Major TOM: Expandable Datasets for Earth Observation (2402.12095)
๐ Developed at European Space Agency ฮฆ-lab in partnership with Hugging Face
๐จ EXPANSION RELEASE: ๐ฆ๐ฒ๐ป๐๐ถ๐ป๐ฒ๐น-๐ญ ๐ถ๐ ๐ป๐ผ๐ ๐ฎ๐๐ฎ๐ถ๐น๐ฎ๐ฏ๐น๐ฒ in the MajorTOM-Core!
Major-TOM/Core-S1RTC
๐ Together with @aliFrancis we've been racing to release the first official expansion to the Major TOM project.
MajorTOM-Core-S1RTC contains 1,469,955 of SAR images paired to Sentinel-2 images from Core-S2.
๐We cover more than 65% of the optical coverage with an average time shift of 7 days.
16 TB of radiometrically calibrated SAR imagery, available in the exact same format as the existing Major-TOM data.
๐บ๏ธ You can explore instantly in our viewing app:
Major-TOM/MajorTOM-Core-Viewer
So, what now?
๐งฑ ๐๐จ๐ฆ๐ฆ๐ฎ๐ง๐ข๐ญ๐ฒ ๐๐ซ๐จ๐ฐ๐ญ๐ก: our community continues to grow! To coordinate the upcoming expansions as well as use cases of the open data, we will organise a meet up on 23 April, you can ๐ซ๐๐ ๐ข๐ฌ๐ญ๐๐ซ ๐ฒ๐จ๐ฎ๐ซ ๐ข๐ง๐ญ๐๐ซ๐๐ฌ๐ญ here: https://forms.gle/eBj8JvibJx9b6PLf9
๐ ๐๐ฉ๐๐ง ๐๐๐ญ๐ ๐๐จ๐ซ ๐๐ฉ๐๐ง ๐๐จ๐๐๐ฅ๐ฌ: Major-TOM Core dataset is currently supporting several strands of ongoing research within and outwith our lab and we are looking forward to the time when we can release models that take advantage of that data!
Major-TOM
๐ ๐๐จ๐ฌ๐ญ๐๐ซ ๐๐ญ ๐๐๐๐๐๐: We will present Major TOM project as a poster at IGARSS in Athens (July) - come talk to us if you're there! You can access the paper here: Major TOM: Expandable Datasets for Earth Observation (2402.12095)
๐ Developed at European Space Agency ฮฆ-lab in partnership with Hugging Face
reacted to
osanseviero's
post with ๐ฅ
about 1 year ago
Post
Diaries of Open Source. Part 5!
๐คฏContextual KTO Mistral PairRM: this model combines iterative KTO, SnorkelAI DPO dataset, Allenai PairRM for ranking, Mistral for the base model, and is a very strong model with Claude 3 quality on AlpacaEval 2.0
Final model: ContextualAI/Contextual_KTO_Mistral_PairRM
Dataset: snorkelai/Snorkel-Mistral-PairRM-DPO-Dataset
Leaderboard: https://tatsu-lab.github.io/alpaca_eval/
Base model: mistralai/Mistral-7B-Instruct-v0.2
๐ค tinyBenchmarks: Quick and cheap LLM evaluation!
Code: https://github.com/felipemaiapolo/tinyBenchmarks
Paper: tinyBenchmarks: evaluating LLMs with fewer examples (2402.14992)
Data: tinyBenchmarks/tinyMMLU
๐จTransformers.js 2.16 includes StableLM, speaker verification and diarization, and better chat templating. Try some fun demos!
- Xenova/video-object-detection
- Xenova/cross-encoder-web
- Xenova/the-tokenizer-playground
๐ดโโ ๏ธ Abascus Liberated-Qwen1.5-72B, a Qwen 72B-based model that strongly follows system prompts
Model: abacusai/Liberated-Qwen1.5-72B
๐Design2Code: benchmark of webpage screenshots to code
Data: SALT-NLP/Design2Code
Project https://salt-nlp.github.io/Design2Code/
Paper Design2Code: How Far Are We From Automating Front-End Engineering? (2403.03163)
๐Data and models around the world
- One of the biggest Italian datasets https://hf.co/datasets/manalog/UsenetArchiveIT
- IndicLLMSuite: argest Pre-training and Instruction Fine-tuning dataset collection across 22 Indic languages ai4bharat/indicllmsuite-65ee7d225c337fcfa0991707
- Hebrew-Gemma-11B, the best base Hebrew model yam-peleg/Hebrew-Gemma-11B
- Komodo-7B, a family of multiple Indonesian languages LLMs Yellow-AI-NLP/komodo-7b-base
You can find the previous part at https://huggingface.co/posts/osanseviero/127895284909100
๐คฏContextual KTO Mistral PairRM: this model combines iterative KTO, SnorkelAI DPO dataset, Allenai PairRM for ranking, Mistral for the base model, and is a very strong model with Claude 3 quality on AlpacaEval 2.0
Final model: ContextualAI/Contextual_KTO_Mistral_PairRM
Dataset: snorkelai/Snorkel-Mistral-PairRM-DPO-Dataset
Leaderboard: https://tatsu-lab.github.io/alpaca_eval/
Base model: mistralai/Mistral-7B-Instruct-v0.2
๐ค tinyBenchmarks: Quick and cheap LLM evaluation!
Code: https://github.com/felipemaiapolo/tinyBenchmarks
Paper: tinyBenchmarks: evaluating LLMs with fewer examples (2402.14992)
Data: tinyBenchmarks/tinyMMLU
๐จTransformers.js 2.16 includes StableLM, speaker verification and diarization, and better chat templating. Try some fun demos!
- Xenova/video-object-detection
- Xenova/cross-encoder-web
- Xenova/the-tokenizer-playground
๐ดโโ ๏ธ Abascus Liberated-Qwen1.5-72B, a Qwen 72B-based model that strongly follows system prompts
Model: abacusai/Liberated-Qwen1.5-72B
๐Design2Code: benchmark of webpage screenshots to code
Data: SALT-NLP/Design2Code
Project https://salt-nlp.github.io/Design2Code/
Paper Design2Code: How Far Are We From Automating Front-End Engineering? (2403.03163)
๐Data and models around the world
- One of the biggest Italian datasets https://hf.co/datasets/manalog/UsenetArchiveIT
- IndicLLMSuite: argest Pre-training and Instruction Fine-tuning dataset collection across 22 Indic languages ai4bharat/indicllmsuite-65ee7d225c337fcfa0991707
- Hebrew-Gemma-11B, the best base Hebrew model yam-peleg/Hebrew-Gemma-11B
- Komodo-7B, a family of multiple Indonesian languages LLMs Yellow-AI-NLP/komodo-7b-base
You can find the previous part at https://huggingface.co/posts/osanseviero/127895284909100
reacted to
osanseviero's
post with ๐
about 1 year ago
Post
Diaries of Open Source. Part 3! OS goes to the moon!
๐ป OpenCodeInterpreter, a family of very powerful code generation models
Models: m-a-p/opencodeinterpreter-65d312f6f88da990a64da456
Paper: OpenCodeInterpreter: Integrating Code Generation with Execution and Refinement (2402.14658)
Demo m-a-p/OpenCodeInterpreter_demo
๐ท๐ถZephyr 7B Gemma, Gemma fine-tuned with the Zephyr recipe
Model: HuggingFaceH4/zephyr-7b-gemma-v0.1
Demo: HuggingFaceH4/zephyr-7b-gemma-chat
GH Repo: https://github.com/huggingface/alignment-handbook
๐ชThe MixedBread folks released a 2D Matryoshka text embedding model, which means you can dynamically change the embedding size and layer counts
Model: mixedbread-ai/mxbai-embed-2d-large-v1
Release blog post: https://www.mixedbread.ai/blog/mxbai-embed-2d-large-v1
๐Microsoft released Orca Math, which includes 200K grade school math problems
Dataset: microsoft/orca-math-word-problems-200k
๐ฅทIBM silently released Merlinite, a cool model trained on Mixtral-generated synthetic data using a novel LAB method https://huggingface.co/ibm/merlinite-7b
๐ Moondream2 - a small vision language model to run on-device!
Model: vikhyatk/moondream2
Demo: vikhyatk/moondream2
๐๏ธCityDreamer: 3D City Generation
Demo: hzxie/city-dreamer
Repo: https://github.com/hzxie/city-dreamer
Model: hzxie/city-dreamer
๐ML in all languages
Sailor, a family of South-East Asian languages models sail/sailor-language-models-65e19a749f978976f1959825
Samvaad dataset, which includes 140k QA pairs in Hindi, Bengali, Marathi, Tamil, Telugu, Oriya, Punjabi, and Gujarati GenVRadmin/Samvaad-Mixed-Language-2
You can see the previous part at https://huggingface.co/posts/osanseviero/674644082063278
๐ป OpenCodeInterpreter, a family of very powerful code generation models
Models: m-a-p/opencodeinterpreter-65d312f6f88da990a64da456
Paper: OpenCodeInterpreter: Integrating Code Generation with Execution and Refinement (2402.14658)
Demo m-a-p/OpenCodeInterpreter_demo
๐ท๐ถZephyr 7B Gemma, Gemma fine-tuned with the Zephyr recipe
Model: HuggingFaceH4/zephyr-7b-gemma-v0.1
Demo: HuggingFaceH4/zephyr-7b-gemma-chat
GH Repo: https://github.com/huggingface/alignment-handbook
๐ชThe MixedBread folks released a 2D Matryoshka text embedding model, which means you can dynamically change the embedding size and layer counts
Model: mixedbread-ai/mxbai-embed-2d-large-v1
Release blog post: https://www.mixedbread.ai/blog/mxbai-embed-2d-large-v1
๐Microsoft released Orca Math, which includes 200K grade school math problems
Dataset: microsoft/orca-math-word-problems-200k
๐ฅทIBM silently released Merlinite, a cool model trained on Mixtral-generated synthetic data using a novel LAB method https://huggingface.co/ibm/merlinite-7b
๐ Moondream2 - a small vision language model to run on-device!
Model: vikhyatk/moondream2
Demo: vikhyatk/moondream2
๐๏ธCityDreamer: 3D City Generation
Demo: hzxie/city-dreamer
Repo: https://github.com/hzxie/city-dreamer
Model: hzxie/city-dreamer
๐ML in all languages
Sailor, a family of South-East Asian languages models sail/sailor-language-models-65e19a749f978976f1959825
Samvaad dataset, which includes 140k QA pairs in Hindi, Bengali, Marathi, Tamil, Telugu, Oriya, Punjabi, and Gujarati GenVRadmin/Samvaad-Mixed-Language-2
You can see the previous part at https://huggingface.co/posts/osanseviero/674644082063278
- 1 reply
replied to
robmarkcole's
post
about 1 year ago
It was a pleasure @robmarkcole !
Really enjoyed the chat and I hope it can encourage more people to build future EO datasets together! ๐๐๐
๐บ Anyone can join our growing community here:
https://huggingface.co/Major-TOM
reacted to
robmarkcole's
post with ๐ค
about 1 year ago
Post
As promised, a video on the new Major TOM dataset on HF: https://youtu.be/KonWxQ1mCpA?si=tp3Cz6-F0exfgdx3
If you prefer audio, there is a podcast too: https://www.satellite-image-deep-learning.com/p/major-tom-expandable-eo-datasets
I think this is just the start for remote sensing datasets on HuggingFace..!
If you prefer audio, there is a podcast too: https://www.satellite-image-deep-learning.com/p/major-tom-expandable-eo-datasets
I think this is just the start for remote sensing datasets on HuggingFace..!
-
- 1 reply
reacted to
osanseviero's
post with ๐
about 1 year ago
Post
Diaries of Open Source. Part 2. Open Source is going brrrrr
๐The European Space Agency releases MajorTOM, a dataset of earth observation covering half the earth. The dataset has 2.5 trillion pixels! Congrats @aliFrancis and @mikonvergence !
Dataset: Major-TOM/Core-S2L2A
Viewer: Major-TOM/MajorTOM-Core-Viewer
๐Re-ranking models by MixedBreadAI, with very high quality, Apache 2 license, and easy to use!
Models: https://huggingface.co/models?other=reranker&sort=trending&search=mixedbread-ai
Blog: https://www.mixedbread.ai/blog/mxbai-rerank-v1
๐งStabilityAI and TripoAI release TripoSR, a super-fast MIT-licensed image-to-3D model!
Model: stabilityai/TripoSR
Demo: stabilityai/TripoSR
๐คTogether AI and HazyResearch release Based
Models and datasets: hazyresearch/based-65d77fb76f9c813c8b94339c
GH repo: https://github.com/HazyResearch/based
๐LaVague: an open-source pipeline to turn natural language into browser actions! It can run locally with HuggingFaceH4/zephyr-7b-gemma-v0.1
Read more about it at https://huggingface.co/posts/dhuynh95/717319217106504
๐Berkeley Function-Calling Leaderboard
Read about it: https://gorilla.cs.berkeley.edu/blogs/8_berkeley_function_calling_leaderboard.html
Leaderboard: https://gorilla.cs.berkeley.edu/leaderboard.html
๐ฌSailor-Chat: chat models built on top of OpenOrca and @sarahooker CohereForAI Aya project. They can be used for South-East Asia languages such as Indonesian, Thai, Vietnamese, Malay and Lao!
Models: sail/sailor-language-models-65e19a749f978976f1959825
Demo: https://huggingface.co/spaces/sail/Sailor-7B-Chat
๐คArabic-OpenHermes-2.5: OpenHermes dataset translated to Arabic 2A2I/Arabic-OpenHermes-2.5
See the previous part here https://huggingface.co/posts/osanseviero/622788932781684
๐The European Space Agency releases MajorTOM, a dataset of earth observation covering half the earth. The dataset has 2.5 trillion pixels! Congrats @aliFrancis and @mikonvergence !
Dataset: Major-TOM/Core-S2L2A
Viewer: Major-TOM/MajorTOM-Core-Viewer
๐Re-ranking models by MixedBreadAI, with very high quality, Apache 2 license, and easy to use!
Models: https://huggingface.co/models?other=reranker&sort=trending&search=mixedbread-ai
Blog: https://www.mixedbread.ai/blog/mxbai-rerank-v1
๐งStabilityAI and TripoAI release TripoSR, a super-fast MIT-licensed image-to-3D model!
Model: stabilityai/TripoSR
Demo: stabilityai/TripoSR
๐คTogether AI and HazyResearch release Based
Models and datasets: hazyresearch/based-65d77fb76f9c813c8b94339c
GH repo: https://github.com/HazyResearch/based
๐LaVague: an open-source pipeline to turn natural language into browser actions! It can run locally with HuggingFaceH4/zephyr-7b-gemma-v0.1
Read more about it at https://huggingface.co/posts/dhuynh95/717319217106504
๐Berkeley Function-Calling Leaderboard
Read about it: https://gorilla.cs.berkeley.edu/blogs/8_berkeley_function_calling_leaderboard.html
Leaderboard: https://gorilla.cs.berkeley.edu/leaderboard.html
๐ฌSailor-Chat: chat models built on top of OpenOrca and @sarahooker CohereForAI Aya project. They can be used for South-East Asia languages such as Indonesian, Thai, Vietnamese, Malay and Lao!
Models: sail/sailor-language-models-65e19a749f978976f1959825
Demo: https://huggingface.co/spaces/sail/Sailor-7B-Chat
๐คArabic-OpenHermes-2.5: OpenHermes dataset translated to Arabic 2A2I/Arabic-OpenHermes-2.5
See the previous part here https://huggingface.co/posts/osanseviero/622788932781684
- 3 replies
reacted to
robmarkcole's
post with โค๏ธ
about 1 year ago
Post
Huge new dataset for remote sensing on HF: Major TOM
More info in this thread https://x.com/mikonvergence/status/1764913256102031402?s=20
Note: video coming soon
More info in this thread https://x.com/mikonvergence/status/1764913256102031402?s=20
Note: video coming soon
- 1 reply
reacted to
aliFrancis's
post with ๐ค
about 1 year ago
Post
๐บ Major TOM: Expandable Datasets for Earth Observation
๐จ RECORD-BREAKING EO DATASET: the largest ever ML-ready Sentinel-2 dataset! It covers almost every single point on Earth captured by the Copernicus Sentinel-2 satellite. @mikonvergence and I are thrilled to finally announce the release of Major-TOM/Core-S2L2A and Major-TOM/Core-S2L1C
๐ About half of the entire planet is covered. That's 2,245,886 patches of 1068 x 1068 pixels, available in both L1C and L2A. At 10 m resolution, we've got 256 million square km with over 2.5 trillion pixels. It's all yours with a few lines of code. See the paper linked below ๐ฝ for more info!
๐งฑ And this is just the beginning. We are currently preparing more datasets from different satellites for the Major TOM org. TOM stands for Terrestrial Observation Metaset - a simple set of rules for building an ecosystem of ML-ready EO datasets, which can be seamlessly combined as if they were Lego bricks.
๐ดโโ๏ธ Want to take the dataset for a spin? We have a viewer app on spaces that lets you go anywhere on Earth and shows you the data, if its available Major-TOM/MajorTOM-Core-Viewer
๐ฐ Preprint paper: Major TOM: Expandable Datasets for Earth Observation (2402.12095)
๐ป Colab example: https://colab.research.google.com/github/ESA-PhiLab/Major-TOM/blob/main/03-Filtering-in-Colab.ipynb
Thank you to the amazing ๐คHugging Face team for the support on this one! @osanseviero @lhoestq @BrigitteTousi
๐จ RECORD-BREAKING EO DATASET: the largest ever ML-ready Sentinel-2 dataset! It covers almost every single point on Earth captured by the Copernicus Sentinel-2 satellite. @mikonvergence and I are thrilled to finally announce the release of Major-TOM/Core-S2L2A and Major-TOM/Core-S2L1C
๐ About half of the entire planet is covered. That's 2,245,886 patches of 1068 x 1068 pixels, available in both L1C and L2A. At 10 m resolution, we've got 256 million square km with over 2.5 trillion pixels. It's all yours with a few lines of code. See the paper linked below ๐ฝ for more info!
๐งฑ And this is just the beginning. We are currently preparing more datasets from different satellites for the Major TOM org. TOM stands for Terrestrial Observation Metaset - a simple set of rules for building an ecosystem of ML-ready EO datasets, which can be seamlessly combined as if they were Lego bricks.
๐ดโโ๏ธ Want to take the dataset for a spin? We have a viewer app on spaces that lets you go anywhere on Earth and shows you the data, if its available Major-TOM/MajorTOM-Core-Viewer
๐ฐ Preprint paper: Major TOM: Expandable Datasets for Earth Observation (2402.12095)
๐ป Colab example: https://colab.research.google.com/github/ESA-PhiLab/Major-TOM/blob/main/03-Filtering-in-Colab.ipynb
Thank you to the amazing ๐คHugging Face team for the support on this one! @osanseviero @lhoestq @BrigitteTousi
- 1 reply