Steeping through data and brewing a database

2026-08-01

I like drinking tea. I have fond memories of the little bags of lemon tea that my father drank when we were young. Around the same time period a wayward cup of coffee somehow made its way down my stomach where it proceeded to cement itself (based on the stomachache), and coffee went way down the list of preferred drinks. Doesn't help that the rest of the family always drank it strong enough that the spoon would stand up unassisted. Nowadays I can go for a late night Irish Coffee or Scanian Kaffegök, but that's it.

Later, around the time I went to university there was this wonderful little shop with metal tins lining the walls, each containing its own interesting blend of tea and smells that would follow you out after leaving. It was a great visit every time and is where my tea drinking started to really pick up.

Though, I've never really got into more of the pure single origin teas, the T.G.F.O.P. and Pu-Ehrs of the world. My favorites are all black tea blends, a good Earl Grey with the citrusy zest, something a bit more floral, or maybe with berries? Ideally with natural flavor and not towards the super synthetic end of the flavor spectrum.

Case in point, I was on vacation earlier this year, and strolling through a market I found a nice little stand selling teas. The guy behind it was great and he let me sniff out a few of the fancy single origin teas he had for sale, but alas, no black blends in sight. So on the ride back my brain was catching on a potential new side project, how hard would it be to make your own tea blend? Just need to source some tea, and some ingredients, which shouldn't be that hard. But the quality is important, so would need to get ahold of some Calabrian bergamot oil for example. And if we're doing all of that, why not be a bit more creative. What else could I put in?

What else have people already tried to put in their tea blends?

So as always, instead of focusing on the core problem of possibly, maybe, making a tea blend, I thought about building a small database of the existing ones. What has been tried? What do people like? What flavor combinations have actually not been tried?

Plucking up leaves of data

To understand the different combinations of tea bases and ingredients I had my sights set on scraping a few different suppliers.

Tehuset Java, the Swedish one where I already know they are up to no good, wildly mixing and matching to make interesting blends. They also carry Kusmi tea, a french tea house with similar proclivities so it becomes our second candidate. Interestingly France seems to have a knack for this style of tea so we're throwing Dammann Frères in the mix as well. While the Germans are more into their fruit infusions we're giving them a shot through Paper & Tea and TeeGschwendner. At the end, the danish Østerlandsk also landed to round out the nordics.

While I'm in for the special blends, being respectable purveyors of tea they all have single origin pure tea as well, so if we're building a library of tea why not grab them all.

Writing all of this down and asking Claude to very kindly create an artifact to understand how far in over our heads we are led to this first prototype.

Teas on one side, details on the other. The pages seem to have enough information for this to be viable, they might be hidden in the description, in json data or in accordions, but they are there. Check on plausibility, water not even up to the ankles.

At this point our data pipeline consists of asking a magic chat to make me a website with teas. It feels magical, but being a single step is neither a data pipeline nor a trustworthy one.

No, we should to save the mana/tokens/spell slots for later. Instead let's have it build its own replacement data pipeline, one where we can re-run parts of it incrementally as we learn all the weird and varied ways that you can describe teas on the internet.

It would be nice if we could just throw a http request at the websites of our chosen suppliers and get everything back in one go, the whole list of teas and all of their details? Actually, Paper & Tea is Shopify based so they do serve a big JSON blob of all products. The rest need a bit more work, though Kusmi and TeeGschwendner have a helpful sitemap we can use. Java and Dammann Frères have neither so we need to navigate through all of their category listings to find all teas. I don't think any of them were fully client side rendered, otherwise we'd have to prep the headless harness as well.

Each site thus gets a different little spider, collecting a list of URLs to all the potentially tasty teas. Then we can filter it to remove things like tasty chocolate (not tea), pretty gifts sets (is tea, but not loose), different size packaging (100g vs 500g of Earl Gray might make a difference if you've run out of essential oil for your bath, but my tea cup is considerably smaller).

With the list of links, fetching the content is pretty simple. For each site we have One script to find them, One script to bring them all to our disk as HTML and JSON bundles, and One script to extract their data and in the darkness bind them, into json.

The reason we do it this way is that a couple of years ago I had Data Engineer as a title and inherited some particularly interesting pipelines, and after that I don't want to hear the word "backfill" anymore, so every intermediate step gets saved on disk.

With pipelines all written, across the six suppliers we're left with over a thousand teas! Plenty to do some fun ad-hoc analysis on, but after having being fetched they are all stored as the raw html and/or json we cold get from the supplier. Next we need to make some choices on how we'd want to model the teas in the extractor.

Let's take a look at an example to get a feeling for what we're working with.

{
  "product": {
    "name": "Abricot Toscane",
    "origin": null,
    "description": "A blend of black teas with the fruity scent of sun-ripened apricots.",
    "form": "loose_leaf",
    "brewing": { "dose_g_per_250ml": null, "temp_c": 90, "time_max": 5, "time_min": 4 },
    "flavor_profile": [ "Apricot" ],
    "in_stock": true,
    "organic": false,
    "price": { "currency": "EUR", "amount": 6.0, "size_g": 100 },
    "product_id": "46",
    "seasonal": false
  },
  "source": "dammann",
  "source_url": "https://www.dammann.fr/en/abricot-toscane-.html",
  "certifications": [],
  "image_url": "https://media.dammann.fr/img/p/4/8/5/3/4853.jpg?t=1771946321&width=375&height=268",
  "ingredients": [
    { "name": "black tea", "role": "base", "organic": false, "percentage": null },
    { "name": "apricot flavor", "role": "functional", "organic": false, "percentage": null },
    { "name": "flower petals", "role": "decorative", "organic": false, "percentage": null }
  ],
  "raw_ingredients": "Black tea (Camellia sinensis), apricot flavor, flower petals",
  "scrape_date": "2026-04-07"
}

Neat.

I want to be able to group together the different tea families and add a navigation that allows me to exclude everything that isn't black tea. So we'd need to populate some kind of tea_family field.

You might see that I've modeled the tea type as a ingredient with role: base, which is populated with what the supplier websites specify as a base tea (and we decided to pull in the extractor), so let's use that to create a filter at the top of the page and check what it gets us. Would expect things like Black Tea, Green Tea, a few other styles at the end.

assam, assam ctc, bai mu dan, bancha, benifuuki, black tea, black tea (decaffeinated), ceylon, chinese black tea, chinese pu-erh tea, chun mee, da bai hao, da hong pao, darjeeling, dong ding, four seasons oolong, fukamushicha, genmaicha, golden downy pekoe, green mate, green rooibos, green tea, houjicha, sencha fukuyu, gu zhang mao jian, gunpowder, gyokuro, hojicha, honeybush, houjicha, jasmine dragon phoenix pearls, jasmine green tea, jeoncha, jin xuan, kabusecha, kamairicha, karigane, keemun, kukicha, lapsang souchong, long jing, lung ching, mao feng, matcha, mate, meng ding gan lu, nepalese black tea, nepalese white tea, nilgiri, oolong, pai mu tan, perles de jade, pi lo chun, pu-erh, qi lan, roasted green tea, rooibos, sencha, sheng pu-erh, shincha, shou pu-erh, silver needle, tamaryokucha, tarry souchong, teguanyin, temple of heaven gunpowder, toro blanco, white peony, white tea, white tea silver tips, white tea tips, yellow tea, yerba mate, yin zhen, yunnan, yunnan golden, zhongbai

Oh, oh no. We're going to need some way of introducing a bit more structure to this field of leaves.

What even is Tea?

The list of teas above is too long to surface as any kind of top bar navigation (which is a thing I want to do), and some of the categories like black tea contains hundreds of items while something like Seogwang Sencha just a handful.

This is a situation where in order to build a better interface, we actually need to understand the underlaying domain and data better so that we can build the right dimensions.

First let's talk True Tea, made with the plant Camellia Sinensis, henceforth graced with an uppercase T to show its superiority over the other jumbled together mess of plant parts people put in their cups.

Tea is grown all over the place, the two main producers being China and India, and as you might expect Asia is well represented overall. What you might not expect is that Africa is also a large producer, with Kenya being the third largest after the two above. And if you're a fan of green tea you might be wondering where Japan places, still in the top 10, but after Argentina surprisingly. I'm getting these numbers (and research below) from "Tea: A User's Guide" by Tony Gebely.

Side-note, I love a good dialect, and interestingly the reason for the split between languages calling it Tea (or Té, Te, The, etc.) versus those leaning Cha(y) seems to have been that early Dutch and English traders were conversing with speakers of the Southern Fujian´s Amoy dialect, and ended up spreading their pronunciation across the world instead of the maybe more common one.

Do we need to worry about this geography though? Mostly not. There are differences in preparation between regions, and different styles being more common in different places. But we can kind of subsume the geography into the tea style. For example: Darjeeling and Assam are tea producing regions in India, and also specific styles of black tea. A similar curiosity is Ceylon, where while the country of Sri Lanka renamed itself in 1972, the tea didn't. The styles are the more important for all three, and for my purposes we can group them as styles under Black Tea.

We still have to deal with Green vs Black, Yellow, Red and the apparent rainbow of Tea colors. Of course, nerds both modern and most ancient have loved making categories. But the approach that makes the most sense to me is looking at the processing processes, with a big part being the oxidation level of the leaves themselves.

You know how when you're sitting in your evil volcano lair monologue-ing, taking a big bite out of a juicy apple for dramatic effect, which then starts to slowly brown as you go on? When a Tea leaf is plucked the same enzyme starts breaking it down, and a key part of the different tea styles is halting that process in different ways.

This is complex and nerdy enough for there to be books on the subject, but to keep things moving along, here's the main breakdown:

Going in order of how far the oxidation is allowed to run, from barely at all to all the way and then some:

• White tea, the least processed, just withered and dried enough to stop the oxidization
• Green Tea, minimally oxidized and the process is stopped with heat, pan-firing or steaming the leaves
• Yellow tea, starts the process same as green, but after the pan-firing they are wrapped up into bundles and left to sit, still a bit humid and warm, until they are finally dried
• Then we come to Oolong (Wulong) teas, halfway oxidized and with many subcategories of processing, aging, roasting, and enough variety to become a category of its own.
• Black Tea, first withered, fully oxidized in a humid environment and then fired dry to remove any moisture
• Pu-Erh or Pu'er also starts a lot like green tea, but the stopping happens at a lower temperature, leaving more of the enzymes active. After steaming it is then shaped into different shapes and just left alone. This causes the microbial processes to continue and ages the tea over time.

We can visualize it something like this

This gives us the main categories, and allows us to group other substyles under the main ones by understanding where the process diverges. For example, matcha starts life as a green tea (yes the plants are also shaded, but like, I can't go at this at that level of detail), which is then ground down to a fine powder, but overall it diverges from green, we have a system, good.

But then you ask, what about when I cozy down on the couch in the evening, preparing to re-read A Wizard of Earthsea and grab that lovely cup of chamomile T--! Nope, maybe a tea, but not a Tea, no matter what the aisles in any local German supermarket would have you think.

Those kinds of infusions of water and things that don't include the tea plant are sometimes called Tisane, and are sometimes also more popular than actual Tea (gasp!).

Some of these plants have their own proper tea styles, which can include blends as well, so it makes sense to me to give them categories. For example we have Rooibos from South Africa where I'll group Aspalathus Linearis and Cyclopia (honeybush). Then there is Ilex Paraguariensis, known as (Yerba) Mate from South America.

After that we have the rest which don't really have a unifying tradition but include fruit infusions or herbal spice blends. To be pragmatic here, these will be one big group (including your chamomile) as they are mainly mixes inside of the group.

That leaves us with if I'm counting correctly, 9 overall groups and a few substyles under most of them. Much more usable, and we've learnt a little bit on the way.

As an alternative, we could have gone full on geography mode (origin x cultivar x style) to try to capture even more of the different tea-space, but I found that harder to navigate, and for a lot of the teas we would be lacking one or more of those.

Scraping wise, the suppliers do provide this information, at minimum we'd know the base tea style. There is another pragmatic choice I did though, which is if we have a blend of black and green tea, it will be listed under both top level styles, it's quite rare and inventing new categories to cover these ones isn't worth it.

But what's that floating on top?

So far we've been concerned with the tea itself, the base ingredient. And there are many blends that are "just" mixing different teas from different places. English Breakfast is a well known example, which is usually made up of black tea from Assam, Ceylon and Kenya. But what I want to discover is what happens when you take the base tea and combine it with something new, not taking anything away but adding something extra on top.

I'd guess that even most people who aren't regular tea drinkers have heard of the Earl Grey, which is for me the archetypal "tea but something on top" kind of style that I'm personally looking for, and has itself spawned oh-so-many variations.

It is unclear if the Earl in question, Mr. Charles Grey, actually sipped on any of the tea now bearing his name (According to the Oxford English Dictionary, whose British-ness I feel makes a trustworthy source in the matter), but the mixture of Chinese tea leaves with extract from the citrus fruit bergamot seems to have been sold since at least 1836. 

Getting back to the data part, representing this in our interface should be fairly simple then, two ingredients, but even this seemingly simple tea makes things interesting. There are two main methods of flavoring the tea leaves, either infusing with bergamot essential oils, or adding the dried bergamot orange rinds to the blend itself.

I want to compare ingredients right, be able to see what kind of blends and flavors have been tried. So I'd like to normalize them so that it's easy to hyperlink and navigate between teas, ingredients and groupings of them.

To make it more concrete, let's list out the variations our suppliers use:

First off we have a mix of languages, then the different ways of physically getting the bergamot into the tea, if it is using real cold pressed oil (and if that's organically/ecologically farmed) or a fragrance, and if we include actual peel or not.

That's only for bergamot, but there are similar challenges across all ingredients, compound ingredient names, listings in parenthesis, (mis)-spellings and different grammatical constructions. You might be wondering why this would be a problem though, maybe we just translate them to English and just show the ingredients as is. That would be a pragmatic approach for a simple database, but not enough for the explorer that I want to have.

One small thing I'd also like to address is flowers. They are pretty and come in many colors. This has led people to pretty-up their teas by adding some in their teas as well. Adding a few Rose Petals will also alter the flavor profile while they are at it, but others, like the commonly added cornflower is mainly there for the oohs-and-aahs factor. They are not so important for filtering, but I'd like to treat them a bit differently visually.

Some of the decorative ones really thread the line between functional and decorative, but I opted for marking them decorative if the description explicitly said so.

With this in mind, for each of the ingredients we want to store and surface:

• The name (surprise!)
• The canonical name, decomposed for filtering without any form included
• The form (e.g. flavor, pieces, petals, oil, etc.)
• The role (is it a base/tea, a functional one that imparts flavor, or purely a decorative one)
• Is it organic or not?

Now, how do we actually do that?

Blending it all together

In the olden days of yore, manually fixing all of these issues would be the point where I'd drop this project and find something else to act as a focus for my procrastination. But, with the advent of LLMs to even out these lows and jump in when the next big chunk is pure tedium I could write a SKILL.md that took the original tea.json files that represented the output after extraction, and throw it to an agent to generate a tea.overlay.json with all of the data fixes and translations.

I'll include the skill below, for the curious readers.

---
name: generate-overlay
description: Generate LLM overlay JSONs for scraped tea data. Use when the user wants to create, update, or regenerate overlay files that enrich mechanically-scraped tea data with factual corrections and enrichments like origin details and type corrections.
user-invocable: true
allowed-tools: Read, Write, Glob, Grep, Bash(jq:*), Bash(ls:*), Bash(mix validate.teas:*)
---

# Generate Tea Data Overlays

You generate overlay JSON files that enrich mechanically-scraped tea data with **factual** corrections and enrichments derived from the product descriptions, ingredients, and breadcrumbs.

Scraped data lives at `data/teas/{prefix}-{handle}.json` where prefix is `pt` (Paper & Tea), `tj` (Tehuset Java), `kusmi` (Kusmi Tea), `tg` (TeeGschwendner), `df` (Dammann Frères), or `ol` (Østerlandsk Tehus). Detect the source from the file.

## Overlay format

Each overlay lives at `data/teas/{prefix}-{handle}.overlay.json` and contains ONLY fields that differ from or add to the base scraped data. The overlay is merged on top of the base data at build time.

```json
{
  "product": {
    "name": "Afternoon Tea Blend",
    "origin": "India",
    "origin_detail": "Darjeeling, India",
    "description": "English translation of description.",
    "original_description": "Original Swedish/German description."
  },
  "ingredients": [
    {"name": "darjeeling", "canonical_name": "darjeeling", "form": null, "role": "base", "organic": true, "percentage": 56},
    {"name": "jasmine flowers", "canonical_name": "jasmine", "form": "flowers", "role": "functional", "organic": true},
    {"name": "rose petals", "canonical_name": "rose", "form": "petals", "role": "decorative", "organic": true, "percentage": 5}
  ]
}
```

Every ingredient in the overlay **must** include `canonical_name`, `form`, and `role` (since the overlay replaces the entire ingredients array, omitting `role` would null it out):
- **`canonical_name`** — the base ingredient identity, lowercase, no form suffix. This is what groups ingredients across teas (e.g., "rose petals" and "rose essential oil" both have canonical_name "rose").
- **`form`** — the physical form suffix, or `null` if the name has no form. Valid forms: `oil`, `essential oil`, `peel`, `petals`, `pieces`, `extract`, `flavoring`, `natural flavoring`, `bark`, `root`, `leaves`, `leaf`, `powder`, `blossom`, `blossoms`, `flowers`, `flower`, `buds`, `seeds`, `seed`, `zest`, `berries`, `flakes`, `chips`, `crystals`, `needles`, `granules`.

Examples of correct decomposition:
| `name` | `canonical_name` | `form` |
|--------|-----------------|--------|
| `"bergamot essential oil"` | `"bergamot"` | `"essential oil"` |
| `"rose petals"` | `"rose"` | `"petals"` |
| `"raspberry flavoring"` | `"raspberry"` | `"flavoring"` |
| `"bergamot natural flavoring"` | `"bergamot"` | `"natural flavoring"` |
| `"bergamot oil"` | `"bergamot"` | `"oil"` |
| `"black tea"` | `"black tea"` | `null` |
| `"darjeeling"` | `"darjeeling"` | `null` |
| `"cinnamon"` | `"cinnamon"` | `null` |
| `"orange peel"` | `"orange"` | `"peel"` |
| `"lemongrass"` | `"lemongrass"` | `null` |
| `"licorice root"` | `"licorice"` | `"root"` |

Use `jq` for all JSON reading and writing.

## What to include in overlays

Focus on **factual enrichments only** — things that can be directly evidenced from the product description, ingredients, or tags.

### 1. Title cleaning (`product.name`)

The scraped title often contains tea type descriptors and ingredient hints after the actual name. Clean the title to just the product name, keeping the Swedish name as-is.

**Pattern:** Titles are Title Case for the name, then switch to lowercase for descriptors. Strip the lowercase tea type suffix and everything after it.

Examples:
- `"Afternoon Tea Blend svart te med bergamott/solros"` → `"Afternoon Tea Blend"`
- `"Brända Mandlar örtte"` → `"Brända Mandlar"`
- `"Darjeeling First Flush FTGFOP svart te"` → `"Darjeeling First Flush FTGFOP"`
- `"Bombay Chai svart/grönt te"` → `"Bombay Chai"`
- `"Karamellen - svart te med karamell och fudge"` → `"Karamellen"`
- `"Rooibos Lemonad ekologisk med grapefrukt/citrus"` → `"Rooibos Lemonad"`

**Keep as-is when the tea type word is Title Case (part of the name):**
- `"Magnolia Oolong"` → `"Magnolia Oolong"`
- `"Rött Julte"` → `"Rött Julte"`
- `"Formosa Fine Oolong"` → `"Formosa Fine Oolong"`
- `"Kinesiskt Gult Te"` → `"Kinesiskt Gult Te"`

**Lowercase suffixes to strip:** `svart te`, `grönt te`, `vitt te`, `örtte`, `svart/grönt te`, `koffeinfritt svart te`, and trailing `med ...` or lowercase `ekologisk`.

**Keep "Ekologisk"/"Ekologiskt" when it appears at the start of the name** — it's part of the product branding, not just a descriptor. E.g., `"Ekologisk Masala Chai svart te"` → `"Ekologisk Masala Chai"`, NOT `"Masala Chai"`.

### 2. Origin and origin_detail

- **origin** — **Country only.** When the scraper has `null` but the description or ingredient names clearly identify a country. Translate to English (`"Indien"` → `"India"`, `"Kina"` → `"China"`, `"Ceylon"` → `"Sri Lanka"`). Never put a region here.
- **origin_detail** — More specific sub-region. **Always use "Region, Country" format** (e.g., "Darjeeling, India", "Yunnan, China", "Uva, Sri Lanka", "Ceylon, Sri Lanka"). Never just the region alone.

### 3. Tea type corrections

Only when the mechanical parser clearly got it wrong. Be conservative.

### 4. Translate non-English fields

For `product.description`, translate to English. Keep the original in `product.original_description`. Ingredient names should be translated to English in the `ingredients` array.

### 5. Ingredients

Provide the full corrected ingredients array when any ingredient needs fixing. This replaces the scraped `ingredients` entirely. Only include when something actually needs changing. Common fixes:

**Translate non-English ingredient names to English.** Per-supplier translation
tables and arom/flavoring conventions live in the `docs/overlays/{prefix}.md`
file (Swedish for `tj-`, Danish for `ol-`).

**Rename verbose names:**
- `"mango pieces (mango, rice flour)"` → `"mango pieces"`
- `"natural bourbon vanilla flavor"` → `"vanilla flavoring (bourbon)"`

**Split compound ingredients into separate entries:**
- `"rose- & svartpeppar"` → two entries: `"pink pepper"` + `"black pepper"`
- `"green tea flavored with jasmine flowers"` → `"green tea"` + `"jasmine flowers"`
- `"passion fruit and pineapple flavoring"` → `"passion fruit flavoring"` + `"pineapple flavoring"`
- `"pineapple, peach, apricot, and vanilla flavoring"` → four separate flavoring entries
- Each ingredient should be a single, atomic item — never combine multiple flavors/items in one entry

**Keep ingredient names clean — no geographic info:**
- Geographic origin belongs in `origin` / `origin_detail`, NOT in the ingredient name
- `"black tea from Assam, India"` → just `"black tea"` (with `origin: "India"`, `origin_detail: "Assam, India"`)
- `"black tea from Sri Lanka"` → just `"black tea"` (with `origin: "Sri Lanka"`)

**Deduplicate after splitting:** When splitting a compound ingredient produces something that already exists in the list, drop the duplicate. E.g., if "vaniljstång" (vanilla bean) is already listed and a later compound "aprikos & vanilj" is split, only add "apricot" — the vanilla is already covered.

**Fix incorrect roles:**
- Flower ingredients (`solros`, `ros`, `malva`, `ringblomma`, `blå malva`) used for decoration → `"decorative"`
- Tea base ingredients classified as `functional` → `"base"`

**Canonical name standardization** (see `docs/glossary.md` "Canonical naming rules" for the full reference):
- **Always check `lib/blends/validator/known.ex`** for the exact canonical name before inventing one
- **Never combine geography + tea type**: `"nepalese black tea"` → `"black tea"` with `origin: "Nepal"`. Geography belongs in origin fields.
- **Never combine scenting + tea type**: `"jasmine green tea"` → split into `"green tea"` (base) + `"jasmine"` (functional)
- **Never use slugified/hyphenated names**: `"shou-pu-erh"` → `"shou pu-erh"`. Canonical names use spaces.
- Always use **singular** form: `"clove"` not `"cloves"`, `"redcurrant"` not `"redcurrants"`, `"pink peppercorn"` not `"pink peppercorns"`
- Use `"flavoring"` as the form, never `"flavor"`: `"raspberry flavoring"` not `"raspberry flavor"`
- Use `"pieces"` not `"bits"`: `"apple pieces"` not `"apple bits"`
- Use `"liquorice"` as canonical (British spelling, consistent across sources)
- Use `"lapsang souchong"` for all smoked souchong-style teas (including Japanese). No standalone `"souchong"` or `"lapsang"`.
- **No herbal bases**: chamomile, verbena, hibiscus, peppermint, rosehip, turmeric, cocoa, etc. are `role: "functional"`, never `"base"`. Only ingredients in `TeaFamily` (`lib/blends/tea_family.ex`) can be `role: "base"`. Herbal tisanes have no base ingredient.

**Zero-ingredient teas:**
When the base data has no ingredients (empty array), the overlay **must** add at minimum the base ingredient inferred from the product title, breadcrumb category, or description. E.g., a pure Darjeeling tea with no ingredients → `[{"name": "darjeeling", "canonical_name": "darjeeling", "form": null, "role": "base"}]`. The loader derives `tea_families` and `tea_base` from the ingredients array, so getting the base ingredient right is what makes the tea show up in the correct family.

**Base ingredient naming — use the most specific style name:**
The base ingredient name should capture the specific tea style, not just the broad type — the broad family (green, black, oolong, …) is derived automatically by `Blends.TeaFamily.families_for_product/1` from the base ingredient's canonical name.

Use the most specific name evidenced by the product title, description, or tags:

| Specificity | Base ingredient name | Resulting family |
|---|---|---|
| **Style known** (from title/description) | `"sencha"`, `"gyokuro"`, `"hojicha"`, `"kabusecha"`, `"kukicha"`, `"genmaicha"`, `"long jing"`, `"assam"`, `"darjeeling"`, `"ceylon"`, `"keemun"`, `"yunnan"`, `"sheng pu-erh"`, `"shou pu-erh"` | green, green, green, green, green, green, green, black, black, black, black, black, pu-erh, pu-erh |
| **Style unknown** (generic or blended) | `"green tea"`, `"black tea"`, `"white tea"`, `"yellow tea"`, `"oolong"`, `"pu-erh"`, `"rooibos"`, `"honeybush"` | green, black, white, yellow, oolong, pu-erh, rooibos, herbal |

This applies to ALL teas, not just pure ones:
- **Pure teas** with no ingredients: add `[{"name": "gyokuro", "role": "base", ...}]`
- **Blends with a named base**: rename generic `"black tea"` → `"assam"` when the title/description identifies the specific style (e.g., "Assam Tea Nandana" → base ingredient is `"assam"`, not `"black tea"`)
- **Blends with a generic base**: keep `"green tea"`, `"black tea"` etc. when no specific style is named (e.g., "Blended Black Tea Jackpot Derby" → `"black tea"`)
- **Blends with multiple named bases**: emit one base entry per named component when the description explicitly lists them (e.g., "A blend of Assam, Ceylon and Keemun" → three base entries: `"assam"`, `"ceylon"`, `"keemun"`). Note: "Ceylon" is a named black tea style (Sri Lanka), not a generic; use `"ceylon"` as the ingredient name, not `"black tea"`. Only split into multiple bases when the source explicitly names the components — do NOT invent bases from the broad family alone.
- **Non-tea herbals/fruits**: no change needed — ingredients like `"chamomile flowers"`, `"apples"`, `"spearmint"` are already specific

### What NOT to include

- **flavor_profile** — Do NOT add this field. Flavor normalization will happen later once we have multiple sources to compare against. This is a hard rule.

## How to generate

Detect the prefix from the existing base file (`data/teas/{prefix}-{handle}.json`),
then **read the per-supplier rules in `docs/overlays/{prefix}.md`** for source-
specific guidance (when an overlay is needed, language-specific translation
tables, naming conventions, edge cases). Apply those rules together with the
general guidelines in this file.

| Prefix | Supplier | Per-supplier rules |
|---|---|---|
| `pt-` | Paper & Tea | `docs/overlays/pt.md` |
| `tj-` | Tehuset Java | `docs/overlays/tj.md` |
| `df-` | Dammann Frères | `docs/overlays/df.md` |
| `ol-` | Østerlandsk Tehus | `docs/overlays/ol.md` |
| `kusmi-` | Kusmi Tea | (no source-specific rules yet) |
| `tg-` | TeeGschwendner | (no source-specific rules yet) |

The general workflow is the same for every source:

1. Read the base JSON with jq.
2. Read the per-supplier rules in `docs/overlays/{prefix}.md`.
3. (PT only) Read the matching entry from `data/pt-products.json` for raw
   Shopify tags / body_html.
4. Generate the overlay using the general rules in this file plus the
   per-supplier rules.
5. Write to `data/teas/{prefix}-{handle}.overlay.json`.
6. Run validation (see below).

## Post-generation validation

After writing the overlay file, **always** run the validator on the base file (not the overlay):

```bash
mix validate.teas data/teas/{prefix}-{handle}.json
```

The validator checks the merged result (base + overlay). Fix any **errors** before returning. Warnings are informational (e.g., unknown ingredients that may need to be added to the known list).

If validation fails, fix the issues in the overlay file and re-run until errors are resolved. Do not return a result with validation errors.

## Argument

A single tea handle (e.g., `golden-earl-n-514`, `bombay-chai-3`, `mon-petit-chocolat`, or `bai-mu-dan-te-okologisk`). Required. The prefix (`pt-`, `tj-`, `kusmi-`, `tg-`, `df-`, or `ol-`) is auto-detected from existing files.

## Guidelines

- Only create an overlay if there's something meaningful and **factual** to add or correct
- Don't duplicate data that's already correct in the base JSON
- Every field in the overlay must be traceable to evidence in the source data (description, tags, title, ingredients)
- Use English for all overlay values
- **All overlay fields go inside `product`** — `description`, `original_description`, `origin`, `origin_detail`, `name` are all under `product`. Only `ingredients` is at the top level. **NEVER** put `origin` or `origin_detail` at the root level.
- `origin` = country only (e.g., "India", "China", "Sri Lanka"). `origin_detail` = "Region, Country" format.
- When in doubt, skip — a missing overlay is better than a wrong one
- For source-specific defaults (which sources always need overlays vs. only when broken), see `docs/overlays/{prefix}.md`

### Overlay hygiene — avoid noise

- **Do NOT include fields that match the base data.** Only include fields that differ or add new information.
- **Do NOT include `organic: false`** — this is the default; only include `organic: true` when the ingredient is organic. This is a HARD rule — never emit `"organic": false` in overlay ingredients.
- **Do NOT include `percentage: null`** — only include percentage when you have a real value.
- **Do NOT set `origin_detail` to the same value as `origin`** — `origin_detail` should only be present when it adds sub-region info (e.g., `origin: "India"`, `origin_detail: "Assam, India"`). If you only know the country, set `origin` only and omit `origin_detail`.
- **Do NOT include `confidence`** — this is set by the loader, not by overlays.
- **Specific tea styles are lowercase**: `"darjeeling"`, `"sencha"`, `"assam"`, `"ceylon"` — never capitalized.
- **Fruit/food pieces are `functional`, not `decorative`**: strawberry pieces, apple pieces, mango pieces are food ingredients. Only flower petals, blossoms, and purely visual garnishes are `decorative`.
- **Primary herbal ingredients are `functional`, not `decorative`**: In herbal tisanes with a single herb (e.g., pure lavender, pure chamomile, pure peppermint, rose bud tea, lime blossom tea), that ingredient is the functional core of the tea, not a decoration. Use `"decorative"` only for ingredients that are added to an existing blend for visual appeal (e.g., cornflower petals scattered in an Earl Grey).
- **Keep specific form names intact**: do NOT rename `"bergamot essential oil"` to just `"bergamot"`. The full name goes in `name`, the decomposition goes in `canonical_name`/`form`.

I mainly used Haiku/Sonnet for this work, but even then they came up with strange artifacts and hallucinations. And I'm a fan of having guardrails do the work for me, so I added a small validation script, that validated both the structure of the final json, but also all the values, raising the alarm if an ingredient was not in the allowed list, and allowing me to make a choice on how to handle it. This really helped the overall quality, and made it much easier to address errors across the entire dataset when I found them.

The final sequence in our data pipeline then looks something like this, adding a cleanup and validation loop after the extraction.

And that was basically it for the data considerations, each supplier has a small document of special considerations, especially for translations. In terms of numbers, this leaves us with still six suppliers and a little bit over a thousand different teas, covering around 200 ingredients in use.

With the data cleared up, we can turn our attention back to the interface and how we present this potential treasure trove to the world, wide web. 

The initial prototype artifact had the now common LLM-built vibe when it comes to the colors, general design and also the overall page contrast (the lack of it). But its much nicer than the "programmer art" of yesteryear, and it already has a visual language that I can build on. I like that it helps across that empty first page problem, but for interfaces.

Armed with the artifact, and a detailed markdown spec, I threw both in an .inspiration folder as the shovel hit the ground. I settled on most of the scraping and data pipeline being built in Elixir, and after some iteration I tried out SolidJs for the frontend parts. Then Astro generates the static bundle with hydration and finally the pages are served by caddy hosted on my homelab.

One thing I had some annoyance with was keeping the visual identity clear. The initial artifact came with one, and I paired it with a markdown UI design system document, but I kept having to jump in and correct the output.

Small examples like: When we display a Teas or Tea Groups name, use the serif font, and tea groups always have a border. This should be true in all places where they are shown, even if they might be in different contexts (meaning shared components might be out), and details like that help things feel coherent.

The different variants fulfill different purposes. In the main list I want an overview to help the user, so big text that catches the eye, and since the whole point of the groups is the grouping I want to show the ingredients that it consists of. While on the detail sidebar it serves as an additional info section to surface similar groupings. This means it needs to be smaller, and instead show the differences.

There are a lot of these small micro decisions that surface specific information for that use, but should still rhyme and feel coherent with the rest. It's one of the places where I felt like I had to step in a lot to guide the LLM, even after writing ui-design system markdown files and components, specs and the whole jam.

With all of this, we have all of the data and we have an interface we can use to browse it with, time to get back to the initial question.

Reading the leaves

At this point we've built a data pipeline and an interface to explore it with, so ignoring all of the fun spent overengineering, now it's time for the fun part, what do they all blend their tea with?

I'm going to focus on Black tea here (as you might expect by now), if you want to do the same for the other types of tea, then you can explore yourself on blends.pv.wtf.

First of all, let's imagine what we'd expect the top ingredients for Black tea to be. Bergamot is most likely far up, due to all the Earl Grays, and probably a lot of other citrus fruits represented as well. After that, I'd say either something floral or maybe even creamy flavors? Let's take a look

Alright! King Bergamot and his court of citrus is well represented, I'm surprised to see orange outpace lemon though. The annoying "flavoring" is still there as well, even though I tried to extract which kind of flavoring via the descriptions, there are still a lot where that wasn't possible. Though I'm a bit surprised to see cinnamon and ginger so high up, there's a lot of Chai mixes but I thought those would still be more herbal than black. The list is rounded out with vanilla, cocoa and berries which matches my guess.

We can also quickly see if any of the ingredients appear a disproportionate time compared to the other bases, black is the most populated one, but we can also control for that.

This shows a slightly different perspective, bergamot on top, 1.8 times more likely on a Black tea than any other. But there are also some more from the long tail, smoke is a bit interesting as it's part of a tea style "Lapsang souchong" which incorporates smoking into the process to create a quite tasty blend. Vanilla, cocoa and the creamy flavors show up a bit more. And out of the fruity flavors, interesting that we have more papaya and maraschino, I'm assuming since a lot of other fruity ones are more evenly tried.

Maybe a quick check on which popular ingredients don't show up on any of our Blacks then?

We've got a lot of fresh minty flavors that don't seem to work with black, surprised that raisins don't show up in some kind of Christmas blend. Several of them do show up on both green and rooibos though, which aren't really that similar to each other in terms of flavour profiles.

I also wanted to take a look at pairings, if there are certain ingredients that always go together, and we can use the concept of lift to measure that. We will define it as lift = (containsBothCount × totalTeas) / (countA × countB) , which is kind of giving us a actual ÷ expected, a measure of how much this combo would stick out compared to chance.

Åhå! This is a list of surprises. The top spot going to Elderflower & Elderberry sounds like it could be a fun band name, but in the end it's the same plant so no points for creativity. There's a couple of Clusters of Chay (another great name combo), with clove practically cleaving through the list. Quite curious to try out some of the redcurrant blends, and someone at Dammann Frères really seems to have found something in Almond+biscuit with it showing up in four different teas.

Next, what if we try the opposite instead, the most lonely ingredients? These are all still teas sold by big respectable brands with online stores, so they are most likely not too far out, but maybe there's one or two surprises there too?

And my eyes immediately go to the Gina Amaretta with Amaretto in it, my interest is piqued and it goes in the cart. Maybe the almond and biscuits person has a friend over at TeeGschwendner.

With all of the different fruity flavors, it's interesting to see cantaloupe, guava and dragon fruit only in there once across the thousand teas. We've got some interesting one-off herbs as well though, but those are less interesting to me.

I think my takeaway from this informal analysis is that we've got several solid centers of just the kind of teas you'd expect, the Earl and his entourage, Clusters of Chay, followed by a tail of variants where there's an extra ingredient or two to spice things up, and finally a smattering of more exotic attempts.

What we have a hard time seeing are the "un-sellable" teas, or what hasn't been attempted at all. But we could take a look at the loneliest ingredients again, pair them up with something new to create a different blend.

For example, sloe is used only in one tea blend called Milde Waldbeeren or "Mild forest berries" together with Juniper, which brings to mind a lot of interesting sloe gins from where more inspiration could be taken, maybe adding a touch of vanilla or cinnamon to bring it halfway to Chay territory?

Or reaching from the world of Perfumes, there's a lonely Neroli based Blanc d'Oranger white tea, while Neroli as a citrus fruit is quite often combined with different floral notes in fragrances. A neroli/jasmine blend maybe? That's pretty much another Earl variant though, can't escape them.

Maybe as a final try we head back home to the north, try to combine the smoke and the forest, mixing Lapsang Souchong with pine to create a campfire mood. The Nordic Earl Grey from Java is something I've enjoyed so this takes that into a different direction.

Actually, then it looks like there's plenty more space for wild new blends in the world, and you can always lean back on good old Bergamot and citrus should inspiration falter.

Putting Down the Cup

This write-up comes a few months after I actually built my little explorer experiment, that time has turned research into tea as packages from both Dammann Frères and TeeGschwendner arrived at the doorstep after the old faithfuls from Tehuset Java ran out.

Dammann Frères hit a good spot between leaves and flavors (though I expect they could be a bit much for some), and the Earl Grey Kerala was my pick of the box. When it comes to TeeGschwendner I'd say that their flavouring was a bit gentler, though then also not as interesting, with potentially higher quality leaves, but that part is hard to judge when I only restock once there's basically only dust left of the last shipment. And to tie back to before, The Gina Amaretta is lovely, even if it's maybe not an everyday kind of cup.

Let's see if I find the inspiration to start up some tea-based alchemy of my own once the current shipment runs out, but for now my little parade of tins are filled a while longer.