🚀 Getting Started

This guide walks you through creating and running your first Hermes pipeline using the Hermes CLI.

By the end, you will: - Initialize a Hermes project - Define a source, destination, and pipeline - Run your first pipeline

---

1. Installation

pip install "git+https://github.com/astral-sh/ruff"

---

2. Initialize a Hermes Project

The easiest way to get started is using the CLI:

hermes init

This will:

• Create a project structure
• Generate a hermes_config.yml file
• Optionally install connectors

Generated structure

.
├── configuration/
├── artefacts/
├── custom_connectors/
└── hermes_config.yml

Configuration file

project_paths:
  configuration_folder: ./configuration
  artefacts_folder: ./artefacts
  custom_connectors_folder: ./custom_connectors

👉 Hermes uses this file to locate your project folders.

---

3. Create Your First Pipeline Configuration

Inside the configuration/ folder, create a file:

configuration/demo.yml

---

4. Define a Source 🛫

sources:
  - name: currency_api
    description: Fetch currency exchange rates
    type: custom
    config:
      extractor: CurrencyExtractor
      module_path: custom_connectors.currency
      tables:
        - name: rates
          data_key: rates
          kwargs:
            endpoint: "https://www.floatrates.com/daily/mad.json"

---

5. Define a Destination 🛬

destinations:
  - name: local_output
    description: Store data locally
    type: local_storage
    config:
      format: json
      bucket: ./artefacts/data

---

6. Define a Pipeline ✈️

pipelines:
  - name: currency_pipeline
    sources:
      - name: currency_api
        tables: [rates]
    destinations:
      - local_output
    schedule: "0 12 * * *"

---

7. Implement the Extractor

Create the file:

custom_connectors/currency.py

import requests


class CurrencyExtractor:
    def extract(self, **kwargs):
        endpoint = kwargs["endpoint"]
        response = requests.get(endpoint)
        data = response.json()

        return {
            "rates": list(data.values())
        }

---

8. Run Your Pipeline

hermes pipeline run currency_pipeline

You should see:

Running pipeline: currency_pipeline
Pipeline currency_pipeline completed successfully!

---

9. Inspect the Output

Your data is now available in:

./artefacts/data/

Example:

artefacts/data/rates.json

---

🔍 Useful CLI Commands

List available pipelines

hermes pipeline list

---

Debug your environment

hermes debug

This shows:

• Installed connectors
• Configuration paths
• System information

---

Install connectors

hermes install

Interactive selection of available connectors.

---

🎉 Success!

You just ran your first Hermes pipeline.

---

🧠 What’s Next?

• Explore the Configuration Reference
• Build more advanced sources
• Use cloud destinations like S3 or Athena Iceberg

---

🛠️ Manual Setup (Optional)

If you prefer not to use the CLI, you can manually define:

export HERMES_CONFIG_FOLDER=./configuration
export HERMES_ARTIFACTS_FOLDER=./artefacts
export HERMES_CUSTOM_CONNECTORS_FOLDER=./custom_connectors

However, using hermes init is recommended.