Skip to content

Fetchers Module

The Fetchers Module is a Go package designed to fetch IP ranges for various cloud providers and services. It is part of the Caddy Defender project, which provides middleware for blocking or manipulating traffic based on IP ranges. This module supports fetching IP ranges from AWS, Google Cloud Platform (GCP), OpenAI, GitHub Copilot, and more.

Features

  • Multiple Cloud Providers: Fetch IP ranges for AWS, GCP, OpenAI, GitHub Copilot, and other services.
  • Region-Specific IP Ranges: Fetch IP ranges for specific AWS regions (e.g., us-east-1, eu-west-1).
  • Dynamic Fetching: IP ranges are fetched dynamically from official sources (e.g., AWS, GCP).
  • Pregenerated Results: Use pregenerated IP ranges from the ranges/data directory for faster setup.
  • Extensible: Easily add new fetchers for additional services or providers.
  • Concurrency: Fetch IP ranges concurrently for improved performance.

Supported Fetchers

Please see the IP ranges on the configuration page.

Pregenerated Results

For convenience, pregenerated IP ranges are available in the ranges/data directory. These ranges are generated by running the fetchers and can be used directly in your project without needing to fetch them dynamically.

Using Pregenerated Results

To use the pregenerated IP ranges, import the data package and access the IPRanges map:

package main

import (
    "fmt"
    "github.com/jasonlovesdoggo/caddy-defender/ranges/data"
)

func main() {
    // Access pregenerated IP ranges for AWS
    awsRanges := data.IPRanges["aws"]
    fmt.Println("AWS IP ranges:", awsRanges)

    // Access pregenerated IP ranges for GCP
    gcloudRanges := data.IPRanges["gcloud"]
    fmt.Println("GCP IP ranges:", gcloudRanges)

    // Access pregenerated IP ranges for OpenAI
    openaiRanges := data.IPRanges["openai"]
    fmt.Println("OpenAI IP ranges:", openaiRanges)

    // Access pregenerated IP ranges for Mistral
    mistralRanges := data.IPRanges["mistral"]
    fmt.Println("Mistral IP ranges:", mistralRanges)
    // ...
}

Available Pregenerated Ranges

The IPRanges map in the data package contains the following keys:

Key Description
vpn Known VPN services
aws Global IP ranges for AWS services.
aws-us-east-1 IP ranges for the AWS us-east-1 region.
aws-us-west-1 IP ranges for the AWS us-west-1 region.
gcloud IP ranges for Google Cloud Platform (GCP) services.
openai IP ranges for OpenAI services (e.g., ChatGPT, GPTBot).
oci IP ranges for Oracle Cloud Infrastructure (OCI) services
githubcopilot IP ranges for GitHub Copilot services.
private IP ranges for private networks (used for testing).
mistral IP ranges for Mistral services.
vultr IP ranges for Vultr Cloud services.
cloudflare IP ranges for Cloudflare services.
digitalocean IP ranges for Digital Ocean services.
linode IP ranges for Linode services.
tor IP addresses of Tor exit nodes (disabled by default).
asn IP ranges for specific ASNs (disabled by default).

Regenerating Pregenerated Results

To regenerate the pregenerated results, run the main.go file in the ranges directory:

cd ranges
go run main.go

This will fetch the latest IP ranges from all supported services and update the generated.go file in the data directory.

Installation

To use the Fetchers Module in your Go project, install it using go get:

go get github.com/jasonlovesdoggo/caddy-defender/ranges/fetchers

Usage

Fetching IP Ranges

To fetch IP ranges for a specific service, create an instance of the corresponding fetcher and call the FetchIPRanges method:

package main

import (
    "fmt"
    "github.com/jasonlovesdoggo/caddy-defender/ranges/fetchers"
)

func main() {
    // Fetch global AWS IP ranges
    awsFetcher := fetchers.AWSFetcher{}
    ranges, err := awsFetcher.FetchIPRanges()
    if err != nil {
        fmt.Println("Error fetching AWS IP ranges:", err)
    } else {
        fmt.Println("AWS IP ranges:", ranges)
    }

    // Fetch GCP IP ranges
    gcloudFetcher := fetchers.GCloudFetcher{}
    ranges, err = gcloudFetcher.FetchIPRanges()
    if err != nil {
        fmt.Println("Error fetching GCP IP ranges:", err)
    } else {
        fmt.Println("GCP IP ranges:", ranges)
    }
}

Using in Caddy Defender

The Fetchers Module is integrated into the Caddy Defender middleware. To use it, configure your Caddyfile with the defender directive:

localhost:8080 {
    defender block {
        ranges aws gcloud openai mistral
    }
    respond "Hello, world!"
}

This configuration blocks requests from IP ranges associated with AWS, GCP, OpenAI, and Mistral.

Adding New Fetchers

To add a new fetcher for a service or provider:

1. Create a New Fetcher:

  • Create a new file in the fetchers directory (e.g., my_service.go).
  • Implement the IPRangeFetcher interface:
package fetchers

import (
  "fmt"
)

// MyServiceFetcher implements the IPRangeFetcher interface for MyService.
type MyServiceFetcher struct{}

func (f MyServiceFetcher) Name() string {
  return "MyService"
}

func (f MyServiceFetcher) Description() string {
  return "Fetches IP ranges for MyService."
}

func (f MyServiceFetcher) FetchIPRanges() ([]string, error) {
  // Fetch IP ranges for MyService
  return []string{"203.0.113.0/24", "198.51.100.0/24"}, nil
}

2. Add the Fetcher to main.go:

  • Update the fetchersList in main.go to include your new fetcher:
fetchersList := []fetchers.IPRangeFetcher{
  fetchers.AWSFetcher{},
  fetchers.GCloudFetcher{},
  fetchers.MyServiceFetcher{}, // Add your new fetcher here
}

3. Rebuild and Test:

  • Rebuild the project and test the new fetcher to ensure it works as expected.

Example Fetchers

AWS Fetcher

Fetches global IP ranges for AWS services:

awsFetcher := fetchers.AWSFetcher{}
ranges, err := awsFetcher.FetchIPRanges()

AWS Region Fetcher

Fetches IP ranges for a specific AWS region (e.g., us-east-1):

awsRegionFetcher := fetchers.AWSRegionFetcher{Region: "us-east-1"}
ranges, err := awsRegionFetcher.FetchIPRanges()

GCloud Fetcher

Fetches IP ranges for Google Cloud Platform (GCP):

gcloudFetcher := fetchers.GCloudFetcher{}
ranges, err := gcloudFetcher.FetchIPRanges()

Contributing

We welcome contributions! If you want to add new fetchers or improve existing ones, follow these steps:

1. Fork the Repository:

2. Create a New Branch:

  • Create a branch for your changes:
git checkout -b my-new-fetcher

3. Make Your Changes:

  • Add your new fetcher or make improvements to existing code.

4. Test Your Changes:

  • Run the tests and ensure everything works as expected.

5. Submit a Pull Request:

  • Submit a pull request with a description of your changes.

License

This module is licensed under the MIT License. See the LICENSE file for details.

Acknowledgments

  • Caddy Server: Built with using Caddy.
  • AWS and GCP: For providing publicly accessible IP range data.
  • OpenAI and GitHub: For their IP range documentation.

Star History

Star History Chart