# go-linkfinder

`go-linkfinder` is a simple web crawling and link extraction tool written in Go, inspired by [https://github.com/GerbenJavado/LinkFinder](https://github.com/GerbenJavado/LinkFinder). It supports fetching content from URLs or files, recursively crawling same-domain links up to a specified depth, and customizing request headers, user-agent, timeout, and delay between requests.

> **WARNING**: This tool is intended for **authorized security testing and web research only**. Unauthorized crawling may violate website terms of service or legal regulations. The author and contributors are not responsible for misuse. Always obtain explicit permission before crawling any site.

## Features

- **Input Sources**: Fetch content from URLs or local files.
- **Recursive Crawling**: Configurable recursion depth to crawl same-domain links.
- **Request Customization**: Set HTTP headers and user-agent strings.
- **Timeout Control**: Customize HTTP request timeout.
- **Delay Between Requests**: Configurable delay (default 4 seconds) to avoid overloading servers.
- **Automatic Gzip Handling**: Supports gzip compressed HTTP responses.
- **URL Normalization**: Resolves relative URLs to absolute form.
- **Duplicate Detection**: Tracks and avoids revisiting URLs.

## Installation

### Prerequisites

- **Go**: Version 1.21 or later.
- **Make**: For building with the provided Makefile.
- **Git**: To clone the repository.

### Steps

- Clone the repository:

```
$ git clone https://cgit.heqnx.com/go-linkfinder
$ cd go-linkfinder
```

- Install dependencies:

```
$ go mod tidy
```

- Build for all platforms:

```
$ make all
```

- Binaries will be generated in the build/ directory for Linux, Windows, and macOS; alternatively, build for a specific platform:

```
$ make linux-amd64
$ make windows-amd64
$ make darwin-arm64
```

- (Optional) Run directly with Go:

```
$ go run main.go -depth <depth> -delay <delay> -header <Header: Value> -input <url/file> -timeout <timeout> -user-agent <ua>
```

## Usage

### Command-Line Flags

```
Usage of ./go-linkfinder-linux-amd64:
  -depth int
        recursion depth for same-domain links (0 disables crawling) (default 0)
  -delay int
        delay between requests in seconds when crawling (only applies if depth > 0) (default 4)
  -header value
        add HTTP header to request (can be repeated, e.g. -header "Authorization: Bearer token")
  -input string
        url or file path (required)
  -timeout int
        timeout for HTTP requests in seconds (default 10)
  -user-agent string
        set User-Agent header (default "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.10 Safari/605.1.1")
```

## Examples

### Crawl a single URL without recursion

```
$ ./go-linkfinder -input https://example.com -depth 0
```

- Extracts and prints all links found on the initial page only.

### Recursively crawl same-domain links with delay

```
$ ./go-linkfinder -input https://example.com -depth 2 -delay 5
```

- Crawls links up to 2 levels deep within the same domain.

- Waits 5 seconds between each HTTP request to avoid server overload.

### Use Custom Headers and User-Agent

```
$ ./go-linkfinder -input https://example.com -depth 1 -header "Authorization: Bearer mytoken" -user-agent "CustomAgent/1.0"
```

## License

This project is licensed under the MIT License. See the LICENSE file for details.

## Disclaimer

`go-linkfinder` is provided "as is" without warranty. The author and contributors are not liable for any damages or legal consequences arising from its use. Use responsibly and only in authorized environments.