Command line args

Usage of ./s3sync-service:
  -config string
        Path to the config.yml (default "config.yml")
  -configmap string
        K8s configmap in the format namespace/configmap, if set config is ignored and s3sync-service will read and watch for changes in the specified configmap
  -http-port string
        Port for internal HTTP server, 0 to disable (default "8090")
  -metrics-path string
        Prometheus exporter path (default "/metrics")
  -metrics-port string
        Prometheus exporter port, 0 to disable the exporter (default "9350")

Example configuration

access_key: AKIAI44QH8DHBEXAMPLE
secret_access_key: je7MtGbClwBF/2Zp9Utk/h3yCo8nvbEXAMPLEKEY
aws_region: us-east-1
loglevel: info
upload_workers: 5
checksum_workers: 5
watch_interval: 30s
s3_ops_retries: 3
sites:
- name: my-precious-data
  local_path: /some/local/path
  bucket: my-s3-backups
  bucket_path: precious-data
  storage_class: STANDARD_IA
  retire_deleted: true
  exclusions:
  - .[Dd][Ss]_[Ss]tore
# Another site, uses the same S3 bucket but different bucket_path and STANDARD storage_class
- local_path: /some/other/local/path
  bucket: my-s3-backups
  bucket_path: less-precious-data
  exclusions:
  - .[Dd][Ss]_[Ss]tore
  - .[Aa]pple[Dd]ouble
  - "[Tt]humbs.db"
# Third site, uses different S3 bucket, region, dedicated access key
# and overrides global watch_interval and s3_ops_retries
- local_path: /some/other/local/path
  bucket: my-synced-data
  bucket_region: eu-central-1
  access_key: BKIAI44QH8DHBEXAMPLE
  secret_access_key: fe7MtGbClwBF/2Zp9Utk/h3yCo8nvbEXAMPLEKEY
  retire_deleted: true
  watch_interval: 1s
  s3_ops_retries: 10
  exclusions:
  - logs

Global configuration options

Variable Description Default Required
access_key Global AWS Access Key n/a no
secret_access_key Global AWS Secret Access Key n/a no
aws_region AWS region n/a no
loglevel Logging level, valid options are - trace, debug, info, warn, error, fatal, panic. With log level set to trace logger will output everything, with debug everything apart from trace and so on. info no
upload_queue_buffer Number of elements in the upload queue waiting for processing, might improve performance, however, increases memory usage 0 no
checksum_workers Number of checksum workers for the service 2 in k8s, otherwise <CPU cores> * 2 no
upload_workers Number of upload workers for the service 10 no
watch_interval Interval for file system watcher in format of number and a unit suffix. Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h". 1000ms no
s3_ops_retries Number of retries for upload and delete operations 5 no

Site configuration options

Variable Description Default Required
name Human friendly site name bucket/bucket_path no
local_path Absolute path on local file system to be synced with S3 n/a yes
endpoint An optional endpoint URL (hostname only or fully qualified URI) that overrides the default generated endpoint for a client. May also become handy when using S3 compatible storage providers. n/a no
bucket S3 bucket name n/a yes
bucket_path S3 path prefix n/a no
bucket_region S3 bucket region global.aws_region no
retire_deleted Remove files from S3 which do not exist locally false no
storage_class S3 storage class STANDARD no
access_key Site AWS Access Key global.access_key no
secret_access_key Site AWS Secret Access Key global.secret_access_key no
watch_interval Interval for file system watcher in format of number and a unit suffix. Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h". global.watch_interval no
exclusions List of regex filters for exclusions, please consult with Go regex syntax for more details n/a no
s3_ops_retries Number of retries for upload and delete operations global.s3_ops_retries no