Command line args
Usage of ./s3sync-service:
-config string
Path to the config.yml (default "config.yml")
-configmap string
K8s configmap in the format namespace/configmap, if set config is ignored and s3sync-service will read and watch for changes in the specified configmap
-http-port string
Port for internal HTTP server, 0 to disable (default "8090")
-metrics-path string
Prometheus exporter path (default "/metrics")
-metrics-port string
Prometheus exporter port, 0 to disable the exporter (default "9350")
Example configuration
access_key: AKIAI44QH8DHBEXAMPLE
secret_access_key: je7MtGbClwBF/2Zp9Utk/h3yCo8nvbEXAMPLEKEY
aws_region: us-east-1
loglevel: info
upload_workers: 5
checksum_workers: 5
watch_interval: 30s
s3_ops_retries: 3
sites:
- name: my-precious-data
local_path: /some/local/path
bucket: my-s3-backups
bucket_path: precious-data
storage_class: STANDARD_IA
retire_deleted: true
exclusions:
- .[Dd][Ss]_[Ss]tore
# Another site, uses the same S3 bucket but different bucket_path and STANDARD storage_class
- local_path: /some/other/local/path
bucket: my-s3-backups
bucket_path: less-precious-data
exclusions:
- .[Dd][Ss]_[Ss]tore
- .[Aa]pple[Dd]ouble
- "[Tt]humbs.db"
# Third site, uses different S3 bucket, region, dedicated access key
# and overrides global watch_interval and s3_ops_retries
- local_path: /some/other/local/path
bucket: my-synced-data
bucket_region: eu-central-1
access_key: BKIAI44QH8DHBEXAMPLE
secret_access_key: fe7MtGbClwBF/2Zp9Utk/h3yCo8nvbEXAMPLEKEY
retire_deleted: true
watch_interval: 1s
s3_ops_retries: 10
exclusions:
- logs
inclusions:
- "subpath/*"
Global configuration options
Variable |
Description |
Default |
Required |
access_key |
Global AWS Access Key |
n/a |
no |
secret_access_key |
Global AWS Secret Access Key |
n/a |
no |
aws_region |
AWS region |
n/a |
no |
loglevel |
Logging level, valid options are - trace , debug , info , warn , error , fatal , panic . With log level set to trace logger will output everything, with debug everything apart from trace and so on. |
info |
no |
upload_queue_buffer |
Number of elements in the upload queue waiting for processing, might improve performance, however, increases memory usage |
0 |
no |
checksum_workers |
Number of checksum workers for the service |
2 in k8s, otherwise <CPU cores> * 2 |
no |
upload_workers |
Number of upload workers for the service |
10 |
no |
watch_interval |
Interval for file system watcher in format of number and a unit suffix. Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h". |
1000ms |
no |
s3_ops_retries |
Number of retries for upload and delete operations |
5 |
no |
Site configuration options
Variable |
Description |
Default |
Required |
name |
Human friendly site name |
bucket/bucket_path |
no |
local_path |
Absolute path on local file system to be synced with S3 |
n/a |
yes |
endpoint |
An optional endpoint URL (hostname only or fully qualified URI) that overrides the default generated endpoint for a client. May also become handy when using S3 compatible storage providers. |
n/a |
no |
bucket |
S3 bucket name |
n/a |
yes |
bucket_path |
S3 path prefix |
n/a |
no |
bucket_region |
S3 bucket region |
global.aws_region |
no |
retire_deleted |
Remove files from S3 which do not exist locally |
false |
no |
storage_class |
S3 storage class |
STANDARD |
no |
access_key |
Site AWS Access Key |
global.access_key |
no |
secret_access_key |
Site AWS Secret Access Key |
global.secret_access_key |
no |
watch_interval |
Interval for file system watcher in format of number and a unit suffix. Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h". |
global.watch_interval |
no |
exclusions |
List of regex filters for exclusions, please consult with Go regex syntax for more details |
n/a |
no |
inclusions |
List of regex filters for inclusions, please consult with Go regex syntax for more details, exclusions have higher priority. |
[".*"] |
no |
s3_ops_retries |
Number of retries for upload and delete operations |
global.s3_ops_retries |
no |