#Linux Use grep to find what you need

“grep” is a wonderful tool to find things in long lists of data. I’ll share an example:

I want to use a specific option in rsync, but the manpage is rather long:

rsync --help,

rsync version 3.0.9 protocol version 30
Copyright (C) 1996-2011 by Andrew Tridgell, Wayne Davison, and others.
Web site: http://rsync.samba.org/
Capabilities:
64-bit files, 64-bit inums, 32-bit timestamps, 64-bit long ints,
socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
append, ACLs, xattrs, iconv, symtimes

rsync comes with ABSOLUTELY NO WARRANTY. This is free software, and you
are welcome to redistribute it under certain conditions. See the GNU
General Public Licence for details.

rsync is a file transfer program capable of efficient remote update
via a fast differencing algorithm.

Usage: rsync [OPTION]… SRC [SRC]… DEST
or rsync [OPTION]… SRC [SRC]… [USER@]HOST:DEST
or rsync [OPTION]… SRC [SRC]… [USER@]HOST::DEST
or rsync [OPTION]… SRC [SRC]… rsync://[USER@]HOST[:PORT]/DEST
or rsync [OPTION]… [USER@]HOST:SRC [DEST]
or rsync [OPTION]… [USER@]HOST::SRC [DEST]
or rsync [OPTION]… rsync://[USER@]HOST[:PORT]/SRC [DEST]
The ‘:’ usages connect via remote shell, while ‘::’ & ‘rsync://’ usages connect
to an rsync daemon, and require SRC or DEST to start with a module name.

Options
-v, –verbose increase verbosity
-q, –quiet suppress non-error messages
–no-motd suppress daemon-mode MOTD (see manpage caveat)
-c, –checksum skip based on checksum, not mod-time & size
-a, –archive archive mode; equals -rlptgoD (no -H,-A,-X)
–no-OPTION turn off an implied OPTION (e.g. –no-D)
-r, –recursive recurse into directories
-R, –relative use relative path names
–no-implied-dirs don’t send implied dirs with –relative
-b, –backup make backups (see –suffix & –backup-dir)
–backup-dir=DIR make backups into hierarchy based in DIR
–suffix=SUFFIX set backup suffix (default ~ w/o –backup-dir)
-u, –update skip files that are newer on the receiver
–inplace update destination files in-place (SEE MAN PAGE)
–append append data onto shorter files
–append-verify like –append, but with old data in file checksum
-d, –dirs transfer directories without recursing
-l, –links copy symlinks as symlinks
-L, –copy-links transform symlink into referent file/dir
–copy-unsafe-links only “unsafe” symlinks are transformed
–safe-links ignore symlinks that point outside the source tree
-k, –copy-dirlinks transform symlink to a dir into referent dir
-K, –keep-dirlinks treat symlinked dir on receiver as dir
-H, –hard-links preserve hard links
-p, –perms preserve permissions
-E, –executability preserve the file’s executability
–chmod=CHMOD affect file and/or directory permissions
-A, –acls preserve ACLs (implies –perms)
-X, –xattrs preserve extended attributes
-o, –owner preserve owner (super-user only)
-g, –group preserve group
–devices preserve device files (super-user only)
–specials preserve special files
-D same as –devices –specials
-t, –times preserve modification times
-O, –omit-dir-times omit directories from –times
–super receiver attempts super-user activities
–fake-super store/recover privileged attrs using xattrs
-S, –sparse handle sparse files efficiently
-n, –dry-run perform a trial run with no changes made
-W, –whole-file copy files whole (without delta-xfer algorithm)
-x, –one-file-system don’t cross filesystem boundaries
-B, –block-size=SIZE force a fixed checksum block-size
-e, –rsh=COMMAND specify the remote shell to use
–rsync-path=PROGRAM specify the rsync to run on the remote machine
–existing skip creating new files on receiver
–ignore-existing skip updating files that already exist on receiver
–remove-source-files sender removes synchronized files (non-dirs)
–del an alias for –delete-during
–delete delete extraneous files from destination dirs
–delete-before receiver deletes before transfer, not during
–delete-during receiver deletes during the transfer
–delete-delay find deletions during, delete after
–delete-after receiver deletes after transfer, not during
–delete-excluded also delete excluded files from destination dirs
–ignore-errors delete even if there are I/O errors
–force force deletion of directories even if not empty
–max-delete=NUM don’t delete more than NUM files
–max-size=SIZE don’t transfer any file larger than SIZE
–min-size=SIZE don’t transfer any file smaller than SIZE
–partial keep partially transferred files
–partial-dir=DIR put a partially transferred file into DIR
–delay-updates put all updated files into place at transfer’s end
-m, –prune-empty-dirs prune empty directory chains from the file-list
–numeric-ids don’t map uid/gid values by user/group name
–timeout=SECONDS set I/O timeout in seconds
–contimeout=SECONDS set daemon connection timeout in seconds
-I, –ignore-times don’t skip files that match in size and mod-time
–size-only skip files that match in size
–modify-window=NUM compare mod-times with reduced accuracy
-T, –temp-dir=DIR create temporary files in directory DIR
-y, –fuzzy find similar file for basis if no dest file
–compare-dest=DIR also compare destination files relative to DIR
–copy-dest=DIR … and include copies of unchanged files
–link-dest=DIR hardlink to files in DIR when unchanged
-z, –compress compress file data during the transfer
–compress-level=NUM explicitly set compression level
–skip-compress=LIST skip compressing files with a suffix in LIST
-C, –cvs-exclude auto-ignore files the same way CVS does
-f, –filter=RULE add a file-filtering RULE
-F same as –filter=’dir-merge /.rsync-filter’
repeated: –filter=’- .rsync-filter’
–exclude=PATTERN exclude files matching PATTERN
–exclude-from=FILE read exclude patterns from FILE
–include=PATTERN don’t exclude files matching PATTERN
–include-from=FILE read include patterns from FILE
–files-from=FILE read list of source-file names from FILE
-0, –from0 all *-from/filter files are delimited by 0s
-s, –protect-args no space-splitting; only wildcard special-chars
–address=ADDRESS bind address for outgoing socket to daemon
–port=PORT specify double-colon alternate port number
–sockopts=OPTIONS specify custom TCP options
–blocking-io use blocking I/O for the remote shell
–stats give some file-transfer stats
-8, –8-bit-output leave high-bit chars unescaped in output
-h, –human-readable output numbers in a human-readable format
–progress show progress during transfer
-P same as –partial –progress
-i, –itemize-changes output a change-summary for all updates
–out-format=FORMAT output updates using the specified FORMAT
–log-file=FILE log what we’re doing to the specified FILE
–log-file-format=FMT log updates using the specified FMT
–password-file=FILE read daemon-access password from FILE
–list-only list the files instead of copying them
–bwlimit=KBPS limit I/O bandwidth; KBytes per second
–write-batch=FILE write a batched update to FILE
–only-write-batch=FILE like –write-batch but w/o updating destination
–read-batch=FILE read a batched update from FILE
–protocol=NUM force an older protocol version to be used
–iconv=CONVERT_SPEC request charset conversion of filenames
-4, –ipv4 prefer IPv4
-6, –ipv6 prefer IPv6
–version print version number
(-h) –help show this help (-h is –help only if used alone)

Use “rsync –daemon –help” to see the daemon-mode command-line options.
Please see the rsync(1) and rsyncd.conf(5) man pages for full documentation.
See http://rsync.samba.org/ for updates, bug reports, and answers

********************************************
I only want to see options that let me delete something before/after/during sync on the target, but I am a lazy sysadmin and the list above is way to long. So:

rsync --help | grep delete

–del an alias for –delete-during
–delete delete extraneous files from destination dirs
–delete-before receiver deletes before transfer, not during
–delete-during receiver deletes during the transfer
–delete-delay find deletions during, delete after
–delete-after receiver deletes after transfer, not during
–delete-excluded also delete excluded files from destination dirs
–ignore-errors delete even if there are I/O errors
–max-delete=NUM don’t delete more than NUM files

And I am happy 🙂

Two small updates :-)

About my Trachycarpus Fortunei palm tree:

image
Blossoms!

About my personal life:

image
Blossoms too <3 🙂 So at the moment I am busy with other things, but some technical stuff is upcoming too (hint: seafile cloud server on RPI).

2705746 / 2416 :-)

Monitoring cafeïne levels with @zabbix

Can somebody bring me some coffee? Please?

image

🙂

Driving to work, at 5472 km/h …

Have fun watching ….

HOWTO: Prune backups made with Relax and Recover #rear #linux

I am backing up my Linux machines bare-metal with ReaR, in the form of weekly full backups and incrementals. ReaR does a great job but doesn’t cleanup old backups by itself, so if you do nothing you’re filing up your backup server sooner or later.

As I am lazy Linux sysadmin I don’t want to cleanup leftovers manualy, so I automated it 🙂 Here is how:

In the picture below you see what happens if you let things run it’s course:

files_lamp02

I have got 11 backup archives of which I only need 4 (today) to recover when needed. The archives with an “F” in their name are full backups, the other ones (with “I”) are incrementals. There are also two important files here, basebackup.txt and timestamp.txt. Those two files actually do sort of the same: tell the system when the last full backup was made. ReaR needs this for restoring the system, using the correct files. I only need timestamp.txt for my cleanup job, but I could also use basebackup.txt for them. What’s in those files is not important to me, I use the time and date they where created or modified. Today is march 17, so every archive created before the last full backup (march 13 in this case) may be deleted.

To find out what files can be deleted you can issue the following command in the terminal:

find /media/nfs/lamp02/*tar.gz -type f ! -newer /media/nfs/lamp02/timestamp.txt

(make sure to adjust the path to your own situation!)

Output:

term_lamp02

To delete them issue the following command:

find /media/nfs/lamp02/*tar.gz -type f ! -newer /media/nfs/lamp02/timestamp.txt | xargs rm -f

(Again, adjust your paths!)

All files created before the last full backup will be gone, keeping your backup server clean 🙂
The only thing you have to do now is create a proper cronjob to automate this. Be sure the command runs AFTER the backups are complete!

For the best results you can do this daily. If you feel you want to keep your files longer, maybe a month, you can tweak around to accomplish that. Maybe I will update this posting with my own solution for that, although in my case I do not need it.

Happy Linux’ing!

(Edit: this posting is now in the official rear user documentation, which is kinda cool)

Stel, je bent uit ’68 @geenstijl

Je hebt een normale baan, en je vervoermiddel is toevallig een Mercedes … (/me)
Dan ben je volgens GeenStijl / reaguurder:

image

image

🙂

Intel outside :-) (drone video)

My icy Trachycarpus Fortunei :-)

After being ill for over a week I was out of the house last afternoon to inspect my Trachycarpus Fortunei palm tree, which had to endure snow, ice, and freezing rain lately.

The result of all that harsh weather is somewhat awkward ..

image

image

image

image

image

image

image

Pretty cool, and it will survive 🙂

Half of #debian died.

Hey Ian,

You shouldn’t have done this. Barriers are never too high.
But since this happened, there is no way back. I will never know your reasons.
The only thing left for me to do is to keep using Debian, and think about you every time I login.

Thanks for everything you brought to the world. Debian, open source, and you, in my memories for ever.

Your name will always be there, in deb-ian.

RIP.
You ruled man, we will miss you very bad.

image