Rsync is a powerful tool often utilized for its proficiency in file synchronization and transfer. Widely adopted in various computing environments, rsync excels in efficiently mirroring data across multiple platforms and minimizing data transfer by sending only changes in files. This utility is not only pivotal for maintaining backups but also ensures that the copies are consistent and up-to-date.
Today’s article is designed to guide you through the steps of setting up a basic yet effective backup system using rsync. By the end of this guide, you’ll have the knowledge to implement a reliable backup solution that can be customized to fit your specific needs.
Daily
The daily backup captures all of the changes for the day, giving you a backup with the fastest cadence.
#!/bin/bash# daily-backup.sh## Basic copy script to perform daily backups of the home dirUSER=$(whoami)HOST=$(hostname)
rsync -aAX\--delete\--rsync-path="mkdir -p /path/to/backups/$HOST/daily/ && rsync"\--exclude-from=/home/$USER/.local/bin/rsync-homedir-excludes.txt \
/home/$USER/ backup-user@backup-server:/path/to/backups/$HOST/daily/
Using whoami and hostname we can generalise this script so that you can reuse it between all of your machines.
The rsync-homedir-excludes.txt file allows you to define files that you’re not interested in backing up.
The switches that we’re sending into rsync are quite significant:
-a puts rsync into archive mode, preserving file structures, and links
-A preserves the ACLs so all of our permissions are preserved
-X any extra attributes that are stored by your file system will be preserved
--delete will delete files in the destination that are no longer present in the source, making the backup a true mirror
Weekly
The weekly backup is very similar to the daily backup. It’ll target different folders, and will have a different execution cadence.
#!/bin/bash# weekly-backup.sh## Basic copy script to perform weekly backups of the home dirUSER=$(whoami)HOST=$(hostname)
rsync -aAX\--delete\--rsync-path="mkdir -p /path/to/backups/$HOST/weekly/ && rsync"\--exclude-from=/home/$USER/.local/bin/rsync-homedir-excludes.txt \
/home/$USER/ backup-user@backup-server:/path/to/backups/$HOST/weekly/
There isn’t much difference here. Just writing to the /weekly folder.
Monthly
The longest cadence that we have is a monthly process which will archive the current state into an archive, and date the file for later use potentially.
There were some important configurations that I needed to put together in order to get this unit to perform correctly.
Problems
After getting everything setup, I had two 4TB BarraCuda drives plugged in ready to go.
I have this system running as a RAID 1 so that I have a mirror of my data.
After starting to transfer data onto the device, I noticed that copy jobs would grind to a halt; as well as the error log being full of the following:
Looking at some discussions I can see that a lot of people were having success with blacklisting the uas driver that the vendor provides, failing back to the standard driver from linux.
So, I did that too.
To do that, you need to pass the IDs as quirks into the blacklist request. To get those IDs, you need to look at the output from lsusb:
$ lsusb
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 003 Device 002: ID 174c:2074 ASMedia Technology Inc. ASM1074 High-Speed hub
Bus 003 Device 003: ID 046d:c548 Logitech, Inc. Logi Bolt Receiver
Bus 003 Device 004: ID 8087:0033 Intel Corp. AX211 Bluetooth
Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 004 Device 002: ID 174c:3074 ASMedia Technology Inc. ASM1074 SuperSpeed hub
Bus 004 Device 003: ID 0480:a006 Toshiba America Inc UAS Controller
The Toshiba, and ASMedia items were of interest to me.
PAM (Pluggable Authentication Modules) is a flexible mechanism for authenticating users, which has become a fundamental part of system security on Linux and other Unix-like operating systems. PAM abstracts a range of authentication tasks into modular plugins that can be tailored to suit the needs of system administrators, providing a way to develop programs that are independent of authentication scheme specifics. This modularity not only enhances security but also simplifies the management of user authentication.
In today’s guide, we will delve into the process of creating a program that interacts with the Linux passwd system using PAM. By the end of this article, you’ll have a clear understanding of how to harness PAM’s capabilities to authenticate users in your own applications, ensuring secure and efficient access control.
Architecture
PAM (Pluggable Authentication Modules) employ a unique architecture that separates the specific implementation of authentication methods from the application programming interface (API).
At the heart of this design is the concept of “conversations,” a mechanism that facilitates communication between the application and the authentication modules.
This conversation-based model allows PAM to present a uniform interface to the application, regardless of the underlying authentication process.
As a result, developers can integrate various authentication technologies into their applications without having to tailor their code to each method.
Instead, they rely on PAM to handle the specifics through configurable modules, each responsible for a different aspect of the authentication process.
This abstraction not only simplifies development but also enhances the flexibility and scalability of security systems, accommodating a wide range of authentication schemes with minimal changes to core application code.
Conversation
The conversation function is the callback that PAM uses for interactivity. In this code snippet, we’re using the password of "password" to stuff into the response to use. You could use the
conversation opportunity to ask for input from the user.
intconverse(intnum_msg,conststructpam_message**msg,structpam_response**resp,void*appdata_ptr){*resp=NULL;if(num_msg<=0||num_msg>PAM_MAX_NUM_MSG){returnPAM_CONV_ERR;}structpam_response*response=malloc(num_msg*sizeof(structpam_response));if(response==NULL){returnPAM_BUF_ERR;}for(inti=0;i<num_msg;++i){if(msg[i]->msg_style==PAM_PROMPT_ECHO_OFF){// supply the password directly hereconstchar*password="password";response[i].resp=strdup(password);}else{// Handle other message styles if neededresponse[i].resp=NULL;}response[i].resp_retcode=0;}*resp=response;returnPAM_SUCCESS;}
You can see that there’s a preference asked for here with the message style of PAM_PROMPT_ECHO_OFF. This would be an indicator back to the user interface to not echo the user’s keystrokes
to the interface as they type.
Test Harness
We can now use this conversation function with the pam library:
The pam_start function begins the authentication conversation. The first parameter supplied
is the service, and in this case it’s set to "login". PAM uses this value and looks for
a configuration file named /etc/pam.d/login. This is pretty standard on any system. This
particular service is for the “Shadow ‘login’ service.
The username variable should contain the name of a registered user.
Our conversation function converse is supplied to pam_start via the conv variable.
We then use pam_authenticate to preform the conversation, and pam_end will do any clean up for us.
A full example
#include<security/pam_appl.h>
#include<security/pam_misc.h>
#include<stdio.h>intconverse(intnum_msg,conststructpam_message**msg,structpam_response**resp,void*appdata_ptr){*resp=NULL;if(num_msg<=0||num_msg>PAM_MAX_NUM_MSG){returnPAM_CONV_ERR;}structpam_response*response=malloc(num_msg*sizeof(structpam_response));if(response==NULL){returnPAM_BUF_ERR;}for(inti=0;i<num_msg;++i){if(msg[i]->msg_style==PAM_PROMPT_ECHO_OFF){// supply the password directly hereconstchar*password="password";response[i].resp=strdup(password);}else{// Handle other message styles if neededresponse[i].resp=NULL;}response[i].resp_retcode=0;}*resp=response;returnPAM_SUCCESS;}intmain(){constchar*username="username";constchar*service="login";pam_handle_t*pamh=NULL;structpam_convconv={converse,NULL};intretval=pam_start(service,username,&conv,&pamh);if(retval!=PAM_SUCCESS){fprintf(stderr,"pam_start failed: %s\n",pam_strerror(pamh,retval));return1;}retval=pam_authenticate(pamh,0);if(retval!=PAM_SUCCESS){fprintf(stderr,"Authentication failed: %s\n",pam_strerror(pamh,retval));return1;}printf("Authentication successful!\n");pam_end(pamh,retval);return0;}
Building
In order to build this test program you need to link with pam and pam_misc.
GMP is a library that will allow you to perform calculations on numbers that extend past the reach of what your standard data sizes can hold. From their website:
GMP is a free library for arbitrary precision arithmetic, operating on signed integers, rational numbers, and floating-point numbers. There is no practical limit to the precision except the ones implied by the available memory in the machine GMP runs on. GMP has a rich set of functions, and the functions have a regular interface.
This means you can embed large math into your programs and will only be limited by the amount of memory on your running system.
In today’s article, we’ll go through a few simple examples on how to use this library.
Getting setup
Get gmp installed locally on your system either by downloading the latest release from their site, or just using your package manager.
sudo apt-get install libgmp10
Building
For any program that will require the gmp library, we’ll need to add the -lgmp switch:
gcc test.c -otest-lgmp
Now, we’re ready to go.
Factorial
As an example implementation, we’ll write a program to calculate the n’th factorial for us. First of all, we’ll implement this using traditional data types offered to us through C, and then we’ll swap this out to get greater numbers.
/* test1.c */#include<stdio.h>
#include<stdlib.h>
#include<assert.h>/** Compute the n'th factorial */intfactorial(intn){inti;intx=1;for(i=1;i<=n;++i){x=x*i;}returnx;}intmain(intargc,char*argv[]){intx;/* check that a number was specified */if(argc<=1){printf("usage: %s <number>\n",argv[0]);return1;}/* convert the input to a number */x=atoi(argv[1]);assert(x>=0);printf("%d\n",factorial(x));return0;}
We build this application not needing the gmp library:
The wheels start to fall off once we want to look at numbers higher than !19.
$ ./test1 19
109641728
$ ./test1 20
-2102132736
We overflowed our integer to where it wrapped into negative numbers. 19 is our limit for traditional data types.
Make the numbers bigger!
Now we can introduce gmp to help us break out of these constraints.
We’ll rewrite the factorial function above to operate on gmp types, and we’ll also convert the body of our program into a input capture function that will parse text into a gmp type for us.
Let’s deal with the input first:
voidatoi_mpz(char*s,mpz_tres){assert(s);intparse_result;/* allocate our number, and set an initial value of 0 */mpz_set_ui(res,0);/* we assume base-10 when parsing */parse_result=mpz_set_str(res,s,10);/* check that parsing was successful */assert(parse_result==0);}
First, you’ll notice that we’re not returning anything here. The mpz_t type is typed as an array, and as such can’t be used as a return. So, we supply it as an output parameter. This pattern will reoccur through these examples.
This function also assumes that res has already had mpz_init run on it, so it’s not magically allocating resources on your behalf.
mpz_set_ui sets the initial state of an mpz_t with a value from an integer (the real world!). mpz_set_str is really doing most of the work for us here, parsing out a string that its given into a mpz_t. The base needs to be supplied.
Now the factorial function will need to change as you’d expect:
/** Compute the n'th factorial */voidfactorial(mpz_tn,mpz_tres){mpz_ti;/* initialize x and set it to n */mpz_set(res,n);/* initialize i to 1 */mpz_init(i);mpz_set_ui(i,1);while(mpz_cmp(i,n)<0){mpz_add_ui(i,i,1);mpz_mul(res,res,i);}/* clean up work vars */mpz_clear(i);}
Again, res is an output parameter with no return value. We do have a “work” variable here, so we set it up and destroy it all within the context of our function so we don’t have a memory leak.
mpz_cmp takes care of the looping for us. We’ve substituted a for loop here for a while loop to accommodate.
Full program!
Now we can use these functions in a program of our own. Here’s a full listing of an application that will give you the factorial of an arbitrary length integer.
/* test2.c */#include<stdio.h>
#include<stdlib.h>
#include<assert.h>#include<gmp.h>/** Compute the n'th factorial */voidfactorial(mpz_tn,mpz_tres){mpz_ti;/* inintialize x and set it to n */mpz_set(res,n);/* initialize i to 1 */mpz_init(i);mpz_set_ui(i,1);while(mpz_cmp(i,n)<0){mpz_add_ui(i,i,1);mpz_mul(res,res,i);}/* clean up work vars */mpz_clear(i);}voidatoi_mpz(char*s,mpz_tres){assert(s);intparse_result;/* allocate our number, and set an initial value of 0 */mpz_set_ui(res,0);/* we assume base-10 when parsing */parse_result=mpz_set_str(res,s,10);/* check that parsing was successful */assert(parse_result==0);}intmain(intargc,char*argv[]){mpz_tx,f;/* check that a number was specified */if(argc<=1){printf("usage: %s <number>\n",argv[0]);return1;}mpz_init(f);mpz_init(x);atoi_mpz(argv[1],x);factorial(x,f);mpz_out_str(stdout,10,f);mpz_clear(x);mpz_clear(f);return0;}
We write the result number here with mpz_out_str. This can redirected to any stream of your choice.
Building this program is just adding the lgmp switch.
gcc test2.c -o test2 -lgmp
And, now you can calculate factorials as high as you like: