Introduction
Extending the Virtual Tape Library (VTL) software capability to export the data on a virtual tape to a file or a set of smaller files, the target files can be on cloud storage. Virtual tapes can therefore be migrated to the cloud. This feature also can be useful for migrating data on physical tapes to the cloud by importing the physical tape to a virtual tapes and then exporting (migrating) the virtual tapes to the cloud.
Tape migration to cloud
rclone install and setup
The VTL requires that rclone be setup to upload files to and download files from the cloud. rclone is available free of cost from https://rclone.org/ and supports all major cloud storage providers. Visit https://rclone.org/downloads/ for setup instructions .
After installing rclone, a rclone remote can be setup by running the command
rclone config
Once a rclone remote is setup a file can be copied to a folder on the cloud by the following simple command
rclone copy {file} {remote}/{dest directory}/{remote file}
For example
rclone copy foo myremote:dir1/bar
To list the file copied
rclone ls foo myremote:dir1/bar
150 bar
In the above example myremote is a remote endpoint setup for a Google drive account and upon completion of the rclone copy command we will have a folder name dir1 in the drive with a file named bar in dir1
VTL configuration
Presently a migration job to the cloud can be triggered by moving the virtual tape cartridge to a free import/export port (mailslot) of the library. The vcartridge is first moved to a virtual vault and the migration job is then started. The steps are below
Download cloudmigrate.sh and copy the script to /quadstorvtl/scripts/
cd /quadstorvtl/scripts wget https://www.quadstor.com/vtlnext/cloudmigrate.sh chmod +x cloudmigrate.sh touch /quadstorvtl/etc/quadstor.conf touch /quadstorvtl/etc/rclone.conf cp /quadstorvtl/scripts/helpers/rcloneops.sh /quadstorvtl/scripts/ cp /quadstorvtl/scripts/helpers/rclonecopy.sh /quadstorvtl/scripts/ chmod +x /quadstorvtl/scripts/rclone*.sh
Edit /quadstorvtl/etc/quadstor.conf and add the following line
PostVaultCmd=/quadstorvtl/scripts/cloudmigrate.sh
The run the following command
/quadstorvtl/bin/vtconfig --reload
Setup rclone job parameters
Following is an example /quadstorvtl/etc/rclone.conf
remotename=gdrive: rcloneflags="--config /root/.config/rclone/rclone.conf" chunksize="10240m" maxdeleteretries=50 firstuploadsleep=0 cleanupsleep=5 remotecleanup=1 skipmd5check=0 createspare=0 deleteexport=0 deleteimport=0 enckey=key.pem
remotename
remotename is the name of the remote specified during rclone config. Note that a rclone copy will be issued as
rclone copy {remotename}/{vcartname}-{dateinfo}/{filename}
For example to export a file CCC001L4.img.000001 to a folder CCC001L4-20200323_163122 in remote 'gdrive' which is a Google drive the remotename will have to be specified as 'gdrive:' and the copy will be issued as
rclone copy CCC001L4.img.000001 gdrive:/CCC001L4-20200323_163122/CCC001L4.img.000001
A remotename for a Google cloud would be something like gcloud:vtlremote1 (remote:bucketname) and the copy would then be
rclone copy CCC001L4.img.000001 gcloud:vtlremote1/CCC001L4-20200323_163122/CCC001L4.img.000001
chunksize is the max size of file in the vcartridge folder. For example a vcartridge with 100GiB of data can be split into 10 chunk files with a chunksize of "10240m" (10GiB). The default is 10240m. Chunk size will have to be specified in m (MB). For example 1024m equals 1GiB
enckey
A keyfile name which will contain a RSA private key. The keyfile will have to be present in /quadstorvtl/keystore directory.. The keyfile can be generated by
cd /quadstorvtl/keystore openssl genrsa -out key.pem 2048
enckey is optional. When a keyfile is specified, the data is encrypted before writing to the chunk file. When this parameter is not specified, the data is written as-is to the the chunk file.
compressdata
When compressdata is set to 1 the data is first compressed prior to writing to the chunk file. compressdata set to 0 disables compression. The default value is 1. When compression is enabled and enckey is specified, the data is first compressed and then encrypted.
deleteexport
Default value is zero. If deleteexport is set to 1 after a vcartridge has been migrated (exported) to the cloud, the local vcartridge is deleted. This will reclaim the space used by the vcartridge
deleteimport
A vcartridge can be imported back from the cloud from the HTML GUI "Job Management" -> "File Export" page. All migration/export jobs are listed and clicking on the "Import" link for a vcartridge job will import the vcartridge back into the VTL. This newly import vcartridge can be ejected again from the VTL without any new changes. In such a case there is no new migration job. deleteimport=1 will delete the vcartridge and reclaim its space, deleteimport=0 will do nothing further and the vcartridge will stay in the virtual vault
createspare
If createspare is set to 1, then a new vcartridge with a generated label will be created in the VTL of the vcartridge being migrated. This is will allow for a constant set of blank vcartridges in the VTL without having to create a new one manually every time a vcartridge is migrated. This is useful when deleteexport=1 since deleteexport when set will delete and reclaim the space used by the vcartridge being migrated
cachedir
During a migration job, data is first written to a temporary file in cachedir upto a maximum of the chunksize. The rclone copy operation is started for the chunkfile and in the meantime another chunkfile is filled up in the cachedir. This allows for multiple chunkfiles to be uploaded at the same time. cachedir by default is /quadstorvtl/migratetmp and this can be changed by setting cachedir=<new location>
skipmd5check
When set to 0, an rclone md5sum of the uploaded object is generated and compared with the computed md5sum. The md5 sums should match for a successful upload. This can be skipped by skipmd5check=1. The default is 0.
Setup VTL job parameters
Following is a sample /quadstorvtl/etc/quadstor.conf file
PostVaultCmd=/quadstorvtl/scripts/cloudmigrate.sh MaxThreadsPerJob=4 MaxRunningJobs=2 ExportRetryCodes=1,256,126 MaxRetryCount=3 CmdRetryPause=1
PostVaultCmd
This parameter will contain the path to the cloudmigrate.sh script. Ensure that the script has execute permission by chmod +x <path script>
After any change to PostVaultCmd value run the following command for the daemon to read the new value
/quadstorvtl/bin/vtconfig --reload
MaxThreadsPerJob
Maximum number of migration or import threads. In other words this value specifies the number of parallel rclone copy jobs
MaxRunningJobs
Maximum number of parallel migration or import jobs.
ExportRetryCodes
These are exit code generated by the upload scripts which give an indication to the daemon that a failed rclone copy should be retried.
MaxRetryCount
The number of times a rclone copy should be attempted
CmdRetryPause
The number of seconds to pause before retrying a rclone copy
Job Management
Jobs can be managed from the HTML GUI "Job Management" -> "Import/Export" page. All running, completed and cancelled jobs are listed in this page.
Click on "Clear Jobs" to clear completed, cancelled or error job information. This option is useful when there are many such jobs and each job information can use upto 32K bytes of application memory.
Jobs can also be managed from the command line
Listing jobs
/quadstorvtl/bin/impexp -l
Or for a longer format
/quadstorvtl/bin/impexp -L
Cancelling a job
/quadstorvtl/bin/impexp -r -j {job id}
Clear completed jobs
/quadstorvtl/bin/impexp --clear