File Splitting with split and Merging with cat on Linux

When network transfer limits require sending large files, it is common to split the file into smaller pieces on a Linux system, transfer them, and then merge them back together.

File Splitting – split

Command syntax

-a: # specify suffix length (default 2: aa, ab...)
-d: # use numeric suffixes instead of alphabetic
-l: # split by number of lines (default 1000 lines)
-b: # binary split mode (supports units: k/m)
-C: # split by size while preserving line integrity

General form: split [-a] [-d] [-l <lines>] [-b <bytes>] [-C <bytes>] [input_file] [output_prefix]

Usage examples

# line-based split
$ split -l 300000 users.sql /data/users_

# numeric suffixes
$ split -d -l 300000 users.sql /data/users_

# split by byte size
$ split -d -b 100m users.sql /data/users_

Help information

# help output
$ split --help
Usage: split [OPTION]... [FILE [PREFIX]]
Output pieces of FILE to PREFIXaa, PREFIXab, ...; default size is 1000 lines, default PREFIX is 'x'.
-a, --suffix-length=N   generate suffixes of length N (default 2)
-b, --bytes=SIZE        put SIZE bytes per output file
-C, --line-bytes=SIZE  put at most SIZE bytes of records per output file
-d, --numeric-suffixes  use numeric suffixes starting at 0
-e, --elide-empty-files do not generate empty output files
--filter=COMMAND        write to shell COMMAND; file name is $FILE
-l, --lines=NUMBER     put NUMBER lines/records per output file
-n, --number=CHUNKS    generate CHUNKS output files
-t, --separator=SEP    use SEP instead of newline as the record separator
-u, --unbuffered       copy input to output without buffering
--verbose               print a diagnostic before each output file is opened
--help                 display this help and exit
--version              output version information and exit

File Merging – cat

Using the cat command on Linux makes it easy to concatenate multiple small files back into a single large file.

Command syntax

-n: # show line numbers
-e: # display $ at end of each line
-t: # show TAB characters (^I)
cat [-n] [-e] [-t] [output_file]

Usage example

# merge files
$ cat /data/users_* > users.sql

Help information

# help output
$ cat --help
Usage: cat [OPTION]... [FILE]...
Concatenate FILE(s) to standard output.
-A, --show-all          equivalent to -vET
-b, --number-nonblank   number nonempty output lines, overrides -n
-e, --show-ends         display $ at end of each line
-n, --number            number all output lines
-s, --squeeze-blank    suppress repeated empty output lines
-t, --show-tabs         display TAB characters as ^I
-v, --show-nonprinting  use ^ and M- notation
--help                  display this help and exit
--version               output version information and exit

For more details, refer to the GNU coreutils online documentation or the local info pages.