Python 3.* is required byte not string to resolve this issue we need to convert the string into UTF-8 then again after encoding we need to convert in to ascii.
token = base64.standard_b64encode(”
samplestring “.encode(‘UTF-8’) + ‘:’.encode(‘UTF-8’) + “samplestring”.encode(‘UTF-8’)).decode(‘ascii’)
Linux commands and their optional parameter are very useful for the user who wants to take advance of the full feature of Linux CLI. In this post, we have discussed the most featured commands are being used extensively in day-to-day activities.
To explore more about commands you can explore all the commands available in using the Linux Inbuild self-help commands (help or man or info) available, Which will give the briefing about the options and uses by precise examples.
The alias command is a way to keep your command /commands in a temporary variable. Unix commands using a shorter name than those that are usually associated with such commands.
Syntax:
alias shortName="your custom command here"
Example:
alias wr=”cd /var/www/html”
APT (Advanced Package Tool) is the command line tool to interact with this packaging system. There is already dpkg commands to manage it. But apt is more friendly way to handle packaging. You can use it to find and install new packages, upgrade packages, clean the packages etc.
Syntax:
sudo apt-get install <package_name>
sudo apt-get remove <package_name>
sudo apt-get purge <package_name>
sudo apt-get clean
sudo apt-get autoclean
sudo apt-get autoremove
Example:
sudo apt-get install pinta
awk command searches files for text containing a pattern. When a line or text matches, awk performs a specific action on that line/text. The Program statement tells awk what operation to do; Program statement consists of a series of “rules” where each rule specifies one pattern to search for, and one action to perform when a particular pattern is found. A regular expression enclosed in slashes (/) is an awk pattern to match every input record whose text belongs to that set.
Syntax:
awk 'pattern {action}' input-file > output-file
Example:
awk '{ print $5 }' table1.txt > output1.txt
Awk is a scripting language used for manipulating data and generating reports.The awk command programming language requires no compiling, and allows the user to use variables, numeric functions, string functions, and logical operators.
Awk is a utility that enables a programmer to write tiny but effective programs in the form of statements that define text patterns that are to be searched for in each line of a document and the action that is to be taken when a match is found within a line. Awk is mostly used for pattern scanning and processing. It searches one or more files to see if they contain lines that matches with the specified patterns and then performs the associated actions.
Awk is abbreviated from the names of the developers – Aho, Weinberger, and Kernighan.
WHAT CAN WE DO WITH AWK ?
1. AWK Operations:
(a) Scans a file line by line
(b) Splits each input line into fields
(c) Compares input line/fields to pattern
(d) Performs action(s) on matched lines
2. Useful For:
(a) Transform data files
(b) Produce formatted reports
3. Programming Constructs:
(a) Format output lines
(b) Arithmetic and string operations
(c) Conditionals and loops
Syntax:
awk options 'selection _criteria {action }' input-file > output-file
Options:
-f program-file : Reads the AWK program source from the file
program-file, instead of from the
first command line argument.
-F fs : Use fs for the input field separator
Sample Commands
Example:
Consider the following text file as the input file for all cases below.
$cat > employee.txt
ajay manager account 45000 sunil clerk account 25000 varun manager sales 50000 amit manager account 47000 tarun peon sales 15000 deepak clerk sales 23000 sunil peon sales 13000 satvik director purchase 80000
1. Default behavior of Awk : By default Awk prints every line of data from the specified file.
$ awk '{print}' employee.txt
Output:
ajay manager account 45000 sunil clerk account 25000 varun manager sales 50000 amit manager account 47000 tarun peon sales 15000 deepak clerk sales 23000 sunil peon sales 13000 satvik director purchase 80000
In the above example, no pattern is given. So the actions are applicable to all the lines. Action print without any argument prints the whole line by default, so it prints all the lines of the file without failure.
2. Print the lines which matches with the given pattern.
$ awk '/manager/ {print}' employee.txt
Output:
ajay manager account 45000 varun manager sales 50000 amit manager account 47000
In the above example, the awk command prints all the line which matches with the ‘manager’.
3. Splitting a Line Into Fields : For each record i.e line, the awk command splits the record delimited by whitespace character by default and stores it in the $n variables. If the line has 4 words, it will be stored in $1, $2, $3 and $4 respectively. Also, $0 represents the whole line.
$ awk '{print $1,$4}' employee.txt
Output:
ajay 45000 sunil 25000 varun 50000 amit 47000 tarun 15000 deepak 23000 sunil 13000 satvik 80000
In the above example, $1 and $4 represents Name and Salary fields respectively.
Built In Variables In Awk
Awk’s built-in variables include the field variables—$1, $2, $3, and so on ($0 is the entire line) — that break a line of text into individual words or pieces called fields.
NR: NR command keeps a current count of the number of input records. Remember that records are usually lines. Awk command performs the pattern/action statements once for each record in a file.
NF: NF command keeps a count of the number of fields within the current input record.
FS: FS command contains the field separator character which is used to divide fields on the input line. The default is “white space”, meaning space and tab characters. FS can be reassigned to another character (typically in BEGIN) to change the field separator.
RS: RS command stores the current record separator character. Since, by default, an input line is the input record, the default record separator character is a newline.
OFS: OFS command stores the output field separator, which separates the fields when Awk prints them. The default is a blank space. Whenever print has several parameters separated with commas, it will print the value of OFS in between each parameter.
ORS: ORS command stores the output record separator, which separates the output lines when Awk prints them. The default is a newline character. print automatically outputs the contents of ORS at the end of whatever it is given to print.
Examples:
Use of NR built-in variables (Display Line Number)
$ awk '{print NR,$0}' employee.txt
Output:
1 ajay manager account 45000 2 sunil clerk account 25000 3 varun manager sales 50000 4 amit manager account 47000 5 tarun peon sales 15000 6 deepak clerk sales 23000 7 sunil peon sales 13000 8 satvik director purchase 80000
In the above example, the awk command with NR prints all the lines along with the line number.
Use of NF built-in variables (Display Last Field)
$ awk '{print $1,$NF}' employee.txt
Output:
ajay 45000 sunil 25000 varun 50000 amit 47000 tarun 15000 deepak 23000 sunil 13000 satvik 80000
In the above example $1 represents Name and $NF represents Salary. We can get the Salary using $NF , where $NF represents last field.
Another use of NR built-in variables (Display Line From 3 to 6)
$ awk 'NR==3, NR==6 {print NR,$0}' employee.txt
Output:
3 varun manager sales 50000 4 amit manager account 47000 5 tarun peon sales 15000 6 deepak clerk sales 23000
More Examples
For the given text file:
$cat > geeksforgeeks.txt A B C Tarun A12 1 Man B6 2 Praveen M42 3
1) To print the first item along with the row number(NR) separated with ” – “ from each line in geeksforgeeks.txt:
$ awk '{print NR "- " $1 }' geeksforgeeks.txt
1 - Tarun 2 – Manav 3 - Praveen
2) To return the second row/item from geeksforgeeks.txt:
$ awk '{print $2}' geeksforgeeks.txt
A12 B6 M42
3) To print any non empty line if present
$ awk 'NF > 0' geeksforgeeks.txt
0
4) To find the length of the longest line present in the file:
$ awk '{ if (length($0) > max) max = length($0) } END { print max }' geeksforgeeks.txt
13
5) To count the lines in a file:
$ awk 'END { print NR }' geeksforgeeks.txt
3
6) Printing lines with more than 10 characters:
$ awk 'length($0) > 10' geeksforgeeks.txt
Tarun A12 1 Praveen M42 3
7) To find/check for any string in any column:
$ awk '{ if($3 == "B6") print $0;}' geeksforgeeks.txt
8) To print the squares of first numbers from 1 to n say 6:
$ awk 'BEGIN { for(i=1;i<=6;i++) print "square of", i, "is",i*i; }'
square of 1 is 1 square of 2 is 4 square of 3 is 9 square of 4 is 16 square of 5 is 25 square of 6 is 36
A portable, fast, open source program that compresses and decompresses files at a high rate, but that does not archive them.
Syntax:
bzip2 option(s) filenames
Example:
bzip2 -z backup.tar
A Unix/Linux command that can read, modify or concatenate text files. The cat command also displays file contents.
Syntax:
cat [OPTION] [FILE]...
Example:
cat /etc/passwd
cat test test1
The cd command changes the current directory in Linux and can conveniently toggle between directories. The Linux cd command is similar to the CD and CHDIR commands in MS-DOS.
Syntax:
cd [-L | -P [-e]] directory
Example:
cd documents/work/accounting
Options
L
Force symbolic links to be followed. In other words, if you tell cd to move into a "directory", which is actually a symbolic link to a directory, it moves into the directory the symbolic link points to.
This option is the default behavior of cd; normally, it will always act as if -L has been specified.
-P
Use the physical directory structure without following symbolic links. In other words, only change into the specified directory if it actually exists as named; symbolic links will not be followed. This option is the opposite of the -L option, and if they are both specified, this option will be ignored.
-e
If the -P option is specified, and the current working directory cannot be determined, this option tells cd to exit with an error. If -P is not specified along with this option, this option has no function.
The chmod command changes the permissions of one or more files. Only the file owner or a privileged user can change the access mode.
Syntax:
chmod [OPTION]... MODE[,MODE]... FILE...
chmod [OPTION]... OCTAL-MODE FILE...
chmod [OPTION]... --reference=RFILE FILE...
[reference]
Reference Class Description
u owner file's owner
g group users who are members of the file's group
o others users who are neither the file's owner nor members of the file's group
a all All three of the above, same as ugo
[operator]
Operator Description
+ Adds the specified modes to the specified classes
- Removes the specified modes from the specified classes
= The modes specified are to be made the exact modes for the specified classes
[mode]
Mode Permission
r Permission to read the file.
w Permission to write (or delete) the file.
x Permission to execute the file, or, in the case of a directory, search it.
[mode]
4 Permission to read the file.
2 Permission to write (or delete) the file.
1 Permission to execute the file, or, in the case of a directory, search it.
0 No Permission
So 7 is the combination of permissions 4+2+1 (read, write, and execute), 5 is 4+0+1(read, no write, and execute), and 4 is 4+0+0 (read, no write, and no execute).
Example:
chmod u=rwx,g=rx,o=r myfile
chmod 754 myfile
The chown prompt changes file or group ownership. It gives admins the option to change ownership of all the objects within a directory tree, as well as the ability to view information on the objects processed.
Syntax:
chown [-c|--changes] [-v|--verbose] [-f|--silent|--quiet] [--dereference] [-h|--no-dereference] [--preserve-root] [--from=currentowner:currentgroup] [--no-preserve-root] [-R|--recursive] [--preserve-root] [-H] [-L] [-P] {new-owner|--reference=ref-file} file ...
| new-owner form | Description |
|---|---|
| user | The name of the user to own the file. In this form, the colon (“:“) and the group is omitted. The owning group is not altered. |
| user:group | The user and group to own the file, separated by a colon, with no spaces in between. |
| :group | The group to own the file. In this form, user is omitted, and the group must be preceded by a colon. |
| user: | If group is omitted, but a colon follows user, the owner is changed to user, and the owning group is changed to the login group of user. |
| : | Specifying a colon with no user or group is accepted, but ownership will not be changed. This form does not cause an error, but changes nothing. |
| Option | Description |
|---|---|
| -c, –changes | Similar to –verbose mode, but only displays information about files that are actually changed. For example: changed ownership of ‘dir/dir1/file1’ from hope:neil to hope:hope |
| -v, –verbose | Display verbose information for every file processed. For example: changed ownership of ‘dir/dir1/file1’ from hope:neil to hope:hope ownership of ‘dir/dir1’ retained as hope:hope |
| -f, –silent, –quiet | Quiet mode. Do not display output. |
| –dereference | Dereference all symbolic links. If file is a symlink, change the owner of the referenced file, not the symlink itself. This is the default behavior. |
| -h, –no-dereference | Never dereference symbolic links. If file is a symlink, change the owner of the symlink rather than the referenced file. |
| –from=currentowner:currentgroup | Change the owner or group of each file only if its current owner or group match currentowner and/or currentgroup. Either may be omitted, in which case a match is not required for the other attribute. |
| –no-preserve-root | Do not treat / (the root directory) in any special way. This is the default behavior. If the –preserve-root option is previously specified in the command, this option will cancel it. |
| –reference=ref-file | Use the owner and group of file ref-file, rather than specifying ownership with new-owner. |
| -R, –recursive | Operate on files and directories recursively. Enter each matching directory, and operate on all its contents. |
Recursive options
The following options modify how a hierarchy is traversed when the -R or –recursiveoption is specified.
| Option | Description |
|---|---|
| –preserve-root | Never operate recursively on the root directory /. If –recursive is not specified, this option has no effect. |
| -H | If a file specified on the command line is a symbolic link to a directory, traverse it and operate on those files and directories as well. |
| -L | Traverse all symbolic links to a directories. |
| -P | Do not traverse any symbolic links; operate on the symlinks themselves. This is the default behavior. |
If more than one of -H, -L, or -P is specified, only the final option takes effect.
Other options
These options display information about the program, and cannot be used with other options or arguments.
| Option | Description |
|---|---|
| –help | Display a brief help message and exit. |
| –version | Display version information and exit. |
Exit status
chown exits with a status of 0 for success. Any other number indicates failed operation.
Example:
sudo chown myuser myfile.txt
sudo chown notme:notmygroup myfile.txt
sudo chown -R myuser:mygroup otherfiles
The cmp utility compares two files of any type and writes the results to the standard output. By default, cmp is silent if the files are the same. If they differ, cmp reports the byte and line number where the first difference occurred.
Syntax:
cmp [OPTION]... FILE1 [FILE2 [SKIP1 [SKIP2]]]
[OPTION]
-b --print-bytes
Print differing bytes.
-i SKIP --ignore-initial=SKIP
Skip the first SKIP bytes of input.
-i SKIP1:SKIP2 --ignore-initial=SKIP1:SKIP2
Skip the first SKIP1 bytes of FILE1 and the first SKIP2 bytes of FILE2.
-l --verbose
Output byte numbers and values of all differing bytes.
-n LIMIT --bytes=LIMIT
Compare at most LIMIT bytes.
-s --quiet --silent
Output nothing; yield exit status only.
-v --version
Output version info.
--help
Output this help.
SKIP1 and SKIP2 are the number of bytes to skip in each file. SKIP values may be followed by the following multiplicative suffixes: kB 1000, K 1024, MB 1,000,000, M 1,048,576, GB 1,000,000,000, G 1,073,741,824, and so on for T, P, E, Z, Y.
If a FILE is '-' or missing, read standard input.
Example:
cmp -b file1.txt file2.txt
cmp -i 10 file1.txt file2.txt
Admins use comm to compare lines common to file1 and file2. The output is in three columns; from left to right: lines unique to file1, lines unique to file2 and lines common in both files.
Syntax:
comm [OPTION]... FILE1 FILE2
Example:
comm file1.txt file2.txt
The cp command copies files and directories. Copies can be made simultaneously to another directory even if the copy is under a different name.
Syntax:
cp [OPTION] Source Destination
cp [OPTION] Source Directory
cp [OPTION] Source-1 Source-2 Source-3 Source-n Directory
Example:
cp Src_file1 Src_file2 Src_file3 Dest_directory
The cpio command copies files into or out of a cpio or tar archive. A tar archive is a file that contains other files, plus information about them, such as their file name, owner, timestamps and access permissions. The archive can be another file on the disk, a magnetic tape or a pipe. It also has three operating modes: copy-out, copy-in and copy-pass. It is also a more efficient alternative to tar.
GNU cpio is a tool for creating and extracting archives, or copying files from one place to another. It handles a number of cpio formats as well as reading and writing tar files.
The following archive formats are supported: binary, old ASCII, new ASCII, CRC, HPUXbinary, HPUX old ASCII, old tar, and POSIX.1 tar. The tar format is provided for compatibility with the tar program. By default, cpio creates binary format archives, for compatibility with older cpio programs. When extracting from archives, cpio automatically recognizes which kind of archive it is reading and can read archives created on machines with a different byte-order.
Copy-Out Mode Syntax:
In copy-out mode, cpio copies files into an archive. It reads a list of filenames, one per line, on the standard input and writes the archive onto the standard output. A typical way to generate the list of filenames is with the find command; you should give find the -depth option to minimize problems with permissions on directories that are unreadable. Copy-Out mode syntax:
cpio {-o|--create} [-0acvABLV] [-C bytes] [-H format] [-M message]
[-O [[user@]host:]archive] [-F [[user@]host:]archive]
[--file=[[user@]host:]archive] [--format=format]
[--message=message][--null] [--reset-access-time] [--verbose] [--dot]
[--append] [--block-size=blocks] [--dereference] [--io-size=bytes]
[--rsh-command=command] [--help] [--version] < name-list [> archive]
Copy-In Mode Syntax:
In copy-in mode, cpio copies files out of an archive or lists the archive contents. It reads the archive from the standard input. Any non-option command line argumentsare shell globbing patterns; only files in the archive whose names match one or more of those patterns are copied from the archive. Unlike in the shell, an initial ‘.‘ in a filename does match a wildcard at the start of a pattern, and a ‘/‘ in a filename can match wildcards. If no patterns are given, all files are extracted. Copy-In mode syntax:
cpio {-i|--extract} [-bcdfmnrtsuvBSV] [-C bytes] [-E file] [-H format]
[-M message] [-R [user][:.][group]]
[-I [[user@]host:]archive] [-F [[user@]host:]archive]
[--file=[[user@]host:]archive] [--make-directories] [--nonmatching]
[--preserve-modification-time] [--numeric-uid-gid] [--rename] [-t|--list]
[--swap-bytes] [--swap] [--dot] [--unconditional] [--verbose]
[--block-size=blocks] [--swap-halfwords] [--io-size=bytes]
[--pattern-file=file] [--format=format] [--owner=[user][:.][group]]
[--no-preserve-owner] [--message=message] [--force-local]
[--no-absolute-filenames] [--absolute-filenames] [--sparse]
[--only-verify-crc] [--to-stdout] [--quiet] [--rsh-command=command]
[--help] [--version] [pattern...] [< archive]
Copy-Pass Mode Syntax:
In copy-pass mode, cpio copies files from one directory tree to another, combining the copy-out and copy-in steps without actually using an archive. It reads the list of files to copy from the standard input; the directory into which it will copy them is given as a non-option argument. Copy-Pass mode syntax:
cpio {-p|--pass-through} [-0adlmuvLV] [-R [user][:.][group]] [--null]
[--reset-access-time] [--make-directories] [--link] [--quiet]
[--preserve-modification-time] [--unconditional] [--verbose] [--dot]
[--dereference] [--owner=[user][:.][group]] [--no-preserve-owner]
[--sparse] [--help] [--version] destination-directory < name-list
CPIO Options
| -0, –null | Read a list of filenames terminated by a null character, instead of a newline, so that files whose names contain newlines can be archived. GNU find is one way to produce a list of null-terminated filenames. This option may be used in copy-out and copy-pass modes. |
| -a, –reset-access-time | Reset the access times of files after reading them, so that it does not look like they have just been read. |
| -A, –append | Append to an existing archive. Only works in copy-out mode. The archive must be a disk file specified with the -O or -F (-file) option. |
| -b, –swap | Swap both halfwords of words and bytes of halfwords in the data. Equivalent to -sS. This option may be used in copy-in mode. Use this option to convert 32-bit integers between big-endian and little-endian machines. |
| -B | Set the I/O block size to 5120 bytes. Initially the block size is 512 bytes. |
| –block-size=BLOCK-SIZE | Set the I/O block size to BLOCK-SIZE * 512 bytes. |
| -c | Identical to ‘-H newc‘; uses the new (SVR4) portable format. If you want the old portable (ASCII) archive format, use ‘-H odc‘ instead. |
| -C IO-SIZE, –io-size=IO-SIZE | Set the I/O block size to IO-SIZE bytes. |
| -d, –make-directories | Create leading directories where needed. |
| -E FILE, –pattern-file=FILE | Read additional patterns specifying filenames to extract or list from FILE. The lines of FILE are treated as if they had been non-option arguments to cpio. This option is used in copy-in mode. |
| -f, –nonmatching | Only copy files that do not match any of the given patterns. |
| -F, –file=archive | Archive filename to use instead of standard input or output. To use a tape drive on another machine as the archive, use a filename that starts with ‘HOSTNAME:‘. The hostname can be preceded by a username and an ‘@‘ to access the remote tape drive as that user, if you have permission to do so (typically an entry in that user’s ‘~/.rhosts‘ file). |
| –force-local | With -F, -I, or -O, take the archive file name to be a local file even if it contains a colon, which would ordinarily indicate a remote host name. |
| -H FORMAT, –format=FORMAT | Use archive format FORMAT. The valid formats are listed below; the same names are also recognized in all-caps. The default in copy-in mode is to automatically detect the archive format, and in copy-out mode is ‘bin‘. bin: The obsolete binary format. odc: The old (POSIX .1) portable format. newc: The new (SVR4) portable format, which supports file systems having more than 65536 inodes. crc: The new (SVR4) portable format with a checksum added. tar: The old tar format. ustar: The POSIX .1 tar format. Also, recognizes GNU tar archives, which are similar but not identical. hpbin: The obsolete binary format used by HPUX’s cpio (which stores device files differently). hpodc: The portable format used by HPUX’s cpio (which stores device files differently). |
| -i, –extract | Run in copy-in mode. (see ‘Copy-in mode‘). |
| -I archive | Archive filename to use instead of standard input. To use a tape drive on another machine as the archive, use a filename that starts with ‘HOSTNAME:‘. The hostname can be preceded by a username and an ‘@‘ to access the remote tape drive as that user, if you have permission to do so (typically an entry in that user’s ‘~/.rhosts‘ file). |
| -k | Ignored; for compatibility with other versions of cpio. |
| -l, –link | Link files instead of copying them, when possible. |
| -L, –dereference | Copy the file that a symbolic link points to, rather than the symbolic link itself. |
| -m, –preserve-modification-time | Retain previous file modification times when creating files. |
| -M MESSAGE, –message=MESSAGE | Print MESSAGE when the end of a volume of the backup media (such as a tape or a floppy disk) is reached, to prompt the user to insert a new volume. If MESSAGE contains the string ‘%d‘, it is replaced by the current volume number (starting at 1). |
| -n, –numeric-uid-gid | Show numeric UID and GID instead of translating them into names when using the ‘–verbose‘ option. |
| –no-absolute-filenames | Create all files relative to the current directory in copy-in mode, even if they have an absolute file name in the archive. |
| –absolute-filenames | This is the default: tell cpio not to strip leading file name components that contain ‘..‘ and leading slashes from file names in copy-in mode. |
| –no-preserve-owner | Do not change the ownership of the files; leave them owned by the user extracting them. This is the default for non-root users, so that users on System V don’t inadvertently give away files. This option can be used in copy-in mode and copy-pass mode. |
| -o, –create | Run in copy-out mode. (see ‘Copy-out mode‘). |
| -O archive | Archive filename to use instead of standard output. To use a tape drive on another machine as the archive, use a filename that starts with ‘HOSTNAME:‘. The hostname can be preceded by a username and an ‘@‘ to access the remote tape drive as that user, if you have permission to do so (typically an entry in that user’s ‘~/.rhosts‘ file). |
| –only-verify-crc | Verify the CRC of each file in the archive, when reading a CRC format archive. Do not actually extract the files. |
| -p, –pass-through | Run in copy-pass mode. (see ‘Copy-pass mode‘). |
| –quiet | Do not print the number of blocks copied. |
| -r, –rename | Interactively rename files. |
| -R [user][:.][group], –owner [user][:.][group] | Set the ownership of all files created to the specified user and/or group in copy-out and copy-pass modes. Either the user, the group, or both, must be present. If the group is omitted but the ‘:‘ or ‘.‘ separator is given, use the given user’s login group. Only the super-user can change files’ ownership. |
| –rsh-command=COMMAND | Notifies cpio that is should use COMMAND to communicate with remote devices. |
| -s, –swap-bytes | Swap the bytes of each halfword (pair of bytes) in the files. This option can be used in copy-in mode. |
| -S, –swap-halfwords | Swap the halfwords of each word (4 bytes) in the files. This option may be used in copy-in mode. |
| –sparse | Write files with large blocks of zeros as sparse files. This option is used in copy-in and copy-pass modes. |
| -t, –list | Print a table of contents of the input. |
| –to-stdout | Extract files to standard output. This option may be used in copy-in mode. |
| -u, –unconditional | Replace all files, without asking whether to replace existing newer files with older files. |
| -v, –verbose | List the files processed, or with ‘-t‘, give an ‘ls -l‘ style table of contents listing. In a verbose table of contents of a ustararchive, user and group names in the archive that do not exist on the local system are replaced by the names that correspond locally to the numeric UID and GID stored in the archive. |
| -V, –dot | Print a ‘.‘ for each file processed. |
| –version | Print the cpio program version number and exit. |
cpio examples
When creating an archive, cpio takes the list of files to be processed from the standard input, and then sends the archive to the standard output, or to the device defined by the ‘-F‘ option. Usually find or ls is used to provide this list to the standard input. In the following example you can see the possibilities for archiving the contents of a single directory:
% ls | cpio -ov > directory.cpio
The ‘-o‘ option creates the archive, and the ‘-v‘ option prints the names of the files archived as they are added. Notice that the options can be put together after a single ‘–‘ or can be placed separately on the command line. The ‘>‘ redirects the cpio output to the file ‘directory.cpio‘.
If you wanted to archive an entire directory tree, the find command can provide the file list to cpio:
% find . -print -depth | cpio -ov > tree.cpio
This will take all the files in the current directory, the directories below and place them in the archive tree.cpio. Again the ‘-o‘ creates an archive, and the ‘-v‘ option shows you the name of the files as they are archived (see ‘Copy-out mode‘). Using the ‘.‘ in the find statement will give you more flexibility when doing restores, as it will save file names with a relative path via a hard-wired, absolute path. The ‘-depth‘ option forces ‘find‘ to print of the entries in a directory before printing the directory itself. This limits the effects of restrictive directory permissions by printing the directory entries in a directory before the directory name itself.
Extracting an archive requires a bit more thought because cpio will not create directories by default. Another characteristic, is it will not overwrite existing files unless specified.
% cpio -iv < directory.cpio
This will retrieve the files archived in the file directory.cpio and place them in the present directory. The ‘-i‘ option extracts the archive and the ‘-v‘ shows the file names as they are extracted. If you are dealing with an archived directory tree, you need to use the ‘-d‘ option to create directories as necessary, something like:
% cpio -idv < tree.cpio
This will take the contents of the archive tree.cpio and extract it to the current directory. If you try to extract the files on top of files of the same name that already exist (and have the same or later modification time) cpio will not extract the file unless told to do so by the -u option (see ‘Copy-in mode‘).
In copy-pass mode, cpio copies files from one directory tree to another, combining the copy-out and copy-in steps without actually using an archive. It reads the list of files to copy from the standard input; the directory into which it will copy them is given as a non-option argument (see ‘Copy-pass mode‘).
% find . -depth -print0 | cpio --null -pvd new-dir
The example shows copying the files of the present directory, and sub-directories to a new directory called new-dir. Some new options are the ‘-print0‘ available with GNU find, combined with the ‘–null‘ option of cpio. These two options act together to send file names between find and cpio, even if special characters are embedded in the file names. Another is ‘-p‘, which tells cpio to pass the files it finds to the directory ‘new-dir‘.
find . -print | cpio -ocv > /dev/fd0
Above, using the find command would list all files and directories and using the cpiocommand copy those files listed to the floppy drive.
find . -print | cpio -dumpv /home/users/hope
In the above example, the find command would list all files and subdirectories of the current directory, and pipe them to the cpio command, which copies those files to the hope user account.
cpio -icuvd < /dev/fd0
The above command would restore the files back from the floppy.
Syntax:
Example:
CRON is a Linux system process that executes a program at a preset time. To use a CRON script, admins must prepare a text file that describes the program and when they want CRON to execute it. Then, the crontab program loads the text file and executes the program at the specified time.
System Cron jobs exist as entries in the /etc/crontab file. Each job is described on a single line by defining a time interval, a user to run the command as, and the command to run. Cron can run any kind of script, command, or executable.
# /etc/crontab: system-wide crontab
# Unlike any other crontab you don't have to run the `crontab'
# command to install the new version when you edit this file
# and files in /etc/cron.d. These files also have username fields,
# that none of the other crontabs do.
SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
# m h dom mon dow user command
17 * * * * root cd / && run-parts --report /etc/cron.hourly
25 6 * * * root test -x /usr/sbin/anacron || ( cd / && run-parts --report / cron.daily )
47 6 * * 7 root test -x /usr/sbin/anacron || ( cd / && run-parts --report / cron.weekly )
52 6 1 * * root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.monthly )Below is the default system crontab file from Debian 9:
The first job in the Cron table is:
`17 * * * * root cd / && run-parts --report /etc/cron.hourly`.
This means at 17 minutes past each hour, change directory to /, the root of the filesystem. Then, as the root user, run the run-parts binary to execute all jobs in /etc/cron.hourly.
Time intervals are denoted by numbers and operators filled in place of each asterisk in a Cron job’s crontab line. From left to right, the asterisks represent:
Admins use cURL to transfer a URL. It is useful for determining if an application can reach another service and how healthy the service is.
Syntax:
Example:
The declare command states variables, gives them attributes or modifies the properties of variables.
This command displays the amount of disk space available on the file system containing each file name argument. With no file name, the df command shows the available space on all the currently mounted file systems.
his command displays the amount of disk space available on the file system containing each file name argument. With no file name, the df command shows the available space on all the currently mounted file systems.
Use echo to repeat a string variable to standard output.
The enable command stops or starts printers and classes.
The env command runs a program in a modified environment or displays the current environment and its variables.
The eval command analyzes several arguments, concatenates them into a single command and reports on that argument’s status.
This function replaces the parent process with any subsequently typed command. The exec command treats its arguments as the specification of one or more subprocesses to execute.
The exit command terminates a script and returns a value to the parent script.
The expect command talks to other interactive programs via a script and waits for a response, often from any string that matches a given pattern.
The export command converts a file into a different format than its current format. Once a file is exported, it can be accessed by any application that uses the new format.
find
The find command searches the directory tree to locate particular groups of files that meet specified conditions, including -name, -type, -exec, -size, -mtime and -user.
The for and while commands execute or loop items repeatedly as long as certain conditions are met.
With the free command, admins can see the total amount of free and used physical memory and swap space in the system, as well as the buffers and cache used by the kernel.
The grep command searches files for a given character string or pattern and can replace the string with another. This is one method of searching for files within Linux.
This is the GNU Project’s open source program for file compression that compresses webpages on the server end for decompression in the browser. This is popular for streaming media compression and can simultaneously concatenate and compress several streams.
The history function shows all the commands used since the start of the current session.
The iconfig command configures kernel-resident network interfaces at boot time. It is usually only needed when debugging or during system tuning.
With ifup, admins can configure a network interface and enable a network connection.
The ifdown command shuts down a network interface and disables a network connection.
iptablesThe iptables command allows or blocks traffic on a Linux host and can prevent certain applications from receiving or transmitting a request.
With kill signals, admins can send a specific signal to a process. It is most often used to safely shut down processes or applications.
The less command lets an admin scroll through configuration and error log files, displaying text files one screen at a time with backward or forward navigation available.
The locate command reads one or more databases and writes file names to match certain output patterns.
The lft command determines connection routes and provides information to debug connections or find a box/system location. It also displays route packets and file types.
The ln command creates a new name for a file using hard linking, which allows multiple users to share one file.
The ls command lists files and directories within the current working directory, which allows admins to see when configuration files were last edited.
Admins use lsof to list all the open files. They can add -u to find the number of open files by username.
The lsmod command displays a module’s status within the kernel, which helps troubleshoot server function issues.
The man command allows admins to format and display the user manual that’s built into Linux distributions, which documents commands and other system aspects.
The man command allows admins to format and display the user manual that’s built into Linux distributions, which documents commands and other system aspects.
Similar to less, more pages through text one screen at a time, but has limitations on file navigation.
This command mounts file systems on servers. It also lists the current file systems and their mount locations, which is useful to locate a defunct drive or install a new one.
Linux mkdir generates a new directory with a name path.
A Gnome GUI tool that allows admins to specify the information needed to set up a network card.
Admins can use netconfig to configure a network, enable network products and display a series of screens that ask for configuration information.
This command provides information and statistics about protocols in use and current TCP/IP network connections. It is a helpful forensic tool for figuring out which processes and programs are active on a computer and are involved in network communications.
A user can enter a host name and find the corresponding IP address with nslookup. It can also help find the host name.
The od command dumps binary files in octal — or hex/binary — format to standard output.
passwd
Admins use passwd to update a user’s current password.
The ping command verifies that a particular IP address exists and can accept requests. It can test connectivity and determine response time, as well as ensure an operating user’s host computer is working.
Admins use ps to report the statuses of current processes in a system.
The print working directory (pwd) command displays the name of the current working directory.
The read command interprets lines of text from standard input and assigns values of each field in the input line to shell variables for further processing.
This command syncs data from one disk or file to another across a network connection. It is similar to rcp, but has more options.
The GNU screen utility is a terminal multiplexor where a user can use a single terminal window to run multiple terminal applications or windows.
Admins use sdiff to compare two files and produce a side-by-side listing indicating lines that are dissimilar. The command then merges the files and outputs the results to the outfile.
The sed utility is a stream editor that filters text in a pipeline, distinguishing it from other editors. It takes text input, performs operations on it and outputs the modified text. This command is typically used to extract part of a file using pattern matching or to substitute multiple occurrences of a string within a file.
This command is the quickest way to start or stop a service, such as networking.
The shutdown command turns off the computer and can be combined with variables such as -h for halt after shutdown or -r for reboot after shutdown.
Like locate, slocate, or secure locate, provides a way to index and quickly search for files, but it can also securely store file permissions and ownership to hide information from unauthorized users.
Snort is an open source network intrusion detection system and packet sniffer that monitors network traffic. It looks at each packet to detect dangerous payloads or suspicious anomalies. Snort is based on libpcap.
This command sorts lines of text alphabetically or numerically according to the fields. Users can input multiple sort keys.
The sudo command lets a system admin give certain users the ability to run some — or all — commands at the root level and logs all the commands and arguments.
SSH is a command interface for secure remote computer access and is used by network admins to remotely control servers.
The tar command lets users create archives from a number of specified files or to extract files from a specific archive.
The tail command displays the last few lines of the file. This is particularly helpful for troubleshooting code because admins don’t often need all the possible logs to determine code errors.
TOP is a set of protocols for networks that performs distributed information processing and displays the tasks on the system that take up the most memory. TOP can sort tasks by CPU usage, memory usage and runtime.
Admins can create a blank file within Linux with the touch command.
This command translates or deletes characters from a text stream. It writes to a standard output, but it does not accept file names as arguments — it only accepts input from standard input.
The traceroute function determines and records a route through the internet between two computers and is useful for troubleshooting network/router issues. If the domain does not work or is not available, admins can use traceroute to track the IP.
This function displays the current operating system name and can print system information.
With uniq, admins can compare adjacent lines in a file and remove or identify any duplicate lines.
The vi environment is a text editor that allows a user to control the system with just the keyboard instead of both mouse selections and keystrokes.
vmstat
The vmstat command snapshots everything in a system and reports information on such items as processes, memory, paging and CPU activity. This is a good method for admins to use to determine where issues/slowdown may occur in a system.
This is a network utility that retrieves web files that support HTTP, HTTPS and FTP protocols. The wget command works non-interactively in the background when a user is logged off. It can create local versions of remote websites and recreate original site directories.
See for.
The whoami command prints or writes the user login associated with the current user ID to the standard output.
Admins use xargs to read, build and execute arguments from standard input. Each input is separated by blanks.
Being a high level language where it can fit into most of the system it has the capability to handle the all data types.To handle this data the operators are categorized into below 7 categories .
Arithmetic operators are used to perform mathematical operations like addition, subtraction, multiplication etc.
| Operator | Meaning | Example |
|---|---|---|
| + | Add two operands or unary plus | x + y +2 |
| – | Subtract right operand from the left or unary minus | x – y -2 |
| * | Multiply two operands | x * y |
| / | Divide left operand by the right one (always results into float) | x / y |
| % | Modulus – remainder of the division of left operand by the right | x % y (remainder of x/y) |
| // | Floor division – division that results into whole number adjusted to the left in the number line | x // y |
| ** | Exponent – left operand raised to the power of right | x**y (x to the power y) |
Relational operators are used to compare values. It either returns True or False according to the condition.
| Operator | Meaning | Example |
|---|---|---|
| > | Greater that – True if left operand is greater than the right | x > y |
| < | Less that – True if left operand is less than the right | x < y |
| == | Equal to – True if both operands are equal | x == y |
| != | Not equal to – True if operands are not equal | x != y |
| >= | Greater than or equal to – True if left operand is greater than or equal to the right | x >= y |
| <= | Less than or equal to – True if left operand is less than or equal to the right | x <= y |
Assignment operators are used in Python to assign values to variables.
a = 5 is a simple assignment operator that assigns the value 5 on the right to the variable a on the left.
There are various compound operators in Python like a += 5 that adds to the variable and later assigns the same. It is equivalent to a = a + 5.
| Operator | Example | Equivatent to |
|---|---|---|
| = | x = 5 | x = 5 |
| += | x += 5 | x = x + 5 |
| -= | x -= 5 | x = x – 5 |
| *= | x *= 5 | x = x * 5 |
| /= | x /= 5 | x = x / 5 |
| %= | x %= 5 | x = x % 5 |
| //= | x //= 5 | x = x // 5 |
| **= | x **= 5 | x = x ** 5 |
| &= | x &= 5 | x = x & 5 |
| |= | x |= 5 | x = x | 5 |
| ^= | x ^= 5 | x = x ^ 5 |
| >>= | x >>= 5 | x = x >> 5 |
| <<= | x <<= 5 | x = x << 5 |
Logical operators are the and, or, not operators.
| Operator | Meaning | Example |
|---|---|---|
| and | True if both the operands are true | x and y |
| or | True if either of the operands is true | x or y |
| not | True if operand is false (complements the operand) | not x |
in and not in are the membership operators in Python. They are used to test whether a value or variable is found in a sequence (string, list, tuple, set and dictionary).
In a dictionary we can only test for presence of key, not the value.
| Operator | Meaning | Example |
|---|---|---|
| in | True if value/variable is found in the sequence | 5 in x |
| not in | True if value/variable is not found in the sequence | 5 not in x |
is and is not are the identity operators in Python. They are used to check if two values (or variables) are located on the same part of the memory. Two variables that are equal does not imply that they are identical.
| Operator | Meaning | Example |
|---|---|---|
| is | True if the operands are identical (refer to the same object) | x is True |
| is not | True if the operands are not identical (do not refer to the same object) | x is not True |
Bitwise operators act on operands as if they were string of binary digits. It operates bit by bit, hence the name.
For example, 2 is 10 in binary and 7 is 111.
In the table below: Let x = 10 (0000 1010 in binary) and y = 4 (0000 0100 in binary)
| Operator | Meaning | Example |
|---|---|---|
| & | Bitwise AND | x& y = 0 (0000 0000) |
| | | Bitwise OR | x | y = 14 (0000 1110) |
| ~ | Bitwise NOT | ~x = -11 (1111 0101) |
| ^ | Bitwise XOR | x ^ y = 14 (0000 1110) |
| >> | Bitwise right shift | x>> 2 = 2 (0000 0010) |
| << | Bitwise left shift | x<< 2 = 40 (0010 1000) |
JavaScript, the language of the web, is the driving force behind interactive and dynamic web applications. Whether you’re a seasoned developer or just starting your coding journey, having the right tools at your disposal can make all the difference. In this blog, we’ll explore JavaScript code editors and guide you through writing your first JavaScript program.
A code editor is a fundamental tool for any developer. It provides a workspace for writing, editing, and organizing code. JavaScript, being a versatile language, can be written in any plain text editor. However, dedicated code editors offer features that streamline development, such as syntax highlighting, auto-completion, and built-in terminal support.
Now that you have a code editor set up, let’s write a simple “Hello, World!” program in JavaScript.
Open your code editor and create a new file named hello.js.
In hello.js, type the following code:
// Our first JavaScript program
let message = "Hello, World!";
console.log(message);Ctrl+S or Cmd+S).hello.js is saved.node hello.jsYou should see Hello, World! printed in the terminal. Congratulations! You’ve written and executed your first JavaScript program.
Let’s break down the code you just wrote:
let message = "Hello, World!";: This line declares a variable named message and assigns it the value "Hello, World!". In JavaScript, let is used to declare variables.console.log(message);: This line logs the value of the message variable to the console. console.log() is a built-in JavaScript function used for debugging and printing output.Now that you’ve taken the first step into the world of JavaScript programming, there’s a wealth of resources to help you deepen your knowledge:
JavaScript is a powerful language that fuels the modern web. With the right code editor and a bit of practice, you can create dynamic, interactive web applications. Writing your first JavaScript program is just the beginning of an exciting journey into the world of web development.
Whether you’re building simple scripts or complex web applications, JavaScript’s versatility and wide adoption make it an invaluable skill for any developer. So, grab your code editor, unleash your creativity, and start crafting innovative solutions with JavaScript!
In the vast landscape of web development, one language stands out as the cornerstone of interactivity and dynamism: JavaScript. From simple web page enhancements to complex web applications, JavaScript is the force driving much of the modern web experience. In this blog, we’ll unravel the mysteries of JavaScript, exploring its origins, features, and importance in the digital realm.
JavaScript was created by Brendan Eich in 1995 while he was working at Netscape Communications. Initially named “Mocha,” it was later renamed “LiveScript” and eventually settled on “JavaScript” to ride on the popularity of Java at the time. Contrary to its name, JavaScript has little to do with Java and was developed independently as a scripting language for web browsers.
Over the years, JavaScript has evolved significantly. The standardization process led by ECMA International resulted in ECMAScript, the official specification for the language. This has seen several versions, with ECMAScript 6 (ES6) in 2015 being a major milestone, bringing numerous enhancements and modern features to the language.
JavaScript is a high-level, interpreted programming language that allows you to implement complex features on web pages. Unlike languages such as HTML and CSS, which are used for markup and styling, respectively, JavaScript adds interactivity and behavior to web pages.
JavaScript is versatile and offers a wide range of features that make it invaluable for web development:
JavaScript has become an integral part of web development for several compelling reasons:
If you’re eager to dive into JavaScript, there are numerous resources available:
JavaScript is the backbone of modern web development, enabling developers to create dynamic, interactive, and user-friendly web applications. Its evolution from a simple scripting language to a versatile powerhouse has revolutionized the way we interact with the web.
Whether you’re building a personal website, an e-commerce platform, or a social media app, JavaScript empowers you to bring your ideas to life. So, if you’re looking to embark on a journey into the world of web development, JavaScript is undoubtedly a language worth mastering. With its endless possibilities and ever-growing ecosystem, JavaScript continues to shape the digital landscape, making the web a more vibrant and engaging place for all.
Gzip or .gz is used to compress a file in order to reduce disk space, it is quite popular in Linux and UNIX operating systems for this reason. Gzip has been around since May 1996 and is still widely used today.
The below examples showing the process of compressing gzip and unzipping .The compression technique can help you save a lot of memory by compressing .
RHEL:
yum install gzip
Debian:
apt-get install gzip
1. Compress a single file
gzip file.txt
This will compress file.txt and create file.txt.gz, note that this will remove the original file.txt file.gzip file.txt
2. Compress multiple files at once
gzip file1.txt file2.txt file3.txt
This will compress all files specified in the command, note again that this will remove the original files specified by turning file1.txt, file2.txt and file3.txt into file1.txt.gz, file2.txt.gz and file3.txt.gzgzip file1.txt file2.txt file3.txt To instead compress all files within a directory, see example 8 below.
3. Compress a single file and keep the original
gzip -c file.txt > file.txt.gz
You can instead keep the original file and create a compressed copy.gzip -c file.txt > file.txt.gz The -c flag outputs the compressed copy of file.txt to stdout, this is then sent to file.txt.gz, keeping the original file.txt file in place. Newer versions of gzip may also have -k or –keep available, which could be used instead with “gzip -k file.txt”.
4. Compress all files recursively
All files within the directory and all sub directories can be compressed recursively with the -r flag
[root@centos test]# ls -laR
.:
drwxr-xr-x. 2 root root 24 Jul 28 18:05 example
-rw-r--r--. 1 root root 8 Jul 28 17:09 file1.txt
-rw-r--r--. 1 root root 3 Jul 28 17:54 file2.txt
-rw-r--r--. 1 root root 5 Jul 28 17:54 file3.txt
./example:
-rw-r--r--. 1 root root 5 Jul 28 18:00 example.txt
[root@centos test]# gzip -r *
[root@centos test]# ls -laR
.:
drwxr-xr-x. 2 root root 27 Jul 28 18:07 example
-rw-r--r--. 1 root root 38 Jul 28 17:09 file1.txt.gz
-rw-r--r--. 1 root root 33 Jul 28 17:54 file2.txt.gz
-rw-r--r--. 1 root root 35 Jul 28 17:54 file3.txt.gz
./example:
-rw-r--r--. 1 root root 37 Jul 28 18:00 example.txt.gz
In the above example there are 3 .txt files in the test directory which is our current working directory, there is also an example sub directory which contains example.txt. Upon running gzip with the -r flag over everything, all files were recursively compressed. This can be reversed by running “gzip -dr *”, where -d is used to decompress and -r performs this on all of the files recursively.
5. Decompress a gzip compressed file
To reverse the compression process and get the original file back that you have compressed, you can use the gzip command itself or gunzip which is also part of the gzip package.
gzip -d file.txt.gz
OR
gunzip file.txt.gz
Both of these commands will produce the same result, decompressing file.txt.gz to file.txt, removing the compressed file.txt.gz file.
Similar to example 3, it is possible to decompress a file and keep the original .gz file as below.
gunzip -c file.txt.gz > file.txt
As mentioned in step 4, -d can be combined with -r to decompress all files recursively.
6. List compression information
With the -l or –list flag we can see useful information regarding a compressed .gz file such as the compressed and uncompressed size of the file as well as the compression ratio, which shows us how much space our compression is saving.
[root@centos ~]# gzip -l linux-3.18.19.tar.gz compressed uncompressed ratio uncompressed_name 126117045 580761600 78.3% linux-3.18.19.tar
[root@centos ~]
# ls -lah -rw-r–r–. 1 root root 554M Jul 28 17:24 linux-3.18.19.tar -rw-r–r–. 1 root root 121M Jul 28 17:25 linux-3.18.19.tar.gz . In this example, a gzipped copy of the Linux kernel has compressed to 78.3% of its original size, taking up 121MB of space rather than 554MB.
7. Adjust compression level
The level of compression applied to a file using gzip can be specified as a value between 1 (less compression) and 9 (best compression). Using option 1 will complete faster, but space saved from the compression will not be optimal. Using option 9 will take longer to complete, however you will have the largest amount of space saved.
The below example compares the differences between -1 and -9, as shown while -1 finishes much faster it compresses around 5% less (approximately 30mb more space required).
[root@centos ~]# time gzip -1 linux-3.18.19.tar real 0m13.602s user 0m12.908s sys 0m0.662s
[root@mirror1 ~]
# gzip -l linux-3.18.19.tar.gz compressed uncompressed ratio uncompressed_name 156001021 580761600 73.1% linux-3.18.19.tar
[root@centos ~]
# time gzip -9 linux-3.18.19.tar real 0m58.129s user 0m57.193s sys 0m0.735s
# gzip -l linux-3.18.19.tar.gz compressed uncompressed ratio uncompressed_name 125064095 580761600 78.5% linux-3.18.19.tar
-1 can also be specified with the flag –fast, while option -9 can also be specified with the flag –best. By default gzip uses a compression level of -6, which is slightly biased towards higher compression at the expense of speed. When selecting a value between 1 and 9 it is important to consider what is more important to you, the amount of space saved or the amount of time spent compressing, the default -6 option provides a fair trade off.
8. Compress a directory
With the help of the tar command, we can create a tar file of a whole directory and gzip the result. We can perform the whole lot in one step, as the tar command allows us to specify a compression method to use. This example creates a compressed etc.tar.gz file of the entire /etc/ directory. The tar flags are as follows, ‘c’ creates a new tar archive, ‘z’ specifies that we want to compress with gzip, ‘v’ provides verbose information, and ‘f’ specifies the file to create. The resulting etc.tar.gz file contains all files within /etc/ compressed using gzip.
tar czvf etc.tar.gz /etc/
9. Integrity test
The -t or –test flag can be used to check the integrity of a compressed file.
On a normal file, the result will be listed as OK, shown below.
[root@centos test]# gzip -tv file1.txt.gz file1.txt.gz: OK
I have now manually modified this file with a text editor and added a random value, essentially introducing corruption and it is now no longer valid.
[root@centos test]# gzip -tv file1.txt.gz file1.txt.gz: gzip: file1.txt.gz: invalid compressed data--crc error gzip: file1.txt.gz: invalid compressed data--length error
The compressed .gz file makes use of cyclic redundancy check (CRC) in order to detect errors. The CRC value can be viewed by running gzip with the -l and -v flags, as shown below.
[root@centos test]# gzip -lv file1.txt.gz method crc date time compressed uncompressed ratio uncompressed_name defla 08db5c50 Jul 28 18:15 40 167772160 100.0% file1.txt
10. Concatenate multiple files
Multiple files can be concatenated into a single .gz file.
gzip -c file1.txt > files.gz gzip -c file2.txt >> files.gz
The files.gz now contains the contents of both file1.txt and file2.txt, if you decompress files.gz you will get a file named ‘files’ which contains the content of both .txt files. The output is similar to running ‘cat file1.txt file2.txt’. If instead you want to create a single file that contains multiple files you can use the tar command which supports gzip compression, as covered above in example 8.
11. Additional commands included with gzip
The gzip package provides some very useful commands for working with compressed files, such as zcat, zgrep and zless/zmore.
As you can probably tell by the names of the commands, these are essentially the cat, grep, and less/more commands, however they work directly on compressed data. This means that you can easily view or search the contents of a compressed file without having to decompress it and then view or search it in a second step.
[root@centos test]# zcat test.txt.gz test example text
[root@centos test]
# zgrep exa test.txt.gz example
This is especially useful when searching through or reviewing log files which have been compressed during log rotation.
RESTful(REpresentational State Transfer) APIs enable you to develop any kind of web application having all possible CRUD (create, retrieve, update, delete) operations. REST guidelines suggest using a specific HTTP method on a specific type of call made to the server (though technically it is possible to violate this guideline, yet it is highly discouraged).
Use below-given information to find suitable HTTP method for the action performed by API.
Table of Contents HTTP GET HTTP POST HTTP PUT HTTP DELETE HTTP PATCH Summary Glossary
Use GET requests to retrieve resource representation/information only – and not to modify it in any way. As GET requests do not change the state of the resource, these are said to be safe methods. Additionally, GET APIs should be idempotent, which means that making multiple identical requests must produce the same result every time until another API (POST or PUT) has changed the state of the resource on the server.
If the Request-URI refers to a data-producing process, it is the produced data which shall be returned as the entity in the response and not the source text of the process, unless that text happens to be the output of the process.
For any given HTTP GET API, if the resource is found on the server then it must return HTTP response code 200 (OK) – along with response body which is usually either XML or JSON content (due to their platform independent nature).
In case resource is NOT found on server then it must return HTTP response code 404 (NOT FOUND). Similarly, if it is determined that GET request itself is not correctly formed then server will return HTTP response code 400 (BAD REQUEST).
Use POST APIs to create new subordinate resources, e.g. a file is subordinate to a directory containing it or a row is subordinate to a database table. Talking strictly in terms of REST, POST methods are used to create a new resource into the collection of resources.
Ideally, if a resource has been created on the origin server, the response SHOULD be HTTP response code 201 (Created) and contain an entity which describes the status of the request and refers to the new resource, and a Location header.
Many times, the action performed by the POST method might not result in a resource that can be identified by a URI. In this case, either HTTP response code 200 (OK) or 204 (No Content) is the appropriate response status.
Responses to this method are not cacheable, unless the response includes appropriate Cache-Control or Expires header fields.
Please note that POST is neither safe nor idempotent and invoking two identical POST requests will result in two different resources containing the same information (except resource ids).
Use PUT APIs primarily to update existing resource (if the resource does not exist then API may decide to create a new resource or not). If a new resource has been created by the PUT API, the origin server MUST inform the user agent via the HTTP response code 201 (Created) response and if an existing resource is modified, either the 200 (OK) or 204 (No Content) response codes SHOULD be sent to indicate successful completion of the request.
If the request passes through a cache and the Request-URI identifies one or more currently cached entities, those entries SHOULD be treated as stale. Responses to this method are not cacheable.The difference between the POST and PUT APIs can be observed in request URIs. POST requests are made on resource collections whereas PUT requests are made on an individual resource.
As the name applies, DELETE APIs are used to delete resources (identified by the Request-URI).
A successful response of DELETE requests SHOULD be HTTP response code 200 (OK) if the response includes an entity describing the status, 202 (Accepted) if the action has been queued, or 204 (No Content) if the action has been performed but the response does not include an entity.
DELETE operations are idempotent. If you DELETE a resource, it’s removed from the collection of resource. Repeatedly calling DELETE API on that resource will not change the outcome – however calling DELETE on a resource a second time will return a 404 (NOT FOUND) since it was already removed. Some may argue that it makes DELETE method non-idempotent. It’s a matter of discussion and personal opinion.
If the request passes through a cache and the Request-URI identifies one or more currently cached entities, those entries SHOULD be treated as stale. Responses to this method are not cacheable.
HTTP PATCH requests are to make partial update on a resource. If you see PUT requests also modify a resource entity so to make more clear – PATCH method is the correct choice for partially updating an existing resource and PUT should only be used if you’re replacing a resource in its entirety.
Please note that there are some challenges if you decide to use PATCH APIs in your application:
HTTP GET /users/1produces below response:{id: 1, username: 'admin', email: '[email protected]'}A sample patch request to update the email will be like this:HTTP PATCH /users/1[There may be following possible operations are per HTTP specification.
[
{ "op": "test", "path": "/a/b/c", "value": "foo" },
{ "op": "remove", "path": "/a/b/c" },
{ "op": "add", "path": "/a/b/c", "value": [ "foo", "bar" ] },
{ "op": "replace", "path": "/a/b/c", "value": 42 },
{ "op": "move", "from": "/a/b/c", "path": "/a/b/d" },
{ "op": "copy", "from": "/a/b/d", "path": "/a/b/e" }
]
PATCH method is not a replacement for the POST or PUT methods. It applies a delta (diff) rather than replacing the entire resource.
Below table summarises the use of HTTP methods discussed above.
| HTTP METHOD | CRUD | ENTIRE COLLECTION (E.G. /USERS) | SPECIFIC ITEM (E.G. /USERS/123) |
|---|---|---|---|
| POST | Create | 201 (Created), ‘Location’ header with link to /users/{id} containing new ID. | Avoid using POST on single resource |
| GET | Read | 200 (OK), list of users. Use pagination, sorting and filtering to navigate big lists. | 200 (OK), single user. 404 (Not Found), if ID not found or invalid. |
| PUT | Update/Replace | 404 (Not Found), unless you want to update every resource in the entire collection of resource. | 200 (OK) or 204 (No Content). Use 404 (Not Found), if ID not found or invalid. |
| PATCH | Partial Update/Modify | 404 (Not Found), unless you want to modify the collection itself. | 200 (OK) or 204 (No Content). Use 404 (Not Found), if ID not found or invalid. |
| DELETE | Delete | 404 (Not Found), unless you want to delete the whole collection — use with caution. | 200 (OK). 404 (Not Found), if ID not found or invalid. |
As per HTTP specification, the GET and HEAD methods should be used only for retrieval of resource representations – and they do not update/delete the resource on the server. Both methods are said to be considered “safe“.
This allows user agents to represent other methods, such as POST, PUT and DELETE, in a special way, so that the user is made aware of the fact that a possibly unsafe action is being requested – and they can update/delete the resource on server and so should be used carefully.
The term idempotent is used more comprehensively to describe an operation that will produce the same results if executed once or multiple times. This is a very useful property in many situations, as it means that an operation can be repeated or retried as often as necessary without causing unintended effects. With non-idempotent operations, the algorithm may have to keep track of whether the operation was already performed or not.
In HTTP specification, The methods GET, HEAD, PUT and DELETE are declared idempotent methods. Other methods OPTIONS and TRACE SHOULD NOT have side effects so both are also inherently idempotent.
References:
This guide discusses how to install packages using pip and virtualenv. These are the lowest-level tools for managing Python packages and are recommended if higher-level tools do not suit your needs.
Note
This doc uses the term package to refer to a Distribution Package which is different from a Import Package that which is used to import modules in your Python source code.
pip is the reference Python package manager. It’s used to install and update packages. You’ll need to make sure you have the latest version of pip installed.
The Python installers for Windows include pip. You should be able to access pip using:
py -m pip --version
pip 9.0.1 from c:\python36\lib\site-packages (Python 3.6.1)
You can make sure that pip is up-to-date by running:
py -m pip install --upgrade pip
Debian and most other distributions include a python-pip package, if you want to use the Linux distribution-provided versions of pip see Installing pip/setuptools/wheel with Linux Package Managers.
You can also install pip yourself to ensure you have the latest version. It’s recommended to use the system pip to bootstrap a user installation of pip:
python3 -m pip install --user --upgrade pip
Afterwards, you should have the newest pip installed in your user site:
python3 -m pip --version
pip 9.0.1 from $HOME/.local/lib/python3.6/site-packages (python 3.6)
virtualenv is used to manage Python packages for different projects. Using virtualenv allows you to avoid installing Python packages globally which could break system tools or other projects. You can install virtualenv using pip.
On macOS and Linux:
python3 -m pip install --user virtualenv
On Windows:
py -m pip install --user virtualenv
Note
If you are using Python 3.3 or newer the venv module is included in the Python standard library. This can also create and manage virtual environments, however, it only supports Python 3.
virtualenv allows you to manage separate package installations for different projects. It essentially allows you to create a “virtual” isolated Python installation and install packages into that virtual installation. When you switch projects, you can simply create a new virtual environment and not have to worry about breaking the packages installed in the other environments. It is always recommended to use a virtualenv while developing Python applications.
To create a virtual environment, go to your project’s directory and run virtualenv.
On macOS and Linux:
python3 -m virtualenv env
On Windows:
py -m virtualenv env
The second argument is the location to create the virtualenv. Generally, you can just create this in your project and call it env.
virtualenv will create a virtual Python installation in the env folder.
Note
You should exclude your virtualenv directory from your version control system using .gitignore or similar.
Before you can start installing or using packages in your virtualenv you’ll need to activate it. Activating a virtualenv will put the virtualenv-specific python and pip executables into your shell’s PATH.
On macOS and Linux:
source env/bin/activate
On Windows:
.\env\Scripts\activate
You can confirm you’re in the virtualenv by checking the location of your Python interpreter, it should point to the env directory.
On macOS and Linux:
which python
.../env/bin/python
On Windows:
where python
.../env/bin/python.exe
As long as your virtualenv is activated pip will install packages into that specific environment and you’ll be able to import and use packages in your Python application.
If you want to switch projects or otherwise leave your virtualenv, simply run:
deactivate
If you want to re-enter the virtualenv just follow the same instructions above about activating a virtualenv. There’s no need to re-create the virtualenv.
Now that you’re in your virtualenv you can install packages. Let’s install the excellent Requests library from the Python Package Index (PyPI):
pip install requests
pip should download requests and all of its dependencies and install them:
Collecting requests
Using cached requests-2.18.4-py2.py3-none-any.whl
Collecting chardet<3.1.0,>=3.0.2 (from requests)
Using cached chardet-3.0.4-py2.py3-none-any.whl
Collecting urllib3<1.23,>=1.21.1 (from requests)
Using cached urllib3-1.22-py2.py3-none-any.whl
Collecting certifi>=2017.4.17 (from requests)
Using cached certifi-2017.7.27.1-py2.py3-none-any.whl
Collecting idna<2.7,>=2.5 (from requests)
Using cached idna-2.6-py2.py3-none-any.whl
Installing collected packages: chardet, urllib3, certifi, idna, requests
Successfully installed certifi-2017.7.27.1 chardet-3.0.4 idna-2.6 requests-2.18.4 urllib3-1.22
pip allows you to specify which version of a package to install using version specifiers. For example, to install a specific version of requests:
pip install requests==2.18.4
To install the latest 2.x release of requests:
pip install requests>=2.0.0,<3.0.0
To install pre-release versions of packages, use the --pre flag:
pip install --pre requests
Some packages have optional extras. You can tell pip to install these by specifying the extra in brackets:
pip install requests[security]
pip can install a package directly from source, for example:
cd google-auth
pip install .
Additionally, pip can install packages from source in development mode, meaning that changes to the source directory will immediately affect the installed package without needing to re-install:
pip install --editable .
pip can install packages directly from their version control system. For example, you can install directly from a git repository:
git+https://github.com/GoogleCloudPlatform/google-auth-library-python.git#egg=google-auth
For more information on supported version control systems and syntax, see pip’s documentation on VCS Support.
If you have a local copy of a Distribution Package’s archive (a zip, wheel, or tar file) you can install it directly with pip:
pip install requests-2.18.4.tar.gz
If you have a directory containing archives of multiple packages, you can tell pip to look for packages there and not to use the Python Package Index (PyPI) at all:
pip install --no-index --find-links=/local/dir/ requests
This is useful if you are installing packages on a system with limited connectivity or if you want to strictly control the origin of distribution packages.
If you want to download packages from a different index than the Python Package Index (PyPI), you can use the --index-url flag:
pip install --index-url http://index.example.com/simple/ SomeProject
If you want to allow packages from both the Python Package Index (PyPI) and a separate index, you can use the --extra-index-url flag instead:
pip install --extra-index-url http://index.example.com/simple/ SomeProject
pip can upgrade packages in-place using the --upgrade flag. For example, to install the latest version of requests and all of its dependencies:
pip install --upgrade requests
Instead of installing packages individually, pip allows you to declare all dependencies in a Requirements File. For example you could create a requirements.txt file containing:
requests==2.18.4
google-auth==1.1.0
And tell pip to install all of the packages in this file using the -r flag:
pip install -r requirements.txt
Pip can export a list of all installed packages and their versions using the freeze command:
pip freeze
Which will output a list of package specifiers such as:
cachetools==2.0.1
certifi==2017.7.27.1
chardet==3.0.4
google-auth==1.1.1
idna==2.6
pyasn1==0.3.6
pyasn1-modules==0.1.4
requests==2.18.4
rsa==3.4.2
six==1.11.0
urllib3==1.22
This is useful for creating Requirements Files that can re-create the exact versions of all packages installed in an environment.
Database is the very vital part of any application.Implementation of the database can boost your application performance.According to the requirement and budget we often choose the database.In the below post we will discuss the advantages and limitation of the database.
The Oracle Corporation is maintain the consistency at the top of the lists in the popular databases. The first version of this database management tool was created in the late 70s, and there are numerous editions of this tool available to meet your organization’s needs.
The newest version of Oracle, 18c, is designed for the cloud and can be hosted on a single server or multiple servers, and it enables the management of databases holding billions of records. Some of the features of the latest version of Oracle include a grid framework and the use of both physical and logical structures.
This means that physical data management has no effect on access to logical structures. Additionally, security in this release is excellent because each transaction is isolated from others.
Pros
Cons
Ideal for: Large organizations that handle enormous databases and need a variety of features.
MySQL is one of the most popular databases for web-based applications. It’s freeware, but it is frequently updated with features and security improvements. There are also a variety of paid editions designed for commercial use. With the freeware version, there’s a greater focus on speed and reliability instead of including a vast array of features, which can be good or bad depending on what you’re attempting to do.
This database engine allows you to select from a variety of storage engines that enable you to change the functionality of the tool and handle data from different table types. It also has an easy to use interface, and batch commands let you process enormous amounts of data. The system is also incredibly reliable and doesn’t tend to hog resources.
Pros
Cons
Ideal for: Organizations that need a robust database management tool but are on a budget.
As with other popular databases, you can select from a number of editions of Microsoft SQL server. This database management engine works on cloud-based servers as well as local servers, and it can be set up to work on both at the same time. Not long after the release of Microsoft SQL Server 2016, Microsoft made it available on Linux as well as Windows-based platforms.
Some of the standout features for the 2016 edition include temporal data support, which makes it possible to track changes made to data over time. The latest version of Microsoft SQL Server also allows for dynamic data masking, which ensures that only authorized individuals will see sensitive data.
Pros
Cons
Ideal for: Large organizations that use a number of Microsoft products.
PostgreSQL is one of several free popular databases, and it is frequently used for web databases. It was one of the first database management systems to be developed, and it allows users to manage both structured and unstructured data. It can also be used on most major platforms, including Linux-based ones, and it’s fairly simple to import information from other database types using the tool.
This database management engine can be hosted in a number of environments, including virtual, physical and cloud-based environments. The latest version, PostgreSQL 9.5, offers larger data volumes and an increase in the number of concurrent users. Security has also been improved thanks to support for both DBMS_SESSION and expanded password profiles.
Pros
Cons
Ideal for: Organizations with a limited budget that want the ability to select their interface and use JSON.
Another free database that also has a commercial version, MongoDB is designed for applications that use both structured and unstructured data. The database engine is very versatile, and it works by connecting databases to applications via MongoDB database drivers. There is a comprehensive selection of drivers available, so it’s easy to find a driver that will work with the programming language being used.
Since MongoDB wasn’t designed to handle relational data models, even though it can, performance issues are likely to crop up if you attempt to use it this way. However, the database engine is designed to handle variable data that isn’t relational, and it can often work well where other database engines struggle or fail.
MongoDB 3.2 is the latest version, and it features new pluggable storage engines. Documents can also now be validated during updates and inserts, and the text search functions have been improved. A new partial index capability also may allow for improved performance by shrinking the size of indexes.
Pros
Cons
This database management system is free, and like many other free offerings, MariaDB also offers paid versions. There are a variety of plug-ins available for it, and it’s the fastest growing open-source database available.
The database engine allows you to choose from a variety of storage engines, and it makes great use of resources via an optimizer that increases query performance and processing. It’s also highly compatible with MySQL, and it is a drop in replacement with exact matching of commands and APIs because many of the developers of MySQL were involved in its development.
Pros
Cons
Ideal for: Organizations looking for an affordable MySQL alternative.
Created by IBM, DB2 is a database engine that has NoSQL capabilities, and it can read JSON and XML files. Unsurprisingly, it’s designed to be used on IBM’s iSeries servers, but the workstation version works on Windows, Linux and Unix.
The current version of DB2 is LUW is 11.1, which offers a variety of improvements. One, in particular, was an improvement of BLU Acceleration, which is designed to make this database engine work faster through data skipping technology. Data skipping is designed to improve the speed of systems with more data than can fit into memory. The latest version of DB2 also provides improved disaster recovery functions, compatibility, and analytics.
Pros
Cons
Ideal for: Large organizations that need to make the most of available resources and handle large databases.
Designed by SAP SE, SAP HANA is a database engine that is column-oriented and can handle SAP and non-SAP data. The engine is designed to save and retrieve data from applications and other sources across multiple tiers of storage. Along with being able to be hosted from physical servers, it can also be hosted from the cloud.
Pros
Cons
Ideal for: Organizations that are pulling data from applications and aren’t under a terribly constrained budget.
If you find we missed any of the important database you can give comments int the below post.
Although we all know that in python def(): is used to create a function,We can use a def() as er our requirement and modify accordingly.Sometimes its necessary to create a one line function known as Lambda.Which can be created by just in demand situation where it can behave like an normal definition. We can see the result: lambda, map() and filter() are still part of core Python. Only reduce() had to go it is moved into the module functools.
Some like it, others feels complicated lambda operator. The lambda operator or lambda function is a way to create small anonymous functions, i.e. functions without a name. These functions are throw-away functions, i.e. they are just needed where they have been created. Lambda functions are mainly used in combination with the functions filter(), map() and reduce(). The lambda feature was added to Python due to the demand from Lisp programmers.
The general syntax of a lambda function is quite simple:
lambda argument_list: expressionThe argument list consists of a comma separated list of arguments and the expression is an arithmetic expression using these arguments. You can assign the function to a variable to give it a name.
The following example of a lambda function returns the average of its two arguments:
xx=lambda y:y*5 print(xx(55))
The above example might look like a plaything for a mathematician. A formalism which turns an easy to comprehend issue into an abstract harder to grasp formalism. Above all, we could have had the same effect by just using the following conventional function definition:
def xx(y):
return y*5
print(xx(55))
We can assure you that the advantages of this approach will be apparent, when you will have learnt to use the map() function.
The lambda operator can be seen when it is used in combination with the map() function.The map() is a function which takes two arguments the first one is the function and second one on which it needs to map the function.
The first argument func is the name of a function and the second a sequence (e.g. a list) seq. map() applies the function func to all the elements of the sequence seq. Before Python3, map() used to return a list, where each element of the result list was the result of the function func applied on the corresponding element of the list or tuple “seq”. With Python 3, map() returns an iterator.
In the below example we are calculating the total travel cost.Our distance=[30,50,80,12,90,40,60,20] list contain the number of distances.We are calculating the distance by car and bike and storing inside the respective list.We are using the lambda for creating inline function.And by using map we are mapping the function over the list of n elements.
distance=[30,50,80,12,90,40,60,20]
costByCar=[]
costByBike=[]
car=lambda x:x*4
costByCar=map(car,distance)
print(list(costByCar))
bike=lambda x:x*2
costByBike=map(bike,distance)
print(list(costByBike))
The below lines of code are the generic way of python definition to create a function and do same thing
def calcostByCar():
for i in distance:
#Per kilomeater i am considering 4$ as traveliing cost
costByCar.append(i*4)
calcostByCar()
print("Cost by car :",costByCar)
def calcostByBike():
for i in distance:
#Per kilomeater i am considering 2$ as traveliing cost
costByBike.append(i*2)
calcostByBike()
print("Cost by bike :",costByBike)
Output of the above program is as below.
[120, 200, 320, 48, 360, 160, 240, 80]
[60, 100, 160, 24, 180, 80, 120, 40]
The function
filter(function, sequence)
offers an elegant way to filter out all the elements of a sequence “sequence”, for which the function function returns True. i.e. an item will be produced by the iterator result of filter(function, sequence) if item is included in the sequence “sequence” and if function(item) returns True.
In other words: The function filter(f,l) needs a function f as its first argument. f has to return a Boolean value, i.e. either True or False. This function will be applied to every element of the list l. Only if f returns True will the element be produced by the iterator, which is the return value of filter(function, sequence).
In the following example, we filter out first the odd and then the even elements of the sequence of the first 11 Fibonacci numbers:
>>> fibonacci = [0,1,1,2,3,5,8,13,21,34,55] >>> odd_numbers = list(filter(lambda x: x % 2, fibonacci)) >>> print(odd_numbers) [1, 1, 3, 5, 13, 21, 55] >>> even_numbers = list(filter(lambda x: x % 2 == 0, fibonacci)) >>> print(even_numbers) [0, 2, 8, 34] >>> >>> >>> # or alternatively: ... >>> even_numbers = list(filter(lambda x: x % 2 -1, fibonacci)) >>> print(even_numbers) [0, 2, 8, 34] >>>
As we mentioned in the introduction of this chapter of our tutorial. reduce() had been dropped from the core of Python when migrating to Python 3. Guido van Rossum hates reduce(), as we can learn from his statement in a posting, March 10, 2005, in artima.com:
“So now reduce(). This is actually the one I’ve always hated most, because, apart from a few examples involving + or *, almost every time I see a reduce() call with a non-trivial function argument, I need to grab pen and paper to diagram what’s actually being fed into that function before I understand what the reduce() is supposed to do. So in my mind, the applicability of reduce() is pretty much limited to associative operators, and in all other cases it’s better to write out the accumulation loop explicitly.”
The function
reduce(func, seq)continually applies the function func() to the sequence seq. It returns a single value.
If seq = [ s1, s2, s3, … , sn ], calling reduce(func, seq) works like this:
If n is equal to 4 the previous explanation can be illustrated like this: 
We want to illustrate this way of working of reduce() with a simple example. We have to import functools to be capable of using reduce:
>>> import functools >>> functools.reduce(lambda x,y: x+y, [47,11,42,13]) 113 >>>
The following diagram shows the intermediate steps of the calculation:
Determining the maximum of a list of numerical values by using reduce:
>>> from functools import reduce >>> f = lambda a,b: a if (a > b) else b >>> reduce(f, [47,11,42,102,13]) 102 >>>
Calculating the sum of the numbers from 1 to 100:
>>> from functools import reduce >>> reduce(lambda x, y: x+y, range(1,101)) 5050
It’s very simple to change the previous example to calculate the product (the factorial) from 1 to a number, but do not choose 100. We just have to turn the “+” operator into “*”:
>>> reduce(lambda x, y: x*y, range(1,49)) 12413915592536072670862289047373375038521486354677760000000000
If you are into lottery, here are the chances to win a 6 out of 49 drawing:
>>> reduce(lambda x, y: x*y, range(44,50))/reduce(lambda x, y: x*y, range(1,7)) 13983816.0 >>>
We learned “Lambda Operator, Filter, Reduce and Map” that Guido van Rossum prefers list comprehensions to constructs using map, filter, reduce and lambda. In this chapter we will cover the essentials about list comprehensions. List comprehensions were added with Python 2.0. Essentially, it is Python’s way of implementing a well-known notation for sets as used by mathematicians.
In mathematics the square numbers of the natural numbers are, for example, created by { x2 | x ∈ ℕ } or the set of complex integers { (x,y) | x ∈ ℤ ∧ y ∈ ℤ }.
List comprehension is an elegant way to define and create list in Python. These lists have often the qualities of sets, but are not in all cases sets.
List comprehension is a complete substitute for the lambda function as well as the functions map(), filter() and reduce(). For most people the syntax of list comprehension is easier to be grasped.
In the chapter on lambda and map() we had designed a map() function to convert Celsius values into Fahrenheit and vice versa. It looks like this with list comprehension:
>>> Celsius = [39.2, 36.5, 37.3, 37.8] >>> Fahrenheit = [ ((float(9)/5)*x + 32) for x in Celsius ] >>> print Fahrenheit [102.56, 97.700000000000003, 99.140000000000001, 100.03999999999999] >>>
The following list comprehension creates the Pythagorean triples:
>>> [(x,y,z) for x in range(1,30) for y in range(x,30) for z in range(y,30) if x**2 + y**2 == z**2] [(3, 4, 5), (5, 12, 13), (6, 8, 10), (7, 24, 25), (8, 15, 17), (9, 12, 15), (10, 24, 26), (12, 16, 20), (15, 20, 25), (20, 21, 29)] >>>
Cross product of two sets:
>>> colours = [ "red", "green", "yellow", "blue" ]
>>> things = [ "house", "car", "tree" ]
>>> coloured_things = [ (x,y) for x in colours for y in things ]
>>> print coloured_things
[('red', 'house'), ('red', 'car'), ('red', 'tree'), ('green', 'house'), ('green', 'car'), ('green', 'tree'), ('yellow', 'house'), ('yellow', 'car'), ('yellow', 'tree'), ('blue', 'house'), ('blue', 'car'), ('blue', 'tree')]
>>>
Generator comprehensions were introduced with Python 2.6. They are simply a generator expression with a parenthesis – round brackets – around it. Otherwise, the syntax and the way of working is like list comprehension, but a generator comprehension returns a generator instead of a list.
>>> x = (x **2 for x in range(20)) >>> print(x) at 0xb7307aa4> >>> x = list(x) >>> print(x) [0, 1, 4, 9, 16, 25, 36, 49, 64, 81, 100, 121, 144, 169, 196, 225, 256, 289, 324, 361]
Calculation of the prime numbers between 1 and 100 using the sieve of Eratosthenes:
>>> noprimes = [j for i in range(2, 8) for j in range(i*2, 100, i)] >>> primes = [x for x in range(2, 100) if x not in noprimes] >>> print primes [2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43, 47, 53, 59, 61, 67, 71, 73, 79, 83, 89, 97] >>>
We want to bring the previous example into more general form, so that we can calculate the list of prime numbers up to an arbitrary number n:
>>> from math import sqrt >>> n = 100 >>> sqrt_n = int(sqrt(n)) >>> no_primes = [j for i in range(2,sqrt_n) for j in range(i*2, n, i)]
If we have a look at the content of no_primes, we can see that we have a problem. There are lots of double entries contained in this list:
>>> no_primes [4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, 32, 34, 36, 38, 40, 42, 44, 46, 48, 50, 52, 54, 56, 58, 60, 62, 64, 66, 68, 70, 72, 74, 76, 78, 80, 82, 84, 86, 88, 90, 92, 94, 96, 98, 6, 9, 12, 15, 18, 21, 24, 27, 30, 33, 36, 39, 42, 45, 48, 51, 54, 57, 60, 63, 66, 69, 72, 75, 78, 81, 84, 87, 90, 93, 96, 99, 8, 12, 16, 20, 24, 28, 32, 36, 40, 44, 48, 52, 56, 60, 64, 68, 72, 76, 80, 84, 88, 92, 96, 10, 15, 20, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 12, 18, 24, 30, 36, 42, 48, 54, 60, 66, 72, 78, 84, 90, 96, 14, 21, 28, 35, 42, 49, 56, 63, 70, 77, 84, 91, 98, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 18, 27, 36, 45, 54, 63, 72, 81, 90, 99] >>>
The solution to this intolerable problem comes with the set comprehension, which we will cover in the next section.
A set comprehension is similar to a list comprehension, but returns a set and not a list. Syntactically, we use curly brackets instead of square brackets to create a set. Set comprehension is the right functionality to solve our problem from the previous subsection. We are able to create the set of non primes without doublets:
>>> from math import sqrt
>>> n = 100
>>> sqrt_n = int(sqrt(n))
>>> no_primes = {j for i in range(2,sqrt_n) for j in range(i*2, n, i)}
>>> no_primes
{4, 6, 8, 9, 10, 12, 14, 15, 16, 18, 20, 21, 22, 24, 25, 26, 27, 28, 30, 32, 33, 34, 35, 36, 38, 39, 40, 42, 44, 45, 46, 48, 49, 50, 51, 52, 54, 55, 56, 57, 58, 60, 62, 63, 64, 65, 66, 68, 69, 70, 72, 74, 75, 76, 77, 78, 80, 81, 82, 84, 85, 86, 87, 88, 90, 91, 92, 93, 94, 95, 96, 98, 99}
>>> primes = {i for i in range(n) if i not in no_primes}
>>> print(primes)
{0, 1, 2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43, 47, 53, 59, 61, 67, 71, 73, 79, 83, 89, 97}
>>>
The following Python script uses a recursive function to calculate the prime numbers. It incorporates the fact that it is enough to examine the multiples of the prime numbers up to the square root of n:
from math import sqrt
def primes(n):
if n == 0:
return []
elif n == 1:
return []
else:
p = primes(int(sqrt(n)))
no_p = {j for i in p for j in xrange(i*2, n+1, i)}
p = {x for x in xrange(2, n + 1) if x not in no_p}
return p
for i in range(1,50):
print i, primes(i)