Python exit gracefully example

03 September 2022

In Python, such as many others pragramming language, it is possibile to implement an exit gracefully from a program.
Note: I created a repository with a full example --> https://gitlab.com/ipsedixit-org/python-exit-gracefully-example

There two concept explained in this post:

System exit
Catch interrupt signal

System exit

In hava there is a special instruction used to force a program to end immediately and it is:

exit(exit_value);

Where:

exit_value is a number >=0: 0 stands for end correctly, otherwise that was an error

Calling exit interrupt immediately the program and the are no more instruction execute.
So please pay attention to use exit carefully because could create a memory leak on opened resources.

Catch interrupt signal

It is possible to register a shutdown hook using signal.signal(signal.SIGINT, handler), where:

signal.SIGINT is the maskable signal that could be intercepted
handler is a function that automatically run if the signal has been captured

References

Gitlab repository

Java exit gracefully example

20 August 2022

In Java, such as many others pragramming language, it is possibile to implement an exit gracefully from a program.
Note: I created a repository with a full example --> https://gitlab.com/ipsedixit-org/java-exit-gracefully-example

There two concept explained in this post:

System exit
Catch interrupt signal

System exit

In hava there is a special instruction used to force a program to end immediately and it is:

System.exit(exitValue);

Where:

exitValue is a number >=0: 0 stands for end correctly, otherwise that was an error

Calling System.exit interrupt immediately the program and the are no more instruction execute, neither inside finally.
Example:

try {
	System.out.println("Printed");
	System.exit(exitValue);
	System.out.println("Not printed");
} finally {
	System.out.println("Not executed");
}

So please pay attention to use System.exit carefully because could create a memory leak on opened resources.

Catch interrupt signal

It is possible to register a shutdown hook using Runtime.getRuntime().addShutdownHook(...).
The Java virtual machine shuts down in response to two kinds of events:

The program exits normally, when the last non-daemon thread exits or when the exit (equivalently, System.exit) method is invoked, or
The virtual machine is terminated in response to a user interrupt, such as typing ^C, or a system-wide event, such as user logoff or system shutdown.

A shutdown hook is simply an initialized but unstarted thread. When the virtual machine begins its shutdown sequence it will start all registered shutdown hooks in some unspecified order and let them run concurrently. When all the hooks have finished it will then halt. Note that daemon threads will continue to run during the shutdown sequence, as will non-daemon threads if shutdown was initiated by invoking the exit method.

References

Java testcontainers SMTP example

13 August 2022

I have created a repository with a full example in Java using testcontainers and mock SMTP server.
You could find a lot of information in README file or in code comments, anyway here I am if you have any question or doubt.

I took much more time than I expected but I am very happy for the final result because in its simplicity is a real runnable example, ready for production code.

References

Bash Conditional Expressions

27 February 2022

Bash offers many built in conditional expressions that could be used to create bash program.

Note: all example below could be execute in a container:

docker run -it  ubuntu /bin/bash

Some Example

File exists and is a regular file

touch /tmp/file-exists.txt && test -f /tmp/file-exists.txt
echo $?
0

test -f not-existing-file.txt
echo $?
1

Arithmetic binary operators (equals)

test 1 -eq 1
echo $?
0

test 1 -eq 2
echo $?
1

References

More information could be found in GNU documentation

Bash times

20 February 2022

In a previous post I described time command, in this post I will show another usefull command that could be used to prints the accumulated user and system times for the shell and all of its child processes.

Note: all example below could be execute in a container:

docker run -it  ubuntu /bin/bash

In this example is execute times before and after a sleep command.

times && sleep 2 && times
0m0.067s 0m0.054s
0m0.013s 0m0.013s
0m0.067s 0m0.054s
0m0.013s 0m0.015s

The accumulated user and system times are:

Before:

0m0.067s 0m0.054s
0m0.013s 0m0.013s

After:

0m0.067s 0m0.054s
0m0.013s 0m0.015s

What is changed is accumulated system times because the system run a sleep of two seconds. As we can see accumulated user times it is not changed because the execution in CPU/time utilization is very fast, under the 0.00s.

References

More information could be found in GNU documentation

Bash exit status

13 February 2022

Every linux or bash program has an exit status, these are the main points:

exit status is an integer value
exit statuses fall between 0 and 255
all of the Bash builtins return an exit status of zero if they succeed and a non-zero status on failure
the shell may use values above 125

These points are also used in programs written in high level language, such as Java, python,... , to inform the caller about the outcome.

References

More information could be found in GNU documentation

Bash special parameters

05 February 2022

Bash shell has some usefull special parameters:

$* Expands to the positional parameters
$# Expands to the number of positional parameters in decimal
$? Expands to the exit status of the most recently executed foreground pipeline
$$ Expands to the process ID of the shell
$! Expands to the process ID of the job most recently placed into the background

References

More information could be found in GNU documentation

Bash readlink

30 January 2022

In the previous post I describe the use of pwd to know the physical directory of a symbolic link.
Another command that could be used to print value of a symbolic link or canonical file name is readlink

Note: all example below could be execute in a container:

docker run -it  ubuntu /bin/bash

Example:

Create a symbolic link:
```
ln -s /etc/init.d/ /init.d
```

List all directories:

ls

bin  boot  dev  etc  home  init.d  lib  lib32  lib64  libx32  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var

Print value of symbolic link:
```
readlink /init.d
/etc/init.d
```

References

Full command descriptions could be found running readlink --help

Bash pwd

23 January 2022

pwd is a command to show the current working directory.
There is also an environment variable ${PWD} that could be used to print the current working directory.

Note: all example below could be execute in a container:

docker run -it  ubuntu /bin/bash

Example:

pwd
/

echo ${PWD}
/

In case of symbolic link could be used to retrieve the physical directory:

Create a symbolic link
```
ln -s /etc/init.d/ /init.d
```
Go inside the symbolic link
```
cd /init.d
```
If execute command without parameter show the current directory
```
pwd
/init.d
```
If execute command with -P parameter show the physical directory
```
pwd -P
/etc/init.d
```

References

Full command descriptions could be found running pwd --help

Bash read

09 January 2022

Bash read builtin is a utility that can read a line from the standard input and split it into fields.

I found it usefull in bash script in order to:
- receive a user response and use for the following process
- separate a string in multiple fields

Note: all example below could be execute in a container:

docker run -it  ubuntu /bin/bash

Receive a user response and use for the following process

CHOOSED_CHOCOLATE=0
while true; do
    read -r -p "Do you want chocolate? (Y/N): " ANSWER
    case $ANSWER in
        [Yy]* ) CHOOSED_CHOCOLATE=1; break;;
        [Nn]* ) CHOOSED_CHOCOLATE=0; break;;
        * ) echo "Please answer Y or N.";;
    esac
done
echo "CHOOSED_CHOCOLATE: ${CHOOSED_CHOCOLATE}"

Explanation about example:
* -r: do not allow backslashes to escape any characters
* -p: output the string "Do you want chocolate? (Y/N): " without a trailing newline before attempting to read
* ANSWER: The line is split into fields as with word splitting, and the first word is assigned to ANSWER

Separate a string in multiple fields

IFS="|" read -r VAR1 VAR2 <<< "ipsedixit|ipsedixit2"
echo "var1: ${VAR1} - var2: ${VAR2}"

Explanation about example:
* IFS="|": set field separator as | character
* <<<: The word undergoes tilde expansion, parameter and variable expansion, command substitution, arithmetic expansion, and quote removal. See also 3.6.7 Here Strings - GNU documentation

References

The best way to understand a command and its options is to use man page or help usage:)

read --help

GNU documentation about read builtin

Happy new year 2022

31 December 2021

2021 is ending and this is the last post for this year...

In this year I am happy to learnt and explored new things and wrote many posts:
38 post, something like 1 every week from march 2021.

2022 is coming and I have many topics that I want to explore and learn, so I will write many others posts ;)

🎉 Happy new year to everyone 🎉

Bash xargs

24 December 2021

In my daily work sometimes I need to execute commands that doesn't handle piped input,
fortunately there is a xargs that could be used.

xargs run COMMAND with arguments INITIAL-ARGS and more arguments read from input,
it is mainly created for use with commands not built to handle piped input or standard input such as cp, rm, du, ... .

Note: all example below could be execute in a container:

docker run -it  ubuntu /bin/bash

Example:

ls /etc/h* | xargs -I {} sh -c 'du -h {}'

The output is:

4.0K	/etc/host.conf
4.0K	/etc/hostname
4.0K	/etc/hosts

Explanation about example:

ls /etc/h*: list all files/folders that start with h in etc folder
-I {} replace INITIAL-ARGS with names read from standard input
sh -c 'du -h {}' execute in a subshell the command du (Disk Usage) to the input argument

References

The best way to understand a command and its options is to use man page or help usage:)

xargs --help

GNU documentation about xargs

Bash time

18 December 2021

When I need to know execution and system execution time elapsed for a script there is a command that I use time

Note: all example below could be execute in a container:

docker run -it  ubuntu /bin/bash

Example:

time sleep 2

The output is:

real	0m2.002s
user	0m0.000s
sys	0m0.002s

Where:

real is the elapsed time, including time slices used by other processes and time the process spends blocked (for example if it is waiting for I/O to complete);
user is the amount of CPU time spent in user-mode code (outside the kernel) within the process. This is only actual CPU time used in executing the process. Other processes and time the process spends blocked do not count towards this figure;
sys is the amount of CPU time spent in the kernel within the process. This means executing CPU time spent in system calls within the kernel, as opposed to library code, which is still running in user-space. Like 'user', this is only CPU time used by the process. See below for a brief description of kernel mode (also known as 'supervisor' mode) and the system call mechanism.

User+Sys will tell you how much actual CPU time your process used. Note that this is across all CPUs, so if the process has multiple threads, it could potentially exceed the wall clock time reported by Real.

References

Java check JAR or WAR validity

12 December 2021

To be sure that a jar or war file built is a valid one, there is a command that come to the rescue:

RESULT=$(jar tvf myfile.jar > /dev/null 2&1)
echo $RESULT

RESULT should be:

Equal to zero if jar or war is valid
Otherwise it is not valid

Let's look at the options and argument used in jar tvf myfile.jar command:

t option: view the table of contents of the JAR file
v option: produces verbose output on stdout
f option: indicates that the JAR file whose contents are to be viewed is specified on the command line
jar-file argument: is the path and name of the JAR file

This command could be used also to view the content of a jar or war file.

References

Oracle documentation on this command

Display non visible characters in a text file

05 December 2021

Sometimes files contains non visible character or UTF-8 character that are similar to ASCII.
In this post I show two ways that helped me.

Note: all example below could be execute in a container:

docker run -it  ubuntu /bin/bash
apt update && apt install -y vim

Via sed

Create a file with not printable and ascii

echo -e '\xe2\x98\xbc Hello\x07World!\xe2\x98\xba' > /tmp/file.txt

Display non visible character via sed
```
sed -n 'l' /tmp/file.txt
```

Via vi

Create a file with not printable and ascii

echo -e '\xe2\x98\xbc Hello\x07World!\xe2\x98\xba' > /tmp/file.txt

Display non visible character via vi
```
vi -b /tmp/file.txt
```

Change a date of a git commit

28 November 2021

Sometimes could be usefull to change a date of a git commit, weekend refactoring ;)

Note: all example below could be execute in a container:

docker run -it  ubuntu /bin/bash
apt update && apt install git
git config --global user.email "you@example.com"
git config --global user.name "Your Name"

Change latest commit to current date

Create repository

cd /tmp && mkdir change-latest-commit && cd change-latest-commit && git init

First commit

echo "Test" > text.txt && git add . && git commit -m "Initial commit"

Git History show the date

# git show --pretty=fuller
commit 1e11854790cbffe9467a1b41a4f0fc1b872d4eb2 (HEAD -> master)
Author:     Your Name <you@example.com>
AuthorDate: Sun Nov 28 06:14:52 2021 +0000
Commit:     Your Name <you@example.com>
CommitDate: Sun Nov 28 06:14:52 2021 +0000

    Initial commit

diff --git a/text.txt b/text.txt
new file mode 100644
index 0000000..345e6ae
--- /dev/null
+++ b/text.txt
@@ -0,0 +1 @@
+Test

Change dte of the commit
Note: the parameter --date change git author date so in order to change git commit date and author date consistently it
is required to set GIT_COMMITTER_DATE and use of --date.
```
(export GIT_COMMITTER_DATE="Thu, 07 Apr 2005 22:13:13 +0200" && git commit --amend --no-edit --date "$GIT_COMMITTER_DATE")
```

Show in git log that a commit changed date

# git show --pretty=fuller
commit b1623b40aa9cf1acb80dddc7714b130983d291fc (HEAD -> master)
Author:     Your Name <you@example.com>
AuthorDate: Thu Apr 7 22:13:13 2005 +0200
Commit:     Your Name <you@example.com>
CommitDate: Thu Apr 7 22:13:13 2005 +0200

    Initial commit

diff --git a/text.txt b/text.txt
new file mode 100644
index 0000000..345e6ae
--- /dev/null
+++ b/text.txt
@@ -0,0 +1 @@
+Test

References

Git documentation on GIT_AUTHOR_DATE and GIT_COMMITTER_DATE

Docker ssh service no root

21 November 2021

Sometimes could be usefull for testing programs or bash scripts to copy files from one docker image to another docker container.

In this example I use:

sender: a docker image based on ubuntu. From this container files are sent to receiver container and it is possible transfer files using the mechanism of known_hosts file, it is also copied a key from sender to receiver without requiring enter password to connect
receiver: a docker image based on httpd daemon. This container has a http and ssh active daemon in order to be able to entrablish ssh connections. I choosed to use a httpd image because files could be browsable and visible via a normal browser, so based on your requirements it is not a mandatory to have http daemon.

Please note that to keep be simple:

entrypoint bash or config files are created in dockerfile but could be mounted externally

docker-compose.yml

version: "3.7"
services:
  sender:
    build:
      context: .
      dockerfile: ./Dockerfile
      target: sender
    volumes:
      - /tmp/docker-ssh-service:/tmp
    links:
      - receiver
    tty: true
    stdin_open: true
    networks:
      - net-docker-ssh-service

  receiver:
    build:
      context: .
      dockerfile: ./Dockerfile
      target: receiver
    ports:
      - "10022:22"
      - "10080:80"
    networks:
      - net-docker-ssh-service

networks:
  net-docker-ssh-service:
    driver: bridge

Dockerfile

FROM ubuntu:21.10 as sender
RUN apt-get update && apt-get install -y openssh-client git sshpass

# In entry point is copied public keys in known_host 
RUN echo '#!/bin/bash\nmkdir --parent /root/.ssh && ssh-keyscan receiver > /root/.ssh/known_hosts\nexec bash --login' > /root/entrypoint && \
    chmod +x /root/entrypoint

# SSH key to be copied in receiver
RUN ssh-keygen -t ed25519 -f /root/.ssh/id_ed25519 -P ""

WORKDIR /tmp
ENTRYPOINT ["/root/entrypoint"]

FROM httpd:2.4.51 as receiver
RUN apt-get update && apt-get install -y openssh-server 
RUN mkdir /var/run/sshd

RUN groupadd -g 501 receiver
RUN useradd -u 501 receiver -d /home/receiver -g receiver -p password

WORKDIR /home/receiver/.ssh
RUN ssh-keygen -t ed25519 -f /home/receiver/.ssh/id_ed25519 -P ""
# SSH config, to keep it simple I preferred in this example execute echo but could be copyied from external resources
RUN cp /etc/ssh/sshd_config /home/receiver/.ssh/sshd_config && \
	echo "HostKey /home/receiver/.ssh/id_ed25519" >> /home/receiver/.ssh/sshd_config && \
	echo "AuthorizedKeysFile       /home/receiver/.ssh/authorized_keys" >> /home/receiver/.ssh/sshd_config && \
        echo "PidFile /home/receiver/logs/sshd.pid" >> /home/receiver/.ssh/sshd_config

# Copying from sender public key
COPY --from=sender /root/.ssh/id_ed25519.pub  /home/receiver/.ssh/authorized_keys

WORKDIR /usr/local/apache2/htdocs/
# In order to avoid 'Set the 'ServerName' directive globally to suppress this message'
# change apache configuration file
RUN sed -i 's/#ServerName www.example.com:80/ServerName localhost:80/g' /usr/local/apache2/conf/httpd.conf
RUN usermod -a -G www-data receiver && \
	chown -R receiver:www-data /usr/local/apache2/


# To keep simple I prefer to run echo command to create entrypoint but could be copyied from external resources
WORKDIR /home/receiver/logs
WORKDIR /home/receiver/bin
RUN echo '#!/bin/bash\n/usr/local/apache2/bin/apachectl -f /usr/local/apache2/conf/httpd.conf\n/usr/sbin/sshd -f ~/.ssh/sshd_config -D -E /home/receiver/logs/sshd-out.log > /home/receiver/logs/sshd-err.log &\ntail -F /home/receiver/logs/sshd-out.log' > /home/receiver/bin/entrypoint && \
	chmod +x /home/receiver/bin/entrypoint

RUN chown -R receiver:receiver /home/receiver/ && \
	chmod 700 /home/receiver/  && \
	chmod 700 /home/receiver/.ssh  && \
	chmod -R 600 /home/receiver/.ssh/*

USER receiver:receiver

WORKDIR /home/receiver/
ENV HOME=/home/receiver

# Expose ssh and http port
EXPOSE 22
EXPOSE 80

ENTRYPOINT ["/home/receiver/bin/entrypoint"]

Example of usage:

Copy the snippets above in /tmp/docker-ssh-service/docker-compose.yml and /tmp/docker-ssh-service/Dockerfile
Create file in /tmp/docker-ssh-service/my-page.html with this content:

<!DOCTYPE html>
<html>
	<head>
		<title>Page Title</title>
	</head>
	<body>
		<h1>Wow my page is published!!</h1>
	</body>
</html>

Open a terminal and move to /tmp/docker-ssh-service/ folder
Execute command

docker-compose up -d --build && docker-compose exec sender bash

We transfer file my-page.html to receiver

scp ./my-page.html receiver@receiver:/usr/local/apache2/htdocs/my-page.html

Open file transferred via browser http://localhost:10080/my-page.html

References

Bash sending email

10 November 2021

Sometimes it is required to send report via mail and it is very simple to do that via bash because it offers a very easy and flexible way to send email.
In this post I will present you three different email cases and for each of them a little scenario:
1. Only body (text/html), without attachment
2. Only attachment, no body (text/html)
3. Both body (text/html) and attachment

To following examples could be execute in dev-mailserver inside example-sender container.

Only body (text/html), without attachment

This case is usefull when you need to communicate to the user a simple text and avoid him/her to open an attachment to read the content.

echo -e "Hello,\n\nThis is a test" > /tmp/mail-body.txt

export MAIL_TO="[INSERT HERE EMAILS, DIVIDED BY ',', THAT RECEIVE THE REPORT. Example: user@example.com,another-user@example.com ]"

mailx -s "Test report: case Only body (text/html), without attachment" "${MAIL_TO}" < /tmp/mail-body.txt

Only attachments, no body (text/html)

This case is usefull when you need to communicate to the user and attach one or more files, such as a reports in xlsx, but without a body text.
Steps:
1. Transform all attachments from unix to Dos (carriage return from LF to CRLF) and then using uuencode transform ASCII representation of files
2. Send mail

Note: for semplicity attachments in this case are all text files, but it works also for other file types.

ATTACHMENT_1=/tmp/attachment-1.txt
echo -e "Attachment 1\nFirst attachment" > /tmp/attachment-1.txt

ATTACHMENT_2=/tmp/attachment-2.txt
echo -e "Attachment 2\nSecond attachment" > /tmp/attachment-2.txt

ATTACHMENTS=/tmp/tmp-attachments
unix2dos < ${ATTACHMENT_1} | uuencode `basename ${ATTACHMENT_1}` > ${ATTACHMENTS}
unix2dos < ${ATTACHMENT_2} | uuencode `basename ${ATTACHMENT_2}` >> ${ATTACHMENTS}

export MAIL_TO="[INSERT HERE EMAILS, DIVIDED BY ',', THAT RECEIVE THE REPORT. Example: user@example.com,another-user@example.com ]"

mailx -s "Test report: Only attachments, no body (text/html)" "${MAIL_TO}" < ${ATTACHMENTS}

Both body (text/html) and attachment

This case is usefull when you need to communicate to the user and attach one or more files, such as a reports in xlsx.
Steps:
1. Transform all attachments from unix to Dos (carriage return from LF to CRLF) and then using uuencode transform ASCII representation of files
2. Concatenate message and attachments
3. Send mail

Note: for semplicity attachments in this case are all text files, but it works also for other file types.

ATTACHMENT_1=/tmp/attachment-1.txt
echo -e "Attachment 1\nFirst attachment" > /tmp/attachment-1.txt

ATTACHMENT_2=/tmp/attachment-2.txt
echo -e "Attachment 2\nSecond attachment" > /tmp/attachment-2.txt

ATTACHMENTS=/tmp/tmp-attachments
unix2dos < ${ATTACHMENT_1} | uuencode `basename ${ATTACHMENT_1}` > ${ATTACHMENTS}
unix2dos < ${ATTACHMENT_2} | uuencode `basename ${ATTACHMENT_2}` >> ${ATTACHMENTS}


echo -e "Hello,\n\nThis is a test" > /tmp/mail-body.txt
BODY_MESSAGE=/tmp/tmp-body-message
cat /tmp/mail-body.txt ${ATTACHMENTS}  > ${BODY_MESSAGE}

export MAIL_TO="[INSERT HERE EMAILS, DIVIDED BY ',', THAT RECEIVE THE REPORT. Example: user@example.com,another-user@example.com ]"

mailx -s "Test report: Both body (text/html) and attachment" "${MAIL_TO}" < ${BODY_MESSAGE}

References

My previous post on dev-mailserver

Dev mailserver

07 November 2021

I have created a repository with a dev mailserver.
You could find a lot of information in README file, anyway here I am if you have any question or doubt.

The main purposes of a dev mailserver are to be able to:

send/receive email via SMTP and IMAP services
read email using an UI that allow show HTML and attachments
send email to a remote SMTP server

I took more time than I expected but I am very happy for the final result because its simplicity and I learned more about sending email.

References

Gitlab repository

Bash tar command

31 October 2021

tar is a Unix/Linux command used to collect many files/directories into one archive file, often referred to as a tarball.
The name tar derived from "Tape ARchive" and its initial release is around January 1979.

I usually use this program to create compressed archive or extract files/directory from an archive.

Note: all example below could be execute in a container:

docker run -it  ubuntu /bin/bash

Create compressed archive

Create a file and subdir folder example

cd /tmp && mkdir test-compress && cd test-compress
echo "Hello World!" > file1.txt
(mkdir subdir && cd subdir && echo "Hello World! [2]" > file2.txt)

Create compressed archive using tar

tar -czvf archive.tar.gz file1.txt subdir

List all files

# ls -latr *
-rw-r--r-- 1 root root   13 Oct 31 05:13 file1.txt
-rw-r--r-- 1 root root  209 Oct 31 05:18 archive.tar.gz

subdir:
total 12
-rw-r--r-- 1 root root   17 Oct 31 05:13 file2.txt
drwxr-xr-x 3 root root 4096 Oct 31 05:18 ..
drwxr-xr-x 2 root root 4096 Oct 31 05:18 .

Decompress and extract archive

Starting from the last step of the previous example, create destination folder

cd /tmp && mkdir test-decompress && cd test-decompress
cp ../test-compress/archive.tar.gz .

Decompress and extract archive using tar
```
tar -xvf archive.tar.gz
```

List all files:

# ls -latr *
-rw-r--r-- 1 root root   13 Oct 31 05:13 file1.txt
-rw-r--r-- 1 root root  209 Oct 31 05:19 archive.tar.gz

subdir:
total 12
-rw-r--r-- 1 root root   17 Oct 31 05:13 file2.txt
drwxr-xr-x 2 root root 4096 Oct 31 05:18 .
drwxr-xr-x 3 root root 4096 Oct 31 05:19 ..

As you can see the list all files show that tar command decompress and extract correctly and it also preserve file attributes, such as: name, timestamps, ownership, file-access permissions, and directory organization.

References

Full command descriptions could be found running command with --help parameter
Wikipedia page

Bash z commands

24 October 2021

In Unix/Linux there are special commands used to manage compressed files, the most common are:

zcat
zgrep
zmore
zless
zdiff

All these commands could be used also for non compressed file.

Note: all example below could be execute in a container:

docker run -it  ubuntu /bin/bash
apt update && apt install -y less

zcat

Like cat show an entire content of a compressed/uncompressed file.

$ cd /tmp && echo -e "test1\ntest2\ntest3" > test.txt

$ gzip --keep test.txt

$ zcat test.txt.gz
test1
test2
test3

$ zcat test.txt
test1
test2
test3

zgrep

Like grep search a string in a compressed/uncompressed file.

$ cd /tmp && echo -e "test1\ntest2\ntest3" > test.txt

$ gzip --keep test.txt

$ zgrep test2 test.txt.gz
test2

$ zgrep test3 test.txt
test3

zmore

Like more show a content of a file one page at a time.

$ cd /tmp && echo -e "test1\ntest2\ntest3" > test.txt

$ gzip --keep test.txt

$ zmore test.txt.gz

$ zmore test.txt

zless

Like less show a content of a file one page at a time but is it possibile search a string or scroll up and down from a file.

$ cd /tmp && echo -e "test1\ntest2\ntest3" > test.txt

$ gzip --keep test.txt

$ zless test.txt.gz

$ zless test.txt

zdiff

Like diff show differences between two compressed/uncompressed files.

$ cd /tmp && echo -e "test1\ntest2\ntest3" > file1.txt && echo -e "test1\ntest--2\ntest4" > file2.txt

$ gzip --keep file1.txt && gzip --keep file2.txt

$ zdiff file1.txt.gz file2.txt.gz
2,3c2,3
< test2
< test3
---
> test--2
> test4

$ zdiff file1.txt.gz file2.txt
2,3c2,3
< test2
< test3
---
> test--2
> test4

References

Full command descriptions could be found running commands with --help parameter

Bash w

17 October 2021

w is a command to show who is logged on and what they are doing.
It is usefull in shared machine, like server, and then interact with other people connected.

$ w
 04:28:16 up  1:04,  1 user,  load average: 1,22, 1,99, 3,12
USER     TTY      FROM             LOGIN@   IDLE   JCPU   PCPU WHAT
test    tty7     :0               03:26    1:01m  8:09   1.47s session

References

Full command descriptions could be found running w --help
W command on Wikipedia

libfaketime docker

26 September 2021

Sometimes a procedure change is behaviour based on the date or time:
* on christmas an engine could give a different value as output (example: christmas date discount ;) )
* an evaluation that consider time as a important parameter inside the formula (example: loan interest)

Using libfaketime is possible to test a different date or time, in a very handy way.
It could be applied for Linux/Unix and Mac systems.

For me two important behaviours are (there are a lot more, see lifaketime README file for further configurations):
* start at date: usefull if you want to start from a specific date time and go further. This option is only available from 0.9.8 version
* absolute date: usefull if it is required always have the same datetime

I normally prefer start at because faketime also effect application logs and it is difficult have a good troubleshooting in case of absolute date.
A minimal log comparison

Example log 'start at' date	Example log 'absolute' date
2021-09-26 05:23:00.000 Starting 2021-09-26 05:23:00.001 Set up connection pool 2021-09-26 05:23:03.123 Extract data	2021-09-26 05:23:00.000 Starting 2021-09-26 05:23:00.000 Set up connection pool 2021-09-26 05:23:00.000 Extract data

docker-compose.yml

version: "3.7"
services:
  service-start-at:
    build:
      context: .
      dockerfile: ./Dockerfile
      target: start-at
    networks:
      - net-libfaketime

  service-absolute:
    build:
      context: .
      dockerfile: ./Dockerfile
      target: absolute
    networks:
      - net-libfaketime

networks:
  net-libfaketime:
    driver: bridge

Dockerfile

FROM ubuntu:21.10 as start-at
RUN apt update && apt install -y faketime

ENV LD_PRELOAD="/usr/lib/x86_64-linux-gnu/faketime/libfaketime.so.1"
ENV FAKETIME_DONT_RESET=1
ENV FAKETIME_DONT_FAKE_MONOTONIC=1
ENV FAKETIME="@2021-09-26 05:30:00"
CMD ["/usr/bin/bash", "-c" , "while [ $SECONDS -lt 5 ]; do date; sleep 1; done"]

FROM ubuntu:21.10 as absolute
RUN apt update && apt install -y faketime

ENV LD_PRELOAD="/usr/lib/x86_64-linux-gnu/faketime/libfaketime.so.1"
ENV FAKETIME_DONT_RESET=1
ENV FAKETIME_DONT_FAKE_MONOTONIC=1
ENV FAKETIME="2021-09-26 05:30:00"
CMD ["/usr/bin/bash", "-c" , "while [ $SECONDS -lt 5 ]; do date; sleep 1; done"]

Example of usage:

Copy the snippets above in /tmp/faketime-docker/docker-compose.yml and /tmp/faketime-docker/Dockerfile
Open a terminal and move to /tmp/faketime-docker/ folder
Execute command

docker-compose up --build

You can see from the terminal output the two different behaviours :)

service-absolute_1  | Sun Sep 26 05:30:00 UTC 2021
service-start-at_1  | Sun Sep 26 05:30:00 UTC 2021
service-start-at_1  | Sun Sep 26 05:30:01 UTC 2021
service-absolute_1  | Sun Sep 26 05:30:00 UTC 2021
service-start-at_1  | Sun Sep 26 05:30:02 UTC 2021
service-absolute_1  | Sun Sep 26 05:30:00 UTC 2021
service-start-at_1  | Sun Sep 26 05:30:03 UTC 2021
service-absolute_1  | Sun Sep 26 05:30:00 UTC 2021
service-start-at_1  | Sun Sep 26 05:30:04 UTC 2021
service-absolute_1  | Sun Sep 26 05:30:00 UTC 2021

References

Bash more

19 September 2021

more is a historical Unix command used to see, but not modify, a content file.

more is a very basic pager, originally allowing only forward navigation through a file, though newer implementations do allow for limited backward movement. (From wikipedia)

Note: all example below could be execute in a container:

docker run -it  ubuntu /bin/bash

Display the content beginning from a specific line

$ more +55 /etc/gai.conf

Display the contents of gai.conf from line 55

Display the content from a specific regexp

$ more +/"scopev[0-4] :" /etc/gai.conf

References

Full command descriptions could be found running more --help
More command on Wikipedia
My previous post on less

Bash cd

12 September 2021

cd is a command used to change the shell working directory.
Using this command with some environment variables helps to navigate accross a system directories.
If directory is not supplied, the value of the HOME shell variable is used.

Note: all example below could be execute in a container:

docker run -it  ubuntu /bin/bash

Open a directory

$ cd etc

Open a directory using CDPATH

If the shell variable CDPATH exists, it is used as a search path: each directory name in CDPATH is searched for directory, with alternative directory names in CDPATH separated by a colon (‘:’).
This feature is usefull if you have some directories visited frequently.

$ export CDPATH='etc:media'
$ pwd
/
$ cd dpkg
/etc/dpkg

Go home

There some alternatives:

$ cd

$ cd ~

$ cd ${HOME}

Parent directory

$ cd ..

Previous directory

$ cd -

References

Full command descriptions could be found running cd --help
More details in GNU bash manual, section 'Bourne Shell Builtins'

Bash less

05 September 2021

Bash provides several commands to show a file, like cat, but frequently I also use less command.

Less is a command wrote by Mark Nudelman initially during 1983–85, in the need of a version of more able to do backward scrolling of the displayed text. (From wikipedia)

Note: all example below could be execute in a container:

docker run -it  ubuntu /bin/bash
# apt update && apt install -y less

Display line number and jump to a specific line

$ less -N /etc/gai.conf

Then press 34g

Go to beginning or end o a file

$ less -N /etc/gai.conf

Then press G to go to the end of a file
Then press g to go to the begin of a file

Search a string (using regex)

$ less -N /etc/gai.conf

Then press / followed by a string (regex) to search, example: /label [a-z0-9]
To navigate through use n next match and N previous match.

References

Full command descriptions could be found running less --help
Wikipedia

Bash grouping commands

21 August 2021

Bash provides two ways to group a list of commands to be executed as a unit:
* () : create a subshell environment and each of the commands in list to be executed in that subshell;
* {} : list of commands, separated by semicolon or newline, to be executed in the current shell context no subshell is created

Note: all example below are execute in a container:

docker run -it  ubuntu /bin/bash

()

In this example you can see that the current directory has no changed at the end of () and a file parentheses.txt is created

$ pwd
/

$ (cd tmp && touch parentheses.txt)

$ pwd
/

$ ls -latr /tmp/parentheses.txt
-rw-r--r-- 1 root root 0 Aug 21 04:34 /tmp/parentheses.txt

{}

In this example you can see that the current directory has been changed at the end of {} and a file curly.txt is created

$ pwd
/

$ { cd tmp; touch curly.txt; }

$ pwd
/tmp

$ ls -latr /tmp/curly.txt
-rw-r--r-- 1 root root 0 Aug 21 04:36 /tmp/curly.txt

References

A full description and some historical reason for difference between () and {}

Bash cat

07 August 2021

cat, stands for concatenate, is a usefull command used to concatenate FILE(s) to standard output.
I normally use it to see the full content of a file or merge two files in a single file.

Note: all example below are execute in a container:

docker run -it  ubuntu /bin/bash

One file and its number lines

$ cat --number /etc/ld.so.conf
     1	include /etc/ld.so.conf.d/*.conf
     2

Multiple files

# cat /etc/ld.so.conf /etc/ld.so.conf.d/libc.conf
include /etc/ld.so.conf.d/*.conf

# libc default configuration
/usr/local/lib

Merge files

# cat /etc/ld.so.conf /etc/ld.so.conf.d/libc.conf > /tmp/test.txt
# cat /tmp/test.txt
include /etc/ld.so.conf.d/*.conf

# libc default configuration
/usr/local/lib

References

Full command descriptions could be found running cat --help

Bash expr command

18 July 2021

In a previous post I described command bc to evaluate mathematical expression.

There is another command that could be used to check if variable is integer, mathematical (only integer) and generic expression:
expr

In general expr print the value of EXPRESSION to standard output.
Note: all example below are execute in a container:

docker run -it  ubuntu /bin/bash

Check if variable is integer

Case positive integer:

$ expr "2" + 0
2

Case NOT integer:

$ expr "foo" + 0
expr: non-integer argument

Case negative integer:

$ expr "-2" + 0
-2

Mathematical operations

Sum:

# expr 2 + 3
5

Division

# expr 10 / 3
3

Check regular expression

# expr 'ipsedixit.org' : '\(.*\)\.org'
ipsedixit

References

Full command descriptions could be found running expr --help
A tutorial with more example

Python testcontainers SMTP example

10 July 2021

I have created a repository with a full example in python using testcontainers and mock SMTP server.
You could find a lot of information in README file or in code comments, anyway here I am if you have any question or doubt.

I took more time than I expected but I am very happy for the final result because in its simplicity is a real runnable example, ready for production code.

References

Bash trap command

04 July 2021

When I wrote complex bash that contains long operations could be necessary to manage a gracefully exit.

In unix/linux there is a command for this purpose: trap.

Note: There is special signals that cannot be caught or ignored, see wikipedia for more information.

Ignoring (do nothing) when received a specific signal

$ trap '' SIGTERM

Do something when received a specific signal

$ trap 'rm -f /tmp/mytmpfile$$; exit' SIGINT

Full Example

Bash script and save in /tmp/example_trap.sh:

#!/bin/bash

trap '' SIGTERM
trap 'echo "Received SIGINT, exit gracefully"; exit' SIGINT

echo "Start long operation"
sleep 120
echo "End long operation"

Then give them execution permission (like chmod +x) for /tmp/example_trap.sh.

docker run -it --mount 'type=bind,source=/tmp/,target=/tmp/' -w /tmp ubuntu /bin/bash

From inside the container:

# ./example_trap.sh &
[1] 27
Start long operation

Nothing happend for SIGTERM, as expected

# kill -SIGTERM 27
#

The program exit gracefully with SIGINT

# kill -SIGINT 27 
Received SIGINT, exit gracefully
[1]+  Done  ./example_trap.sh

References

Full command descriptions could be found running trap --help
A good article in linux journal

Bash calculator

27 June 2021

From time to time it happens in my work to do some calculation and I use bc command.

The cool features bc are:

it could be possible do mathematical operations with arbitrary precision numbers
it could be done in a standard terminal, so no window change or focus
use a domain specific language, a DSL

Example of usage

Just to have a clean environment start a container

docker run -it ubuntu /bin/bash
# apt update && apt install -y bc

Addition

# echo "2+1" | bc
3

Division no scale

# echo "scale=0;10/3" | bc
3

Division scale=9

# echo "scale=9;10/3" | bc
3.333333333

References

Full command descriptions could be found running bc --help
Good article on bc
More examples

Bash background Jobs

20 June 2021

Sometimes it happen that inside the same terminal I need to do different things, so I need to stop some process and recalling some others.
Fortunately a Unix command come to the rescue: jobs.

Example of usage

Just to have a clean environment start a container

docker run -it ubuntu /bin/bash
# apt update && apt install -y vim

Show current top process

# top
top - 05:13:23 up 30 min,  0 users,  load average: 1.73, 1.95, 1.80
Tasks:   2 total,   1 running,   1 sleeping,   0 stopped,   0 zombie
%Cpu(s): 26.7 us,  6.2 sy,  0.0 ni, 65.8 id,  0.0 wa,  0.0 hi,  1.2 si,  0.0 st
MiB Mem :  2124.0 total,  2024.2 free,      100.0 used,   1539.4 buff/cache
MiB Swap:  164.0 total,    164.0 free,        0.0 used.    984.7 avail Mem

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
      1 root      20   0    4116   3468   2916 S   0.0   0.0   0:00.07 bash
     10 root      20   0    6108   3360   2824 R   0.0   0.0   0:00.01 top

Then press Ctrl + z to send it to background

[1]+  Stopped                 top

Check process stopped via jobs:

# jobs
[1]+  Stopped                 top

Start a new process in background using & at the end of the command:

# vi /tmp/baground.txt &
[2] 608
[2]+  Stopped                 vi /tmp/baground.txt

Using jobs could be find both:

# jobs
[1]-  Stopped                 top
[2]+  Stopped                 vi /tmp/baground.txt

Recall second task (vi /tmp/baground.txt) using %2, so if we want recall the first command we need to use %1:

# fg %2

Exit to vi using :q! (exit without saving)

# jobs
[1]-  Stopped                 top

The last command we use in this example is to close top sending a SIGTERM using kill command:

# kill %1

References

Full commands descriptions could be found running
jobs --help
fg --help
bg --help
kill --help
Good article on jobs

Jbake Commentics integration

13 June 2021

I use Jbake as static content generator and commentics for comments, I described the reasons in a previous post and here.

In this post is described how I integrate commentics in Jbake templates.

templates/post.ftl

Before the line <#include "footer.ftl">

Add the following lines:

	<#if (config.render_enablecommentics)??>
	<script src="https://ipsedixit.org/commentum/embed.js"></script>
	<div id="commentics"></div>
	<#else>
	<h1>Commentics disabled.</h1>
	</#if>

Makefile

# Used to set up languages in date, like 1 April 2021
export JBAKE_OPTS="-Duser.language=en"

# Rule used to clean previous generated file, prod-site is a folder where I store my final files
clean:
	rm -fR ../prod-site/blog ../prod-site/css ../prod-site/fonts  ../prod-site/js

# Rule used to clean up and create static content
# Set up a environment variable used in post.ftl to render commentics.
build: clean
	export JBAKE_OPTS="-Drender.enablecommentics=true "${JBAKE_OPTS} && jbake -b . ../prod-site

# Rule used to create static content & startup a server @ http://localhost:8820 & watch for changes on files
up:
	rm -fR output && jbake -b -s

So when execute:
* make up --> Environment variable render.enablecommentics is not set and a message show 'Commentics disabled.'
* make build --> Environment variable render.enablecommentics is set and import commentics JavaScript and create div

References

Mock SMTP server

03 June 2021

From time to time I need to simulate sending email via bash script or a specific software.

I found a mock SMTP server that perfectly fit my needs mantained in james SMTP server:

a project in very good health (inside from Apache Foundation and actively maintened)
could be used also for automatic tests and in future post I will explore how to use this mock server using testcontainers
simple to startup and usages

To startup locally is very simple:

docker run --rm -p 2525:25 -p 8000:8000 linagora/mock-smtp-server

Above you could find examples of succesfully and failed, by a specific rule, delivering e-mail.

Case successfully deliver e-mail

docker run -d linagora/mock-smtp-server && docker exec -it $(docker ps | grep mock-smtp-server | head -1 | awk '{print $1}') bash

# apt update && apt install -y python

# curl --GET http://127.0.0.1:8000/smtpMails
[]

At the moment there are no email stored in mock server mail.
In order to send email we use python for a simple example:

# python
Python 2.7.13 (default, Apr 16 2021, 14:02:03) 
[GCC 6.3.0 20170516] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import smtplib
>>> server = smtplib.SMTP(host='127.0.0.1', port=25)
>>> server.sendmail(from_addr='my@example.com', to_addrs='you@example.com', msg='Subject: My mail subject\n\nHello World!!!')
{}
>>> server.quit()
(221, 'Bye')
>>> exit()

# curl --GET http://127.0.0.1:8000/smtpMails
[{"from":"my@example.com","recipients":["you@example.com"],"message":"Received: from [172.17.0.2] (localhost [127.0.0.1])\r\n        by 6fc4bb93a031\r\n        with SMTP (SubEthaSMTP 3.1.7) id KP9AQRFY\r\n        for you@example.com;\r\n        Sat, 29 May 2021 05:11:31 +0000 (UTC)\r\nSubject: My mail subject\r\n\r\nHello World!!!\r\n"}]

As expected the API show the email sent via python.

Case failed deliver e-mail

docker run -d linagora/mock-smtp-server && docker exec -it $(docker ps | grep mock-smtp-server | head -1 | awk '{print $1}') bash

# apt update && apt install -y python

# curl --GET http://127.0.0.1:8000/smtpMails
[]

At the moment there are no email stored in mock server mail.
Set a rule that response with a failed deliver:

# curl -X PUT -H "Content-Type: application/json" -d '[{"command": "MAIL FROM","condition": {"operator": "contains","matchingValue": "fail-deliver-case"},"response": {"code": 521,"message": "Server does not accept mail"}}]' http://127.0.0.1:8000/smtpBehaviors

# curl --GET http://127.0.0.1:8000/smtpBehaviors
[{"condition":{"operator":"contains","matchingValue":"fail-deliver-case"},"response":{"code":521,"message":"Server does not accept mail"},"numberOfAnswer":null,"command":"MAIL FROM"}]

The rule is set correctly, emulate failed send mail scenario using python:

# python
Python 2.7.13 (default, Apr 16 2021, 14:02:03) 
[GCC 6.3.0 20170516] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import smtplib
>>> server = smtplib.SMTP(host='127.0.0.1', port=25)
>>> server.sendmail(from_addr='fail-deliver-case@example.com', to_addrs='you@example.com', msg='Subject: My mail subject\n\nHello World!!!')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/smtplib.py", line 736, in sendmail
    raise SMTPSenderRefused(code, resp, from_addr)
smtplib.SMTPSenderRefused: (521, 'Server does not accept mail', 'fail-deliver-case@example.com')
>>> server.quit()
(221, 'Bye')
>>> exit()

# curl --GET http://127.0.0.1:8000/smtpMails
[]

As expected the SMTP server response with error in sending email and it is not stored.

References

Bash countdown script

30 May 2021

Sometimes when I have limited time to do something I check time to time the clock or I work more than I planned.
This behaviour have some disadvantage: I check a lot the clock, distracting me to do a great job, and I work for more time than I planned.

So I created a script that help me :)

countdown.sh

#!/bin/bash

MINUTES=0
HALT_SYSTEM="no"

while [[ ${#} -gt 0 ]]; do
    case "${1}" in
	--minutes)
	    if [ ${#2} -ne 0 ]; then
		MINUTES="${2}"
		if ! [[ ${MINUTES} =~ ^[1-9][0-9]*$ ]]; then
		    echo "Invalid minutes value, must be positive > 0. Please read --help for more information"
		    exit 1
		fi
		shift 2
	    fi
	    ;;
	--halt-system)
	    if [ ${#2} -ne 0 ]; then
		HALT_SYSTEM="${2}"
		if [ "${HALT_SYSTEM}" != "yes" -a "${HALT_SYSTEM}" != "no" ]; then
		    echo "Invalid halt system value  must be equal to yes/no. Please read --help for more information"
		    exit 1
		fi
		shift 2
	    fi
	    ;;
	--help)
	    echo "usage: countdown.sh [--help] [--minutes <positive value>] [--halt-system <yes/no>]"
	    echo "OPTIONS"
	    echo "    --minutes"
	    echo "        Countdown counter minutes."
	    echo ""
	    echo "    --halt-system"
	    echo "        Halt system when time is over yes/no."
	    echo ""
	    echo "    --help"
	    echo "        Read this manual."
	    exit 0
	    ;;
	*)
	    echo "Invalid options found ${1}, please read --help for more information."
	    exit 1
	    ;;
    esac
done


COUNT=${MINUTES}

while [ $COUNT -gt 0 ]
do
	MESSAGE="${COUNT} minutes left."
	echo "${MESSAGE}"
	if [ ${COUNT} -eq 5 -o ${COUNT} -eq 2 ]; then
		# Alert in the last 5/2 minutes using text to speech software.
		spd-say "${MESSAGE} ${MESSAGE}"
	fi
	sleep 60
	COUNT=$((COUNT-1))
done

spd-say "Time is over."

if [ "${HALT_SYSTEM}" == "yes" ]; then
	halt --force
fi

Example of usage

countdown.sh --minutes 6 --halt-system no

References

Bash shell parameter expansion

23 May 2021

I use quite often in my scripts parameter expansion.
So in this post I will explore what I use more but there are a lot more to explore and use :) , please see references for full documentation.

${parameter:-word}

If parameter is unset or null, the expansion of word is substituted. Otherwise, the value of parameter is substituted.

$ parameter="Test"
$ echo "${parameter:-word}"
Test
$ echo "$parameter"
Test

$ unset $parameter
$ echo "${parameter:-word}"
word
$ echo "$parameter"

${parameter:=word}

If parameter is unset or null, the expansion of word is assigned to parameter. The value of parameter is then substituted.

$ parameter="Test"
$ echo ${parameter:=word}
Test
$ echo "$parameter"
Test

$ unset parameter
$ echo ${parameter:=word}
word
$ echo "$parameter"
word

${parameter:offset:length}

This is referred to as Substring Expansion. It expands to up to length characters of the value of parameter starting at the character specified by offset.

$ string=ipsedixit.org
$ echo ${string:4:5}
dixit

References

Full documentation

Basename command

15 May 2021

Basename is a command that could be used to strip directory and suffix from filenames.
The interesting part is that basename does not check the directory or file existence, so the only side effect is to print out the result :) .
Note: the following examples are based on command man/--help page.

Strip directory

Example of usage could be extract a filename from an absolute path.

$ docker run -it ubuntu /bin/bash
# basename /usr/bin/sort
sort

Remove suffix

Example of usage could be exctract file name in the next step rename a file changing its suffix.

$ docker run -it ubuntu /bin/bash
# basename --suffix .h include/stdio.h
stdio

Multiple argument

$ docker run -it ubuntu /bin/bash
# basename --suffix .h -a include/stdio.h include/my_file.h
stdio
my_file

References

Full documentation
The best way to understand a command and its options is to use man page or help usage :)

basename --help

Bash read input parameters

08 May 2021

Normally a bash script has some input parameters used to processing data and there are at least two ways to read them:

Read by name

In this way you make explicit the parameter name with its value.
Example:

./print_phrase_by_name.sh --first-word "Hello" --second-word "World!"

Bash script:

#!/bin/bash

FIRST_WORD=""
SECOND_WORD=""

while [[ ${#} -gt 0 ]]; do
	case "${1}" in
		--first-word)
			if [ ${#2} -ne 0 ]; then
				FIRST_WORD="${2}"
				shift 2
			fi
			;;
		--second-word)
			if [ ${#2} -ne 0 ]; then
				SECOND_WORD="${2}"
				shift 2
			fi
			;;
		--help)
			echo "usage: print_phrase_by_name.sh [--help] [--first-word <word>] [--second-word <word>]"
			echo "OPTIONS"
			echo "    --first-word"
			echo "        First word to print."
			echo ""
			echo "    --second-word"
			echo "        Second word to print."
			echo ""
			echo "    --help"
			echo "        Read this manual."
			exit 0
			;;
		*)
			echo "Invalid options found ${1}, please read --help for more information."
			exit 1
			;;
	esac
done

# TODO check FIRST_WORD and SECOND_WORD are not empty
echo "${FIRST_WORD} ${SECOND_WORD}"

exit 0

Read by position

For me read by position is not the first choice because it obfuscated the meaning of the input parameter.
The advantage of this approach compared to by name is more concise.
Example:

./print_phrase_by_pos.sh "Hello" "World!"

Bash script:

#!/bin/bash

FIRST_WORD="${1}"
SECOND_WORD="${2}"

# TODO check FIRST_WORD and SECOND_WORD are not empty
echo "${FIRST_WORD} ${SECOND_WORD}"

exit 0

Example in a containerized environment

Copy previous example in /tmp/print_phrase_by_pos.sh and /tmp/print_phrase_by_name.sh
Then give them execution permission (like chmod +x) for both scripts.

docker run -it --mount 'type=bind,source=/tmp/,target=/tmp/' -w /tmp ubuntu /bin/bash

From inside the container:

# ./print_phrase_by_name.sh --first-word "Hello" --second-word "World!"
Hello World!

# ./print_phrase_by_name.sh --help                                     
usage: print_phrase_by_name.sh [--help] [--first-word <word>] [--second-word <word>]
OPTIONS
    --first-word
        First word to print.

    --second-word
        Second word to print.

    --help
        Read this manual.

# ./print_phrase_by_pos.sh "Hello" "World!"
Hello World!

References

Gnu shell bash parameter

Commentics

02 May 2021

As described in a previous post I used a static content generator for my blog.
What I missed with static content generator is to allow leaving a comment to a post, and I think this is important because it possible for the readers to share thoughts or suggestion about the specific topic.

There are a lot available options for blogging comments, among all I choose what I retain the best for my purpose and scope:

commentics

The reasons are:

is a free and open source software (free as in speech but also free as free beer ;), but you could in a one-off payment to support the project and have more features )
all data has been stored in your website, no external resources including captcha part
easy to install and minimum requirements (MySQL, PHP 7+ and web server, like apache server)
flexible: it is possible customize all settings, such as fields to show, mandatory fields, ...

JBake - Static site/blog generator

30 April 2021

For my blog I prefer to generate a static contents (HTML, Javascripts, ...) instead of dynamic pages because I update not so
frequently and my pages are very simple.

Among all static contents generator I choose to use (JBake)[https://jbake.org/], I found it very simple to use and very well documented and I like the default template :) .

I created a simple Makefile to help me to create and run locally Jbake:

# Used to set up languages in date, like 1 April 2021
export JBAKE_OPTS="-Duser.language=en"

# Rule used to clean previous generated file, prod-site is a folder where I store my final files
clean:
	rm -fR ../prod-site/blog ../prod-site/css ../prod-site/fonts  ../prod-site/js

# Rule used to clean up and create static content
build: clean
	jbake -b . ../prod-site

# Rule used to create static content & startup a server @ http://localhost:8820 & watch for changes on files
up:
	rm -fR output && jbake -b -s

References

My post on makefile

QR Code WiFi

24 April 2021

QR code is a matrix barcode based that contains some information and could be used also to setup your smartphone wifi.
This functionality is very handy if you have a public wifi, you have several devices to setup or if you sometimes lost your settings.

In order to setup a qrcode for your wifi, following these steps:
1. Create QR-code as described in containerized environment
2. Print on a paper the output image
3. Add Wi-Fi to your smartphone scanning the qr-code image

Create QR-Code in a containerized environment

$ mkdir /tmp/wifi-qrcode && docker run -it -v /tmp/wifi-qrcode:/tmp -w /tmp python:3.9.4-alpine3.13 sh
# apk add build-base jpeg-dev zlib-dev
# pip install wifi-qrcode-generator==0.1
# python
>>> import wifi_qrcode_generator
>>> wifi_qrcode_generator.wifi_qrcode(ssid='Home wifi', hidden=False, authentication_type='WPA', password='very complicated password').save('output.png')

You will find qrcode generated in /tmp/wifi-qrcode/output.png

References

Bash read only variables

17 April 2021

It is possible to make immutable variables or functions in bash script using readonly command, in this way it could be possible avoid to overwrite variables in complex environment.
If you try to overwrite a readonly variable then an error message will be appear bash: ${VARIABLE NAME}: readonly variable.
Example:

Set readonly variables

$ readonly MY_VAR=test

List readonly variables

$ readonly -p

Example in a containerized environment

$ docker run -it ubuntu /bin/bash
root@9a44663a36f0:/# readonly -p
declare -r BASHOPTS="checkwinsize:cmdhist:complete_fullquote:expand_aliases:extquote:force_fignore:globasciiranges:histappend:hostcomplete:interactive_comments:progcomp:promptvars:sourcepath"
declare -ar BASH_VERSINFO=([0]="5" [1]="0" [2]="17" [3]="1" [4]="release" [5]="x86_64-pc-linux-gnu")
declare -ir EUID="0"
declare -ir PPID="0"
declare -r SHELLOPTS="braceexpand:emacs:hashall:histexpand:history:interactive-comments:monitor"
declare -ir UID="0"
root@9a44663a36f0:/# readonly MY_VAR=test
root@9a44663a36f0:/# MY_VAR=test2
bash: MY_VAR: readonly variable
root@9a44663a36f0:/# echo ${MY_VAR}
test
root@9a44663a36f0:/# readonly -p
declare -r BASHOPTS="checkwinsize:cmdhist:complete_fullquote:expand_aliases:extquote:force_fignore:globasciiranges:histappend:hostcomplete:interactive_comments:progcomp:promptvars:sourcepath"
declare -ar BASH_VERSINFO=([0]="5" [1]="0" [2]="17" [3]="1" [4]="release" [5]="x86_64-pc-linux-gnu")
declare -ir EUID="0"
declare -r MY_VAR="test"
declare -ir PPID="0"
declare -r SHELLOPTS="braceexpand:emacs:hashall:histexpand:history:interactive-comments:monitor"
declare -ir UID="0"

References

The best way to understand a command and its options is to use man page or help usage:)

readonly --help

Docker ssh service

10 April 2021

Note:

I published a new blog post about start SSH service as a no root user

Sometimes could be usefull for testing programs or bash scripts to copy files from one docker image to another docker container.

In this example I use:
* sender: a docker image based on ubuntu. From this container files are sent to receiver container and it is possible transfer files using the mechanism of known_hosts file, it is also copied a key from sender to receiver without requiring enter password to connect
* receiver: a docker image based on httpd daemon. This container has a http and ssh active daemon in order to be able to entrablish ssh connections. I choosed to use a httpd image because files could be browsable and visible via a normal browser, so based on your requirements it is not a mandatory to have http daemon.

Please note that to keep be simple:
* I use root user (not reccomended for production environment!)
* entrypoint bash are created in dockerfile but could be mounted externally

docker-compose.yml

version: "3.7"
services:
  sender:
    build:
      context: .
      dockerfile: ./Dockerfile
      target: sender
    volumes:
      - /tmp/docker-ssh-service:/tmp
    links:
      - receiver
    tty: true
    stdin_open: true
    networks:
      - docker-ssh-service

  receiver:
    build:
      context: .
      dockerfile: ./Dockerfile
      target: receiver
    ports:
      - "10022:22"
      - "10080:80"
    networks:
      - docker-ssh-service

networks:
  docker-ssh-service:
    driver: bridge

Dockerfile

FROM ubuntu:20.10 as sender
RUN apt-get update && apt-get install -y openssh-client git sshpass

# In entry point is copied public keys in known_host 
RUN echo '#!/bin/bash\nmkdir --parent /root/.ssh && ssh-keyscan receiver > /root/.ssh/known_hosts\nexec bash --login' > /root/entrypoint && \
    chmod +x /root/entrypoint

# SSH key to be copied in receiver
RUN ssh-keygen -t ed25519 -f /root/.ssh/id_ed25519 -P ""

WORKDIR /tmp
ENTRYPOINT ["/root/entrypoint"]

FROM httpd:2.4.46 as receiver
RUN apt-get update && apt-get install -y openssh-server 
RUN mkdir /var/run/sshd

RUN ssh-keygen -t ed25519 -f /root/.ssh/id_ed25519 -P ""
RUN sed -i 's/#   IdentityFile ~\/.ssh\/id_ed25519/   IdentityFile ~\/.ssh\/id_ed25519/g' /etc/ssh/ssh_config

# Copying from sender public key
COPY --from=sender /root/.ssh/id_ed25519.pub  /root/.ssh/authorized_keys
RUN chmod 600 /root/.ssh/authorized_keys

WORKDIR /usr/local/apache2/htdocs/
# In order to avoid 'Set the 'ServerName' directive globally to suppress this message'
# change apache configuration file
RUN sed -i 's/#ServerName www.example.com:80/ServerName localhost:80/g' /usr/local/apache2/conf/httpd.conf

# To keep simple I prefer to run echo command to create entrypoint but could be copyied from external resources
RUN echo '#!/bin/bash\napachectl start\n/usr/sbin/sshd -D' > /root/entrypoint && \
    chmod +x /root/entrypoint

# Expose ssh and http port
EXPOSE 22
EXPOSE 80

ENTRYPOINT ["/root/entrypoint"]

Example of usage:

Copy the snippets above in /tmp/docker-ssh-service/docker-compose.yml and /tmp/docker-ssh-service/Dockerfile
Create file in /tmp/docker-ssh-service/my-page.html with this content:

<!DOCTYPE html>
<html>
	<head>
		<title>Page Title</title>
	</head>
	<body>
		<h1>Wow my page is published!!</h1>
	</body>
</html>

Open a terminal and move to /tmp/docker-ssh-service/ folder
Execute command

docker-compose up -d --build && docker-compose exec sender bash

We transfer file my-page.html to receiver

scp ./my-page.html receiver:/usr/local/apache2/htdocs/my-page.html

Open file transferred via browser http://localhost:10080/my-page.html

References

Makefile commands

31 March 2021

Makefile is a build automation tool for Unix created around April 1976 by Stuart Feldman at Bell Labs.
It use is widespread in Unix and Linux culture and is used a lot in computer science histories: from building First versions of Unix to build modern software such as linux kernel.

Makefile is a great tool because:
- text file, could be edit with your favourite editor
- dependencies: can be specified dependencies among different directive
- define rules: it is possible write in a easy way how to do things
- flexible: it is not a tool related only for building software but could be used also to help in day by day work (like know how many days left from the next weekend ;) )

In this post I will show you a basic example of using Makefile.
Example:

Copy this file in /tmp/Makefile

current_day=$(shell date +%u)
normal_day=$(shell if (test ${current_day} -lt 5); then echo "yes"; else echo "no"; fi)
countdown=$(shell expr 6 - ${current_day})

hello:
	echo "hello world!"

weekend:
ifeq "$(normal_day)" "yes"
	echo 'Not today :( ... ${countdown} left'
endif
ifeq '${current_day}' '5'
	echo 'Weekend is coming ;)'
endif
ifeq '${current_day}' '6'
	echo '!!! Saturday :) !!!'
endif
ifeq '${current_day}' '7'
	echo '!!! Sunday :) !!!'
endif

Move to /tmp
execute commands like make hello make weekend

Example in a containerized environment

Copy previous example in /tmp/Makefile

$ docker run -it --mount 'type=bind,source=/tmp/Makefile,target=/tmp/Makefile' ubuntu /bin/bash
root@c6fb5678babc:/# apt update && apt install make && cd /tmp
root@c6fb5678babc:/tmp# make hello
echo "hello world!"
hello world!
root@c6fb5678babc:/tmp# make weekend
echo 'Not today :( ... 3 left'
Not today :( ... 3 left
root@c6fb5678babc:/tmp#

References

Change file date

25 March 2021

In Unix/Linux operating system it is possible to modify or change timestamp of a file using touch command.
Example:

$ touch -t 197001020304.05 example-change-date.txt

Where the date format is [[CC]YY]MMDDhhmm[.ss] :
- [[CC]YY] = 1970, Year
- MM = 01, Month
- DD = 02, Day of the month
- hh = 03, Hour of the day in 24 format, so 1AM --> 01 and 1PM --> 13
- mm = 04, Minutes
- [.ss] = 05, Seconds

Example in a containerized environment

$ docker run -it ubuntu /bin/bash
root@37748a763085:/# cd /tmp
root@37748a763085:/tmp# touch example-change-date.txt
root@37748a763085:/tmp# ls --full-time
total 0
-rw-r--r-- 1 root root 0 2021-03-25 20:32:39.077535888 +0000 example-change-date.txt
root@37748a763085:/tmp# touch -t 197001020304.05 example-change-date.txt 
root@37748a763085:/tmp# ls --full-time
total 0
-rw-r--r-- 1 root root 0 1970-01-02 03:04:05.000000000 +0000 example-change-date.txt

References

The best way to understand a command and its options is to use man page :)

man touch

Core utils

Random password

17 March 2021

Sometimes I need to create a random password so I created an alias, saved in ~/.bashrc, for my needs that use openssl:
alias random-psw='$(openssl rand -base64 8)'

Example in a containerized environment:

$ docker run -it ubuntu /bin/bash
root@2f07ea84c99b:/# apt update && apt install -y openssl
root@2f07ea84c99b:/# openssl rand -base64 8
0BUp74ivXSa6AA

Change password linux - passwd

13 March 2021

passwd is the command in linux to change password for your user or for other users if you are root,
Example as simple user:
*

$ passwd --status
ipsedixit P 2021-03-10 0 99999 7 -1

$ passwd
New password: 
Retype password: 
passwd: password updated successfully

$ passwd --status
ipsedixit P 2021-03-11 0 99999 7 -1

Example as root:
*

# passwd ipsedixit --status
ipsedixit P 2021-03-10 0 99999 7 -1

# passwd ipsedixit
New password: 
Retype password: 
passwd: password updated successfully

# passwd ipsedixit --status
ipsedixit P 2021-03-11 0 99999 7 -1

Some Notes:
* The best way to understand a command and its options is to use man page :)

man passwd

If you want test in a containerized environment try:

docker run -it ubuntu /bin/bash

Docker compose exec

02 March 2021

Sometimes docker compose has a lot of services and there is a cool command to go directly to one container for a specific service:

docker-compose exec my-pretty-service bash

If you need to login as a root

docker-compose exec -u 0 my-pretty-service bash

Example:

Create a file called docker-compose.yml

version: "3.7"
services:
  my-pretty-service:
    image: ubuntu:20.10
    tty: true
    stdin_open: true

  another-my-pretty-service:
    image: ubuntu:20.10
    tty: true
    stdin_open: true

Start up all services:

docker-compose --file ./docker-compose.yml up

In another terminal enter in your pretty service

docker-compose exec my-pretty-service bash

$ echo 'Hello World!'

28 November 2020

Hello World!

Older posts are available in the archive.

Blog

Python exit gracefully example

System exit

Catch interrupt signal

References

Java exit gracefully example

System exit

Catch interrupt signal

References

Java testcontainers SMTP example

References

Bash Conditional Expressions

Some Example

File exists and is a regular file

Arithmetic binary operators (equals)

References

Bash times

References

Bash exit status

References

Bash special parameters

References

Bash readlink

References

Bash pwd

References

Bash read

Receive a user response and use for the following process

Separate a string in multiple fields

References

Happy new year 2022

🎉 Happy new year to everyone 🎉

Bash xargs

References

Bash time

References

Java check JAR or WAR validity

References

Display non visible characters in a text file

Via sed

Via vi

Change a date of a git commit

Change latest commit to current date

References

Docker ssh service no root

docker-compose.yml

Dockerfile

Example of usage:

References

Bash sending email

Only body (text/html), without attachment

Only attachments, no body (text/html)

Both body (text/html) and attachment

References

Dev mailserver

References

Bash tar command

Create compressed archive

Decompress and extract archive

References

Bash z commands

zcat

zgrep

zmore

zless

zdiff

References

Bash w

References

libfaketime docker

docker-compose.yml

Dockerfile

Example of usage:

References

Bash more

Display the content beginning from a specific line

Display the content from a specific regexp

References

Bash cd

Open a directory