Pages

Showing posts with label script. Show all posts
Showing posts with label script. Show all posts

August 22, 2017

Beautiful Mac OS Terminal

This post is about making your Terminal window beautiful and easy to use. I use all of them on Mac OS, I guess they are compatible with all UNIX systems.

1. HTTPie for working with HTTP requests

This tool can be used instead of cURL. It just makes responses beautiful (but it's quite powerfull tho):

2. Oh My Git for working with GIT repositories

This tool prints information about Git repo status: branch which you are currently using, number of unpushed commits, tags, untracked files etc. Official readme has a lot of screenshots:

3. ~/.bash_profile for configuring prompt

I configured my promp according to this article, so in my ~/.bash-profile file I have the folloring part:

Which makes prompt look like this:

4. ~/.vimrc for configuring Vim

If you already use Vim - read A Good Vimrc. If not - see Vim isn’t that scary. Here are 5 free resources you can use to learn it.

Here is my ~/.vimrc file:


5. ~/.bash_aliases for quick commands

I have already wrote about aliases that I use here, here and here, but as I already mentioned Git here, I want to share one more alias I have:

It transforms git log output into something like this:

February 11, 2017

Automatic W3C Validation Using YAML Conf in GitLab Continues Integration

As I already wrote I host my Profile Page on GitLab and use Runner to deploy the code. Runner is quite powerful tool, than can be managed with YAML .gitlab-ci.yml conf file.

Basically, it means that after every commit some automatic actions happen (well, actually, not on every commit, depends on how you configure it). In my case .gitlab-ci.yml looks like this:


It means that after I commit something into master branch it performs deploy stage (script move-content-to-public.sh) and test stage (script w3c-html-validation.sh). Both scripts are my own custom sripts and in this post I want to talk about the second one – W3C validation of HTML file.

You can find the script content in my repo: w3c-html-validation.sh. The good thing is that W3C has public API for validating some URL. So all I need is just requesting this API with URL of my page as an input parameter. And that's the reason why I perform the test stage after the deploy stage: new changes need to be published to check them. Also, W3C errors are not verty critical, so I dond't want to fail the whole deploy because of them.

It looks like a very simple thing (it actually is), but it took a time for me to configure this validation (as I was not very familiar with YAML), so I decided to leave some notes here.

November 19, 2016

Use Oracle Data Base in Command Line


Cadmus Asks the Delphic Oracle Where He Can Find his Sister, Europa by Hendrik Goltzius, 1615


Sometimes I need to make some quick changes in data base. Or I need to do several changes using some patterns and templates. In that case I don't want to open editor (I use DataGrip) and wait for loading all resources, connecting to DB, opening SQL window and inserting or copying SQL query. I am that kind of person who keeps open only those applications that I am going to use in next 30 min. If I don't plan to use something in next half an hour I close it, so every opening takes some time and effort.

The best way to save time and effort is command line or, actually, scripts. SQLPlus is a great tool for using Oracle data base in the console. You can find instructions about installing SQLPlus on MacOS on StackOverflow: Oracle Sqlplus client on Mac.

The annoying thing about using data base in the terminal is connecting: you have to remember and write all credentials of data base every time you want to connect to it. But, as usually, scripting resolves this problem. I wrote simple Bash script connect-to-oracle-db.sh, that opens connection using small alias as an input. For example, I write sql test in terminal and it opens connection with test data base where I can write some SQL right ahead. Instead of test it can be any base that I work with: demo, live, production etc.

Second script that I use is changing some value in specific data base in specific table. Some times one of the clients asks me to change specific data in their DB, so now instead of opening an editor I can just run a script with parameter given by client and it changes all the data, which saves me time and doesn't disturb my attention on other tasks so much. The script looks something like that:


Of course, SQLPlus doesn't replace editors like SQL Developer or DataGrip, but it can save a lot of time and effort in performing small and routine tasks.

September 7, 2016

Checking Deployments on Tomcat Server Without Web Manager

In some cases Tomcat web manager is disabled (for example, in production for security reasons). Then the only way to see deployments and their statuses is to use Tomcat API. To list deployed applications you may do following request:
http://localhost:8080/manager/text/list
And you will get something like this:
The problem is that if you have a lot of applications such output is not very easy to read – you can't say is there any stopped applications without reading all rows. For that case you can use awk to color the output, which is much more informative:
The next problem is that awk command is quite long and you don't want to type something like this every time:
curl http://localhost:8080/manager/text/list | sort | grep ^/ | awk '{ gsub("running", "\033[32m&\033[0m"); gsub("stopped", "\033[31m&\033[0m"); gsub("\\:[0-9]+", "\033[34m&\033[0m"); gsub("^/.+:", "\033[36m&\033[0m"); gsub("[0-9]+$", "\033[33m&\033[0m"); print }'
So you can use a script, that returns you colorful list of deployments without typing any URLs and regexps:
./show-status-of-applications-tomcat.sh

There is one other case, when script is even more convinient than web manager – if your environment is running on many servers and clusters (like production, again). In that case script can show information about all clusters and servers on one page:
I have separate script show-status-of-prod-applications-tomcat.sh for that, but it's possible to merge them into one file or modify it to work with parameters (for example, give a manager URL as a parameter).

All scripts are available in my GitLab repo gitlab.com/irina-ivanova/scripts and don't forget about aliases!

July 8, 2016

Bachelor Thesis "Version Update Automation Using Scripting Language Bash"


I've finally finished University of Tartu, so I am a Bachelor of Science in Engineering now!

My work was about version updater bash script, that I made on my work for updating Java applications on different web servers. I once wrote a post about first version of this script – Scripting For Automated Update (Tomcat 6) [DEPRECATED], – the final version has more features and much more code.

The thesis is written in Estonian and can be found in GitHub – Version Update Automation Using Scripting Language Bash.pdf (or in UT registry).

The script itself is open-source (currently only Tomcat 8 part) and can be used in other projects. It's located in GitHub repo – github.com/iriiiina/version-updater, the manual about install, configuration and usage is in GitBook – iriiiina.gitbooks.io/version-updater-manual.

And here are slides of defence (also in Estonian). I don't know why you may need them, but since we are talking about thesis I'll leave them here:

I feel proud that I got A for the thesis, but I don't have any good feeleings about finishing the university itself. Two years ago I wrote a post Secrets of a Buccaneer-Schoolar by James Marcus Bach where I explained my opinion about the university and hight schools, so it wasn't the priority for me. I decided to finish just to not loose already gotten points for finished courses – I've already passed 99% of the programm, so it would be a shame to spend that time for nothing.

June 17, 2016

Bash Scripts for Transfering Files Between Server and Local Computer


Yes, Bash scripts again! Two scripts for copying files from server to local computer and vice versa. Usually I use them if I want to work with some text files in graphical editors, not in Vim. So I can download the file to my local computer and then upload it back to the server.

./copy-file-from-server.sh [SERVER] [FILE]
./copy-file-to-server.sh [SERVER] [FILE]

Basically, it's the same as using scp command, but scripts allow you to define short names for servers. For example, instead of username@192.168.1.1, username@some.hostname.com or something1-23-45-678-9.eu-east-0.compute.amazonaws.com you can use short names, like test, demo, latest etc.

To set your own hosts in the script you need to modify function setHost(). In the copy-file-from-server.sh script you can also set a fixed directory where you want to download files. Currently it downloads to the same directory where script was run, but you can change it in the downloadFile() function.

Also, don't forget about aliases! You can set some short name to the script and use it in every directory.

June 7, 2016

Script for Sending E-mails About Certain JIRA Issues


"Still life with various Unix shells" by Bartholomeus van der Ast (source)


Recently I wrote a new Bash script, that can be useful for others.

./notify-about-issues.sh
It sends e-mail when some specific JIRA JQL query returns any result. You can create any JQL query you want: return issues that were created in specific project, return issues that were commented by some specific user, return issues where status was changed etc. Probably you want to limit the query by period to monitor only recent changes. This period can be the same as the frequency of the e-mail sending. For example, if you want to check this filter every 10 minutes, then you can check changes in last 10 minutes.

To implement this script you need to define following variables in the code:
$jiraUrl – URL of your JIRA
$filter – JQL query, encoded for URL (you can use some online URL encouder/decoder)
$user – JIRA username
$password – JIRA password (authentication is very primitive and not safe, but you can use some general credentials or keep the script file in safe place)
$to – e-mail where notification should be sent
$from – e-mail from whom notification should be sent (could be the same as to)
$subject – subject of the e-mail

Automatic running can be configured with crontab – read post 10 Mac OS X Terminal Commands for Testers for details.

The most complex part is to write a propper JQL query that returns exactly what you want. You can use Advanced searching documentation for complex queries or ask an advice in Atlassian Answers.

October 23, 2015

10 Mac OS X Terminal Commands for Testers


I really like terminals, command lines, bash scripting etc. They are handy and easy way to automate some routine tasks in my everyday work. Thats why after I saw article Eight Terminal Utilities Every OS X Command Line User Should Know I decided to write something like this, but for testers.

These are OS X commands, but some of them are also working on Linux systems and Windows (if use some UNIX-like terminal like Babun, CygWin or Cmder).

This is defently not the complete list. Here you can find the complete list of all OS X commands, but this post is about my favourite and frequently used (almost) commands.

1. alias
It's the most useful command of all – it allows to replace some command with a short word. That means, that if you have some long command that you regularly run in the terminal you can assign alias to it and use some simple short substitution. For example, you always monitor application logs using tail -1000f application.log, but instead of writing this whole string every day you can create alias like logs and write only 4 letters into the command line. Moreover, you can use autocomplete with aliases (TAB key), just like with regular commands. You can see my previous post Bash Scripts for Working With Documentation for more examples about using alias.

There is one trap that many beginners fall into – you can just type following command into your command line:
alias logs='tail -1000f application.log'
but in that case you create alias only for your current session. That means it'll disappear after you close it. To create permanent aliases you need to put them into specific file, which is usually ~/.bash_aliases. You may also need to add following code into your ~/.bash_profile file (you can add it just to the end):
# Aliases
if [ -f ~/.bash_aliases ]; then
. ~/.bash_aliases
fi
And you need to restart your session for taking these changes into use (just close-open you terminal or exit-login to server).

2. crontab
This one allows you to automate some commands. For example, you have some documentation on SVN (or some similar version control system) and you need to update it regulary to get the latest state. You can add following job into your cron schedule, which updates SVN directory every 10 minutes:
*/10 * * * * svn update ~/Documents/SVN-documentaion
To open crontab editor, where you can insert previous command, you need to run:
crontab -e
(or sudo crontab -e if you are not under admin user)
See Ubuntu CronHowTo article for understanding what are these asterisks mean.

3. grep
Grep is a very powerful tool for finding strings in any output (log files, query responses, script output etc), that you can perform. I use it to filter log files. For example, I can get all rows, where ERROR word is present:
tail -1000 application.log | grep ERROR
Or I can get all rows, where todays date is mentioned in the beginning of the row:
tail -1000 application.log | grep ^23.10.2015
Grep works with regular expressions, so you may need to understand them to perform some complex filtering, but simple search by one word may be enough for you.

4. find
Find searches files in directory. Again, in my previous post Bash Scripts for Working With Documentation I wrote about scripts for finding documents by name or by content – the whole funtionality is done by find there. So I use it in by daily work for finding documents with required data. You can find (ha-ha) a lot of examples in the following article: Find Command in Unix and Linux Examples.

5. open
Simply opens a file, a folder, an application or URL, just as if you had double-clicked the file's icon. It's useful if you first have done some complicated search of required file (using find), got the path of it and open it (with default or some other application).
Second useful case – opening directories (especially hidden ones). For example, open . opens current directory.

6. awk
AWK is actually programming language designed for text processing. It can be used as reporting tool and I usually use it to beautify my scripts' output – for example, to color some words. I am not sure that it's useful for everyday life, but if you want to write your own script, it better has meaningful output.

7. scp
It stands for Secure Copy. It is useful for somebody who works with servers (application updating, logs monitoring etc). With this command you can copy files from one server to other, or from server to your computer and vice versa. You can find some examples in Example syntax for Secure Copy (scp), I copy here one of them:
scp your_username@remotehost.edu:foobar.txt /some/local/directory

8. pbcopy / pbpaste
These two commands may save you from endless scrolling: pbcopy copies output to the clipboard, pbpaste inserts content from the clipboard. For example, you have some log file, that you want to investigate on your computer in you favorite editor. The simplest way is to copy its content to clipboard and paste it into another document. With these commands you don't need to scroll anything to select the text, or google "how to select the whole text in vi" or anything like this, just type:
cat application.log | pbcopy

9. screencapture
Honestly, I don't use this command in my daily work, but I have a feeling that it's important. It does what it says – captures the screenshot. You can set time delay to it and you can write a script for time lapse (taking screenshot every 10 seconds or something like this). It just happens, that I use screenshots quite rarely in my work, but I guess it can be interesting command for testers, who use them a lot.
You can see examples here: Take a screen capture from the command line.

10. sips
Last (and least) one is SIPS – Scriptable Image Processing System. You can use it for processing files, for example, resize all PNG files in current directory to 1024×768 (ignoring aspect ratio):
sips -z 768 1024 *.png
Again, I don't work with screenshots and picture files a lot, but it's the simplest and the fastest way (and cheapest one, comparing to some Photoshop) to process a large amount of picture files (or if you want to work with pictures in your bash script).



Don't forget the alias! You don't need to know all these commands by heart – you can just figure out how you can use them and which parameters suits for your case, put it into the alias thereafter use only short and nice commands.

BONUS!
telnet towel.blinkenlights.nl
This one is just for fun (once you've already opened the terminal). Some guys did pretty awsome ASCII version of Star Wars IV. You can run it over the Telnet. I don't know how long does it last – I've watched 15 minutes and Luke only meets Chubaka and Han Solo, so I guess they did the whole movie.

September 30, 2015

Bash Scripts for Working With Documentation

Recently I wrote a couple of bash scripts for working with specification documents. They solve sort of general problems, so I decided to share them here.

All scripts can be downloaded from my GitLab repo: gitlab.com/irina-ivanova/scripts

All scripts can be executed in UNIX terminal (for Windows users there are alternatives such as Babun, Cygwin, Cmder). I also recommend to use aliases: just pick up short and unique command, specify the path to the script and you can excecute this script with short command from wherever you are (like spec bug.01.1 for finding specification with such text in the title).


find-specification-by-name.sh
./find-specification-by-name.sh [text_in_file_title]
Script looks for .doc files (extension can be changed) in a specific directory (which can be also changed) where title contains text, that user gives to the script.
In the example I have directory specifications with 4 documents, but only 3 of them contain word bugs in the title. There is also directory Archive with old documents, which script also finds, but shows Archive folder in red, so user can easily differ old documents from new ones (you can also change words that should be colored).

Rows, that you may want to change to use this script in your way:
  • 6 path="/Users/irina/Desktop/specs" – here you should specify the absolute path (not relative, because you want to execute this script from different places) of the directory, where you have all your specifications (usually it's CVS of SVN root directory).
  • 31 find $path -iname "*$text*" | awk -v text="$text" -v lower="$lower" -v upper="$upper" '{ gsub(text, "\033[36m&\033[0m"); gsub(lower, "\033[36m&\033[0m"); gsub(upper, "\033[36m&\033[0m"); gsub("Arhiiv", "\033[31m&\033[0m"); print }' – the awk part is responsible for coloring. Codes like \033[36m are colors – particualary this one means cyan, but you can change it. Also you need to change word "Arhiiv", which helps to differ old documents. Or you can remove this part, if you don't want to color anything.

open-last-specification.sh
./open-last-specification.sh [text_in_file_title]
This one is doing basically the same as the previous one, except that it automatically opens lastly modified file. You can use it, if your components have unique identifiers and specs about these components have many versions, like this:
Rows that you may want to change:
  • 6 path="/Users/irina/Desktop/specs" – same as in the previous one, absolute path to directory with your specs.
  • 43 function checkIsFileIsDoc(){} – in my case I want to open only .doc files – I don't want to open folders or excel files, but you may want to remove this check.

find-text-in-specification.sh
./find-text-in-specification.sh [text_in_the_file_content]
This one is also very similar to first one, only it finds files with given text in the content.
For example, I want to know in which specification is mentioned data base column BUG.STATUS:
Rows that you may need to change are the same as in the first one.

nortal-logo.sh
I always wanted to print some pictures with bash script. So here are my first experiences.
Logo of company where I work. It was picked as easy to reproduce, because it has strict lines.

bug.sh