All opinions expressed are those of the authors and not necessarily those of OSNews.com, our sponsors, or our affiliates.
  Add to My Yahoo!  Subscribe with Bloglines  Subscribe in NewsGator Online

published on 2017-11-11 21:00:07

published on 2017-11-11 21:00:07

published on 2017-11-11 21:00:07

published on 2017-11-11 21:00:07

published on 2017-11-11 21:00:07

published on 2017-11-11 21:00:07

published on 2017-11-11 21:00:07

published on 2017-11-11 21:00:07

published on 2017-11-11 21:00:07

published on 2017-11-11 21:00:07

published on 2017-11-11 21:00:07

published on 2017-11-11 21:00:07

published by noreply@blogger.com (Joe Marrero) on 2017-10-31 02:37:00 in the "c" category

Try It Out Yourself

The code for this project is hosted on Github and can be cloned from here.

At End Point Corporation, our team is spread out across 10 time zones. This gives us the advantage of being able to work around the clock on projects. When one co-worker leaves for day, another can take over. Consider this scenario. It's Monday evening and Martin needs to continue installing software on that Linux cluster, but it's already 6pm and his wife is going to murder him if he's not ready to go out for their anniversary dinner. Let's see who can take over... Ah, yes, Sanjay in Bangalore can continue with the maintenance. Tuesday morning, the client wakes up to be surprised that 16 hours of work was completed in a day. With respect to software development, the same efficiencies can be realized by parallelizing tasks across time-zones. Code reviews and further development can be continued after normal business hours.

With all the blessings of a distributed engineering team, collaborating with co-workers can be, occasionally, challenging. Some of these challenges stem from complexities of our system of time. Every co-worker may be operating in a timezone that is different than yours. Time-zones have an associated offset relative to Coordinated Universal Time (UTC). These offsets are usually in whole hour increments but they may be any real-valued number.

For example, Eastern Standard Time (EST) has an offset of -5 (five hours behind UTC) and Indian Standard Time (IST) has an offset of 5.5 (five and half hours ahead of UTC). Furthermore, these UTC offsets can be completely arbitrary. In 1995, Kiribati, an island nation in the Pacific, changed its UTC offset from -10 to +14 so that all of its outlying islands can share the same time. To further complicate things, some regions may not observe daylight savings time (DST) while other regions do. In fact, in the United States, Indiana started observing DST on April 2, 2006. Some states like Arizona and Hawaii do not observe DST. Other countries, like Australia, have a similar situation where it's left to local governments to decide whether DST is observed or not. Moreover, although DST usually accounts for adding or subtracting an hour of time, it isn't always one hour. This has historically changed from time to time.

Now you may begin to imagine the headaches that arise when you need to coordinate with anything involving multiple time-zones. To make all of this easier, you can use a utility that we wrote to do all the time conversions for you. First, you have to add each co-worker's information to a configuration file stored at ~/.timezoner. This configuration file will describe all of your co-worker's contact information and their associated IANA time-zone. As an example, this is what the configuration file looks like:

# Timezone            Email                  Name              OfficePhone        MobilePhone
America/New_York      "edward@example.com"   "Edward Teach "   "n/a"              "+1 731 555 1234"
America/New_York      "henry@dexample.com"   "Henry Morgan"    "+1 646 555 5678"  "+1 954 555 5678"
America/New_York      "john@example.com"     "John Auger"      "n/a"              "+1 902 555 1234"
America/Denver        "sam@example.com"      "Samuel Bellamy"  "+1 347 535 1234"  "+1 994 555 5678"
America/Los_Angeles   "william@example.com"  "William Kidd"    "+1 330 555 5678"  "+1 305 555 1234"
America/Los_Angeles   "israel@example.com"   "Israel Hands"    "+1 507 555 1234"  "+1 208 555 5678"

Now when I need to coordinate a meeting, I can run the utility with the -T option to see each team member's local time.

With the -U option, you can display each contact separated in groups based on UTC offset.

Let us know what you think and if you found this tool helpful.


Comments

published by noreply@blogger.com (Peter Hankiewicz) on 2017-10-27 23:00:00 in the "hot-deployment" category

Introduction

Java development can be really frustrating when you need to re-build your project and restart a server every time you change something. I know about JRebel, but while it’s a good tool, it’s also pretty expensive. You can use the open-source version, but then you need to send project statistics to the JRebel server, which is not a viable option for your more serious projects.

Fortunately, there is an open-source project called HotSwapAgent and it does the same thing as JRebel, for free (thank you, guys!).

I will explain how to combine it with Widlfly in order to hot-deploy Java classes as well as how to hot-deploy other resources (Javascript, CSS, images).

Wildfly configuration

Let’s assume that we use the standalone-full.xml configuration file.

We need to use exploded deployment instead of deploying WAR or EAR. You can do this in production as well to allow for application changes with zero downtime.

Start by configuring the metaspace size; we had to increase defaults for our application, but it’s possible that it will be just fine in your case. It’s encouraged that you play with these values after completing all steps.

In:

WILDFLY_DIR/bin/standalone.conf

set:

-XX:MetaspaceSize=256M -XX:MaxMetaspaceSize=512m

so it looks like this:

JAVA_OPTS="-Xms512m -Xmx1024m -XX:MetaspaceSize=256M -XX:MaxMetaspaceSize=512m".

Now, look for the deployment-scanner node in:

WILDFLY_DIR/standalone/configuration/standalone-full.xml

Replace it so it looks like this:

<deployment-scanner path="PATH_TO_DEPLOYMENT_DIR" relative-to="RELATIVE_TO_PATH" scan-enabled="true" scan-interval="2000" auto-deploy-exploded="false" runtime-failure-causes-rollback="${jboss.deployment.scanner.rollback.on.failure:false}"/>

Note:

PATH_TO_DEPLOYMENT_DIR is WILDFLY_DIR/standalone/deployments

RELATIVE_TO_PATH is, as the name suggests the dir that the PATH_TO_DEPLOYMENT_DIR is relative to.

HotSwapAgent installation and configuration

We need to download and install the latest release of DCEVM Java patch from here: https://github.com/dcevm/dcevm/releases . Why it’s needed? It will allow us unlimited redefinition of loaded classes at runtime. This is not possible with the original Java HotSpot VM. Make sure you update to the same Java version that you’re going to use to run the Wildfly server.

Now, download the latest release of the Hotswap agent from here:

https://github.com/HotswapProjects/HotswapAgent/releases .

The only thing that you need to do is get the JAR and put it anywhere on your hard drive (I recommend to add it to your Java project).

Ok, great, now just some configuration.

Open:

WILDFLY_DIR/bin/standalone.conf

and add new Java opts:

-XXaltjvm=dcevm -javaagent:PATH_TO_HOTSWAPAGENT_JAR.

What does this do?

  • The altjvm option sets an alternative Java Virtual Machine.
  • The javagent is just an interceptor on the top of your classes that allows the HotSwapAgent library to manipulate your code on the fly.

That’s all you need. It’s a good idea to create a configuration file for the HotSwapAgent. This is well explained here:

http://hotswapagent.org/mydoc_configuration.html

Basically create a new file, name it hotswap-agent.properties, set all needed configuration inside and add it to the classpath of the application.

If you use Netbeans, Eclipse or Intellij you should check the HotSwapAgent page for some helpful plugins here: http://hotswapagent.org/mydoc_setup_intellij_idea.html , http://hotswapagent.org/mydoc_setup_eclipse.html  and http://hotswapagent.org/mydoc_setup_netbeans.html .

Application configuration

Now that we have everything in place, I will explain how to put it all together.

I doesn’t really matter which build-tool you use (Ant, Gradle or Maven). The process should look like this (you can do it in many ways, in our case, it’s pretty specific as our build process is really complicated):

  1. Build your application and deploy it to the PATH_TO_DEPLOYMENT_DIR in the exploded version,
  2. Create a script that will look for changes in the application directory (this one is interesting: https://gist.github.com/peter-hank/3ecf7fc285ba4b9c50cf8cace1badaf4 ),
  3. On change, trigger a job that will:
    1. Copy all resources like JSP, JavaScript, CSS and copy to the PATH_TO_DEPLOYMENT_DIR,
    2. Compile classes and copy them to the PATH_TO_DEPLOYMENT_DIR.

That’s it, after you replace files in the PATH_TO_DEPLOYMENT_DIR HotSwapAgent and Wildfly will do the rest really fast. We have a ton of assets and classes and the whole process takes only a few seconds!

Summary

I feel this process is really worth doing. It doesn’t take a lot of time to configure everything and saves a lot of manual work. Just multiply the number of manual deployments and the number of developers in your team and you understand how much time you lose everyday without hot-deployment.

From now on, focus on development, forget about deployment!

Lastly, good luck!


Comments

published by noreply@blogger.com (Greg Sabino Mullane) on 2017-10-23 10:30:00 in the "postgres" category

(Photo of Turtle Island by Edwin Poon)

Someone recently asked me something about tail_n_mail, a program that watches over your log files, scans for certain patterns, and sends out an email if matches are found. It is frequently used to watch over Postgres logs so you can receive an automatic email alert when Bad Things start happening to your database. The questioner wanted to know if it was possible for tail_n_mail to change its behavior based on the time of day - would it be able to do things differently outside of "business hours"? Although tail_n_mail cannot do so directly, a simple solution is to use alternate configuration files - which get swapped by cron - and the INHERIT keyword.

To demonstrate the solution, let's spin up a Postgres 10 instance, route the logs to syslog, setup tail_n_mail, and then create separate configuration files for different times of the week. First, some setup:

$ initdb --version
initdb (PostgreSQL) 10.0
$ initdb --data-checksums data
$ cat >> data/postgresql.conf << EOT
log_line_prefix=''
log_destination='syslog'
EOT
$ echo 'local0.*  /var/log/postgres.log' | sudo tee -a /etc/rsyslog.conf > /dev/null
$ sudo systemctl restart rsyslog
$ pg_ctl start -D data -l logfile

Grab the latest version of tail_n_mail and verify it:

$ wget --no-verbose https://bucardo.org/downloads/tail_n_mail{,.asc} 
2017-03-03 10:00:33 URL:https://bucardo.org/downloads/tail_n_mail [98767/98767] -> "tail_n_mail" [1]
2017-03-03 10:00:33 URL:https://bucardo.org/downloads/tail_n_mail.asc [163/163] -> "tail_n_mail.asc" [1]
FINISHED --2017-03-03 10:00:33--
Total wall clock time: 0.3s
Downloaded: 2 files, 96K in 0.1s (702 KB/s)
$ gpg --verify tail_n_mail.asc
gpg: assuming signed data in `tail_n_mail'
gpg: Signature made Sun 01 Oct 2017 11:14:07 AM EDT using DSA key ID 14964AC8
gpg: Good signature from "Greg Sabino Mullane "
gpg:                 aka "Greg Sabino Mullane (End Point Corporation) "
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: 2529 DF6A B8F7 9407 E944  45B4 BC9B 9067 1496 4AC8

The main way to configure tail_n_mail is through its configuration file, which is always the first argument given to the program. This file describes where the log files are, what to look for, and a few other important items. In addition, it automatically updates itself each time tail_n_mail is run to keep track of where the last run left of, so the next run can start at the exact same file, and the correct place within that file. In this example, let's assume the DBA wants to get email for every error that pops up in the database (in practice, this means any severity levels that are ERROR, FATAL, or PANIC). The configuration file would look like this:

$ cat > tnm.conf << EOT
FILE: /var/log/postgres.log
PGLOG: syslog
EMAIL: greg@example.com
MAILSUBJECT: HOST Postgres errors NUMBER
INCLUDE: PANIC:
INCLUDE: FATAL:
INCLUDE: ERROR:
## Okay, we don't want to get emailed on *every* error:
EXCLUDE: could not serialize access due to concurrent update
EXCLUDE: canceling statement due to user request
EOT

To test it out, we will generate some errors, and then run tail_n_mail from the command line. If all goes well, it sends out an email and then rewrites the configuration file to indicate how far along it got. The --dry-run option can be used to view the email without actually sending it.

$ for i in 2 4 6 8; do psql -tc "select $i/0"; done
ERROR:  division by zero
ERROR:  division by zero
ERROR:  division by zero
ERROR:  division by zero
$ perl tail_n_mail tnm.conf --dry-run
Subject: localhost.localdomain Postgres errors 4
Auto-Submitted: auto-generated
Precedence: bulk
X-TNM-VERSION: 1.30.0
To: greg@example.com

Date: Tue Oct  3 03:19:22 2017 EDT
Host: localhost.localdomain
Unique items: 1
Matches from /var/log/postgres.log: 4

[1] (between lines 139 and 142, occurs 4 times)
First: Oct   3 03:19:00 localhost postgres[28483]: [6-1]
Last:  Oct   3 03:19:00 localhost postgres[28495]: [6-1]
ERROR: division by zero
STATEMENT: select ?/0
-
ERROR: division by zero
STATEMENT: select 2/0

  DRYRUN: /usr/sbin/sendmail 'greg@example.com' < tnmBWaG6QA1.tnm2

Running it in normal mode rewrites the configuration file:

$ perl tail_n_mail tnm.conf
$ cat tnm.conf
## Config file for the tail_n_mail program
## This file is automatically updated
## Last updated: Mon Oct  2 12:09:29 2017
PGLOG: syslog
EMAIL: greg@example.com
MAILSUBJECT: HOST Postgres errors NUMBER

INCLUDE: PANIC:
INCLUDE: FATAL:
INCLUDE: ERROR:
## Okay, we don't want to get emailed on *every* error:
EXCLUDE: could not serialize access due to concurrent update
EXCLUDE: canceling statement due to user request

FILE1: /var/log/postgres.log
LASTFILE1: /var/log/postgres.log
OFFSET1: 333

Note how the file was rewritten to include state information about the files we are tracking, but leaves the exclusion rules and their comments in place. Tail_n_mail also attempts to "flatten" similar queries, which is why the four division-by-zero errors all appear as "SELECT ?/0". A sample of one of the literal errors appears below the normalized version.

You are not limited to a single configuration file, however, as the main config file can read in other configuration files via the INHERITS keyword. This allows you to import one or more other configuration files. Not only does this allow different tail_n_mail invocations to share common items to search for, but (as you will see in a bit) can solve the problem at the top of this post: how to change what is being looked for based on the time of day.

Using INHERITS also allows us to store files in version control, without worrying about them getting rewritten on each invocation, as we can store the ephemeral data in one file, and the constant data in a separate, version controlled file. Let's apply that idea to our example:

$ cat > tnm.global.conf << EOT
PGLOG: syslog
EMAIL: greg@example.com
INCLUDE: PANIC:
INCLUDE: FATAL:
INCLUDE: ERROR:
## Okay, we don't want to get emailed on *every* error:
EXCLUDE: could not serialize access due to concurrent update
EXCLUDE: canceling statement due to user request
EOT
$ git add tnm.global.conf && git commit tnm.global.conf 
  -m "Global config for tail_n_mail"
[master 2441df8] Global config for tail_n_mail
 1 file changed, 7 insertions(+)
 create mode 100644 tnm.global.conf
$ cat > tnm.conf << EOT
FILE: /var/log/postgres.log
MAILSUBJECT: HOST Postgres errors NUMBER
INHERIT: tnm.global.conf
EOT

After another run, we observe that the inherited file does not change:

$ perl tail_n_mail tnm.conf
$ git status
On branch master
nothing to commit, working tree clean
$ cat tnm.conf

## Config file for the tail_n_mail program
## This file is automatically updated
## Last updated: Mon Oct  2 12:28:10 2017
MAILSUBJECT: HOST Postgres errors NUMBER

INHERIT: tnm.global.conf

FILE1: /var/log/postgres.log
LASTFILE1: /var/log/postgres.log
OFFSET1: 13219

Another advantage to moving common items to another file is that we can run multiple tail_n_mails, with slightly different purposes, but all sharing some of the same underlying rules. A common usage is to get an immediate email about almost all database problems, as well as a daily report about all problems. To do this, we create two configuration files and set them up in cron:

$ cp tnm.conf tnm.fatals.conf
$ mv tnm.conf tnm.errors.conf
$ perl -pi -e 's/Postgres errors/Postgres fatals/' tnm.fatals.conf
$ crontab -e
## Every five minutes, check for important problems
*/5 * * * * perl tail_n_mail tnm.fatals.conf
## Once every morning, generate a report of all errors in the last 24 hours.
30 6 * * * perl tail_n_mail tnm.errors.conf
## Note: it is usually easier to have separate "fatals" and "errors" exclusions.

What if we want to change the rules depending on the time of day, per the question that started this article? Simple enough - we just create two "inherited" configuration files, then have cron swap things around as needed. For example, let's say that after 5pm on weekdays, and all weekend, we do not want to receive emails about "division by zero" errors. First, create files named tnm.global.hometime.conf and tnm.global.workday.conf:

$ cp tnm.global.conf tnm.global.workday.conf
$ cp tnm.global.conf tnm.global.hometime.conf
$ ln -sf tnm.global.workday.conf tnm.global.conf
$ echo "EXCLUDE: ERROR:  division by zero" >> tnm.global.hometime.conf

Finally, have cron swap the files around at the start and end of business hours:

$ crontab -e
## May need to use 1-5 instead of Mon-Fri on some systems
0 9 * * Mon-Fri ln -sf tnm.global.workday.conf tnm.global.conf
0 17 * * Mon-Fri ln -sf tnm.global.hometime.conf tnm.global.conf

Voila! We've changed the way tail_n_mail runs depending on the time of day. There are many other tricks you can do with tail_n_mail - check out the documentation or post to the mailing list for more help and/or inspiration.


Comments