All opinions expressed are those of the authors and not necessarily those of, our sponsors, or our affiliates.
  Add to My Yahoo!  Subscribe with Bloglines  Subscribe in NewsGator Online

published by (Wojtek Ziniewicz) on 2017-08-14 13:49:00 in the "git" category

Ever wondered how to split your Git repo into two repos?


First you need to find out what files and directories you want to move to separate repos. In the above example we're moving dir3, dir4 and dir7 to repo A, and dir1, dir2, dir5 and dir8 to repo B.


What you need to do is to go through each and every commit in git history for every branch and filter out commits that modify directories that you dont care about in your new repo. The only flaw of this method is that it will leave those empty, filtered out commits in the history.

Track all branches

First we need to start tracking all branches locally:

for i in $(git branch -r | grep -vE "HEAD|master" | sed 's/^[ ]+//');
  do git checkout --track $i

Then copy your original repo to two separate dirs: repo_a and repo_b.

cp -a source_repo repo_a
cp -a source_repo repo_b

Filter the history

Following command will delete all dirs that exclusively belong to repo B, thus we create repo A. Filtering is not limited to directories. You can provide relative paths to files, dirs etc.

cd repo_a
git filter-branch --index-filter 'git rm --cached -r dir8, dir2 || true' -- --all

cd repo_b
git filter-branch --index-filter 'git rm --cached -r dir3, dir4, dir7 || true' -- --all

Note that the `|| true` prevents git from failing to filter our dirs mentioned in the `rm` clause in early stages of the git history where the dirs did not yet exist.

Look at the list of branches once again (in both repos):

git branch -l

Set new origins and push

In every repo, we need to remove the old origin and set up new origin. After it's done, we're ready to push.

Remove old origin:

git remote rm origin

Add new origin:

git remote add origin

Push all tracked branches:

git push origin --all

That's it!


published by (Muhammad Najmi Ahmad Zabidi) on 2017-08-08 07:21:00 in the "bash" category

In certain cases we might want to get a certain process' elapsed time for our own reason. Turns out "ps" command could easily assist us in that. According to "ps" manual, etime could put the duration of time in [[DD-]hh:]mm:ss. format, while etimes in seconds.

From "ps" manpage:

etime       ELAPSED   elapsed time since the process was started, in the form [[DD-]hh:]mm:ss.
etimes      ELAPSED   elapsed time since the process was started, in seconds.

To use that, we could use (in [[DD-]hh:]mm:ss. format):

ps -p "pid" -o etime
or in seconds:
ps -p "pid" -o etimes

In this case the "pid" should be replaced with your intended process ID.

The following will help to nicely reporting the output. We can put -o etime or -o etimes with other argument, that is "command", in order to show the executed command along with its very own absolute path:

ps -p "28590" -o etime,command
21:45 /usr/bin/perl ./ 7999

We can also get the start date of the process' execution:

najmi@ubuntu-ampang:~$ ps -p 21745 -o etime,command,start
    ELAPSED COMMAND                      STARTED
 1-19:47:45 /usr/lib/firefox/firefox      Aug 02

What if we do not want to manually parsing the PID, instead (since we are very sure) to just get the name of the running application? We could just simply use pgrep or pidof

najmi@ubuntu-ampang:~$ ps -p $(pgrep firefox) -o pid,cmd,start,etime
  PID CMD                          STARTED     ELAPSED
21745 /usr/lib/firefox/firefox      Aug 02  2-04:29:36

najmi@ubuntu-ampang:~$ ps -p $(pidof firefox) -o pid,cmd,start,etime
  PID CMD                          STARTED     ELAPSED
21745 /usr/lib/firefox/firefox      Aug 02  2-04:29:42

What if the command issued many processes? Take an example of the Chrome browser:
najmi@ubuntu-ampang:~$ ps -p $(pidof chrome) -o pid,comm,cmd,start,etime
error: process ID list syntax error

 ps [options]

 Try 'ps --help '
  or 'ps --help '
 for additional help text.

For more details see ps(1).

The best way (so far) that I could get is by creating a loop. It seems pidof is much more accurate when parsing the exact application (string) that we feed into it.

With pgrep:
najmi@ubuntu-ampang:~$ for i in `pgrep chrome`;do ps -p $i -o pid,comm,cmd,start,etime|tail -n +2;done
 2255 chrome          /opt/google/chrome/chrome - 08:05:43    02:55:17
 4990 chrome          /opt/google/chrome/chrome - 10:39:16       21:44
 5567 chrome          /opt/google/chrome/chrome - 10:53:13       07:47
 9448 chrome          /opt/google/chrome/chrome -   Jul 31  3-12:25:08
10033 chrome          /opt/google/chrome/chrome     Jul 27  8-10:43:42
10044 chrome          /opt/google/chrome/chrome -   Jul 27  8-10:43:42
10050 chrome          /opt/google/chrome/chrome -   Jul 27  8-10:43:42
10187 chrome          /opt/google/chrome/chrome -   Jul 27  8-10:43:39
10234 chrome          /opt/google/chrome/chrome -   Jul 27  8-10:43:37
10236 chrome          /opt/google/chrome/chrome -   Jul 27  8-10:43:37
19440 chrome          /opt/google/chrome/chrome - 22:30:34    12:30:26
20229 chrome          /opt/google/chrome/chrome -   Aug 03  1-09:51:25
20514 chrome          /opt/google/chrome/chrome - 22:52:25    12:08:35
20547 chrome          /opt/google/chrome/chrome - 22:52:36    12:08:24
21009 chrome          /opt/google/chrome/chrome -   Aug 03  1-09:27:11
22458 chrome          /opt/google/chrome/chrome -   Jul 27  8-03:44:07
22474 chrome-gnome-sh /usr/bin/python3 /usr/bin/c   Jul 27  8-03:44:07
23681 chrome          /opt/google/chrome/chrome -   Aug 03  1-03:33:45
23691 chrome          /opt/google/chrome/chrome -   Aug 03  1-03:33:45
23870 chrome          /opt/google/chrome/chrome - 00:15:15    10:45:45
24544 chrome          /opt/google/chrome/chrome - 00:40:17    10:20:43
25116 chrome          /opt/google/chrome/chrome - 00:51:31    10:09:29
25466 chrome          /opt/google/chrome/chrome - 00:59:55    10:01:05
29060 chrome          /opt/google/chrome/chrome - 02:15:42    08:45:18
With pidof:
najmi@ubuntu-ampang:~$ for i in `pidof chrome`;do ps -p $i -o pid,comm,cmd,start,etime|tail -n +2;done
29060 chrome          /opt/google/chrome/chrome - 02:15:42    08:47:40
25466 chrome          /opt/google/chrome/chrome - 00:59:55    10:03:27
25116 chrome          /opt/google/chrome/chrome - 00:51:31    10:11:51
24544 chrome          /opt/google/chrome/chrome - 00:40:17    10:23:05
23870 chrome          /opt/google/chrome/chrome - 00:15:15    10:48:07
23691 chrome          /opt/google/chrome/chrome -   Aug 03  1-03:36:07
23681 chrome          /opt/google/chrome/chrome -   Aug 03  1-03:36:07
22458 chrome          /opt/google/chrome/chrome -   Jul 27  8-03:46:29
21009 chrome          /opt/google/chrome/chrome -   Aug 03  1-09:29:33
20547 chrome          /opt/google/chrome/chrome - 22:52:36    12:10:46
20514 chrome          /opt/google/chrome/chrome - 22:52:25    12:10:57
20229 chrome          /opt/google/chrome/chrome -   Aug 03  1-09:53:47
19440 chrome          /opt/google/chrome/chrome - 22:30:34    12:32:48
10236 chrome          /opt/google/chrome/chrome -   Jul 27  8-10:45:59
10234 chrome          /opt/google/chrome/chrome -   Jul 27  8-10:45:59
10187 chrome          /opt/google/chrome/chrome -   Jul 27  8-10:46:01
10050 chrome          /opt/google/chrome/chrome -   Jul 27  8-10:46:04
10044 chrome          /opt/google/chrome/chrome -   Jul 27  8-10:46:04
10033 chrome          /opt/google/chrome/chrome     Jul 27  8-10:46:04
 9448 chrome          /opt/google/chrome/chrome -   Jul 31  3-12:27:30
 5567 chrome          /opt/google/chrome/chrome - 10:53:13       10:09
 4990 chrome          /opt/google/chrome/chrome - 10:39:16       24:06
 2255 chrome          /opt/google/chrome/chrome - 08:05:43    02:57:39

There is other tool, called as stat which records the timestamp of a file but for slightly different purpose. Stay tune for the next blogpost!


published by (Peter Hankiewicz) on 2017-07-31 23:00:00 in the "Java" category


Recently, we have taken over a big Java project that ran on the old JBoss 4 stack. As we know how dangerous for a business is outdated software, we and our client agreed that the most important task is to upgrade the server stack to the latest WildFly version.

It?s definitely not an easy job, but it?s worth to invest to sleep well and don?t worry about software problems.

This time it was even more work because of a complicated and not documented application, that?s why I wanted to share some tips and problem resolutions for issues I encountered.

Server configuration

You can set it up using multiple configuration files in the standalone/configuration directory.

I can recommend to use the standalone-full.xml file for most of setup, it contains a default full stack as opposed to standalone.xml.

You can also set up an application specific configuration using various configuration XML files ( to keep the appliction specific configuration in the Classpath.

Quartz as a message queue

The Quartz library was used as a native message queue in previous JBoss versions. If you struggle and try to use its resource adapter with WildFly just skip it. It?s definitely too much work, even if it?s possible.

In the latest WildFly version (as of today 10.1) the default message queue library is ActiveMQ. It has almost the same API as the old Quartz has, so it?s easy to use it.

Quartz as a job scheduler

We had multiple cron-like jobs to migrate as well. All the jobs used Quartz to schedule runs.

The best solution here is to update Quartz to the latest version (yes!) and use a new API ( to create CronTriggers for the jobs.

trigger = newTrigger()
    .withIdentity("trigger3", "group1")
    .withSchedule(cronSchedule("0 42 10 * * ?"))

You can use the same cron syntax (e.g 0 42 10 * * ?) as in the 12 years old Quartz version. Yes!

JAR dependencies

In WildFly you can set up an internal module for each JAR dependency. It can be pretty time consuming to create declarations for more than 100 libraries (exactly 104 in our case). We decided to use Maven to handle dependencies of our application and skip declaring modules in WildFly. Why? In our opinion it?s better to encapsulate everything in an EAR file and keep WildFly configuration minimal as we won?t use our server for any other application in the future.

Just keep your dependencies in the Classpath and you will be fine.


I really prefer the bin/ interface to the web interface to handle deployments. It?s a powerful tool and much faster to work with than clicking through the UI.

JNDI path

If you can?t access your JNDI definition try to use the global namespace. Up to JAVA EE6 developers defined their own JNDI names. These names had a global scope. This doesn?t work anymore. To access a previously globally scoped name use this pattern: java:global/OLD_JNDI_NAME.

The java:global namespace was introduced in JAVA EE 6.

Reverse proxy

To configure a WildFly application with a reverse proxy you need to, of course, set up a virtual host with a reverse proxy declaration.

In addition, you must add an attribute to the server?s http-listener in the standalone-full.xml file. The attribute is proxy-address-forwarding and must be set to true.

Here is an example declaration:



If you consider to upgrade to WildFly I can recommend it, it?s much faster than JBoss 4/5/6, scalable and fully prepared for modern applications.


published by (Elizabeth Garrett Christensen on 2017-07-27 03:53:00

co:collective is a growth accelerator that works with leadership teams to conceive and execute innovation in the customer experience using a proprietary methodology called StoryDoing©.

Doable, one of co:collective?s recent innovations, is a cloud based platform, designed to empower employees to meaningfully contribute and collaborate on ideas that move their business forward. The tool allows companies to solicit ideas from employees at all levels of an organization, filter down those ideas, make decisions as a team, and then implement a project - all the while collaborating in a fun and easy-to-use application. Over 200 companies across multiple sectors use Doable to create new products, new features, and problem solve to keep their business growing.

co:collective engaged End Point?s front-end developer, Kamil Ciemniewski, to work with their in-house development team led by Tommy Dunn. Kamil joined the Doable effort to refactor the Doable application which was moving from a Ruby on Rails-based application to an Angular frontend application with a Ruby on Rails backend. Kamil has been working with Doable since March to complete the application re-write and the project has gone immensely well. Kamil brings extensive frontend and backend knowledge to co:collective and together they?ve been able to refactor their site to be more efficient and powerful than before.


published by (Greg Sabino Mullane) on 2017-07-22 00:16:00 in the "postgres" category

Upgrading Postgres is not quite as painful as it used to be, thanks primarily to the pg_upgrade program, but there are times when it simply cannot be used. We recently had an existing End Point client come to us requesting help upgrading from their current Postgres database (version 9.2) to the latest version (9.6 - but soon to be 10). They also wanted to finally move away from their SQL_ASCII encoding to UTF-8. As this meant that pg_upgrade could not be used, we also took the opportunity to enable checksums as well (this change cannot be done via pg_upgrade). Finally, they were moving their database server to new hardware. There were many lessons learned and bumps along the way for this migration, but for this post I'd like to focus on one of the most vexing problems, the database encoding.

When a Postgres database is created, it is set to a specific encoding. The most common one (and the default) is "UTF8". This covers 99% of all user's needs. The second most common one is the poorly-named "SQL_ASCII" encoding, which should be named "DANGER_DO_NOT_USE_THIS_ENCODING", because it causes nothing but trouble. The SQL_ASCII encoding basically means no encoding at all, and simply stores any bytes you throw at it. This usually means the database ends up containing a whole mess of different encodings, creating a "byte soup" that will be difficult to sanitize by moving to a real encoding (i.e. UTF-8).

Many tools exist which convert text from one encoding to another. One of the most popular ones on Unix boxes is "iconv". Although this program works great if your source text is using one encoding, it fails when it encounters byte soup.

For this migration, we first did a pg_dump from the old database to a newly created UTF-8 test database, just to see which tables had encoding problems. Quite a few did - but not all of them! - so we wrote a script to import tables in parallel, with some filtering for the problem ones. As mentioned above, iconv was not particularly helpful: looking at the tables closely showed evidence of many different encodings in each one: Windows-1252, ISO-8859-1, Japanese, Greek, and many others. There were even large bits that were plainly binary data (e.g. images) that simply got shoved into a text field somehow. This is the big problem with SQL_ASCII: it accepts *everything*, and does no validation whatsoever. The iconv program simply could not handle these tables, even when adding the //IGNORE option.

To better explain the problem and the solution, let's create a small text file with a jumble of encodings. Discussions of how UTF-8 represents characters, and its interactions with Unicode, are avoided here, as Unicode is a dense, complex subject, and this article is dry enough already. :)

First, we want to add some items using the encodings 'Windows-1252' and 'Latin-1'. These encoding systems were attempts to extend the basic ASCII character set to include more characters. As these encodings pre-date the invention of UTF-8, they do it in a very inelegant (and incompatible) way. Use of the "echo" command is a great way to add arbitrary bytes to a file as it allows direct hex input:

$ echo -e "[Windows-1252]   Euro: x80   Double dagger: x87" >
$ echo -e "[Latin-1]   Yen: xa5   Half: xbd" >> 
$ echo -e "[Japanese]   Ship: xe8x88xb9" >>
$ echo -e "[Invalid UTF-8]  Blob: xf4xa5xa3xa5" >> 

This file looks ugly. Notice all the "wrong" characters when we simply view the file directly:

$ cat
[Windows-1252]   Euro: �   Double dagger: �
[Latin-1]   Yen: �   Half: �
[Japanese]   Ship: ? 
[Invalid UTF-8]  Blob: ����

Running iconv is of little help:

## With no source encoding given, it errors on the Euro:
$ iconv -t utf8 >/dev/null
iconv: illegal input sequence at position 23 

## We can tell it to ignore those errors, but it still barfs on the blob:
$ iconv -t utf8//ignore >/dev/null
iconv: illegal input sequence at position 123

## Telling it the source is Window-1252 fixes some things, but still sinks the Ship:
$ iconv -f windows-1252 -t utf8//ignore 
[Windows-1252]   Euro: €   Double dagger: ‡
[Latin-1]   Yen: ¥   Half: ½
[Japanese]   Ship: è?¹
[Invalid UTF-8]  Blob: ô¥£¥

After testing a few other tools, we discovered the nifty Encoding::FixLatin , a Perl module which provides a command-line program called "fix_latin". Rather than being authoritative like iconv, it tries its best to fix things up with educated guesses. Its documentation gives a good summary:

  The script acts as a filter, taking source data which may contain a mix of
  ASCII, UTF8, ISO8859-1 and CP1252 characters, and producing output will be
  all ASCII/UTF8.

  Multi-byte UTF8 characters will be passed through unchanged (although
  over-long UTF8 byte sequences will be converted to the shortest normal
  form). Single byte characters will be converted as follows:

    0x00 - 0x7F   ASCII - passed through unchanged 
    0x80 - 0x9F   Converted to UTF8 using CP1252 mappings
    0xA0 - 0xFF   Converted to UTF8 using Latin-1 mappings

While this works great for fixing the Windows-1252 and Latin-1 problems (and thus accounted for at least 95% of our table's bad encodings), it still allows "invalid" UTF-8 to pass on through. Which means that Postgres will still refuse to accept it. Let's check our test file:

$ fix_latin
[Windows-1252]   Euro: €   Double dagger: ‡
[Latin-1]   Yen: ¥   Half: ½
[Japanese]   Ship: ?
[Invalid UTF-8]  Blob: ����

## Postgres will refuse to import that last part:
$ echo "SELECT E'"  "$(fix_latin"  "';" | psql
ERROR:  invalid byte sequence for encoding "UTF8": 0xf4 0xa5 0xa3 0xa5 

## Even adding iconv is of no help:
$ echo "SELECT E'"  "$(fix_latin | iconv -t utf-8)"  "';" | psql
ERROR:  invalid byte sequence for encoding "UTF8": 0xf4 0xa5 0xa3 0xa5

The UTF-8 specification is rather dense and puts many requirements on encoders and decoders. How well programs implement these requirements (and optional bits) varies, of course. But at the end of the day, we needed that data to go into a UTF-8 encoded Postgres database without complaint. When in doubt, go to the source! The relevant file in the Postgres source code responsible for rejecting bad UTF-8 (as in the examples above) is src/backend/utils/mb/wchar.c Analyzing that file shows a small but elegant piece of code whose job is to ensure only "legal" UTF-8 is accepted:

pg_utf8_islegal(const unsigned char *source, int length)
  unsigned char a;

  switch (length)
      /* reject lengths 5 and 6 for now */
      return false;
    case 4:
      a = source[3];
      if (a < 0x80 || a > 0xBF)
        return false;
      /* FALL THRU */
    case 3: 
      a = source[2];
      if (a < 0x80 || a > 0xBF)
        return false;
      /* FALL THRU */
    case 2:
      a = source[1];
      switch (*source)
        case 0xE0:
          if (a < 0xA0 || a > 0xBF)
            return false;
        case 0xED:
          if (a < 0x80 || a > 0x9F)
            return false;
        case 0xF0:
          if (a < 0x90 || a > 0xBF)
            return false;
        case 0xF4:
          if (a < 0x80 || a > 0x8F)
            return false;
          if (a < 0x80 || a > 0xBF)
            return false;
      /* FALL THRU */
    case 1:
      a = *source; 
      if (a >= 0x80 && a < 0xC2)
        return false;
      if (a > 0xF4)
        return false;
  return true;

Now that we know the UTF-8 rules for Postgres, how do we ensure our data follows it? While we could have made another standalone filter to run after fix_latin, that would increase the migration time. So I made a quick patch to the fix_latin program itself, rewriting that C logic in Perl. A new option "--strict-utf8" was added. Its job is to simply enforce the rules found in the Postgres source code. If a character is invalid, it is replaced with a question mark (there are other choices for a replacement character, but we decided simple question marks were quick and easy - and the surrounding data was unlikely to be read or even used anyway).

Voila! All of the data was now going into Postgres without a problem. Observe:

$ echo "SELECT E'"  "$(fix_latin"  "';" | psql
ERROR:  invalid byte sequence for encoding "UTF8": 0xf4 0xa5 0xa3 0xa5

$ echo "SELECT E'"  "$(fix_latin --strict-utf8"  "';" | psql
  [Windows-1252]   Euro: €   Double dagger: ‡+
 [Latin-1]   Yen: ¥   Half: ½                +
 [Japanese]   Ship: ?                       +
 [Invalid UTF-8]  Blob: ???? 
(1 row)

What are the lessons here? First and foremost, never use SQL_ASCII. It's outdated, dangerous, and will cause much pain down the road. Second, there are an amazing number of client encodings in use, especially for old data, but the world has pretty much standardized on UTF-8 these days, so even if you are stuck with SQL_ASCII, the amount of Windows-1252 and other monstrosities will be small. Third, don't be afraid to go to the source. If Postgres is rejecting your data, it's probably for a very good reason, so find out exactly why. There were other challenges to overcome in this migration, but the encoding was certainly one of the most interesting ones. Everyone, the client and us, is very happy to finally have everything using UTF-8!


published by Eugenia on 2017-07-13 23:02:11 in the "Metaphysics" category
Eugenia Loli-Queru

I had my usual nap yesterday, and had a very weird dream again. Dreams are by nature surreal because our brain can not understand easily the physics of another dimension. So it resorts to things it knows (e.g. an advanced AI machine might look like a dishwasher), and to symbolism. I feel that I was allowed a peek into the plan somehow. Maybe because I insisted in knowing “what the hell is going on” on my various meditations.

So anyway, I was in a place that had compartments separated by veils (curtains). I understood in my dream this was a UFO, and that my brain was simply showing it as such. In one of the compartments, there were the workers. Short creatures, looked androgynous, waiting for their next task (obviously representing the Greys). One word that could round up their psyche: resentment. They didn’t want to do all these jobs they were assigned to do, they just wanted their freedom.

On the other compartment, it was full of hybrids, with a veiled (cloaked?) insectoid overseeing them. The hybrids were holding a competition as to which one would appear more human (the whole setup felt like the TV set of Jeopardy).

Good luck fighting the agenda with monkey-cowboy mentality

Next thing I knew, I’m looking into a concentration camp, on Earth. It’s filled with hybrids, mostly male. Apparently, the aliens has dumped them on Earth, and the military had rounded them up on FEMA camps (maybe that’s why they were actually built?).

The alien spacecraft were visible to Earth people, who were afraid of them. The media gets into the camps, to interview the hybrids. They pledge allegiance to the aliens, however, as time goes by, their change their position, because they feel used and abandoned by the aliens, stuck in a prison. They say that they feel human, that they would like to be part of the society, and most importantly, they’d join the humans to FIGHT the aliens, in exchange for their freedom.

And they do, purporting “important strategic and technical information”. The aliens and their stationed spacecraft are “defeated” and they are leaving. Hurrah for the hybrids (now dubbed “New Humans”) who have just won their freedom.

The alien plan is now complete. Most of the humans feel that these hybrids helped them win against an immediate alien danger, while the alien plan was such all along: integrate the hybrids into the society in a way that the humans accept them by seeing them as heroes. What’s easier than that to spread their seed?

The aliens obviously are never leaving. They were always here, and will always be here, even if we can’t see them.

Of course, none of these public stunts have to happen, since the US military is already in cahoots (has no choice, really) and could integrate these populations immediately, without any fanfare. However, not every military is US (these people are going to be dumped worldwide), and without the appearance of an alien threat, the world can’t come together (as it should, really). Someone would call it a New World Order, but call it how you want it and paint it as negative or as positive as you want, it’s something that humanity needs in order for the planet to survive.

And of course, 30-40 years down the line, a virus would appear, that would decimate the homo sapiens, but leave unscathed the hybrids and their offspring. This is needed for a quicker and smoother switch to the new species.


published by Eugenia on 2017-07-10 02:02:45 in the "Metaphysics" category
Eugenia Loli-Queru

1. The Pleiadians propaganda

There are so many people “channeling” info about the Pleiadians/Nordics type of aliens, the ones that look 99% human. I’m here to tell you that such a group does not exist. No one is here to rescue you from the Greys/Mantids and their takeover agenda. You are tools, and you have fallen for false propaganda. “Starseeds”, my ass.

Here is why such a benevolent human-like group doesn’t exist.

Because absolutely NO ONE has seen them on psychedelic trips. As I wrote in the past, psychedelic trips are not hallucinations in their core, they are simply altered states of awareness, similar to hypnosis or trance. And in such “spiritual” states, people HAVE seen the Mantis, the Reptilians, and the Greys. In that order of frequency, which denotes how advanced they are (the more advanced they are, the easier reach the DMT states, so people see them in hyperspace). So if you accept alien abductions as real, you MUST accept psychedelic information as real too.

But NO ONE has seen ANY human-based alien in such a state. Yes, people have seen entities changing their appearance to humans, but they are not human in reality (the tripper knows that this was just a screen memory to make the encounter easier).

This means two things:

1. The so-called Nordics/Pleiadians are simply late stage hybrids, lying to you about their identity. They are of the same group as the Greys/Mantis. Since they are very low in the hierarchy ladder, they never reach the DMT states, so no one sees them there.

2. These are Greys, creating screen memories about being human-like, so they can give you false hope (reverse propaganda). Pure disinformation: “Don’t pay attention to negativity, only Follow the Light”. Ridiculous.

No one is coming to rescue you, pal. And EVEN if such a group exists, the fact that they don’t make it on high psychedelic states to reach you, it means that they are insignificant as a species and they don’t pose a threat to the Mantids. That’s like Uganda trying to stop the USA from invading Iraq. Good luck with that.

2. Why abduct children?

Secondly, one last bit of information, that there isn’t out there: why do the Greys abduct children, as young as months old?

The answer: To create a second, split personality through conditioning. When you’re abducted physically, it’s “you” who get abducted. When they abduct you ethereally, to “teach you” things, that’s when your alter comes out. That alter is brainwashed, and subvertied to the Greys, not to humans. When the Event will happen, your alter will take over.

Why do you think you don’t remember what happened to you? It’s not just because of your altered state, or because the Greys magically removed your memory. It’s because that memory belongs to your alter, not to you. This also explains why humans often help the Greys with their work, and then the humans can’t understand under hypnosis why they went along with that. It’s because it wasn’t “you” who followed orders, it was your alter.

3. Praying Mantis Encounter

Something that I realized recently, about a lucid dream I had over 2 years ago.

So, I became lucid, and I called Heva, my Higher Self. Heva appeared, but he soon changed his appearance into a very tall demon, that had a purple cloak. The demon was pissed off at me, that I had “promised” them to do something, but I didn’t do it. He showed me on a tablet what I was supposed to do, and all I could see was myself running around as in a computer game. My first thought was “why do demons have tablets?”, but before I even finished that thought, he telepathically punched me on the liver, which hurt me a lot. I immediately woke up, and my liver was hurting a lot, in REAL LIFE.

Obviously, there are no freaking “demons” (as in, religious fallen-from-grace angels/demons). What they do exist, is highly advanced hyper-dimensional beings, and this one was obviously the Praying Mantis type, disguised itself as a demon (they’re known to wear purple robes with a medallion on the front).

In at least 3 lucid dreams I’ve also encountered the imp (kind of like the Irish elf), which is also seen in alien abductions, although more rarely. As for my secondary “spirit guide”, calling herself “One”, I wouldn’t be surprised if she was a Tall Grey (she appears with a long, white dress, like they do).

4. Final Conclusion

So, what I’ve discovered is all that business about higher selves, spirit guides and shit, these are all THEM. Greys, Tall Greys, and Mantis entities. I guess we’d do the same if we had their powers, so I don’t hold that against them. One thing is for sure, while a lot of New Age beliefs are solid, there’s a lot of crapola around it (added by these beings via channeling etc), just like they’ve done with any past religion.

I was calling myself spiritual so far, but I’m now leaning towards atheist theist. Meaning, I believe that there’s a Source of Consciousness that everything emanates from, but absolutely everything else, and I mean, everything else, is not spiritual in nature (apart from being emanating from the Source, like everything is). It’s just aliens, at different stages of evolution posing as angels, guides, demons, and whatever else bullshit, fucking with our head.

Also, I’ve deleted all my past content on the subject, and I’ll delete this one soon too. They got to me. They sent me a crystal clear dream about what they’re going to do to me and my life if I continue keeping this up.

Three days ago, they also sent me a telepathic message, while I was in a hypnagogic state (just as I was falling asleep — that’s the second time in my life that I receive a message that way). Their voice was like the voice of 1000 entities together, it was a hive that messaged me, not just one individual. Their voice was cold and alien. They just said that they will take over, because we fucked up. It was about 3 sentences long, but I don’t remember it exactly, I just remember the gist of it.

Immediately after, there was one sentence with a human collective voice, if I could put a name on it, it’d be the collective of the Secret Government, pleading to them to have mercy on us and not punish us severely. So if that’s true, there’s neither “resistance”, nor “working with them”. It’s TOTAL CAPITULATION.


published by Eugenia on 2017-07-05 22:52:16 in the "Metaphysics" category
Eugenia Loli-Queru

According to various hypnotists that deal with UFO abductions, the typical Grey alien is a worker being, a hybrid itself. There are several types of beings up in their command chain, including the Reptilians, and the Tall Greys. However, there is a being that is seemingly at the top of all these other beings, an entity that resembles a praying mantis insect (mostly on the head/eye shape and its arms, not the rest). It’s usually seen wearing a purple cloak, which probably denotes position.

My collage, Mantis encounter

Now, here’s the most interesting thing about this being: it’s been seen not only in a conscious, unconscious (e.g. dreams), or hypnotic/trance states, but also under the influence of psychedelics and meditation. In fact, on break-through doses of psychedelics, these insectoid entities are the most common type of alien seen. Greys are occasionally seen too using psychedelics, however these are fewer and between, and also, entering their reality, looks a lot like our reality. While when entering the reality of the praying mantis beings, things become way more advanced and crazy-looking – hinting that the Mantis’ consciousness level is way more advanced than that of the Greys’.

And then there’s meditation, where a smaller number of travelers have seen them (usually during a meditation-induced OBE). I’ve read 3 reports so far of seeing a praying mantis alien, and only 1 of seeing a Grey while on meditation.

Someone could argue that these entities are archetypes of the human collective psyche, and that’s why everyone keeps seeing them. However, I fail to see how the very specific abductions (which all have the same structure and sequence of events) can account for something simple and one-dimensional, as an “archetype”. To me, it looks more likely that these are real beings, living outside of our perception (something that quantum physics and String Theory accepts).

Same goes for the Jester alien, which is along the insectoids, the two most-seen entities on psychedelics. The difference here seems to be that the insectoids have technology that can occasionally punch through to our reality and also appear physically, while the jesters are only available to us via consciousness travel.


published by (Ben Witten) on 2017-07-03 15:50:00 in the "cesium" category

We just returned from Nashville after bringing Liquid Galaxy to the 2017 BOMA International Annual Conference & Expo. Commercial real estate has always been a seamless fit for Liquid Galaxy due to the system's ability to showcase real estate and property data in an interactive and panoramic setting. Within real estate, Liquid Galaxy was first used by CBRE and has since been adopted by Hilton, JLL, and Prologis to name a few.

In preparation for BOMA (Building Owners and Managers Association), we prepared sample commercial real estate content on our content management system to be displayed on the Liquid Galaxy. This included the creation of content about Hudson Yards, the new development being built in lower Manhattan.

The content that was created demonstrates how brokers and directors at commercial real estate companies can tell an immersive, panoramic, and interactive story to their potential clients and investors. Future developments can be shown in developing areas, and with the content management system you can create stories that include videos, images, and 3D models of these future developments. The following video demonstrates this well:

We were able to effectively showcase our ability to incorporate 3D models and mapping layers at BOMA through the use of Google Earth, Cesium, ArcGIS, Unity, and Sketchfab. We were also able to pull data and develop content for neighboring booths and visitors, demonstrating what an easy and data-agnostic platform Liquid Galaxy can be.

We?re very excited about the increasing traction we have in the real estate industry, and hope our involvement with BOMA will take that to the next level. If you?d like to learn more about Liquid Galaxy, please visit our website or email

published by Eugenia on 2017-06-26 23:46:29 in the "Metaphysics" category
Eugenia Loli-Queru

Some random additional Q&A on the alien agenda series, first part was here.

Q: How can you say that alien life exists at all?

A: If you don’t believe that intelligent alien life exists, you’re not just blindly selfish, but also the ultimate pessimist — you basically believe that the Universe is a trap.

If you do believe they exist, but you don’t believe they’re already here, you essentially deny their intelligence, and hence their very existence. Philosophically, your argument wouldn’t work. Either you believe they’re here, or you don’t believe that aliens exist at all.

Q: Who are the Greys?

A: The Greys seem to be an engineered species, made to do a specific work. The bosses are the “praying mantis” 7 ft tall insectoids, not the Greys, and not the Reptilians (which seem to be hybrids themselves, so what is being done to humanity was done to them before). From all these races, in psychedelic trips the praying mantis type is among the most popular encounters. It requires a higher level of consciousness to reach the levels that LSD, mushrooms or DMT can reach. The other races are encountered much less often (but they still do).

Here’s how to setup the whole operation:
Step 1: Create a prototype hybrid between your race, and the humans. Two important keys you’re after: to be intelligent to do the job right, to be able to control humans but not the insectoids, and most importantly, to be able to survive on the Earth atmosphere, at least for a while. The moment you have a successful hybrid with the desired characteristics (even if sterile), you clone it. These are the Greys.

Step 2: You raise the Greys and teach them how to do the job. For different operations (e.g. for surveillance instead of abductions), a few more Grey types exist (e.g. the tall ones). They’re loyal because they have no alternative anyway. They don’t have a planet. The insectoids do, but the Greys are between worlds, entities without a real home. As such, they can only continue to exist in the new humanity. So that becomes their goal.

Step 3: The Greys start the human mass abductions, to create a new hybrid between them, and the humans, that looks completely human but has additional features that they have (e.g. telepathy), albeit diminished in ability.

Step 4: When the project finishes, the Greys are not cloned or replaced anymore. Their role would be fulfilled, they would be a dying race. They were never meant to last. That explains why the Greys are not seen in ancient drawings. Just like you create a special robot for a specific usage on an one-of-a-kind factory, that is unlike any other robot for other factories, same way, the Greys are made for the Earth project only. For a different planet, they’d create a different cloned hybrid to carry out the plan.

Please note that the Greys are not robots, neither they’re “biological robots”. They’re just engineered to do a specific job, and not much else. Mechanical AI/robots wouldn’t be able to do what’s needed, because the job requires to be able to adapt fast, because dealing with humans can be violent, and unpredictable. Only consciousness-assisted AI can reach high levels of performance, pure software can’t.

Q: What about alien implants?

A: What about them? They’re just tracking and health monitoring devices that get powered from the body itself. They emit some radiation or signal when inside the body, but when they are removed, they immediately go OFF. And yes, we Earthlings recently got a similar a technological achievement too!

Q: When is the abduction project finishing?

A: It already has. Since early 2000s, the number of alien abductions is way, way lower than what it used to be in the second part of the last century. The project has now moved to a new phase, which is the integration to human society of thousands of final stage hybrids, which look indistinguishable from humans.

Q: …and loyal to them, not to us.

A: It will take a few generations before they become loyal to Earth. Just like when Spaniards moved in South America, it took them a while before they felt Colombian or Argentinian, and not Spanish. Or the Australians not feeling English anymore. In the meantime, after the initial massacres that took place on the aboriginals, the local population was given options to join the new order. Even if they were a few steps behind in choices and often the objects of racism.

Q: Some say that the Reptilians are evil and they run the whole operation.

A: Nonsense of David Icke and his conspiratorial, religiously idiotic followers, that mix New Age crapola in the whole operation. Reptilians get the bad wrap because humans have evolved to fear reptiles more than insects. But on each and every hypnosis session made by a reputable hypnotist (from Dr Mack to Dr Jacobs), Reptilians are just in the middle of the pack in the hierarchy. People fear them just because they look scary, not because they do anything really different than the Greys do. And no, they won’t eat you, jeez.

Some of the best analysis of the phenomena by a trained professional

Q: What do they get out of the whole deal?

A: Evolution. At the end of the day, nothing really changes about why anyone does anything (from microbes, all the way to humans, and to aliens). It always comes down to evolution at the end.

What if after one point, “evolution of consciousness” only happens artificially, and it MUST happen because the civilization must play catch up with its own technological achievements? If you don’t stay ahead of your own tech curve, you are going to get wiped out by your own creations. This is exactly also why Elon Musk wants to create a “neural link” between the human brain and the upcoming age of AI. Because he wants to keep the humans in the loop, and not “lose out” to the AI.

What this means in a galactic scale, is that evolved races must constantly find new ways to enrich their consciousness, and the best way to do that is to intermingle with other races that have had a different evolutionary path, which had made them better at different things. For example, an eagle can see fine, but a rat, not so well, but the rat has other qualities (e.g. it can swim, chew etc). Mix the two, and you get millions of years of evolution in a few steps.

The Greys have mentioned to the US government that human evolution was intervened 65 times in the past. Not all of these times were for the benefit of the aliens, but occasionally only to us. You will have to see the project as a “set of projects, as needed”, and not as a single project, to understand their longer term benefit too (basically we get immediate benefits, while they get benefited long term).

No one is coming here to “take our planet and its resources”. If they needed that, they would have done so thousands of years ago already. It’s evolution of consciousness they’re after, and it’s a two way road. Which is why the Greys always say to their abductees that: “stop complaining, you agreed to this before you came to this life”. It’s because from a higher vantage point, anyone with a pure logic would have agreed to such evolution, even if from the human incarnated perspective the whole thing looks and sounds so ominous, and terrible. It’s the vantage point that makes all the difference.

Q: Why don’t they reveal themselves?

A: Because no human would accept the medicine they offer.

They’re not here to offer us free energy, or cancer treatments. They are here to give us the tools to find these ourselves. Just like an educational system, they would give you the tools necessary to find the solutions to homework on your own. Because if they just served you the answers, you never learn anything. You can’t progress.

It only makes sense to reveal themselves to the population after we’re dealt with immediate suicidal problems, e.g. wars, nuclear weapons, global warming etc., and only after the descendants of their final stage hybrids have taken root to ensure safe passage. I don’t think they will reveal themselves in less than 50-100 years from now, unless:
A. We are close to global war, which this time around will have them mobilize due to the threat of nuclear weapons, or,
B. the government spills the beans earlier, just to piss them off (or as a strategically-placed scare tactic, after the whole Muslim fear dies down).

Q: We didn’t ask for their help!

A: First, you need to read and deeply understand this. And then, I will leave you with a quote from one of the smartest people today, Mrs Naomi Klein, from a recent Vox interview about global warming:

“For decades, there was a huge emphasis on these just small consumer changes that you can make. It created a kind of dissonance where you present people with information about an existential threat and then say, ?Well, change your light bulb,? or, ?Drive a hybrid.? You don’t talk at all about public policy. And if you do, it’s a very tiny carbon tax and that’s going to do it.

Then I think there are some liberals who do understand the implications of climate change and the depth of change it requires from us. But because they believe humans are incapable of that kind of change, or at this stage in human evolution, I suppose, they think we’re basically doomed. I think contemporary centrist liberalism does not have the tools to deal with a crisis of this magnitude that requires this level of market intervention. And I worry that can lead to a kind of a nihilism around climate change.”

Clearer now?


published by (Mark Johnson) on 2017-06-26 13:00:00 in the "CommonAdjust" category

Product pricing can be quite complex. A typical Interchange catalog will have at least one table in the ProductFiles directive (often products plus either options or variants) and those tables will often have one or more pricing fields (usually price and sales_price). But usually a single, static price isn't sufficient for more complex needs, such as accessory adjustments, quantity pricing, product grouping--not to mention promotions, sales, or other conditional features that may change a product's price for a given situation, dependent on the user's account or session.

Typically to handle these variety of pricing possibilities, a catalog developer will implement a CommonAdjust algorithm. CommonAdjust can accommodate all the above pricing adjustments and more, and is a powerful tool (yet can become quite arcane when reaching deeper complexity). CommonAdjust is enabled by setting the PriceField directive to a non-existent field value in the tables specified in ProductFiles.

To give an adequate introduction and treatise on CommonAdjust would be at a minimum its own post, and likely a series. There are many elements that make up a CommonAdjust string, and subtle operator nuances that instruct it to operate in differing patterns. It is even possible for elements themselves to return new CommonAdjust atoms (a feature we will be leveraging in this discussion). So I will assume for this writing that the reader is familiar generally with CommonAdjust and we will implement a very simple example to demonstrate henceforth.

To start, let's create a CommonAdjust string that simply replaces the typical PriceField setting, and we'll allow it to accommodate a static sales price:

ProductFiles products
PriceField 0
CommonAdjust :sale_price ;:price

The above, in words, indicates that our products live in the products table, and we want CommonAdjust to handle our pricing by setting PriceField to a non-existent field (0 is a safe bet not to be a valid field in the products table). Our CommonAdjust string is comprised of two atoms, both of which are settors of type database lookup. In the products table, we have 2 fields: sale_price and price. If sale_price is "set" (meaning a non-zero numeric value or another CommonAdjust atom) it will be used as it comes first in the list. The semicolon indicates to Interchange "if a previous atom set a price by this point, we're done with this iteration" and, thus, the price field will be ignored. Otherwise, the next atom is checked (the price field), and as long as the price field is set, it will be used instead.

A few comments here:
  • The bare colon indicates that the field is not restricted to a particular table. Typically, to specify the field, you would have a value like "products:price" or "variants:price". But part of the power of ProductFiles holding products in different tables is you can pick up a sku from any of them. And at that point, you don't know whether you're looking at a sku from products, variants, or as many additional tables as you'd like to grab products from. But if all of them have a price and sales_price field, then you can pick up the pricing from any of them by leaving the table off. You can think of ":price" as "*:price" where asterisk is "table this sku came from".
  • The only indicator that CommonAdjust recognizes as a terminal value is a non-zero numeric value. The proposed price is coerced to numeric, added on to the accumulated price effects of the rest of the CommonAdjust string (if applicable), and the final value is tested for truth. If it is false (empty, undef, or 0) then the process repeats.
  • What happens if none of the atoms produce a non-zero numeric value? If Interchange reaches the end of the original CommonAdjust string without hitting a set atom, it will relent and return a zero cost.

At this point, we finally introduce our situation, and one that is not at all uncommon. What if I want a zero price? Let's say I have a promotion for buy one product, get this other product for free. Typically, a developer would be able to expect to override the prices from the database optionally by leveraging the "mv_price" parameter in the cart. So, let's adjust our CommonAdjust to accommodate that:

CommonAdjust $ ;:sale_price ;:price

The $ settor in the first atom means "look in the line-item hash for the mv_price parameter and use that, if it's set". But as we've discussed above, we "set" an atom by making it a non-zero numeric value or another CommonAdjust atom. So if we set mv_price to 0, we've gained nothing. CommonAdjust will move on to the next atom (sale_price database settor) and pick up that product's pricing from the database. And even if we set that product's sale_price and price fields to 0, it means everyone purchasing that item would get it for free (not just our promotion that allows the item to be free with the specific purchase of another item).

In the specific case of using the $ settor in CommonAdjust, we can set mv_price to the keyword "free", and that will allow us to price the item for 0. But this restricts us to only be able to use $ and mv_price to have a free item. What if the price comes from a complex calculation, out of a usertag settor? Or out of a calc block settor? The special "free" keyword doesn't work there.

Fortunately, there is a rarely used CommonAdjust settor that will allow for a 0 price item in a general solution. As I mentioned above, CommonAdjust calculations can themselves return other CommonAdjust atoms, which will then be operated on in a subsequent iteration. This frees us from just the special handling that works on $ and mv_price as such an atom can be returned from any of the CommonAdjust atoms and work.

The settor of interest is >>, and according to what documentation there is on it, it was never even intended to be used as a pricing settor! Rather, it was to be a way of redirecting to additional modes for shipping or tax calculations, which can also leverage CommonAdjust for their particular purposes. However, the key to its usefulness here is thus: it does not perform any test on the value tied to it. It is set, untested, into the final result of this call to the chain_cost() routine and returned. And with no test, the fact that it's Perly false as numeric 0 is irrelevant.

So building on our current CommonAdjust, let's leverage >> to allow our companion product to have a zero cost (assuming it is the 2nd line item in the cart):

    $Items->[1]{mv_price} = '>>0';

Now what happens is, $ in the first atom picks up the value out of mv_price and, because it's a CommonAdjust atom, is processed in a second iteration. But this CommonAdjust atom is very simple: take the value tied to >> and return it, untested.

Perhaps our pricing is more complex than we can (or would like to) support with using $. So we want to write a usertag, where we have the full power of global Perl at our disposal, but we still have circumstances where that usertag may need to return zero-cost items. Using the built-in "free" solution, we're stuck, short of setting mv_price in the item hash within the usertag, which we may not want to do for a variety of reasons. But using >>, we have no such restriction. So let's change CommonAdjust:

CommonAdjust $ ;[my-special-pricing] ;:sale_price ;:price

Now instead of setting mv_price in the item, let's construct [my-special-pricing] to do some heavy lifting:

UserTag my-special-pricing Routine <<EOR
sub {
    # A bunch of conditional, complicated code, but then ...
    elsif (buy_one_get_one_test($item)) {
        # This is where we know this normally priced item is supposed to be
        # free because of our promotion. Excellent!

        return '>>0';
    # remaining code we don't care about for this discussion

Now we haven't slapped a zero cost onto the line item in a sticky fashion, like we do by setting mv_price. So presumably, above, if the user gets sneaky and removes the "buy one" sku identified by our promotion, our equally clever buy_one_get_one_test() sniffs it out, and the 0 price is no longer in effect.

For more information on CommonAdjust, see the Custom Pricing section of 'price' glossary entry. And for more examples of leveraging CommonAdjust for quantity and attribute pricing adjustments, see the Examples section of the CommonAdjust document entry.


published by (Ben Witten) on 2017-06-19 18:59:00 in the "Conference" category

End Point had the privilege of participating in The Ocean Conference at the United Nations, hosted by the International Union for Conservation of Nature (IUCN), these past two weeks. The health of the oceans is critical, and The Ocean Conference, the first United Nations conference on this issue, presents a unique and invaluable opportunity for the world to reverse the precipitous decline of the health of the oceans and seas with concrete solutions.

A Liquid Galaxy was set up in a prominent area on the main floor of the United Nations. End Point created custom content for the Ocean Conference, using the Liquid Galaxy?s Content Management System. Visiting diplomats and government officials were able to experience this content - Liquid Galaxy?s interactive panoramic setup allows visitors to feel immersed in the different locations, with video and information spanning their periphery.

Liquid Galaxy content created for The Ocean Conference included:
-A study of the Catlin Seaview Survey and how the world's coral reefs are changing
-360 panoramic underwater videos
-All Mission Blue Ocean Hope Spots
-A guided tour of the Monaco Explorations 3 Year Expedition
-National Marine Sanctuaries around the United States

We were grateful to be able to create content for such a good cause, and hope to be able to do more good work for the IUCN and the UN! If you?d like to learn more, please visit our website or email


published by (Ben Witten) on 2017-06-16 20:44:00 in the "cesium" category
This past week, End Point attended GEOINT Symposium to showcase Liquid Galaxy as an immersive panoramic GIS solution to GEOINT attendees and exhibitors alike.

At the show, we showcased Cesium integrating with ArcGIS and WMS, Google Earth, Street View, Sketchfab, Unity, and panoramic video. Using our Content Management System, we created content around these various features so visitors to our booth could take in the full spectrum of capabilities that the Liquid Galaxy provides.

Additionally, we were able to take data feeds for multiple other booths and display their content during the show! Our work served to show everyone at the conference that the Liquid Galaxy is a data-agnostic immersive platform that can handle any sort of data stream and offer data in a brilliant display. This can be used to show your large complex data sets in briefing rooms, conference rooms, or command centers.

Given the incredible draw of the Liquid Galaxy, the GEOINT team took special interest in our system and formally interviewed Ben Goldstein in front of the system to learn more! You can view the video of the interview here:

We look forward to developing the relationships we created at GEOINT, and hope to participate further in this great community moving forward. If you would like to learn more please visit our website or email


published by Eugenia on 2017-06-15 01:03:45 in the "Metaphysics" category
Eugenia Loli-Queru

It was sometime in October of 2000, about a month after I’ve come back from my first visit to the US. I was living in Guildford, UK at the time, working as a web developer and database analyst.

I was sleeping, and suddenly I woke up. I don’t know what time it was, but if I was to make a guess, I’d probably say it was about 2:30 AM or so. I immediately got completely awake, and alarmed. I woke up because I felt something moving on my feet on the bed, on top of the blankets. Whatever it was, it did not have the full weight of a human, it felt more like a dog or cat (there were no pets in the house where I was renting a room at the time, and my door was locked).

I got extremely scared, since I couldn’t understand what that was. I didn’t make a peep, neither I moved. I don’t know if they had paralyzed me, but my idea at the time was that I consciously decided not to move, as to supposedly mimic that I was still sleeping. I didn’t open my eyes either.

Within 20-25 seconds of all that happening, something else touched my head. It seems there was more than one of them in the room. As it touched me, within 2-3 seconds, I was back at sleep like nothing had happened. Which is crazy to think that I’d go back to sleep so easily, since I was wide awake and in sheer terror.

I do not believe this to be a case of sleep paralysis. I have had a few cases of sleep paralysis over the years, and they all happened while in an altered state (in a lucid dream state which clearly differs from my waking state). Instead, that night I was 100% awake.

The next morning, the alarm clock woke me up, at my usual time at 7 AM. I sprang out of the bed, and I was feeling completely OFF. Like, REALLY off. You know how you might feel sometimes when you’re really tired and you need to sleep 8 hours to recover, but you only sleep about 2 hours or so, and so you wake up and you’re kind in between of two states, feeling really off? That’s how I felt.

I got ready and went to work. I was feeling sick and had pain in my abdomen.

For a month, I was extremely scared to sleep at night. I was sleeping along a picture of Virgin Mary that my mother had bought for me years before to… protect me (lol). I was shaking and shivering for at least an hour every night until sleep would finally arrive. The fear went away gradually after talking to a monk (I believe he was Catholic, but I’m not sure), who used to setup two chairs in the middle of Guildford’s High Street on Saturdays, waiting for people to talk to him if needed (he never spoke in the middle of the street as some fanatics do, he just patiently waited all day for anyone who might needed to talk). I approached him as he almost started to pack to leave, at the end of the day. He listened to me, he prayed with me, and he told me to not fear. That’s all he did, but he helped me. I thanked him, and I asked him if he talked to many more people that day. He said I was the only one!

Since then, I haven’t experienced anything else like that consciously. I never had any missing time. If they’re still coming for me, they definitely don’t screw it up anymore. ‘Cause if they’ll screw it up again, I will break their big, fat skulls.

Subconsciously though, it’s another story. I’ve had many dreams and lucid dreams about aliens and UFOs. The most recent one, just a couple of weeks ago, where I was lifted off my house where I currently live, along a few others, and we were placed on a hangar of a big UFO. We were all waiting for the other shoe to drop when the aliens arrived (about 20 of them), but they had altered our perception to not see them with their true form. So the Greys looked like this kinda. Big wig, big head, large eyes, feminine. I immediately realized that they were putting screen memories instead of revealing themselves in their true form, and I thought that they looked ridiculous.

I don’t quite find the Greys dreams that interesting though. They’re kind of same-old, same-old. What’s more interesting is that I’ve seen 2-3 times in dreams the same type of spaceships appearing out of nowhere on our skies worldwide, and kinda stay there without communicating. Their shapes are very irregular, without any symmetry, they’re not like saucers or triangles that we normally associate with the Greys. They’re just strange design-wise, like random scraps of metal. The closest I could find can be seen here. I don’t know who the occupants of these spaceships are, I never seen them.


published by (Greg Sabino Mullane) on 2017-06-06 21:18:00 in the "Amazon" category

Many of our clients at End Point are using the incredible Amazon Relational Database Service (RDS), which allows for quick setup and use of a database system. Despite minimizing many database administration tasks, some issues still exist, one of which is upgrading. Getting to a new version of Postgres is simple enough with RDS, but we've had clients use Bucardo to do the upgrade, rather than Amazon's built-in upgrade process. Some of you may be exclaiming "A trigger-based replication system just to upgrade?!"; while using it may seem unintuitive, there are some very good reasons to use Bucardo for your RDS upgrade:

Minimize application downtime

Many businesses are very sensitive to any database downtime, and upgrading your database to a new version always incurs that cost. Although RDS uses the ultra-fast pg_upgrade --links method, the whole upgrade process can take quite a while - or at least too long for the business to accept. Bucardo can reduce the application downtime from around seven minutes to ten seconds or less.

Upgrade more than one version at once

As of this writing (June 2017), RDS only allows upgrading of one major Postgres version at a time. Since pg_upgrade can easily handle upgrading older versions, this limitation will probably be fixed someday. Still, it means even more application downtime - to the tune of seven minutes for each major version. If you are going from 9.3 to 9.6 (via 9.4 and 9.5), that's at least 21 minutes of application downtime, with many unnecessary steps along the way. The total time for Bucardo to jump from 9.3 to 9.6 (or any major version to another one) is still under ten seconds.

Application testing with live data

The Bucardo upgrade process involves setting up a second RDS instance running the newer version, copying the data from the current RDS server, and then letting Bucardo replicate the changes as they come in. With this system, you can have two "live" databases you can point your applications to. With RDS, you must create a snapshot of your current RDS, upgrade *that*, and then point your application to the new (and frozen-in-time) database. Although this is still useful for testing your application against the newer version of the database, it is not as useful as having an automatically-updated version of the database.

Control and easy rollback

With Bucardo, the initial setup costs, and the overhead of using triggers on your production database, is balanced a bit by ensuring you have complete control over the upgrade process. The migration can happen when you want, at a pace you want, and can even happen in stages as you point some of the applications in your stack to the new version, while keeping some pointed at the old. And rolling back is as simple as pointing apps back at the older version. You could even set up Bucardo as "master-master", such that both new and old versions can write data at the same time (although this step is rarely necessary).

Database bloat removal

Although the pg_upgrade program that Amazon RDS uses for upgrading is extraordinarily fast and efficient, the data files are seldom, if ever, changed at all, and table and index bloat is never removed. On the other hand, an upgrade system using Bucardo creates the tables from scratch on the new database, and thus completely removes all historical bloat. (Indeed, one time a client thought something had gone wrong, as the new version's total database size had shrunk radically - but it was simply removal of all table bloat!).

Statistics remain in place

The pg_upgrade program currently has a glaring flaw - no copying of the information in the pg_statistic table. Which means that although an Amazon RDS upgrade completes in about seven minutes, the performance will range somewhere from slightly slow to completely unusable, until all those statistics are regenerated on the new version via the ANALYZE command. How long this can take depends on a number of factors, but in general, the larger your database, the longer it will take - a database-wide analyze can take hours on very large databases. As mentioned above, upgrading via Bucardo relies on COPYing the data to a fresh copy of the table. Although the statistics also need to be created when using Bucardo, the time cost for this does NOT apply to the upgrade time, as it can be done any time earlier, making the effective cost of generating statistics zero.

Upgrading RDS the Amazon way

Having said all that, the native upgrade system for RDS is very simple and fast. If the drawbacks above do not apply to you - or can be suffered with minimal business pain - then this way should always be the upgrade approach to use. Here is a quick walk through of how an Amazon RDS upgrade is done.

For this example, we will create a new Amazon RDS instance. The creation is amazingly simple: just log into, choose RDS, choose PostgreSQL (always the best choice!), and then fill in a few details, such as preferred version, server size, etc. The "DB Engine Version" was set as PostgreSQL 9.3.16-R1", the "DB Instance Class" as db.t2.small -- 1 vCPU, 2 GiB RAM, and "Multi-AZ Deployment" as no. All other choices are the default. To finish up this section of the setup, "DB Instance Identifier" was set to gregtest, the "Master Username" to greg, and the "Master Password" to b5fc93f818a3a8065c3b25b5e45fec19

Clicking on "Next Step" brings up more options, but the only one that needs to change is to specify the "Database Name" as gtest. Finally, the "Launch DB Instance" button. The new database is on the way! Select "View your DB Instance" and then keep reloading until the "Status" changes to Active.

Once the instance is running, you will be shown a connection string that looks like this: That standard port is not a problem, but who wants to ever type that hostname out, or even have to look at it? The pg_service.conf file comes to the rescue with this new entry inside the ~/.pg_service.conf file:


Now we run a quick test to make sure psql is able to connect, and that the database is an Amazon RDS database:

$ psql service=gtest -Atc "show rds.superuser_variables"

We want to use the pgbench program to add a little content to the database, just to give the upgrade process something to do. Unfortunately, we cannot simply feed the "service=gtest" line to the pgbench program, but a little environment variable craftiness gets the job done:

$ export PGSERVICEFILE=/home/greg/.pg_service.conf PGSERVICE=gtest
$ pgbench -i -s 4
NOTICE:  table "pgbench_history" does not exist, skipping
NOTICE:  table "pgbench_tellers" does not exist, skipping
NOTICE:  table "pgbench_accounts" does not exist, skipping
NOTICE:  table "pgbench_branches" does not exist, skipping
creating tables...
100000 of 400000 tuples (25%) done (elapsed 0.66 s, remaining 0.72 s)
200000 of 400000 tuples (50%) done (elapsed 1.69 s, remaining 0.78 s)
300000 of 400000 tuples (75%) done (elapsed 4.83 s, remaining 0.68 s)
400000 of 400000 tuples (100%) done (elapsed 7.84 s, remaining 0.00 s)
set primary keys...

At 68MB in size, this is still not a big database - so let's create a large table, then create a bunch of databases, to make pg_upgrade work a little harder:

## Make the whole database 1707 MB:
$ psql service=gtest -c "CREATE TABLE extra AS SELECT * FROM pgbench_accounts"
SELECT 400000
$ for i in {1..5}; do psql service=gtest -qc "INSERT INTO extra SELECT * FROM extra"; done

## Make the whole cluster about 17 GB:
$ for i in {1..9}; do psql service=gtest -qc "CREATE DATABASE gtest$i TEMPLATE gtest" ; done
$ psql service=gtest -c "SELECT pg_size_pretty(sum(pg_database_size(oid))) FROM pg_database WHERE datname ~ 'gtest'"
17 GB

To start the upgrade, we log into the AWS console, and choose "Instance Actions", then "Modify". Our only choices for instances are "9.4.9" and "9.4.11", plus some older revisions in the 9.3 branch. Why anything other than the latest revision in the next major branch (i.e. 9.4.11) is shown, I have no idea! Choose 9.4.11, scroll down to the bottom, choose "Apply Immediately", then "Continue", then "Modify DB Instance". The upgrade has begun!

How long will it take? All one can do is keep refreshing to see when your new database is ready. As mentioned above, 7 minutes and 30 seconds is the total time. The logs show how things break down:

11:52:43 DB instance shutdown
11:55:06 Backing up DB instance
11:56:12 DB instance shutdown
11:58:42 The parameter max_wal_senders was set to a value incompatible with replication. It has been adjusted from 5 to 10.
11:59:56 DB instance restarted
12:00:18 Updated to use DBParameterGroup default.postgres9.4

How much of that time is spent on upgrading though? Surprisingly little. We can do a quick local test to see how long the same database takes to upgrade from 9.3 to 9.4 using pg_upgrade --links: 20 seconds! Ideally Amazon will improve upon the total downtime at some point.

Upgrading RDS with Bucardo

As an asynchronous, trigger-based replication system, Bucardo is perfect for situations like this where you need to temporarily sync up two concurrent versions of Postgres. The basic process is to create a new Amazon RDS instance of your new Postgres version (e.g. 9.6), install the Bucardo program on a cheap EC2 box, and then have Bucardo replicate from the old Postgres version (e.g. 9.3) to the new one. Once both instances are in sync, just point your application to the new version and shut the old one down. One way to perform the upgrade is detailed below.

Some of the steps are simplified, but the overall process is intact. First, find a temporary box for Bucardo to run on. It doesn't have to be powerful, or have much disk space, but as network connectivity is important, using an EC2 box is recommended. Install Postgres (9.6 or better, because of pg_dump) and Bucardo (latest or HEAD recommended), then put your old and new RDS databases into your pg_service.conf file as "rds93" and "rds96" to keep things simple.

The next step is to make a copy of the database on the new Postgres 9.6 RDS database. We want the bare minimum schema here: no data, no triggers, no indexes, etc. Luckily, this is simple using pg_dump:

$ pg_dump service=rds93 --section=pre-data | psql -q service=rds96

From this point forward, no DDL should be run on the old server. We take a snapshot of the post-data items right away and save it to a file for later:

$ pg_dump service=rds93 --section=post-data -f

Time to get Bucardo ready. Recall that Bucardo can only replicate tables that have a primary key or unique index. But if those tables are small enough, you can simply copy them over at the final point of migration later.

$ bucardo install
$ bucardo add db A dbservice=rds93
$ bucardo add db B dbservice=rds96
## Create a sync and name it 'migrate_rds':
$ bucardo add sync migrate_rds tables=all dbs=A,B

That's it! The current database will now have triggers that are recording any changes made, so we may safely do a bulk copy to the new database. This step might take a very long time, but that's not a problem.

$ pg_dump service=rds93 --section=data | psql -q service=rds96

Before we create the indexes on the new server, we start the Bucardo sync to copy over any rows that were changed while the pg_dump was going on. After that, the indexes, primary keys, and other items can be created:

$ bucardo start
$ tail -f log.bucardo ## Wait until the sync finishes once
$ bucardo stop
$ psql service=rds96 -q -f 

For the final migration, we simply stop anything from writing to the 9.3 database, have Bucardo perform a final sync of any changed rows, and then point your application to the 9.6 database. The whole process can happen very quickly: well under a minute for most cases.

Upgrading major Postgres versions is never a trivial task, but both Bucardo and pg_upgrade allow it to be orders of magnitude faster and easier than the old method of using the pg_dump utility. Upgrading your Amazon AWS Postgres instance is fast and easy using the AWS pg_upgrade method, but it has limitations, so having Bucardo help out can be a very useful option.