This is currently a proof of concept and not in production code.

Scenario

The scenario is this: A user wants to have a time consuming undertaking performed. He inputs the start parameters (e.g. a web site url to be processed and an e-mail address to send the results to) into a form on a web page and clicks "submit". The web page immediately returns informing the user that the undertaking has been accepted and that he will get an e-mail later on its completion.

Architecture

The web server that received the undertaking is on a relatively expensive server which we pay for having in a good datacenter with great uptime. We will call this machine the control machine. We do not want it to churn through any tasks since its precious computing resources are needed for front-end work. Instead the undertaking should be done on inexpensive back-end servers.

The back end servers starts churning, having the undertaking divided into two tasks. If all goes well a report will be sent to the user. If something goes wrong, the undertaking is set aside and staff notified that something is either wrong in the code or in the data.

Implementation

Step 1 - Make a worker

For this example we will use two tasks that do jobs. For simplicity in this example, they will just add numbers and multiply numbers. We will also define an error handling task that will handle the buggy add task. This module, called "test1.py" should be available both on the worker machine and on the control machine. But it won't be actually running on the control machine. It will just be reachable with an import by other scripts.

For simplicity, the error handling will just be a print statement with an apology, although it is unlikely that the user is looking at the worker machine's terminal output.

# Change these two to your back end. Here it is set to use a Redis server

# at 192.168.1.21, running on the Redis standard port, using database number 1

BROKER_URL = 'redis://192.168.1.21:6379/1'
CELERY_RESULT_BACKEND = 'redis://192.168.1.21:6379/1'

from celery import Celery

celery = Celery('test1', backend= CELERY_RESULT_BACKEND, broker=BROKER_URL)

@celery.task
def add(x, y):
    why = x/0  # An error in the code!!
    return x + y

@celery.task
def mul(x, y):
    return x * y

@celery.task
def onerror(uuid):
    print "We apologize and sweat profusely over the sub standard processing of job %s" % uuid

Start the worker with:

celery -A test1 worker

Step 2 - Make the control script

This only needs to be installed on the control machine

The control script has a client that puts stuff into the system. It also specifies what should happen if a worker throws an exception.

# Change these two to your back end. Here it is set to use a Redis server

# at 192.168.1.21, running on the Redis standard port, using database number 1

BROKER_URL = 'redis://192.168.1.21:6379/1'
CELERY_RESULT_BACKEND = 'redis://192.168.1.21:6379/1'

from test1 import add, mul, onerror

res = add.apply_async((2, 2), link=mul.s(16), link_error=onerror.s())
print res.get(propagate=False)

So, thats it. Control calls add first, and links it to mul. This means that Celery will execute add first, and whenever that is ready, will execute mul with the result value of add as part of the input to mul. However, if add throws an error, the task specified by link_error will be executed instead. You should see the apology being printed in the terminal window of the worker. Normally you wouldn't wait for the result with get, since that is a blocking operation. The get here has propagate set to false, which means it will not re-raise the error from the worker.

Untested by me

Womack is a service that you can use to push realtime events between your regular, plain-old, non-websockety web application and clients. It is built on top of gevent-socketio and redis.

Read more: Womack: Push real-time events to the browser with python

If you start flower - a web interface that allows you to monitor, inspect and edit things in the Celery work queue system - it will assume you are running RabbitMQ or similar on localhost. As of currently, doing

celery flower --help

...will not reveal that there is a --broker option which allows you to direct flower to monitor another back end (such as Redis on another server for example). Use the celery style url for that. e.g:

celery flower --broker=redis://192.168.1.14:6379/1

This would use database 1 on a Redis server running on its standard port of 6379 on the machine at IP number 192.168.1.14.

celery flower --broker option does not work

Read more: celery flower --broker option does not work · Issue #11 · mher/flower

In Google Chrome, when you copy an address out of the URL filed, you get it as rich text, which makes it a cinch to paste in the URL in for example a blog like the one you are reading right now. In Firefox this is not possible. There are a couple of Firefox add.ons that manipulate copied text but surprisingly none of the do what Google does. However a gentleman posting at Stackoverflow has taken the time to extract javascript from one of the add-ons and combine that with an add-on that allows you to add shortcuts. I followed the instructions and it works and I can now with a key combination get the title of the current page inside of an A element pointing to the page, all wrapped up in rich text so that it works pasting it in to WYSIWYG editors.

I need to automate the copying of a HTML link to the current page that is viewed in the current Firefox Tab into other WYSIWYG editors.

Read more: key bindings - Firefox Extension that copies HTML link to current web page to clipboard and not just the URL - Stack Overflow

I have just finished writing a message queue based system and I used Redis as the message queue system. I have just run the system in production for a couple of days, but I already have some thoughts on how I would like a message queue system to work, and what seems to be available. There seems to be two names floating around to describe these kinds of queues: Message queues as a more general term and work queue as a more specific term.

A work queue system helps with distributing and deferring work. It can be so that a user of a web site can ask for some processing to be done, but does not have to wait for it to happen while the web page is slowly loading the result. Instead a quick confirmation is given and later a message - usually an e-mail, is sent to the user. A message queue system can also help with scalability and reliability, see further down in the text how it does this.

Here is a list of some interesting systems I have found that are freely available. I will divide them in three categories: Messaging toolkits, message queue systems and work queue systems. This is just a rough categorization. With messaging toolkits you have complete freedom in how you want your system to work and behave, message queue systems have the fundamentals to build work queue systems (and oftentimes form the core of the work queue systems) and finally work queue systems contain a lot of the functionality out of the box.

Messaging toolkits

zeromq - Toolkit targeted towards those who want to build their own high-speed systems.

Message queue systems

Lend themselves as a starting point for building work queue systems

Redis - Well documented and easy to get started with. Does not dabble in higher level constructs so you may need to write your own code on that level, or use any of the frameworks that build on Redis. Redis seems to have started as an improvement on memcached, but has since taken on more functions also suitable for messaging systems. A lot of the use of Redis on the web is though as a cache and session store. The building stones of Redis - keys, lists, sets, sorted sets, transactions and timeouts - seem well thought out. Redis is what I currently use. Huge user base: A Google search for pages mentioning "Redis" yields  7.5 million results. However a large part of the base is probably using the caching stuff more than the queueing.

RabbitMQ - Advanced system that follows the AMQP standard (which exists in different revisions, RabbitMQ is at 0.91, but there is a 1.0 version of the standard). Used by many large operations. Not a caching system. Has an acknowledge function that means that a job can be requeued if it does not get completed in a configurable amount of time. Also has the concept of exchanges, which are a couple of pre-defined routing algorithms. There are other AMQP systems available such as ActiveMQ and HornetMQ, and also commercial message queue systems that use the AMQP protocol. AMQP originated in the world of banking.

Work queue systems

Have features built in for managing jobs

Resque - Github's framework running on top of Redis, written in Ruby. It is used massively on github.com, and GitHub is awesome. There is a python clone on the net somewhere too, called pyres. (Resque does not seem to have a logo, so I just snatched an octocat from GitHub:s front page). There is a java version called jesque, and a javascript/Coffeescript version called coffee-resque. The latter seems to run server-side on node.js.

sidekiq is a work queue system written in Ruby similar to Resque and in many ways API compatible with it. Uses threads. Can run on the JVM with JRuby, might even be the recommended way of running it.

beanstalkd Seems fairly advanced in the work queue department. Does not have that extensive documentation, but there is a wiki.

Celery - Amibitious high level messaging system in Python that can run on top of RabbitMQ, Redis, beanstalkd or pretty much anything. Celery is extensively documented. It can still be a bit hard to find one's way through the documentation on what it really is and how it works. It's here: User Guide — Celery 3.0.11 documentation . This slideshow- Advanced task management with Celery helps a lot after having read through the docs. Here's a little thing I've been writing after initial tests:  How to create a sequential workflow with error handling in Celery.

Besides running on the standard CPython (both 2 and 3), Celery can also run on the JVM with Jython, and with Pypy.

kombu - factored out of Celery - has a very nice API. Runs on top of Rabbitmq, Redis, beanstalkd or pretty much anything. Kombu is used in OpenStack - an infrastructure as a service project (basically automatically deploying virual servers).

Gearman has been around for a while, originally written in Perl, but there is now also a C version. It was mentioned in a Reddit discussion pertaining to this post. Documentation is a bit sketchy and I cannot say much about it, but seems to have a user community and many client implementations. The main documentation does not give a good overview of Gearman as far as I can see, but these do:

• http://gearman.org/index.php?id=gearman_php_extension
• http://en.wikipedia.org/wiki/Gearman

Popularity of the different systems

I did an unscientitic ranking by checking how many days back it took to get the 50 latest questions about each system from stackoverflow.com, and then calculate questions per week for each system. The results as of 2012-10-28 were with most popular sorted first:

1. Redis 35
2. RabbitMQ 13
3. Celery 9.5
4. Resque 5.8
5. ZeroMQ 5.4
6. Gearman 2.9
7. Kombu 1.6
8. sidekiq 1.3
9. Beanstalkd 1.1

I would say though that at least half of the Redis questions on stackoverflow are fielded by people who are using Redis as a cache or session store rather than as a basis for a message queue or work queue.

I am surprised that beanstalkd ranked so low and Celery so high. It may well be that some of these systems have their communities do Q/A somewhere else than at stackoverflow or that some just do not generate that many questions, but still it is a rough estimate.

I checked the tag "beanstalk" too for beanstalkd, but that one was 98% about Amazon beanstalk, which is something else.

For other AMQP systems than RabbitMQ: ActiveMQ would have ranked similar to RabbitMQ, HornetQ somewhere near Kombu.

Features and concepts

In this text:

• task means the overarching thing you are trying to achieve with your application, e.g. do some searches, analyze the results and then email a report to the user
• subtask is a part of the task, e.g. e-mailing out a report
• worker is a program that does the subtask, e.g. a worker that is an emailer
• job is an instance of a subtask, e.g an emailer worker emailing out the report to a specific user
• Control is some kind of central control function, e.g. a supervisor that checks for bad jobs

Scalability and reliability

Distributing and deferring work can help with scalability and reliability. With a work queue, several worker processes can feed from the same queue and hence you can get an automatically scalable system, where each worker just needs to connect and snatch something from the queue and go to work. Reliability can be improved by revoking jobs from malfunctioning workers and reschedule them to other workers. Just the fact that you have queues means that is not a biggie if most your complex web based system temporarily goes down; as long as the web interface and the process that puts things into the queues are up and running, the rest of the system can display a bit of volatility without jeopardizing the entire application.

Caching

Message queue systems also seem to be used as a cache, similar to memcached. This is the function I am least interested currently. In caching, the message queue system supplies a number of constructs, such as queues obviously, that can help to serve out fast-changing information quicker, than what would be possible by handling it with slower back end services. It is basically a cache with a bit of intelligence, that can sort, slice and dice. For this application speed is of the essence, with sub-millisecond replies being the order of the day.

Modularity

If you divide your application into separate workers, they need a way to communicate. Often JSON is used for this in message queues, and since there are JSON parsers for all major languages you can mix and match workers written in completely different languages.

Resilience and inspectability

Having the process divided into several steps becomes a bit like having break points in your code. You can check the state of the data at the end and beginning of each step; they provide a snapshot of the state of your application at that point. You can correct data or code to get jobs unstuck in the processing chain, which gives the system a higher service level, and can help when demands are that every task should get through.

Revoking and rescheduling

Sometimes a job does not finish, or it does not finish the way you like. It would be good if the message queue system could help in handling this. In Redis I simply rescheduled the job by putting it back last in the processing queue. However the processing in my system is fairly deterministic and the job is most likely not going to fare better on a second run. Hence jobs that fail are now taken out of the queue and go to human inspection.

I figured out there could be different causes for a job not finishing and how it should be handled. It is assumed that a worker at least can catch its own exceptions. Here is what I came up with:

• Bad job: Data makes worker throw an exception, detectable from worker, can reach control
• Bad storage: Server makes worker throw an exception, detectable from worker, can reach control
• Bad network: Server makes worker throw an exception, detectable from worker, cannot reach control
• Power failure or other catastrophic failure: Server makes worker lock or crash, undetectable from worker

zeromq has some reasoning along the same lines. Since they have more experience than me I'll quote their take on it:

"So let's look at the possible causes of failure in a distributed ØMQ application, in roughly descending order of probability:

• Application code is the worst offender. It can crash and exit, freeze and stop responding to input, run too slowly for its input, exhaust all memory, etc.

• System code - like brokers we write using ØMQ - can die for the same reasons as application code. System code should be more reliable than application code but it can still crash and burn, and especially run out of memory if it tries to queue messages for slow clients.

• Message queues can overflow, typically in system code that has learned to deal brutally with slow clients. When a queue overflows, it starts to discard messages. So we get "lost" messages.

• Networks can fail (e.g. wifi gets switched off or goes out of range). ØMQ will automatically reconnect in such cases but in the meantime, messages may get lost.

• Hardware can fail and take with it all the processes running on that box.

• Networks can fail in exotic ways, e.g. some ports on a switch may die and those parts of the network become inaccessible.

• • Entire data centers can be struck by lightning, earthquakes, fire, or more mundane power or cooling failures."

Some ideas from me on remedies on the things on my list above:

Bad job: The job should be taken out of the job queue and a bad job mail be sent to a human

Bad storage: The job should be resubmitted to another worker and the malfunctioning worker should be taken out of commission, i.e. terminate itself.

Bad network:  The job should time out and be resubmitted and the worker should be taken out of commission, i.e. terminate itself .

Power failure: The job should time out and the job should time out and be resubmitted and the worker should be taken out of commission from the point of control, since the malfunctioning worker can't do it. If it comes to life and sends in a job that has already been processed by another worker, this job is ignored.

• The control server must make reasonable assumptions of  how long a job could max take

Work queue strategies

The three systems I have found that seems to be able to help out-of-the-box with these kinds of things are Resque from GitHub,  beanstalkd and Celery. After reading through the AMQP 0.91 standard as explained on the RabbitMQ site it seems RabbitMQ should also be able to contribute out-of the box on this level.

zeromq and Redis on the other hand are more like toolkits, especially zeromq.

Resque puts in an abstraction layer with a parent worker process and a child worker process, where the parent worker process starts the child process for the actual work and watches it and changes its own state depending on whether the child process concludes or something else. Some quotes from their pages:

"Resque assumes your background workers will lock up, run too long, or have unwanted memory growth."

"If you want to kill a stale or stuck child, use USR1. Processing will continue as normal unless the child was not found. In that case Resque assumes the parent process is in a bad state and shuts down."

And beanstalkd from its FAQ, on its buried state:

"For example, this [buried state] is useful in preventing the server from re-entering a timed-out task into the queue when large and unpredicatble run-times are involved. If a client reserves a job, then first buries it, then does its business logic work, then gets stuck in deadlock, the job will remain buried indefinitely, available for inspection by a human — it will not get rerun by another worker."

sidekiq has an interesting take on the same theme. Instead of burying a job for human inspection it retries with a timeout that gets longer and longer so that you should have time to fix the problem:

"Sidekiq will retry processing failures with an exponential backoff using the formula retry_count**4 + 15 (i.e. 15, 16, 31, 96, 271, ... seconds). It will perform 25 retries over approximately 20 days. Assuming you deploy a bug fix within that time, the message will get retried and successfully processed."

beanstalkd also sports configurable time-outs that can give control a signal that a job is probably hung or unreachable. RabbitMQ uses acknowledgements (acks) to track finished jobs. Redis has time-outs on its key data type, but no buillt-in detection or event handler for timeouts, so you would have to construct something like that on a higher level than in Redis itself. A pop/push timeout in Redis would have been helpful methinks.

Celery has advanced concepts in this department. Some concepts from Celery that seem to be of interest: linked workers, revoke, inspect, chains, groups, chords, scheduling. Chains, groups and chords are a way of stringing together subtasks into units, that can fan out into parallel processing and for example do map-reduce.

Some concepts from kombu that seem to be of interest: message priority, retry_policy, recover, heartbeat_check.

There seems to be some systems that are pull-based when it comes to queues such as Redis, while RabbitMQ seems to support both pul and push-based interactions. With pull based systems you need less intelligence centrally but the downside then is that you do have less intelligence centrally of course. I guess you will need a bit of both: Workers know better how to distribute the load between them, but they cannot manage states where they do not function anymore. Then control needs to do that and be able to bury jobs.

In my current application, there is a different job queue for each kind of subtask, and it is each worker's responsibility to move the results of a job on to the next queue so that the next subtask can be executed.

 I discovered that you can specify in Celery a mapping between task signatures and queue names, see: Routing Tasks. Still, it seems as the responsibility for moving stuff onwards does not fall onto the worker, although I will have to look into that.

Update: I have looked into how Celery can be used for handling work queues in a way that you can move a task from worker to worker and branch out on error conditions, see my blog post: How to create a sequential workflow with error handling in Celery.

Parts and names

From Zeromq again with my boldface:

"sinks (process the messages without any response), proxies (send the messages on to other nodes), or services (send back replies)"

Side effects

Furthermore you want to avoid having the workers produce side effects that cannot be revoked or at least gather them. It is a good idea I believe if you are going to send out an email, to make that worker as simple and reliable as possible, and not trigger it unless everything else has lined up right. And do make sure it does not get stuck in a loop. In fact its should probably have a memory of what is has sent out before and refuse to send again if it detects same content and recipient. Unless you are running a huge system or a spam operation or something else nefarious you don't need to have many e-mailing workers and can get by with just a singleton, which means you do not need to worry about spawning e-mail clients all over the place (and other places).

If a subtask doesn't have any side effects, it means that it can be run several times without any ill consequences. It is then said to be idempotent, a word that pops up here and there in the documentation for the different systems. Idempotency does allow side effects too, as long as it only happends on the first run. So a singleton e-mailer that refuses to send an e-mail it has already sent would also be called idempotent.

Fallbacks & graceful degradation

One thing I think would be useful in a work queue system is fallbacks. If the biggest source for interruption of a job is a conflict between code and data (a nice way of saying that the code is buggy and/or bad data has been allowed in), then re-running  the job will give the same result. Another way of handling that is to re-run the job with different code, that may not give as good a result but is more robust.

I am right now in the process of replacing one bit of worker code in CheckMyCSS.com written in python, with code written in javascript including a headless browser called phantomjs running webkit, the web browser engine that 40% of all the world's web browsers are built on top of (Chrome, Safari, Android's web browser). Phantomjs throws exceptions just like the python code but what if it will just hang in some situations? Then it can be terminated with a time out but that won't help the end user. Now, if hypothetically some jobs would fail on the new worker, why not fall back on the old one and keep the task moving to completion? One could call the fallback worker a naive worker or something. A naive e-mailer might send an SMS over the GSM network telling someone to send an e-mail for example.

Feature ratings

Could be time to start defining what would be interesting to have in a work queue system for me, right now. One should note that many things may best be left to the coder. There is no use in having a framework which concepts are just far enough from what you actually want to make the code a bit cumbersome.

I'll start with Redis, which I have used and I will also fill in some preliminary info on Celery, but I need to run more tests on that one. Please note that the rating will change as I find out more. Do also note that since for example Celery can use Redis as a back end, by definition Redis can "do" everything Celery can, as long as e.g. Celery is doing the work.

I've also started putting in info on RabbitMQ and its implementation of the AMQP, which seems to have enough knobs and levers to express many of the concepts listed below. It can however be a blurry line between what is built-in and is merely a configuration issue, and what starts looking like bona fide programming.

A solid authentication layer

Redis claims to not be overly secure if exposed directly to the Internet. One can put an SSL tunnel in front with certificates, which I have done. That may be a better solution than what could have been built in.

Celery has built in support for signing and certificates right into the messages themselves, see Security — Celery 3.0.11 documentation, however the Celery documentation says that Celery should be treated as an "unsafe component", so SSL tunneling might be a good idea anyway.

A work queue that workers can take jobs from

Redis has this with e.g. a blocking pop/push (BRPOPLPUSH), but the only way to find out if an item is in a list is to delete it (LREM)

Celery has this.

RabbitMQ has this. It can make it look like workers are pulling jobs from a queue, by using the configuration prefetch_count=1.

Possibility to bury jobs

Redis does not have this, you will have to build that on top

Celery has something called a persistent revoke, which if you specify a storage file, seems to do the trick, see: Workers Guide — Revoking tasks

Beanstalkd has this

Sidekiq retires the job with an exponentially rising delay, so that a fix can be applied, instead of burying it

According to the RabbitMQ docs here, there is something called a dead letter extension, that as I can read it, could be helpful in implementing this.

Time-outs after which jobs are rescheduled, revoked or buried

Redis does not have this, you will have to build that on top. There is a time-out option for keys though.

Sidekiq retires the job with an exponentially rising delay, so that a fix can be applied

Celery has this, with acks_late option, see: Workers Guide.

Monitoring of processes

Redis does not have this, you will have to build that on top, but there are third party modules although it unlikely that they work out-of-the-box with how you have designed your system

Celery, se Monitoring and Management Guide — Celery 3.0.11 documentation

Group operations into atomic transactions

Redis has this

RabbitMQ has this

Persistence

Redis has this

Celery has this as long as the back end is configured to persist

RabbitMQ has this.

Flexible reporting and web interface, pubsub logging

• There is a PHP web interface, among others. Pubsub exists and there is e.g a python logger that publishes to Redis.
• Celery has a very ambitious support for this, in the shape of flower and django plugins, see: Monitoring and Management Guide — Celery 3.0.11 documentation
  There is even a limited curses interface for Celery.

Sequential workflow with error handling

Redis does not have this, you will have to build that on top

Celery has this, see How to create a sequential workflow with error handling in Celery

sidekiq can send exceptions to an exception notification service

Fallbacks

Redis does not have this concept, you will have to build that on top

Celery does not have this concept, you will have to build that on top

Language agnostic, easy to mix and use different languages

Redis has a defined wire protocol and support in a plethora of languages, and as long as you use a data format available on all platforms such as JSON, interoperability should not be a problem

Celery has a defined protocol as far as i can see, but it is not implemented in plethora of clients. It uses python's pickle format for serializing data as default, but it can be switched to e.g. JSON or Yaml. Celery can together with Django be used with a simple http protocol called Webhook, a protocol that according to Google searches seems to have been causing some enthusiasm back in 2009, and which today forms a part of Github's API. Celery can also operate with http posts and gets, see: HTTP Callback Tasks (Webhooks), celery/examples/celery_http_gateway.

Resque seems to run its child workers as system processes and that could open up a possibility for it to run code in other languages, but that does not seem to be how it is used. I guess one could write a worker that used STDOUT and STDIN to communicate with the child process I guess.

RabbitMQ works with AMQP and has standardization and interoperability built right in.

Documentation

Redis -well documented in a concise way. Redis itself is pretty concise, which helps. For every command there is information on how well that command scales, in "Big O" notation. there are also pages covering other aspects of Redis.

Celery is very well documented, from high level all the way down to the internals and message format.

beanstalkd - An FAQ that is one page on a five page wiki.

Resque

Zeromq - Lots of documentation and examples, given in parallel in several computer languages

I've just made a site called CheckMyCSS.com . It checks your web site for what CSS selectors in your style sheets aren't actually used and mails a report to you. It's free. 

CheckMyCSS.com . Take it for a spin!

Ubuntu 10.04 is still supported but that does not mean that you always want the software versions that are still supported. In contrast to Debian 6, there is no official backports repository for Ubuntu 10.04 containing a newer version of Redis than what the distribution is shipped with.

However David Murphy has made a newer redis (2.2) available for 10.04. I have only tested redis-cli so far but that did the job of connecting to newer servers, which the 2.1 in 10.04 did not.

PPA description Backport of redis to 10.04 LTS

Read more: Redis : David Murphy

./configure --prefix=/home/auser/where-you-want-stunnel

I'm writing  this in frustration so I might go back here and do some edits later. Anyways: I tried to use the stunnel packages supplied with the Precise Pangolin distribution. Setting up two stunnel daemons between two machines with those, is like trying to play the violin with two cold, dead fish. They start, but they do not read any configuration files (sheet music in the case of the fish). Or they might, but I doubt it since writing garbage in the config files does not make the stunnel daemons react. In fact I think the 12.04 stunnel daemons are some of the most stable things on earth in their complete insensitivity to anything happening around them, like config files in the right place or having any kind of arguments with them. And by that I mean command line arguments.

In contrast, the source tarballs are all rainbows and unicorns, printing sane and helpful stuff to stdout and stderr, and I got up and running with them in 20 mins. Not gonna dwell how big a dollop of my life the the debs disappeared with.

The included version for Redis in Debian6 (Debian Squeeze) is a bit long in the tooth (version 2.1) and lacks some of the commands you may want to use. A more modern version (currently 2.4) can be found in Debian backports for Debian 6.

Due to high demand I have prepared official Redis packages for Debian "squeeze":

Read more: Official Redis packages for Debian "squeeze" « lamby

http://packages.debian.org/squeeze-backports/redis-server

Paramiko is an ssh library for python.

The forward.py script expects one non-keyword argument, and a number of keyword arguments.

python forward.py example.com:22 -r 80 -p 8080 -r example.com:80 --user username --key secret_key_file_name

The above example takes the remote port 80 and makes it available locally as 8080.

The non-keyword argument above is example.com:22 . It connects to the ssh server at example.com running at port 22. 22 is not the port the forward goes to. It is simply the port where the ssh server is, and can be changed from 22 if you run an ssh server on a non-standard port.

The -p indicates the local port for the tunnel, i.e. the port your applications are going to connect to, i.e. your end of the tunnel.

The -r indicates the host and port that the remote server is going to forward to. The server indicated by -r may be different than the server you are connecting to, e.g.:

python forward.py example.com:22 -r 80 -p 8080 -r anotherhost.example.com:80 --user username --key secret_key_file_name

...in which case you are using example.com to reach anotherhost.example.com .

In normal ssh, an equivalent tunnel to the first example above could be written as:

ssh -L 8080:localhost:80 username@example.com

Start redis-cli that should have come with your redis distribution. Start redis-cli and type "keys *" (without the quotes) and hit return. That will show you what keys (queues) have data in them. See this stackoverflow discussion:

you can explore the Redis dataset using the 'redis-cli' tool included in the Redis distribution. Just start the tool without arguments, then type commands to explore the dataset. For instance KEYS will list all the keys matching a glob-style pattern, for instance with: keys * you'll see all the keys available.

Read more: ruby - Dumping all key/value pairs in a Redis db - Stack Overflow

 jcampbelly:

1. Try to forget the existence of "==" and use "===" instead. There are use cases for double-equals, but those are exceptions to the rule (usually cute hacks for brevity).
2. Truthy and falsy values can be difficult to grasp. Falsy: "", 0, undefined, null, NaN (0/0). Truthy: "0", 1/0, {}.
3. Type coercion is weird: (new Date() + 0) is ""Sat Oct 06 2012 12:56:45 GMT-0400 (Eastern Daylight Time)0"", (new Date() - 0) is 1349542608499. Incidentally, with implicit type coercion, the string form is local time and the numeric form is UTC.
4. Always parse integers with a base: parseInt('07') is 7, parseInt('08') is 0, parseInt('08',10) is 8.
5. Use console.log() instead of alert().
6. Primitives are passed by value, objects are passed by reference.
7. Declaring a variable without "var" makes it a global variable.
8. object['property'] is identical to object.property except that "dot notation" cannot be used with language keywords (depends on browser) or numbers and bracket notation can be used with arbitrary strings (including spaces, for example).

9. There is no scope inside of loops. Declaring a variable within a loop scope is the same as declaring it in the containing block. This can be hard to demonstrate, but take this very contrived example. The number "10" will be printed 10 times, because at the end of the loop execution there exists only one variable "i" whose value is 10, which is what the callback is referring to.

   for (var i = 0; i < 10; i++) {   setTimeout(function(){ console.log(i); }, 1000); } 

Read more: jcampbelly comments on In my first week of a JavaScript course -- Is there anything in particular I should watch out for that consistently trips up newcomers?

A free and open-source javascript shopping cart that easily integrates with your current website.

Read more: Javascript Shopping Cart - simpleCart(js)

Updated 2016-02-29

javascript - At underscore js, Can I get multiple columns with pluck method after input where method as linq select projection - Stack Overflow

ESLint - Pluggable JavaScript linter

JavaScript’s Apply, Call, and Bind Methods are Essential for JavaScript Professionals | JavaScript is Sexy

JavaScript Nested function - Stack Overflow

How can I pretty-print JSON using JavaScript? - Stack Overflow

lodash documentation

DOM libraries

Libraries that solve things that "everybody needs", by making constructs that make commonly done things simpler, and things more compatible between browsers over different platforms and versions. Usually you only use one of these, but there are ways of making them not be in conflict with one another. One of these basic libraries is often required by the frameworks.

jquery

An ubiquitous library that makes stuff easier and more cross browser compatible. Concentrates on the DOM (Document Object Model). Has a plugin ecosystem

Other basic libraries are Prototype, MooTools, Dojo and YUI.

Utility libraries

Libraries that make specific tasks easier.

atomizejs

A library that implements STM (Software transactional memory), a fairly novel way of doing concurrency

underscore.js

Functional idioms, required by Backbone

Amino

Animation and interactive graphics on the HTML5 Canvas

Google closure library

The foundation for GMail, Google documents, among other projects. Strong on widgets. Could arguably be called a framework, but according to the documentation emphasizes modularity and being usable in small parts in other code

D3.js - Data-Driven Documents

Data visualization and interaction, with some amazing examples

Functional programming libraries

Ramda Documentation

fn.js | A JavaScript library built to encourage a functional programming style & strategy.

The functional JavaScript library | functional.js

Form validation

Livevalidation

Validation library with immediate feedback

jquery.validate

Validation library that might be one step better for me, in initial tests

Frameworks

Put some overall structure onto the code and how to get things done. You choose one over the others. Tend to deal with data modelling and how to bind together an application with views, control, routes, and data persistence. Goal is to make larger scale applications doable and manageable.

backbone

powers many of the big sites. Has models, views routes and can serialize data into JSON and talk RESTfully with the server back end.

• Backbonification: migrating a large JavaScript project from DOM spaghetti to Backbone.js.:
• Developing Backbone.js Applications,
• Backbone patterns.

canjs

A framework in the same vein as backbone and ember

ember

A framework in the same vein as backbone, methinks

spine

backbone written in coffeescript, methinks

Angularjs

Like backbone but operating purely(?) with markup, it looks like.

Obviel

zope.interface and adapter stuff on the client side, in javascript

Live bindings

Helps make things update immediately in the UI when data changes, like in a normal GUI on the desktop. Some of the frameworks already contain live bindings, such a Ember, Angularjs.

knockout

makes data and presentation update immediately, like in a normal GUI on the desktop

knockback

compatibility layer between knockout and backbone

Backbone.ModelBinder

I believe it makes data and presentation update immediately, like in a normal GUI on the desktop

Slideshows

jquery cycle

Build/test/deploy frameworks

Grunt

Can apply different transformations to your code before deployment with the aid of modules, such as unit tests, minification and conformance to coding standards

Twitter Bootstrap

Responsive web design with eminently usable page layouts and css. Also contains javascript targetting the look and feel.

PhantomJS

Let's you control a headless webkit engine. The webkit engine underlies Google Chrome, Apple's Safari, the Android's standard web brwoser and many more. Webkit originally comes from KDE, a popular desktop environment on e.g. Linux. PhantomJS ought to make it possible todo very realistic testing and simulation.

Languages

Languages that are translated into javascript before being run in the browser. Debugging is going to get better when these languages start using the new source map functionality of the recent versions of Chrome/Chromium, where a breakpoint in the executed javascript code is associated with a line in the non-javascript source code. The other main browsers will hopefully follow suit implementing source maps.

coffeescript

Makes javascript look more like python. The kickstarter founded fork  CoffeeScriptRedux supports source maps, according to this blog post.

clojurescript

Lisp-like language that also exists on the server side running on the JVM. Fairly big in the JVM world, so does not live or die by the browser. Source map tweet indicates that source maps may be on the way.

typescript

Light-handed enhancement of standard javascript to aid with development in larger projects. Tries to anticipate new versions of javascript (a.k.a. ECMAScript).  Prevents a lot of late binding in order to make things more predictable and bugs easier to spot automatically. Gives more ambitious editing help than what is possible with javascript.

pyjs

Templating

Handlebars

Templating

Dust

Javascript date handling

Moment.js | Docs

datejs

Moment Timezone | Home

Moment Timezone | Docs

Time · mbostock/d3 Wiki

dosx/timezone-picker

Node.js Express

Using template engines with Express

Git

How do you merge selective files with git-merge? - Stack Overflow

Git Tip: How to "Merge" Specific Files from Another Branch [jasonrudolph.com]

Javascript testing

Intern: Software testing for humans

Buster.JS overview — Buster.JS 0.7 documentation

List of unit testing frameworks - Wikipedia, the free encyclopedia

coderwall.com : establishing geek cred since 1305712800

Jasmine: Behavior-Driven JavaScript

Cookbook | QUnit

QUnit

Sinon.JS - Documentation

Javascript testing frameworks — jorgenmodin.net

node.js

Mocha - the fun, simple, flexible JavaScript test framework

The Node Way - Testing Essentials

jhnns/rewire

mfncooper/mockery

Node.js Handbook - Testing Essentials | FredKSchott

Nightwatch.js | Node.js powered End-to-End testing framework

Vows « Asynchronous BDD for Node

testing framework node.js promise chains - Google Search

dareid/chakram

Eradicating Non-Determinism in Tests

StrongLoop | Testing and Documenting Node.js APIs with Mocha and Acquit

acquit

Home - Chai

Using Tests to Document a Node.js Module

How To Use Mocha With Node.js For Test-Driven Development to Avoid Pain and Ship Products Faster

cloudhead/hijs

Asynchronous programming

Implementing Actors in JavaScript

benlau/nactor · GitHub

mental/webactors · GitHub

AndrewBelt/hack.chat

events - How is Node.js evented system different than the actor pattern of Akka? - Stack Overflow

Embracing Async With Deferreds and Promises - Sebastian's Blog

Reactive programming - Wikipedia, the free encyclopedia

JavaScript Promises: There and back again - HTML5 Rocks

Does async/await solve a real problem?

chopachom/syncio

We have a problem with promises

Promise anti patterns · petkaantonov/bluebird Wiki

Taming the asynchronous beast with ES7

JSX in Depth | React

petkaantonov/bluebird

javascript asynchronous styles - Google Search

Testing Asynchronous JavaScript

Javascript Promises - Being Agile

You're Missing the Point of Promises

domenic/chai-as-promised

javascript - How do you properly promisify request? - Stack Overflow

node request promise - Google Search

Passing multiple arguments in promises | Better world by better software

JavaScript with Promises - Daniel Parker - Google Books

Resolving JavaScript Objects Containing Promises - Will Anderson

javascript - asynchronous or promised condition for array filter - Stack Overflow

Patterns for Asynchronous Programming with Promises | Joe Zim's JavaScript Blog

monet.js - For when you have no choice but to use JavaScript

prequest

promise-useful-utils/API.md at master · fanatid/promise-useful-utils

JavaScript with Promises - Daniel Parker - Google Books

Resolving JavaScript Objects Containing Promises - Will Anderson

javascript - asynchronous or promised condition for array filter - Stack Overflow

Ecmascript 6 & higher and typescript

 

Sign in or Register | edX

TagClass - jsdoc-toolkit - @class - A documentation generator for JavaScript. - Google Project Hosting

Language Safety Score Mark 2 - Deliberate Software

Programming Language Safety Score - Deliberate Software

ECMAScript 6: New Features: Overview and Comparison

ES6 | Node.js

ECMAScript - Wikipedia, the free encyclopedia

ES7 async functions - JakeArchibald.com

Use ECMAScript 6 Today - Tuts+ Code Article

Which TypeScript IDE - Steve Fenton

Using TypeScript in Visual Studio Code - TypeScript - Site Home - MSDN Blogs

ES6 | Node.js

lukehoban/es6features

Returns a random v4 UUID of the form xxxxxxxx-xxxx-4xxx-yxxx-xxxxxxxxxxxx, where each x is replaced with a random hexadecimal digit from 0 to f, and y is replaced with a random hexadecimal digit from 8 to b.

Read more: generate random UUIDs — Gist

With a standard Plone setup that this site used to have (and for many other systems), one new blog post on this blog would invalidate 250 pages or so, by pushing one blog post off the front page and on to the next "older blog posts >>" page. That in its turn would push one blog post on the the next older page etcetera all the way back to changing 250 pages for the total of 2500 or so blog posts on this blog.

Numbering batches in reverse with the high numbers first, and having the orphans in the beginning rather than the end saves you a lot of cache invalidation.

If you look at the "older blog posts >>" at the front page of this blog you will notice that the numbers in the url for the first page of slightly older posts are very high. That is because the blog posts are numbered backwards.

This saves a lot of cache invalidation when a new blog post is posted. Furthermore, even though the batch size of the site is 10, the front page expands to 19 items before "giving birth" to a new batched page. this means that when a new blog post is made, one page . the front page - is invalidated in the Varnish cache, and not like before, 250 pages.

This blog has about 2500 blog entries as I type this, and you can go back in time, browsing through them with the "older blog posts >>" link. You will get 10 blog posts per page. The url for each batch contains info on what sequence numbers of blog posts you want to see.

However, for some reason it is customary to number these batches starting with newest first. This means that whenever a new item is added to the sequence, every single item gets a new sequence number. Furthermore for aesthetic reasons one does not want to have a small number of posts on the last page (this is called to have "orphans"). So instead the last page is expanded to accommodate for these "orphaned" posts. If you let the first page take care of the orphans, suddenly all your batched pages will be stable forever, barring edits.

I am doing a lot of javascript programming this week so here some things I have picked up.

Javascript debuggers in Firefox and Chrome

Chrome's built in seems the better one compared to Firebug on Firefox, but some things are hidden behind miniscule non-descript widgets in Chrome. i still cannot see how to select an element with the mouse on the page and get the source code to jump there though. Other way works fine.

jquery

When you select for an element in jquery, even if it is for an element with a specific and hopefully unique id, it will return an array, so you will have to extract the one you want. The slice method works well for that: result.slice(0) for first (and possibly only) element, and result.slice(-1) for last.

The clone() method is very nice.

If you miss python's "for .. in" syntax for iterating over things, there is an ".each()" method in jquery that accomplishes the same more or less.

javascript and prices

When handlig currencies with decimals, it seems to be ok to use parseFloat() to convert it into something numerical, but that can introduce rounding errors when doing arithmetic on the numbers. That seems to be handled by using var.toFixed(2) in the end. A bit of a patchwork.

Another way would be to multiply the prices by 100 and then use parseInt and just work with integers I suppose.

Submitting two forms at once

Can be done in at least two ways in javascript. You can submit one with XML-RPC or similar but the method I used was to have the first one submit to an invisible iframe with an appropriate target atribute in the form.  In that way you are still on the original page when the second form submits. Otherwise you would be somewhere else and the second submit would not work. I found this iframe technique in a discussion forum but I have closed that page alas.

I have been doing a lot of javascript programming this week, so here are what I have learned so far about live validation.

Form validation with Livevalidation

I am using a library called Live Validation, which I so far like.

Conflict with jquery when submitting

However if you are using it in conjunction with jquery, which I do, there is one conflict. When livevalidation is turned on for a field, livevalidation also takes care of the submit handler of that form, inserting itself in a chain with the original handler placed after it. In this way livevalidation can stop the submitting of a form. However in my use case I had a jquery form submit event handler set up on loading the page, and this one fired even when validation failed since it somehow managed to sneak in before livevalidation. Since my code submits another form as part of that code, that did not work out well.

Solution was to assign my code to the onsubmit handler of the form, instead of using fancy jquery event interception. Then livevalidation can then sneak in and stop the shenanigans.

An empty field with an email validator still validates

Secondly a thing that is logical on scrutiny when it comes to livevalidation, is that even if you have marked a field that it must contain a valid e-mail address, an empty field also validates.

Why? I believe it is because there is the use case that you would like to validate optional fields too. The solution is to stick two validators on the field, one for e-mail and one for at least something (Validate.Presence in livevalidation speak).

Liveavalidation tries to validate non-existing removed  fields

If you remove input fields from your form with jquery's remove(), while these inputs still have Livevalidation validators on them, Livevalidation will still remember them and refuse to submit the form if they have any validators on them that checks for empty. Solution is to go through the inputs before you remove them and do something like this:

var fieldid = $(v).attr('id');
var aField = new LiveValidation(fieldid);
aField.disable();

Calling the Livevalidation destroy() methid is not enough unless you manage to get hold of the original Livevalidation object(s) for that field, so use disable().

Last week I had to look into a problem with a Plone production server that had performed unsatisfactory a couple of times during the week under heavy load, with lagging response times. Now, how to trouble shoot what was the problem? I had a copy of the server that looked fine when clicking around in it, or when using the Apache benchmarking tool. The solution is of course to hit the testing server with lots of requests and see what actually happens.

However since the slowness only happened under certain load patterns, wouldn't it make sense to just use the log files from the times when the problems manifested themselves? I googled a bit and found repache, that is supposed to replay Apache log files to a server. It is written in C++ and does not compile on newer Ubuntus unless you add a couple of includes. Once that was done it promptly crashed on my access_log log file.

So, time to whip something together (note that the playlogfiletoserver github repository may have newer versions of the scipts than posted below). First a pre processor that takes a log file in the Apache combined log file format and outputs a python module containing a tuple of tuples with info on when to fire each request and what request that would be:

import datetime
import re

def get_timestamp(date):
    timestamp = datetime.datetime.strptime(date, '%d/%b/%Y:%H:%M:%S')
    return timestamp
    
def parseline(line):  
    linedict = {}
    line = line.replace('\\"', '%22')
    quoteprep = line.split('"')
    try:
        (linedict['ipnumber'], tmp, tmp, linedict['date'], tmp, tmp) = quoteprep[0].split(" ")
        (linedict['method'], linedict['path'], linedict['protocol']) = quoteprep[1].split(" ")
        (tmp, linedict['returncode'], linedict['bytestransferred'], tmp) = quoteprep[2].split(" ")
        linedict['url'] = quoteprep[3]
        linedict['date'] = linedict['date'][1:]
        
    except ValueError:
        print line
        raise
    return linedict
    
fi = open('access_log', 'r')
fo = open('generatedrequests.py', 'w')
fo.write('requests = (')

first_timestamp = 0
for line in fi:
    linedict = parseline(line)
    if not first_timestamp:
        first_timestamp = get_timestamp(linedict['date'])
        time_offset = 0
    else:
        time_offset = (get_timestamp(linedict['date']) - first_timestamp).seconds
    lineout = '''(%s,"%s"\n),''' % (str(time_offset), linedict['path'])
    fo.write(lineout)
    
fo.write(')')

Secondly, some kind of user agent is needed to do the actual requesting, and it needs to be able to run requests in parallel, lest we would not be load testing the server much if we just kindly wait for it to duly process each request in sequence. I found Pete Warden's pyparallelcurl that uses pycurl/libcurl to do its magic (pyparallelcurl is also included in the playlogfiletoserver github repository for convenience).

Thirdly and lastly the script to do the actual load testing:

import datetime
import time
from generatedrequests import requests as therequests
import pyparallelcurl
import pycurl

server = 'http://www.example.com'
speedup = 4
maxparallelrequests = 100

sleepduration = 0.5/speedup
t = datetime.datetime.now()
starttime = time.mktime(t.timetuple())


curloptions = {
    pycurl.SSL_VERIFYPEER: False,
    pycurl.SSL_VERIFYHOST: False,
    pycurl.USERAGENT: 'Play log file to server user agent',
    pycurl.FOLLOWLOCATION: True
}

#c.setopt(c.MAX_RECV_SPEED_LARGE, 10000)

parallel_curl = pyparallelcurl.ParallelCurl(maxparallelrequests, curloptions)

def go(request):
    parallel_curl.startrequest(server + request[1], on_request_done)
    return

   
def on_request_done(content, url, ch, search):
    
    if content is None:
        print "Fetch error for "+url
        return
    
    httpcode = ch.getinfo(pycurl.HTTP_CODE)   
    transfertime = ch.getinfo(pycurl.TOTAL_TIME)
    if httpcode != 200:
        print "%0.3f seconds fetch error %s for %s" % (transfertime, str(httpcode),url)
        return
    else:
        print "%0.3f seconds fetch worked %s for %s" % (transfertime, str(httpcode),url)
    print "********"


def timetogo(request):
    reqtime = request[0]
    t = datetime.datetime.now()
    now = time.mktime(t.timetuple())
    return reqtime  <= (now - starttime) * speedup


for request in therequests:
    while True:
        if timetogo(request):
            go(request)
            break
        else:
            time.sleep(sleepduration)
    
    

There are probably some faulty assumptions hiding in these few lines of code, feel free to point them out in the comments below in the issue tracker or fork the code on github.

Always rely on monitor tools! Some example?

Read more: Slow Plone site? Let's use haufe.requestmonitoring — RedTurtle's Blog