For me at least. Looking at different backup solutions I have been evaluating Obnam, a python based backup application. Using Obnam v 1.15 on Ubuntu 15.10 on a laptop, backing up to a server over the network. I have now also tried version 1.19.1, same problem

Today, looking at the logs I noticed it is stuck in a loop

2016-02-19 13:19:25 INFO Locking client client1
2016-02-19 13:19:25 CRITICAL Node 0x1026 cannot be found in the node store chunksums: 2: No such file: None
Traceback (most recent call last):

It does not seem to recover from this looping. I am running obnam through a python script under Supervisord that uses subprocess to start obnam every two hours or so. Before each invocation I do a "force-lock" to get rid of any hanging locks (possibly from interrupted backup sessions, it's a laptop). There is some help here: Missing node or KeyError problems on the obnam web site, but I used more recent versions of obnam, namely 1.15 and 1.19.1.

Furthermore, yesterday I noticed obnam hanging on certain patterns in the config file. Specifically, it doesn't like patterns of this form:

exclude = /node_modules/

It will just hang when it encounters this pattern in the file system. I have not verified this extensively but it seems to hang when there is a trailing slash. This mitigated the problem:

exclude = /node_modules$, /node_modules/

I.e. first match the directory and then any underlying files.

To an extent, it seems it does. Microsoft has checked on a number of projects and found that these metrics do correlate with the quality of the software developed:

Precision is lack of false positives, i.e. 100% means it was right about all the projects pinpointed (but may have missed some)

Recall is lack of false negatives, i.e. 100% means it found all projects, but also potentially many that were false positives

Organizational structure and Code churn are different beasts than the other ones. They both are metrics of things that are to an extent inevitable. They cannot be designed away, at least not code churn. Organizational structure is about measuring turnover of people and if participants are from disparate organizations. That can be handled to some extent, but not in code.

The other four one can take a shot at, at the design level of the code.

Explanations from the paper of Code Churn Model and  Code  Complexity  Model:

Code Churn Model:

• Total Churn: Total added, modified and deleted lines of code
• Freq:  The  number  of  time  that  a  binary  was  edited during  its  development  cycle.
• Repeat  Freq:  The  number  of  consecutive  edits performed  on  a  binary. 

Code  Complexity  Model:

• (Max)(Total)  Cyclomatic  complexity
• (Max)(Total) Fan-In: # of functions calling function
• (Max)(Total) Fan-Out: # of functions called by function
• (Max)(Total) Lines of Code (LOC)
• (Max)(Total) Weighted methods per class (if any)
• (Max)(Total) Depth of Inheritance (if any)
• (Max)(Total) Coupling between objects (if any)
• (Max)(Total) Number of sub classes (if any)
• Total Global variables.

Microsoft Word - Org_Ver15_TechReport.doc - tr-2008-11.pdf

Shell script patterns for bash

Pro memoria.

http://sandstorm.io sätter ett gemensamt autentiserings- och webbgränssnitt ovanpå olika befintliga OSS webapplikationer. Om Sandstorm eller liknande lyckas får vi en massa privata och semi-privata "hubs" där ingen central aktör kan majna data

Om man går hit: https://apps.sandstorm.io/ så ger de en en tillfällig server (1h)  där man kan installera chat, fildelning osv. Bara klicka på "Demo" under valfri applikation. Fort och enkelt. MAn kan installera på egen valfri Linuxmaskin också. Kör en hemma nu.

Sandstorm flyttar datat från Facebook/Google osv till ens egen server, men integrerar ändå en massa applikationer (Rocket Chat, Gitlab, fildelning osv).
 

Introduction to the Zig Programming Language - Andrew Kelley

I've made a server that burns backup data sessions to disc, by a simple transfer of directory trees (with scp, rsync, ftp, sftp or other means). It can also burn only updated files. It runs on Linux.

Check it out here on GitHub: dvd-data-burn-server.

The purpose of DVD-data-burn-server is to burn data to a DVD disk, just by a simple transfer of a directory tree (with scp, rsync, ftp, sftp or anything else that can move files) to the server. DVD-data-burn-server will wait to burn the tree until the transfer is complete, waiting for a special file to say it's time to burn. More transfers of directory trees can be done, burning new sessions to the disk, until the disk is full.

Synopsis

python burn_server.py 

or

python burn_server.py -c <config_file>

The config file should by default be in etc/burn_server.config. Use etc/burn_server.config.example as a template.

Dependencies

You need the xorriso DVD-burning program installed. On Ubuntu Linux that is:

apt-get install xorriso

Tested on Ubuntu 15.04 and Ubuntu 15.10 with Xorriso 1.3.2 with a Samsung SE-208GB external portable USB DVD burner drive.

Use cases

• Any data you want to commit to a read-only media while the data is fresh

• Logs to be burned soon after the events are happening

• New photos or other newly created stuff (e-mails, text messages, documents) that you want to have a read-only copy of if hard disks or cloud storage get inaccessible or involuntarily encrypted

How it works

Directories and files below can have any names you like

• There is a watched directory that DVD-data-burn-server watches, for sessions to burn. Set the path to it in the config file

• Inside of the watched directory you put session directories

• Inside each session directory you make a data directory, where you put the file tree you want to burn in a session

• Finally, when it is time to burn the session, you leave a time to burn file in the session directory, one step above your data. This file can have any name and contain anything

• DVD-burn-server detects the time to burn file and burns the data directory to disc in a session

If you are using DVD+R, you can burn 153 sessions before it's time to switch in a new disk. For DVD-R it is 99 sessions.

After burning, the server removes the time to burn file and the session directory gets "-burned" and a time stamp appended to its name

An example, burning the files of my_files_dir via rsync:

Example

Move my_files_dir over to the server and nest it inside an empty dir, e.g. myfiles-session-dir:

rsync -r my_files_dir server:/path/to/watched_dir/myfiles-session-dir/

Wait for the files to be transferred over to /path/to/watched_dir/myfiles-session-dir/my_files. When ready, put some kind of file inside the /path/to/watched_dir/myfiles-session-dir to trigger the burning (Here we are transferring the file /etc/issue which is always available on Fedora, CentOS, RHEL, Debian and Ubuntu systems):

scp /etc/issue server:/path/to/watched_dir/myfiles-session-dir/

That file will trigger the burning!

So in essence, data should be wrapped two levels deep, like this:

-watched_dir (already on the server)
    -session_dir (created by you)
        -dir_to_be burned (sub directory with your data)
           files.txt
           -dir1
           -dir2
            ...and so on

With the time to burn file it looks like this and is ready for burning:

-watched_dir (already on the server)
    -session_dir (created by you)
        a_file.txt (signals ready for burning)
        -dir_to_be burned (sub directory)
           files.txt
           -dir1
           -dir2
            ...and so on

Error handling

No much. If the server stops, it may be because the disc is full. Then it is time for manual intervention, put a new disc in the DVD-burner and restart the server.

Running it as a service

I use supervisor for this. This is an example configuration file for supervisor, to be put in /etc/supervisor/config.d/ or equivalent:

[program:dvd_data_burn_server]
command = /path/to/python /path/to/burn_server/burn_server.py
process_name=%(program_name)s
numprocs=1
directory=/path/to/burn_server
stopsignal=TERM
user=auser

Config

Look at the example config

session_server.py

session_server.py is not needed to use burn_server.py, but can be useful. session_server.py watches a directory and every n minutes it checks if there are any new files, in which case it transfers only those files to a new session directory in the watched directory of burn_server.py, and signals it to burn them to disc. The idea is to skim off any new files appearing in a backup and burn only those.

Check etc/session_server.config.example for configuration settings. If you would be using rsync to update your backup, use the -u or -c switch so that only newly transferred files are timestamped as new. See /index_html/rsync-and-the-resulting-time-stamps

Config file should by default be in etc/session_server.config.

It may be possible to do something similar with xorriso directly, but the man page for xorriso 1.3.2 lists 344 command line switches, and some of them can be combined...

Running session_server as a service

I use supervisor for this. This is an example configuration file for supervisor, to be put in /etc/supervisor/config.d/ or equivalent:

[program:dvd_data_burn_server]
command = /path/to/python /path/to/session_server/session_server.py
process_name=%(program_name)s
numprocs=1
directory=/path/to/session_server
stopsignal=TERM
user=auser

session_server.py config

Look at the example config

Future directions

Some ideas:

• Better error handling

• Check how many sessions and how much space is left before burning (however the server should just terminate on error, and after replacement of the disc and restart of the server, it should commence burning again. Hopefully)

• Having DVD-data-burn-server leave signal files of its own in the watched directory when something goes wrong. E.g. a disk gets full and DVD-data-burn-server can leave a "Disk is full" signal file, which can then be picked up by a notifier that e-mails or texts the admin, or files a ticket that it is time to change disks.

• Supporting several DVD burners so one can move on to the next when one gets full

• You can already use several DVD burners in parallel by having this program run in multiple instances with different config files with different device settings

I needed a way of getting rsync to timestamp the transferred files in such a way that the time stamp on the destination side reflects when the file appeared in its current form on the destination side.

This is so that I can find the files that are new or modified with a "find -cmin n" command, that should get all files that have appeared new or modified during the last n minutes, and burn them to disc.

I tested with no flags (besides -r), with -t flag, with the -u flag and with the -c flag. Both the -u and -c flags seem to do the job. -r flag was switched on for all.

no flags (besides -r)
All files get the timestamp of the most recent transfer, whether they are new/modified or not.

-t flag
All files get the timestamp they have on the source side

-u flag
New files get the timestamp of the transfer time, old files keep the timestamp of when they were transferred. Files with an updated modification date (e.g. with touch), get an updated date

-c flag
New files get the timestamp of the transfer time, old fileskeep the timestamp of when they were transferred. Files with an updated modification date (e.g. with touch), and no other changes will not get an updated date

Untested by me. From the man page:

 Configuration files and settings
       --config=FILE
              add FILE to config files

       --dump-config
              write out the entire current configuration

       --dump-setting-names
              SUPPRESSHELP

       --help-all
              show all options

       --list-config-files
              SUPPRESSHELP

       --no-default-configs
              clear list of configuration files to read
 

Which ought to mean that this should work:

obnam backup $HOME --no-default-configs --config=FILE

But what about if one goes completely the other way and tries to make it so easy to fake messages that no one can be sure the screen shot published, is actually authentic?

Summary: Make a chat app with built in tools for editing received messages and taking screen shots of the edited versions, and make sure these tools are very easy to use.

In that way any screen shot from the app will be accused of being faked and edited, and there is no way of knowing if it was indeed the original chat message received. This gives plausible deniability to the person sending the chat message. Instead of  expiring the chat message, one is expiring the trust in the screen shot being original.

There are a number of chat applications that try to preserve the secrecy of messages by expiring them at the receiver's end after a short amount of time. However, one can of course always make a screen shot on the receiving end and preserve it forever, and then present it at a later inconvenient time for the sender.

But what about if one goes completely the other way and tries to make it so easy to screenshot and also fake messages that no one can be sure the screen shot published, is actually authentic?

That could be an interesting chat app, although it of course could be severely misused as well.

These researchers have discovered the perfect password that’s also easy to remember - The Washington Post

their paper

Online generator here

A very good 1 hr talk on simplicity in programming http://www.infoq.com/presentations/Simple-Made-Easy … by Rich Hickey. Declarative, queues, no objects, no conditionals

Hat tip to Mikael Ståldal.

About 15 seconds evaluation time per app from Apps · Sandstorm Oasis, using sandstorm.io's demo account that is automatically created for you, try it out yourself - it is very quick and seamless!

Just select install for an app and you will get the option to try it out on their demo server.

I have starred my favorites below with ****, and I have not rated the other ones; they may still be good. Please note that 15 seconds of testing time per app is not a lot of time :)

The reason so many get stars is probably because I selected the ones I was interested in.

• MediaGoblin - no bulk upload
• Lychee - Photo manager with albums and drag-and-drop upload ****
• Davros - OwnCloud compatible file manager with drag-and-drop and desktop clients, seems awesome ****
• Telescope - posting of links
• Hacker CMS - Disarmingly simple, actually quite nice ****
• Tiny RSS reader - Works fine, can pick up feeds automatically from parsing HTML Pages, works at least with this blog! ****
• Apache Wave - Old Google Wave, or new Google Wave I suppose.
• Framadate - Your own doodle for scheduling meetings. confusing use of check marks and a bit worse handling of times than Doodle, but otherwise fine ****
• Giftr - an app for coordinating your friends gift-giving to you, simple but probably does the job, hard to gauge in a single-user demo environment
• Let's chat - "Error: remote exception: remote exception: remote exception: remote exception: remote exception: remote exception: connect(): Connection refused C++ location:(remote):?? type: disconnected"
• Dillinger - Markdown editor works, but then again how could it not? It is basically all client side, nothing stored server side afaict.
• FileDrop - Simple, can see use for it just between your own computers. The Hacker CMS and the Media Goblin could use this drag-and-drop stuff ****
• Rocket chat - impressive, haven't tried the voice chat ****
   

Untested by me

 

IPE R2 – Industrial Perennial Environment Release 2 aims to be a blackout-proof Linux distribution-alike for the Raspberry PI. It is based on buildroot, and available both as an Image and as an SDK, where you can compile your own Image.

Read more: Link - IPE R2 | NutCom Services, Inc.

Backups with Obnam

Expansion with ~ doesn't work in config files. On the Backups with Obnam page, you can read:

As an example, your first backup might have the following configuration:

obnam backup -r /media/backups/tomjon-repo ~ \
    --exclude ~/Downloads

...but that doesn't work in a config file:

exclude = ~/Downloads

Not working! Reason I guess: There is no shell to expand the ~ character. So you'll have to write:

exclude = /full/path/to/Downloads

On an Ubuntu 15.04 computer I could not install linux-headers for the right kernel versions. Turned out that the kernel installed was from an older Ubuntu, that did not have any corresponding kernel headers in the enabled repositories. Doing the apt-get update && apt-get upgrade dance made no difference.

Sneaky.

I did sudo apt-get install linux-generic and bob's your uncle.

 

This is a bug in the distribution upgrade script. You can install the correct kernel by running: sudo apt-get install linux-generic After a reboot you should have the 4.2 kernel.

Read more: Link - upgrade - Ubuntu kernel not updating with 15.10 - Ask Ubuntu

Cap'n Proto: Introduction

Something to do with protocol buffers and used as a general data channel in sandstorm.io . They themselves compare it favorably to google protocol buffers.

From their front page:

• Incremental reads: It is easy to start processing a Cap’n Proto message before you have received all of it since outer objects appear entirely before inner objects (as opposed to most encodings, where outer objects encompass inner objects).
• Random access: You can read just one field of a message without parsing the whole thing.
• mmap: Read a large Cap’n Proto file by memory-mapping it. The OS won’t even read in the parts that you don’t access.
• Inter-language communication: Calling C++ code from, say, Java or Python tends to be painful or slow. With Cap’n Proto, the two languages can easily operate on the same in-memory data structure.
• Inter-process communication: Multiple processes running on the same machine can share a Cap’n Proto message via shared memory. No need to pipe data through the kernel. Calling another process can be just as fast and easy as calling another thread.
• Arena allocation: Manipulating Protobuf objects tends to be bogged down by memory allocation, unless you are very careful about object reuse. Cap’n Proto objects are always allocated in an “arena” or “region” style, which is faster and promotes cache locality.
• Tiny generated code: Protobuf generates dedicated parsing and serialization code for every message type, and this code tends to be enormous. Cap’n Proto generated code is smaller by an order of magnitude or more. In fact, usually it’s no more than some inline accessor methods!
• Tiny runtime library: Due to the simplicity of the Cap’n Proto format, the runtime library can be much smaller.

All your photos, documents and other digital memories. Seems to be a bit rough around the edges still, not sure what to do with it once it is installed and stuff is imported. Images are not correctly rotated and other fineties. You can retrieve stuff via tags at least in the web browser. Will see if I'll explore it further.

It is all written in golang.

Camlistore is a set of open source formats, protocols, and software for modeling, storing, searching, sharing and synchronizing data in the post-PC era. Data may be files or objects, tweets or 5TB videos, and you can access it via a phone, browser or FUSE filesystem.

Camlistore (Content-Addressable Multi-Layer Indexed Storage) is under active development. If you're a programmer or fairly technical, you can probably get it up and running and get some utility out of it.

./bin/camlistored - for starting it

./bin/camput file -filenodes <your files> - for importing stuff (at least I did it that way)

./bin/cammount - for mounting a FUSE file system, but you cannot view much intelligible right now

Camlistore

The Zika virus has spread to much of South America and it seems to be linked to brain damage in children born to infected mothers. An interesting debate on genetically modified mosquitoes has emerged:

If you search for Zika and genetic modification, at the moment you get links to articles from a number of news sites.

The more reputable ones (The Guardian, Bloomberg, etc.) write that genetically modified mosquitoes can help curb the spread of the Zika virus, while some other news sites that starts with "daily" in the UK, and Russia Today put forward the hypothesis that the genetically modified mosquito contributed to the spread of The Zika virus.

I do not know which hypothesis is correct, but the interesting thing is that what later will be proved true may greatly influence public attitudes to genetically modified organisms in general. If it appears that an increased use of these mosquitoes helps, it will be more sympathetic. If it is found that the mosquito has instead contributed to the epidemic, it might be a nail in the coffin for these experiments.

It seems that The Zika virus started moving from its previous heartland a few years ago by way of French Polynesia, and it may simply be that it is the jet age with its rapid transportation of humans over large distances that is the decisive factor behind the epidemic: The disease simply thrives in new areas, once it was offered the chance to get there.

That said, I do not know myself what is true in this case, but I think generally you should not genetically modify things that are:

• Small
• Can multiply

When you are unsure about the odds, take a look at the potential outcomes and it becomes quite clear that self-replicating stuff should not be meddled with, or created.'

And even if these modified mosquitoes aren't meant to breed in any meaningful sense, I am not convinced of the accuracy and stability of what has been created. I found this comment from Reddit and I can not assess the veracity of it, but it gives pause for thought and I find that the reasoning in it would be worthy of further exploration, with the help of someone more knowledgeable in the area:

sky_s 2 points

2 days ago  

I'm personally pretty skeptical about the claims here, but you are the person who'se made the most incorrect claim I've read so far. Your claim that DNA is just DNA is too simplistic, and because of that you're just wrong. The way that most modifications on non-bacteria are made is by using either a polymerase or virus and then inserting an active sequence of DNA.

A few major concerns pop up with lab genetic modifications vs natural modification Firstly, most of the methods lead to DNA which is orders of magnitude more unstable than normal DNA, leading to horizontal transfer across the genome. This is in part due to the tails used needing to initially bind with foreign DNA, since they had to once readily unbind from something else for the initial implanting they are generally much more prone to repeating that later.

Another effect is that promotion is much more poorly regulated in GMO than natural mutations, since in nature besides base sequence there are steric effects that lead to proper regulation of promotion. These genes have sequences called promoters which bind due to some stimuli, and this binding allows for gene readout to begin after it. In viral techniques, there is a growing concern since it seems that in some cases active viral genes is also transferred to the organism. Sometimes it's because we thought they were inert, other times its just carelessness.

For animals the two primary sources of mutation are a single base pair being incorrectly replaced, or an extra insertion of a base pair. Both are much less likely to result in adverse steric effects and the cell can trigger aptosis if the error is particularly egregious. If not aptosis, then it still has to get through natural selection if a gamete mutated poorly. GMO is a very interesting technology, heck I've done bacterial genetic modification before, but I think it's far too underdeveloped a field for people to foolishly claim that it's indistinguishable from nature.

Zikaviruset har spritt sig mycket i Sydamerika och det tycks som att det eventuellt kan orsaka hjärnskador på foster. Då har en intressant debatt om genetiskt modifierade myggor dykt upp:

Om man söker på Zika och genetisk modifiering på engelska just nu så får man träffar på artiklar från ett antal nyhetssajter.

De mer välrenommerade (The Guardian, Bloomberg, m fl) skriver att en genetiskt modifierad mygga kan hjälpa till med att stävja Zikavirusets framfart, medan några andra nyhetssajter som startar på "daily" i Storbritannien samt Russia Today framkastar hypotesen att den genetiskt modifierade myggan bidragit till spridningen av Zikaviruset.

Jag vet inte vilken hypotes som är riktig, men det intressanta är att vad som senare visar sig vara sant kan komma att kraftigt påverka allmänhetens inställning till genetiskt modifierade organismer: Om det visar sig att ett utökat bruk av dessa myggor hjälper, kommer den att bli mer välvilligt inställd, om det visar sig att myggan bidragit till epidemin kan det istället bli spiken i kistan för dessa experiment.

Det tycks som att Zikaviruset började röra på sig för några år sedan via Franska Polynesien och det kan helt enkelt vara så att det är jetåldern med sina snabba transporter som är faktorn bakom epidemin och att sjukdomen helt enkelt sprider sig över nya områden, inte för att den muterat eller bärs fram av genmanipulerade mygg, utan för att den nu fått fotfäste där och att det sedan går snabbt.

Som sagt, jag vet inte själv vad som är sant i det här fallet, men jag anser allmänt att man inte ska genmanipulera saker som är:

• Små
• Kan föröka sig

Och även om det är tänkt att den modifierade myggan inte ska föröka sig, så är jag inte övertygad om precisionen och stabiliten i hur de gör det hela. Hittade denna kommentar från Reddit och kan inte bedöma kompetensen på personen, men finner resonemanget värt att utforska med hjälp av någon kunnig i området:'

sky_s 2 points

2 days ago  

I'm personally pretty skeptical about the claims here, but you are the person who'se made the most incorrect claim I've read so far. Your claim that DNA is just DNA is too simplistic, and because of that you're just wrong. The way that most modifications on non-bacteria are made is by using either a polymerase or virus and then inserting an active sequence of DNA.

A few major concerns pop up with lab genetic modifications vs natural modification Firstly, most of the methods lead to DNA which is orders of magnitude more unstable than normal DNA, leading to horizontal transfer across the genome. This is in part due to the tails used needing to initially bind with foreign DNA, since they had to once readily unbind from something else for the initial implanting they are generally much more prone to repeating that later.

Another effect is that promotion is much more poorly regulated in GMO than natural mutations, since in nature besides base sequence there are steric effects that lead to proper regulation of promotion. These genes have sequences called promoters which bind due to some stimuli, and this binding allows for gene readout to begin after it. In viral techniques, there is a growing concern since it seems that in some cases active viral genes is also transferred to the organism. Sometimes it's because we thought they were inert, other times its just carelessness.

For animals the two primary sources of mutation are a single base pair being incorrectly replaced, or an extra insertion of a base pair. Both are much less likely to result in adverse steric effects and the cell can trigger aptosis if the error is particularly egregious. If not aptosis, then it still has to get through natural selection if a gamete mutated poorly. GMO is a very interesting technology, heck I've done bacterial genetic modification before, but I think it's far too underdeveloped a field for people to foolishly claim that it's indistinguishable from nature.

Den kända mytkrossarsajten Snopes.com har skrivit om den Reddit-tråd ur vilken ovanstående kommentar är hämtad, och är skeptisk men inte kategoriskt avvisande till hypotesen: Zika Virus Caused by GMO Mosquitos? : snopes.com

This server untested by me:

Deploy a private npm registry in 5 minutes · cnpm/cnpmjs.org Wiki

cnpm/cnpmjs.org: Private npm registry and web for Enterprise (China mirror: http://npm.taobao.org);

cnpmjs.org: Private npm registry and web for Company

Other options:

sinopia