2012-08-30

Attenuating Linux dmesg USB Spam

For some reason, my usb mouse and keyboard, hooked up to a kvm, freeze intermittently when switching back and forth to the system. After wasting way too much time trying to figure out why, my workaround is to switch to a console before switching off system.

However, the console becomes flooded with the USB modules noisy spam, as does dmesg, of course.

Here are my naive, uninformed, lazy kludges to deal with this, and other noise.

First, an example of the usb printk noise:

[87934.587528] usb 2-2.2: USB disconnect, device number 20
[87944.107520] usb 2-2.2: new low-speed USB device number 21 using uhci_hcd
[87944.248817] input: Logitech Trackball as /devices/pci0000:00/0000:00:1d.0/usb2/2-2/2-2.2/2-2.2:1.0/input/input27
[87944.248917] hid-generic 0003:046D:C404.0014: input,hidraw0: USB HID v1.10 Mouse [Logitech Trackball] on usb-0000:00:1d.0-2.2/input0
[91382.140601] usb 2-2.2: USB disconnect, device number 21
[91390.507608] usb 2-2.2: new low-speed USB device number 22 using uhci_hcd
[91390.650062] input: Logitech Trackball as /devices/pci0000:00/0000:00:1d.0/usb2/2-2/2-2.2/2-2.2:1.0/input/input28
[91390.650176] hid-generic 0003:046D:C404.0015: input,hidraw0: USB HID v1.10 Mouse [Logitech Trackball] on usb-0000:00:1d.0-2.2/input0
A glance at /usr/src/linux/.config shows:
# CONFIG_USB_DEBUG is not set
# CONFIG_USB_ANNOUNCE_NEW_DEVICES is not set
(this is for kernel 3.5.3)
And in /usr/src/linux/drivers/usb/core/hub.c:

#ifdef CONFIG_USB_ANNOUNCE_NEW_DEVICES
static void show_string(struct usb_device *udev, char *id, char *string)
{
        if (!string)
                return;
        dev_printk(KERN_INFO, &udev->dev, "%s: %s\n", id, string);
}

So, why the spam? Because these aren't wrapped:
        dev_info(&udev->dev, "USB disconnect, device number %d\n",
                        udev->devnum);

                dev_info(&udev->dev,
                                "%s %s USB device number %d using %s\n",
                                (udev->config) ? "reset" : "new", speed,
                                devnum, udev->bus->controller->driver->name);
Solution: Wrap:
#ifdef CONFIG_USB_ANNOUNCE_NEW_DEVICES
        dev_info(&udev->dev, "USB disconnect, device number %d\n",
                        udev->devnum);
#endif

For Xorg.0.log, every switch back and forth results in this:

[ 51354.054] (II) config/udev: Adding input device Logitech Trackball (/dev/input/mouse0)
[ 51354.054] (II) No input driver specified, ignoring this device.
[ 51354.054] (II) This device may have been added with another device file.
[ 51354.054] (II) config/udev: Adding input device Logitech Trackball (/dev/input/event6)
[ 51354.054] (**) Logitech Trackball: Applying InputClass "evdev pointer catchall"
[ 51354.054] (II) Using input driver 'evdev' for 'Logitech Trackball'
[ 51354.054] (**) Logitech Trackball: always reports core events
[ 51354.054] (**) evdev: Logitech Trackball: Device: "/dev/input/event6"
[ 51354.054] (--) evdev: Logitech Trackball: Vendor 0x46d Product 0xc404
[ 51354.054] (--) evdev: Logitech Trackball: Found 3 mouse buttons
[ 51354.054] (--) evdev: Logitech Trackball: Found scroll wheel(s)
[ 51354.054] (--) evdev: Logitech Trackball: Found relative axes
[ 51354.054] (--) evdev: Logitech Trackball: Found x and y relative axes
[ 51354.054] (II) evdev: Logitech Trackball: Configuring as mouse
[ 51354.054] (II) evdev: Logitech Trackball: Adding scrollwheel support
[ 51354.054] (**) evdev: Logitech Trackball: YAxisMapping: buttons 4 and 5
[ 51354.054] (**) evdev: Logitech Trackball: EmulateWheelButton: 4, EmulateWheelInertia: 10, EmulateWheelTimeout: 200
[ 51354.054] (**) Option "config_info" "udev:/sys/devices/pci0000:00/0000:00:1d.0/usb2/2-2/2-2.2/2-2.2:1.0/input/input24/event6"
[ 51354.054] (II) XINPUT: Adding extended input device "Logitech Trackball" (type: MOUSE, id 8)
[ 51354.054] (II) evdev: Logitech Trackball: initialized for relative axes.
[ 51354.054] (**) Logitech Trackball: (accel) keeping acceleration scheme 1
[ 51354.054] (**) Logitech Trackball: (accel) acceleration profile 0
[ 51354.054] (**) Logitech Trackball: (accel) acceleration factor: 2.000
[ 51354.054] (**) Logitech Trackball: (accel) acceleration threshold: 4

[ 58510.586] (II) config/udev: removing device Logitech Trackball
[ 58510.587] (II) evdev: Logitech Trackball: Close
[ 58510.587] (II) UnloadModule: "evdev"

The solution for this spam didn't require a recompile --- only digging through the usual delightful lack of anything resembling documentation for X.
$ X -help
yields:

-verbose [n]           verbose startup messages
-logverbose [n]        verbose log messages

Great! So, what should [n] be to do less log messages? Oh, right, who'd want to know that? 

It turns out it's somewhat similar to kernel logging, so higher numbers are noisier.  Based on the source:
/* Flags for log messages. */
typedef enum {
    X_PROBED,                   /* Value was probed */
    X_CONFIG,                   /* Value was given in the config file */
    X_DEFAULT,                  /* Value is a default */
    X_CMDLINE,                  /* Value was given on the command line */
    X_NOTICE,                   /* Notice */
    X_ERROR,                    /* Error message */
    X_WARNING,                  /* Warning message */
    X_INFO,                     /* Informational message */
    X_NONE,                     /* No prefix */
    X_NOT_IMPLEMENTED,          /* Not implemented */
    X_UNKNOWN = -1              /* unknown -- this must always be last */
} MessageType;
/* Returns the Message Type string to prepend to a logging message, or NULL
 * if the message will be dropped due to insufficient verbosity. */
static const char *
LogMessageTypeVerbString(MessageType type, int verb)
{
    if (type == X_ERROR)
        verb = 0;

    if (logVerbosity < verb && logFileVerbosity < verb)
        return NULL;

    switch (type) {
    case X_PROBED:
        return X_PROBE_STRING;
    case X_CONFIG:
        return X_CONFIG_STRING;
    case X_DEFAULT:
        return X_DEFAULT_STRING;
    case X_CMDLINE:
        return X_CMDLINE_STRING;
    case X_NOTICE:
        return X_NOTICE_STRING;
    case X_ERROR:
        return X_ERROR_STRING;
    case X_WARNING:
        return X_WARNING_STRING;
    case X_INFO:
        return X_INFO_STRING;
    case X_NOT_IMPLEMENTED:
        return X_NOT_IMPLEMENTED_STRING;
    case X_UNKNOWN:
        return X_UNKNOWN_STRING;
    case X_NONE:
        return X_NONE_STRING;
    default:
        return X_UNKNOWN_STRING;
    }
}

So, the easy solution to kill the spam:
$ vi /etc/X11/xinit/xserverrc
change
exec /usr/bin/X -nolisten tcp "$@"
to
exec /usr/bin/X -logverbose 1 -nolisten tcp "$@"

(YMMV. This is with Debian (#!) )
This allows the usual startup messages for debugging, but gets rid of the constant 'Look at me!" kvm switch messages.

And a bit of bonus irritation, on boot, I was getting these friendly messages:

udevd[25342]: failed to execute '/lib/udev/mtp-probe' 'mtp-probe /sys/devices/pci0000:00/0000:00:1d.0/usb2/2-2/2-2.2 2 20': No such file or directory
udevd[25656]: failed to execute '/lib/udev/mtp-probe' 'mtp-probe /sys/devices/pci0000:00/0000:00:1d.0/usb2/2-2/2-2.2 2 21': No such file or directory
udevd[26105]: failed to execute '/lib/udev/mtp-probe' 'mtp-probe /sys/devices/pci0000:00/0000:00:1d.0/usb2/2-2/2-2.2 2 22': No such file or directory
udevd[10556]: failed to execute '/lib/udev/mtp-probe' 'mtp-probe /sys/devices/pci0000:00/0000:00:1d.0/usb2/2-2/2-2.2 2 23': No such file or directory

My first thought was to remove mtp, since it's a worthless Microsoft abortion... but, since vlc, which is occasionally useful, depends on it, the more proper fix seems to be this:
$ vi /lib/udev/rules.d/69-libmtp.rules
and comment out this line:

# Autoprobe vendor-specific, communication and PTP devices
ENV{ID_MTP_DEVICE}!="1", ENV{MTP_NO_PROBE}!="1", ENV{COLOR_MEASUREMENT_DEVICE}!="1", ENV{libsane_matched}!="yes", ATTR{bDeviceClass}=="00|02|06|ef|ff", PROGRAM="mtp-probe /sys$env{DEVPATH} $attr{busnum} $attr{devnum}", RESULT=="1", SYMLINK+="libmtp-%k", MODE="660", GROUP="audio", ENV{ID_MTP_DEVICE}="1", ENV{ID_MEDIA_PLAYER}="1"

And done! Needles are easier to find when the haystack isn't augmented by rotten weeds...



2012-05-23

A Sledgehammer For Chrome

I upgraded Chrome in my VM, to version 19.0.1084.46.

Which, apparently, forces use of the GPU Process task.

And, in a nice lovely screw you from the kind folks at Google, there's no way to disable this.

The end result is having the VM slow to a crawl (2 second pause between mouse movements, ignored clicks, etc...).  Fixable by the agony of getting to the task manager and killing the GPU process by hand, and then, doing it again when it restarts.

My naive effort was to use Chrome's commandline switches... none of which solved it, and I eventually got to where I ran out of room in the shortcut target, and then in the Windows run dialog.  So, I found myself with this on a commandline:

C:\Documents and Settings\Administrator\Local Settings\Application Data\Google\Chrome\Application=>chrome --disable-webgl --disable-accelerated-2d-canvas --disable-accelerated-layers --disable-accelerated-plugins --disable-accelerated-video --blacklist-webgl --disable-threaded-compositing --disable-gl-multisampling --disable-threaded-animation --disable-accelerated-compositing --blacklist-accelerated-compositing --disable-accelerated-painting --disable-accelerated-filters --disable-3d-apis 

And, still, chrome://gpu-internals happily showed everything (and I mean everything, except the CSS bit) disabled... and yet, still it started.

So much for a big hammer.

But, ... a kludge was found... using --gpu-startup-dialog.  This pops up a window (with only 'ok'), but since it doesn't switch modes, it's easy to kill the GPU Process in the task manager.

However, the final solution to running the broken, retarded, I-loved-this-browser-why'd-they-have-to-ruin-it Chrome version in a VM?

--gpu-launcher=pieceofshit

Using that, I didn't even have to do any of the other disabling; in a VM, leaving everything accelerated, the GPU Process never starts, everything's smooth, and fast, just the way it should be. (I'm assuming it works because by passing the launcher garbage, it bails instead of trying to keep going. Hopefully this 'feature' doesn't get fixed.)

Reference:
Chrome Commandline Switches

2012-05-15

Using Conky's Template[x]

Conky's neat, but doesn't appear to allow user variables, and the documentation leaves a lot to be desired when it comes to templates.  I found myself desperate for some way to clean up a .conkyrc when using Teo's weather scripts. (They work well, but are insane to edit by hand --- and the gui program (conky companion) appears to require using a mouse to drag. every. single. element. to one's configuration.)

Here's the stock USA Image Accuweather script from TEO:
Screenshot of Teo's Accuweather Conky Script
Stock Accuweather Conky

TEXT
${font Arial:size=12}${color ffffe5e59595}WEATHER ${font}
${hr 2}$color${texeci 500 bash $HOME/Accuweather_Conky_USA_Images/acc_usa_images}
${image $HOME/Accuweather_Conky_USA_Images/cc.png -p 0,70 -s 180x108}
${font Arial:size=10}${execpi 600 sed -n '3p' $HOME/Accuweather_Conky_USA_Images/curr_cond}${font}
${goto 200}${color ffe595}TEMP:$color${alignr}${execpi 600 sed -n '4p' $HOME/Accuweather_Conky_USA_Images/curr_cond}°F (${execpi 600 sed -n '5p' $HOME/Accuweather_Conky_USA_Images/curr_cond}°F)
${goto 200}${color ffe595}WIND:$color${alignr}${execpi 600 sed -n '6p' $HOME/Accuweather_Conky_USA_Images/curr_cond} ${execpi 600 sed -n '14p' $HOME/Accuweather_Conky_USA_Images/curr_cond}
${goto 200}${color ffe595}HUM:$color${alignr}${execpi 600 sed -n '7p' $HOME/Accuweather_Conky_USA_Images/curr_cond}
${goto 200}${color ffe595}PRESS:$color${alignr}${execpi 600 sed -n '8p' $HOME/Accuweather_Conky_USA_Images/curr_cond}
${goto 200}${color ffe595}CLOUD COVER:$color${alignr}${execpi 600 sed -n '9p' $HOME/Accuweather_Conky_USA_Images/curr_cond}
${goto 200}${color ffe595}UV INDEX:$color${alignr 10}${execpi 600 sed -n '10p' $HOME/Accuweather_Conky_USA_Images/curr_cond}
${goto 200}${color ffe595}DEW POINT:$color${alignr}${execpi 600 sed -n '11p' $HOME/Accuweather_Conky_USA_Images/curr_cond}
${goto 200}${color ffe595}AM. OF PRECIP.(1hr):$color${alignr}${execpi 600 sed -n '12p' $HOME/Accuweather_Conky_USA_Images/curr_cond}
${goto 200}${color ffe595}VISIB.:$color${alignr}${execpi 600 sed -n '13p' $HOME/Accuweather_Conky_USA_Images/curr_cond}
${goto 200}${color ffe595}SUNRISE:$color${alignr}${execpi 600 sed -n '15p' $HOME/Accuweather_Conky_USA_Images/curr_cond}
${goto 200}${color ffe595}SUNSET:$color${alignr}${execpi 600 sed -n '16p' $HOME/Accuweather_Conky_USA_Images/curr_cond}
${hr 1}
${goto 50}${font Arial:size=10}${color ffe595}TODAY$color$font${goto 230}${font Arial:size=10}${color ffe595}TONIGHT$color$font${image $HOME/Accuweather_Conky_USA_Images/today.png -p 0,230 -s 120x72}${image $HOME/Accuweather_Conky_USA_Images/tonight.png -p 190,230 -s 120x72}
${font Arial:size=8}${execpi 600 sed -n '1p' $HOME/Accuweather_Conky_USA_Images/messages_curr}${goto 195}${execpi 600 sed -n '3p' $HOME/Accuweather_Conky_USA_Images/messages_curr}$font
${font Arial:size=8}${execpi 600 sed -n '2p' $HOME/Accuweather_Conky_USA_Images/messages_curr}${goto 195}${execpi 600 sed -n '4p' $HOME/Accuweather_Conky_USA_Images/messages_curr}$font
${color ffe595}TEMP: $color${execpi 600 sed -n '28p' $HOME/Accuweather_Conky_USA_Images/tod_ton}°F${goto 195}${color ffe595}TEMP: $color${execpi 600 sed -n '32p' $HOME/Accuweather_Conky_USA_Images/tod_ton}°F
${color ffe595}REAL FEEL: $color${execpi 600 sed -n '29p' $HOME/Accuweather_Conky_USA_Images/tod_ton}°F${goto 195}${color ffe595}REAL FEEL: $color${execpi 600 sed -n '33p' $HOME/Accuweather_Conky_USA_Images/tod_ton}°F
${hr 1}
${font Arial:size=9}${goto 40}${color ffe595}${execpi 600 sed -n '6p' $HOME/Accuweather_Conky_USA_Images/tod_ton}${goto 155}${execpi 600 sed -n '11p' $HOME/Accuweather_Conky_USA_Images/tod_ton}${goto 270}${execpi 600 sed -n '16p' $HOME/Accuweather_Conky_USA_Images/tod_ton}$color${font}
${image $HOME/Accuweather_Conky_USA_Images/7.png -p 0,390 -s 90x54}
${font Arial:size=8}${goto 100}H:${execpi 600 sed -n '9p' $HOME/Accuweather_Conky_USA_Images/tod_ton}°F${goto 210}H:${execpi 600 sed -n '14p' $HOME/Accuweather_Conky_USA_Images/tod_ton}°F${goto 330}H:${execpi 600 sed -n '19p' $HOME/Accuweather_Conky_USA_Images/tod_ton}°F
${font Arial:size=8}${goto 100}L:${execpi 600 sed -n '10p' $HOME/Accuweather_Conky_USA_Images/tod_ton}°F${goto 210}L:${execpi 600 sed -n '15p' $HOME/Accuweather_Conky_USA_Images/tod_ton}°F${goto 330}L:${execpi 600 sed -n '20p' $HOME/Accuweather_Conky_USA_Images/tod_ton}°F
${execpi 600 sed -n '1p' $HOME/Accuweather_Conky_USA_Images/messages}${goto 128}${execpi 600 sed -n '3p' $HOME/Accuweather_Conky_USA_Images/messages}${goto 255}${execpi 600 sed -n '5p' $HOME/Accuweather_Conky_USA_Images/messages}
${execpi 600 sed -n '2p' $HOME/Accuweather_Conky_USA_Images/messages}${goto 128}${execpi 600 sed -n '4p' $HOME/Accuweather_Conky_USA_Images/messages}${goto 255}${execpi 600 sed -n '6p' $HOME/Accuweather_Conky_USA_Images/messages}
${image $HOME/Accuweather_Conky_USA_Images/12.png -p 115,390 -s 90x54}${image $HOME/Accuweather_Conky_USA_Images/17.png -p 230,390 -s 90x54}
${font Arial:size=9}${goto 40}${color ffe595}${execpi 600 sed -n '21p' $HOME/Accuweather_Conky_USA_Images/tod_ton}${goto 155}${execpi 600 sed -n '1p' $HOME/Accuweather_Conky_USA_Images/last_days}${goto 270}${execpi 600 sed -n '6p' $HOME/Accuweather_Conky_USA_Images/last_days}$color${font}
${image $HOME/Accuweather_Conky_USA_Images/22.png -p 0,505 -s 90x54}
${font Arial:size=8}${goto 100}H:${execpi 600 sed -n '24p' $HOME/Accuweather_Conky_USA_Images/tod_ton}°F${goto 210}H:${execpi 600 sed -n '4p' $HOME/Accuweather_Conky_USA_Images/last_days}°F${goto 330}H:${execpi 600 sed -n '9p' $HOME/Accuweather_Conky_USA_Images/last_days}°F
${font Arial:size=8}${goto 100}L:${execpi 600 sed -n '25p' $HOME/Accuweather_Conky_USA_Images/tod_ton}°F${goto 210}L:${execpi 600 sed -n '5p' $HOME/Accuweather_Conky_USA_Images/last_days}°F${goto 330}L:${execpi 600 sed -n '10p' $HOME/Accuweather_Conky_USA_Images/last_days}°F
${execpi 600 sed -n '7p' $HOME/Accuweather_Conky_USA_Images/messages}${goto 128}${execpi 600 sed -n '9p' $HOME/Accuweather_Conky_USA_Images/messages}${goto 255}${execpi 600 sed -n '11p' $HOME/Accuweather_Conky_USA_Images/messages}
${execpi 600 sed -n '8p' $HOME/Accuweather_Conky_USA_Images/messages}${goto 128}${execpi 600 sed -n '10p' $HOME/Accuweather_Conky_USA_Images/messages}${goto 255}${execpi 600 sed -n '12p' $HOME/Accuweather_Conky_USA_Images/messages}
${image $HOME/Accuweather_Conky_USA_Images/N2.png -p 115,505 -s 90x54}${image $HOME/Accuweather_Conky_USA_Images/N7.png -p 230,505 -s 90x54}
${font Arial:size=9}${goto 40}${color ffe595}${execpi 600 sed -n '11p' $HOME/Accuweather_Conky_USA_Images/last_days}${goto 155}${execpi 600 sed -n '16p' $HOME/Accuweather_Conky_USA_Images/last_days}${goto 270}${execpi 600 sed -n '21p' $HOME/Accuweather_Conky_USA_Images/last_days}$color${font}
${image $HOME/Accuweather_Conky_USA_Images/N12.png -p 0,620 -s 90x54}
${font Arial:size=8}${goto 100}H:${execpi 600 sed -n '14p' $HOME/Accuweather_Conky_USA_Images/last_days}°F${goto 210}H:${execpi 600 sed -n '19p' $HOME/Accuweather_Conky_USA_Images/last_days}°F${goto 330}H:${execpi 600 sed -n '24p' $HOME/Accuweather_Conky_USA_Images/last_days}°F
${font Arial:size=8}${goto 100}L:${execpi 600 sed -n '15p' $HOME/Accuweather_Conky_USA_Images/last_days}°F${goto 210}L:${execpi 600 sed -n '20p' $HOME/Accuweather_Conky_USA_Images/last_days}°F${goto 330}L:${execpi 600 sed -n '25p' $HOME/Accuweather_Conky_USA_Images/last_days}°F
${execpi 600 sed -n '13p' $HOME/Accuweather_Conky_USA_Images/messages}${goto 128}${execpi 600 sed -n '15p' $HOME/Accuweather_Conky_USA_Images/messages}${goto 255}${execpi 600 sed -n '17p' $HOME/Accuweather_Conky_USA_Images/messages}
${execpi 600 sed -n '14p' $HOME/Accuweather_Conky_USA_Images/messages}${goto 128}${execpi 600 sed -n '16p' $HOME/Accuweather_Conky_USA_Images/messages}${goto 255}${execpi 600 sed -n '18p' $HOME/Accuweather_Conky_USA_Images/messages}
${image $HOME/Accuweather_Conky_USA_Images/N17.png -p 115,620 -s 90x54}${image $HOME/Accuweather_Conky_USA_Images/N22.png -p 230,620 -s 90x54}

That just shouts "There's got to be a better way.". Here's my use of Conky templates.  I was also playing with fonts, so yes, it's most likely pretty putrid (the screenshot and colors), but the point is more editable (ie, hand tweakable) code which is functionally equivalent to the above:
Screenshot of Greyproc's Accuweather Conky
My Modified Version
#custom colors
color0 85e5ef
color1 42a5AA
color2 75dfd5
# Current Conditions
template0 ${execpi 3600 sed -n '\1' $HOME/.Accuweather_Conky_USA_Images/\2}
# Formatting boilerplate
template1 ${goto 285}${color0}\1:$color${alignr \2}
# Image boilplate (get path here to make cleaner) : assumes all images are pngs
template2 ${image $HOME/.Accuweather_Conky_USA_Images/\1.png -p \2 -s \3}
# Temp/Feels Like
template3 ${color0}${goto 75}\1: $color${template0 \2 \3}°F${goto 265}${color0}\1: $color${template0 \4 \3}°F
# template4 : Day + Day + Day label (ie, "Tomorrow Thursday Friday")
template4 ${font Caston:size=10}${color0}${alignc 100}${template0 \1 \2}${goto 210}${template0 \3 \4}${goto 355}${template0 \5 \4}$color$font
# template5 : High/Low temperature prediction text
template5 ${font Terminus:style=bold:size=10}${goto 125}${color0}\1:\ ${color}${template0 \2 \3}°F${goto 285}${color0}\1:\ $color${template0 \4 \5}°F${goto 420}${color0}\1:\ $color${template0 \6 \5}°F
# template6 : message aka weather description
template6 ${font Caston:size=9}${goto 40}${template0 \1 messages}${goto 195}${template0 \2 messages}${goto 345}${template0 \3 messages}
TEXT
${font Metal Lord:style=bold:size=20}${color1}W E A T H E R  ${font}$color${hr 2}$color${texeci 3600 bash $HOME/.Accuweather_Conky_USA_Images/acc_usa_images}
${template2 cc 50,70 180x108}
${offset 50}${font Terminus:style=bold:size=10}${template0 '3p' curr_cond} 
${template1 TEMP}${template0 '4p' curr_cond}°F (${template0 '5p' curr_cond}°F)
${template1 WIND}${template0 '6p' curr_cond} ${template0 '14p' curr_cond}
${template1 HUMIDITY}${template0 '7p' curr_cond}
${template1 PRESSURE}${template0 '8p' curr_cond}
${template1 CLOUD\ COVER}${template0 '9p' curr_cond}
${template1 UV\ INDEX 10}${template0 '10p' curr_cond}
${template1 DEW\ POINT}${template0 '11p' curr_cond}
${template1 PRECIPITATION}${template0 '12p' curr_cond}
${template1 VISIBILITY}${template0 '13p' curr_cond}
${template1 SUNRISE}${template0 '15p' curr_cond}
${template1 SUNSET}${template0 '16p' curr_cond}
${hr 1}
${goto 85}${color1}${font Venus Rising:style=bold:size=10}TODAY${goto 265}TONIGHT$color$font
${template2 today 40,250 120x72}${template2 tonight 225,250 120x72}
${alignc 95}${voffset 60}${template0 '1p' messages_curr}${goto 270}${template0 '3p' messages_curr}
${template0 '2p' messages_curr}${goto 265}${template0 '4p' messages_curr}
${template3 TEMP '28p' tod_ton '32p'}
${template3 FEELS\ LIKE '29p' tod_ton '33p'}
${hr 1}
# row 1
${voffset 5}
${template4 '6p' tod_ton '11p' tod_ton '16p'}
${template2 7 15,435 90x54} ${template2 12 175,435 90x54} ${template2 17 310,435 90x54}
${template5 H '9p' tod_ton '14p' tod_ton '19p'}
${template5 L '10p' tod_ton '15p' tod_ton '20p'}
${voffset 15}
${template6 '1p' '3p' '5p'}
${template6 '2p' '4p' '6p'}
# row 2
${voffset 5}
${template4 '21p' tod_ton '1p' last_days '6p'}
${template2 22 15,565 90x54}${template2 N2 175,565 90x54}${template2 N7 310,565 90x54}
${template5 H '24p' tod_ton '4p' last_days '9p'}
${template5 L '25p' tod_ton '5p' last_days '10p'}
${voffset 15}
${template6 '7p' '9p' '11p'}
${template6 '8p' '10p' '12p'}
# row 3
${voffset 5}
${template4 '11p' last_days '16p' last_days '21p'}
${template2 N12 15,700 90x54}${template2 N17 175,700 90x54}${template2 N22 310,700 90x54}
${template5 H '14p' last_days '19p' last_days '24p'}
${template5 L '15p' last_days '20p' last_days '25p'}
${voffset 15}
${template6 '13p' '15p' '17p'}

While I'm hardly an expert with conky, or weather data scraping (I'd likely not be using someone else's script, if I were), I'd say the template way works quite a bit better for hand editing; after converting, it became trivial for me to quickly change the positioning of elements.

Reference:
Teo's Conky Weather Scripts
Conky - A light-weight system monitor

2012-04-20

Notes to self: ICH7 - i975x - BIOS

My notes on BIOS settings for the old validation board.  Since the non-archived encrypted container I use for this sort of thing is on the system itself, it's somewhat annoying to reference when something's borked.   
Some of these are right buggers to find, and my guesses are more than likely wrong -- but I'll think of them wrongly consistently, going forward, rather than this somewhat 3 year cycle of looking them up again when I need to revisit them and have forgotten what Intel's delightful abbreviations mean.
CPU Configuration:
Max CPUID Value Limit
Determines the values that the OS can write to EAX to obtain processor information.  When enabled, the processor will limit the max CPUID input value to 0x03 when queried. Disabled, processor returns actual CPUID input value. (1)
GV3 Functionality
Likely EIST (Enhanced Intel SpeedStep, from Geyserville codename?) - Would be V3.2 for my Kentsfield(?) - P-State power management (in effect while processor is active) - Dynamically adjust the clock speed and automatically switch between maximum performance mode and battery-optimized mode. (1)
C1 Enhanced
Enhanced Processor Halt State. Reduces power consumption in idle mode (by turning off some unused parts of the processor, reducing the supply voltage and frequency) - Disable to increase overall determinism of real-time applications. (Intel: "Allows the system to change voltage level (lower) of processor when no work is being done.).
No-Execution Page Protection
Later XD Technology (?) aka Execute Disable Bit (xdbit) - Sets bounds to ban execution in areas of memory where code shouldn't be running. (1)
VT
Intel Virtualization Technology
PECI
Platform Environment Control Interface - Thermal Management - Allows CPU to report temperature.
Hardware Prefetcher
Tries to predict which instructions and data are required in the near future and prefetches data/code into L2 cache to reduce latency with memory reads. (1)
Application Processor(s)
One CPU is the Bootstrap Processor (BSP) and others are Application Processors (AP). This setting controls whether to enable the additional cores. (1) (2)
FSB Bus Park
Intel Dynamic Bus Parking(?) - Allows the chipset to shut down during inactivity to save energy. (1) (2)
IDE Config:
IDE Detect Time Out (Sec)
Amount of time to pause.
Hard Disk Write Protect
Prohibits recording on hard drives.
ATA(PI) 80Pin Cable Detection
Order of type checking for IDE cable (40pin, 80 conductor). Host & Device = checked from mobo and drive. Host = only from mobo, device = only from hdd.
SATA Link Power Management
Enable aggressive management and slumber and partial (AIPE&ASP) on all ports - Puts physical layer of link into low-power state. Potentially buggy with some linux utilities(?) (1) (2)
Stagger Spinup Support
Enabled: The BIOS will clear the staggered speed up supported bit in AHCI generic 'has capabilities' register.
Set HPCP Bit
Hot Plug Capable Port - When set to '1', indicates port's signal and power connectors are externally accessible. (1)
ARMD Emulation Type
ATAPI Removable Media Device - Devices which use removable media (LS-120, MO, Zip) (1)
SuperIO Config:
Assume PS2 Mouse Presence
Likely equivalent to PS/2 Mouse Function Control (?) - If enabled, reserves IRQ12 for PS/2 mouse.
USB Configuration:
USB Mass Storage Reset Delay
BIOS will wait [x] seconds for device to initialize.
Boot Settings Config:
AddOn ROM Display Mode
If enabled, logo screen will be followed by AddOn ROM initial screen (showing add-on card BIOS message). Keep Current - 3rd party only displayed if manufacturer set to do so; Force BIOS forces display. (1)
Interrupt 19 Capture
Allows host adapters to capture int 19 during boot process.
Chipset - Northbridge:
Device 2 (IGD) Enable
Device 2 (IGD) Enable - Auto: IGD enabled/disable based on GFX card detection and primary video device setup option. Disable: IGD is disabled regardless of card detection.
GMCH (Graphics Memory Controller Hub) Operating Mode
Paging modes(?) (1)
TOLUD (Top of Low Usable DRAM)
Contains address one byte above maximum DRAM memory below 4gb that is usable by the OS.  TOLUD is the lowest address above both Graphics Stolen memory and TSEG. BIOS determines the base of Graphics Stolen Memory by subtracting the GSM size from TOLUD and further decrements by TSEG size to determine base of TSEG. All the bits in this register are locked in Intel VT-d mode. Choices: 3.25gb / 2.75gb / 2.50gb/ 2.25gb / 2gb / 1.75gb. Note: grub doesn't like anything over 2.75gb.
Memory Reclaim
Allows reclaiming physical memory overlapped by the MMIO logical address space. MCH remaps physical memory from the TOLM boundary up the 4gb boundary to an equivalent sized logical address range located just above the top of physical memory. (1)
DRAM Throttling Threshold
Reduces heat; if threshold temp is exceeded, uses additional cycles; can adversely affect computer performance -- disable.
Chipset - Integrated Graphics Options:
IGD BARs Programming - ?
Device 2, Function 1
? (Related to IGD parellization?)
Chipset - South Bridge Configuration:
CPU BIST Enable
Built-In Self Test - "To enter BIST, software sets CPU_BIST_EN bit and then does a full processor reset using the CF9 register."
SSC Enable
Spread Spectrum Clocking (?)
SLP_S4# Assertion Width
Minimum assertion width of the SLP_S4# signal to guarantee that the DRAMs have been safely power-cycled or... S4 Sleep Control: SLP_S4# is for power plane control; this signal shuts power to all non-critical systems when in the S4 (suspend to disk) or S5 (Soft Off) state. Pin must be used to control the DRAM power to use the ICH7's DRAM power-cycling feature.
80h,84-6h,88h,8C-Eh Routing
Set to LPC -- functionality for the Port 0x80 Codes(?)
LPC 4Eh-4Fh Decode
CNF2_LPC_EN - R/W - Microcontroller Enable #2
LPC 2Eh-2Fh Decode
CNF1_LPC_EN - R/W - Super I/O Enable
SPI Prefetch During Shadowing
SPI Read Configuration(?) - 00b - No prefetching, but caching enabled, 01b - No prefetching and no caching, 10b - prefetching and caching enabled, 11b - reserved. (BIOS_CNTL - BIOS Control Register)
ASF (Alert Standard Format) Management Controller Support
Allows controller to collect and send information from system components to a remote server and accept commands back from the remote console and execute on local system.
Energy Lake Support
Disables or enables Energy Lake power management technology. Introduces two main end user features: "CE" like device power behavior and maintaining system state and data integrity during power loss events. (Now known as "Intel Quick Resume Technology"?)
Intel® AMT BIOS Extension
Active Management Technology - Remote pwnage. (1) (2)
Force IDE-R (Integrated Device Electronics - Redirect) on Tekoa
Forces redirect to the network card(?)
HPET (High Precision Event Timer)
Replaces 8254 PIT and RTC.
PCIe Config:
VC1/TC Map
VC1 will be mapped to TC specified - Field indicates TCs (traffic classes) that are mapped to the VC (virtual channel) resource. Bit locations within this field correspond to TC values. For example, when bit 7 is set in this field, TC7 is mapped to this VC resource. (i975x reference of field this BIOS setting manipulates.) (1)
Remove non-POR (Power on reset(?)) TCs from VC0
Default is enabled - Removes traffic classes that don't come back after reset from virtual channel 0(??)
Lock PCIE Credits Register
Writes once to the PCIE credit register to prevent them from being modified. (src: C2D Q45 Express Chipset Development Kit)
Northbridge PCIe:
PEG Port
PCI Express Graphics port 16x - auto/disable/enable
PEG Scrambler Bypass
GFX card: BIOS forces the scrambles to be bypassed without a link disable. Enable: BIOS forces the scrambles to be bypassed by disabling and enabling the link.
PEG Retry Buffer Depth
If forced, Retry Buffer is forced to less than optimal value. (Reduces retry buffer to a lower than typically required value)
PEG Force x1
Related to Force Link Width(?) - Auto / X1 / Reserve / X4 / X8
PEG Isoch (isochronous) Flush Page
Unknown, but chipset spec has: "IFPBAR - Any write to this window will trigger a flush of the MCH's GLobal Write Buffer to let software provide coherency between writes from an isochronous agent and writes from the processor (4-kb window)" which might be related.
PEG Active State PM (power management)
Controls level of active state power management supported on the given link: 00 = Disabled, 01 = L0s Entry Supported, 10 = Reserved, 11 = L0s and LI Entry Supported. (1)
SERR# on Fatal (and Non-Fatal) Error
System Error is generated if (error type) is detected on the PEG port and will result in NMI (non-maskable interrupt) or SMI depending upon state of NMI to SMI setup option.
Link Stability Alogrithm
Used for verifying PCIe Link is up and running for x16 slot for x16 graphics cards.
Southbridge PCIe:
PCI Express PME (Power Management Event) SCI (System Control Interrupt) Enable
Routes PMEs through SCIs(?) (1)
VC1 for HD Audio - ???
Northway Training W/A - ???
Northway S4 W/A - ???
ACPI Advanced:
BIOS->AML ACPI Table
Allows update of AML. When set, the ACPI APIC table pointer is included in the RSDT pointer list.
ACPI PCI0 _S1D (S1 Device State)
The highest D-state supported by the device in the S1/S2/S3/S4 states, respectively. It should be noted that higher D-state values correspond to lower power consumption states, with D0 being the highest powered state and D5 being the lowest power state (off).
ACPI Video Extensions
Mostly for laptops(?) (1 (pg 877))
Headless Mode
Setting updates FACP (Fixed ACPI Description Table) - "System Type Attribute - If set indicates the system cannot detect the monitor or keyboard/mouse devices."
SV (Silicon Validation) Options:
Program ICH Native IDE BARs
Unknown - (1)
PCI Exp Bridge ISA Enable
Needs to be disabled to support DOSODI and NDIS2. Detail: "This problem appears to be related to the specific I/O address assigned to the Tekoa LAN function by the BIOS. As configured by the BIOS, I/O reads to adapter registers always fail, returning 0xFFFFFFFF. This causes the driver to be unable to read the adapter EEPROM or configure the adapter for operation and thus it fails to load."
Global SMI
CPU SMBASE relocation will not occur if set to disable.

2012-04-16

Using Python to Extract Data From a Smereka Created SQLite Database

I've been migrating my snippets of stuff to thinkery.me, but for more private/personal data, I've been using Smereka: It's open source, with a (for me) intuitive GUI, fast --- and uses sqlite for its backend database.

Unfortunately, its export absolutely sucks, and it looks like Smereka's author is trying to direct folks to his commercial version, and probably won't be adding many features/fixes to the free version.

So, having become set on doing something with all my random files (dot configs, equipment tracking, cake recipes (all the more important when navigating portals), contacts, TMF Yearmix track listings, pppd dial scripts, etc) beyond dumping them all in a backup directory in an encrypted container (and backing up to dropbox and Ubuntu One), I wanted to make sure I could liberate all the data, if I needed to, from whichever program I used. (Siderant: I looked at Evernote, Treeline, etc... why are there so many programs that do such a simple thing so wrongly?)  Smereka was nearly perfect, except for putting all items into blobs, which my limited abilities don't allow me to retrieve directly with just an interactive sqlite3 session.

Enter Python. (And ni saying knights, spam, laden swallows and show tunes sung from crosses.)

I've been meaning to 'take a look' at python for the better part of a decade, but never had a big enough incentive to push me into doing so.  In one of those weird instances of perfect timing, a good friend linked me to Alex Munroe's rant about PHP, unaware I was bracing myself for some soul destroying C++ spelunking. I read through it, nodding my head and agreeing with most of the points, and was ready to drink the koolaid by the time I got to the end.

What follows is my first ever python program; it's the result of (literally!) a couple days of learning the language.  I'm definitely sold on Python; php was always, "Yeah, it's flawed, but what else is there, for quick scripts and web backends?" --- now, I know the answer is "Python."

This code is likely incredibly naive.  There are almost certainly better/more elegant ways of accomplishing what I'm doing.  However, for my purposes, all I needed was a way of being certain the stuff I was putting into Smereka could be liberated in the future, should I ever need to.

Thus, I'm simply extracting all the data (which, in my case, is text. Smereka also allows images, rich text, etc) and dumping it into a single file.  It would be trivial to dump each record to its own file, or use the parent folder as a containing file, or whatever.  Since the searching inside of Smereka works just fine, the only point for me was to be able to have something greppable, and zipped, 'just in case'.  So, I'm not even bothering to grab the tags or descriptions, again which would be trivial. (Technical: I ignore the DataType: File, as well as the tags in StringsTbl.) I'm skipping error checking, or even politeness, like adding 'global var' to defs. (Obviously, I'm only reading...).  I don't worry about memory management, or if it'd be better to create a table with folder names, rather than walk backwards each time... and, finally, I half-assed dealing with the unicode 'issues'. (Windows directory listings from over a decade ago have the obligatory broken bytes to make python kick up a UnicodeDecodeError.)

Note: This code was tested with Python 2.7.1+. YMMV.

#!/usr/bin/python
# -*- coding: utf-8 -*-

import sqlite3 as sql
import sys
import zlib
import codecs

DATABASE = "db.sqlite"

DELIM = "\\"
folders = {}
titems = {}

FQUERY = "SELECT ItemId, ParentId, KeyText FROM Items WHERE \
    DataType = 'Folder|24A1B35E-B391-184F-12C3-FBB2A40B9B37'"
DQUERY = "SELECT ItemId, ParentId, KeyText FROM Items WHERE \
    DataType = 'Plain Text|49F238EA-38B0-1744-CD64-1947630FA367'"

def traverse_tree(node, p):
    """traverse_tree -
    : Walks backward with the parentID until no
    : more parents are found.
    """
    while node['parent'] != 0:
        p.append(node['keytext']+DELIM)
        node = folders[node['parent']]
    p.append(node['keytext']+DELIM)

def debug_folderwalk():
    """Debug function to verify tree iteration."""
    for v in folders.values():
        if v['parent'] == 0:
            print "DEBUG: Path: %s" % v['keytext'] + DELIM
        else:
            pathexpand = []
            traverse_tree(v, pathexpand)
            print "DEBUG: Path: "+ "".join(reversed(pathexpand))

def debug_rawdata(d):
    """Debug function to show raw data from database"""
    for key, value in d.iteritems():
        print "DEBUG: %s-%s" % (key, value)
    
def path_parse(id):
    """path_parse - 
    : Returns the folder tree in reverse
    : order of depth for the given item.
    """
    if titems[id]['parent'] == 0: 
        return DELIM
    else:
        pathexpand = []
        traverse_tree(folders[titems[id]['parent']], pathexpand)
        return "".join(reversed(pathexpand))

def decorate(p, i, v):
    """decorate - 
    : Decorates output for each text file
    : to make it clear which tree node this is,
    : as well as establing the start and end of item.
    """
    t = "/* "+"*"*63+"\n" # top
    r = " "*3+"* " # right
    b = " "*3+"*"*63+" */\n" # bottom
    s = "\n/* "+"-"*63+" */\n" # eof marker
    return t+r+"Path: %s\n" % (p)+r+"Item: %s\n" % (i)+b+"%s" % (v)+s

def populate_dict(c, q, d):
    """populate_dict -
    : Takes a cursor, query and dictionary, and populates 
    : the dictionary with the results of the query.
    """
    c.execute(q)
    while True:
        row = c.fetchone()
        if row == None: break
        d[row[0]] = dict({"parent":row[1], "keytext":row[2]})

byteswritten = 0

f = codecs.open("data", "wb", 'utf-8')

with sql.connect(DATABASE) as con:
        cur = con.cursor()
        populate_dict(cur, FQUERY, folders)
        populate_dict(cur, DQUERY, titems)
        for item in titems:
            cur.execute("SELECT TheData FROM Blobs WHERE ItemId = ?", (item,))
            cblob = cur.fetchone()
            zobj = zlib.decompressobj()
            dblob = unicode(zobj.decompress(cblob[0])[6:], 'utf-8')
            buf = decorate(path_parse(item), titems[item]['keytext'], dblob)
            f.write(buf)
            byteswritten += len(buf)

f.close()

print "Wrote {:,d} bytes from {:,d} item records.\n"\
    .format(byteswritten, len(titems))

#debug_rawdata(folders)
#debug_rawdata(titems)
#debug_folderwalk()

Example output (ignore the cobwebs...):

/* ***************************************************************
   * Path: Equipment\APC Technical Information\
   * Item: APC Backups BX1500G Original
   *************************************************************** */
Model: Back-UPS BX1500G
Serial number: ------------
Firmware revision: 866.L4 .D
USB firmware revision: L4 
Result of last manual self-test: Not Recorded
Last manual self-test date: Not Recorded
Last battery replacement: 9/12/2010

/* --------------------------------------------------------------- */
/* ***************************************************************
   * Path: Equipment\APC Technical Information\
   * Item: APC Backups BX1500G Replacement
   *************************************************************** */
Model: Back-UPS BX1500G
Serial number: -----------
Firmware revision: 866.L5 .D
USB firmware revision: L5 
Result of last manual self-test: Not Recorded
Last manual self-test date: Not Recorded
Last battery replacement: 12/4/2010

/* --------------------------------------------------------------- */
/* ***************************************************************
   * Path: Scripts\
   * Item: uudecode
   *************************************************************** */
grep -v ":\ " Alt.binaries.multimedia.repost | grep -v "From\ " | grep -v '^$' | uudecode -c

/* --------------------------------------------------------------- */
/* ***************************************************************
   * Path: deprecated\linux_varg\rc\
   * Item: linux_winmx_forward
   *************************************************************** */
ipmasqadm portfw -l
prot localaddr            rediraddr               lport    rport  pcnt  pref  
TCP  xxxxxxxxxx.xxxxxxxx.dyndns.org xxxxxxxx.xxxxxxxx.dyndns.org     6699     6699    10    10
UDP  xxxxxxxxxx.xxxxxxxx.dyndns.org xxxxxxxx.xxxxxxxx.dyndns.org     6257     6257    10    10
ipmasqadm portfw -a -P tcp -L 10.0.0.1 4000 -R 10.0.0.3 4000
iptables -t nat -A PREROUTING -p tcp -i ppp0 --dport 6699 -j DNAT --to 10.0.0.3:6699
iptables -t nat -A PREROUTING -p udp -i ppp0 --dport 6257 -j DNAT --to 10.0.0.3:6257

/* --------------------------------------------------------------- */

General References:

Smereka - "Extensible Personal Freeform Database and Personal Information Manager"
SQLite Database Browser - Very useful app to play with sqlite databases graphically.

Python References:

PHP - A Fractal of Bad Design - Wonderful php hate manifesto.
Why Python? - Advocacy post by Eric S. Raymond, from all the way back in 2000!
The Python Tutorial - Start Here.
Learn Python The Hard Way - More Beginner Python
SQLite Python Tutorial - Useful for the database bits.
All About Python and Unicode - A staggeringly helpful document.