2012-04-20

Notes to self: ICH7 - i975x - BIOS

My notes on BIOS settings for the old validation board.  Since the non-archived encrypted container I use for this sort of thing is on the system itself, it's somewhat annoying to reference when something's borked.   
Some of these are right buggers to find, and my guesses are more than likely wrong -- but I'll think of them wrongly consistently, going forward, rather than this somewhat 3 year cycle of looking them up again when I need to revisit them and have forgotten what Intel's delightful abbreviations mean.
CPU Configuration:
Max CPUID Value Limit
Determines the values that the OS can write to EAX to obtain processor information.  When enabled, the processor will limit the max CPUID input value to 0x03 when queried. Disabled, processor returns actual CPUID input value. (1)
GV3 Functionality
Likely EIST (Enhanced Intel SpeedStep, from Geyserville codename?) - Would be V3.2 for my Kentsfield(?) - P-State power management (in effect while processor is active) - Dynamically adjust the clock speed and automatically switch between maximum performance mode and battery-optimized mode. (1)
C1 Enhanced
Enhanced Processor Halt State. Reduces power consumption in idle mode (by turning off some unused parts of the processor, reducing the supply voltage and frequency) - Disable to increase overall determinism of real-time applications. (Intel: "Allows the system to change voltage level (lower) of processor when no work is being done.).
No-Execution Page Protection
Later XD Technology (?) aka Execute Disable Bit (xdbit) - Sets bounds to ban execution in areas of memory where code shouldn't be running. (1)
VT
Intel Virtualization Technology
PECI
Platform Environment Control Interface - Thermal Management - Allows CPU to report temperature.
Hardware Prefetcher
Tries to predict which instructions and data are required in the near future and prefetches data/code into L2 cache to reduce latency with memory reads. (1)
Application Processor(s)
One CPU is the Bootstrap Processor (BSP) and others are Application Processors (AP). This setting controls whether to enable the additional cores. (1) (2)
FSB Bus Park
Intel Dynamic Bus Parking(?) - Allows the chipset to shut down during inactivity to save energy. (1) (2)
IDE Config:
IDE Detect Time Out (Sec)
Amount of time to pause.
Hard Disk Write Protect
Prohibits recording on hard drives.
ATA(PI) 80Pin Cable Detection
Order of type checking for IDE cable (40pin, 80 conductor). Host & Device = checked from mobo and drive. Host = only from mobo, device = only from hdd.
SATA Link Power Management
Enable aggressive management and slumber and partial (AIPE&ASP) on all ports - Puts physical layer of link into low-power state. Potentially buggy with some linux utilities(?) (1) (2)
Stagger Spinup Support
Enabled: The BIOS will clear the staggered speed up supported bit in AHCI generic 'has capabilities' register.
Set HPCP Bit
Hot Plug Capable Port - When set to '1', indicates port's signal and power connectors are externally accessible. (1)
ARMD Emulation Type
ATAPI Removable Media Device - Devices which use removable media (LS-120, MO, Zip) (1)
SuperIO Config:
Assume PS2 Mouse Presence
Likely equivalent to PS/2 Mouse Function Control (?) - If enabled, reserves IRQ12 for PS/2 mouse.
USB Configuration:
USB Mass Storage Reset Delay
BIOS will wait [x] seconds for device to initialize.
Boot Settings Config:
AddOn ROM Display Mode
If enabled, logo screen will be followed by AddOn ROM initial screen (showing add-on card BIOS message). Keep Current - 3rd party only displayed if manufacturer set to do so; Force BIOS forces display. (1)
Interrupt 19 Capture
Allows host adapters to capture int 19 during boot process.
Chipset - Northbridge:
Device 2 (IGD) Enable
Device 2 (IGD) Enable - Auto: IGD enabled/disable based on GFX card detection and primary video device setup option. Disable: IGD is disabled regardless of card detection.
GMCH (Graphics Memory Controller Hub) Operating Mode
Paging modes(?) (1)
TOLUD (Top of Low Usable DRAM)
Contains address one byte above maximum DRAM memory below 4gb that is usable by the OS.  TOLUD is the lowest address above both Graphics Stolen memory and TSEG. BIOS determines the base of Graphics Stolen Memory by subtracting the GSM size from TOLUD and further decrements by TSEG size to determine base of TSEG. All the bits in this register are locked in Intel VT-d mode. Choices: 3.25gb / 2.75gb / 2.50gb/ 2.25gb / 2gb / 1.75gb. Note: grub doesn't like anything over 2.75gb.
Memory Reclaim
Allows reclaiming physical memory overlapped by the MMIO logical address space. MCH remaps physical memory from the TOLM boundary up the 4gb boundary to an equivalent sized logical address range located just above the top of physical memory. (1)
DRAM Throttling Threshold
Reduces heat; if threshold temp is exceeded, uses additional cycles; can adversely affect computer performance -- disable.
Chipset - Integrated Graphics Options:
IGD BARs Programming - ?
Device 2, Function 1
? (Related to IGD parellization?)
Chipset - South Bridge Configuration:
CPU BIST Enable
Built-In Self Test - "To enter BIST, software sets CPU_BIST_EN bit and then does a full processor reset using the CF9 register."
SSC Enable
Spread Spectrum Clocking (?)
SLP_S4# Assertion Width
Minimum assertion width of the SLP_S4# signal to guarantee that the DRAMs have been safely power-cycled or... S4 Sleep Control: SLP_S4# is for power plane control; this signal shuts power to all non-critical systems when in the S4 (suspend to disk) or S5 (Soft Off) state. Pin must be used to control the DRAM power to use the ICH7's DRAM power-cycling feature.
80h,84-6h,88h,8C-Eh Routing
Set to LPC -- functionality for the Port 0x80 Codes(?)
LPC 4Eh-4Fh Decode
CNF2_LPC_EN - R/W - Microcontroller Enable #2
LPC 2Eh-2Fh Decode
CNF1_LPC_EN - R/W - Super I/O Enable
SPI Prefetch During Shadowing
SPI Read Configuration(?) - 00b - No prefetching, but caching enabled, 01b - No prefetching and no caching, 10b - prefetching and caching enabled, 11b - reserved. (BIOS_CNTL - BIOS Control Register)
ASF (Alert Standard Format) Management Controller Support
Allows controller to collect and send information from system components to a remote server and accept commands back from the remote console and execute on local system.
Energy Lake Support
Disables or enables Energy Lake power management technology. Introduces two main end user features: "CE" like device power behavior and maintaining system state and data integrity during power loss events. (Now known as "Intel Quick Resume Technology"?)
Intel® AMT BIOS Extension
Active Management Technology - Remote pwnage. (1) (2)
Force IDE-R (Integrated Device Electronics - Redirect) on Tekoa
Forces redirect to the network card(?)
HPET (High Precision Event Timer)
Replaces 8254 PIT and RTC.
PCIe Config:
VC1/TC Map
VC1 will be mapped to TC specified - Field indicates TCs (traffic classes) that are mapped to the VC (virtual channel) resource. Bit locations within this field correspond to TC values. For example, when bit 7 is set in this field, TC7 is mapped to this VC resource. (i975x reference of field this BIOS setting manipulates.) (1)
Remove non-POR (Power on reset(?)) TCs from VC0
Default is enabled - Removes traffic classes that don't come back after reset from virtual channel 0(??)
Lock PCIE Credits Register
Writes once to the PCIE credit register to prevent them from being modified. (src: C2D Q45 Express Chipset Development Kit)
Northbridge PCIe:
PEG Port
PCI Express Graphics port 16x - auto/disable/enable
PEG Scrambler Bypass
GFX card: BIOS forces the scrambles to be bypassed without a link disable. Enable: BIOS forces the scrambles to be bypassed by disabling and enabling the link.
PEG Retry Buffer Depth
If forced, Retry Buffer is forced to less than optimal value. (Reduces retry buffer to a lower than typically required value)
PEG Force x1
Related to Force Link Width(?) - Auto / X1 / Reserve / X4 / X8
PEG Isoch (isochronous) Flush Page
Unknown, but chipset spec has: "IFPBAR - Any write to this window will trigger a flush of the MCH's GLobal Write Buffer to let software provide coherency between writes from an isochronous agent and writes from the processor (4-kb window)" which might be related.
PEG Active State PM (power management)
Controls level of active state power management supported on the given link: 00 = Disabled, 01 = L0s Entry Supported, 10 = Reserved, 11 = L0s and LI Entry Supported. (1)
SERR# on Fatal (and Non-Fatal) Error
System Error is generated if (error type) is detected on the PEG port and will result in NMI (non-maskable interrupt) or SMI depending upon state of NMI to SMI setup option.
Link Stability Alogrithm
Used for verifying PCIe Link is up and running for x16 slot for x16 graphics cards.
Southbridge PCIe:
PCI Express PME (Power Management Event) SCI (System Control Interrupt) Enable
Routes PMEs through SCIs(?) (1)
VC1 for HD Audio - ???
Northway Training W/A - ???
Northway S4 W/A - ???
ACPI Advanced:
BIOS->AML ACPI Table
Allows update of AML. When set, the ACPI APIC table pointer is included in the RSDT pointer list.
ACPI PCI0 _S1D (S1 Device State)
The highest D-state supported by the device in the S1/S2/S3/S4 states, respectively. It should be noted that higher D-state values correspond to lower power consumption states, with D0 being the highest powered state and D5 being the lowest power state (off).
ACPI Video Extensions
Mostly for laptops(?) (1 (pg 877))
Headless Mode
Setting updates FACP (Fixed ACPI Description Table) - "System Type Attribute - If set indicates the system cannot detect the monitor or keyboard/mouse devices."
SV (Silicon Validation) Options:
Program ICH Native IDE BARs
Unknown - (1)
PCI Exp Bridge ISA Enable
Needs to be disabled to support DOSODI and NDIS2. Detail: "This problem appears to be related to the specific I/O address assigned to the Tekoa LAN function by the BIOS. As configured by the BIOS, I/O reads to adapter registers always fail, returning 0xFFFFFFFF. This causes the driver to be unable to read the adapter EEPROM or configure the adapter for operation and thus it fails to load."
Global SMI
CPU SMBASE relocation will not occur if set to disable.

2012-04-16

Using Python to Extract Data From a Smereka Created SQLite Database

I've been migrating my snippets of stuff to thinkery.me, but for more private/personal data, I've been using Smereka: It's open source, with a (for me) intuitive GUI, fast --- and uses sqlite for its backend database.

Unfortunately, its export absolutely sucks, and it looks like Smereka's author is trying to direct folks to his commercial version, and probably won't be adding many features/fixes to the free version.

So, having become set on doing something with all my random files (dot configs, equipment tracking, cake recipes (all the more important when navigating portals), contacts, TMF Yearmix track listings, pppd dial scripts, etc) beyond dumping them all in a backup directory in an encrypted container (and backing up to dropbox and Ubuntu One), I wanted to make sure I could liberate all the data, if I needed to, from whichever program I used. (Siderant: I looked at Evernote, Treeline, etc... why are there so many programs that do such a simple thing so wrongly?)  Smereka was nearly perfect, except for putting all items into blobs, which my limited abilities don't allow me to retrieve directly with just an interactive sqlite3 session.

Enter Python. (And ni saying knights, spam, laden swallows and show tunes sung from crosses.)

I've been meaning to 'take a look' at python for the better part of a decade, but never had a big enough incentive to push me into doing so.  In one of those weird instances of perfect timing, a good friend linked me to Alex Munroe's rant about PHP, unaware I was bracing myself for some soul destroying C++ spelunking. I read through it, nodding my head and agreeing with most of the points, and was ready to drink the koolaid by the time I got to the end.

What follows is my first ever python program; it's the result of (literally!) a couple days of learning the language.  I'm definitely sold on Python; php was always, "Yeah, it's flawed, but what else is there, for quick scripts and web backends?" --- now, I know the answer is "Python."

This code is likely incredibly naive.  There are almost certainly better/more elegant ways of accomplishing what I'm doing.  However, for my purposes, all I needed was a way of being certain the stuff I was putting into Smereka could be liberated in the future, should I ever need to.

Thus, I'm simply extracting all the data (which, in my case, is text. Smereka also allows images, rich text, etc) and dumping it into a single file.  It would be trivial to dump each record to its own file, or use the parent folder as a containing file, or whatever.  Since the searching inside of Smereka works just fine, the only point for me was to be able to have something greppable, and zipped, 'just in case'.  So, I'm not even bothering to grab the tags or descriptions, again which would be trivial. (Technical: I ignore the DataType: File, as well as the tags in StringsTbl.) I'm skipping error checking, or even politeness, like adding 'global var' to defs. (Obviously, I'm only reading...).  I don't worry about memory management, or if it'd be better to create a table with folder names, rather than walk backwards each time... and, finally, I half-assed dealing with the unicode 'issues'. (Windows directory listings from over a decade ago have the obligatory broken bytes to make python kick up a UnicodeDecodeError.)

Note: This code was tested with Python 2.7.1+. YMMV.

#!/usr/bin/python
# -*- coding: utf-8 -*-

import sqlite3 as sql
import sys
import zlib
import codecs

DATABASE = "db.sqlite"

DELIM = "\\"
folders = {}
titems = {}

FQUERY = "SELECT ItemId, ParentId, KeyText FROM Items WHERE \
    DataType = 'Folder|24A1B35E-B391-184F-12C3-FBB2A40B9B37'"
DQUERY = "SELECT ItemId, ParentId, KeyText FROM Items WHERE \
    DataType = 'Plain Text|49F238EA-38B0-1744-CD64-1947630FA367'"

def traverse_tree(node, p):
    """traverse_tree -
    : Walks backward with the parentID until no
    : more parents are found.
    """
    while node['parent'] != 0:
        p.append(node['keytext']+DELIM)
        node = folders[node['parent']]
    p.append(node['keytext']+DELIM)

def debug_folderwalk():
    """Debug function to verify tree iteration."""
    for v in folders.values():
        if v['parent'] == 0:
            print "DEBUG: Path: %s" % v['keytext'] + DELIM
        else:
            pathexpand = []
            traverse_tree(v, pathexpand)
            print "DEBUG: Path: "+ "".join(reversed(pathexpand))

def debug_rawdata(d):
    """Debug function to show raw data from database"""
    for key, value in d.iteritems():
        print "DEBUG: %s-%s" % (key, value)
    
def path_parse(id):
    """path_parse - 
    : Returns the folder tree in reverse
    : order of depth for the given item.
    """
    if titems[id]['parent'] == 0: 
        return DELIM
    else:
        pathexpand = []
        traverse_tree(folders[titems[id]['parent']], pathexpand)
        return "".join(reversed(pathexpand))

def decorate(p, i, v):
    """decorate - 
    : Decorates output for each text file
    : to make it clear which tree node this is,
    : as well as establing the start and end of item.
    """
    t = "/* "+"*"*63+"\n" # top
    r = " "*3+"* " # right
    b = " "*3+"*"*63+" */\n" # bottom
    s = "\n/* "+"-"*63+" */\n" # eof marker
    return t+r+"Path: %s\n" % (p)+r+"Item: %s\n" % (i)+b+"%s" % (v)+s

def populate_dict(c, q, d):
    """populate_dict -
    : Takes a cursor, query and dictionary, and populates 
    : the dictionary with the results of the query.
    """
    c.execute(q)
    while True:
        row = c.fetchone()
        if row == None: break
        d[row[0]] = dict({"parent":row[1], "keytext":row[2]})

byteswritten = 0

f = codecs.open("data", "wb", 'utf-8')

with sql.connect(DATABASE) as con:
        cur = con.cursor()
        populate_dict(cur, FQUERY, folders)
        populate_dict(cur, DQUERY, titems)
        for item in titems:
            cur.execute("SELECT TheData FROM Blobs WHERE ItemId = ?", (item,))
            cblob = cur.fetchone()
            zobj = zlib.decompressobj()
            dblob = unicode(zobj.decompress(cblob[0])[6:], 'utf-8')
            buf = decorate(path_parse(item), titems[item]['keytext'], dblob)
            f.write(buf)
            byteswritten += len(buf)

f.close()

print "Wrote {:,d} bytes from {:,d} item records.\n"\
    .format(byteswritten, len(titems))

#debug_rawdata(folders)
#debug_rawdata(titems)
#debug_folderwalk()

Example output (ignore the cobwebs...):

/* ***************************************************************
   * Path: Equipment\APC Technical Information\
   * Item: APC Backups BX1500G Original
   *************************************************************** */
Model: Back-UPS BX1500G
Serial number: ------------
Firmware revision: 866.L4 .D
USB firmware revision: L4 
Result of last manual self-test: Not Recorded
Last manual self-test date: Not Recorded
Last battery replacement: 9/12/2010

/* --------------------------------------------------------------- */
/* ***************************************************************
   * Path: Equipment\APC Technical Information\
   * Item: APC Backups BX1500G Replacement
   *************************************************************** */
Model: Back-UPS BX1500G
Serial number: -----------
Firmware revision: 866.L5 .D
USB firmware revision: L5 
Result of last manual self-test: Not Recorded
Last manual self-test date: Not Recorded
Last battery replacement: 12/4/2010

/* --------------------------------------------------------------- */
/* ***************************************************************
   * Path: Scripts\
   * Item: uudecode
   *************************************************************** */
grep -v ":\ " Alt.binaries.multimedia.repost | grep -v "From\ " | grep -v '^$' | uudecode -c

/* --------------------------------------------------------------- */
/* ***************************************************************
   * Path: deprecated\linux_varg\rc\
   * Item: linux_winmx_forward
   *************************************************************** */
ipmasqadm portfw -l
prot localaddr            rediraddr               lport    rport  pcnt  pref  
TCP  xxxxxxxxxx.xxxxxxxx.dyndns.org xxxxxxxx.xxxxxxxx.dyndns.org     6699     6699    10    10
UDP  xxxxxxxxxx.xxxxxxxx.dyndns.org xxxxxxxx.xxxxxxxx.dyndns.org     6257     6257    10    10
ipmasqadm portfw -a -P tcp -L 10.0.0.1 4000 -R 10.0.0.3 4000
iptables -t nat -A PREROUTING -p tcp -i ppp0 --dport 6699 -j DNAT --to 10.0.0.3:6699
iptables -t nat -A PREROUTING -p udp -i ppp0 --dport 6257 -j DNAT --to 10.0.0.3:6257

/* --------------------------------------------------------------- */

General References:

Smereka - "Extensible Personal Freeform Database and Personal Information Manager"
SQLite Database Browser - Very useful app to play with sqlite databases graphically.

Python References:

PHP - A Fractal of Bad Design - Wonderful php hate manifesto.
Why Python? - Advocacy post by Eric S. Raymond, from all the way back in 2000!
The Python Tutorial - Start Here.
Learn Python The Hard Way - More Beginner Python
SQLite Python Tutorial - Useful for the database bits.
All About Python and Unicode - A staggeringly helpful document.