Sunday, September 27, 2015

Flattening Python Code with "continue"

Flat code is better than nested.  It's more readable, easier to update, and easier to debug.  All of the linux kernel source code uses 8-space indents because, as I've seen it explained by Linus, if you can't use enough indents with 8-space, you have too many indents and need to refactor your code so it's more simple.

I do some data analysis type stuff at work fairly often.  Sometimes I'll have a half-dozen conditional statements to satisfy in a loop where I'm ticking over rows of data.  Using the "continue" statement is a really simple way to keep code flat and manageable.

Bad form would be like this:
for row in my_data:
    if row[val] == 'blah':
        if row[val2] == 'test':
            if row[val3]:
                yield row

Better would be like this:
for row in my_data:
    if row[val] != 'blah':
    if row[val2] != 'test':
    if row[val3]:
        yield row

Search Filters

I really wish that I could apply always-on filters to my google searches so that I never see results from the filtered pages.   For example, when I look for a recipe, I'm never going to click on a bettycrocker link.  It's just pollution hiding all the good stuff.

Monday, August 31, 2015

Raspberry Pi Projects

For years, I've been listening to people talk about the projects that they want to use the Raspberry Pi boards for.  I've never really had any good ideas... until now!

Automotive Data-Logger

This would include a dash cam and a reverse cam and would be a fairly involved project.  The RPi is a great candidate for this project because of it's low power requirements, hardware compatibility, and just because it runs Linux and is easy to program for!

What I envision is a fully contained unit that affixes to the ceiling in the cab of my truck up by the rear-view mirror.  It would:

  • House front and rear-facing cameras.  The RPi has a special connector for a specific camera that works well with it, and there's a cable splitter available to enable connecting more than one camera.  Would use this.
  • The Pi board
  • Sensor accessory board (has accelerometers and stuff)
  • Time/clock accessory board - this enables the RPi to keep track of time even when it's powered off, which is apparently an issue since it doesn't have a battery built in. 
  • Bluetooth and 802.11 b/g/n dongles.  It might be reasonable to even use an external monopole antenna mounted atop the truck for better reception. 
  • A notebook HDD
  • USB GPS dongle
  • A backup battery
I'd have to write some software for this one... I'm unsure if I could throw it together in bash or if it would be complicated enough that I'd want to do it in python.  Software Features would include:
  • Event Awareness:
    • Automatic detection of car starting to trigger logging. 
    • Detection of poweroff to end logging
    • Sudden stops trigger additional logging? 
  • Logging while vehicle is on:
    • Front and rear video
    • GPS data, as available, but expectedly including position and velocity. 
    • Accelerometer data
    • Log wifi access points observed, signal strength, etc (wardriving)
  • Network Services:
    • Seek out and connect to open wifi networks 
    • Auto-connect to home network and automatically sync logged data over to home server. 
    • Tether from phone for internet - look at usb devices connected and auto-start the tether software when phone connects... could even adb issue command to the phone to start the tether software!
    • Share available network via LAN to other devices in the truck...  For example, if I ever get around to building a Nexus tablet CD Deck replacement, it would provide a network connection to it when available.   

References - there are lots of these builds online:

SDR Radio Server

This is a pretty simple one.  Just install linux to the Pi, install the SDR drivers and software, and configure it for the network.  SDR servers allow clients to connect and control the frequency that the SDR radio listens on and, via the network, listen to the radio.

The SDR receivers a lot of people use are cheap, like $20, and work on all kinds of bands, from HF and shortwave, up through the FM stations, VHF, UHF, and beyond.    They are limited by their inability to filter signals received from the antenna, so powerful FM broadcasts can pollute the rest of the spectrum without filters in front of the receiver.  Of course, having a quality antenna is really important as well.

Ideally, I'd have a random wire antenna set up somewhere on my property, running into a window or something, and the RPi would serve the radio over the local network so i could listen from anywhere at home, w/o a tether.  It'd make it easy to listen to SW broadcasts.

Sunday, July 19, 2015

Property Invesetment - Finding ROI

So I've been looking at buying a duplex or something and am trying to understand what is actually a good investment.  I put together a spreadsheet, here, that allows values to be plugged in and it will show the annual increase in value and %return on total investment in the property.

Plug in values in yellow and look at cells in green on the right for return.  Red shows my cost each month toward the mortgage.

It includes mortgage insurance automatically for down payments of less than %20 of the total value, and then stops them once the M.I. cutoff has been reached, configurable in yellow.  I'm not sure if I'm dong this right, but it looks good to me :p

Friday, July 10, 2015

Outlook 2011 Mac OSX Problems

Just some hints for others incase them run into similar issues.  I went on vacation last month and when I returned, Outlook would crash constantly.  Local IT support recommended that I rebuild my profile and, second, try removing my Exchange account and rebuilding the profile again.

This didn't do anything, and somewhere along the way, all of my local folders, the ones that show up under "on my computer", disappeared.  I was still seeing crashes and I had time machine backups, so I charged forward anyway.  I've heard reports of problems resulting from the 14.5.0 update.

What fixed the crashing for me was to completely reinstall Office 2011 (including Outlook).  Google around for the Office Uninstall Tool.  Microsoft provides one.  Reinstall worked fine, although the Update tool had a hard time and I had to grab updates from the Microsoft site to get Office and the Update Tool up to a recent version.

I was hoping I'd be able to try and use an earlier Outlook release, in case one of the updates that my system grabbed when I first got back from vacation had caused my issues, but it complained that my Profile was only compatible with current outlook releases, so I went ahead and updated to the most recent 14.5.2.

My "On my computer" folders were still missing!!!  I restored my profile from a few days ago using a time machine backup, and still nothing.  I started digging through settings to see if I could find a way to load them manually, and something happened that fixed it; it was one of two things:

  • I tried to do an export of all of my contacts, calendar, mail, to a .olm file, just in case, but ended up canceling it because it was too slow (I was trying to write to an SD card where I dump general data).  
  • It's possible that I accidentally clicked the check box in the Preferences -> General -> Folder list -> [ ] Hide On My Computer folders.  By this time I noticed this, I had been through settings once... this may be all that I needed to make these folders show in the first place, but who knows.  If I check it now, they disappear!  
There wasn't much good info coming up in Google about my problem with the local mail, so hopefully this helps someone!

Saturday, October 18, 2014

Notes on OSX Virtalization, booting raw disks

I picked up a Mac to use for work, primarily, a couple weeks ago.  It's a late 2013 Macbook Pro Retina 13" w/ 8GB ram and 256GB SSD.  I wanted to be able to dual boot with Linux for home and dev type stuff, but being short on time needing to encrypt my primary hdd, I didn't get to figure out boot camp before finalizing my setup for work, so I'm not about to mess with my primary disk's partition/efi config and risk downtime at work.

Sooo, I'm trying to get set up to boot to boot Linux on an SD card.  This has been a bumpy road.  I purchased a Nifty SD card adapter and a Sandisk 80MBps 64G card to install to, and I'm doing the install via a VM while I work on other stuff.  This presents first problem - support for booting SD card by Parallels, Fusion, or VirtualBox.

I'm defaulting to Parallels since everyone else on mac uses it at work, but Fusion looks pretty comparable.  I've used VirtualBox a lot in the past, but I got tired of bugs and crashing at some point and switched to VMWare workstation at home.

That said, VirtualBox seems to be the most flexible and capable when it comes to booting SD cards on Mac.  It is the only virtualization solution of the three that doesn't have  "dumbed down" interface for people who expect things to just work without understanding them on mac.  It comes with command line tools for creating virtual disk files, etc.  This is what you need if you want to make a "bootcamp" disk image to use with Fusion.

Parallels and Fusion only support booting raw disks via "bootcamp".  You can't just point them to a block device and have them use it without tricking them.

  • To trick Fusion into booting to the SSD, set up a linux VM with a virtual disk that you'll replace.  Then use virtualbox command line tools to provision a raw disk vmdk and copy this over in place of the virtual disk that Fusion created.   That's it. It should boot fine. 
  • To trick Parallels, and note that I haven't found a way to do a raw disk vmdk like with Fusion, create a VM without a disk and enable efi.  You may want to remove Network from the boot order.  In the VM settings menu, attach the SD card to the VM.  Power up your VM and wait for it to drop you to the efi shell.  Then run:
    • fs0:
    • cd EFI\ubuntu
    • grubx64.efi
Getting the macbook to boot the SSD is another matter, and it's where I'll look at the actual Ubuntu build.  

Macbook Pro Dualboot Ubuntu SD Card & Virtual in OSX

This is just a quick post outlining how I finally was able to get ubuntu installed on an SD card that would boot from the option menu when I power on the Macbook and would also boot in either Fusion or Parallels in OSX.

For reference I'm on a Q4 2013 Macbook Pro 13" Retina with a SanDisk Extreme Pro 64GB SD card.  Not the fastest, but nearly the fastest.  I finally had success using VMWare Fusion Pro 7.0.  This may work with Parallels, which seems to have better EFI support, but has not been very stable for me.  This solution would not boot in Fusion 6.x.

  • Install the SD card in the laptop.
  • Use VirtualBox to create a .vmdk disk file that points to the SD card (/dev/disk2, in my case).  This effectively creates a "bootcamp" disk image to use with Parallels or Fusion. 
  • Download the rEFInd (the more recent version of rEFIt) usb stick disk image and dump it to the SD card:
    • Disk Manager to see what the device name is for the SD card.  It was /dev/disk2 for me. 
    • Use mount to see which partitions of the SD card are mounted:
      • mount | grep /dev/disk2
      • blah blah /dev/disk2s1...
    • Use diskutil to unmount:
      • diskutil unmount /dev/disk2s1
      • ...
    • Write the rEFInd image to the card:
      • dd if=rEFInd_blah.img of=/dev/disk2
    • Use diskutil info to take note of the name of the disk2s1 partition name. Something like ElTorito. 
  • Download your choice of linux distro.
  • Create a VM in VMWare Fusion and:
    • Attach a 20GB IDE hdd to it.
    • Attach the Linux install CD and set it as default boot. 
    • In the VM config directory, remove the 20G disk image vmdk files and rename the .vmdk that was created in VirtualBox to match. 
    • Add firmware="efi" to the vmx config file for the VM. 
  • Boot the new VM and when it loads into the Linux Live CD:
    • Dump /dev/disk2s1 (this is the 4.5MB partition that was created when we dumped the rEFInd image file to the card) to a tmp file:
      • dd if=/dev/sda1 of=./sda1.dd
    • Use gdisk (the GPT equivelant of fdisk) to:
      • Take note of the starting and ending sectors  and the partition type of the refind partition (sda1)
      • Wipe the GPT table (under expert optoins somewhere) and then re-open gdisk.
      • Create a new partition of for rEFInd using the same starting and ending sectors we used earlier and the same partitoin type. 
      • Set the name of this partition to the same as it was before (ElTorito?)
      • Create a linux partition using the remaining space.  You may want to leave some buffer space after the first partition in case you ever need to update refind and don't want to have to resize or rebuild the linux partition. 
      • save and exit gdisk
    • Restore our image of sda1 to the new partition we created along with the new GPT partition table:
      • dd if=./sda1.dd of=/dev/sda1
  • Reboot the VM with SD card as default and verify that it comes up in the rEFInd boot menu. 
  • Reboot the VM and boot to the CDROM again and start the install. 
    • I'd recommend using ext2 filesystem, since it doesn't use journaling and is fast, and no swap, to conserve writes to the SD card.  
    • Tell Ubuntu (or your distro) to install the bootloader to sda2!!! this is critical, so it doesn't bork your rEFInd bootloader.  
  • When the Linux install completes, reboot the VM to SD card and verify that rEFInd loads and that, from there, you can select the linux image and boot to it.  
  • if that works, then reboot the laptop and hold "option" to get the boot menu.  You should see the SD card and be able to boot Linux via rEFInd!
So Simple!    You don't want to know how many days of tinkering and repeated linux installs it took to find a setup that would work like this.  Fack!

Trying to boot to CD and install ubuntu directly to the SD card never worked for me, as the MAC bootloader wouldn't see the installation and present it in the boot menu when the laptop powers up, despite Ubuntu installing EFI boot junk on the SD card.  It wasn't too difficult to get the SD card to boot in EFI or legacy mode inside of a VM once figuring out how to do the bootcamp vmdk, which raises another important thing to note:

Fusion and Parallels are playthings only.  They really have no advanced tools included via command line or the standard UI.  There's no way to "bootcamp" to an SD card with them without hand-editing their config files and using 3rd party tools to create the disk image files.  VirtualBox is indispensable when manipulating VM images.  It comes with a bunch of great command line utilities for creating and managing disk images, and who knows what else.  I don't use it normally, though, because I used to run into issues with stability when I used it in Linux.  Meh... It worked great before Oracle bought it.  :P

Update:  Just saw the following, about creating raw disk iamges with vmware tools... might be helpful:

And this, which talks about using "tmpfs" to redirect IO-intensive filesystem activity to ramdisks and cut down on SD card activity:
He suggests:

tmpfs    /tmp    tmpfs    defaults,noatime,nosuid,size=100m    0 0
tmpfs    /var/tmp    tmpfs    defaults,noatime,nosuid,size=30m    0 0
tmpfs    /var/log    tmpfs    defaults,noatime,nosuid,mode=0755,size=100m    0 0
tmpfs    /var/run    tmpfs    defaults,noatime,nosuid,mode=0755,size=2m    0 0
tmpfs    /var/spool/mqueue    tmpfs    defaults,noatime,nosuid,mode=0700,gid=12,size=30m    0 0

Might be useful... but my system wouldn't boot with all of these enabled... will have to actually think about this and do some testing later.