Monday, June 6, 2011

A solution to speed up an old Droid

Is your Droid phone running so slow that you can barely answer a phone call before it rolls to voice mail? Yea that was my phone last week! I couldn't accomplish anything with my phone without long delays and that included just answering the phone! I have a venerable but, much loved Droid 1 and with 4 months remaining on the contract there was just no way I could "tough it out" at the current response time. Discussing the issue with a few Verizon reps got me the well worn mantra of the old Windows OS days, Reset (Just like Reinstall when you had Windows speed problems right?).

Fortunately, I found a better solution. I didn't want to invest the time into re-configuring and downloading everything on my phone, it's just the way I like it and I have better things to do such as working on my Game Programming or gaming with my son! I had freed up more memory then the average amount of free memory found on a Droid so I knew that the OS had enough space to do it's thing which lead me to suspect something was getting in the way of Androids ability to do things quickly. The first thing that came to mind? Cache. Nothing like a nice fat Cache full of useless junk to keep an old processor spinning its wheels right? Sure enough!

So my suggestion, before you reset your Android, clear the system Cache, it just might save you a couple hours!  Here's the surprisingly easy Steps (Getting into the System Menu is specific to the Droid 1, but the steps should be the same on other Droids)

1. Turn the Droid off, take the battery out then place it back in.

2. Open the keyboard and press the X key while pressing and holding Power. The device will power on. Keep holding the buttons down until a yellow triangle appears on the screen.

3. Press and hold the Volume Up and then also press Camera button button to get to the "Android System Recovery" Menu.

4. Use the direction pad to the right of the keyboard to select the "wipe cache partition"  (This is it right here!)

5. Once complete, use the direction pad to select the first option "reboot"


Believe it or not this did the trick! My phone has been running much faster, it's still a near two year old phone so its not as peppy as it once was but hey we all get a little bit slower with age...

Tuesday, April 26, 2011

Moving to Siena from Play Framework JPA

I Recently built a REST service using the Play Framework and once complete I started looking for a place to deploy the app. Not having a hosted server that runs Java meant I needed a cloud solution and the first to come to mind was GAE. I have used GAE before but this was the first time with Play. (Play and GAE go great together btw.)

Having done little deployment planning from the beginning, it wasnt until a blew through the coding and did a successful deploy that I realized that JPA just doesn't seem to work well with Play's JPA (nothing against Play, GAE just does JPA differently). After a bit of reading that I had skipped past the first time it became evident that I needed to use the Play Siena Module with my GAE app. This turned out to be easier then I though but, I had to hunt around a little to piece together the details so I wanted to share the migration here. This should also be helpful if you are planning to use Siena from the start and already know Play's JPA.

Moving to Siena from the Play JPA was surprisingly simple, I spent more time thinking over weather I wanted to mess with it and looking up the details then actually doing it!

1. Install Siena into Play. (I'm assuming you have Play set up and GAE already installed.)
play install siena-1.5

2. Include in App.conf. Add Siena right under the GAE module like so.

# ---- MODULES ----
module.gae=${play.path}/modules/gae-1.4
module.siena=${play.path}/modules/siena-1.5


3. Change your Models from play.db.jpa.Model to siena.Model. Your just changing an import statement here.

//import play.db.jpa.Model;
import siena.Model;


4. Add an ID field. Play takes care of this for you in the Model but you'll have to add it in for Siena, or make a new Object to inherit from.

  @Id
  public Long id;


5. Update method calls. A few methods have slightly different names but they are pretty obvious and should slow you down.
  1. Model.save() calls become Model.insert()
  2. You'll need to change up your searching, I usually just code methods in the Model to wrap the search queries which means I only have to change the search code in one spot, the Model. Here's a sample of a new search:
  3. Model.all(MonitoredFeed.class).filter("suspended", false).fetch();
  4. Change findAll(), all() calls to Model.All(Your.class) or simply provide a helper method like below.(This could again go in some type of BaseModel reducing your code changes.)
  5. public static Query all() {     return Model.all(MonitoredFeed.class); }
6. Update your Test Fixtures! Of course you have to update the Tests! If your using Fixtures you'll just have to change your imports from Fixture to SienaFixture.

That's all there its to it! Now you can deploy your app on GAE!

Saturday, April 16, 2011

Serve Up Static Content Even When Mapping /* to a FrontController Servlet.

I am blogging this to share the info as much as to remind myself in the future how simple setting this up really is. It's the easy solutions that can be the hardest to remember..

Say you have a RESTful services framework running in a web app and all of your requests get routed to CXF or some other Framework for supporting such implementations. Simply put, there isn't a UI for the app and the Controller Servlet does't support a UI. Now say that you really want to be able to serve up say.. a JS file and some css to style generated API documentation for your services. No problem since this is Tomcat except you have a /* url-pattern routing all traffic to a REST Servlet.

Here's the solution: Most Java servers have a default servlet of some sort which will simply handle HTTP requests. If you set up a url-pattern to /static or some other path you can direct that request to your static files. One note, you'll be requesting /static/js/jquery-min.js but, the fiel will reside at /js/jquery-min.js this is because the /static is really acting as a indicator to rout to the default servlet. It's almost like a query string in the path.. no it makes since, really.


<servlet-mapping>
    <servlet-name>default</servlet-name>
    <url-pattern>/static/*</url-pattern>
</servlet-mapping>

Thursday, April 14, 2011

Scripting Groovy to process csv files.

I have a lot of experience with um, "manhandling" flat file data into usable formats. Usually this "data" comes to me in the form of excel spreadsheets, crafted with presentation in mind. They look great but, have little flexibility when it comes to importing the data into relational databases. There are hundreds of solutions for doing this sort of manipulation in bulk and if that's your lot in life by all means beg the IT department to buy you a good data transformation tool!

For some reason my situations are frequently one offs of user data that is begging to get into a database but, as of yet is still running in the wild. My solution has come down to quick Groovy scripts and splicing spreed sheets into multiple CSV files Groovy to chew through. This can be done with Python and many other script languages but, I'm fully fluent in Java so Groovy is my fastest path to a database at this point in time. This is a great place to learn a new scripting language if you have the time but, I usually want this over in a flash. On that note, if you would like to just grab the code and run jump to the bottom to grab My CsvUtils code which takes care of this whole process in a more formal utility object.

Reading a file and working with it is pretty darn simple in Groovy, the following code will open up a CSV and move all the data into nicely structured arrays. (This provides the same results as calling CSVUtils.csvToList())

 //load room types from file
def f= new File('roomtypemap-min.csv');

//set up a List of 'Rows' which will contain a list of 'columns' or you can think of it as array[][]
roomTypes = [];
def lines = 0;
f.splitEachLine(',') {
    def row = [];
    row = it;
    roomTypes << row;
    lines++
}
println "Processed $lines Lines"

The above chewed through a CSV file with name,value pairs of data and placed them in a multi-dimensional array types as a Java list. If your not up on Java collections, it's a whole lot like an array but, better.

Now lets say we did the above code with a few files then process all that data into a new CSV file ready for your code. The code below shows the building of the final results. You'll notice that along with the roomTypes list imported from a CSV file there are now a few others resortList and amenities. All these were pulled from random table dumps and such and I turned them in to usable data.


def aId = 0;    
roomTypes.each{
  def indx = roomTypes.indexOf(it);
  def resortList = roomResortMap[indx];
  def roomType  = it[0];
  
  resortList.each() {
    def facIdx = it.toInteger();
    def facilityId = resorts[it.toInteger()];
    
    amenities.each() {
      def ai = amenities.indexOf(it);
      def avail = values[ai];
      if(avail[facIdx] == true) {
        def description = it;
        println "\"$facilityId\",$roomType,$aId,\"$description\"";
      aId++;
      }
    }
  }

There are a couple things to point out if your new to Groovy. While there is a whole lot of Java in Groovy it also looks a lot like JavaScript making Groovy pretty easy for JavaScript developers to pick up despite deep Java knowledge.

The first thing to note is the iteration I am performing with the .each callback. Personally, I think it looks a good bit like a JavaScript closure. Your simply passing in code to the .each(){} iterator to execute on each item in the List. OH the () is optional when creating this closure so add if you wish.

Second, in the version of Groovy I am using, 1.7.10, you have to do a little bit of work to get the current index position of the for loop. To do this simply call yourListName.indexOf(it). It looks like the upcoming 1.8 is going to have an additional closure called eachWithIndex() which will provide you it and i variables where the i will be the current index. I'm looking forward to that feature!

Finally I just want to point out one of my favorite things for keeping life simple, string replacement. Notice on the next to list line before the braces cascade I'm calling println but, the cool part is the replacements in the string using $fieldName. When your trying to print out complicated strings nothing is worse for readability and bugs then concatenating or calling tons of append() methods. It just leads to errors. You can do simple string replacement with variables using the $fieldName inside your string or for more complicated inserts ${} will work for things like ${Object.field}.

Hope this will help you to simplify the tedious task of data mining text files!

My CSVUtils Object:

/**
 * CSVUtils is released into the public domain, do what ya want with it.
 * By Lee Clarke
 */
public class CSVUtils {  
 
  /**   
   * Removed quotes from around imported csv values if present.
   */
  public static String stripQuotes(String strIn) {  
    def valOut = strIn;  
    if(valOut==null)  
      valOut == "";  
    if(valOut.startsWith("\""))   
      valOut = valOut.substring(1);  
    if(valOut.endsWith("\""))   
      valOut = valOut.substring(0,valOut.length()-1);  
    return valOut;  
  }
    
  /**   
   * Load the csv into a Map using indexed position of values for key and value. 
   * @param keyPos - col index of the key value in a row
   * @param valPos - pull value data from index/col position in row
   * @param csvFilePath - full path to file
   */  
  public static Map csvToMap(int keyPos, int valPos, String csvFilePath) {  
    def fp= new File(csvFilePath);  
    def rtnMap = [:];  
  
    def palines = 0;  
    fp.splitEachLine(',') {  
      if(palines > 0)//skip col header line  
      {  
        def row = [];  
                row = it;  
        def key = CSVUtils.stripQuotes( row[keyPos]);  
        def val = CSVUtils.stripQuotes( (valPos < 0)?row.last():row[valPos]);  
        rtnMap.putAt(key, val);  
      }  
      palines++  
    }  
    return rtnMap;  
  }  
  
  /**
   * loads values in csv file into multi-dimentional like List of rows and cols.
   * @param csvFilePath - full file path.
   * @param skipFirstRow - skip first row if it contains column names.
   */
  public static List csvToList( String csvFilePath, boolean skipFirstRow) {
    def fr= new File(csvFilePath);
    def rtnList = [];
    def rlines = 0;
    fr.splitEachLine(',') {
      if(!(skipFirstRow && rlines == 0)) {
        println "row= $it"
        def row = [];
        row = it;
        rtnList << row;
      }
      rlines++
    }
    return rtnList;
  }
  
  /**
   * loads values in csv file into multi-dimentional like List of rows and cols.
   * @param csvFilePath - full file path.
   */
  public static List csvToList( String csvFilePath) { 
    return csvToList(csvFilePath,true);
  }
}  

Friday, April 8, 2011

New hRecipeHelper for Web Browsers released!

Do you like to blog recipes but don't use Chrome? No problem! Now you can use my new web version of hRecipeHelper in any moder browser. I know it works with FireFox 3.6 and up as well as IE8, I can only assume it works with IE9 if IE8 works. I haven't tested on IE9 and Safari because I don't have them and can't even run IE9. If you trying it out and find bugs please post a comment here and I'll have it fixed asap.

Just like the Chrome Extension its free, Enjoy!

Friday, March 25, 2011

Validating your microformat pages

I posted some information about validating microformats over on the hRecipeHelper wiki but, I felt that it was worth a quick blog posting just to share with others that aren't likely to find a wiki page buried on Guthub!
I built a Chrome extension that will format recipes for blog postings using the hRecipe microformat and once done I wanted to test my results to make sure they were hRecipe friendly as well as to ensure that Google would find them tasty and digestible. To my surprise I did find a few very useful resources for the task.
The first handy resource is of course the microformat hRecipe where the specification is being defined. The format is a work in progress and currently I am supporting version 0.22 though I plan to continue to update the hRecipeHelper extension as the specification matures.
The second resource is a nice validation tool that Google has made available that will review your web page and try to parse out the hRecipe data as well as other microformats. It's simple to use, just paste any link into the form field on the page right here Rich Snippets Testing Tool and you'll see how it holds up. 
One cool thing about the testing tool is that not only will it validate your hRecipe formated pages but also many other microformats including hCard  which is the the markup spec for representing personal identification and hAtom, for blogs.
 Finally it you want to learn more about how Google interprets the hRecipe just take a look at their Recipe help page. If you cClick around a bit more and you'll find data element info for hCard as well. Happy microformatting, lets make a semantic web that's easier to discover and put to use!

Thursday, March 17, 2011

hRecipeHelper - new Chrome Extension

A little while back Google announced a new Recipe Search function that can search just recipes and then filter the results on various components in the recipe as well as cook time and calories. Now that's how you can rapidly dissolve the agonizing question of "What's for dinner?" into immediate action! To do this Google is relying on a nifty standard called microformats and specifically in this case the hRecipe format.

What hRecipe provides is a standard of coding HTML which reveals to bots or other programs the exact meaning of the web pages content, specifically the data values it contains. As a programmer who has written website scraping programs more then once in the past, this sort of formatting is a wonderful and accessible way to publish data. Unfortunately HTML is not exactly everyone's best skill, in fact I am willing to bet that most people don't know HTML and would very much like to avoid ever knowing HTML. Honestly, those people are perfectly reasonable, why should they?

I started thinking of people I know who are advanced computer users who simply don't have the time or interest in learning to write web pages and realized that while microformats are freaking awesome to a developer they are still a bit demanding to the cook that just wants to share her favorite recipes with the world. I saw this as a chance to give a little something back to the blogging world and perhaps selfishly get even more great recipes turning up in Google's search. I've written a nifty Chrome Extension called hRecipeHelper to help recipe bloggers easily post nice looking; hRecipe (and Google) compliant recipes by simply filling out a recipe card and copying the code into their favorite blogging site. All the site needs to do is support posting of basic HTML which Blogger, LiveJournal and most other blog platforms support.


Click to get it now!


I hope this will help get even more recipes on the web and further the microformats cause.

Tuesday, March 8, 2011

The Garden Droid 1.0 Update

I started thinking that if anyone wanted to know more about how my Garden Droid was working out they would be left wanting because I'm not posting the updates to this blog. My logic is that this is my Tech blog for programming and Electronics and since this project leans towards my more green and leafy interested I have posted the updates on my other blog. If your interested click on over and check out my other blog.

Yes,  much could be debated on the logic behind 2 blogs... I debate it  often but, hey it makes some logical since none the less so I stick with it.

Tuesday, February 1, 2011

The Garden Droid 1.0

Well after many months in the lab toiling away with a soldering iron and C compiler I have finally gotten the first version of my Arduino project done! OK really It just took months to get the time to finish the project but that doesn't sound as impressive really.. This is my first Arduino project and first really large electronics project so mistakes were made but overall I'm really happy that it all works!

I really  started the project way back in February of last year and a little over a week ago I planted the GardenDroids first crop, Spinach. I mostly chose spinach because I believe it will grow fairly well in the 75 degree room upstairs where the Droid has been parked as well as its ability to grow in partial sun which I am assuming is about what the LED lighting will produce.. tho I may be mistaking on that. We will see!

So what is a mini-greenhouse? Well in one sentence it's an electronically maintained greenhouse which can be maintained indoors and monitored from your web browser.
This is my mini-greenhouse which I am calling The Garden Droid (you know like garden gnome?)

What you see above is my Garden Droid which boasts a whopping 3 sq. ft. of well lit gardening space, soil moisture monitoring, temperature logging, LED grow lights and wireless transmission of data to my computer. Coming soon it will also be self watering, monitor humidity and ambient light as well as have an upgraded wireless protocol so it can transmit small pictures of the plant growth and support multiple Droids. 
Has I mentioned the website where I can monitor everything going on? Below is a quick snap of the web application which is used to monitor not just the Garden Droid plants but, your whole garden.


The whole project is open source and built on free software with open hardware that anyone can buy and build. I will be trickling out technical details and how to's in the following weeks for anyone interested and heck if my plants grow well I might even sell kits but, first let's see if I can get spinach to grow.

Part of my motivation for this project was to be able to grow salads year round but here in FL most of the summer is not kind to regular spinach and other leafy greens, its just too darn hot and all my efforts have been a failure. I figured that since I am keeping my house artificially cool all summer and the temps would be great for growing greens, why not give it  ago? I plan to eventually get this little box powered by a solar panel collecting dust int he garage as well then there will be a net gain of energy savings though just growing at home saves oil I would think.

Anyway I am really excited about version 1 coming together and I am looking forward to the next version with the enhanced capabilities. To check out the code jump over to my GitHub where you can see the code for the Arduino, the data logging deamon and the full web app.  I will be posting the real technical details just as soon as I can throw together some schematics and I'll keep you posted on the plants progress as well.

Thursday, January 13, 2011

Those crazy Sparkfun guys


Sparkfun is giving it away for free again this year! Last year their servers were so overloaded that I never got past the log-in page to try for the free goods, it was quite a bummer but a good lesson in server traffic. This year I am excited to say that I have managed to wait out the heavy server traffic and pick up a customer loyalty credit! :D I've been a member for 4 years so $40 worth of electronics gear will be great fun to have, I think I have an xbee mesh network coming up i my future, that should be fun when I add my GardenDroid and future home automation system all reporting back to my web server.

Thanks Sparkfun! [I'm sure I'll go over the 40... ;)]