Myriad Reflections

thoughts on software engineering and other interesting things

about me

software architect. engineer. manager. mentor. student. husband. father of four. somewhere between intj and infp. robotics mentor. singer. gamer. reader. tinkerer. gentleman farmer.

recent public projects

Status updating...

found on

My Architectural Philosophy

- - posted in architecture, philosophy, requirements, scale, startup

I was recently putting together a presentation to discuss the architecture of a system, and I found myself wanting to elaborate on the foundational principles that drove my decisions about technologies, deployment topology, domain modeling, service definition, APIs, and so on. The need to focus the discussion on concrete issues ultimately led me to cut those sections, but now I am compelled to write this blog post.

What is architecture in the context of software?

The architecture of software systems is difficult to define pithily. My favored definition comes from a classic paper that introduces the Perry-Wolf Architectural Model Foundations for the Study of Software Architecture.

Software Architecture = {Elements, Form, Rationale}

“Elements are either processing, data, or connecting elements. Form is defined in terms of the properties of, and the relationships among, the elements - that is, the constraints on the elements. The rationale provides the underlying basis for the architecture in terms of the system constraints, which most often derive from the system requirements.”

So much context is required to comprehend this definition that, beyond telling you to go read the paper, this becomes a jumping off point for a lengthy discussion.

It may be simpler to begin with what architecture is not. We do not typically talk about the architecture of functions or methods. So we can agree that architecture does not apply to programming in the small. It should however be noted that architecture is sometimes quite concerned with decisions made in the implementation of specific methods - such as when an algorithmic or data structure choice has an impact on system performance.

This leads back to the first part of the Perry-Wolf definition: the elements in an architecture may be data, processing, or connecting. That sounds like data structures, algorithms, and interfaces (whether they be simple method calls or API endpoints). The elements are where architecture and construction meet. Ultimately we must be able to implement architectural ideas lest we become what Joel Spolsky derisively calls architecture astronauts. There is no doubt there is a danger of attempting to solve a problem that is too abstract and too far removed from actual domain requirements, and I will address the goodness of architecture in the next section.

The form of an architecture is concerned with how elements are organized. My preferred approach to system architecture is to first focus exclusively on the problem domain and produce a Domain Architecture that defines a set of elements without regard to technology. Those elements represent the business domain and have names that the business stakeholders would use when discussing the problems we are attempting to solve. As we add the technology concerns, the organization of the domain objects into components, subsystems, services, etc. produce a form of connected elements to satisfy both the domain requirements and the technical nonfunctional requirements.

The need to balance constraints using judgment based on experience leads to different architects producing different solutions. That is, the elements and form will vary based on an architect’s biases. Two different architectures can both be good, but it is critical that an architect provide the rationale behind the set of decisions leading to a given solution. This is essential to engage the business stakeholders as partners in making the various trade-offs that must be balanced in an efficiently designed system.

What makes an architecture good?

The short answer is that a good architecture balances all of the key system constraints. These include the functional requirements, but also the nonfunctional requirements including budget, availability, scalability, maintainability, and security. Additionally, the need to satisfy regulatory or industry rules can promote some requirements from nonfunctional into functional, as noncompliance can mean significant penalties for the business!

For modern internet-facing web services a typical set of nonfunctional requirements would include ensuring high availability by the use of redundant web/app-tier servers with a replicated database tier. For an early stage startup still looking for product-market fit, this is probably sufficient to ensure 99.99% availability. That may mean triple the cost of a single-node deployment, however, so this is a good example of a business trade-off that needs to be discussed. Is the extra 0.09% availability worth the impact to the burn rate? My bias is toward the additional availability. For most deployments on Amazon AWS today, the triple cost amounts to $200 - $300 additional per month. Hopefully that is a negligible sum for a properly capitalized startup!

Here too we must recognize that the goodness of the architecture may only be defined in the business context. If the business will happily accept minutes or hours of downtime as a new server is spun up to replace a failed one, then the extra cost of the highly available configuration represents resources that may be better spent elsewhere.

To my thinking this is a contrived example, as there are other important factors to be considered - such as the fact that the highly available cluster may be scaled out to many web nodes in minutes. A single node configuration would be much more troublesome at responding to a traffic spike scenario.

Rationale

But the central point is valid. Ultimately the architect’s job is to communicate trade-offs with all of the stakeholders and design a system that balances all of the requirements. Communicating those whys is an essential part of creating a shared metaphor for what the system should be from the technical and domain perspectives. This is especially critical on agile teams where user stories represent thin slices of functionality developed in isolation. That shared metaphor is what fills in the gaps as the team works through the backlog to combine those slices into an effective solution.

Mobile VOIP Part 2

- -

It has been a month now without AT&T 3G service on my iPhone 4, and I have not missed it even a little bit. The few times I have been out and about and needed to make a call the Clear Wireless mobile hotspot has worked like a champ!

It seems like I get less time to answer a call before it goes to voicemail - sometimes Skype will only ring once before the caller gets ungraciously dispatched to recording land. But I never have any trouble calling them right back - so I am willing to check off that weakness and then think about the massive reduction in my mobile costs! :)

Once I can get my bluetooth headset working with Skype - everything will be perfect.

An additional note on the 4G hotspot - when out and about it works very well, but in the house or in my office it does not do so great. I know they offer a different device for home use (which presumably can handle weaker signals better), but for now we are sticking with Cable for the home internet. That caps our savings at around $80/month for now. And that feels really good!

Pushing the Envelope on Mobile VOIP

- -

I just took the plunge - and took my family with me - into the wild west of mobile voice. We just abandoned our iPhone cellular plan in favor of turning these wonderful devices into fat iPod Touches, and it looks like we will save at least $80 per month on our family cellular bill.

The current primary experiment is to use Skype over wifi at home and the office. That works well. The two numbers are $30/year each with a Skype Premium plan. That works out to $11/month for 2 accounts.

We also added a Clear Wireless mobile hotspot that provides 4G and 3G connectivity for cellular data. Early experiments with that have worked well too. This service is running $55/month + $9/month for the personal wifi hotspot. We could save a lot here if we went 4G only, but one of our requirements was to be able to make calls over 3G when we travel around the country.

That adds up to about $75/month it seems (not clear on the tax differences yet - though it seems we will avoid the many taxes for USF and 911 service).

Total Savings: about $80 per month!

If the 4G data rates are good and stable, we will be looking at replacing the home cable modem with a second service from Clear. They need to prove themselves at our location first. And the additional savings from that switch would probably only amount to about $20/month. Nothing to sneeze at for sure, but hardly the dramatic budgetary axing we are getting with cell service!

There are certainly glitches. I tried several times this morning to call a work colleague over a weak wifi connection where only one of us could hear the other. Once I got things setup to connect to a strong hotspot all went well.

We have to make sure to add the local police/fire/hospital to our contacts - just like when I was a kid - since you don’t get 911 access on a setup like this.

I have to say though that the voice quality - when it works right - is spectacular.

The solutions we are experimenting with so far in addition to Skype are Google Voice and Pinger.

I am using Google Voice to handle text messages with Skype, since for some reason they don’t offer any kind of unlimited SMS plan and charge over 11 cents per message. Seems bizarre to me.

Pinger gets kudos for free SMS and Voice sponsored by ads, jumping through hoops, or cold, hard cash. I still have 100 Pinger minutes I need to use up, so I will experiment with them some more most likely.

So far though, Skype gets the nod for best voice quality and overall call experience.

What are your experiences with internet-based calling?

Research and Writings

- -

This will be an evolving post providing summary info about software development topics of interest to me. I will include links to papers written by me, but most of the others are IEEE or ACM journal articles that cannot be freely distributed. I will strive to provide links to the abstracts where possible.

The process of building software systems involves many different activities, but it seems to me that the rubber meets the road with the architecture. This coarse, componetized view of the system is where the requirements first run into an objective measure. Yet it is the architecture itself that must be evaluated with respect to the requirements. The artifacts produced by the architect often anchor communication between the stakeholders on a project, and they will serve as the entry point for team members to begin understanding the system.

I will add some more detail on each of these aspects soon: Systems Engineering Process Activities (SEPA), ATAM, Evaluating, Software Development Process/Lifecycle, Tool for Communication

 

My graduate work focused on a couple of key architectural components - the construction of a Domain Object Model (this is akin to the Domain Reference Architecture in the SEPA process) from a set of requirements elicitation interviews, and the evaluation of that model against standards of completeness and correctness.

Thesis: Establishing the Completeness and Correctness of a Domain Object Model

Another very interesting subject area for me was Data Mining. Here I used tools like Weka (now absorbed into the open-source Pentaho suite) to extract novel, actionable knowledge from data warehouses. I had an opportunity there to collaborate on an interesting case-study using convenience store data.

KDD (Knowledge Discovery in Databases)/Data Mining: Knowledge Mining Convenience Store Sales Data

Software testing generates a great deal of zealous discussion among developers. 100% code coverage, black box, white box, mutation detection, unit, functional, acceptance, test driven development (TDD), integration, system - all of these aspects of testing software must be stacked up against the businesss value delivered. What we need is a good return on investment for the time and effort spent writing, executing, and maintaining the tests. My ideas on the subject have evolved quite a bit over the years. One interesting project I worked on involved implementing the Korat unit testing algorithm with a GUI to allow incremental improvement of the key evaluation routine. Interestingly enough, I later came across a commercial product (Agitar) that did a very nice job with a similar solution. There is no Korat in the mix (as far as I know), but it does a nice job of allowing the developer to manage the state-space explosion problem nonetheless.

Verification and Validation of Software Systems: Korat Visual Finitization Editor

More to come…

A Brief History of Mytoons

- -

Since Mytoons.com has been offline for some time now, I think it would be nice to archive some of the key features of the site and its related applications here.

Way back in the summer of 2006 the founders of Bauhaus Software decided it was time to move beyond the animation tools market and deliver a new capability for animators to showcase their work, share it with the world, and potentially earn something for their efforts. I was hired that August as VP of Web Engineering to build this vision into a working product/platform.

From the outset the founders identified a threefold strategy for growing the business that could be summed up as create-share-sell. We would provide tools with which animation students could create, provide a web portfolio site for them to share their work, and provide an ecommerce platform they could use to monetize that work.

This ambitious vision was initially tackled with a programming team of two and a talented graphic designer. The other programmer was a gifted recent college grad named Jason Sims, who has since gone on to do a variety of freelance work. The designer (who also dove into some of the front-end coding work) was Franklin Lyons, who has also gone freelance and produced some nice work.

Jason and Frank were the first, but I also want to acknowledge the great contributions of the other developers on the engineering team over the couple of years that we built the Mytoons applications.

  • Paul Dobbins - html/css/js/ruby/rails coder extraordinaire
  • Todd Williams - puts the engineer in software engineering, smart and gets it done - whatever the job
  • Josh Helpert - Adobe Flash & ActionScript, helped implement our flashy vision
  • Sam Bagot - animation tools Java coder, another smart and gets it done developer, deserves a medal for enduring the animation tools project :)
  • Jinkuk Nahidi - ruby/rails - front to back - great coder all around
  • Mark Mayfield - world-class object modeler, made our e-commerce and catalog system totally awesome
  • James Jennings - 3D modeler, QA/tester, and the fastest keyboard/mouse in the west
  • Jenné Leigh (née Speegle) - QA/tester, learned it as we went, doc'ed lots of scenarios and created/ran lots of automated Selenium tests for us
  • Jorge Miramontes - built our Facebook app and tied it to our ruby/rails system

The following screenshot is of one of the last revisions of the Mytoons home page. I have been searching for an early mockup to contrast with it, but I have yet to lay my hands on one. Note the HD resolution option. We were ahead of the curve where video quality was concerned.

From the outset there were many facets to the development of the site. I spent quite a bit of time in the early days deciding on the solutions for video transcoding, DRM/Copy Protection, development processes, the physical and logical deployment topologies, licensing and copyright issues, and a thousand other things. I think a future writeup of the post-mortem analysis would be interesting, but for this first article on the Mytoons history I want to focus on the major features of the site.

The Digital Portfolio System evolved into a very nice widget-based CMS for animators to showcase their work with all of the flair and attention to detail that an artist requires. The following shot is a page in edit mode allowing the animator to select and edit widgets for the page.

The same page in view mode as the world would see it. Animators had a direct link into their personal store catalog if they had animations for sale.

This is a list page that illustrates how store content and free content were mixed on the site. An animation fan could browse and watch for free or purchase the premium content for offline viewing.

Another example of the premium and free blend - this is a play page where a user would have just finished watching a free animation that had associated premium content. The Flash player and the page content both direct the fan to the store for additional content.

We held a contest to kick off our launch of HD Animation in June of 2008. The winner got a trip to New York and got to see their animation play in Times Square. This picture represents the achievement of the Mytoons vision - a talented animator got to showcase their work and gain some exposure.

Allan Dye’s account of his winning trip is available here:

https://allandye.blogspot.com/2008/11/mytoons-get-with-times-trip-to-ny.html

This is the ecommerce pipeline in action. Revenue at last!

More revenue…

The Digital Safe is where you went to grab all of your purchases. Animations sold through the mytoons store were available in a variety of formats from 3gp to iPhone to 1080P HD. Once you purchased a title, you could at your leisure download it whenever you liked.

There are many good stories in the history of Mytoons - including several of the sprints we executed to deliver the HD and store milestones, the NDB Cluster fiasco, searching with ferret/solr/google, melting databases, sneaky ads wreaking havoc, caching, outages, and more. I will try to hit the high points in another post.

Getting the Songs Back Online

- -

Back in the good old days when I seemed to have more spare time for that sort of thing I played with selling some mp3 and CD recordings of multi-track hymns I recorded.

Well, crocketthopper.com has been defunct for a couple of years now, and I don’t see myself ever getting back to attempting to sell the music - so here it is:

Free for your listening pleasure

oh-jesus-my-savior

tallis-evening-hymn

return-oh-god

oh-tell-me-no-more-of-this-worlds-vain-store

supplication

samanthra

almighty-king-whose-wondrous-hand

The Curse of Object-Relational Mapping

- -

I had a professor in my software engineering program at UT who said that “Code is the desiccated relic of a long intellectual process.” I have never heard it put better. I am a big believer in the “Say it in code” principle - that your code should be simple and readable enough for others to understand (or at least yourself when you look at it again a year later and wonder “Who wrote that mess?”). Experience tells me this is not sufficient. Object-Oriented Analysis, Design, & Programming (OOA,OOD,OOP,OOAD,OOADP, or OOADD for those with short attention spans) has proven itself to me as the best available technique for maintaining the original intent behind code - after all, the behaviors and the state live together in modules that encompass a domain concept - if we have done our jobs.

OOADP also gives us more tools in our toolboxes for tackling complexity problems - if you look at the classic Object Oriented Analysis you can see that while functional, procedural, and data-flow techniques are suitable for certain classes of problems, an OOADP approach encompasses those and adds flexibility - especially in complex interactions of part-whole, is_a, has_a, and acts_like_a domain relationships.

By the way - the most under-read book on OOP is Streamlined Object Modeling.

Streamlined gets us the rest of the way there on being able to look at a system years later and understand what is going on. The appropriately-named DIAPER process “catches all the crap” and the whole book presents a dozen patterns (most of you probably think of these as analysis patterns) based on about a hundred principles - along with sample code for Java and Smalltalk.

I think the beauty of Streamlined is that by taking some decisions away (constraining our design and coding to the dozen patterns), we are freed to focus on the problem domain. It helps by reducing the arbitrary complexity that Brooks bemoans in The Mythical Man Month - because when you go look at a Streamlined object you know where to find stuff and you can quickly grasp how it interacts with other objects to get the job done.

And all is well with the world. Well, not quite. You see, in the real world these days - even if you are cool like us and write Ruby on Rails Web Applications - most of your system state resides in a relational database. Sigh. The Object Relational Mapping problem has served as some great fodder for theses and dissertations - not to mention lots of consulting dollars. Because at the end of the day all of that extra power I was bragging about object-think having over data-flow or functional or procedural or whatever older programming technique your technology stack forced upon you - all of that gets diced to shreds when you start to bolt your system onto the relational database.

Inheritance is a problem for databases. Single-Table or Multi-Table - you want to be able to collect, select, inject on your sets of objects without doing an 8-way-join? Sorry. Cost of doing business. Cache your app like mad and keep going.

OK. That works fine for toy apps. And I good-naturedly include any app where the domain complexity does not overwhelm the technical issues - even if you are generating a billion hits-per-second or whatever you lucky dogs are doing these days ;) no jealousy at all there friends!

What I am trying to get at here is that even with a powerful abstraction like Active Record (see Martin Fowler’s book Patterns of Enterprise Application Architecture for the general case and Ruby on Rails for an incredible implementation) - any time we have to think about the fact that there is a database under there it detracts from the ability of the development team to maintain momentum on new features and the evolution of the system as a whole.

The nature of a business application is to change - I cannot recall the name of the classic paper - but business apps were categorized a long time ago as a system which is not simply simulating the world - it simulates some things sure - but it also changes the world as soon as it is released. A good business application provides the business with capabilities it did not have before. And as soon as a business has new capabilities it will capitalize on them - you guessed it - there will be immense pressure to change the system to provide new capabilites built upon those just created.

That is a good summary I think of why software is a lucrative field and why thinking practicioners of the software art need never fear running out of cool work to do. Each time we innovate for our customers or our employers (they are our customers too after all) - every time a new system leaves the abnormal state of the lab and starts breathing in the real world, it better be solving some key business problem. Then those customers are going to look around and say “Wow! If we can do this, then what about …”.

So we need to stay fast and agile - ready to respond to those wonderful changing business requirements that let us know the system is still alive and still being used to do stuff. And a relational database makes that problem a little harder.

Am I just ranting? Mostly. But there is hope for us OO purists. Seaside can run on Gemstone - an object store that persists state across servers and on disk (the way most of us use the database) without having to wrestle with the ORM problem. And I have long had a desire to really learn the Smalltalk way of doing things - but so far I have not gotten around to it. So what really melts my butter these days is Maglev - a project to make the Gemstone architecture support Ruby (or a project to make Ruby run on the Gemstone architecture if you like).

It bugs me that the Rails community is not talking about it more of late - my impression is that since it is not open and free (yes, Gemstone is made by a real company and they stubbornly wish to keep the system proprietary) that the community wants to ignore it. I have been scouring the web for press on the RubyConf2008 talk about Maglev, and I have found only a couple of tweets! I am waiting with bated breath for Confreaks to release their videos so I can finally see what I missed.

I think we can at least talk about the technology. Gemstone does have a free license for many classes of application. And for those of us with big systems like mytoons.com - we’ll just have to negotiate :)

Finally a little encouragement to the Maglev team themselves - lets get some blogs going and an alpha or a beta out there to experiment with. I am ready to turn ActiveRecord into a no-op! Well, at least an abstraction where I don’t have to worry about inadvertantly introducing a database crippling query into the mix by asking my object model an interesting question. Cheers for turtles all the way down!

About the Blog

- -

Greetings friends, family, cohorts, peers, etcetera, etcetera, etcetera!

I expect the focus of this blog will evolve over time - but for the moment I plan for this public diary to span personal, political, and professional topics ranging from Software Engineering to liberty to the value of human life - body, soul, and spirit.

I have read that most blogs are read by at most a dozen people - so I will approach this public/private endeavor with an expectation that most of my readers are already acquainted with me in some way. Clay Shirkey makes some great observations about the weirdness of this phenomenon in his book Here Comes Everybody - writing a blog is like having a conversation in the food court of a mall - it is sort of private, but eavesdroppers abound. In the blogosphere however, this behavior is encouraged :)